Whether or not it turns out to be statistically significant is another matter, but it's a good visual to show the difference between the two operating modes that caused me to post the findings.

...

I hadn't really studied the above graphs when I posted them, but the data within them is actually quite interesting.

That's why I like charts. The human brain can assimilate information very quickly from images. As I mentioned earlier, the easiest way to check whether the samples are statistically different is to plot the 95% confidence intervals for the regression line and checking whether they overlap or not. This can be done both in R and in python. Here's my infamous 2025 setback vs no setback energy in vs OAT data plotted in R with 95% confidence intervals:

It appears to show the daily energy use is significantly lower when running a setback, at least at lower OATS, but I think the data may be biased. For a start, spring is not the same as Autumn and winter. Other unknown unknowns could also be in play. This in turn led me to the idea of using 'matched pairs' ie match setback days to non-setback days where the variables we do know about are matched. For a start, that means from the same season. I already have the Spring 2025 data, which was a period of setback running, and I am still collecting this Spring's data, which is a period on non-setback running. Once I have all the data (around about the end of April) I will do the plots and see what appears.

The oversizing issue, as shown via the heat loss data graph earlier in the thread, is largely a MCS modelling issue, consistently applied by several heat loss surveys from a number of providers.  I pushed back on the calculations as much as I felt able during the quotation process to try to mitigate any potential oversizing, but when you are presented with multiple assessments telling you broadly the same thing, there is a point where you have to rely on their information.  It is only with the benefit of analysing actual post-installation data that I can quantify the potential scale of error in the heat loss assessment.

Finally, just to repeat my usual caveat, the most important stat for me is the cost of running figure, and the system has delivered cost savings of 40-50% to date compared to the gas boiler system it replaced, despite the very modest stats, thanks to my ability to integrate it with the existing PV and battery system via the Octopus Intelligent Go tariff.  Fundamentally, that is why my conclusion as to fitness for purpose for my own system is very different to yours.

I hope that you are able to satisfactorily resolve your system issues in due course.

I didn't have those graphs but they were easy enough to create.  I've done one for the tracking period (currently 102 days) along with separate graphs for the weather comp running (45 days) and the auto adaptive (57 days).  I wasn't sure how compelling they would be, but there's a distinct difference in what they show, with the weather comp data being broadly aligned, as for the output graphs.  By comparison, the Auto Adaptive data has a much more noticeable difference between the two lines, such that you can actually visualise the point of intersection.

Whether or not it turns out to be statistically significant is another matter, but it's a good visual to show the difference between the two operating modes that caused me to post the findings.

Quick note: R squared is not the correlation coefficient, r, but that value squared, and it tells us the extent to which the independent variable explains the dependent variable. High R squared, and the data points will be close to the line, lower R squared and they will be more spread out. If all the points are on the line, ie the independent variable 100% explains the dependent variable, then R squared will be 1, as happens for the red data and regression line on your chart.  

Can you do a similar chart for daily energy in vs mean daily OAT, comparing emoncms data with Ecodan data?        

NB: (and something of a digression for this thread) The main issue identified, and I suspect this is endemic across the industry, is how different the actual heat loss data is compared to the heat loss assessment done during the installation.  Short of appointing @jamespa as Government Tsar for this area of MCS compliance, the methodology for these calculations is always likely to produce such anomalies.  In my case, I had several heat loss surveys carried out by various providers, all of which were consistently in the 8-10kW range, so persuading any of them to rely on other data to reduce their heat pump sizing recommendations would have been a challenge, particularly from the position of being a potential purchaser of a heat pump system and therefore being somewhat reliant on what you are being consistently told by various heat-loss assessments.

For the purposes of the query raised by @ecoste on his own installation, the main point of sharing this data is to attempt to quantify the scale of any error in the output data recording, which the two regression line formulae could potentially be used to assess.  Relative to the issues with input measurement, and the general uncertainty in being able to accurately verify output data, I would suggest that this is a lesser source of overall concern.  As in my case, the true heat loss relative to the assessed one might be indicative of potential heat pump oversizing, but those inefficiencies are separate to the ones he's trying to demonstrate on the basis of the heat pump's measured performance vs specification.

This chart makes me suspicious, because of the data spread. The regression line looks off to me as well. Assuming the Y axis is the daily mean OAT, then look at the range of X axis values, which I assume is the daily energy produced, for a mean OAT of around 9°, which shows a spread from about 60kWh to 100 kWh. Why is there such variation?

I also am not too confident.  There is not much data and the resolution of the data is poor.  However there are other reasons to suspect the house loss is actually more like 7 than 10kW. 

However one thing linked to this is clear, and I say it above, namely

With a fairly high degree of certainty one of two things is the case namely 

a) the actual loss is quite a lot less than the surveyed loss   or

b) the sensors are underrepresenting the delivered energy

Which of these we dont know, but care is needed in interpreting results without knowing

Otherwise I agree with everything you say (not that the above is disagreeing, more like clarifying).

Here is a plot of OAT vs total energy produced, suggesting a house loss of maybe 7kW at -2.  Given that it was ticking along at 3.5-4kW at 2-4, I would say that this may well be about right, but more data needed to be sure.  If so it may well be oversized, therefore running at a relatively low efficiency point on the compressor efficiency curve.

This chart makes me suspicious, because of the data spread. The regression line looks off to me as well. Assuming the Y axis is the daily mean OAT, then look at the range of X axis values, which I assume is the daily energy produced, for a mean OAT of around 9°, which shows a spread from about 60kWh to 100 kWh. Why is there such variation?

Thank you for the reply. I recognise that the data that I am using is not very accurate but it is good enough to show that the efficiency of the heat pump is poor.

It doesn't need to be very inaccurate to be misleading. Given the variability in your data, your COP estimates could be way out.

I can try refining the measurement accuracy, it will take time and cost, but it is just going to more accurately tell me that the performance is poor and I still won't know why.

I don't think you know performance is poor. It is possible your current data is far enough out to mean it is effectively GIGO. To know whether your performance is a poor as you believe it is, you need more accurate monitoring.

I don't know how to get to a clear understanding of the cause.

See above (in this comment and @jamespa's previous comments): more accurate monitoring.

My heating cost is about £1,500 per year therefore the extra due to the lack of efficiency is about £500 per year.

There are other factors to consider. Have you gone from timed heating with gas to always (or almost always) on with the heat pump? This can increase costs because, in short, the property stays warmer for longer. 

But really it all comes back to the same thing, to make confident assertions about efficiency you need accurate energy in and energy out data. As previously noted, even a small error in say delta t of 0.5° for a delta t of 2.5 is still a large percentage error, and that means the derived energy out figure will be way out, more than enough to have a major impact on COP.        

Thank you, I have validated the temperature sensors and found them to be accurate. 

As a matter of interest how and to what level are they accurate?  What would be the deviation between the reading of the two sensors if placed adjacent on the same pipe so they are at the same temperature. 

I am interested in suggestion c) that there are specific conditions under which the specification is measured (eg flow rate) and yours are, for some reason, materially different, but unfortunately can not think of any cause and hence any test

The most obvious ones are flow rate and deltaT.  The heat transfer across the internal heat exchanger will likely be most efficient when rad deltaT (and thus deltaT across the heat exchanger) is fairly high (which of course has the opposite effect on system efficiency - ie it makes it lower).  This typically wont occur (in a real system) at low load (where you are mostly operating), but they might measure it that way eg at a standard deltaT which is not load dependent.  They might even measure it at high/standard flow rate and high deltaT which again wont happen in a real life system but is the sort of condition one might standardise on for doing the measurement.  I dont know if any of the international/national standards specify this.

b) that the heat pump doesn't meet its specification, remains the obvious choice Mitsubishi will not accept this suggestion and I don't know how to get irrefutable evidence to support this. Is there a third party organisation that could provide this?

There are independent test houses that the manufacturers use, but the cost will be exorbitant.  Otherwise its diligent data collection on your part which would need thinking through very carefully and some questions to Mitsubishi about their test conditions.

There is some discussion about the limitations of Mitsubishi energy monitoring here  You should read this.  

This is a very interesting topic.  Heat pump manufacturers have the easy get out of 'system configuration' but in reality you can (as you are doing) measure the COP of the heat pump alone.  Provided it is running for a reasonable amount of time, the only influences (so far as I can see) of 'system configuration' on this are the flow temps, flow rates, OATs and deltaT flow to return (exception - around freezing point OAT humidity has an influence).  Since they give figures for a variety of flow temps and OATs, you can take this out leaving only flow rate and deltaT.  Provided these are reasonable I cant see a reason why the hardware would not be expected to perform to spec.