@cathoderay

Could you please provide a detailed description of how you arrived at the constituent parts of this formula:-

y = 0.0093x2 - 0.3911x + 4.3461

@cathoderay From your Obs vs Exp worksheet here is a pivot of actual and expected for the whole of the heating season (I omitted the days when the heating was obviously off).

Note that the delta (expected-actual) is almost all negative prior 19/10 and almost all positive after 19/10.  You advise us that setback commenced on 4/11 so that's not the cause.  It rather looks like something else changed on 19/10 to disturb the profile and reduce your consumption (real or apparent) at any given OAT.

@derek-m - it's in the post (and is repeated in a reply to @jamespa I am about to post). In short, it is the equation given by the spreadsheet for the cleaned up trend line, see sheet "OAT-E_in" in the spreadsheet.

Posted by: @jamespa

↑

The ratio between the two is 1.18, which is obviously more than a little suspicious

That potentially makes the difference between expected and actual even bigger

When a specific ratio mysteriously reappears something is usually not right! Now i think I might know what it is, and it is the old gremlin of having too much data in too many places. I think I may well have applied the correction factor twice. If so, it is my fault for doing the correction in situ, ie the result ends up in the same column as the original. I do this to reduce the number of columns (a good thing) but it hides whether the correction has been done (a bad thing). I should be able to check, and if necessary correct, this, by looking at past backups. You may recall a short while ago I was in a flap about how and when to apply the correction to live data, this sort of thing (did I or didn't I?) is the reason for being in a flap!

If I did apply the 1.18 correction factor twice, then as you say removing it will make the difference between observed and expected even greater, perhaps to the point where even I will find my credulity strained. Savings of around 20% do not necessarily have to be absurd, given that a six hour setback means the heat pump is off for 25% of a 24 hour period.      

Posted by: @jamespa

↑

In your pivot table how did you get the data to display by date rather than by date and hour.  I cant seem to reproduce that.

I don't know, it just happened. According to 'Format Cells' they are formatted as 'Custom' with the pattern 'dd/mm/yyyy hh:mm' but obviously they are not appearing with that format. I think it is possibly the 'Group by > Day' that is responsible, because after doing that there are no hours, only days, and before the 'Group by' they did look like dates, and if I undo the grouping, they go back to being dates, with an underlying date number. Even weirder, if I copy and paste special as value one of the grouped dates, I get a string rather than the underlying date number, suggesting the spreadsheet doesn't think it is a date, even though the underling ungrouped column is a date! As I have said before, I find spreadsheets to be very peculiar when it comes to dates (and python is even worse!). Add a Pivot table into the mix and it seems mysterious things happen. I'm not that bothered though, as long as the string is the right string for the date in question (ie 4 Nov appears where it should etc), even if part of me would like an explanation of what is going on.

Posted by: @jamespa

↑

Also can you explain how you got the trendline.  The Mk1 eyeball method almost certainly ignores overlapping points, particularly those on the axis (heating energy =  zero).  But at least some of these these matter because there are quite a few hours when the heating energy is zero (or reduced) because there is DHW heating going on or for other reasons, and these need to be counted otherwise the expected energy might be overestimated.  If I get excel to put an order 2 trendline on the data, it falls a long way below the 'eyeball' trendline, but I grant that this may include zero values that should be excluded.

I used the Mk 1 Eyeball to deal with the problem you mention (including zeros in the trendline making it nonsensical - I get a U shaped curve...). The zeros (no energy in) are there because I used all the data including the summer months (couldn't be bothered to sort it and remove the zeros, again there is a small hidden virtue, less meddling), however it is only heating, as I use the heating, not all energy in, column. The steps are: initial plot of all OAT vs heating energy in data, visually.manually add the Mk 1 trendline, read off the energy in for each degree OAT, make a second plot of those OAT.energy in readings, which is now a 'cleaned' plotm visually confirm it looks right, and if it does, which it did, then add a trend line to that cleaned plot, and get the equation for the line. I'm not too bothered overlapping points because I think for each OAT they are normally distributed, or at least near enough (see the 10 degree OAT histogram) and so the shape of the data stays the same, with or without overlapping points. This is a key part of my logic: if my assumption that the energy in for each OAT is normally distributed, then the mean does have meaning, ie it really is the average, and the variance will be spread roughly equally on either side. Individual expected hourly values may be too high or to low, but the aggregate of many expected values over time will approach the mean. I suppose in effect I'm making use of regression to the mean.      

Posted by: @jamespa

↑

Also, using the Mk1 eyeball average, your IAT during the setback period was typically about 19.5C whereas during October it was more like 20-20.5 for quite a lot of the time.  That is not insignificant and is, for laudable simplicity, ignored in your calculation.  We don't know what the IAT was in March/April but the data is included in the trendline.

As I said before, I don't directly use the IAT at all in the calculations, instead I only use a visual assessment of the IAT over the setback period to check the net energy balance remains constant, which it does, apart from the brief previously noted start and end exceptions. The only calculations used are amps x volts ( x 1.18) to get the energy in, and then the expected energy in is calculated from the OAT using the trend line equation, and that is it, and the trend line equation only 'knows' about the OAT and energy in. That said, I do agree that if the OAT vs energy in trend line also includes periods in which the IAT is higher (or for that matter lower) than the 'standard' IAT (19 degrees), then the line will be shifted somewhat. But I am not persuaded it is shifted by a huge amount: periods when the IAT was a bit higher will be balanced by periods when the IAT was a bit lower. In fact, I do have IAT data for March and April, but it comes from the standalone RC4 data logger which (a) is in a (slightly) different place to the current MD02 sensor (b) doesn't always give the same reading as the MD02 sensor and (c) it records hourly data (point or average?) at 30 minutes 38 seconds past the hour which makes it very awkward to add it to the modbus data (needs some sort of horizontal inner join on 'datetime like' which I have not worked out how to do). That said, the average IAT from the RC4 data logger for the period 22 Mar 2023 to the 3rd April 2023 (last data download I did) is 19.7 degrees C, while the MD02 average over the setback period is actually 18.2 degrees C, pulled down as it was by the setbacks. But, crucially, what we are interested in is not the actual hourly or average IAT with a setback (which we expect to be lower because of the setback), but what the expected energy in would be without a setback (and so a higher average IAT). The point is all that matters is the average IAT stays constant over longer setback time frames, the actual value (which we expect to be lower) doesn't matter, as long as it doesn't go so low as to be absurd/compromise comfort.             

In short (I haven't checked this, and perhaps I should) I suspect the average trend line based on all the OAT/energy in data probably represents the OAT/energy in relationship when the IAT is at or just above 19 degrees, ie very close to the desired IAT. Over time, as more data comes in, the accuracy of this line in representing the true OAT/energy in relationship should improve.  

Posted by: @jamespa

↑

As I say above the difference you appear to show is, I think, too great to be explained only by savings in losses from the house during setback.  That being the case I also believe that we need to look for other explanations to get a fuller picture of whats going on.

I agree, this is still very much work in progress, and I have never claimed that my current method represents the definitive answer, it is just another way of tackling the question that needs further work to improve it (or bin it, if it turns out it has a fatal flaw).

Thanks for the responses.

Posted by: @cathoderay

↑

I agree, this is still very much work in progress, and I have never claimed that my current method represents the definitive answer, it is just another way of tackling the question that needs further work to improve it (or bin it, if it turns out it has a fatal flaw).

I don't think it has a fatal flaw in principle, although omitting to take into account IAT means that you will need to collect much more data to take out yet another 'noise' factor, relative to the amount you would need to collect if you do take into account IAT.  Whilst I agree that its better to meddle as little as possible with the data, some meddling may be useful to make the experimental collection of results more practical.  There is a trade off, and at some point it becomes sensible to take out the predictable effect of known variables in order to get more easily/quickly to the answer.  

However see my post above which I suspect crossed yours and calls into question the explanation for the apparent difference between actual and expected, on the basis that it appears to be present from 19th October, not 4th November when setback was said to start.

There are some rather odd things going on! The long post I have just posted (I think) still remains as a draft, despite having clicked ADD REPLY, so I hope it has been published. It starts with "When a specific ratio mysteriously reappears something is usually not right!" and ends with "I agree, this is still very much work in progress, and I have never claimed that my current method represents the definitive answer, it is just another way of tackling the question that needs further work to improve it (or bin it, if it turns out it has a fatal flaw)."

If that isn't enough, almost all the cells in the raw data worksheet in the spreadsheet I posted earlier (midea_hr_data - Copy (3) - htg_in_corrected.xls) have mysteriously gone to zero...

I will try and sort it out, and answer what (if anything) happened on or about 19th Oct when I have got things sorted out. 

Posted by: @cathoderay

↑

There are some rather odd things going on!

I quite often get database errors recently and have taken to copying text to the clipboard before pressing 'add reply' to ensure I dont have to retype it all!

Posted by: @cathoderay

↑

@derek-m - it's in the post (and is repeated in a reply to @jamespa I am about to post). In short, it is the equation given by the spreadsheet for the cleaned up trend line, see sheet "OAT-E_in" in the spreadsheet.

Could you please explain why, in the 4th Nov. data, during the period before setback commenced, when the heat pump should have been operating in a stable condition, your calculation method appears to show that the 'expected' Energy In is over 19% higher than the measured value? Should they not have been the same at this point?

I don't see how your proposed method can produce results anywhere near the true values, since OAT is not the only parameter which affects the energy consumption of a heat pump. How  do you account for changes in LWT?

Posted by: @jamespa

↑

Posted by: @cathoderay

↑

There are some rather odd things going on!

I quite often get database errors recently and have taken to copying text to the clipboard before pressing 'add reply' to ensure I dont have to retype it all!

If you hit the back arrow on your browser it usually recovers, and yes I have been getting the error message quite frequently for the past couple of days.

Posted by: @derek-m

↑

Could you please explain why, in the 4th Nov. data, during the period before setback commenced, when the heat pump should have been operating in a stable condition, your calculation method appears to show that the 'expected' Energy In is over 19% higher than the measured value? Should they not have been the same at this point?

The setback did start on the 4th November, at 2100 hours. As noted earlier, the first and the last days are not typical, they only include 3 rather than 6 setback hours, as the days run midnight to midnight, not noon to noon. This was discussed in detail earlier. 

Posted by: @derek-m

↑

I don't see how your proposed method can produce results anywhere near the true values, since OAT is not the only parameter which affects the energy consumption of a heat pump. How  do you account for changes in LWT?

This is modelling thinking becoming muddling thinking. I am deliberately NOT modelling anything (apart from in effect doing a regression of energy in on OAT, which I concede in some people's minds might be a model). A core assumption is that all of the variables and their minutiae are baked into that relationship (equation), which, assuming a normal distribution for the energy in values for a given OAT, gives a meaningful average energy out for that given OAT. If that is true, then I don't need to account for all the other variables, they are already accounted for. If I then assume regression to the mean occurs, then over aggregate periods of time, the values will regress towards a mean that is a not too inaccurate representation of the true mean. 

The italics are to emphasise that this is a proposal, a work in progress, something put up for discussion, not a definitive answer.

Posted by: @jamespa

↑

Thanks for the responses....

However see my post above which I suspect crossed yours and calls into question the explanation for the apparent difference between actual and expected, on the basis that it appears to be present from 19th October, not 4th November when setback was said to start.

@derek-m @cathoderay until we get to the bottom of this and either explain it or conclude that i have goofed in my analysis I really don't think its worth much more discussion/analysis.

Posted by: @cathoderay

↑

I am deliberately NOT modelling anything (apart from in effect doing a regression of energy in on OAT, which I concede in some people's minds might be a model). A core assumption is that all of the variables and their minutiae are baked into that relationship (equation), which, assuming a normal distribution for the energy in values for a given OAT, gives a meaningful average energy out for that given OAT. If that is true, then I don't need to account for all the other variables, they are already accounted for. If I then assume regression to the mean occurs, then over aggregate periods of time, the values will regress towards a mean that is a not too inaccurate representation of the true mean. 

For the avoidance of doubt I accept that this is (most probably) true but it does mean that it will be necessary to collect much more data in order to take out effects (noise) which could easily be taken out by simple arithmetic.  That's obviously a choice that the experimental scientist has to make.  

OK, I suppose the expected energy in is also a modelled value, but the point is that it is a very simple process, using two variables, OAT and observed energy in, to model what the expected energy in would be, for any given OAT.