Posted by: @jamespa

↑

Finally we have to eliminate confirmation bias (which I think is broadly the physics equivalent of the placebo effect) which is almost certain to exist with the experiment and quite possibly also in the theory.

I can at least clear this one up! They are related, but not the same. Confirmation bias is in effect seeing in results what you already believe to be the case - the results confirm your beliefs. We are all to a greater or lesser extent prone to this across many situations. More generally a bias occurs when results are wrong because of systematic errors, ie a there is a reason for the error, as as opposed to random error. A good example is lead time bias, which suggests medical screening is effective when it isn't:

No screening: patient gets a clinical diagnosis of X on 1st Jan 2024, dies 1st Jan 2025, survival time one year.

Screening: patient gets screened for X on 1st Jan 2023 and is positive, but still dies 1st Jan 2025, survival time two years. 

But actually they still died on the same day, all you did was lengthen their period of illness. Well done!

The placebo effect, which is fascinating, is in effect a benefit that occurs because a patient believes as treatment will benefit them. A sugar pill can have a useful effect if I tell you it is a strong opioid with remarkable pain killing abilities. Since there is no opioid in the pill, any beneficial (analgesic) effect is a placebo effect. There is also the opposite effect, the lugubriously named nocebo effect, where a patient's negative beliefs about a treatment worsen the outcome.

Since confirmation bias is pretty universal, it can be assumed to be present, and can be dealt with by applying more rigorous independent analytical methods, ie ones that don't rely on seeing what we expect to see.

Posted by: @jamespa

↑

The experimental results, such as they are, tell a variable story, and as haven't yet been reconciled even with conservation of energy, which we can be absolutely certain applies, extrapolating from the results specific to one system to expectations for another would be irresponsible.

The conservation of energy certainly applies, but have we applied it correctly - that is the question. Given robust enough methods, empirical results should always be taken seriously, even when they are at odds with the theory, because they may be telling us something is wrong with the theory, or more accurately, the application of theory to the problem at hand.

I think we are all agreed that one 24 hour period does not a conclusion make. In fact that is partly why I posted two different periods, two different ambient conditions, two different results. The fundamental problem with the empirical method is how to estimate the expected value had the setback not occurred. What I am hoping is that as time goes by, and more data is collected, we may be able to find 24 hour periods where the ambient conditions were very similar, where some of those periods had a setback, and the others didn't. In that way, we might get pretty close to a like for like comparison, though antecedent conditions may matter. Nonetheless, it will be as close as we can get to a controlled experiment: same building, same conditions, only difference is whether a setback was used or not. Take this period of steady state running:

If I can find a similar period which did have a setback, then we might be onto something.   

Edit: here is a pretty similar period with a setback: 

I think it is very hard to argue I didn't save around 5kWh (not 6, because there is a small increase in energy in immediately after the setback) with that setback. @jamespa, I wonder what your model predicts for these conditions?     

Posted by: @cathoderay

↑

The conservation of energy certainly applies, but have we applied it correctly - that is the question. Given robust enough methods, empirical results should always be taken seriously, even when they are at odds with the theory, because they may be telling us something is wrong with the theory, or more accurately, the application of theory to the problem at hand.

Agreed.  For the avoidance of doubt I'm not suggesting anything else.  However...

Posted by: @cathoderay

↑

I think we are all agreed that one 24 hour period does not a conclusion make. In fact that is partly why I posted two different periods, two different ambient conditions, two different results.

What I also say above, perhaps for the first time in explicit terms, is that one system does not a general conclusion make, unless we can explain the results so that we know, at least to some extent, under what circumstances they can be applied to which other systems.

To use a crude analogy: Fred dies at 75 for reasons we cant explain.  Fred is a human being.  It therefore follows that humans die at 75. Probably not!

Somehow, if we are to make progress in predicting what a reasonable subset of systems might be expected to do, we need either to match theory to the few experiments we have, or alternatively have tolerably controlled results from a sufficient sample of diverse systems that we can reasonably expect them to be representative. 

The latter seems unlikely to happen, unless you are Octopus or someone else that has access to the big data.  Thus, until one of those who does have access to the big data spills the beans with sufficient detail that we can be confident they are telling a truth not just a marketing spin, our only viable approach (assuming our aim is to make some sort of general statement rather than one specific to one system) is to try to match theory with experiment and/or vice versa.

Alternatively just give up, say we don't know for certain and that ToU tariffs make the whole question academic anyway in many cases.

Posted by: @cathoderay

↑

@jamespa, I wonder what your model predicts for these conditions?

I dont know although, just eyeballing it, its certainly appears to be in the space where it might predict a saving.  Do you have the data in numerical form that I could compare, ideally also for the day before as the precedent conditions certainly can matter.

Posted by: @jamespa

↑

Alternatively just give up, say we don't know for certain and that ToU tariffs make the whole question academic anyway in many cases.

Basically, all agreed, apart from the above. As I posted earlier, 80% of homes are still on standard variable tariffs, and of the 20% who aren't, most are on fixed tariffs (as indeed I am). ToU tariffs may be common here on the forum, but in the rest of the country they are very rare, and I don't see that changing overnight. And having got so far, I don't think we should give up!

Posted by: @jamespa

↑

I dont know although, just eyeballing it, its certainly appears to be in the space where it might predict a saving.  Do you have the data in numerical form that I could compare, ideally also for the day before as the precedent conditions certainly can matter.

I certainly have the data, and can let you have it, either as minute data or hour data, or both, just say which and it is yours. In fact, you may already have the data for the second chart, it is from the 2023/24 heating season, and I think I have already given you a copy of that.

Posted by: @cathoderay

↑

And having got so far, I don't think we should give up!

I probably agree, although the thought I had about setback beingat least in part equivalent to cycling does concern me as I dont see much prospect of modelling cycling in the near future. 

Posted by: @cathoderay

↑

I certainly have the data, and can let you have it, either as minute data or hour data, or both, just say which and it is yours. In fact, you may already have the data for the second chart, it is from the 2023/24 heating season, and I think I have already given you a copy of that.

I have the hour data for the period in 2024, but not for the period in 2025, so would appreciate the latter.

Posted by: @jamespa

↑

I have the hour data for the period in 2024, but not for the period in 2025, so would appreciate the latter.

Here it is, 2025 year to date, csv format, column headings as before:

Thanks.  

I seem to have several spreadsheets in various formats and with various columns, many of which I dont doubt I created myself.  It would be good (for me) to standardise on a format (possibly different for Hr data and min data.

Is this the appropriate format to standardise on, and if so is it easy for you to produce previous seasons data in the same format.  Its time I had a bit of a clear out!

Posted by: @jamespa

↑

Its time I had a bit of a clear out!

I know the feeling! I have all sorts of files in various places. The data I have just posted comes from the raw csv data file, I just copied the 2025 data, and it is the most original unmanipulated version of the data, even if some of the column headings are a bit cryptic eg MD02_tmp is the room (IAT) temp, because the sensor I use is an MD02 sensor. If it were me, I would be inclined to keep read only copies of the raw csv data, either as one big running file, or split into heating seasons, and then use xls/xlsx files for data that has been worked on, with in an ideal world each file having a meaningful name which will still make sense next year when I have forgotten all about it!

OK, I think that makes sense.  Can I have the other raw csv file(s) please and Im going to have that clear out!

@jamespa - I'll DM you the entire raw csv data files (minute and hour and 24 hour) from day one to date, with some notes on what's what.

Posted by: @sunandair

↑

I thought we’ve been here…. It’s fact isn’t it?

https://renewableheatinghub.co.uk/forums/postid/27514

@cathoderay Im just not sure what you are trying to find out beyond what already is showing as realistic savings. Also I do disagree with a statement you said upthread….

Posted by: @cathoderay

↑

Possibly. Or possibly not. We still don't have a cast iron empirical methodology that all can accept as giving a definitive answer, robust enough to trump theoretical objections. For sure, both your data and my data suggest savings are possible, but we are plagued with three main problems. Firstly, if we get the saving by merely by having a lower mean 25 hour IAT, how do we demonstrate that that lower IAT does not come with a penalty, ie IAT too cold for some of the day? Both of us have failed to do that.

I have to say that, for my part your statement is not true. Here is one sentence from my definition of our setup.-

“In summary the purpose of these trials have been to see if there are any energy savings in having an automated setback over a 7 hour period of night time, while still guaranteeing a comfortable room temperature of 20.5c between the hours of 8.00am (or earlier) and 10.30pm.“

The post then goes on to clarify the schedule used to ensure the IAT guarantees a minimum room temp of 20.5c.-

“We are operating a setback from daytime 21c back to 17c between the hours of 9pm and 4am. The ramp up in the morning starts at 4am and is achieved in 1c increments with 1.5 hours between each 1c rise taking our set temperature up to 20c. The incremental rises are designed to keep the Auto Adaptive controller operating at a minimal output so that the feedback loop doesn’t see large temperature rise targets. The 21c room temp is achieved by Auto Adaptive always over running the target by half or one degC.”

the page I’ve posted at the top of this post is from my topic “Creative trials in night time setbacks….”

perhaps you haddent read it but the top of page 2 describes definitions and performance conditions of the trial which needed to meet in order to be a valid trial. There were over 3 months of trials and 5 trials of 24 hour operation with a spread of outside air temperatures to give a range of results. 
It was a period of intense testing which I think proved a distinct trend in addition to some new insights which were not expected.

there are example graphs of 24 hour operation showing the guaranteed room temperatures during the trials. Furthermore there is an explanation of graduated temperature rises so that the HP did not race off in high flow temps. And how this could be scheduled for automation.

@sunandair - you may have confidence in the findings from your creative trials but I lack confidence in mine and I have to say that I remain somewhat nonplussed about the conclusions that can be drawn from the data you have published both here and in your creative trials thread.

No one doubts that setbacks save energy use while they are in effect. The problem is what happens after the setback. Specifically, during the recovery period:

(a) how much extra energy is used over and above what would have been used without a setback?

(b) how long does the recovery (say to within 1°C of the desired IAT) take?

Unless I have missed something, the data you have published on energy use during the recovery is minimal, and, I think I am right in saying, only covers 24 hour periods. The problem here is that as in politics, 24 hours is a long time, and a lot can change. A particular 24 hour period might use more or less energy for all sorts of reasons, not just the OAT, but that is the big one, and frankly the most difficult one to control for, because the OAT is out of our hands. For example, two identically cold nights, one followed by a warm day, the other by a cold day, will use very different amounts of energy. If, for example, you happened to have a setback running on the first (warm day) period and no setback in the second (cold day) period, How do you tease out the energy saving due to the setback, and the energy saving due to it being a warmer period?

The mean OAT only goes some of the way towards but I am not even sure you use the mean. Instead, and again I think I read this right, you use a temperature that is one third of the daily range above the lowest OAT for that day:

"1. I select 1third of the temperature variation rather than 50:50 middle mean temperature. In other words if there is a 6c variation in a day I select a +2c from the lower temperature rather than the mid way point of 3c. This is still not very refined but it irons out some of the inaccuracies of the high consumption differential which would otherwise create an inaccurately higher outside ambient temperature for the day’s energy consumption."

The middle of a range is not the mean, and I am not at all sure what a value that is one third up from the bottom of a range is, but whatever it is, I am pretty sure it is not a reliable measure of the central tendency of the OAT for that day. This inevitably calls into question your conclusions. Take two days, both with a range of nine degrees between 5 and 14 degrees, where one spent most of the time cool and only warmed up at the end, whereas the other warmed up at the beginning and stayed that way. Using your system, both would get the same 24h OAT (8 degrees) but the second day would have a higher mean OAT than the first and would use far less energy.  

Nor I am not sure you have established the IAT recovers in good time. So far as I can see, you have only published two charts showing recovery periods during a 24 hour period, both undated (the date visible top left looks like the current date/time, bot the date for the chart), and neither are particularly demanding (the OATs are mostly moderate, with no low OATs):

and it is a moot point whether the recoveries shown meet the criteria for adequate recovery, though I do accept that if you were comfortable enough, then on one level that is an adequate recovery. But the real problem is only two samples, which may or may not be representative.

Likewise, if I have read your account of what you have done correctly, you only have four days of continuous running, which is a tiny sample. Add in the uncertain OAT assessment method, and it really is very hard for me to come to any firm conclusions from this data. Even if we could, we still have the n=1 problem: you have described what happens on one property run one way.   

To recap, the fundamental problem is how to compare like with like. To make a valid straightforward comparison between energy us on a setback day and a no-setback day, the hourly OATs need to be very similar, and the IATs need to be within acceptable limits when it matters (mean IATs can vary, indeed that is supposed to be how the saving, if there is one, happens). Because days with very similar hourly OATs are rare, and for the time being can only be identified using the Mk 1 Eyeball, a very tedious and boring activity, one option is to use some form of adjustment eg what would the energy use have been had both days, one with setback, one without, had the same hourly OATs, and that comes with it's own set of problems.

Meanwhile, I have been trying to find a valid way of comparing the two samples (setback and no setback) I have from the 2023/24 heating season to to determine whether they come from the same population or not (put another way, could the differences have arisen by chance, ie due to random variation?). What I thought would be a simple statistical test is turning out to be rather a challenge, not least because polynomial regression provides the best fit!   

The great thing about science is that it doesn't care one iota about what I think, or what you think, or for that matter what Donald Trump or Elon Musk think.  The facts are the facts, end of. 

In many ways that's extremely liberating, we are all free to have our own interpretation of the facts but, in most cases, it wont make the blindest bit of difference, except to the extent that it changes our behaviours or the behaviour of others.  Most of the time it doesn't materially.

Unfortunately there are some people who have sufficient influence that their (deliberate mis-) interpretation of the facts does change behaviours of many others in a way that is potentially very destructive and arguably malign.   Fortunately none of us fit into that category so far as I know!

@sunandair I am genuinely interested in your data because I would like to get to the bottom of this to my own satisfaction.  Thus if you would be prepared to share and it (eg the table you posted) in a machine readable form and explain what the various annotations mean, I would be most grateful.  Of course if you don't wish to I fully understand.