Posted by: @jamespa

↑

Can you post the actual data in a usable form for 48rs before and 24hrs after each day of interest (and of course also for the day itself) and the calculation you did to get to the figures. 

Can and will do. I'll post the minute data for the last two weeks shortly. The calculations I have explained before but I expect that signal may have got lost in the general noise, so here we go again:

1. The raw data is collected by a python script over modbus at minute intervals from the wired controller and MD02 indoor air temperature sensor, and logged to the minute data csv file. No calculations are done to this data (apart from things like a decimal point adjustment eg a flow rate of 1.43 sent as an integer of 143 is corrected to 1.43 by dividing by 100), apart from setting up the logic to allocate the energy parameters to either space heating or DHW heating. This is done by using the binary value for the position of the three port two way valve (0 or 1): if the value is 0, allocate the amps_in to htg_amps, if it is 1, allocate amps_in to dhw_htg. This value is also used as a multiplier (see below), to 'confirm' a value when it is 1 or zero it when is 0 (a value x 1 = value, same value x 0 = zero).     

2. Every hour, another script takes the last hour's minute data, and calculates the various derived hour values, and logs the results to the hour data csv file. These are hourly trailing values for each hour. The same script, after a short pause, sums the latest 24 hours worth of data from the hour data file, and logs the results to the 24 hour data csv file. Again, the values represent trailing 24 hour values eg the noon value is the 24 hours from the previous noon to the current noon. The 'Total past 24h kWh in to noon on 26 Nov: 48.25' and 'Total past 24h kWh in to noon on 1 Dec: 70.04' reports are from the 24 hour data csv file.

The hourly calculations are mostly done using basic python and the pandas module. The latter can read csv files and then manipulate the data eg select the last 60 rows of minute data to get last hours worth of data, and do thing like convert a string datetime to a true datetime that is treated as a real date and time. The actual calculations are as follows:

## for total energy in:

calc_kWh_in = round(((df.amps_in.mean() * df.volts_in.mean() * 1) / 1000) * dcf, 2)

The dcf variable is the difference correction factor, ie the correction factor to get the calculated value to match the external kWh meter value, and is currently 1.18, and no, I have no idea why I multiplied the amps x volts by 1. The rest should be obvious, bearing in mind that df is the pandas 'data frame' that holds the data, thus df.amps_in.mean() says return the mean for the amps_in column in the dataframe and so on. The '/ 1000' converts Wh to kWh. I did try calculating the values on a minute by minute basis, and then summing them, but got the same result as using the mean (makes sense if you think about it: 5A x 240V for 60 min is the same 0A for 30 min followed by 10A for 30 min x 240 etc) and using the mean makes for simpler shorter code. I also tried various fancy area under the curve type calculations, but again got the same (or very very close) answer, so I have stuck with the simpler use of the means. Exactly the same calculation is used for the allocated space and dhw energy in values, using their allocated raw data.

The actual modbus readings are presumably momentary values and the above assumes they are representative for the minute. I think this assumption is reasonable, as visual monitoring of the amps in on the wired controller shows the value doesn't jump all over the place, and only changes relatively slowly over time.

The energy out calculation is similar: 

## for total energy out

calc_kWh_out = round((df.flowrate.mean()/3.6) * spht * (df.LWT.mean() - df.RWT.mean()), 2)

The flow rate division by 3.6 is to correct for the fact that it is reported in m^3/h, and the spht variable is the specific heat of the circulating fluid (4.05 for 10% glycol). 

The space and DHW allocated energy values use the more complex 'calculate for each minute then sum' approach. It was the first method I tried, and I never got round to updating the code to use means. For example, this adds a new column to the minute data frame (not the file) ('* (1/60) as this is minute data, and (see (1) above for logic) htg_on_off is either 1 if on and leaves the value unchanged or 0 if off, making the value equal to zero):

df['htg_kWh_in'] = ((df['amps_in'] * df['volts_in'] * (1/60) * df['htg_on_off'])/1000) * dcf

The energy out is done in similar fashion, and then these new minute columns are then summed over the hour to get the hour values.

The 24 hour space and DHW values in the 24 hour data csv file are even simpler, they are just sums the values for the last 24 hours in the hour data file, which are then added to the 24 hour data file. I should mention the total energy in and out values are are not directly calculated, but are the sums of the relevant space and DHW values, but tests on the hour data file show this gets the same or very close results as if the value had been directly calculated eg 79.944 vs 80.08 for energy in for the current past 24 hours, the minor differences probably being due to rounding errors. 

@jamespa - and here's the data:

@cathoderay Not an answer just some more analysis to ponder.

Here is a table, taken directly from the minute by minute data, of heating energy (from flow rate * dt), dhw energy (ditto), electrical energy (from volts and amps), degree-minutes (sum of IAT-OAT for any given day), heating energy in per degree minute and average COP.

25th & 26th stand out as having low heating energy per degree minute, but note either side its a) much warmer and b) the energy in per degree minute is higher (much higher in the case of 27th and even 28th).  Are these two days making up for setback as the fabric slowly recovers, but you didn't notice it because the OAT was higher?

30th-2nd (cold, no setback) are actually slightly better than average in terms of heating energy per degree minute.  Not sure if this actually tells us anything, other than again the difficulty of making inferences from individual daily data.

Ive also uploaded the spreadsheet so you can check my workings.

This doesn't answer the question but illustrates nicely how complex any analysis is!

If I try to correlate my energy use with OAT its definitely complex and delayed by 12-24hrs, which is why Im thinking a bit about the history in your data.  

@cathoderay @derek-m

I think I am making some progress untangling the data above provided by @cathoderay. 

The raw data showed a tolerable, but not strong, correlation between heating energy delivered to the house and 'degree minutes = sum (IAT-OAT).  Above I talk above vaguely of 'history' and its effect on consumption and speculate on some of the reasons for the larger divergences from good correlation in the data provided.  That's all very well but its hardly quantitative.

The 'history' should, I think, be manifested in the energy held in the fabric which changes depending on the IAT (and, to a much lesser extent assuming that the house is well insulated and the thermal mass is inside the insulation, on the OAT). 

So in a first quantitative attempt to account for this I have, for each day of the measurement, worked out the difference between the starting IAT and the ending IAT and applied this, multiplied by an adjustable factor (which is essentially a proxy for the heat capacity of the house) to the heating energy in.  This results in a much stronger correlation between heating energy in and degree minutes.

The first chart shows the scattergram of energy to house vs degree minutes without adjustment

The second graph is the scattergram of energy to house vs degree minutes with an adjustment of -15000 Wh/degree of IAT temperature rise (that is the right sense, the energy required will increase if the house temperature rises, so subtracting this increase takes out from the measured energy delivered the increase).

Its immediately obvious that the second graph is much tighter, implying (probably) that the adjustment has gone some way to taking into account the change in energy stored in the fabric.

Here is the data from which the second chart was plotted

Comments welcome, it would be good to do this with a larger dataset.

@jamespa - interesting! I have made a small change, changed the degree minute calculation from (IAT -OAT) to (baseline - OAT) with a baseline of 15.5, as the IAT fluctuates and furthermore is a bit higher than the baseline (in fact, it doesn't make that much difference in practice, just shifts the values by a small amount, overall pattern remains very similar), and then plotted my baseline daily degree minutes vs daily energy in (couldn't get the labels to format to dd-mmm, so a bit messy):

Firstly, this does appear to confirm that overall, energy in is proportional to degree minutes, but there are also some outliers, more easily spotted on a chart than in a table. The 24h periods To Noon 30 Nov and 2 Dec used notably more energy, the 24h periods to Noon 25 and 26 Nov used less energy (broadly the periods you picked out, just easier to see on the chart). On paper, this should, I suggest, mean that the building gained heat (warmed up, IAT increased) during the first two periods, and lost heat (cooled down, IAT decreased) in the second two periods. What actually happened?

24h to To Noon 30 Nov (expect warming) got slight cooling (19.0 - 18.8)

24h to To Noon 2 Dec (expect warming) got moderate cooling (17.9 - 17.3)

24h to To Noon 25 Nov (expect cooling) got marked cooling (18.7 - 17.7)  

24h to To Noon 26 Nov (expect cooling) got marked cooling (17.7 - 16.7*)

Not that I am sure what that tells us! - except that if there is an energy deficit (energy in is less than expected for given degree minutes), then IAT falls, but the reverse, energy excess, doesn't cause the IAT to increase, in fact, it actually fell by a small amount. 

Edit: *typo corrected

Posted by: @cathoderay

↑

@jamespa - interesting! I have made a small change, changed the degree minute calculation from (IAT -OAT) to (baseline - OAT) with a baseline of 15.5, as the IAT fluctuates and furthermore is a bit higher than the baseline (in fact, it doesn't make that much difference in practice, just shifts the values by a small amount, overall pattern remains very similar), and then plotted my baseline daily degree minutes vs daily energy in (couldn't get the labels to format to dd-mmm, so a bit messy):

-- Attachment is not available --

 

Firstly, this does appear to confirm that overall, energy in is proportional to degree minutes, but there are also some outliers, more easily spotted on a chart than in a table. The 24h periods To Noon 30 Nov and 2 Dec used notably more energy, the 24h periods to Noon 25 and 26 Nov used less energy (broadly the periods you picked out, just easier to see on the chart). On paper, this should, I suggest, mean that the building gained heat (warmed up, IAT increased) during the first two periods, and lost heat (cooled down, IAT decreased) in the second two periods. What actually happened?

24h to To Noon 30 Nov (expect warming) got slight cooling (19.0 - 18.8)

24h to To Noon 2 Dec (expect warming) got moderate cooling (17.9 - 17.3)

24h to To Noon 25 Nov (expect cooling) got marked cooling (18.7 - 17.7)  

24h to To Noon 26 Nov (expect cooling) got marked cooling (17.7 - 18.7)

Not that I am sure what that tells us! - except that if there is an energy deficit (energy in is less than expected for given degree minutes), then IAT falls, but the reverse, energy excess, doesn't cause the IAT to increase, in fact, it actually fell by a small amount. 

 

Can you clarify whether this was composed after or before my post at 10.05.  I have the impression before, but I could be wrong.  Essentially my 10.05 post attempts to quantify the effect of 'building warmed up'/'building cooled down'

Posted by: @jamespa

↑

Can you clarify whether this was composed after or before my post at 10.05. 

During, in effect we cross-posted, your post appearing first, but I didn't see it until after i had posted.

Also note the typo correction.

Posted by: @cathoderay

↑

Posted by: @jamespa

↑

Can you clarify whether this was composed after or before my post at 10.05. 

During, in effect we cross-posted, your post appearing first, but I didn't see it until after i had posted.

Also note the typo correction.

Thanks.  I think your adjustment has the effect of moving the curve up or down, and mine the effect of changing the distribution.  They are thus different (I think) in nature and I think mine attempts to compensate for the observations made by you about the plot as it is following yours. 

I will give some more thought, I think that the reality there is both a) an offset (due to other heat sources in the house) and b) a history held in the fabric.

Have you got a larger (but still manageable) dataset I could analyse, maybe a couple of months worth?

Posted by: @jamespa

↑

I think your adjustment has the effect of moving the curve up or down,

It's actually left or right, because we have the axes the other way round, I put degree minutes on the x axis as to my mind it is the predictor variable, but yes, it just shifts the line this way or that, the very minor changes in the distribution are too small to be seen.

Posted by: @jamespa

↑

I will give some more thought, I think that the reality there is both a) an offset (due to other heat sources in the house) and b) a history held in the fabric.

So will I. The 'other sources' (a) will be fairly constant in my house at this time of year. The history held in the fabric (b) - this is a tricky one because I appear to have both a high loss building (basically hard to insulate it well) and a high thermal mass (what materials there are eg solid stone walls and solid ground floor are slow to heat up, and (relatively) slow to cool down).

Posted by: @jamespa

↑

Have you got a larger (but still manageable) dataset I could analyse, maybe a couple of months worth?

The complete data set goes back to late Sept this year, when I installed the modbus IAT sensor. Before that, there is no modbus IAT, only a separate hourly IAT data logger that requires very awkward migration to get it into the minute data set, as the hours and minutes don't line up. In effect it needs a sort of horizontal inner join on 'datetime like around the hour', but I can't figure out how to do that! Plus it's hourly data, so we loose that very useful ability to see what is happening at the minute level, inside the hour. I think best to stick with the late September onward data set.

I can post that, but it still has the old pre-correction factor (the 1.18 adjustment) energy in values until very recently. Because it is updated every minute, the file is very lively, and I am still working on the best way to update all the historical values - its not just the raw energy in, it is also all the calculated values that use energy in - without nuking the whole shebang. I'll see what I can do.

@jamespa

You need to remember that although comprehensive, the raw data does have certain weaknesses specific to CathodeRay's system.

There is still a PHE between the primary circuit and the secondary circuit, which I suspect may vary the temperature measurements under varying load conditions.

The other factor is the cooler exhaust air being drawn into the heat pump intake, which appears to vary the measured OAT in relation to the true OAT, again under changing operating conditions.

It would be useful to measure both the LWT and the water temperature actually at the heat emitters, and also the heat pump measured OAT against the true OAT measured independently.

In the Modeling Tool I found that a Thermal Mass/Heat Capacity value of 260kWh seemed to match the heat loss rate. Also by factoring in a 5C temperature reduction across the PHE, and an increasing variance between true and measured OAT as loading increased, I came close to matching the measured IAT values throughout the day.

Posted by: @cathoderay

↑

The 'other sources' (a) will be fairly constant in my house at this time of year. The history held in the fabric (b) - this is a tricky one because I appear to have both a high loss building (basically hard to insulate it well) and a high thermal mass (what materials there are eg solid stone walls and solid ground floor are slow to heat up, and (relatively) slow to cool down).

The data should be capable of telling us the amount of energy contributed by other sources, it should appear as an intercept in the correlation curve, net of any energy in lost in the pipework etc.  So I think on balance I prefer, for now and for this exercise only, to leave it out of the calculations and see what the graphs tell us.

The loss/fabric energy is interesting and I spent several hours thinking about it. 

In principle both the IAT and the OAT affect the stored energy, but in different ways.  Currently I'm only considering the effect of the IAT, which is a reasonable approach for a house where the main thermal mass is within the insulating layer, and this is large compared with the thermal mass of the insulating layer itself.  A house with loft insulation, underfloor insulation beneath a slab and external wall insulation would be the ideal example, but most reasonably well insulated houses will be fairly close I think.  In this case the thermal gradient is largely in the insulation, and its a reasonable approximation that there is just a mass at IAT, ie the 'fabric adjustment factor' can be based on the IAT only.

In a house where the thermal gradient runs through massy fabric elements it becomes more complicated (which is why I ignored it, at least for now).  In this case the energy stored in the fabric is materially affected by the OAT as well as the IAT, and heat transfer from outside to fabric can occur if the OAT increases, because in this circumstance the outer part of the fabric will be colder than the OAT. What happens is that the thermal gradient within the fabric itself changes as the OAT/IAT changes and thus the amount of energy stored ditto.  The 'ideal' example (in the model sense, not in the practical sense) of such a house is a massive one minimally insulated (eg one with solid walls and floors with little or no insulation).  I cant quite work out how to compensate for this because of the potential two-way transfer of energy.

Posted by: @cathoderay

↑

I can post that, but it still has the old pre-correction factor (the 1.18 adjustment) energy in values until very recently. Because it is updated every minute, the file is very lively, and I am still working on the best way to update all the historical values - its not just the raw energy in, it is also all the calculated values that use energy in - without nuking the whole shebang. I'll see what I can do.

Although I do calculate electrical energy in the sheet its not used in the current analysis, so Id by happy with the raw minute by minute data as you posted for the shorter period.  Preferably just a couple of months though, or alternatively (if you think its representative) I could probably cope with Sept-Mar last year, ie one season.  The formulae are all set up so its really just a copy and paste exercises once the data is imported (which I do via libre office because it natively recognises the date-T-Time format and translates it to a date number, whereas my version of excel currently doesn't).

If the improvement in correlation made by taking account change in IAT in the crude way I do applies to a larger dataset, I think its potentially quite a step forward.

Posted by: @derek-m

↑

@jamespa

You need to remember that although comprehensive, the raw data does have certain weaknesses specific to CathodeRay's system.

There is still a PHE between the primary circuit and the secondary circuit, which I suspect may vary the temperature measurements under varying load conditions.

The other factor is the cooler exhaust air being drawn into the heat pump intake, which appears to vary the measured OAT in relation to the true OAT, again under changing operating conditions.

It would be useful to measure both the LWT and the water temperature actually at the heat emitters, and also the heat pump measured OAT against the true OAT measured independently.

In the Modeling Tool I found that a Thermal Mass/Heat Capacity value of 260kWh seemed to match the heat loss rate. Also by factoring in a 5C temperature reduction across the PHE, and an increasing variance between true and measured OAT as loading increased, I came close to matching the measured IAT values throughout the day.

Thanks for the data health warning.  I was struck by just how much difference adjusting for change in IAT made to the degree of correlation (have a look at the two graphs I posted around 10am, its quite obvious).  Given this I'm prepared to try with a bigger sample notwithstanding the limitations of the system, as the technique, which has a reasonably sound base in theory, has the potential to take out a lot of what is currently perceived as 'noise' in the correlation.  Ideally I would still prefer to do it with a cleaner set up though.

The small data set I used suggested a thermal mass of 15-20kWh/C (ie that was when the correlation was best).  Im not sure at what temperature your 260kWh is, but assuming its baselined at 0 C, the two values, from two completely independent methods, are not so different.