@majordennisbloodnok Whereas Peter Brough (sp?) would have educated Archie with a “Gottle of geer’ I suppose.🤨

Posted by: @toodles

↑

I suppose that it is not impossible

Not impossible, true, but when an academic dept publishes results or a commercial concern makes use of and publishes academic results, the methodology should also be published. Otherwise, who knows what went on between the devil and the deep blue sea? 

@cathoderay Neither is it impossible that the ‘sans-Homely’ set up was carefully optimised before the test. Presumably, such information should be declared, but, we don’t know if it was do we? Or perhaps, as the information that was published might suggest, both trials were conducted based on a ‘face value’ assessment. Quizzically Toodles.

Posted by: @toodles

↑

Neither is it impossible that the ‘sans-Homely’ set up was carefully optimised before the test.

The £444 claim is based on a comparison with a 'Fixed Flow Temperature Base Case', not exactly a carefully optimised set up, which is then given a 'factor X' treatment which is supposed to bring the results in line with the 'average home'. The reasoning in a footnote is rather impenetrable. The key bit from the blog is this:

£444 is the difference between the top and bottom amounts in the last column of the table. In percentage terms it is a saving of 43%, very impressive. Note the bit about the testing running for over three months. But according to the University of Salford Energy House Case Study report, the study was done over two 24 hour periods, with only 'approximately 20% cost savings were achieved with the Homely system':

"As part of the ERDF- supported Energy House 2.0 project, the Homely system was tested in the Smart Meters > Smart Homes Laboratory at the University of Salford which has a choice of heating systems available, including an Air Source Heat Pump (ASHP). A test compared two 24-hour periods with and without the Homely control system. With electricity priced according to the Octopus Energy dynamic tariff, the external temperature, room temperature, electricity demand, and cost were recorded. During the test period external temperatures were between 5 and 10⁰C and comparison of the two test periods showed that approximately 20% cost savings were achieved with the Homely system. This was despite the fact that external temperatures were lower during the 24-hour period where the Homely control app was used."

Approx 20% and 43% are quite a long way apart, as are two 24 hour periods and over three months. 

Posted by: @toodles

↑

@cathoderay Neither is it impossible that the ‘sans-Homely’ set up was carefully optimised before the test. Presumably, such information should be declared, but, we don’t know if it was do we? Or perhaps, as the information that was published might suggest, both trials were conducted based on a ‘face value’ assessment. Quizzically Toodles.

That’s very true, @toodles, @cathoderay. Nor will we know for the foreseeable. When I asked Salford, they said the report of the study wasn’t in the public domain at the moment.

That begs the question, in that case, of who does own the report; who commissioned it and who dictates who can read it. I don’t have the answers, btw.

This is a really strong piece of work, @majordennisbloodnok. I’ll be honest, I wasn’t aware of some of the detail you’ve uncovered here, and it’s pretty disappointing reading once you step back and join the dots. Fair play for taking the time to dig into it properly and call it out so clearly.

What really stands out to me is that you’re not dismissing these products outright, you’re simply asking for transparency and evidence. That feels like a very reasonable position for any homeowner to take, yet it’s amazing how often that gets lost once marketing takes over. We see the end result of this all the time on the forums: people coming in expecting 20-40% savings, for example, because that’s what they’ve been told, only to find they can’t verify anything independently and don’t really understand what’s changed, if anything, on their system.

The baseline issue you raise is a big one. Comparing a third-party controller against a deliberately poor setup, like a fixed high flow temperature that no competent installer should be leaving behind, doesn’t feel like a fair or honest comparison. If the industry itself doesn’t recommend running systems that way, then using it as the reference point for “savings” is, at best, questionable. It starts to look like a problem being exaggerated so a solution can be sold.

Your breakdown of the Homely study is particularly eye-opening. Two short snapshots over a 24-hour period tell us very little about real-world performance across a heating season, especially when weather, occupancy and behaviour play such a big role. Presenting that data as indicative annual savings, and then pushing those figures through installer sales material, explains a lot about why expectations and reality so often don’t line up.

The data access point is also worrying. Any product that genuinely improves performance should be comfortable with independent scrutiny. If it works, it should stand up to heat meters, OpenEnergyMonitor data or third-party analysis. When homeowners are restricted from accessing their own system data, it inevitably raises questions about what’s being protected and why.

I agree with you on Melpump too. It feels less controversial precisely because it doesn’t over-promise. Better insight, better diagnostics, better information... those things genuinely help homeowners understand and improve their systems. That’s very different from headline savings claims that can’t be easily verified.

For me, this all comes back to homeowners needing clear, comparable information so they can make informed decisions. Claims need proper context, baselines need to be realistic and performance needs to be verifiable. 

It’s a real shame, because these tools could absolutely have a place if they were honest about what they do and don’t do. But overstated savings and fuzzy comparisons just leave more homeowners feeling misled, and we already see far too much of that playing out here.

Thanks again for the detailed post and forensic analysis. Hopefully this will post will gain traction, and homeowners will be able to manage their expectations better.

Posted by: @majordennisbloodnok

↑

In some ways, @sheriff-fatman, my inclusion of Melpump was to show there is another way; that a company can successfully fulfil a need without making inflated claims. @toodles is right - it is rife in the industry but I see no reason why it has to be.

I think that heat pumps and their ancillary systems are too measurable for an industry that’s spent years getting away with loose claims. When everything can be logged, graphed and benchmarked, there’s real accountability, and not everyone likes what that exposes. So instead of fixing the underlying issues, we’re already seeing the response... bigger, bolder numbers, selective reporting and performance figures that look great on a slide or website but fall apart under proper scrutiny.

That’s why I’m increasingly sceptical of headline claims full stop. As soon as performance becomes a marketing battleground, over-reporting is inevitable. You choose the boundary, you choose the timeframe, you ignore the bits that don’t flatter the system and suddenly you’ve got a “market-leading” result. 

Posted by: @majordennisbloodnok

↑

Posted by: @toodles

↑

@cathoderay Neither is it impossible that the ‘sans-Homely’ set up was carefully optimised before the test. Presumably, such information should be declared, but, we don’t know if it was do we? Or perhaps, as the information that was published might suggest, both trials were conducted based on a ‘face value’ assessment. Quizzically Toodles.

That’s very true, @toodles, @cathoderay. Nor will we know for the foreseeable. When I asked Salford, they said the report of the study wasn’t in the public domain at the moment.

That begs the question, in that case, of who does own the report; who commissioned it and who dictates who can read it. I don’t have the answers, btw.

Just as an update, I got clarification that the Homely report was not part of the ERDF-funded Energy Home project. The ERDF-funded bit was setting up the Energy Home, and the Homely study was one of a number of separate research projects that then made use of the Energy Home facilities and research team. This information strongly suggests the study wasn't commissioned and/or funded by Salford University, but it still doesn't say who did. Homely refer to the report as "Independent testing with Salford Energy Labs", so either the commissioning and funding were courtesy of a third party or Homely commissioned and funded it and Salford did the testing (which is not quite so independent).

I hasten to add that I have no reason to suggest Salford are in any way to be criticised. However, if whoever paid for it only provided the funds for two days of testing when a longer period might've been more meaningful, that's an inherent bias Salford would have no way of avoiding and so might affect the true independence of the results. If, however, that two days was all Salford could offer, the independence remains for good or ill.

Posted by: @majordennisbloodnok

↑

However, if whoever paid for it only provided the funds for two days of testing when a longer period might've been more meaningful, that's an inherent bias Salford would have no way of avoiding and so might affect the true independence of the results.

I think the length of testing is the big factor here, and it’s almost certainly driven by budget rather than best practice. I’ve been told it costs in the region of £5,000 per day to use the facility, so if funding only stretches to one or two days, that automatically constrains what can be tested and how representative the results can be.

@editor They couldn’t afford to run to a second heat?

Posted by: @editor

↑

Posted by: @majordennisbloodnok

↑

However, if whoever paid for it only provided the funds for two days of testing when a longer period might've been more meaningful, that's an inherent bias Salford would have no way of avoiding and so might affect the true independence of the results.

I think the length of testing is the big factor here, and it’s almost certainly driven by budget rather than best practice. I’ve been told it costs in the region of £5,000 per day to use the facility, so if funding only stretches to one or two days, that automatically constrains what can be tested and how representative the results can be.

Agreed. My personal feeling is that if someone asks for a study to be run about something they have a vested interest in, they should not then call that study independent. From what I can see, either someone completely independent has done Homely a huge favour in asking for this study to be run (and then allowing Homely access to it) or Homely commissioned the report and are now erroneously claiming independence.