@iancalderbank

I fully agree with your sentiment, that whilst there are some that may wish to explore 'the far end of a fart', most are not interested in re-inventing the wheel.

I am a firm believer that it only requires one person to carryout the investigation and development, and then the required information should be made available to others, but it needs to be in a simple to follow, step by step format. A guide does not need to inform the reader of the all the processes, good or bad, that were carried out to get to the end result, it merely needs to contain sufficiently detailed instructions of what to do at each stage, what result to expect, and what may need to be done if the achieved result is not as expected.

Others may find the process of interest, which can also be documented elsewhere, but as you state it may not be suitable for beginners. When helping forum member's with their heat pump problems, I often initially spend time assessing the level of technical knowledge of the poster by asking simple questions, so that I can tailor any advice, and guide them through the process in the most productive manner.

@iancalderbank  and @derek-m - I agree with both of you. This thread has unintentionally turned into something of a monster, perhaps not yet Frankensteinian, but heading that way. Little did I know when I plugged my lightening powered electrodes into HA's unbreathing bandaged corpse what horrors would await me...

My original intention when starting this thread was to have maybe half a dozen hands-on how-to posts, with some background on what I had chosen the route I had eg why a mini PC rather than a raspberry Pi. The idea was to be very beginner-focused, very 'do this', because a lot of the problems beginners face is too many choices swirling around behind too much jargon. Then along came HA, and midea_ca_lan, and the fateful decision to run with them.

Months later, we are where we are. This thread will stay, as I am not one for deleting published material. It will be renamed (Frankenstein's Assistant? 50 Shades of Python? Home Alone with Home Assistant?) and the reader can perfectly well make up their own mind whether they want to read it or not. Most will not.

Once I have got a reliable working wired modbus simple python system set up I will do another thread on how to set that up, and that will become the new Beginner's Guide thread.       

Posted by: @william1066

↑

Maybe because you have not written this yet  

Feel free to make HA "even better", experience the power of open source community projects!

I note that you paste the panel as an image, without a link that people can click "View given feedback" to see what happened. HA seems not a friendly community in my experience. If one tries to write something to cover a gap in the documentation, if you can even extract the information on it from people to do so, 7 people will tell you that it should be done in a different way and 3 will tell you (more or less) to go fondle yourself for daring to suggest their beloved project is less than perfect yet.

I remain of the opinion that Open Energy Monitor is better open software for monitoring energy use... I'd make the python script output to MQTT, then let emoncms collect it and draw pretty pictures.

Posted by: @mjr

↑

I note that you paste the panel as an image, without a link that people can click "View given feedback" to see what happened.

I'm sorry, I am not sure what you are getting at here. They are meant to be images, images that show the appearance of the panels, not intended to be interactive in any way here (though they are in part in real life, you can scroll and zoom in and out).

Posted by: @mjr

↑

HA seems not a friendly community in my experience. If one tries to write something to cover a gap in the documentation, if you can even extract the information on it from people to do so, 7 people will tell you that it should be done in a different way and 3 will tell you (more or less) to go fondle yourself for daring to suggest their beloved project is less than perfect yet.

I couldn't agree more. Their forum (or 'community' in their distorted language) is terrible, hostile, incoherent and almost never provides any practical answers. It is another major reason for me saying don't use HA unless you are a masochist.

Posted by: @mjr

↑

I remain of the opinion that Open Energy Monitor is better open software for monitoring energy use

You may well be right. To my eye (just a personal thing), their charts are a little on the chunky side, rather too blocky, as if intended for low res screens. It's a pity there isn't, as far as I know, a reasonably simple non-bloatware generic charting program that can take csv/spreadsheet data and turn it into pretty pictures. The best aesthetics, including getting the quality of any images/pdfs printed out to be of publication quality, I have come across so far are in R. Even there, the quality is not native to R, you have to add a package (cairo-pdf) to get the quality, and R itself is yet another non-intuitive program with a steep learning curve.      

I think we could do with stepping back for a moment and reassessing.

@cathoderay took an initial decision to posit a particular scenario he wanted to explore from the perspective of someone with little prior knowledge of energy systems monitoring. Nowt wrong with that.

@cathoderay's detailed log of the steps and decisions since taken have now demonstrated that the initial stance of requiring everything to be easily explainable to a beginner is no longer entirely appropriate. Also fine, since threads evolve, but that certainly doesn't invalidate what has been documented; far from it.

We seem to have developed a bit of an offshoot argument at the same time about shortcomings of various open source energy monitoring software choices, and in particular Home Assistant. In my opinion, how accessible it is to a beginner is not a valid criticism but is a valid observation. In contrast, how good its documentation and support offerings are is eminently fair game for criticism since those are shortcomings and not just related to target audience.

Nonetheless, I suspect the Renewable Heating Hub could potentially create a very useful resource - particularly for newcomers - if a more comprehensive and broader-scoped article were to be written about monitoring of renewable energy systems. However, my feeling is that it would need to start from a point of "this is what is normally available to the new owner out of the box, this is what you can do with it and these are the limitations". From that starting point, the article could then explore progressive changes that could be made to improve on those various limitations and what those changes would entail both in terms of new hardware/software and in terms of what extra knowledge or familiarity the owner would need to gain to go down that path. Such changes could be related to better reporting, better reliability, more comprehensive monitoring, future-proofing or whatever, and I'm sure this forum can draw from a broad enough range of expertise in different fields to end up with good and accurate information. Moreover, the particular path @cathoderay has taken in this thread already provides a pretty comprehensive structure for several of those pathways.

Even so, whether @editor decides to commission such an article or set of articles, props should be given to @cathoderay for the results achieved so far. Well done.

@majordennisbloodnok - thanks as ever for your welcome thought and comments. I think this thread, and the 'journey' it documents, is a sort of necessary evil, I needed to go down this road to find out what doesn't work (and hopefully stop others chasing similar unicorns).

I've learnt for example that:

(1) manufacturer filtered data, whether on a local network or in the cloud, is often limited (we only get to see what they deign to let us see)

(2) in Midea's case, and very possibly other manufacturer's cases, getting networked data, either from the local LAN of the cloud is far from straight forward, in fact on many occasions it is beyond us mere mortals

(3) Home Assistant is overkill for simple monitoring and even basic controls. In fact it is worse than that, its code, as @derek-m put it so well, is written by python sadists, its forum is a Stygian bear pit where the Home Assistanistas forever torment the novices, and it is bloatware beyond measure, code without end. I discovered the other day its version of python (the main HA programming language) has over 1000 add-on packages (add-on modules that supposedly add extra functions). For heaven's sake...

(4) wired modbus connections are very do-able even with basic electronic DIY skills. If that is too much for a particular beginner, an electrician could make the connection in 20 minutes plus any time needed to route the cable through the house. Both Midea and Samsung and I am sure many other brands of heat pumps have modbus built in, meaning this solution is a genuine generic solution.

I am going through the final throes of completing the code to collect the data. This is Midea specific, but it should be easily adaptable to other brands of heat pumps. I might even get it completed by 1st April, my original 'do or die' deadline date for getting something that works. 

As a taster, here is a chart showing recent hourly energy in/out and COP done using modbus and simple python/spreadsheet code (sorry it is a bit washed out): 

In due course, I will do a proper write up on how to do this.    

Posted by: @cathoderay

↑

Home Assistant is overkill for simple monitoring and even basic controls.

If your endgame, is simple monitoring of just your heat pump, then yes there are many other better alternatives.  My suggestion is that you just shell out the money for something relatively "turn key" such as the open energy monitor heat pump kits and be done with it.

Posted by: @cathoderay

↑

I discovered the other day its version of python (the main HA programming language) has over 1000 add-on packages

I would be very interested in why this is an issue.  As an analogy, the builder who built my extension, did not make the bricks for it on site, he just bought them and laid the bricks, ditto for the windows, tiles, reinforcing bars etc.  A modern car likely has upwards of 10,000 components, most of which are the equivalent of "libraries".  Toyota does not make its own transistors, and likely does not make any of the components of the braking system, buying those from Bosch.   A car in which every component was built from scratch at low volume by the car manufacturer would be insanely expensive and of very low quality.

I would much rather use a mature library that is very widely used as it is likely to be much more robust and feature rich due to the code coverage and feedback that results from that use, than a system that is 100% bespoke.

Posted by: @cathoderay

↑

[HA] is written by python sadists ....

HA is made up of a core product, and many many plugins developed by a pretty dedicated community, much of the development is done for free, and no one using the product is required to pay for it.  I think it very unfair to make very broad based statements against a group that puts a huge amount of effort into something as robust and powerful as HA, and makes it available for free.

Posted by: @cathoderay

↑

its forum is a Stygian bear pit where the Home Assistanistas forever torment the novices,

I have never personally experienced that.  The HA community forum has a very clear code of conduct and a mechanism for reporting abuse.  Of course, we are talking about humans here, so there will likely be the occasional person who may not be very nice as well as a grey area where low level "abuse" may happen. 

@william1066 - it goes without saying (at least I hoped it did) that what I write about HA is based on my observations experience and opinions.  

Posted by: @william1066

↑

If your endgame, is simple monitoring of just your heat pump, then yes there are many other better alternatives.  My suggestion is that you just shell out the money for something relatively "turn key" such as the open energy monitor heat pump kits and be done with it.

You forget I may actually like the challenge of making something work. I also like the challenge of keeping costs under control. For me, third party monitoring kits aren't a necessary solution for achieving what I am currently trying to do. They may however have a very important role if I decide that current energy out data is suspect (I already have an independent third party meter monitoring energy in). As I have mentioned before, relying on a manufacturer's software for performance data is in effect allowing them to mark their own homework, and we know where that ended in another sector (dieselgate). I'm just applying sensible scepticism.

Posted by: @william1066

↑

I would be very interested in why this [1000s of added packages] is an issue.

It is not an 'issue' for me, it was an observation to back up my description of HA as bloatware. My own python installation for example has 82 packages, and that is after adding quite a few. One of the things I do like about python is that very often someone has already done what you want to do before, and written a package for it. The minimalmodbus package I am currently using to get the Midea data is a good example (it's also a good package, and well documented). But the key thing is it is one package, not dozens. If it had turned out it had loads of dependencies, I would have uninstalled it. I have a natural aversion to bloatware, partly because it smacks of lack of discipline, and partly because it maintenance hell much more likely.

My personal experience of HA is that it is not robust. It has on more than one occasion destroyed data on it's own. My PC was either off at the time, if it was on, I wasn't doing anything HA related. The logs aren't that helpful, entries that amount to database corruption, but no clues as to how that happened. I did subsequently discover the corrupt databases do have data in them, but it is slog getting it out, and worse, you can't add it back into HA's database (see threads on the HA forum where people have asked about adding historical data, which is what the recovered data is as far as HA is concerned: it can't be done. HA only knows about current time, the past isn't a foreign country, it simply doesn't exist in HA's world.

Posted by: @william1066

↑

HA is made up of a core product, and many many plugins developed by a pretty dedicated community, much of the development is done for free, and no one using the product is required to pay for it. 

I appreciate that, but neither is anyone required to use it.

Posted by: @william1066

↑

I have never personally experienced that [the Stygian bear pit].

My observation (which I am not alone in making) about the hostility on HA's forum (Stygian bear pit may include a bit of journalistic licence) is based on my observation of the experience of others. I do find it depressing to come across such a high proportion of posts (not all by any means) where the OP asks a reasonable question and the HA experts react in rather contemptuous tones. Worse still from a practical point of view, viable solutions are rare.

The bottom line is that based on my experience, HA has not worked for me, and I do not recommend it. This is a pity: it does have great potential, and some things, like the history explorer card are already good. As I am putting that opinion (HA not recommended) in the public domain, in the particular context of a beginner's guide, I think I should add some explanation of why it hasn't worked for me, and why, in this context of a beginners guide, I suggest it is not the way to set up a heat pump monitoring system. Perhaps at times i do write a bit flamboyantly, but that is just my style, and like a leopard, I cannot change my spots.

On a more positive note, I am getting close to the point of thinking I have set up a robust enough modbus/basic python monitoring system that I can go public with. I still don't know about longevity (only wrote the code over the last week), and need to do something to deal with if (when) it crashes, and what happens in the event of say a power cut. But there is some slack in the system for that. The two python files I have currently written come to a total of less than 9Kb. See my 12:15 post above for a chart of the collected data to show the sort of thing it can already do. I think it is probably over-egging the COP a bit, possibly because Midea's energy in may only include compressor energy use, and my calculated kWh use definitely only includes compressor use, because that's what comes down the modbus telegraph line. I'm hoping I can add a correction factor once I have a reliable estimate of the difference between the internal Midea data and the actual external kWh meter data.         

Posted by: @cathoderay

↑

I think it is probably over-egging the COP a bit, possibly because Midea's energy in may only include compressor energy use

I posted some photos of the minute by minute ASHP circuit power readings during the startup of the heating a few days ago. I doubt it is very helpful to include anything other than the actual pump consumption when calculating the COP. I observe background consumption of 8-10 watts including the wired controller. When the secondary pump for the rads starts   there is an extra 140-150W. 

More interestingly, as the heat load from the rads decreases below 70% of rated maximum, the COP drops. At about 35% it is down to 3.6. Shortly afterwards the compressor stops. I suppose if it is cold this won’t happen. I can’t see the point of running the ASHP 24/7 when the load is so low. Just my opinion for my house etc.

Phil