@editor - even just thinking about it makes me go weak at the knees. But there is some good news in the pipeline. This morning I managed to get a wired modbus connection set up, and the data collected logged to a csv file, thus completely bypassing HA and its merry band of delinquent add-ons. I might yet have the whole up and running by my deadline, 1st April. Costs are still well controlled, and the modbus route gives access to far more data. It is also potentially more generic, in theory any heat pump with a modbus connection should be able to use it.

@cathoderay the beginners commentary for HA , if it was to be written up as a full end-2-end solution , would need to include things like IP addressing, to make it beginner-able. The extensibility and scalability of HA has a lot going for it, I know you had bad experience but there are several others on here who've had good. Yes your right, if you end up needing to using a code thread where the members aren't desperately helpful, or committed, you can lose heart - but they aren't all like that. And sometimes you figure it out yourself and says to yourself "Hah, didn't need them anyway...."

My apologies, when I pointed you at HA a month or two back, I must have assumed (without even realising that it was an assumption, the worst kind in project-land) that IP addressing awareness on your part was a given. I design IP networks for a living, so in my case, it is. Everything that matters on my home network has a static IP.

I'm now running HA to control my heating (still the boiler, for various reasons) and intend to run it for the ASHP as soon as I am done. there will be a fallback to something analogue though.

Posted by: @iancalderbank

↑

would need to include things like IP addressing, to make it beginner-able

Thanks for your comments, much appreciated. I did in fact go though IP addresses at various points, on the assumption (this time perhaps OK!) that beginners can't be assumed to know about it. Interesting to read that, for you, "Everything that matters on my home network has a static IP" - that goes a long way to removing my concerns about using static addresses, stemming from all the dire warnings on the web.

I am sure that in a 'community' as large as the HA community that there are good eggs among the bad, but often the environmental odour situation (another Clive James-ism) from a bad egg can overpower the sweeter smell from the better eggs. Another problem I think is the "extensibility and scalability of HA", it does indeed have a lot going for it, but it can also be overkill, and more importantly overwhelm. Using modbus over a wired connection with my own ultra simple code removes whole swathes of complexity. My current python code has about 60 lines, and some of them come from the way I have formatted it (multi-line code for readability). Compare that to the ESPHome in HA yaml file I mentioned earlier in another thread, which runs to 2441 lines. Another problem is yaml is yet another 'language' (more accurately, format?) to learn, and to make matters worse it comes in many dialects.

One of the big take homes for me from the last 48 hours or so is that it is possible using a wired connection outside HA to do in hours that which proved impossible in HA using midea_ac_lan even after weeks of effort. OK, I've got a bit more coding to do, notably deriving hourly values, and logging them, but I don't anticipate spending weeks on it.        

Posted by: @cathoderay

↑

Posted by: @iancalderbank

↑

would need to include things like IP addressing, to make it beginner-able

Thanks for your comments, much appreciated. I did in fact go though IP addresses at various points, on the assumption (this time perhaps OK!) that beginners can't be assumed to know about it. Interesting to read that, for you, "Everything that matters on my home network has a static IP" - that goes a long way to removing my concerns about using static addresses, stemming from all the dire warnings on the web.

I am sure that in a 'community' as large as the HA community that there are good eggs among the bad, but often the environmental odour situation (another Clive James-ism) from a bad egg can overpower the sweeter smell from the better eggs. Another problem I think is the "extensibility and scalability of HA", it does indeed have a lot going for it, but it can also be overkill, and more importantly overwhelm. Using modbus over a wired connection with my own ultra simple code removes whole swathes of complexity. My current python code has about 60 lines, and some of them come from the way I have formatted it (multi-line code for readability). Compare that to the ESPHome in HA yaml file I mentioned earlier in another thread, which runs to 2441 lines. Another problem is yaml is yet another 'language' (more accurately, format?) to learn, and to make matters worse it comes in many dialects.

One of the big take homes for me from the last 48 hours or so is that it is possible using a wired connection outside HA to do in hours that which proved impossible in HA using midea_ac_lan even after weeks of effort. OK, I've got a bit more coding to do, notably deriving hourly values, and logging them, but I don't anticipate spending weeks on it.        

if you get static IP's wrong its a disaster. Stuff won't work and you can spend hours trying to figure out why . You must separate your network so that the range for statically assigned devices is not part of the DHCP range. A useful hybrid is to reserve (and thus make constant and the same functionality as static) DHCP assigned addresses for the important devices using the DHCP server. Most routers have the capability to do this.

Sounds like you got a bad starting point with the midea code. The code for reading samsung modbus via HA+ESPHOME that I've grabbed from others on this forum is a 100 odd lines of YAML, easy to follow and worked first time.

Posted by: @iancalderbank

↑

A useful hybrid is to reserve (and thus make constant and the same functionality as static) DHCP assigned addresses for the important devices using the DHCP server. Most routers have the capability to do this.

Thanks for reminding me, that is what I did. I recall also having a moan about the fact that the HA static ip address documentations totally fails to mention that using an IP address in the DHCP (local LAN range, what i think of as the 192.168.x.x addresses?) is a no no, see near the top of this page under "Part 5: Dealing with a Major Flaw" for what I did and next few posts for my moan. I think the advice I have given (set a fixed/reserved ip address via your router) is the right advice, having just modified it to say don't use a static address, but perhaps I can improve it. I'd welcome your thoughts on that.

Edit 1807: arrgghh! had two edit windows open at the same time, saved one and lost the other, now corrected.

@cathoderay if you've set a reserved IP address for the HA machine, using the DHCP configuration page of your router, then that will take care of it for you and is the safest way for a non network techie. stick with that.

192.168 is just a part of a set of IP addresses that are reserved for private use, they are neither DHCP nor non-DHCP nor LAN nor non-LAN.

@iancalderbank - thanks again for the clarifications, others will no doubt benefit too. Brief summary:

DO use a reserved IP address set in your router

DON'T use a static IP address set in HA or anywhere else

Sorry to say, @cathoderay, but this whole thread in my mind poses something of an oxymoron. I mentioned in a previous post my concerns about calling this a beginner’s guide, but I think I now need to reiterate it.

The starting point for a beginner should be to understand what’s available with the initial installation, usually meaning a manufacturer’s cloud-based app or web site. I also think that anyone finding the limitations of that will have to accept they need to do get a little more hands-on and that means learning stuff.

You’re more familiar than me with modbus, so you’re happy to consider it “beginnerable”. However, I’m far more familiar with IP addressing than you, so what I would consider “beginnerable” is at times stuff you are berating HA for as too complex. It’s not HA’s remit to sort out your network, but if it’s going to exist on your network then it should have at least the configuration options any other network device would offer - your printer, your router, your PC or phone all have the ability to be given an address dynamically or told to use a static one, and the choice as to what’s appropriate depends on what you’re trying to achieve.

You’ve just simplified IP addressing down to reserved = good, static = bad. Trouble is that some people may read that and take it as gospel. There’s nothing “wrong” with using static addresses; you just need to know enough to do it right.

In summary, I think you’re still trying too hard to present this for beginners when the whole scenario of local collection and monitoring of data requires an assumption of basic understanding in enough different concepts that it’s already worthy of the “intermediate” classification at very least. And, if that’s so, expecting the installer to have a basic idea of the network settings required is not a big ask, even though I agree with you about the need for readily available documentation.

@majordennisbloodnok - all good points, as ever. I think part of the problem, perhaps the main problem, is that I made an early decision to present all this as a 'journey', the long long road with many a winding turn, with 'postcards' sent from the various ports of call along the way. My reason for choosing this serialised narrative format was that it would allow me to show my reasoning along the way, why I selected this and not that etc. That approach may well have been a mistake with hindsight, because it also reveals all the messy stuff as well.

In the last few days I have added the ability to use a modbus connection over a wired connection, and written my own python script to poll the wired controller every minute and log the data to a csv file which can be opened, manipulated and charted in any spreadsheet. I have learnt that all said and done a wired modbus connection is almost infinitely simpler and easier to use than a network, be that network local or in the cloud. In Midea's case, and I suspect for many other brands, the second problem with using these network based approaches is you only have access to what the manufacturer chooses to send or make available over the network, and in Midea's case this is further complicated by the lack of documentation, deliberate obfuscation and use of atomic bomb levels of security. Using modbus bypasses all that, and is far better documented, with tables describing which register address holds what data. I've managed to do over a few days (maybe 12 hours back to back) what I was unable to achieve using networks even after months of work.

I've just very recently set up the python script on the mini PC and got it running (on the Home Assistant OS for now, but it is running totally independently of HA). Here's the first few lines of midea_data.csv, the file logging the data. The column headings at the top are largely self-explanatory:

datetime,on_off_bin,htg_on_off,dhw_on_off,RWT,LWT,ambient,room_temp,dhw_temp,amps_in,volts_in,capacity,setLWT,flowrate,ability,energy_in,energy_out
2023-03-21T11:58:18,10000001101000,1,0,36,40,11,25,39,5,245,14,39,1.35,5.11,11344,30915
2023-03-21T11:59:18,10000001101000,1,0,37,40,12,25,39,5,245,14,39,1.36,5.34,11344,30915
2023-03-21T12:00:18,10000001101000,1,0,37,40,12,25,39,5,245,14,39,1.35,5.11,11344,30915
2023-03-21T12:01:18,10000001101000,1,0,37,40,12,25,39,5,242,14,39,1.34,5.23,11344,30915
2023-03-21T12:02:18,10000001101000,1,0,37,40,12,25,39,5,241,14,39,1.37,5.46,11344,30915
2023-03-21T12:03:18,10000001101000,1,0,37,41,12,25,39,5,243,14,39,1.35,5.58,11344,30915
2023-03-21T12:04:18,10000001101000,1,0,37,41,11,25,39,5,243,14,39,1.36,5.93,11344,30915
2023-03-21T12:05:18,10000001101000,1,0,37,41,11,25,39,5,244,14,39,1.37,6.04,11344,30915
2023-03-21T12:06:18,10000001101000,1,0,38,42,12,25,39,5,244,14,39,1.37,6.28,11344,30915
2023-03-21T12:07:18,10000001101000,1,0,38,42,11,25,39,5,244,14,39,1.37,6.62,11344,30916
2023-03-21T12:08:18,10000001101000,1,0,38,43,11,25,39,5,244,14,39,1.34,6.74,11344,30916
2023-03-21T12:09:18,10000001101000,1,0,39,43,11,25,39,5,245,14,39,1.38,7.32,11344,30916       

The second column, on_off_bin, is a temporary work in progress record of the binary output that has the heat pump mode (space heating or DHW heating) in it. Capacity (fixed at 14) is I suspect the unit's nominal capacity, and ability is I'm pretty sure midea-speak for current output power in kW.

Over the past few days I have come to appreciate that my earlier forebodings about Home Assistant, that it would massively over-complicate things, have been realised. What I have done over the last few days has in contrast been remarkably simple both in concept and in execution. I really am coming round to the belief that it is suitable for a 'normal beginner' (whoever that fictional person is). It also has another major advantage. All my previous efforts were largely tied to Midea (and Midea clone) units, because I was using Midea networking and coding protocols, potentially limiting its generalisability. On the other hand, the modbus approach is far more generic. If your heat pump has a modbus connection (as many do), and you have a table with the register addresses, then you are practically there. The python script is short, comprehensible, and easily modified to suit other heat pumps' data. And finally, it has all been very affordable.

Modbus isn't esoteric, it's a very easy to get your head round way to read data from (and also control) almost anything. It is a cousin of the old serial connections and current USB connections, and they are all serial connections, and all that means is that the data is sent one bit after another, ie in serial, as opposed to a parallel connection. Over a wired connection - literally just two wires for data and one for earthing - it is extremely robust. It really does have a lot to commend it.     

Full write up to follow, but even at this stage I am now 99% sure that a wired modbus connection is the way to go.   

@cathoderay I just think that you've perhaps got a bee in your bonnet about HA and the sleepless nights its caused you and conflated matters..sorry, nothing personal. I too have a wired modbus connection to my heat pump. From an ESP controlled by HA. It worked first time.

I would suggest that this thread is retrospectively retitled "a Treatise on various experiments into methods of ASHP monitoring" and then at some point in the future once there are some clear solutions (perhaps more than one, with options) those could be written up in beginner friendly form, under the original title?

@iancalderbank - I have got a bee in my bonnet (actually it is more of a hornet's nest) about HA, based on my experience, but I fully accept that others will have found the HA experience to be as smooth as Horlicks. The reason I make such a noise about my experience is that this thread started life as a beginner's guide, and I want beginning folks to be in no doubt that there is a very good chance HA will drive them insane. Nor am I the only one who has found HA to be impossible and frustrating: there are over 100,000 posts (or maybe threads, who knows, this is HA) on the HA forum, and the vast majority of them are about HA not working, and the vast majority of those never get a solution. I don't know what the denominator is (the total number of people using HA), but my hunch is that a significant proportion have problems.

Note on the denominator (total number of active HA installations worldwide): short answer: no one knows, because there is no way of knowing. Slightly longer answer is around 234,000 active signed up to analytics installations (see here). It's anyone's guess as to whether your average HA user will be keen to sign up to anayltics: concerns over sharing data doing battle with the love of big data.

Posted by: @iancalderbank

↑

I would suggest that this thread is retrospectively retitled "a Treatise on various experiments into methods of ASHP monitoring" and then at some point in the future once there are some clear solutions (perhaps more than one, with options) those could be written up in beginner friendly form, under the original title?

I think this is probably a sound idea. Treatise however might be rather too grandiloquent a word for the incoherent ramblings of a beginner whose brain has been turned into useless putty by the HA sadists...

Posted by: @cathoderay

↑

 is that this thread started life as a beginner's guide, and I want beginning folks to be in no doubt that there is a very good chance HA will drive them insane.

     

given what this thread evolved into, the end result in a new thread for beginners would be a write up of "here's the set of things that works for different goals, pick the one you want, here's the  method to get there". 

opinion: If people are wanting to improve on manufacturers monitoring at little/no cost they will be prepared to tolerate some "fiddling about" and not expect commercial standard fully working / tested/ documented. HA is a really good base for doing that kind of thing. But on the other hand , it greatly helps someone trying to do that if someone non-beginner has done the painful experimenting bit (yourself, in this case) and written the answer down for the beginner to start from (the method the experimenter used to get to the answer doesn't matter to the beginner).

So I disagree somewhat on "HA will drive you insane". If you are following someone's previously written down method , to get HA to do a thing that its already known that it can do, then most of the people who are going to want to try this kind of thing, will be able to get there. The difference is they WONT be inventing as they go along (which I fully agree, can well drive you insane). I'm only suggesting that you try to separate the two things that you've had to do, from scratch in a short space of time 1: "getting HA to a point that you understand how to make it dance on its head" - which is hard and 2: "debugging difficult midea stuff" which is double-hard. Both of which I give you a huge "well done" 👍 for persevering with. but neither of those are tasks that a beginner would take on.

to give a reasonably comparable analogy from my point of view:

I'd already done the HA learning stage 2 years ago. I'm not a ninja but I'm comfortable. So that part - not an issue any more.,

Recently I tried to get samsung modbus working with HA. Other people have already figured out how to get modbus working with samsung, some with HA, some not. There are pages of discussion and code on other forums, that I have tried to read but do not understand. Those people have published "the answer" (sometimes work in progress, but definately a working answer), I have taken their answer and pasted it into my system. It worked for me first time. I am eternally grateful to those people. now all I (as a modbus-samsung beginner) need to do is work out how best to use it for my needs (which I am fine with doing) , and occasionally ask (mostly in this forum) "how does the next bit work". I've not had any of the pain of how to make the bloomin things talk to each other in the first place - which a beginner wouldn't either. so I think (in a little while) I'd be in a  position to write up "beginners guide to samsung modbus monitoring"