@cathoderay I'm not that type of technical. However if you want to show me what to do in happy to recreate your progress on my system at home. It might be useful as a control for your tinkering and improvement. You've done amazingly so far!! My big question, is the app rounding up the power usage?!?

Posted by: @batalto

↑

My big question, is the app rounding up the power usage?!?

Thanks @batalto. To answer your question, I think it must be, but it is nothing to do with the app, this is all being done in python and javascript on my local PC, with the data retrieved locally from the wired controller via wifi. The reason I think it is rounding is that actual data retrieved (from the wired controller via wifi by the midea_ac_lan component) is total kWh used, and that number is always a whole number currently 7,954 kWh. The actual data recovered from the HA sqlite database looks like this in csv format:

15/01/2023 12:00,7949
15/01/2023 12:57,7950
15/01/2023 13:39,7952

The History Explorer javascript then does some jiggery-pokery maths on it to get power consumed over certain intervals, but I think it is rather basic. If you hover over those bars in the chart you get date/time and usage, and the date/times are the above 2nd and 3rd csv lines floored to the nearest five minutes, and the usage is current total minus previous total and because the totals are whole numbers, the result is a whole number. Although the chart says it is showing ten minute interval data, each bar actually shows usage since the last bar. This is why it is so useful being able to download the data in csv format, without having to use fancy sql queries on the underlying sqlite database.

The underlying problem interval usage is calculated from the the data retrieved from the Midea controller, which is total lifetime energy use, and that is always in whole numbers, and furthermore it isn't at regular intervals. Here's a filtered snapshot of the contents of the states table in the HA sqlite database (\config\home-assistant_v2.db, which logs just about everything, including when the dog last farted if you have the right sensor) showing the whole number kWh values in the 'state' column, and data/time stamps on the right:

The setup to get the midea_ac_lan out of the box sensors is fairly straightforward, instruction are on the component's github page, but basically you:

1. download the release zip file of the component, unzip and add it's folder to your HA custom_components folder

2. it should then appear as an addable device in HA, add it (I let it auto-detect my Midea wired controller, which it did successfully)

3. you then add sensor entities to the device. Unfortunately, the only out of the box one relevant to use is the DHW tank temperature.

You have to hack the midea_ac_lan code to add other custom sensors. This is not for the faint-hearted, but is worth it because if it can be achieved for other sensors, then we will have locally retrieved data without ever going near the Midea servers, and that blasted Midea app can be deep-sixed. The 'instructions' for total energy consumed are here (<=link), the only comment I would add is you don't need to modify the doc/c3.md file because it isn't present locally. Even with pretty rudimentary understanding of python, you can see what is going on, but the mystery bit is the code I quote above, specifically this bit:

self.total_energy_consumption = (
body[data_offset + 4] +
(body[data_offset + 3] << 8) +
(body[data_offset + 2] << 16))

Body is the message body that comes back from the Midea controller, a meaningless string to human eyes, the code pulls the data out of it. How did the author know which offsets to use??? Why do three of them need to be added together??? And why the << bitwise operators??? 

PS meant to add, the wired controller has current draw and voltage, if they can be retrieved, then we can calculate average amps over the hour etc and so watts, which should avoid the rounding errors, but it will depend on having frequent enough readings.              

@cathoderay

I am not familiar with Python, but I do have experience with data storage and transmission. To store a number, be it an integer or floating point, requires several bytes. A number is therefore transmitted as several sequential bytes of data. The data_offset + 4 is probably the least significant byte of the whole number, data_offset + 3 would therefore be the next least significant byte, with data_offset + 2 being the most significant byte. When the three bytes are reconstituted in the correct order, they will provide a 24 bit number. This I believe is indicated by the 'bitwise' operator, which positions the bits in the correct position before adding together. The most significant bit is normally used to denote whether the number is positive or negative. 

self.total_energy_consumption = (
body[data_offset + 4] +
(body[data_offset + 3] << 8) +
(body[data_offset + 2] << 16))

@derek-m - I've tried to do some debugging on this, and managed to get the values for the data offsets:

class C3Notify1MessageBody(MessageBody):
def __init__(self, body, data_offset=0):
super().__init__(body)
self.total_energy_consumption = (
body[data_offset + 4] +
(body[data_offset + 3] << 8) +
(body[data_offset + 2] << 16))
self.test01 = body[data_offset + 1]            gets 0
self.test02 = body[data_offset + 2]            gets 0
self.test03 = body[data_offset + 3]            gets 31
self.test04 = body[data_offset + 4]            gets 72

31 << 8 = 7936 (from an online calculator)

7936 + 72 = 8008 - which amazingly is exactly the same as my current total energy use!

But apart from getting the right result, it is still all Ancient Greek to me. Someone somewhere must have access to the data/tables/descriptions that identify what is where in the message bodies, and how the actual reading value needs to be extracted from the data. 

Posted by: @cathoderay

↑

@derek-m - I've tried to do some debugging on this, and managed to get the values for the data offsets:

class C3Notify1MessageBody(MessageBody):
def __init__(self, body, data_offset=0):
super().__init__(body)
self.total_energy_consumption = (
body[data_offset + 4] +
(body[data_offset + 3] << 8) +
(body[data_offset + 2] << 16))
self.test01 = body[data_offset + 1]            gets 0
self.test02 = body[data_offset + 2]            gets 0
self.test03 = body[data_offset + 3]            gets 31
self.test04 = body[data_offset + 4]            gets 72

31 << 8 = 7936 (from an online calculator)

7936 + 72 = 8008 - which amazingly is exactly the same as my current total energy use!

But apart from getting the right result, it is still all Ancient Greek to me. Someone somewhere must have access to the data/tables/descriptions that identify what is where in the message bodies, and how the actual reading value needs to be extracted from the data. 

I assume that the number is represented by 4 bytes in Hexadecimal format, with decimal 8008 appearing as Hex 00001f48. It may therefore be possible that all the number are transferred in the same format. It could be that all you need to do is look for 4 bytes, where the two most significant bytes are 0000.

If so, then use the HEX2DEC converter as described below.

One way that you may be able to decipher some of the data stream is as follows:-

Extract the first 4 bytes from the data stream.

Use the HEX2DEC converter in Excel to convert to a decimal number.

Check the decimal number obtained against those displayed by your Midea controller, to see if you obtain a match.

If no match is obtained then extract another 4 bytes, starting 1 byte to the right from the original starting point.

Repeat the above procedure and hopefully you will be able to identify the location of some of the Midea data.

If you are not certain what to do then send a copy of the data along with the data displayed on the Midea controller.

@derek-m - I partially understand what you are getting at, some of it is beyond stuff I have done before, but I can always learn.

Here's two adjacent entries in the debug log:

Received: {'header': 'aab9c300000000000004', 'body': '040100001f4f0000521f02231e323031ff01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000ef000000000000000000000000000000000014', 'message type': '04', 'body type': '04'}

Status update: {'total_energy_consumption': 8015}

The first one contains, I think, the undecoded hex string as received, the second one is the current Total Energy Consumption, recovered from the hex string, and the value, 8015, agrees with what is currently showing in the Midea wired controller. It looks to me like there is more data in the hex string than has been extracted.

8015 decimal is hex 1f4f and that is in the string above, starting at position 9. I can't believe that is a coincidence. But then why not just do something like 'hex2dec(mid(data,9,4))' instead of all that complicated bitwise stuff? Here are all the other groups of four characters converted from hex to decimal up to the long sequence of zeros:

0401    1025
0000    0
1f4f      8015
0000    0
521f     21023
0223    547
1e32    7730
3031    12337
ff01      65281

none of which looks like a credible reading, apart from the 1f4f/8015 one... Mostly we are looking for temps in the 0-100 range, or flow rates current draw etc and none of the decodes look like that. The other problem is there is a time lag between the data appearing in the wired controller and in the debug log, due to the polling interval. I have not been able to find out what sets that polling interval, but it is long enough to mean more volatile values will have changed by the time I have extract them from the log and done the hex2dec conversion.  

Posted by: @derek-m

↑

It may therefore be possible that all the number are transferred in the same format.

I strongly suspect this is the case. The routines for extracting human readable data all basic look as if they are doing the same sort of thing, see the code at https://github.com/georgezhao2010/midea_ac_lan/tree/master/custom_components/midea_ac_lan The key files are midea_devices.py in the top level directory, and device.py and message.py in the midea/devices/c3 sub-directory. c3 is I think short for 0xc3, a hex code that identifies heat pumps. 

@cathoderay the midea app does track hot water and heating separately. Likewise its also tracking heat delivered. Could some of those numbers be those? Likewise its probably tracking it own operational hours. Might be best to work thorough the controller to find values close to, be not exactly the same as some of those - they might be good suspects.

Posted by: @batalto

↑

Could some of those numbers be those?

I don't think so, not in the right ball park. The problem with monitoring the controller is a lot of the key variables are very volatile, though they do tend to be within a range, eg the LWT/RWT. I've spent ages going through the midea_ac_lan python code, and just can't see how or where they send the query, or how it is made up. The result that comes back is painfully thoroughly encoded/encrypted, Midea's own brand by the looks of it, again very difficult to see how to decode it without having the key to the encoding/encryption. It is just so tantalising, knowing it should be there, should be accessible, but isn't.

Remember none of this is happening through the app, it is all local on my PC, and the app doesn't get involved. That said, the HA/python approach is using the same data, so it would be rather surprising if when they do show the same thing, the values were different! 

I may have to resort to using modbus, effectively doing over wires what I am now trying to do over wifi. At least we have the address registers for modbus, and as far as I know, the data isn't encrypted. Someone managed to do something very similar with Grant (Chofu) ASHPs and modbus, see here:

https://github.com/aerona-chofu-ashp/modbus

It may be most heat pumps use something similar, given Grant, Samsung and Midea do.   

Here's another cryptic bit of code for anyone to have a go at decrypting. It's from the Air Conditioning section of the code, but it is retrieving something of interest to ASHP owners, outdoor temp. I'm wondering if this can be adapted for use in the ASHP code, to get outdoor temp from a heat pump:

if body[14] == 0xFF:
   self.outdoor_temperature = None
else:
   temp_integer = int((body[14] - 50) / 2)
   temp_decimal = (((body[18] & 0xF0) >> 4) * 0.1) if len(body) > 20 else 0
   if body[14] > 49:
      self.outdoor_temperature = temp_integer + temp_decimal
   else:
      self.outdoor_temperature = temp_integer - temp_decimal

I get the first if - if body[14] (item 14 in the array body?) is 0xFF, then outdoor temp is none, else...

But the rest??? Seems to get left and right side of the decimal point and then add/subtract the decimal to/from the integer. I sense the -50 and >49 might be clues, but can't see how.
  

@cathoderay The value 0xff is equivalent to 255 in unsigned decimal, -127 in signed decimal, and 11111111 in binary - So I am guessing thats an error state?

0xF0 = 240 

len is the function to find the length of a string of text/number. So if its longer than 20 use the formula, else its zero.

Have you tried extracting the "body" (if you can do that), to see what it contains?

@batalto - I agree with your observations, my level of understanding of code gets me that far, eg the len property/function. But why does it need to use the length of the string? Normally it would be something like if len > 0 then formula else zero etc. My guess on 0xff is it's the highest possible value, and so is being used as a switch (case) sort of thing, in this case to say the value is none (nothing). But it's the bizarre maths/operators that have really got me baffled.

Body is the message body, which arrives heavily encrypted, I think it is the strings I have posted previously. For example, a 50 character body string yields 25 name:value pairs in what may be json format.I can see code that does the encoding/encrypting and decoding decrypting, but can't make out what exactly it is doing. And still no hint of the way of working out what is where in the 'body', and how to process it to get the right numbers. Why can't outdoor temp in the above example just be the raw numbers? Split into integer/decimal if need be, so 11.24 arrives as 11 and 24, but no absurd encryption. This data is hardly the key to launching nuclear missiles!

That said, I have found a fork of the midea_ac_lan project that looks as though it has done a bit more. HA blew up when I tried to substitute it for the original midea_ac_lan code, I think too many names overlapped. I will have a clear out on HA and see if I can get it working. The fork has recent activity on it, I am hopeful I might be able to get all the key variables included, and then we will have the option of app independent data collection from our Midea controllers.

@batalto - I have now got total energy produced as well as consumed into HA, and it looks like this, first image is a card, second is from the History Explorer (HE):

Some comments:

1. The card: I am confident the total  energy consumed is right, it matches the value in the wired controller, but I am not sure how credible energy produced is, it gives me a lifetime COP of 2.62, which I think (and hope!) is lower than it really is. Unfortunately the wired controller doesn't have a total lifetime energy produced reading, so no way of checking it. The combined COP from daily readings on the app since I started recording them on 25th Nov 22 is 3.38. The DHW tank temp matches that on the wired controller.

2. The charts: HE joins and smooths the dots for the DHW, and they don't necessarily make sense for a period when my PC was off eg overnight. But overall, the DHW is about right, eg it shows the water heating up between 1300 and 1400 daily as it is set to do on a schedule. The big dip yesterday was me having a bath, after which the temp is lower than it should be, I think because the sensor is halfway down the tank (I still had usable hot water, despite the chart showing around 20 degrees, the water coming out of the taps was certainly hotter than that yesterday evening.

3. The two energy bar charts work by subtracting the immediate past hourly reading from the current reading so although it says total in the headings, they are actually hourly readings - except when my PC is off. That is why the bar at the start of each period is so large, the value comes from current minus when my PC was last on, which might have been 12 or more hours ago.

Nonetheless, I have some data, but it is not clear how reliable the energy out data is. I have left a post on the github page of the developer who added the energy readings asking if he can add, or reveal how to add, LWT/RWT/flow rate so we can independently calculate the energy out on an hourly basis.

Readings will be 24/7 once I have a dedicated HA box, still dithering about what to get.