Posted by: @derek-m

↑

I presume you have carried out a virus check?

The HA mini PC is behind the router firewall, which I think should should keep nasties out, and I only have access to it via a local IP address (192.168 etc etc), meaning it shouldn't be visible to the outside world. @iancalderbank may be able to expand on this.

Also, so far, I haven't come across any warnings about HA suffering from virus attacks, doesn't really need them, it has more than enough ability to do itself in without external aid. 

Posted by: @derek-m

↑

Am I correct in assuming that HA is written in Python

Certainly large chunks of it are written in Python, and it is v3, probably v3.8 or 3.9. There is also some javascript, presumably needed for the browser interface.

Posted by: @derek-m

↑

If it is not too onerous, it may be necessary to wipe the disc and start again.

OSes that need the disk to be wiped more than once in a blue moon should probably just be wiped, and consigned to history. I am prepared to give HA another chance, now that I have established I can get most if not all of the data back from the 'corrupt' database files. The data seems to be there, all that has happened is that HA has thrown a hissing fit, and and lobbed its toys out of the pram.

I have also now set up a nightly backup to a remote server which means I should be able to recover HA and its data if/when it goes tits up again.

I've just used a spreadsheet to calculate the daily COP from the recovered data, and made a chart:

Credible (see below), so much easier than using HA, and what's more, the sums are totally transparent. I know what they do, because I did them.

Outdoor temps (degrees F) over the same period:

The COP increases in milder weather, decreases in colder weather. 

Posted by: @cathoderay

↑

I am prepared to give HA another chance

Given you had one failure on that disk [which I assume is a new disk], without really knowing the cause.  I would get a USB to SATA connector (for your HDD/SSD) and run the smartctl tools from your PC, make sure there are no problems with the disk.

Alternatively put Ubuntu on an 8GB USB stick, boot that (if your PC supports USB boot) and check the disk from there.

@william1066 - I have already run smartctl on the disk, see previous post at 12/02/2023 5:05 pm, 100% pass, and total lifetime hours confirms disk was new when I got it. I think it is very unlikely the disk is the cause of the database corruption. The reality seems to be that HA is often flaky, the myriad of forum posts attest to this, and then there is this from the developer of a widely used backup addon: "Home Assistant is notorious for failing silently". OK, he is plugging his addon, but no one has stepped in to say he is wrong. Maybe HA is just delicate: some installations land the right way up as it were, others have flaws that break from time to time. Sqlite has as you mentioned earlier a much longer and better history, and I can't think of any other sqlite based application I have used that regularly suffers database corruption. From what I can see, HA is holding the smoking gun.

Scene: The Apprentice Boardroom, Week 3. The candidates have had to design an app to monitor their heat pumps. Midea have managed to smuggle one of their reps in as a candidate.

Midea Rep: According to our app, our heat pump only used 226 kWh over the last week. 

Lord Sugar: Look, I've spoken to Karen here, to get the real figures. You're not making any sense. You're fired. 

And indeed it is so. Here are some 'interesting' figures comparing the Midea app's reported kWh used (App column) with those shown on the external dedicated heat pump meter (Meter column, which we may count as the gold standard): 

Until I did these figures, I only had the lifetime use, and it appears from those lifetime numbers that the Midea app over-estimates usage by 13%, a rather bizarre finding, as it would suggest the estimated COP is lower than it should be, hardly to Midea's advantage. But if we look at recent weekly use, a very different picture emerges. The app usually under-estimates usage, typically by about 4%, meaning the COP will be higher, and so better, than it should be. Go figure - or is that a whiff of dieselgate I can smell in the air?

Without independent third party monitoring tools, there is no way of knowing what the true output is, and how it compares to to the apps estimated output. 

@cathoderay 

I think it is more complicated and will be hard to investigate the differences.

Here is my 3-phase pump (6kW) power consumption over night running cycle, start scheduled 10PM:

in heat production production it is running on one phase, but every 40-50 minutes it is defrosting and using all 3-phases, so question does Midea calculation includes these defrosting periods on all 3-phases?

Then, does it include the indoor unit running power? according the per manual it is consuming approx 80W for running.
Would be interesting to identify precisely where and how power consumption is measured.

Don't have answers, just wondering.

Posted by: @marekk

↑

I think it is more complicated and will be hard to investigate the differences.

Exactly, we don't know what goes on inside the box, in particular how the controller works out power in and power out. As you say, what is/isn't included? What sensors are used? Are proxy measures used, rather than direct measurements? The only reliable answers are via third party sensors, complicated and expensive.

I do at least have the simplicity of single phase, not that is simplifies things very much...

The only evidence I saw that might have been a defrost cycle was a short dip in a rising DHW tank temp. It may have been a defrost cycle happening during a period of DHW tank reheating, during the defrost the heat pump took heat from the how water, when normally it would take it from the rads. The defrost cycles are not visible in any shape of form that I can see in the energy use charts, but as the resolution is effectively 1 kwh and 1 hour, I am not surprised. 

Even if the defrost cycles were in the data, they remain an intriguing case because they are a period in which energy is still consumed, but heat (energy) production is in effect negative, in that heat is being withdrawn from the house. If it is nonetheless treated as positive production, then it will distort the reported figures even more.  

@cathoderay Thanks for your earlier response. I have discovered two things:

1. The operational parameters accessible using the Midea panel include the current power and the output power computed from the flow rate and delta T (also available).

2. Freedom HP have a monitoring App and easy to fit interface available to installers, which does what you have been doing with HA and more.

My guess is that the Midea Cloud used by MSmartHome does the same except its accuracy may be questionable. The energy consumption may be based on the stream of data being correctly integrated. One could at least estimate the COP using the panel and see how it varies. 

Freedom HP also sell high grade energy measuring equipment suitable for charging in residential areas. 

I suspect the packet you have been decoding is just the operational parameters that the panel is continuously displaying. 

There are quite a few Freedom  HP YouTube videos that explain the some more details. 

I wonder whether under MCS installers should have provided more monitoring kit? Interested to hear the views of others.

I am interested to know how your CH circulation pump is controlled. Freedom HP show it controlled from the ASHP whereas my installer starts the pump when any zone thermostat fires. This means there is no overrun to remove heat from the plate heat exchanger.

Phil

@filipe - yes, there are a lot of potentially useful parameters on the operational parameter pages, but they remain only potentially useful for as long as the wired controller remains the only way they can be accessed. Even I am not going to set up a camp bed by the controller and manually write down the values every five minutes! Hence the drive to get the data automatically.

My hunch is that the data, wherever it appears, is the same. It just doesn't make sense to have parallel data sources and streams. In broad terms, I think the wired controller gets data from the heat pump, does some processing on it, and then (a) makes it available on the operation parameter pages and (b) sends it to the Midea cloud over the internet. Midea then have what they want (our data) and in return they make some of it available to us through the app. Optionally, we can also (a) get data wirelessly from the wired controller over our local area network using the midea_ac_lan custom component in Home Assistant, which is what I have been doing here recently, and/or (b) get data over a wired modbus/rs-485 connection to the H1/H2 terminals in the wired controller. I am sure as can be that this is what the Freedom 'incredibly easy to install' bit of kit is, it is a two wire modbus/RS-485 connection to the wired controller, with, presumably, some sort of wifi/internet connection that gets the data to their servers. Homely also use the same connection, and interestingly (proves it can be done) they control the heat pump through that connection. I've said it before, and I'll say it again, Freedom should allow heat pump users ready access to that data, either for free, or for a nominal one off installation fee (for fitting a something which, by their own words, incredibly easy to fit, if done at installation time it would take less than five minutes). But Freedom don't care about consumers, they just want to cosy up with installers, and sell more heat pumps, so they make their more detailed and much more useful cloud/app only available to installers. There is rumoured to be a cut down consumer app, but I have yet to see it for real, on that web page it looks like it is designed for 4 year olds. The installer cloud interface is visible after logging in here (<=link), you can sign up using any email/password and then see the installer dashboard which has no data (because you are logged in but not able to access any data), but nonetheless looks like this: 

It's all there, they just don't want us consumers to have any of it. 

That said, and going back to the data and more specifically data quality, wherever and however it is accessed, it is all still a black box. I do know the Midea reported energy consumed data usually under-estimates the amount used (which of course works to Midea's advantage, a scandal we might call compressorgate) but how do they measure it? Multiply the rather volatile comp. current (p6 of the Operational Parameters, parameter name suggests it only reports compressor draw, so will under-estimate total draw)) by the voltage (p7) and sum it over time to get power consum (p3)? This is lifetime consumption, it seems likely all interval values are done by simple subtraction, end value minus start value for the interval. How about energy produced (on p3, as heat pump capacity). The standard flow x specific heat x LWT/RWT delta T equation? If so, is any allowance made for antifreeze in the system, or does it assume 100% water? If the latter, then it will over-estimate energy produced when there is antifreeze in the system, again an error in Midea's favour.

Compressorgate is very simple: you just under-estimate energy consumed a bit and over-estimate energy produced a bit. Example (round numbers): real situation is energy in 20, energy out 60, COP of 3; under-estimate energy in by 5% and over-estimate energy by 5% gives energy in 19, energy out 63, giving a compressorgate COP of 3.32. Wow! It must be so tempting... 

There is no general MCS requirement for monitoring (another MCS blunder). The only time it is required, I think, is when you have RHI combined with a hybrid system, so an estimate can be made of what the heat pump has actually done (because perhaps the RHI shouldn't be paying for your gas heating...).

My secondary (rad) circulating pump is room thermostat controlled, exactly as it would be on a fossil fuel system. In practice, this means it is always on when the heating is on, because I have the room stat set high, and control the heating using weather compensation.

@cathoderay you might find your problems with HA stem from the deployment you're using. I'm not sure what 'HA OS' really is, how it's configured or how well it's run. I went for the self install option on FreeBSD (which isn't officially supported so wouldn't recommend on that basis, although it does all work) which has been bulletproof since install several years ago. Certainly never had a problem with the SQL database.

@timmcc - thanks, interesting. HA OS is Home Assistant Operating System. Originally, I thought it was supposed to be the purest/fullest way of installing HA, the OS being a linux kernel optimised to work with HA. I now see it as a crippled version of linux, stripped of core functions, because the developers think we need to be protected from ourselves. One serious limitation (which really does cripple it) is you are prevented from installing third party software. To run smartctl (to check the SSD for errors), for example, I had to boot from a bootable USB stick and run it from there.

I did look at installing it on a standard linux distro, but, as far as I could see, that always means running it in a virtual machine, and I have never liked them. I wonder, have you manged to install it on FreeBSD 'as is', without using a virtual machine? 

@cathoderay yes, I just installed it as a Python program on FreeBSD - you can do the same with Linux. I'm a big FreeBSD advocate, but you'll have an easier time with Linux because that's officially supported. Sometimes I end up needing to find workarounds for example to get functionality working on FreeBSD where on Linux you could easily install whatever library you need off the shelf.

The 'VENV' (virtual environment) install is the best method in my opinion if you want the full freedom to do what you want, but it comes with the cost of maintaining compatibility yourself. For example, every now and then they switch Python version, so if you're on this method you need to obtain whatever the newer version of Python is before upgrading. Thankfully HA always give notice before this happens, and indeed overlapping support for more than one Python version so you can upgrade Python with HA X and run with it, before upgrading to HA Y which requires it.

A Python VENV is a virtual environment with its own set of Python packages, it allows you to run multiple Python programs on the same system with conflicting requirements. It isn't a virtual machine, more a Python library sandbox. Consider a situation where Python program A requires library myLib at exactly version 2.0.0, but Python program B requires the same library myLib at a version > 2.1. These are mutually exclusive requirements, and you wouldn't be able to install both programs on the same system. With VENV you put each program into its own Library sandbox, and each downloads the version of myLib it needs and runs with it without interfering with the other program.

All you do to create a venv is run:

python -m venv /path/to/venv

Then you install homeassistant via the pip package manager:

pip install homeassistant

If you later want to upgrade you can use 'pip install --upgrade homeassistant' to move up to the latest version.