@majordennisbloodnok - you make a very good point, I haven't made it clear enough that this is all voluntary, you don't have to do it. I will make an addition to the first post making this clear.

There are two reasons why I prefer local collection and storage. The first, which is really two linked reasons, is redundancy and reliability. If I do local collection and storage, then with the cloud as well, I have two independent (as in where they are) sources of data. One can self-destruct, and I still have the data in the other place. I suppose I could also deal with that by just making conventional backups. Reliability - the internet is not always reliable, as I have already seen, and when it drops out, as it did the other day, then the cloud has no data for that period. Another third linked thing is that Midea make the cloud data (above and beyond their feeble app) even harder to get at than the local data. If the local data has Fort Knox level security, then the cloud data has Fort Knox squared level security. Midea absolutely do not want pesky punters poking holes in their APIs.

The second main reason is almost philosophical, I am old enough to have a deep seated bias against my data going into 'the cloud', which is a misnomer, it is not some woolly thing on the horizon, but, in most cases, a server controlled by a commercial organisation. I do about half of my grocery shopping at Tesco's, and despite the gross bribes (price reductions) offered to encourage me to do so, I have have not signed up to their Club Card scheme. Why should I let Tesco's record the fact I routinely buy 14 bottles of single malt whisky every fortnight? Why is it any business of Tesco's if I want to sit down of an evening in front of my crackling log fire, with a glass or two of single malt, remembering the good times. What if one day all the big data gets joined up - purely in the interests of better efficiency and safety, of course - and my Tesco's data gets joined up with my NHS data - purely to keep an eye on your diet, sir, to make sure you are eating safely, not to much red meat, and of course strict limits on alcohol consumption? As Adam Curtis said, a brave, and terrifying, new world where we are All Watched Over by Machines of Loving Grace.

Absurd and alarmist? Not necessarily so. The covid passports were the beginnings of such schemes, which is why I fought so hard against them. The credit card companies are already hard in on the act. Mastercard has a carbon footprint tracker that can track your spending, and when your monthly carbon foot print goes over a certain limit (currently set voluntarily, but think about where those 15 minute cities are heading - yet another big data thing using ANPR - nothing voluntary about them) because that's the tenth megapack of T-bone steaks you bought this month, the card says no. 

I am quite capable of working out I don't need ten megapacks of T-bone steaks every month. I don't need a credit card company telling me so.

Now that I have got that off my chest, back to the monitoring: the next stage, after going through how to set up local area network monitoring is to attempt to use a wired connection (modbus over RS-485 for the technically minded). This should be simple - 'two wires and a converter card' - and I suspect it will bypass all the Midea security hurdles, as the modbus registers can be queried directly. If that all works, then stage three is adding simple not too expensive third party monitoring, thus bypassing Midea's data entirely, because who knows for sure the heat pump manufacturers don't have their own version of dieselgate?

The final stage, which actually goes beyond monitoring, is to attempt to control the heat pump, in particular, develop a load adapted weather curve, along the lines of the Ecodan Auto Adaptive control.

For a bit of light - or perhaps not so light - relief, this two and a bit minute video is absolutely brilliant:

Thank you, @cathoderay

I should probably clarify that I am also very wary - both personally and professionally - of how data is used. You are not alone in avoiding "reward" cards and I too have none of them. However, I do recognise the benefit of aggregated and shared non-personal data in providing much fuller information and the field of energy consumption/generation is a particularly good candidate. To put it into context, I have no problem with Mitsubishi being kept abreast of my ASHP's performance figures and sharing those stats with the rest of the industry to be able to understand (for instance) the effect of weather fluctuations on the energy requirement of their kit. However, I have a big problem if that sharing were to contain any of my personal information, and here the UK/EU legislation on GDPR is a real and big benefit.

As a result, the sharing of my kit's data with the manufacturer is a compromise between my privacy and the greater good, with a significant dollop of commercial benefit to Mitsubishi being gained in the process, but thankfully not a benefit that will realistically help them sell me more stuff; it's a benefit instead in terms of product improvement which I see as a good thing. At the moment, what I share is a balance I'm happy with, so I have no problem availing myself of their reporting tools. I am less happy that their API is undocumented for the general public and therefore a hidden resource that could provide much more benefit to me the consumer.

I suppose the point I'm trying to make is that I agree we should be very cautious before sharing our data. However, I think it is well worth making the distinction between "my data" as details about me personally (name, age, gender, contact details and so forth) and "my data" as non-personal data I have generated. The former I guard very jealously, but the latter - if it is kept disconnected from the former - I am much more happy to share (but not lose access to for my own use).

Part 4: Getting Home Assistant up and running

I had hoped this would be a quick and easy post, but the reality has been very different, so much so that I have even found myself on many occasions considering abandoning the whole project, on the grounds that Home Assistant (HA) is just too flaky, just too not ready for prime time, to bother with. The final straw that almost broke the camel's back was HA corrupting it's database, making it look like I had lost all my data - a no no in my books for a monitoring system. But it turned out that although HA thought the database was corrupt, and refused to use it, the data was still present in the file, and by a bit of jiggery pokery I managed to retrieve about ninety percent of it. On the basis the data wasn't actually destroyed, I decided to give HA another chance, on strict probation.

I must also add that using HA will drive you to distraction, and then some. I am not stupid, but I found almost everything I had to do to get The Beast (as HA is known in this parish) to come alive and start functioning as I wanted was well nigh impossible, for two reasons. Firstly, the UI is crap. It is the opposite of intuitive, nothing flows seamlessly into the next thing. Secondly, if the UI is crap, then the documentation is the biggest pile of steaming 5-H-1-T you have ever seen in your life. I would go so far as to say 90% is either wrong, or impenetrable, or both. You have been warned.

But HA can be made to work, and this post is an account of how I did it, step by step, which inevitably means a rather long post. We ended the last post at the point where HA was installed on the mini PC, but had not yet itself been accessed. Instead, we had the HA OS command line start up screen on the temporary monitor connected to the mini PC, looking something like this (generally, I am going to avoid screen grabs, because HA has a habit of not looking the same on different PCs and over time, but this is a very generic image):

But this is of no use, because the monitor is only a temporary one. To access HA, we need to access it from another device, in my case a desktop PC, via a browser over the local network. To do this, take the following steps:

(1) note the Home Assistant URL given on the screen, it will be http://homeassistant.local:8123. This is the address you are meant to use in your browser to connect to HA (but it may not work, as we shall see). You can now remove the monitor keyboard and mouse from the HA mini PC.

(2) Turn off the mini PC, and then connect it to your router. I used a wired network (ethernet) cable, simple and reliable. It may be possible to connect wirelessly, but I prefer simple and reliable.

(3) Turn your mini PC on again, and give it a few minutes to start up again. Once it has had a few minutes, open your browser and type in the address you noted in step (1). The HA login screen is supposed to appear — but, in my case, it didn't, and I got a can't connect screen. You might be lucky and get to the login screen, but if, like me, you can't connect, you will need to use the actual IP (internet protocol) address of your mini PC to access it. This will be a string of numbers separated into four groups by dots, something like 192.168.1.150, which acts as the unique address for your mini PC on the network.

(4) The simplest way to get this IP address, if you can do it, is to login in to your router, and list the connected devices. Failing that, you need to use the command line to get the mini PC's IP address. Open a command prompt on your PC (if you are trying to do this on a Mac, you will have to find the Mac equivalent commands) and type at the prompt:

arp -a

and press return. You will get a list of IP addresses. Those that start 192.168 are on your local network, and one of them is your HA mini PC. To find out which one it is, for each IP address that starts 192.168 type at the prompt:

nslookup 192.168.x[xx].x[xx]

replacing the x[xx] (they can be one, two of three numbers) with the numbers you got from arp -a, and press return. When you enter the HA mini PC's IP address, you will get back something that includes 'homeassistant'. That IP address is the IP address for your HA mini PC.

(5) Go back to your browser, and enter 1982.168.x[xx].x[xx]:8123 in the address bar and press return. This time, you should get the HA login screen.

(6) As this is the first time you have accessed HA from your browser, you will need to create an account, with a username and password. Ignore all the HA 'awesome bullshit' and just follow the instructions. You will be taken through a series of screens that set some basic options for your account. Don't try to be too anonymous, eg when it asks for your location (which you set by zooming out, and then dragging the map to your location) select somewhere nearby, because HA does use this to get local weather data. Text entry and drop down boxes are not obvious, they appear as greyed text, clicking on the greyed text will either cause a drop down to appear, or allow you to enter text. If HA offers to add any devices it has found, look at what they are and decide if you want them. Instead, just carry on until you get to the HA dashboard, HA's main screen. Like a dashboard on a car, is gives you a summary of your current system. Unlike a dashboard on a car, you have to put it together yourself.

(7) At this stage, the dashboard will be empty, or may just have one or two random cards, cards being HA's term for the rectangular boxes that present information about, well, pretty much anything you want information on that is available in HA. To achieve this, you need to take a bit of a dive into HA jargon, much of which is unintuitive and confusing. For now, the important terms are integrations, devices, entities and add-ons. In plain English, I see these as follows:

(a) integrations are the building blocks that connect HA to various services (eg a local weather service) and devices (see below). Usually you have to add an integration to be able to use it to connect to something

(b) devices are harder to grasp. They can be a physical device (like a heat pump) or they can be a sort of virtual device or service that is a 'box' that contains related 'stuff', for example a Met Office weather service that has data about your current and forecast weather

(c) entities are the individual data items within a device. For a physical device like a heat pump, they might include energy consumed and energy produced, while for a virtual device or service like a weather service, they might include current outdoor temperature and wind speed.

You need to get your head round these concepts, because they are all needed to get HA to do what you want it to do. Confusingly, some things can appear to be both an integration and a device, for example the Met Office weather. Instead, there are two things, a Met Office integration, which connects to a Met office device (service). In brief, an integration connects to a device which in turn makes available data entities. The Met Office integration connects to the Met Office device (service) which in turn makes available weather data entities, like outdoor temperature and wind speed. In due course, a Midea integration will connect to your heat pump device, which in turn will make available heat pump data, like energy consumed and energy produced.

(d) lastly, we have add-ons. These add extra functions, rather than data. For example, there is an add-on that allows you to view the HA database directly, while another one provides some networking capability (you can view and work with files on your mini PC from your main PC) and yet another provides a way to automate remote backups. Remote backups go to a remote location, so if your HA mini PC packs up, you still have a backup of your HA installation and, crucially, data.

(8) Before we add any integrations devices and entities, we are going to add some add-ons. These are all but essential for even basic stuff. As a minimum, I suggest you add Samba Share, which will give your PC limited (HA is very controlling) network access to some of the files and folders on your mini PC, which is needed to install the Midea (and many other) custom components, and Google Drive Backup, which automatically makes and copies HA backups to both your mini PC and Google Drive. You will need (and probably already have) a Google account to do this, if not, you will need to sign up to Google.

Later on, you might want to add some other add-ons. I have the SQLite Web Community add-on, which allows me to get at the HA database from inside HA, and the Terminal and SSH Official add-on that allows seriously limited (HA is very controlling…) remote (from your desktop) command line access on your mini PC. With hindsight, it is so limited as to be almost pointless…

To add add-ons, click Settings on the left hand menu on HA's screen, and then click Add-ons. You will get a blank screen, as you have yet to add any add-ons. Down in the bottom right hand corner, there is an 'Add-on Store' button (confusing, it is actually for adding, not buying, add-ons… Perhaps the HA developers suffer from monologaphobia, what they really mean is 'Add an add-on'). Click on it, and you will get a screen with HA's 'Official' add-ons. These are approved by HA add-ons, and so should work and be reliable. There are also, further down the page, HA Community add-ons, which might or might not work, but which offer the possibility of adding extra functionality.

Start by adding Samba Share ("Expose Home Assistant folders with SMB/CIFS..." - well that is all crystal clear, isn't it?) Just ignore the tagline jargon, all Samba does is allow your PC to view and manipulate a limited but necessary set of files and folders on your HA mini PC from inside Windows Explorer. Click on the Samba Share button, and a screen will open that allows you to install it. Go ahead, and install, and set up a share, providing shared access from your PC. You will need a username (I used 'homeassassin' 'homeassistant'), a password, your local workgroup name, usually 'WORKGROUP' (right click My Computer on your PC to confirm) and the share name (I used 'share'), so you can map the network share (ie mini PC shared folder) to a drive letter in Windows. I also set (on the add-on main page) both 'Start on boot' and 'Watchdog' to be on, so Samba is always available.

Once Samba is up and running, go to Windows Explorer, and click 'Map network drive' (near top left). In the box that opens, the drive letter to may will default to Z:, leave it as that (or whatever it is if it is not Z:), and then in the 'Folder' box enter \\ followed by the IP address for your mini PC (which you obtained earlier, and will still be visible in your browser address bar) followed by \share. The whole thing will look something like this: \\192.168.1.101\share (though your 1 and 101 numbers may be different). Then click 'Finish', and you should see the HA folders appear in Windows Explorer under the drive letter Z. The two important folders are config, HA's configuration files, and backup, your local copies of your backups (empty until you make backups).

(9) Next, add Google Drive Backup. This is a bit more complicated, because it is not an official add-on, despite being very popular and useful. You will also need a Google account to do this, create one if you don't already have one here (<=link). Now, as it happens, Google Drive Backup is very much the happy exception, in that it has exemplary instructions (why can't other developers do this?) available here (<=link). Rather than duplicate them here, I recommend you follow that link and then the instructions on that page to install the add-on. There are quite a few steps, and you do have to set up Google authentication, but the instructions are admirably clear, follow them to the letter and all will go well. I set it up to do one daily backup at 0100 hours, and to keep the three backups, the most recent, and the immediate past two backups. I also added it to the HA left hand menu, so my backups are always easily available.

(10) Lastly, for this post, we are going to install something called History Explorer. HA's own graphing capabilities are pretty hideous, and I am one of those pesky individuals that appreciate visually appealing charts. History Explorer does this, and, as it's name implies, it allows you to look back in time, to see how things change over time.

History Explorer is similar to an add-on, but is in fact a 'custom card', or 'view', a way of looking at HA, or more particularly, its data. A card, you may recall, is the generic term for the boxes in HA that display data, and History Explorer is just another card, albeit a large, elaborate and rather impressive one. But installation, surprise surprise, is a PITA.

The recommended way of installing History Explorer (HE) is to use something called HACS, or the Home Assistant Community Store. I strongly recommend you avoid HACS like the plague, as it is a superlative bit of pointless bloatware. Instead, take control, and install HE manually. Doing a manual installation helps you to gain an understanding of what is going on under the hood in HA, and avoids hidden bloatware dumps that clog up your system. To do a manual installation:

(a) start by accessing your HA files and folders in Windows Explorer. If you haven't mapped them to a drive letter, in Explorer click Tools (top left hand menu) and then Map network drive and then add your HA share (see above for details) and finally click on the shared folder to open it, and then navigate to the config folder if you are not already there. You will see a number of core HA files, including configuration.yaml, home-assistant.log (the current log file) and home-assistant_v2.db (your HA database). Create a new folder in HA's config folder called www, and then open it

(b) go to History Explorer's main web page here (<=link), click on releases on the right hand side, and download history-explorer-card.js to the www folder you just created in HA. Files with the extension js are javascript files, javascript being a very common programming language used on the web and elsewhere

(c) back in HA's main screen, click Settings in the left hand menu, then Dashboards, and then click the blue Add Dashboard button in the bottom right hand corner

(d) add a title for the dashboard, I called mine History Explorer, and select an icon, I used mdi:history. The url field will be filled in automatically, do not alter it. Leave Show in sidebar on, and then click Create

(e) History Explorer should now appear near the top of HA's left hand menu. Click on it, and you will get a dashboard filled with HA crap. To get rid of the crap, click on the three dots in the top right hand corner and then click Edit Dashboard. HA assumes you don't know what you are doing, and produces a dire warning about taking control. Taking control is exactly what you want to do, and furthermore you want to get rid of all that HA crap. Click on the Start with an empty dashboard button, and then click on Take Control. You will get a mercifully blank dashboard

(f) click on the three dots in the top right hand corner, and then click on Manage Resources, and then on the blue Add Resource button in the bottom right hand corner. This is where you add the History Explorer 'custom card'. Ignore the dire warnings, you know what you are doing, and type /local/history-explorer-card.js into the url field, then select Javascript Module, and finally Create. For some reason known only to itself, HA wants you to access the \config\www folder using /local/… Note the slashes, Windows, where we added the files, uses backslashes, HA, which is Linux based, and runs the files, uses forward slashes

(g) now return to your empty History Explorer dashboard (click on it in the left hand menu), click again on the top right three dots and then on edit dashboard. Next click on the blue Add Card button, in the screen that opens scroll down to the bottom, where you should see Custom: History Explorer Card. Click on it, and then on Save in the box that appears. An empty (you will add things to it later) History Explorer Card will appear in your dashboard. Finally, as it is the only card we want to show in this dashboard, click the pencil icon by Home near the top left hand corner, and in the box that opens, click Panel (1 card) and then Save.

That's it! You have now completed the initial HA setup. Next time we will add the core Midea custom components needed to get some of the Midea data. Clearly this will only work with Midea and other manufacturer re-branded Midea units, but the general principles will apply to all heat pumps. If you have a way of accessing data from your non-Midea heat pump, then you stand a working chance of getting it into HA.

Running total costs to date: £47.00 (plus quite a lot of hair pulled out…)

@cathoderay given your knowledge and experience level and you’re pulling hair out I’m not going to even consider trying this 🤣

Posted by: @morgan

↑

you’re pulling hair out

That's why I put that in, to warn people that, however good a head of hair people have when they start working on Home Assistant, within short order they will lose a lot of it. It's a pity, because when you do get it working, Home Assistant really is quite useful, even impressive. 

Posted by: @cathoderay

↑

Posted by: @morgan

↑

you’re pulling hair out

That's why I put that in, to warn people that, however good a head of hair people have when they start working on Home Assistant, within short order they will lose a lot of it. It's a pity, because when you do get it working, Home Assistant really is quite useful, even impressive. 

I'm sure it is, but I decided I like what's left of my hair, so I abandoned Home Assistant and used Domoticz instead. It's not as flash and doesn't have as many extension plugins as HA, but it can set clock and daylight-relative timer schedules with randomness and other unusual patterns without installing a load of add-ons or laboriously describing them in text, plus it does have plugins and event scripting that are enough for me.

But that's for control and I use emoncms for ASHP monitoring, as mentioned previously. It's intended as a monitoring system primarily and I believe its emonSD "low write" memory card setup (less likely to go past a memory card's lifespan, or corrupt it by being half-written if the power fails), the use of time-series-specialised database formats and its automatable backups mean it's less likely to lose data than most smart home control systems. (I've lost data once because I had messed up the backups and didn't realise: test your backups!) Its documentation isn't always totally up-to-date and complete (as so often is the way for actively-developed systems) but what is there is fairly easy to follow, especially compared with HA's confusing mess (and I say that as no beginner to IT or even things like Message Queueing — of course, I've had to decipher worse for work and could probably do so for HA, but nobody is paying me to monitor my own home and it's not fun so why would I?).

Edit to add: I know you're trapped into HA by the midea interface only being available for HA, but this means that later parts of your guide are hard work, scary to beginners and I feel not the best approach for owners of other brands. Maybe this discussion should be retitled "A Beginner's Guide to Midea ASHP Monitoring"?

@mjr - thanks for your welcome contribution. It's good that people know there are other options out there that might not involve so much Trichotillomania.

Posted by: @mjr

↑

I know you're trapped into HA by the midea interface only being available for HA, but this means that later parts of your guide are hard work, scary to beginners and I feel not the best approach for owners of other brands. Maybe this discussion should be retitled "A Beginner's Guide to Midea ASHP Monitoring"?

This is a good point, one I have been addressing behind the scenes by trying to find out what other heat pumps have python modules similar to the Midea one. It seems re-branded Midea units often use the same protocol, so they can use the same python code. Among the other brand python modules, Melcloud seems a  possibility, but rather limited integration with HA, and Daikin. I'll look for more, but if I can't find any, I will probably re-title the post as you suggest (or something similar).  

@cathoderay several heat pump interfaces speak MQTT or echonetlite which are fairly easy to connect to. I know HA can control MQTT devices and I'd be astonished if it can't do echonetlite. Then there's also openTherm. That all gives both a wide range of possible pumps, but also simpler options than HA. Anyone writing a heating interface should probably target one of the general interfaces and not HA only, both to increase the number who can benefit, but also to reduce the risk of breaking when a new version of HA is published.

Posted by: @mjr

↑

reduce the risk of breaking when a new version of HA is published

I have heard that breaking changes are a real problem with HA (nuvver tuft of hair lands on the floor). My goal when starting out with this thread was to provide a solution, not options. If we go back to earlier discussions, one of the major problems for beginners is there are too many options, none of which make any sense to a beginner. Most of us do not want to get a degree in computing science or electrical engineering so we can monitor our heat pumps. I started out trying to document the steps I took so that a beginner in all this but otherwise averagely capable and intelligent person could follow the steps, and get a result/solution. I then came up against the proprietary problem...

A quick google search doesn't suggest there is much MQTT/Midea common ground. I'm not even sure what MQTT is. Usual jargon problem serious lack of plain English. In said plain English (and a few symbols), you have

A heat pump  <==>  some sort of communication set up <==> home monitoring/automation software that display data etc (in my case Home Assistant)

The middle bit (communication): at the moment, for Midea units, I know this can be done

(a) over my LAN, wired in my case, using a custom python module

(b) over modbus/RS-485 (wired), still yet to be sure what else is needed

I can't see how MQTT fits into this, or what it replaces.       

Posted by: @cathoderay

↑

Posted by: @mjr

↑

reduce the risk of breaking when a new version of HA is published

I have heard that breaking changes are a real problem with HA (nuvver tuft of hair lands on the floor). My goal when starting out with this thread was to provide a solution, not options.

the above are true. however, I would like to respectfully suggest that you are very much still at the options stage.  Once you've gone through lots of options, tried them, and arrived at a stable(ish) position, then you're in a position to write it up as a solution.

I have half an inkling that those of us who are IT geeks (of one sort or another), might between us be able to put together and recommend something that the non-IT-geek can "just deploy", but I don't think we're there yet!

Posted by: @iancalderbank

↑

the above are true. however, I would like to respectfully suggest that you are very much still at the options stage.  Once you've gone through lots of options, tried them, and arrived at a stable(ish) position, then you're in a position to write it up as a solution.

I have half an inkling that those of us who are IT geeks (of one sort or another), might between us be able to put together and recommend something that the non-IT-geek can "just deploy", but I don't think we're there yet!

I very much agree! I haven't made any final decisions yet, so I do have options, but my goal is still to end up with a 'just deploy' solution. I am sure you experts (IT geeks) will all help in making decisions. By far the most time consuming and often rather overwhelming part of all this is investigating the options, and trying to work out the best solution. Horses for courses doesn't really help because it's really just a three word way of saying options again, it isn't a decision/solution.

One of the big questions I have yet to answer is whether I am prepared to rely on only Midea's data (underlying question: can a manufacturer's claims about the performance of their kit be trusted?), or whether include third party hardware for monitoring the critical parameters, which boil down to just two, energy in and energy out. The former is relatively straight forward, I already have an external dedicated heat pump meter with pulsing LED, meaning I just need a pulse counter. Energy out is more complicated, as I need a flow meter plus flow and return temperature sensors.

I wonder what people's views are on whether a manufacturer's data can be trusted? Isn't it a classic case of marking their own homework? 

Posted by: @cathoderay

↑

.

One of the big questions I have yet to answer is whether I am prepared to rely on only Midea's data (underlying question: can a manufacturer's claims about the performance of their kit be trusted?), or whether include third party hardware for monitoring the critical parameters, which boil down to just two, energy in and energy out. The former is relatively straight forward, I already have an external dedicated heat pump meter with pulsing LED, meaning I just need a pulse counter. Energy out is more complicated, as I need a flow meter plus flow and return temperature sensors.

I wonder what people's views are on whether a manufacturer's data can be trusted? Isn't it a classic case of marking their own homework? 

well its interesting. the heat pump must have an accurate flow meter and accurate flow and return temp sensors, for its own control purposes. So IMO that information , if its available and reliable to get at, should be good enough - if your wet circuit is a single loop design.  If you have a primary / secondary split system then metering the secondary side becomes a valid thing to potentially also need to do. Which would require additional hardware.

I had some conversations with marko cosic at meterpoint a couple of months back, they can do commercial quality heat meters, designed for heat pump flow rates, with modbus, for around £300. needs a nice long straight pipe to fit is as the meter itself is 260mm plus you need 280mm before and 140mm after as best practice to avoid turbulence. have not purchased as not convinced I need one yet.