@ianmk13 I think ours fails to ‘phone home’ every so often! This leaves OE with a ‘hole’ in the data for one or two days. Regards, Toodles.

Posted by: @ianmk13

↑

It is in the news again today that there are almost 4 million smart meters that aren't working in smart mode.  This just adds fuel to the fire.  I suspect that they may be SMETS1 meters (my meter is SMETS2) and it would be interesting to know just what the problem is with these, too.  Is it an incorrect communications protocol, an issue with the security credentials, or something else? Can it be fixed with a firmware update? If so, does it require a site visit? How did the industry get itself in this situation?

There are a great many layers in the questions you're posing here, Ian.
Allow me to first give some clarifications.
We can then better understand the issue you've raised regarding the data loss from your Gas Smart Metering Equipment (GSME).

1: SMETS1 meters were licensed to be installed before the national smart meter network was designed and built.
They used a variety of different wireless mechanisms to send their data to your chosen Energy Supplier.
That's why some became un-smart if you switched Supplier to one who had implemented a different network path.

2: All SMETS1 sites should by now have been changed to operate using the SMETS2 specification.
That was actioned by Energy Suppliers sending an over-the-air software update to their customers.
Since meters are manufactured by a number of different companies, the relevant software for each meter was provided to Suppliers by the manufacturers.

Following the software update, each SMETS1 meter was migrated across to the national network, operated on our behalf by DCC (the Data Communications Company).
In some cases that migration failed. There are a number of possible reasons, but only rarely has it been necessary to implement an exchange which requires a site visit.

3: No data transfer can be initiated by your Smart Meter (or more properly, by the Communications Hub).
To obtain consumption data, a SMETS Command is sent to it by your Energy Supplier.
The Comms Hub responds to that Request.

Since the Comms Hub can hold the last 13-months of data within it (48-readings per day for electricity & gas), any period when there has been a loss of communication can be 'made good'.

If, in the meantime your Supplier's Billing Software has used estimated readings, then these are substituted with the actual readings, and an adjustment made to the bill.

4: A Gas Meter (GSME) acts like a sloth to conserve battery life.
It wakes up every 1800-seconds (a half-hour) to establish communication with the Comms Hub using the short-range Home Area Network (HAN).
Any data request messages in either direction are executed, and it then goes back to sleep.

Since data transfers will themselves occupy a finite amount of time, the GSME awakening will gradually move forwards in time.

Your Energy Supplier's data request will always be responded to by the Comms Hub, not the GSME itself.
If the Supplier requests the 'previous days readings' too soon after midnight, for example, then there will be occasions when only 47 of the 48 readings have been stored.
That's why you sometimes see a missing entry or an estimated reading for the last period of that day.

Occasionally there have been mistakes made by the Suppliers' billing software which makes a real mess of handling an estimate for the last reading of the day.
Here's one I saved from 15th Feb 2021 where the billing software applied a low estimate of the 48th reading, and then assumed that low figure applied to the whole day, even tho' it had actually received 47 correct readings!

The billing software obviously 'noticed' the error when it collected readings for 16th Feb just after midnight on 17th.
It then apportioned the missing gas consumption from 15th across the next 5 days, artificially making those day's bars on the histogram larger than they really were. (It was a warm week).

Just as the software was getting itself back in sync, the Supplier then collected the data for 21st too early again (too soon after midnight).
So it too has a very low estimate.

I'm afraid this sort of issue is well beyond the level of technical understanding you're likely to receive by telephoning Customer Services.
For that reason I always recommend taking screenshots over several days, and sending them in by email.
You then stand a chance of getting the error escalated to an in-house engineer who does actually understand the communications between a GSME and a Comms Hub!

5: If/when your Supplier needs to send software updates or change the configuration settings in your GSME, it will inevitably be asleep!

For that reason, each Comms Hub has a Virtual Gas Meter within it!
That acts as if the gas meter itself has received the software and/or commands, which means an acknowledgement is sent to the Supplier via DCC.

When the GSME next awakens, the Virtual Gas Meter transfers whatever is required to the actual meter.
Should that transfer across the HAN fail, it could theoretically leave the GSME in a state where correct daily readings are no longer available.

Once you tell your Supplier that something appears to be wrong, they can send a SMETS Reset command...
... and then attempt the software upgrade again.

Personally I think it shouldn't be necessary for the customer to contact the Supplier.
By now the Billing Software itself should be using a sanity-check to test if valid data is being received.
For goodness sake... this is 2024!

@transparent 

I am particularly interested in how the DCC fits into the picture. Does my energy supplier send a request for meter data through the DCC and receive meter data back via the DCC? Your diagram suggests this is the case. Do third parties (e.g. n3rgy) access meter data through the DCC in the same way? You made an interesting point regarding the gas meter sleeping for a fixed period, rather than waking at a fixed period, which I hadn't considered.

The final section of your reply seems to be a good explanation of what probably went wrong in my case. I'm surprised, though, that the communications protocol between the GSME and Gas Proxy in the Communications Hub isn't more robust, to include a verification message back from the GSME to confirm a correct upload of data. Given that I've now received three 'Welcome to Octopus' messages on my In Home Display in the past few days, should I conclude that this is considered to be an unreliable process. As you say, it's 2024! Maybe we have to wait until the next iteration of Comms Hub.

I've been looking very closely at my gas consumption this winter to help me to determine future heat pump requirements. Whereas I can usually get gas volume figures to tally, getting my kWh figures to agree with my bill is another matter. This may be a result of the asynchronicity of the meter readings, but could be exacerbated by the variations in calorific value (or even a different precision in the calculations). If only I could be as confident in my gas bill as I am with my electricity bill.

Yes. The only route to communicate commands to a Smart Meter and receive data from its registers is via DCC.
All communications paths in/out of DCC are encrypted.

Each Energy Supplier (or other customer of DCC) must implement a secure pathway from their in-house billing software to DCC using a Business Orchestration Layer (BOL).
In practice many 'buy in' that interface from the likes of ESG/Utilisoft, who manage the security aspects on their behalf.

Posted by: @ianmk13

↑

Do third parties (e.g. n3rgy) access meter data through the DCC in the same way?

Yes. However little data you wish to pass through DCC, each party must

• be a member/shareholder of the Smart Energy Code (SEC)
• undergo a data security audit by an independent 3rd party, which takes several months

The calibre of security in the national Smart Meter network is very strong, and is overseen by the National Cyber Security Centre (NCSC).

That's in sharp contrast to the way in which many Energy Suppliers actually ensure the validity of the consumption data which they retrieve.
If there's an error, it's far more likely to be within the Billing System software than the Smart Meter architecture itself.
Billing software still seems to have few 'sanity checks' within it.

The Smart Meter errors which so often get reported by the press, are unfairly tarnishing the reputation of the Smart Meter network and its design.

If the in-house billing software

• retrieves the day's consumption figures from the Comms Hub before the GSME has transmitted the 48th reading of that day
• incorrectly estimates usage for a period when consumption data is missing
• bills a householder for consumption which is more than double the forecast figure

... then that's not a fault of the Smart Meter system!

Posted by: @ianmk13

↑

I'm surprised, though, that the communications protocol between the GSME and Gas Proxy in the Communications Hub isn't more robust, to include a verification message back from the GSME to confirm a correct upload of data.

The GSME operates on the Home Area Network (HAN).
This usually operates at 2.4GHz, using a secure derivative of the Zigbee communications protocol (v2.5).
Not only is it secure and robust, but it incorporates a 4-way handshake with acknowledgements that include CRC (Cyclic Redundancy Check) to ensure data integrity.

If your Energy Supplier's billing software hasn't incorporated common-sense rules to verify that it has retrieved 48 gas-consumption readings within a 'sanity window' then that's not the Smart Meter's error!

Of course there will be glitches and drop-outs in wireless communication routes from time to time.
If you place a metal dustbin between your GSME and the Comms Hub, then don't be surprised if no data gets through!

Posted by: @ianmk13

↑

Whereas I can usually get gas volume figures to tally, getting my kWh figures to agree with my bill is another matter.

There are 69 gas supply sampling points in the UK which report the Calorific Value (CV) by noon on the following day.
You can obtain this data directly from the National Grid Gas Data Portal which looks like this

Further sample points are based out in the regions.
That data is combined to provide a CV for each of the 19 gas regions of the UK.

Each Energy Supplier imports that CV data, selects the test point(s) which apply to your Meter Point Reference Number (from the location database held by Xoserve) and apply it to the reading they obtained from your Smart Meter for the previous day.

Prior to noon, it's likely that your gas consumption will have been calculated based on a 'typical' CV of around 39.2MJ/cu.m, which is probably also used on your IHD.

If 'the system' fails to obtain a valid reading, then the Supplier must create your bill based on the lowest possible CV of 37MJ/cu.m.

To derive a gas bill in kWh from the meter readings in m³ (cubic metres)

x 1.02264  (volumetric correction - a constant, based on barometric pressure and temperature)

x CV  (the calorific value for that day to 1 decimal place)

÷ 3.6  (convert to kWh)

Here's the CV for my region (SW England) during March 2024

I grabbed that data before noon today (29th).
So the figures for today and yesterday (28th) are defaulting to 39.2MJ/m³ and will get corrected later.

@transparent So in short, I don't stand a snowball's chance in hell if accurately verifying my gas bill 🤣 🤣 🤣 . I had been trying to use my regional gas calorific value from the  National Grid Gas Data Portal and the conversion factor to replicate my bill but it seems there are too many known unknowns.  The daily breakdown of my bill reads like a work of fiction, although my calculation of the monthly total is within half a percent. For my own sanity, I think I'll continue to use my own calculated figures for power and energy while I continue to use gas.

@ianmk13 Never mind the ‘Unknown Knowns’ 😉

I have several apps that indicate my electrical consumption and export; I find that the reports of my export are somewhat lower on the apps than from my SMETS1 meter reports to the DCC. I am sure the meter is more accurate than my apps and so leave it that way! Regards, Scrooge err… I mean Toodles.

@toodles The difference on my bill is similarly beneficial, so I'm happy to let sleeping dogs lie 😉

@ianmk13 😉

Posted by: @ianmk13

↑

I had been trying to use my regional gas calorific value from the  National Grid Gas Data Portal and the conversion factor to replicate my bill but it seems there are too many known unknowns.

It should be possible to verify your gas bill by using a 'rolling' spreadsheet.
You'd have to insert formulae that apply adjustments

• after noon the following day, once the CV has been published
• after any day when all 48 readings weren't obtained
• after any period of multiple days when no data was collected at all (WAN fault)

The advantage of doing this in your own spreadsheet is that you can keep separate columns for the original readings/calculations, and those which you adjusted.

That's a problem with the way gas bills are presently provided to consumers.
The Energy Suppliers overwrite the original readings with the adjusted figure.
You can't tell which figures are

• tentative (pending an actual CV being available)
• an estimate (based on some unknown time period)
• a final figure with all adjustments applied

Using different colours for these categories would help!

On the plus side, there are fewer billing errors for gas consumption than there are for electricity for houses with Smart Meters †

I have provided the statistical analysis for electricity billing errors to the Commons Select Committee for Energy.
They forwarded the document to DESNZ... who have been investigating by writing to Energy Suppliers.

That process needs following up.
My MP is aware.

† Gas consumption is measured against a static price.
Electricity consumption might be assigned to different price points throughout the day (eg Economy-7), and those periods aren't synchronised to UTC

@transparent   I have tried this for my billing period since 9th February. Whilst I have half-hourly meter readings from the National Grid Gas Data Portal, I only have daily energy readings, in kWh, from my supplier.  Correlating the data sets would be quite time-consuming, I think, particularly when one of my known unknowns is the calorific value. Xoserve data places Milton Keynes in LD Zone SO (South) whereas Energy Solutions data places Milton Keynes in LD Zone EA (East Anglia). My calculations suggest it to be likely that my supplier uses CV data (and charging rates) for LD Zone EM (East Midlands).

OK. So the problem stems from your Energy Supplier not giving you the raw consumption figures.
You are only presented with the kWh figures after they may (or may not) have applied the CV and corrections.

Have I understood that correctly?

Coincidentally I've just written to the Financial Conduct Authority on the matter of energy bills.
I believe that energy contracts & payments fall within their remit.