@korwraith - please expand on the comment about EDF circumventing the Randomised Offset to the Tariff Table 'in the same way as Octopus'.
I'm not aware that Octopus has 'circumvented' the Randomised Offset, merely that their internal billing system (Kraken) has demonstrated a lower level of billing error than EDF did with your particular account.
My analysis of the correspondence between you and EDF suggested that they had tried using two internal registers with the Electricity Smart Meter Equipment (ESME) as a 'fudge'. However, in your case that fudge factor had operated in the opposite direction to that which they'd intended. It amplified the Randomised Offset error to a much greater extent.
Without further evidence we have no idea what their fudge factor did for other users. I'm wary of drawing conclusions whilst the investigation is based on only two major foundations:
the theoretical mathematical modelling undertaken by Simon & I
the actual calculated billing error which EDF have admitted to in your case
I don't think that Announcement you've just published from EDF about them changing the hours during which your cheap rate is valid is related to the Incorrect Billing error.
In fact I think it will make matters more difficult for them, because they've moved too close to midnight!
At some point after 00:00:01 every ESME will start a new tariff period. That's when the Smart Meter loads a reading into the final (48th) slot of the previous day's consumption figures. The latest point in time when that 48th slot could be completed is 00:29:59. That's defined in the Smart Meter specification.
Unconnected to that issue, at some point after 00:00 the gas meter (GSME) will wake up, exchange pleasantries with your Communications Hub and transfer the 48th gas consumption reading of the previous day to the 'virtual GSME' in the Hub.
The transfer takes a finite amount of time, depending on how weak is the Zigbee network between the two devices. The GSME then goes to sleep for another 1800-secs. So the time period between each GSME wake-up will gradually creep forward by 1 or 2 seconds.
The GSME also has a Randomised Offset to the Tariff Table, despite that fact that we are currently charged the same for gas, whenever we use it.
In the worst case there will be a customer with a GSME Randomised Offset of 1799-secs and a GSME wake-cycle of 1802-secs.
The less time that has elapsed after midnight before your Energy Supplier sends the "Read Usage Registers" command to your meter, the greater is the likelihood that your ESME and GSME will not yet have switched across from the 48th/final period of the previous day.
When I was with OVO four years ago, I compared the gas consumption readings which they used for billing purposes against the same readings obtained by the Bright App from Hildebrand.
OVO was so fast in retrieving readings soon after midnight that their customers' gas usage pattern was shifted one 30-min period forward in time.
On the graph above you will see the first (left-most) gas consumption reading, as declared by OVO, was actually the 48th period of the previous day.
That was a crucial piece of evidence which allowed Simon and I to confirm that the Randomised Offsets stipulated in the Smart Meter specification, were indeed being implemented by the meter manufacturers.
@korwraith - please expand on the comment about EDF circumventing the Randomised Offset to the Tariff Table 'in the same way as Octopus'.
EDF said their future billing system wouldn't suffer from the same 'doesn't start exactly on the hour' issue, so I'm assuming this email is indicative that this new system is now in place (although who can be sure of anything they say?).
In any case, as far as I can tell, Octopus billed me perfectly. My off-peak period started at midnight, my usage spiked at midnight, my bill reflected this as expected, without any leakage into the adjacent periods. Indeed, Octopus' bills identically detailed the same 30 minute UTC usage data that I downloaded from my smart meter. None of this peak / off-peak 24 hour counter stuff that didn't actually align with the advertised off-peak times that EDF's system was based on (presumably because of the random offset).
My non-expert understanding (that may very well be incorrect) is that in my manually controlled case, the Octopus approach to billing is perfectly adequate, from a consumer point of view. However, smart devices that co-ordinate with the randomised off-set clock in the smart meter would presumably still have issues as we're now billing purely by UTC time and disregarding local offsets. By ignoring the random offset it discourages devices to be synced to the random offset clock and so does away with the intended feature of the random offset, which is to improve grid stability.
At least, that's my non-expert take. Please correct any misinformation 😀
Please keep adjusting the time at which you start drawing 'cheap rate' electricity @korwraith It seems that you have a meter where the Randomised Offset to the Tariff table is particularly large, and therefore easier to work out.
I've had a further bill issued since, covering mid June to mid July.
The data from my meter indicates:
512.9kWh off peak 8.85kWh peak (521.75kWh combined, 98.3% off peak)
EDF Billing for the period:
512.89kWh off peak 9.33kWh peak
(522.22kWh combined, 98.2% off peak)
I can't say what time of day EDF polled the counters at so for all intents and purposes I consider this to be a match BUT ONLY BECAUSE I MANUALLY ADJUSTED FOR MY METER'S RANDOM OFFSET.
My battery inverter and car charger were set to charge at 15 minutes AFTER the supposed start of my off peak period at the time (i.e. 01:15). My previous bill on page 4 of this thread had these devices set to charge 10 minutes after the start of off peak period (01:10).
Therefore I conclude my randomised offset to be between 10 and 15 minutes, likely closer to the latter than the former. Once I've another bill and more information regarding EDF's potential change to their billing system I will possibly experiment with moving my device charging times to 5 minutes passed the stated start of the off peak period.
Email from EDF today indicating a change to their billing system, I suspect relevant to the content of this thread...
Sadly no detail beyond that which is displayed in the image, but I expect the change may (for billing purposes in manually set systems) circumvent the random off-set issue (perhaps in the same manner as Octopus). At least, that is what their previous communications with me indicated.
This is the first communication I've had from them regarding the change. I've been operating under the information that my BST off-peak period is (officially) 1am to 6am. No info as to the date at which this was switched to 12am to 5am, will be interesting to see how they account for that when billing...
Bad bad bad development!
I contacted EDF on the 22nd of August after receiving the above email to have the above change in my off-peak period confirmed in writing (prior to this communication my off peak period was 1am-6am during summer (BST), and 12am-5am in winter (GMT), when working with 'real world time'):
"Hello Greig,
I can confirm that your EV tariff off peak will [be] 12.a.m - 05.00a.m the change has only just been made and as stated in the email:
We've got you covered
We know you might not have been aware of these times which changed when we moved you to our new system. So we'll be reaching out soon to make it right.
Kind Regards
Lisa"
Alas, the subsequent alteration of my EV charger and battery inverter operational timings has resulted in a horrendously incorrect bill for the following period - with my bill being roughly 2/3 off peak and 1/3 peak - a stark contrast from my ~98% off peak and ~2% peak real world usage (as confirmed by my direct smart meter data)!
I perhaps shouldn't be overly surprised given the previous content of this thread, but I had hoped that this relatively simple change to the terms of my tariff (from 1am-6am off peak to 12am-5am off peak) would have been successfully managed by EDF.
As this is such a glaringly obvious oversight that will undeniably impact thousands of other EDF customers, my hope is that it will result in a decidedly swifter technical escalation and resolution than previous discussions around EDF misbillings (although it should be noted those previous misbillings remain somewhat unresolved...).
It is very disappointing that we have yet another example of supplier error impacting on the effective utilisation of smart meters.
This post was modified 3 months ago 4 times by KoRWraith
For the avoidance of doubt, as I realise I omitted to unequivocally state the root of the problem, EDF continued to treat my off peak hours as 1am-6am, rather than implement the change specified in the August emails.
I need a few hours to consider what you've just posted @korwraith and what EDF mean when they write "the change has only just been made"
I should let you know that the DESNZ contacted me unexpectedly last week and have provided some useful insights from the Smart Metering Implementation Team (SMIT).
Please note that DESNZ staff and technical members of the SMIT now have the URL for this topic and can read what is being posted. It would be really helpful if we can stick to the technical issues... ... and ensure that statements are adequately backed up with source references.
For the benefit of SMIT, I'm asking @korwraith to state whether his EV charger is being operated according to times which are set manually, or by EDF sending the 'ON' command with the charger in 'Smart mode'?
For the benefit of SMIT, I'm asking @korwraith to state whether his EV charger is being operated according to times which are set manually, or by EDF sending the 'ON' command with the charger in 'Smart mode'?
My EV charger is manually set, aligned with my off peak hours (with further manual tweaks detailed previously to adjust for my meter's random offset due to prior (and ongoing) EDF misbilling).
This post was modified 3 months ago 2 times by KoRWraith
@transparent, many thanks for your illuminating contributions on this thread. Having looked at the information you have presented and the SMETS2 Specifications, I have one query and one possible explanation of the security aspect that puzzled you.
I am working from the 2 November 2023 version of the specifications (referred to here as the "Specifications"), which I found at https://smartenergycodecompany.co.uk/document-download-centre/ (search for "Smart Metering Equipment Technical Specifications 2").
First, the query. This is in relation to the suggestion that the gas meters (GSMEs) have a Randomised Offset like electricity meters (ESMEs). In the Specifications, there is no mention in section 4.6 (Data requirements) of the GSME storing a Randomised Offset Number or a Randomised Offset Limit, or the Randomised Offset that is the product of the first two. Nor in section 4.4.8 (Pricing) is there any mention of adjusting time-of-use periods by a Randomised Offset, as there is in section 5.5.8 for ESMEs.
So I cannot see that the Specifications require GSMEs to have a Randomised Offset - am I missing something?
A further pointer that GSMEs don't have it comes from the SMETS2 command that you referred to, which can be used read the Randomised Offset remotely. As far as I can work out, this is the ReadDeviceConfigurationRandomisation service request detailed at section 3.8.47 of the DCC User Interface Specification v5.3. Note that, as set out in that section, this is a request that is applicable only to ESMEs (unlike some other requests that apply both to ESMEs and GSMEs), which further supports the idea that GSMEs do not have a Randomised Offset.
I see that your OVO experience led you to believe that the underlying cause was a randomised offset in the half-hourly gas data. But might it have been something else, e.g. a programming error in OVO's billing software that caused OVO to shift data by one half-hourly interval?
Puzzlingly, though, given the apparent absence of a Randomised Offset in the GSME Specifications, I have also observed in my own property behaviour that is consistent with an offset in my gas data. When I use gas, it is recorded in my half-hourly gas usage as having been used about 15-20min earlier than it was in fact used. I have eliminated all possible causes that I can think of other than it lying in the way my smart meter (or something downstream of it) is recording data. Having since found and read this thread, I found it entirely believable that gas meters had a random offset, and that mine was something like 18min. But now that I have looked at the underlying documents as set out above, I am less sure - but in that case I can't explain the offset apparent in my data.
Second, there is the possible explanation of the security aspect that puzzled you. This arose from the only mention of anything random in the GSME part of the Specifications, which is of a "Random Number Generator" in section 4.3, which (as for the ESME) is within the "Secure Perimeter" - you included the corresponding snippet about this for ESME on the first page of this thread in your "Part 3" post. Was it this mention of a Random Number Generator that prompted your comments, as follows?
The Smart Meter specification (2014) prohibits the Randomised Offset from being accessed externally.
although I appreciate the need for the security features within most of the ESME specifications, I don't understand why a customer shouldn't know their own Randomised Offset
If not, apologies for having misunderstood. But if it was this that prompted your puzzlement over security features, I think this may be a misapprehension since the Random Number Generator appears to be unconnected with the Random Offset. As the Specifications make clear, the generator is "A component used to generate a sequence of numbers or symbols that lack any predictable pattern". But there is no need for the meter to generate any (pseudo-)random numbers in relation to the Random Offset, since the Randomised Offset Number is fixed at the time of manufacture, and the Randomised Offset Limit is not a random number (but rather the constant that defines the upper bound of the Randomised Offset).
Instead, I believe the Random Number Generator (in both GSME and ESME) is related to the cryptographic aspects of the communication protocols. I have found National Cyber Security Centre documents that shed light on this. Looking at the GSME one ("CPA Security Characteristic - Gas Smart Metering Equipment" v1.3 at https://www.ncsc.gov.uk/files/CPA%20SC%20GSME%20V1.3.pdf), it explains that "A Random Number Generator ... is used in the generation of the Public-Private Key Pair on the Device". In other words, it is used to generate the cryptographic keys that allow the communications to be kept secure. It is the same in the ESME document. That would explain why there is a concern to keep secret the details of the way that the random numbers are generated for this purpose.
In conclusion I can see nothing in the Specifications that means the Randomised Offset could not be displayed to the consumer (although that doesn't mean any meters are actually designed to do this). Equally there is nothing in the Specifications that would prevent a utility company from disclosing the Randomised Offset to the consumer.
It is certainly very concerning that: 1. Utility companies are incorrectly billing customers because they fail to take into account the Randomised Offset, as you have so clearly explained; 2. Customers cannot even find out what their Randomised Offset is (short of estimating it from experimentation); and 3. DENSZ appears not to understand (or else not to care) about this.
Thanks for posting and for your interest in this subject @jamesw
Firstly, I am a Member (shareholder) of the SEC... and I'm glad you're sufficiently diligent to seek out official source documents.
Secondly, the version of the SMETS2 Spec which I usually refer to dates from Nov 2014, because that's when Parliament approved the legislation. It was that version on which manufacturers started designing the ESMEs and Communication Hubs.
And it was in Spring 2018 that SMETS2 meters were first installed in GB homes... long before the Nov'23 version which you've been reading!
I see that your OVO experience led you to believe that the underlying cause was a randomised offset in the half-hourly gas data. But might it have been something else, e.g. a programming error in OVO's billing software that caused OVO to shift data by one half-hourly interval?
Not quite.
There were several Members of the OVO Forum who analysed bills and discovered a wide range of anomalies.
We were looking at GSMEs because OVO were collecting data from the Comms Hubs too soon after midnight. The consumption figures for gas were indeed half-an-hour out, because the physical GSME hadn't yet awoken to send the real data to the Virtual-GSME which resides in the Comms Hub.
That's a separate issue to the matter of incorrect billing due to the Randomised Offset affecting electricity meters.
Yes, there were errors in the Billing Software (Orion), and I actually have the Job-Numbers which were assigned to some of them. That's not unexpected. Code develops over time. But the way in which errors were handled by the programmers in Kaluza made it very difficult for OVO to implement Time of Use Tariffs.
Suppliers retrieve consumption data from their customers' meters in different ways.
Some have given up on using the registers which record consumption that's delayed by the Randomised Offset. Instead their tariffs are aligned to UTC... ... and that's increasingly becoming a problem because it causes demand surges as various tariffs kick-in at half-hour intervals.
Abandoning the RO may make the Billing Systems correct, but it throws the problem at the DNOs, who must now upgrade the Distribution Grid capacity to exceed demand peaks.
It's fundamentally the wrong approach. The time offsets are there for a good reason, and we should be using them.
This post was modified 2 months ago 3 times by Transparent
