Posted by: @transparent

↑

Posted by: @transparent

↑

The main reason we cannot rule out an attack on the Iberian grid by a malicious 3rd party is the Press Release by Reuters on 14th May this year.

It was a story carried by three UK-based media outlets on the following weekend, but they misidentified the threat as an embedded "Kill Switch".
See this article by LBC

 

 

Ok, that's really old news. It was resolved before it was made public - all the kit (which included Sunamp & other big names) has been patched and sorted. It was never suggested it was 'deliberately installed' either, it was just as likely to be slackness or similar.  Who knows? 

Erm... seeing this from an electronics viewpoint, you can't have two entirely separate control channels implemented on a commercial inverter by anything other that it being deliberate!

Slackness can't create a cellular communications system which isn't mentioned in the hardware specifications, nor been certified.
The very nature of there being two possible ways to send commands means that there must also be software present to resolve potential conflicts.

It's not like having two separate communication paths for monitoring an inverter.
That would only be sending out data (a duplicate) on each channel.

Apparatus which has two separate command and control pathways is deliberate, not an oversight.

Posted by: @transparent

↑

That provides a crude protection against surge demand in an area.

I am not sure if many people would be running aircon or similar high powered devices, relying on PV and the grid.. i assume they would not keep a system that would trip frequently going. Even with a battery people would not like to constantly have to check if it tripped.     

But fair point if at very scale, could it cause localised issues? 

Posted by: @transparent

↑

If Zero Export is the norm, then I don't (yet) understand why the latest ENTSO-e report suggests the possibility of small embedded (solar) inverters becoming disconnected from the grid.

[... ]there was a loss of 208 MW identified distributed wind and solar generators in northern and southern Spain, as well as an
increase in net load in the distribution grids of approximately 317 MW, which might be due to the disconnection of small
embedded generators < 1 MW (mainly rooftop PV) or to an actual increase in load or to a combination of both.

Reading through the report (I did not notice all the recent posts that seem to be commenting on it 😳), it seems that there were various issues on the day and before impacting the frequency of various parts of the network. 

The rooftop PV systems disconnecting seems like a red herring. I am not saying it could not be an issue, but wishful thinking for someone else to try to hook a solution to that to the much bigger solution. I mean what will come out of a major investigation that is looking into much bigger topics. It is serious that these faults did not get isolated by localised blackouts in SW Spain and instead caused so much load to be disconnected, then bringing the whole Iberian system and 0.1% of the French down. And why it could not be brought back up in max a few hours.   

Posted by: @batpred

↑

if at very scale, could it cause localised issues? 

Yes, it could.

The observation from the expert panel regarding the 'loss' of 208MW of embedded generation fits with what we might expect from that demand suddenly shifting to the grid.

Posted by: @batpred

↑

it seems that there were various issues on the day and before impacting the frequency of various parts of the network. 

Quite.

And I note that the ENTSO-e panel are investigating these two earlier voltage fluctuations separately from the final collapse which started at 12:32.

They may be not directly related.

All three might, for example, be caused by particularly low inertia on the grid on 28th April.
But that doesn't mean the first set of fluctuations was the cause of the second set.

@batpred We run our aircon in the summer not just to keep fresh and have a pleasant indoor temperature, but to use up some of the spare energy we generate. Doesn't use much anyway, each system under 400watts.

@batpred The rooftop PV systems disconnecting seems like a red herring. I am not saying it could not be an issue, but wishful thinking for someone else to try to hook a solution to that to the much bigger solution

You are correct, it is a red herring.

[As are any 'cyber threat' suggestions which have been repeatedly ruled out by absolute experts.] 

Damien Cortinas head of ENTSO-E board: "neither Spain’s increasing reliance on renewable energy nor its limited power interconnections with other countries had played a role". 

The report emphasises that wind and solar production that day were no different to previous days and seasonal averages. 

We discussed this in Brussels and the transmission system operators were present (as I said before, so was NESO). 

I'm not going to write an essay here because: a) it's my job and I don't wish to do it for free because that doesn't pay my mortgage (😁); b) I'll be spending time with both the Spanish & Portuguese grid operators (amongst others) & writing about it all in considerable depth in the future; c) it's Sunday! 

First off, the report does not seek to identify a 'cause' beyond indicating it was an accumulation of interconnected events. 

A report into 'causes' will be issued in the future. 

However, one fundamental 'event' involved a thermal plant which decided not to make an offer to the balancing mechanism the night before (this is the critical intersection of half hourly markets and technology which function according to 'bids' by plant operators) thus there was a shortage of thermal power plants connected to provide essential voltage control services at the time of the incident.

Interestingly the un-named thermal plant revised their position just as the cascade started - too late obviously. 

In the half hour preceding the cascade the 'Iberian' (as opposed to Spanish) grid had two moments of intense oscillations which the grid transmission operators attempted to rectify but these actions in themselves increased the voltage levels across the network. There was a series of oscillations and disconnections (including disconnecting solar). 

Who has to do dynamic tension control? Conventional generation (thermal power plants). And what happened? A cascading collapse. 

In the Iberian grid conventional thermal plants are responsible for dynamic tension control. In Portugal this functioned correctly, unfortunately in Spain it didn't. 

This became a problem because of the lack of voltage control due in part to the non-operational thermal plant. 

It seems that in Portugal it did behave as expected in tension control. But not Spain... 

Sorry if I'm repeating myself here but I think this is absolutely fundamental to understanding why safeguards didn't kick in. 

Please do not read this as 'causal' because it really was a conglomeration of 'things'.  This is why there will be a separate report on 'causes' although data collection in some areas is proving to be a problem. 

To be honest, the ENTSO-E report pretty much reflects the redacted public version of the Spanish report. 

One more thing, there are already bills going through the Spanish parliament to change laws and put out to public consultation new operating procedures to better reflect Spain's rapidly evolving energy production. 

Spain is incredibly ahead of the game in some ways which is why all the Transmission System Operators are so interested. One Brussels official wistfully said to me that if the collaboration and reflection that European TSOs share was applied to other sectors Europe would be on top of the world.

Seeing them in action interacting really changed how I see the European grid - Britain never left that market. [🙀🫢😉🤫😂]

Oh and once again I was told the recovery was a 'master class in black starts'. Big up little Morocco who played a quick acting role in this and never gets a credit for it outside of the Peninsula. 

@transparent Slackness can't create a cellular communications system which isn't mentioned in the hardware specifications, nor been certified.
The very nature of there being two possible ways to send commands means that there must also be software present to resolve potential conflicts.

Just a very quick one on this....

[I might have to beg an expert quote off you further down the project line 😊] 

First off, as you know, what are euphemistically called 'vulnerabilities' occur with shocking frequency across our tech - be it Apple or Oracle or whatever... And that includes back doors that Apple, for example, put it in their own systems inadvertently that were so bad there were public information warnings issued to news rooms. 

(We won't get into the NSA threats that Snowden (heroically imo) exposed).

Never mind that much to my horror, the number of major organisations running systems on Windows XP or whatever dinosaur kit they have, is absolutely beyond belief. 

There have actually been a few cyber attacks on grid kit - all dealt with fairly quickly (unlike the British Library ... grrrr!) and while some may have been 'state linked' they've tended to be criminal organisations including European criminals. 

One study (of many) investigated a range of inverter, battery and other kit manufacturers and Huawei, for example, came out squeaky clean. 

My point is that on one hand 'everyone's at it' - Israel being streets ahead of anyone else - on the other hand because of this it's very closely monitored.

Back in June there was a major UK conference on grids and cyber security with Ed Miliband and National Grid. But my point is the need to not fall for the clickbait that gets manufactured out of this 'Reds under the bed' narrative. 

Look at the recent front page story on "Russia jammed von Der Leyen's plane's GPS when it was coming in to land" which even the FT repeated uncritically. It was a fiction. Her office put it out - she had an agenda to sell. 

Shockingly, no one bothered to fact check the story before printing it. 

Planes don't need GPS to land, they have other systems. Flight Radar (I love that site 😁) tracked her plane's GPS and it was functioning perfectly normally, the whole story was whipped up for nefarious reasons because it's so easy to blame Russia/China - tick applicable box - and it sells like hot cakes. The rebuttal ends up a fortnight later at the bottom of page 12. 

Meanwhile, cyber attacks were the first thing investigated and ruled out in Spain's 'el Apagón' drama. 

And all the SunAmp owners here in our happy green forum can relax - the kit was patched before the story even went public. 😜😁

Posted by: @lucia

↑

One more thing, there are already bills going through the Spanish parliament to change laws and put out to public consultation new operating procedures to better reflect Spain's rapidly evolving energy production. 

Seems that they had to use the impetus of the publishing of the report to alert that over the last couple of weeks, things have not been working as expected. But never too late if they implement those 6 weeks period with better standards. I can imagine commercial operators would be in a bind. If it works, they will have to accept new rules. 

Posted by: @lucia

↑

Oh and once again I was told the recovery was a 'master class in black starts'. Big up little Morocco who played a quick acting role in this and never gets a credit for it outside of the Peninsula. 

I am not sure about what would have happened if the Spanish grid had not been helped to recover by Morocco and France. Have all the black-starts worked as expected? 😉

As what happened in Portugal was transparent in that weekend´s newspapers and it teams were shipped on hour trips carrying generators to start a hydro plant with black-start and that was the backbone of the autonomous recovery. This report at some points seems to overlook these little things, but common sense is that recovering predictably in a few hours can only happen when black-start capability is real and so demonstrated in real tests regularly. REN were already going to double the number of generators able to do a black-start. And hopefully Morocco will be able to help Portugal directly in the future. 

And RE seems to be pushing to reign in the commercial factors soon. But in that larger market is the same level of transparency possible?

Posted by: @lucia

↑

Never mind that much to my horror, the number of major organisations running systems on Windows XP or whatever dinosaur kit they have, is absolutely beyond belief. 

It may seem shocking, but it is not a problem per se to have old kit. As having compensating suitable controls may reduce the risk for those organisations. Clearly if these would be laptops moving around the organisation, then swapping them for new and if essential, moving whatever needs Windows XP elsewhere behind security layers, etc comes to mind as an option to consider. Surely Windows XP hardware cannot be the backbone of internet facing solutions. 

Now applying the same thinking to the electricity domain, how can it be justified to have oil cooled electrical distribution plants still in use? 😀

Posted by: @batpred

↑

Posted by: @lucia

↑

Never mind that much to my horror, the number of major organisations running systems on Windows XP or whatever dinosaur kit they have, is absolutely beyond belief. 

It may seem shocking, but it is not a problem per se to have old kit.

…

Old hardware - no.

Old software - no.

Old (or even new) software that’s out of support and not being patched any more - yes.

Posted by: @batpred

↑

As having compensating suitable controls may reduce the risk for those organisations. Clearly if these would be laptops moving around the organisation, then swapping them for new and if essential, moving whatever needs Windows XP elsewhere behind security layers, etc comes to mind as an option to consider.

That sadly ignores the way our interconnected world works. An old, unpatched and vulnerable computer is only “safe” if either turned off or not connected to a network. More importantly, any such device locked down enough to render it relatively protected is then so hamstrung it’s almost guaranteed to be effectively incapable of doing the job it’s supposed to do.

Posted by: @batpred

↑

Surely Windows XP hardware cannot be the backbone of internet facing solutions. 

I wish your optimism matched reality. Sadly I have seen plenty of examples of exactly that; perhaps not “backbone”, but certainly “key component”.

Posted by: @majordennisbloodnok

↑

I wish your optimism matched reality. Sadly I have seen plenty of examples of exactly that; perhaps not “backbone”, but certainly “key component”.

 

Agree...but worse still is a human.  Many hackers gain entry initially by exploiting the weakness of human beings.  Eliminate those and we would be much safer.  

@batpred

And RE seems to be pushing to reign in the commercial factors soon. But in that larger market is the same level of transparency possible?

 

Yeah... we won't mention 'missing data'... 

Dunno, Sara Aagesen & Concha Sanchez have had a few squabbles - over responsibility for the balancing mechanism as I mentioned above - but ultimately they're on the same page, I think.

But the anonymous plant operators are a whole otra cosa!  

It's not just Spain, the entire 'energy transition' is like the wild west. 

REN was a big presence at the Brussels gig. It's really interesting how Portugal is moving forwards. 

Meanwhile, Sara Aagesen has just had to issue 'keep calm and carry on' noises because the slightest ripple now and everyone jumps sky high. 

Posted by: @lucia

↑

my point is the need to not fall for the clickbait that gets manufactured out of this 'Reds under the bed' narrative. 

That's an important point for this entire topic: Renewables in the News

For the record, I don't read 'news' stories pushed in my direction by social media.
So you won't find me raising such sources here!

When I come across a story from a sector of the 'regulated press' including TV News and documentaries, my habitual response is to first check their sources.

That's why I referenced the Reuters announcement of 14th May rather than merely quoting the Times and Telegraph articles which were based on it.
And I then cross-check technical issues with technical reports, two of which I've cross-referenced on this Forum.

As it happens, the potential for malicious use of grid-connected generation/storage was an issue I was already well aware of.
I had already been in communication with the Commons Select Committee on Energy regarding threats from commercial inverters in December'23 (18 months earlier).

Within those communications I provided an illustration of a cascade event taking down a section of the GB Grid.

That example does not require any form of cyber attack.
Just because a cascade event takes out a whole section of the distribution grid (as occurred in Spain) doesn't mean it should be investigated as if the reason was 'cyber'.

Don't bother trying to find those Select Committee documents from 2023, because they're not in the Parliamentary Library, for obvious reasons!

Posted by: @batpred

↑

I am not sure about what would have happened if the Spanish grid had not been helped to recover by Morocco and France. Have all the black-starts worked as expected?

Erm... It isn't possible for Spain or Portugal to have recovered without assistance from neighbouring countries.

The whole point of re-starting a grid in Europe is that you have to synchronise with others!
You can't have two different 50Hz frequencies operating on the same grid.

Equally, that's why a black start has to commence with

• running up generation sites with rotating turbines
• connected to the neighbouring country's grid via a 50Hz analogue interconector

You can't restart a grid using the high-efficiency 1-million-volt DC interconnectors.
They're basically a pair of inverters back-to-back in different countries.
Each inverter cannot (must not) have its own concept of 50Hz!

The DC interconnectors rely on having an operational grid at 50Hz at each end before they can be re-started.

Britain, for example, could not commence a black-start using the North Seal Link with Norway because it's DC.

We would first need to use a 50Hz interconnector, such as BridNed from Holland, to bring up our transmission grid.