Plant startup is only the first step.[1] It's load pickup.
Here is a PJM training module on load pickup during system restoration.[2] It gives a sense of how touchy the process is. Power network control has good control over generation and transmission, but limited control over load.
When load is turned on, there are transient loads with different time constants. There's a huge load for the first second as inductors, capacitors, and incandescent filaments start up. That tails off in under a second. There's a second load as motors wind up to speed. Ten seconds or so. Then there's "cold load", where everything in HVAC starts trying to get temperatures back to normal. Maybe half an hour.
There's no mention of computer control. Listening to this, you visualize people running around reading meters and throwing big switches. It's probably people looking at display boards and sending commands to remote big switches, but the concept is the same.
Botching this means voltage or frequency goes out of tolerance, protective devices shut things down, and the system operators have to start over.
More PJM training modules on related subjects.[3]
Unclear what caused this yet. Something caused enough system instability to trip protective devices, but there's no good info yet. Once everybody has a chance to compare all the logging data from different points, it will make more sense.
[1] https://pjm.adobeconnect.com/_a16103949/p622tuwooba/
There are costly means to compensate for the lack of spinning baseload but actually building these devices have been neglected, to no ones surprise.
Looking at power generation reports it's the plants relying on inertia (like nuclear) which were disconnected and haven't been reconnected since:
In a control system when you have an actuator that is much more responsive than the load, you can get into instable operating regimes where. On a small scale, when you are setting up servo motors, there is an inertia matching ratio for stability. In some of the systems I set up it was 10:1 load inertia:motor inertia. If you exceed that, you end up with elasticity between the motor and load that the motor can locally be above set point, and the control tells the motor to slow down, while the load has not yet reached set point. The motor is then too slow, and the control tells it to go faster, and you get the motor oscillating around the speed of the load.
On the grid you can have similar elasticity. An inverter many km away can get too far ahead in the AC cycle, then when that signal reaches the local inverter, the local inverter slows down, but the remote inverter has already slowed down, and by the time that slower wave gets to the local inverter, it is already too slow, and so the local inverter starts to speed up. If you get the frequency "right" you can end up with a positive feedback loop. With a high inertia rotating generator, the system is damped and slow enough to respond you don't change set point faster than the signal can propagate.
At the start of solar and wind integration, it didn't matter. Large rotating generators were the main source on the grid, and provided stability. We're in a different world with an all-inverter grid control perspective-wise.
I think there is also still an information issue. The grid is definitely not point-to-point. It also isn't all generators. You also have loads switching on and off at indeterminate times. There is an overall predictability to it, but not at the phase governing resolution. If you have a local inverter and the phase slows down, you don't know if it is a remote inverter that has over-slowed, to which you shouldn't respond as it will fix itself, or a near-by load that suddenly switched on, to which you should respond as demand is now greater, and needs more supply. With big rotating generators, it doesn't really matter, as they are time averaging with the large inertia to stabilize the load/generator balance.
-react immediately
-use its full capacity all the time
Imagine you're trying to coordinate a choir of ten million singers that are scattered across a radius of a thousand kilometer, that all sing into their individual Ham radio. And you need it to come out in a perfect unison that sounds good for those who tune into the mixed broadcast.
"A problem of coordination" is perhaps correct, but it neglects the difficulties involved
Also we know the locations of the generators, so we can calculate the phase shift and control them accordingly. At thousand kilometres it's around 60 degrees anyway, so not catastrophically huge.
I'm pretty sure in grid terms, 60 degrees out of sync is catastrophic. I'm pretty sure you'll be disconnected by protection circuits before it gets that far out.
And even if we did have the superluminal grid sync signal the propagation of electricity itself is slower than C and you'll need to consider perceived phase synchronization over distance.
If I can sync ALL my [important] servers across the globe to the EXACT same time (under 2 microseconds variance in the last 7 days), and all that on the off-the-shelf hardware, surely a NATION can devise something better?
(also replaces the need for synchronous condensers where they would typically be sited for grid ancillary services)
https://www.abc.net.au/news/2021-05-26/queensland-blackout-c...
https://www.pscconsulting.com/australia-leans-in-on-syntheti...
https://aemo.com.au/-/media/files/initiatives/engineering-fr...
https://www.ercot.com/files/docs/2024/01/12/Tesla%20BESS%20G...
Our recent govt installation is reported at $1.6bn for 2000MWh, which is incredibly expensive, and can't scale to run the grid, simply smooth load/demand diurnal curves.
https://news.ycombinator.com/item?id=43624308
https://www.vox.com/climate/408381/energy-transition-renewab... | https://archive.today/sZFxF
https://www.eia.gov/todayinenergy/detail.php?id=64586
https://about.bnef.com/blog/lithium-ion-battery-pack-prices-...
I'm not saying "at some point" batteries won't be viable as grid "storage", I'm just saying today they aren't.
You are still correct in the sense that storage is still incredibly expensive per hour of demand stored, but batteries may very well already be the cheapest option out there
https://en.m.wikipedia.org/wiki/Snowy_2.0_Pumped_Storage_Pow...
More importantly, according to the REE, PV played a mayor role in the current and successful black start.
If the power is out, do you imagine that all the load that is out there disappears too?
If I'm in a blackout and put my multimeter into the grid socket, will the measurement be something that my local inverter can drive an AC load through with no problems?
And if that's not possible, how do we get an AC wave out to my 5000 local solar panel neighbors? And if we don't get any carrier wave to synch it to, then how do all these thousands of individual inverters decide on the unison synchronization necessary to all start jumping together?
And if they try and fail, will all the connected load accept a graceful out of synch mixed power noise at 0-220v until things properly latch together?
The problem isn't like playing a sine tune of acoustic sound that's merely audible to your local neighbor, who can dial in his own tune, and then daisy chaining this until it matches on a grid sized level.
Then depending on how out-of-sync a given station is with the signal, they ramp production up or down accordingly.
Is the problem here that there is inadequate or slow feedback to the supply side regarding load and therefore you'd end up with under-voltage?
It's not about getting the generator to spin at 50hz, it's about getting it to spin at 50hz under dynamic load. Most generators are mostly fixed speed I think for the sake of efficiency, so the mechanism the grid has to regulate the frequency of power is largely by controlling the number of generators running at any given time, and how much of the grid is connected to them. Ramping up and down based on the current observed frequency of the grid is exactly how this all works. However it's more complicated because what if two or more power stations see a low frequency grid and decide to ramp up more generators? Well now the grid frequency is too high. If they them respond to that by ramping down, it'll go too low. Rinse and repeat and you get undesirable frequency oscillation, so there needs to be more communication across the grid than just responding to the locally observed frequency.
Connect a stopped, or sufficiently out of phase generator to a full scale power grid and the puny thousands of pounds of metal that make up a generator are going to get ripped apart by the sheer inertia of the power grid forcing it to match speed and phase in an instant. This is also a problem when attempting to connect two isolated grids, of they're too out of phase with each other, catastrophic physical damage will occur.
Any individual station that gets out of sync either works hard to catch up, or gets disconnected.
The problem, as I see it, is that the load signal is equal and equivalent to the sync signal and maybe they could be separated. (e.g. the FM modulated signal wouldn't have to be a sine wave, nor would it even have to be 50Hz)
> what if two or more power stations see a low frequency grid and decide to ramp up more generators?
Seems to me it would solve this problem - they both have a single invariant sync signal.
> Connect a stopped, or sufficiently out of phase generator to a full scale power grid and the puny thousands of pounds of metal that make up a generator are going to get ripped apart
Seems to me this problem is solvable with electronics, but I don't know enough to say how much out-of-phase is problematic. What if, as you say, some heavy load is switched on, the local nuclear turbines slow by a fraction for a few 10s of seconds, and so a home PV array starts leading by 0.01Hz (e.g. home PV at 50.00Hz, local power station at 49.99Hz, so after 10s the PV is fully 36º or 0.6rad ahead). Does this result in the inverter (or the small local windfarm) exploding? Is that a realistic scenario?
> there needs to be more communication across the grid than just responding to the locally observed frequency
Precisely - a grid-wide sync signal, modulated at high frequency.
A millisecond latency will cause a significant out of phase shift to the AC signal too.
The AC data you want is already codified in the power signal
Ooh, good point. But then, we have I think micro- or even nano-second clock sync over public internet now. Or use (reliable) GPS timestamps codified in the sync signal.
> The AC data you want is already codified in the power signal
You're right, but:
> there needs to be more communication across the grid than just responding to the locally observed frequency
Maybe large stations need a grid-wide sync signal that is independent of load, but small (e.g. domestic, community) suppliers just latch on the observed AC (which is easy for an inverter).
Thermal and hydro plants use synchronous rotating generators that have physical inertia, and the generator rotational speed is exactly locked to the electrical frequency. (proportional to the number of poles in the generator).
On-topic: Wind power brings in about 20% of Spain's electricity, and that one fluctuates a lot indeed. I hope they put out a detailed post-mortem, I'd be an enjoyable read.
Some even borrowed the essence of it, posted it elsewhere and still downvoted, ¯\_(ツ)_/¯.
(also we're a bit early for a proper postmortem, but that's an inevitable reality of social media, got to get the talking points in before the facts)
I agree as well. And I think the gp (or the ggp?) post noticed that there is a lack of "spinning mass" to do frequency damping would have agreed as well. They could be merely suggesting to add just spinning mass synchronizers. These could be independent units, doing just that or maybe even convert some old power plants: "just" decouple the turbine from the generator.
The solution in either case is the same. Put enough smarts in the inverter for it to have "synthetic inertia". If we didn't have the ability to do something like that in inverters, the solution would be something like flywheels to augment the chemical batteries.
It dropped from about 32GW to about 8GW
source: https://transparency.entsoe.eu/generation/r2/actualGeneratio...
though admittedly https://en.wikipedia.org/wiki/Black_start mentions that partial shutdown may be also requiring a black start?
It looks to me like the author of this article has fundamentally misunderstood what a black start is. The fact that some power generation remained both on the peninsula, but also on imported power being available means that to my understanding this cannot be classified as a "black start". A lot of overtime, a lot of careful coordination yes, a black start, no
it's that in Factorio there are no problems with phase synchronization, and the devices, consumers, producers, so the machines don't get damaged from a lot of abrupt starting/stopping.
still, it's strange that operators and grid controllers have to do the restart manually.
Dyson sphere program is the by far most forgiving in that regard. You can overdraw to something like 200% before it becomes really problematic, and even that quickly resolves itself the moment you get more power online
Same thing with all the coal burning... how does that coal get ignited? Do we secretly have an infinite lighter in our pocket? Are we rubbing sticks together but not showing it for the sake of gameplay?
In satisfactory, all pumps need water and there is a special kind of power plant that doesn't need pumps/electricity to kickstart the factory. It's special because it needs items the player has to manually gather and only runs "on demand" (shuts off if enough power from regular plants is provided)
Strong recommendation to give it a try if you haven't already. It's very different to factorio, but great in it's own way
My girlfriend had more fun collecting leaves for the biomass burners and playing the tetris game in the workshop while I built the conveyors and calculated ratios... I let her keep adding leaves even once we were burning coal, but she eventually noticed leaves weren't burning anymore :)
In Space Age, more than once I've needed to fly across the solar system to hand-restart a generator when I didn't have my fail-safe setup correctly. Normally I setup a SR latch to fall back to the power generators (whatever type they are) _only_ on a sub-grid when capacity falls below 20% (and alert me) since if you lose power completely, hand-feeding is the only way to fix (unless you have another air-gapped power storage/generation set up). It sucks when you hand-feed and then see all your power generation still struggling because you accidentally left in 1 copper wire to a pole outside your "air-gapped" grid. Having to slowly rebuild on Aquilo solar->nuclear->fusion is tedious, after making that mistake once I make sure I didn't have to travel back again to fix the grid... After the 2nd time it happened I finally fixed it for good.
During that time, you are producing power, but none of the inserters work, none of the miners work, etc. and the distributed nature of the roboports means it isn’t straightforward to simply disconnect them.
https://www.youtube.com/watch?v=uOSnQM1Zu4w
from Practical Engineering
As a result, the government invested heavily in installing gas turbines all over the country.
When we had another blackout event in 2018(?), it only took an hour or so to get things back again.
The downside is that the choice was made to focus on smaller 10-20 MW projects, with the occasional 100+ MW installation.
Hence, despite new projects entering service every couple months solar only accounts for less than 3% of electricity production nationwide.
In short: Heat Inertia of large molten bodies is massive with good insulation. If the time is too long tho, only dynamite will dismantle a solid chunk of material again.
https://www.blindex.com.br/-/media/blindex/site-content/xx-n...
The Texas winter storm resulted in a lot of scrap at semiconductor lines due to power loss. There are industries that are completely dependent on grid power. You can't generate enough on site to back up 100% of your operation. Think about how much power one EUV source consumes.
The only reason Samsung is building additional factories in Texas is because the local utilities are effectively treating them as a critical load. The new plants are right by the ERCOT operation center and likely have access to the same cranking paths that the grid uses for black starts.
https://www.asml.com/en/news/stories/2024/a-sustainability-m...
There are thousands of tools. Material handling robots, compressors, HVAC, pumps, an entire data center within the factory, etc.
> For example, a lot of that 1.3 MW went into cooling the system and handling the hydrogen flow used to prevent contamination build up.
Actually, it’s not if you read the linked article. Try harder with your indignant reply next time to an entirely factual post. Sheesh!
Further, here’s a >one hundred megawatt generator installation (yes, used, but it’s always call for quote otherwise online) for a small fraction of the price of the EUV machine in case I need to further prove my point.
https://www.uspeglobal.com/listings/1705462-used-100-mw-2004...
There’s also multiple GE LMS100 generators installed in Texas and they each provide ~115MW of power. So not unprecedented for installation in the state either!
First hit off google.
Assuming they run them 24/7 that amounts to a power usage a bit over 1 megawatt, but they could be pulsed for all I know and use quite a bit more
They are also frequently situated close to reliable power sources such as nuclear or hydro, usually fed by more than one generating station.
A friend of mine lived close to a brick plant his power NEVER went out.
The one thing that surprised me is how quickly rumors started about power being out in Portugal (true), France (true to an extent), Belgium (false), and the UK (false). Walking back home from work you heard the names of Trump being mentioned as well.
It's a bit scary to personally experience how quickly people can start panicking. On the other hand the bars that had some power had a great afternoon with tons of beer being served before it got too warm :-)
On the upside, more people at all levels are going to be more prepared if it happens again.
Solar flares or temperature anomalies are much easier explanations. https://en.meteorologiaenred.com/What-is-induced-atmospheric...
Refrigeration is sort of a scourge to those who enjoy food. We are so used to simply freezing our esophagus and not tasting anything! Food is kept cold because it lasts longer, not because it tastes better.
Sure, refrigeration makes restaurants and grocery stores possible. But it also presents challenges to cooking and it's very resource-intensive -- think about it -- basically everyone everywhere is always running their refrigeration -- that's a huge load on any system.
Unless you drink Schultenbrau, which tastes worse cold than at room temperature (/s, kind of).
The same goes for wine, by the way: cellar temperature is usually optimal.
Do they? But why? Why would they? Are they simply accustomed to tasting it that way? I should say "not tasting it" because chilling a beverage basically kills its flavours, and they never make it to your tongue correctly.
I would say that many people would be surprised and amazed if they tasted such things as Pepsi without ice, and without coming right out of the fridge.
Unfortunately, any lack of refrigeration really messes up supply chains and will eat into anyone's bottom line, so it's not a realistic goal.
Beer also does not require refrigeration throughout the entire supply chain, they happily sit in a warm warehouse waiting to be picked up by an unrefrigerated truck, only to be cooled at the outlet or consumer.
Smell, touch, look, mouthfeel, nostalgia, temperature, color, other various human factors (ie I always eat this with my friends), sounds, cost
ALL of these affect your experience of consuming something.
Your insistence on ignoring the holistic experience because cold things don't react as well to produce taste sensations is misguided.
Never mind that plenty of things a DESIGNED to be consumed cold, and eating it room temp or warm would be contrary to the design. Beer and other soft drinks are often in this category as they intend to be "refreshing".
You don't drink your beer at 85 Celsius right? Why not? You will "taste" it more!
Yes, in fact, there is nothing to kill my enjoyment more than a watered-down icy beverage that murders my tongue and esophagus as it goes down.
Since my youth I’ve been a choral singer and musician. I’m quite sensitive to keeping “my instrument” in top condition. Therefore I take offense from anyone who says that [Scoville levels] of spice should be calibrated for murder, or that beverages should be as close to freezing as possible.
This is doing violence to the consumer, and it’s simply unjust across the board. If I had a nickel for every butt-hurt counter girl just because I asked for “no ice please”; if it really hurts your bottom line then charge double? Consumers aren’t actually afforded much choice about the temperature of food we consume, are we?
There is basically no scientific or cultural reason for food or drink to be consumed while it is cold [below 40°F]. Simply marketing and capitalism because of food safety and profit. Therefore I need not kowtow to someone’s haughty decree that soda is somehow designed to be drunk that way.
People have been panicking, internally, for quite some time. What happened is that the threshold to externalize that panic was dropped with this event. So this is a very real example of how people are always looking for confirmation of what they already believe, and for a relatively brief time after they receive that confirmation, can live in a reality that is not at all aligned with base reality. I find this kind of thing fascinating, and it's been happening more and more in recent years due to changing dynamics in the way we communicate.
Sounds simple, and I appreciate from experience a huge amount of prep and validation needs to be done in the background.
Because if not sequenced and interlocked, islanding behavior can result in back-powering parts of the grid that are supposed to be disconnected and down for maintenance, or creation of desynchronized islands that then cause more damage when reconnected.
This is absolutely a problem that operators need to solve. Radio synchronization is possible, neighborhood islanding could be beneficial, etc. There are entire village-wide micro-grids in remote places where everyone's inverters do perform these functions, it just hasn't been embraced in monopoly environments yet.
For instance, relays that close isolated parts back onto the main network would need to be phase-aware, to only reconnect if it's safe to do so. You could mandate that all island-mode inverters center themselves around 50.1 or 60.1Hz in the absence of outside influence so there's certain to be phasing opportunities. Distribution networks would need more switches to provide positive isolation during maintenance. And all of these things would need huge amounts of interoperability testing. It's possible, but it has a cost.
I’m guessing this could be more of a thing in the future to make the grid more robust.
Didn’t Australia have a similar issue a few years back, which Tesla built a large grid battery to resolve, not just for storage but for stability.
If everyone's inverter can feed the line you have a situation where you can never be confident the line is dead and you must always work the line as if it were hot.
In the end it took about 12 hours to restore most of the power in two countries. There are some knock-on effects from trains being in the wrong places, but a solid result overall.
I operate a microgrid facility in Hispaniola and have wonderfully cooperative users backed by a separately powered communications and control system. Even for us, a facility serving a small neighborhood and farm, a black start must be performed as a careful choreography of systems and loads…and we can just pick up a radio and tell people to turn the main breaker of their house on or off, and to leave their AC units off until we finish bringing everything up. In 12 years we’ve only had to do it twice, but even for us it’s a tedious process.
Oh dear.
