RandomizedDelaySec=12hr
That, exponential backoffs, and local cache for external resources needed to operate the CDN (such as configuration) were the foundation of running a large network reliably.
Amusingly, you can also observe periodic bumps if your metrics don’t have enough resolution, due to aliasing effects dominating signal (guestures vaguely at Nyquist frequency).
The specific resolution is the Nyquist frequency - that part is accurate.
Similarly, sample count/interval limits in tsdbs like prometheus are not downsampling. They're just skipping samples. This also creates aliasing. Can be very noticeable especially if you combine it with rate/derivative functions.
Yes of course, there was the check at 2 am local time before ...
I wished cron had a similar feature. I missed that on acouple of occasions.
Seems like they did not find the exact manufacturer or the exact devices and were not able to change firmware then?
"following our communication with the manufacturer, a software update was issued that removed the strict periodicity of the current peaks in one battery block of the investigated UPS system"
Given the issue is so widespread (international, affecting pretty much every grid), it's likely that multiple manufacturers use similar approaches.
Btw, one of those numbers is not like the others (has more prime factors that evenly divide 60).
Systemd timers have some, for example RandomizedOffsetSec=30m on 30m job will spread it nicely.
I'd also like if schedulers just had option "just fit it in with other jobs and fire it with this rough interval when there is least amount of other jobs running"
The funny thing is, modern electrical codes already account for this phenomenon. In order to implement Net Metering (selling your power to the grid), modern standards implement two kinds of jitter:
- Sandia Frequency Shift: a minute shift in the power frequency generated by inverters. If the grid is connected, this is effectively eliminated; if the inverter is disconnected from the grid, it detects the frequency has shifted and disconnects the inverter from the grid.
- Slip-Mode Frequency Shift: basically the same thing, but phase shift (which I don't understand).
Both of these are used for anti-islanding (don't create an island of power). When the inverter detects it's isolated and stops outputting current, it then has to check every 5 minutes, with jitter, for whether the grid is restored or not. It has to add random jitter, because otherwise all the mini battery/solar installs will come online at the same time and swamp the grid, causing another outage.
Obviously cron jobs are not all equal in terms of load and duration, and hours are not all equal in terms of usage, but that still helps prevent trivial spikes similar to the one mentioned in the article (disclaimer: I'm not aware of a recent counterfactual experiment proving this, but that seems a fair assumption).
* * * * * sleep 17 && …
That way I still maintain a per-job 60-second cadence consistently (which random would not), but don’t do it right on the top of the minute.
If the answer to that is all-yes, then my compliments to you. Alas ... at least my experience says that's not how things are done.
Journalctl with SQLite or a columnar embedded database would be amazing for sysadmins. Current format is entirely useless
give it number and you will get random value up to that number, same every run for a given host, but different per host (basically, hashed host fqdn)
add a label and you will get different number but similarly consistent. Makes it real easy to randomize run across the fleet.
Would be easy script to make.
Also systemd timers have the randomization builtin, just need to specify time
RandomizedDelaySec=12hhttps://shoppress.dormanproducts.com/hunting-tooth-gearsets/
Analog systems naturally have a dispersion to them, digital ones, especially with everybody on NTP, much less so
There was an post a couple of days ago about a telco debrief of an interesting outage that happened. Several things went wrong like the audible bell was manually silence from being heard as a step in routine working but was forgotten to put back into normal mode once work was complete. The actual loss in power triggered the automatic switch to the batteries, but something else seemingly benign happened at the same time so that when power was restored, the batteries had been depleted to the point the charging process could not start because of something similar to this. I wish I could remember the write up, as it was really well written about the cascades of failures that occurred.
The turbines/generators set the grid frequency itself.
Having the batteries disconnect/connect would cause the generators to slow down and then speed up (or vice versa) which makes the rest of the grid frequency fluctuate as well.
It may also trigger the FCAS part of the grid that is paid to keep the frequency within the required range which is +/- 10mHz-200mHz.
Doesn't seem that complicated from the article though. They say that they immediately knew it was clock based due to the strict interval. With REA etc. it might become easier to proof, but I guess it's a measurement problem that only appears in large networks?
Can someone with more understanding explain why this was apparently so difficult to find out?
It's just bulk flows, so when a weird flow shows up, you've got to move up or down the line, past a few producers or consumers, and see if the flow changes. For a subtle effect, that might not even be so clear.
Worse, for an effect that's most visible at the medium-voltage transmission level, simply placing a monitor might take weeks. These are planned installations with incredible safety considerations -- you don't just plug in.
Finally they found a research campus with a bunch of battery storage systems, AND the ability to install a bunch of monitors, and persuaded them to install dozens of monitoring points. Finally a high-resolution view of the effect allowed them to confirm the behavior.
FFT gain = 10*log(M/2)
where M is the number of points processed in the FFT. Doubling the number of points, lowers the apparent noise floor by 3 dB.
(I don't know if that's what the researchers actually did. I'm just explaining what the comment presumably meant.)
Because the grid as a whole is dumb as fuck. Sure, you got measurement stations doing very fine frequency analysis, to the point you can use grid hum to timestamp audio recordings [1].
But you rarely have realtime insight at individual busbar level for each substation or heaven forbid consumers. Best chance you get is 15-minute consumption pings and voltage/frequency metering at large plants.
That means if you want to track down bad actors like the batteries here or the Kosovo/Serbia bullshit [2] that dragged down the frequency on the entire ENTSO-E grid (which is quite the challenge!), it takes months if not years.
[1] https://en.wikipedia.org/wiki/Electrical_network_frequency_a...
[2] https://www.entsoe.eu/news/2018/04/03/frequency-deviations-c...
This is kind of understandable, the amount of data would be huge. When faults happen though, recordings are automatically made available.
Some of these measurements can have surprisingly high frequency sampling, even though the grid itself feels "slow".
Depends on the fault. Without dedicated measurement in place it's still a lot of (literal) legwork to install current and voltage loggers.
> Some of these measurements can have surprisingly high frequency sampling, even though the grid itself feels "slow".
Sure. But especially in a world dominated by more and more solar and wind, where something like a cloud passing by can easily shed hundreds of kilowatts in a matter of seconds, the grid needs much, much more data across the board and everywhere - and much, much more battery storage, both centralized at substations and decentralized in people's basements.
Likely because the source of the noise was a class of energy providers that themselves react to fluctuation in the grid. Electricity may travel at the speed of light but all the little switches and transistors that connect/disconnect suppliers do not. It would have been noise within noise.
https://en.wikipedia.org/wiki/Velocity_factor#Typical_veloci...
Oh I’m sure there is a German phrase for this.
Similar to “allzuschön um wahr zu sein”, “allzugroß” etc.
Well, then it seems we don’t have any single German word for all too regular anymore :(
My new proposal would be überregelmäßig. Über+adjective is still one word only, post-reform. Sounds strange, but grammatically correct, and still conveys the “unusual” part.
Still, I think we can push the limits to Bürokratieapokalyptischhyperregelmäßigkeitsgötterungeheuerhaft and feel pretty comfortable with it.
https://shoppress.dormanproducts.com/hunting-tooth-gearsets/
You have to consider that when power goes down and everything goes back up at same time you get same pattern back, just spike every 59 seconds instead of 60.
Why though? Due to increased torque?
To some extent, this is a good thing. Because it means the spinning mass of all turbines in the grid actually provide a buffer for sub-second mismatches in demand and supply. But it does mean that big demand spikes (or drops) put some wear on the systems. They are made for this, but if it gets very periodic it might be more than designed for.
Note that this means that all turbines in a power network are perfectly synchronized. And if they are not, they are subject to massive torques. This is why a cold start is so hard. You need to bring on generators from one central point to keep everything in sync. And as you bring on new generation capacity you also need to bring on new consumption, that electricity has to go somewhere.
The reason for this synchronization is because the generators are spinning a magnet past 3 coils, and those coils are directly wired into the power network. If your magnets are out of phase then effectively the entire power network will use those coils to generate a field to get that magnet back in sync. All of those coils on all those generators really are all hard-wired together, it still kind of blows my mind.
Broadly speaking, it’s a collective momentum problem. If part of the system inertia sags, another part needs to step in to compensate for the inertia loss, otherwise you have brownouts and eventually, a blackout.
No good. I am out that week. How about the 3rd week of November?
That's no good. Second week of December should work for all of us.
--near actual conversation with (State) government coordination team to have a 20 minute discussion.