In another life I worked as an engineer commissioning oil rigs and I’ve seen how tricky even a small-scale black start can be. On a rig, we simulate total power loss and have to hand-crank a tiny air compressor just to start a small emergency generator, which then powers the compressors needed to fire up the big ~7MW main generators. It's a delicate chain reaction — and that's just for one isolated platform.
A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.
I remember talking to my ex's dad about his job, which involved planning refuels of a large nuclear-powered generation station in the Lower Midwest.
The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. It's a system basically built to produce "common accidents". It's amazing that it doesn't on a regular basis.
> The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true.
Funny thing is, those are the exact words I use when talking to people about networking. And realistically anytime I dig deep into the underlying details of any big enough system I walk away with that impression. At scale, I think any system is less “controlled and planned precision” and more “harnessed chaos with a lot of resiliency to the unpredictability of that chaos”
This is one of the key insights in my early SRE career that changed how I viewed software engineering at scale.
Components aren’t reliable. The whole thing might be duct tape and popsicle sticks. But the trick for SRE work is to create stability from unreliable components by isolating and routing around failures.
It’s part of what made chaos engineering so effective. From randomly slowing down disk/network speed to unplugging server racks to making entire datacenters go dark - you intentionally introduce all sorts of crazy failure modes to intentionally break things and make sure the system remains metastable.
Everything is chaos, seek not to control it or you will lose your mind.
Seek only to understand it well enough to harness the chaos for more subtle useful purpose, for from chaos comes all the beauty and life in the universe.
We would instead have HaaS, with monthly subscriptions for a license to use the house. Which can be randomly revoked at any moment if the company doesn't feel like supporting it is profitable enough, or if an AI thinks your electricity usage is suspicious and permabans you from using a home in the entire town.
A bit of a tangent, but I don't think this is it. There are plenty of species with plenty of shared norms, expectations, and trust - but no civilization. And, vice versa, many of the greatest societies have been riddled with completely incompatible worldviews yet created amazing civilizations. Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart!
The reason people work together is fundamentally the same reason you go to work - self interest. You're rarely there because you genuinely believe in the mission or product - mostly you just want to get paid and then go do your own thing. And that's basically the gears of society in a nutshell. But you need the intelligence to understand the bigger picture of things.
For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps. They even wage war over them and in efforts to expand them or control their territory. This [1] war was something that revolutionized our understanding of primates behaviors, which had been excessively idealized beforehand. But they lack the intelligence to understand how to bring their little societies up in scale.
They understand full well how to kill the other tribe and "integrate" their females, but they never think to e.g. enslave the males, let alone higher order forms of expansion with vassalage, negotiated treaties, and so on. All of which over time trend towards where we are today, where it turns out giving somebody a little slice of your pie and letting him otherwise roam free is way more effective than just trying to dominate him.
> There are plenty of species with plenty of shared norms, expectations, and trust
Citation needed on that one.
> Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart!
They spoke the same language, shared the same literature, practiced the same religion, had a long history of diplomatic ties. When the Persians razed Athens, they took refuge with the Spartans.
> For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps.
Again, I don't think this claim stands to evidence. The so called chimp war you mention is about a group of about a dozen and a huge fight that broke out among them. That doesn't support the idea that they are capable of 200-strong 'intricate' groupings.
"They spoke the same language" ... not exactly, the Spartans spoke Doric, while the Athenians Attic. (Interestingly, there is a few Doric speakers left [0].) While those languages were related, their mutual intelligibility was limited. Instead of "Greek" as a single language, you need to treat it as a family of languages, like "Slavic".
"shared the same literature" ... famously, the Spartans weren't much into culture and art, and they left barely any written records of their own. Even the contemporaries commented on just how boring Sparta was in all regards.
If we delve deeper into ideas about how a good citizen looked like, or how law worked, the differences between Sparta and Athens are significant, if not outright massive.
While those two cities weren't entirely alien to each other, had some ties, same gods, and occassionally fought on the same side in a big war, there was indeed a huge political and cultural distance between them. I would compare it to Poland vs. Russia.
Not "entirely alien, had some ties" is not it. They were part of the same cultural cluster, participated in the same games, traveled to the same sanctuaries, had mutual proxenies. The very fact that we know the opinions of several Athenians about Spartans is telling. We don't know what they thought of inhabitants of Celtic population centers, or Assyrian cities, or Egyptian ones. But we know what they thought of individual Spartans that they mention by name, biographical detail and genealogy.
I stand by my comparison to the Slavic nations of today.
Yeah, we have a lot of opinions of one another, yes we understand basic vocabulary of our cousins, though details in fine speech are another matter, yes, we are technically Christian, but still the political and societal difference between, say, Czechs and Russians is quite big.
As was the difference between the Spartans and the Athenians. Constitutionally, the poleis were all over the map, from outright tyrannies, through oligarchies and theocracies, to somewhat democratic states.
So your argument is: Athens and Sparta had things in common but were different. Like Czechia and Russia. Czechia and Russia are quite different. So were Athens and Sparta?
Try to speak holistically. I have no idea what you're trying to argue. I could expand or provide evidence for everything I said, but providing a citation or proving that there are indeed social groups of upwards of 200 chimps, or whatever, isn't going to do much, because you're not really formulating any argument or contrary view yourself.
Put another way, you're arguing against an example and not a fundamental premise. Proving the example is correct doesn't really get us anywhere since presumably you disagree with the fundamental premise.
That sounds very much like "Just believe me." or even more "The rules were that you guys weren’t gonna fact-check"
> I have no idea what you're trying to argue.
Presumably you know what you are trying to argue. That is what the questions were about.
> Proving the example is correct doesn't really get us anywhere
You would have solid foundations to build your premise from. That is what it would get us.
First we check the bricks (the individual facts), then we check if they were correctly built into a wall (do the arguments add up? are the conclusions supported by the reasoning and the facts?). And then we marvel at the beautiful edifice you have built from it (the premise). Going the other way around is ass-backwards.
> you're not really formulating any argument or contrary view yourself.
I don't know what viewpoint namaria has. I know that "Sparta and Athens [..] couldn't possibly have been further apart" is ahistorical. They were very similar in many regards. If you think they were that different you have watched too many modern retellings, instead of reading actual history books. That's my contrary view.
> For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps.
Here the question is what do we believe to be "societies". The researchers indeed documented hundreds of chimps visiting the same human made feeding station. Is that a society now? I don't think so, but maybe you think otherwise. What makes the Chimps' behaviour a society as opposed to just a bunch of chimps at the same place?
The preppers can only buy themselves a small amount of time, though—no more than a year or two. Eventually, their stockpiled supplies will run out, or some piece of equipment will need a replacement part.
I'd much rather focus on "prepping" by building social resiliency, instead. The local community I'm plugged into is much stronger together than anything I could possibly build individually.
I am an ex-scientist and an engineer and had a look at the books of my son who studies finance in the best finance school in the world (I am saying this to highlight that he will be one of the perpetrators, possibly with influence, of this mess)
The things in there are crazy. There are whole blocks that are obvious but made to sound complicated. I spent some time on a graph just to realize that they ultimately talk about solving a set of two linear equations (midfle school level).
Some pieces were not comprehensible because they did not make sense.
And then bam! A random differential equation and explanation as it was the answer to the universe. With an incorrect interpretation.
And then there are statistics that would make "sociology science" blush. Yes, they are so bad that even the, ahem, experts who do stats in sociology would be ashamed (no hate for sociology, everyone needs to eat, it is just that I was several times reviewer of thesises there and I have trauma afterwards).
The fact that finance works is because we have some kind of magical "local minimum of finance energy" from which the Trumps of this world somehow did not maybe to break from (fingers crossed) by disrupting the world too much.
I did a lot of work for a major airline earlier in my career and came away with same impression. I just couldn’t see how they kept planes in the air based on my experiences through out the organization. I think in a big enough org the sheer momentum keeps things moving despite all the fires happening constantly.
"Funny thing is, those are the exact words I use when talking to people about networking"
Computer networking is not the same. Our networks will not explode. I will grant you that they can be shite if not designed properly but they end up running slowly or not at all, but it will not combust nor explode.
If you get the basics right for ethernet then it works rather well as a massive network. You could describe it as an internetwork.
Basically, keep your layer 1 to around 200 odd maximum devices per VLAN - that works fine for IPv4. You might have to tune MAC tables for IPv6 for obvious reasons.
Your fancier switches will have some funky memory for tables of one address to other address translation eg MAC to IP n VLAN and that. That memory will be shared with other databases too, perhaps iSCSI, so you have to decide how to manage that lot.
You tried to nerdsnipe someone without mentioning L2 is effectively dead within datacenters since VXLAN became hardware accelerated in both Broadcom and "NVIDIA"(Mellanox) gear. And for those that don't need/care about L2 they don't even bother and run L3 all the way.
EVPN uses BGP to advertise MAC addresses in VXLAN networks which solves looping without magic packets, scales better and is easier to introspect.
And we didn't even get into the provider side which has been using MPLS for decades.
A problem with high bandwidth networking over fiber is that since light refracts within the fiber some light will take a longer path than other, if the widow is too short and you have too much scattering you will drop packets.
So hopefully someone doesn't bend your 100G fiber too much, if that isn't finicky idk what is, DAC cables with twinax solve it short-range for cheaper however.
I built control computers for nuclear reactors. Those machines are not connected to a network and are guarded by multiple stages of men with automatic machine guns. It was designed to flawlessly run 3x boards each with triple-modular-redundant processors in FPGA fabric all nine processors instruction-synced with ECC down to the Registers (including cycling the three areas of programmable fabric on the FPGAs). They cycle and test each board every month.
Well, the news says that doge randos are potentially exfiltrating the details of systems like that as well as financial details of many Americans, including those who hold machine guns and probably suffer from substandard pay and bad economic prospects/job security as much as anyone else does.
Perhaps the safest assumption is that system reliability ultimately depends on quite a lot of factors that are not purely about careful engineering.
A bit off topic, but my uncle used to be security at a nuclear plant. Each year the Delta Force (his words) would conduct a surprise pentest. He said that although they were always tipped off, they never stopped them.
I guess the biggest security advantage of any of these old critical systems is fact that they are not connected to the internet. At least I hope they are not.
The regulations around parts sourcing, required maintenance, and training has more to do with how well/safe modern aviation is than anything else. If those aren’t done properly, all sorts of weird things start happening. Pretty much the only reason aerospace safety records aren’t worse in third world countries is because of how obviously bad the consequences are quickly - and even then….
I love the "analog" handcranked air compressor to 7MW generator escalation, it really captures human ingenuity.
I wonder however how being part of the "continental Europe synchronous grid" affects this, and how it isolates to Portugal and Spain like this.
But yeah there are a lot of capacitors that want juice on startup that happily kills any attempt to restore power. My father had "a lot" of PA speakers at home and when we tripped the 3680w breaker (16A 220v) we had to kill some gear to get it back up again. I'm also very sure we had 230v because I lived close to the company I worked for and we ran small scale DC operations so I could monitor input voltage and frequency on SNMP so through work I had "perfect amateur" monitoring of our local grid. Just for fun I got notifications if the frequency dropped more than .1 and it happened, but rarely. Hardly ever above though since that's calibrated over time like Google handle NTP leap seconds.
I saw some ancient footage of an Me-109 fighter engine being started. A tech jumped on the wing and inserted a hand crank into a slot on the side. He threw all his might into turning it, and then after a delay the propeller started turning and coughed into life.
I realized the tech must have been winding up a flywheel, and then the pilot engaged a clutch to dump the flywheel's inertia into the engine.
The engineer in me loves the simplicity and low tech approach - a ground cart isn't needed nor is a battery charger (and batteries don't work in the cold). Perfect for a battlefield airplane.
---
I saw an exhibit of an Me-262 jet fighter engine. Looking closely at the nacelle, which was cut away a bit, I noticed it enclosed a tiny piston engine. I inferred that engine was used to start the jet engine turning. It even had a pull-start handle on it! Again, no ground cart needed.
---
I was reading about the MiG-15. American fighters used a pump to supply pressurized oxygen to the pilot. The MiG-15 just used a pressurized tank of air. It provided only for a limited time at altitude, but since the MiG-15 drank fuel like a drunkard, that was enough time anyway. Of course, if the ground crew forgot to pressurize it, the pilot was in trouble.
You are correct, the official moniker is Bf-109, but the Allies referred to it as the Me-109.
BTW, since we are Birds of a Feather, I bet you'd like the movie "The Blue Max". It's really hard to find on bluray, but worth it! The flying sequences are first rate, and no cgi.
Blackstart assumes *no* power is available, period. Nothing but human muscle power. Thus the first stage is always either a human pulling a starter cord or the like, or a human building up energy in some fashion that is then dumped into the system to produce a bigger surge than is possible by direct muscle power.
And, despite the news reports, this is not a true blackstart. Some power survived.
> have to hand-crank a tiny air compressor just to start a small emergency generator
Similarly, the US Navy maintains banks of pressurized air flasks to air-start emergency diesels. Total Capacity being some multiple of the required single-start capacity
Random fact: Those starters are a plot point in the 1965 film The Flight of the Phoenix, where the protagonists are trying to start a plane that’s stranded in the Sahara, but only have a small supply of starter cartridges left.
I lived for a while on a sailboat equipped with an ancient Saab tractor engine (8 whole horsepower!). Was designed for cartridge starts in cold weather, though someone had fitted an electric starter by the time I saw it
That would be charging up the spring to throw the breaker. High voltage breakers need to switch on (or off) very quickly, to avoid damage from arcing. It's common for them to have some kind of spring or gas piston arrangement that you pump up first to give them enough energy to do that quickly.
No, he's winding up a spring to close the circuit breaker quicker than a human hand could, which reduces/prevents and arc from forming as the electrical contacts close.
If you were the right age when it came out in theaters in '93 (roughly between 11-15), Jurassic Park was a huge deal. Titanic was another of those in that era (although mainly to certain females).
I can appreciate the ability to revert to hand cranking an air compressor, yet I can't help but feel that the 99.99% of events, you'd be better served with keeping a two stroke gas engine ready to go. Air compressors tend to have parts just as or more vulnerable to environmental factors, and you get a lot more power for less elbow grease out of a two stroke.
In 99.99% of real-world scenarios, the rig would have other options to bootstrap a black start—like fully charged air tanks, backup power from a support vessel, or even emergency battery systems. The hand-cranked air compressor is really a last resort tool. We test it during commissioning to prove it could work, but in most cases, it’s never used again in the rig’s working life. It’s there for the rarest situations—like if a rig was abandoned during a hurricane, drifted off station, and someone somehow ended up back onboard without normal support. It’s a true "everything else failed" kind of backup.
Nice to see that at least in some places people are actually thinking to almost-impossible scenarios and taking them into account. I have the feeling that it's quite unlike most infrastructure development nowadays, unfortunately.
The key is the responsible party's skin in that particular game. A drilling rig is a very large, very expensive, and very lucrative man-made island. The backed-up backups have backups. Not only could it be very far away from support vessels, capable of bringing it online in every situation, every minute not in production is money thrown overboard.
Very true, although I think that economic arguments can apply to most infrastructure. What are the actual costs of a day-long nationwide blackout? I have no actual idea, but I'd not be surprised if they exceed 1 billion {EUR|USD}.
The part you are missing is ‘paid by whom’. Unlikely the power companies or regulator is going to be paying that amount here. It’s all the poor saps who didn’t have sufficient backup capacity.
There will be costs/losses by the various power companies which weren’t generating during all this of course, but also fixing this is by definition outside of their control (the grid operators are the ones responsible).
I’m sure public backlash will cause some changes of course. But the same situation in Texas didn’t result in the meaningful changes one would expect.
That’s because there is no effective regulation of the state’s power industry. Since they’re (mostly) isolated from the national grid, they aren’t required to listen to FERC, who told them repeatedly that they should winterize their power plants. And a state-level, the regulators are all chosen by the Governor, who receives huge contributions from the energy industry, so he’s in no rush to force them to pay for improvements.
The real irony was the following summer during a heatwave, when they also experienced blackouts. Texas energy: not designed for extreme cold, not designed for extreme heat. Genius!
Same thing happened in south Texas last year. Years of deferred maintenance on transmission lines resulted in almost two weeks of power outages from two major storms, that could have largely been avoided. The utility provider is mostly allowed to regulate itself (while donating to the campaigns of the dominant political party), and allowed to keep excess profits/return dividends to shareholders, rather than re-invest in infrastructure. There is very little regulatory structure or checks in place to ensure the grid is being maintained. And there have essentially been no consequences, other than an apology and excuses, with an attempt to raise delivery rates even higher. As a home owner, its on me to bear the additional cost of a backup generator, because I can’t rely on the state to regulate the utility to provide the service I’m forced to pay them for.
Based on how difficult it can be to start my chain-saw, snow-blower, and motorcycle after they've sat without being run for a while, I'd not recommend a gasoline-powered engine to be the only thing on stand-by.
Air compressors in adverse environments don't hold up that well either, without basic maintenance. I've had engines run seasonally for decades. It doesn't take much for them to keep working well, though doing nothing at all is an easy way to clog up the carburator.
Compressor pistons/screws that ingest grit/dirt, or aren’t run often enough to boil the water out of them, also tend to not last long. I used to help run a volunteer workshop with an Atlas Copco screw compressor, and it died in a few years because it wasn’t being run hard enough and the screws rusted (doh!).
It shouldn't be that bad. A little fogging oil when put away and drain all the fuel. Then a little starting fluid on the first couple start attempts. Usually they start fairly quickly if they're in decent shape. And that's just for pull starts. My electric start mower starts right up after even 5 months of not running with stabilizer in the fuel.
As an ex small engine mechanic, I'd advise against using a 2 stroke for something like that. A 4 stroke would be a better bet. Better yet would be a natural gas/propane 4 stroke, since gasoline goes stale and plugs carburetors.
Small diesels could be an option but they're harder to pull start for a given size.
> Small diesels could be an option but they're harder to pull start for a given size.
I once needed to jump-start a small marine diesel, many miles from land...
There was a small lever that cuts compression. You have to get it spinning really fast before restoring compression! It's definitely a lot of work!
EDIT - Here is a cheap modern small marine diesel [1]. The operation manual suggests that you don't have to do anything to get it spinning quickly, you just have to crank it 10 times, put away the crank handle, and then flip the compression switch. That's progress!
Lister diesel generators are much the same - half a dozen cranks, restore compression and off they go. The hand cranking can easily break your arm if you get it wrong though.
Even gas engine pull starts have a compression release function built in. That's why you need special cylinder pressure tools to check compression on most pull starts.
I did that too and crank got stuck on flywheel. To stop engine I had to climb over the engine where now-removed stairs were since my mate was clueless. Fortunately the crank handle stayed on.
Cranks and decompression levers are gone for at least 30-40 years now tho.
Not being at all qualified to comment (though I work for a power company), I'd think the hand crank air compressor wouldn't suffer from no spark or bad gas.
If stale gas is a concern, then all of the other steps in-between zero power and your full start are also screwed.
Air compressors have more valves and gaskets that are vulnerable to oxidation, especially in salty environments, so I'd have thought the upkeep between the two, the two stroke would be easier.
But it's an emergency system, not a general operation system. Thus it's not going to be exposed to the salty environment most of the time. You could certainly put the whole thing in an airtight box.
Look at how the military builds surface-based missiles these days: it's in a factory-sealed box. Molten salt batteries so they last for decades. (You don't see molten salt in most purposes because once it's been triggered it's lifespan is in minutes or even less. They're used in applications that only need to deliver power once.)
Diesel will run on mostly anything if it’s running… including methane in the air intake, so you need to think quickly when presented with a generator that keeps running after cutting the fuel
Oil leaking around a turbocharger rotor seal also makes for good diesel fuel, if you define "good" as an exciting uncontrolled disassembly of the engine.
Crude oil from various wells has properties varying from ‘thick, stinky, corrosive goo’ to ‘explosive, barely liquid, bubbly mess’. Also, rigs need to be careful about ignition sources, as methane leaks can be a common emergency condition for some wells/crude.
It’s not the type of thing that using directly is economically feasible, even for emergency situations.
Batteries are great when they have charge. What happens if the generator doesn't want to start the first, second, and third time? How many start attempts do you get before the batteries are dead?
The hand-pumped air compressor is the tool of last resort. You can try an engine start if there's someone there who's able to pump it. You don't have to worry about how much charge is left in your batteries or whether or not the gasoline for the 2-stroke pump engine has gone stale. It's the tool that you use as an alternative to "well, the batteries are dead too, guess we're not going to start the engine tonight... let's call the helicopters and abandon ship"
The data center where I work has large diesel generators for power cuts. They are electric (battery) start. There is no capability to start them manually. The batteries are on maintenance chargers that keep them in good condition. The generators are started and tested every two weeks.
Could the batteries be dead and the generators not start? I guess but it's very unlikely. I get that on an oil rig it might be a matter of life and death and you need some kind of manual way to bootstrap but there's not much that's more reliable than a 12V lead-acid battery and a diesel engine in good condition.
Also, the data center is probably in a city, surrounded by infrastructure that could be used if necessary. An oil rig is in the middle of an ocean, and has to rely on itself.
Lead acid batteries are not exactly what I would call reliable. They require a lot of constant maintenance to ensure that they will work when you need them and they can easily degrade in such a way that they maintain voltage and appear to be good but then fail to deliver the needed amps when you demand them. This is made much worse in cold weather. Finally, if allowed to freeze when they are moderately drained, then the accumulated water inside will freeze and drastically shorten their life span.
I think I'd take Lithium Ion batteries over lead acid for almost every conceivable use-case. They are superior in almost every way. Lighter, less likely to leak acid everywhere, better long term storage (due to a low self-discharge) and better cold weather discharge performance. The only drawback would be a slightly increased risk of fire with Lithium.
I worked with a telecom provider's data center that ended up having a quad redundant diesel generator failure during the first cold snap that took the Texas grid offline a few years back. They had at three fuel supplies gel and then failed to start. The fourth, as I remember, just didn't try to fire.
It's unrealistic, and if one power station is unable to use their batteries to start their emergency generator (through the absurd incompetence you describe, or more likely through a major fire, flood or assault) the grid can be started from a different one.
Black out on a rig or ship is very different to black start of a national electricity grid.
Most vessels will experience a blackout periodically and the emergency generator start fine, normally on electric or stored air start, and then the main generators will come up fine. It's really not delicate, complex or tricky - some vessels have black outs happen very often, and those that don't will test it periodically. There will also be a procedure to do it manually should automation fail.
There are air starters on some emergency generators that need handling pumping. These will also get tested periodically.
The most complex situation during black out restoration would be manual synchronisation of generators but this is nothing compared to a black start.
The point isn't to make a system that is easy. The point is to make a system that is guaranteed to work in any remotely realistic circumstance.
In a real black start, the guys might very well grab a portable generator and just use that instead. But having the option to hand crank something rather than rely on batteries that might run flat is good.
and if the entire thing depends on it, you'll give that generator a handcrank as a backup too instead of assuming the batteries ever dying or getting flooded or whatever is entirely impossible.
bringing islands together requires one to synchronize both -- frequency and phase. It is super difficult for large generators and transmission lines. transient heat dissipation can be a real bummer.
How hard is getting each island within .1Hz of correct? The full grid doesn't have much trouble, but I don't know how much cutting things down impacts that.
And then phase will align itself a couple times a minute so what's difficult about that part?
The compressor pressurizes an air tank. When the pressure in the tank is nice and high, use the compressed air to turn a turbine connected to the crankshaft of the engine.
You can also directly feed the compressed air into a cylinder (or even the intake manifold!) to force the engine to turn. No extra turbine required, though the plumbing might get a little odd. [https://en.m.wikipedia.org/wiki/Air-start_system]
That tends to be for very large engines, where the extra plumbing isn’t a problem.
This technology of starting a diesel engine using a turbine driven by compressed air was used in Russian T-34 tank during the WWII. While Germans could not start the tanks in the cold of winter 1941 from the frozen batteries the Russians were using compressed air (hand-crank) to start T-34s just fine.
As an engineer who has spent time commissioning electrical systems on oil rigs, I've had the opportunity to witness some truly fascinating tests. However, out of all the tests I've seen, the black start test is by far the most intriguing.
Similar to the article, but at a smaller scale, it takes power to start up one of the ~7MW main generators on an oil rig. During commissioning, a black start test simulates a scenario where all power sources on the rig are exhausted, and the emergency backup systems must be used to bring the rig back to life.
Starting the main generators on an oil rig is no easy feat. It takes a significant amount of power to get these massive machines running, and it's not as simple as pulling a cord like you would with a lawnmower. In normal circumstances, the rig uses tanks of compressed air to start the generators, but during a black start test, these tanks are assumed to be empty.
So, how do you start the main generators in this situation? The answer is with a special emergency hand-cranked air compressor. By cranking (and cranking, and cranking) this compressor, you can generate enough air to start a small air compressor, which in turn is used to pump up the air tank and start the 1.5MW emergency generator. Once the emergency generator is running, it can be used to power the compressors that fill the large tanks needed to start the main generators.
Watching this process unfold is truly a unique experience. To see a massive oil rig slowly come to life, all thanks to one person cranking away at a small air compressor, is truly impressive.
Very much reminds me of the stories of my dad how they'd have to start really big bulldozers back in the day of draining german marshes. You'd first hand-crank a small generator to fill up a small battery for the electric starter motor. Once the battery is sufficiently full, you'd engage the starter motor, which in turn spins up a flywheel at higher torque than the crank could. Once that flywheel was up to a certain amount of speed and momentum, you'd engage that with the main crankshaft of the big old engine and hope it had enough power to push the big engine through one or two strokes. That's what it needed to start running on its own. Otherwise you'd have to start over.
Big machines and complex startup machines are surprisingly fascinating.
I don't get why you need to hand pump a bunch of air just to start a small generator/air compressor when I can pull a rip cord and have 10kw of power in seconds and use that to bootstrap the remainder of this startup process.
It would seem you only need to do a bunch of pumping to startup a larger engine where the smaller starter/pony motor is not working. I wouldn't describe such an engine as 'small' though.
Unrelated but fun fact: some heavy duty air-brake trucks can also use compressed air to start the engine, but what if your tank is flat? You can hook up one of the tires and use that to start the engine! Really neat.
Also, many trucks have both an electric starter and an air starter. I one day aspire to those kind of redundancies in a vehicle I own.
So if I'm understanding the description above correctly, the chain is:
turn crank -> bootstrapping air compressor -> air tank 1 -> 1.5MW emergency generator -> bigger air compressor -> air tank 2 -> 7MW main generators
Pretty rube goldbergian.
Here's an interesting parallel: the process that a computer goes through when it booted is itself an incredibly complex, intricate, and rube goldbergian process. Unlike a power station black start, this process happens completely automatically every day when someone turns a computer on, yet it is no less rube goldbergian and in many ways probably more so.
The boot processes of computers have always fascinated me for this reason, and I suspect black start processes fascinate me for much the same reason. It's not unlike why people enjoy watching videos of rube goldbergian contraptions.
It is similar how planes starts. First small turbine engine (APU) starts. When APU fails to start, they use specialized airport cars with compressed air. Then the turbine helps to start first main engine. Then the main engine helps to start second main engine. Repeat until all main engines are started.
(https://www.youtube.com/watch?v=GzhdxSsoT0g)
This seems like great advice, and it resonates with me. I've seen people make a big impact by following this path.
> It’s ok to spend some of your time on snacks to keep yourself motivated between bigger accomplishments, but you have to keep yourself honest about how much time you’re spending on high-impact work versus low-impact work. In senior roles, you’re more likely to self-determine your work and if you’re not deliberately tracking your work, it’s easy to catch yourself doing little to no high-impact work.
In my own personal experience, that boost of actually accomplishing something right now instead of slowly starting the process of impactfully pushing another rock up a hill is very tempting.
Does anyone have any experience or recommendations for effectively tracking your own work and putting yourself in the right headspace to tackle these more long lived impactful tasks? This mental game seems to me to be one of the huge factors that determine outcomes.
I tend to have some challenges with attention at the best of times. My interests tend to run hot and cold. I can make a huge impact and move a project significantly forward when I get into it and hyperfocus on it. But other times managing to focus my attention on a tasks that I know would be high impact is mental torture.
A couple strategies that I've employed, that have helped me:
1. Do a start-of-week plan, and end-of-week review. Pick a few milestones that are achievable this week. Hold yourself accountable and check in with yourself to see if you completed them. If you didn't, review why not. Did you snack too much? Did you get pulled into lower-priority meetings? Did you work on some other urgent stuff that is actually OK to drop your tasks for? Keep any insights at the top of your "weekly plans" doc so you can remind yourself of them and try to avoid making the same errors repeatedly. Breaking your long-term goals into milestones also gives you some "snack-like" satisfaction before you get to the finish line and earn the big payoff.
2. Every day, pick a task that you're going to do "hell or high water". Try to get that done before you snack. Typically this is (a piece of) one of your weekly tasks. If your calendar is prone to getting filled up with meetings, block off some "maker time" on your calendar to get this task done. I find it helpful to preemptively schedule timeslots for my project work at the beginning of the week even when my calendar is likely to remain open; it keeps me honest.
3. Timebox your snacking. If you feel like you need a break from longer-range tasks, you want to get an energy boost, etc., set a timebox, say "1 hours refactoring these tests", and try to return to your hell-or-high-water task after that timebox. I find it easy to go down the rabbit hole when I start snacking, especially if I get deep into the flow state. Flow is good! But it can lead you astray from your longer-term goals if you're flowing on something that's not your #1 priority.
As for the mechanics of tracking your work, I have used a personal Trello, todo.txt doc, Roam, GDoc, pen & paper -- this is immensely personal but just having a single place where you can go back and remind yourself what you were supposed to be working on is really helpful.
This is great advice. I do something similar and it’s been working. The only thing I dont fully agree with is the tracking of work. I’ve found that tracking work has very little value to me, so I just don’t spend time on it.
The only time when tracking work has been really useful was when the Icelandic banks went bust during the crash of 2008. Having a log of work-done helped a little in trying to get paid after my client went down.
But, in other cases, I’ve found that the result of the work is the log of work done.
I didn't use to track my work and I noticed that at yearly review time it's been pretty difficult to remind what I actually did for last year. I'd go through merged PRs in the main projects, but it's only a tip of the iceberg.
Some time ago we moved daily standup to Slack, and since then I have a pretty good history of what I did each day. I created a wiki page where every 1-3 months I try to summarize all the stuff I did in the quarter (code shipped, dashboards created, docs written & non code work done, like helping X ship Y -- with screenshots, links etc.) and then it gives me a bit of confidence boost during the perf review time.
Cannot upvote this enough. These are excellent ideas for managing work, constantly making forwards progress, and showing to yourself, and the world, that you are having an impact.
I would add that always double check on regular basis that at least one of the things you are working on is a priority for your manager. Use your 1:1 meetings to confirm this and make sure you are aligned, and where you arent, that there is an over-riding company-acceptable reason. The bigger the company, the more this matters.
I find the Pomodoro timer technique (25-minute timeboxes) helpful maintaining short-term focus (and for breaking my focus, giving me a chance to review whether I am snacking or need to move to a different task for my next 25-minute timebox).
Checklists are useful for breaking a task down and providing incremental steps to progress through.
During my PhD I would write down what specific task I was working on in a work journal every half hour or so and it would help me refine the specific problem/question I was working on. At the end of the day I would have made a steady progression through a relatively abstract problem which might not have been apparent at the beginning.
I take the same approach with working on the various projects now as a software engineer. I find the act of documenting my progress/thoughts as I go extremely calming. There’s a balance to be struck here - I only take this approach when developing something new and I have to record all the things which didn’t work.
Definitely agree on snacking being very seductive - it makes you feel useful and doing stuff, but when you zoom out, it's usually very inconsequential
The mental game is very important. Definitely hard to work on the things that actually matter - they are usually difficult, new, more intricate to setup, etc. What I've found to work is to "just start on it" - starting is the hardest part. Telling yourself you'll do 15 minutes of it or something, so you will actually start. Usually, when the 15 minutes are up, I won't be stopping. The inertia goes from "not going it" to "doing it" and it's hard to change ha
Trick that often works for me: create a ticket in JIRA, explain well what has to be done, how, document gotchas etc.; assign to yourself; if it's a code ticket, create a local branch with ticket number.
I usually procrastinate when it's not clear what exactly has to be done and how, and writing it down somewhere helps immensely.
In my personal life, I adopted a "broad front" approach to long term projects.
By broad front, I mean that I'm inching a lot of things forward, instead of focusing all of my attention on a single breakthrough.
If my motivation is low, I'll do small things that will be helpful when my motivation returns. This includes planning, buying supplies, cleaning up the workspace (physical or digital) or implementing quality of life improvements. For example, I might not be ready to extract a bolt that broke in a motorcycle engine ([expletive]), but I can disassemble the engine, buy tools, or figure out how it's done.
Sometimes, this gets me right back into it. Sometimes it just makes it easier for another day. To keep with the analogy, I call it stabilizing the front.
If you do this, it's critical that you leave yourself an easy reintroduction to the project. This might be a 95% finished commit, or a really good readme. You shouldn't dread getting back into it because of the project's state.
I've found focusing on the hardest problems first when enthusiasm is high nets the best results as the difficulty over time of remaining tasks coincides with waning enthusiasm and fatigue. Focusing on small tasks first and delaying large problems has the opposite effect - unless there's not enough context to complete the larger tasks without completing smaller parts first.
I was once working for a small company building electrical equipment. We mostly worked on "medium voltage" equipment, you know 2400 to 69000 VAC.
For one project we had large banks of ultracapacitor in a cabinet. Fully charged it was around 1200 VDC. This thing was in the prototyping stage, and we were testing a control system on a Saturday morning.
So we charge it using a large AC/DC converter, fully charged, everything worked beautifully. We start a discharge cycle converting the DC back to AC. Uh oh, it starts pulling way too much current. Flames start to shoot out of the AC/DC converter. Fuck. BANG. Fuse blown.
We assess the damage... the AC/DC unit is totally shot. And someone (me) is going to have to analyze what caused the failure. Otherwise everything with the capacitor cabinet seems okay, but the thing is still charged to 1090 VDC and the fuse is blown. Check with the mechanical engineer that designed the cabinet. Turns out the fuse can't be changed (can't be accessed) while the cabinet is charged and the cabinet can't be discharged because the fuse is blown. Well that isn't good.
The only thing we could do was discharge it into a load bank (think large toaster) by connecting something directly to the copper busbar live at 1090 VDC. So one of the commissioning guys volunteered. He put on some high voltage gloves, stood on a plastic mat, and connected some jumper cables someone had in their car to the bus bar. He stepped back and someone else threw the switch on the load bank and it discharged without incident.
You would think if you guys were working on those AC voltages, you'd have an arc flash suit on hand and he would have also put on an arc flash suit to do that.
Perhaps the solution is more reporting instead of less reporting. Some sort of real time (or hourly, daily, whatever) metrics about a company instead of quarterly reports. I feel like a certain frequency makes it harder to game, and much more routine, so people aren't as likely to make decisions that are detrimental to the long term. I think faster reporting and quicker feedback loops are the way the world is going, why not for public markets?
The problem with reporting is it's actually a liability. You're legally responsible for reporting accurately to shareholders so there's a non-trivial amount of work to ensure you get it right. You could do that in real-time but there would be a significant overhead to it. There are also other problems, for example the sales team will always make sure their paperwork is completed in time for the end of quarter to hit their targets, if you're reporting continuously you're going to see a lot more noise in the sales reporting figures that gets averaged out by only reporting the quarter. You aren't gaining information by reporting more often, you're just exposing your investors to a noisier signal.
But wouldn't that be more productive for the business to have people always producing value, rather than rushing things in for a deadline and slacking off immediately afterwards?
A quote from a letter from Chairman Khrushchev to President Kennedy during the Cuban missile crisis.
I see, Mr. President, that you too are not devoid of a sense of anxiety for the fate of the world understanding, and of what war entails. What would a war give you? You are threatening us with war. But you well know that the very least which you would receive in reply would be that you would experience the same consequences as those which you sent us. And that must be clear to us, people invested with authority, trust, and responsibility. We must not succumb to intoxication and petty passions, regardless of whether elections are impending in this or that country, or not impending. These are all transient things, but if indeed war should break out, then it would not be in our power to stop it, for such is the logic of war. I have participated in two wars and know that war ends when it has rolled through cities and villages, everywhere sowing death and destruction.
I worry that the current nationalist trend that seems to be simultaneously happening in many countries is going to lead down the long path to war. I don't see many leaders today that I think would have the diplomatic resolve to write a letter like this. I feel people are interpreting others actions in the least charitable way which causes all kinds of rifts. The deeper these rifts grow, the more likely we are to see someone ignite the spark of war. I honestly don't know the reasons, or if maybe its just my perception that has changed as I've gotten older.
Both Krushchev and Kennedy had gone through World War II. They had seen what war is. It wasn't just stuff in a history book to them. That's why Krushchev could write such a letter, and why Kennedy could understand it as more than just an academic statement.
Yes but the Soviets saw by far the worst of that war among the allies and Kruschev was in Stalingrad during the siege. I don't discount the possibility that Kennedy could have seen some horrors in the Pacific but Kruschev saw them visited on the civilians in his homeland. Its a whole different frame of reference.
Not to minimize the destruction that Soviet Union (and much of Eastern Europe) had experienced, but Kennedy was not that much of a stranger of the horrors of war. In addition to whatever he saw on the Pacific front (which, if you'll recall, might have included some of the most horrendous atrocities of the war), his elder brother was killed during the war, too -- just like Khruschev lost his son during the war. IIRC, it's a fact that Khruschev himself reminded him of, during their private talks in Viena: they both knew, on a personal level, the kind of toll that war levies.
The "burden" (Kennedy's words, not mine) of avoiding similar destruction was something that he carried every step of the way during the Berlin crisis and then, later, during the missile crisis. Both Kennedy and Khruschev, and others in their immediate circle, had this "more than academic" understanding of the tragedy of war.
(edit: FWIW, my native country was no stranger to destruction in Europe. My grand-grandfather lost a brother and several friends at Stalingrad. I'm not some patriotic 'murican trying to save face for his president :-). I do think that losing a brother qualifies as having a pretty practical understanding of what war means.)
The irony is that our nation (I am an American) has been at war for over a decade, but much of our society has not. I am a staunch advocate for peaceful conflict resolution, informed in part by my experiences in the war in Iraq. If a larger part of our society had been actively involved in the wars in Iraq and Afghanistan (serving in the military, having a loved one deploy for months/years, paying higher taxes to support the war, etc), I wonder if our nation would be more wary of a force-first approach to foreign policy.
If there are any of the moderators reading this, I would love to know what this post did wrong to get shuffled to the very bottom of the conversation so I don't do it again. There must be some mod magic involved as it has a number of up-votes.
In a lot of industrial sites software security is a joke. Embedded systems tend to use very old, well proven technology, which in itself isn't a problem, it fits the market well, but the side effect is that security isn't always properly considered as it wasn't a concern when the software/hardware was developed.
I was involved in a project a few years ago delivering a series of monitoring systems running Windows XP to a brand new 700 million dollar oil rig. This was at the request of the client, they had software they needed that would only run on Windows XP. They had a fit when we had trouble sourcing Windows XP licenses. The expectation is that these systems will have a 20 - 30 year life.
It used to be that keeping every air gapped was enough, but organizations want easier monitoring, so more systems are being networked in an ad-hoc way without a lot of thought about security.
I expect we are going to see more things like this happening in the future until we start taking security in systems / embedded space more seriously. And even then there will be exploits of older systems for years afterwords since the replacement cycle is so long.
I wonder what a secure embedded system even looks like when I think about it. The environment isn't suitable to the kind of continuous patching that is done in the web world, but exploits will be found and dependencies will need to be updated. How do you square keeping things up to date with stringent testing requirements in systems that can kill people. Many of these systems / plants are unique, there is only one plant like it in the world, so testing becomes very hard.
From what I've heard, one of the problems here is that Windows 10's support of parallel ports (widely used in automation) is really bad, and there's no way that live updates are going to be acceptable on something that needs to be running 24/7. What ways are there to properly air-gap something like this?
And of course once nobody's using Windows, attackers will pivot to target whatever everyone's switched to in its place. And they'll find holes. They always do.
We're talking about the difference between barely usable and unusable here. No system in the world has ever been slimefest of the scale of Windows. Even those which are deployed more than Windows.
It is much to easy to blame Windows for everything. If this is another TRITON level attack then it did not matter what operating system is used, the attackers would have come in anyway. With enough resources you can break anything.
Check out the DEFCON 2018 talk "Through the Eyes of the Attacker" to see to what lengths the TRITON attackers went. These guys were dumping eeproms from boards running obscure MIPS processors and were looking at raw ethernet packets in Wireshark and flipping bits in the packets to see what would happen. That their command&control was running on Windows was just coincidence.
I'll never understand selecting windows for any embedded system that is expected to have a 10+ year life. If you look at their history, they frequently blow up the development environment every 5-6(?) years. Yes they'll provide support, but it comes at being increasingly held hostage to whatever level of long term support costs are in vogue at the time. For a 20-30 year support life, that's a guaranteed headache.
You don't install any updates or change anything once the machine is setup and performing its duty. If the hardware fails you replace it with similar hardware and install the original software from CD again. I see windows NT on ships, windows 2000 in power stations, etc.
Improper maintenance of worn out components can cause a failure resulting in a crash, be they computer components, electrical components, or mechanical components.
Linux doesn't necessarily help here. Ubuntu LTS lifetime is 3-5 years, others are similar; outside this, distro maintainers think nothing of just dropping packages from the distro that the deployment may be relying on, so upgrading is not simple.
The difference is that with Linux you can self-support, or pay for someone else to do it, since the whole source code is available to everyone and the licence allows it.
Also, a better comparison would be with RHEL, which has a lifetime of 10 years or more.
How are mechanical components tested against requirements in critical systems? What is the process for changing those components, like upgrading pipes to a fancy new composite?
In my mind, we'll start treating silicon the same way... formal verification, rigorous real-world testing, trusted suppliers, and an expectation that change is slow, expensive, and risky.
Seems like there is a huge opportunity here for a startup that can navigate the industry and manage to solve some of these problems. There are huge players in the space, but from what I've seen they aren't solving these problems very effectively.
I've friends who work as instrument technicians/engineers on embedded systems in manufacturing and have worked in Europe, UK and Australia in food manufacturing, mechanical, water collection and water processing.
The one constant I've heard is that all of their hardware is almost or is out of support, when it breaks, they expect band aid fixes and ironically none or very little of them can accept any downtime. There's no hardware redundancy for their production lines and when anything breaks, it's all hands on deck. Yet there's no funding going back into the production lines to pro-actively repair or minimise their risks. Manufacturing it seems to me is 100% a reactive industry.
The industries above work on incredibly small margins of profit and the sheer expense to outfit and refit these aging, decrepit (but still working!) production lines are quite honestly, massive.
These manufacturers won't invest in these engineering faults (whether it's security or production focused) until they've been fucked.
NB: This might be with the exception of Lego, my cousin who got employed by Lego after finishing his masters in Industrial Design & CS, and after reading and watching some articles on Lego. I'm convinced Lego's margins are a lot larger than most. They might be the closest thing to a FAANG company when it comes to investment in phu7sical engineering and manufacturing.
How are these places insured against industrial accidents?
I wonder if the way to get your foot in the door would be to partner with an insurer who would provide discounts to the client if they installed security technology meeting a certain standard. You could come up with some kind of bump on the wire type Linux device that proxied access to the old insecure system bringing them up to the standard. Then sell it as a return on investment through savings on industrial accident insurance premiums.
You're right, but my experience suggests that anything broadly in the 'manufacturing' field is treated as a cost center, and not something people want to spend money on.
In most places a rubber stamp of 'yep, definitely secure because we have a password (over http, oh by the way everyone uses the default which is in the manual)' is good enough. Then when something goes wrong there's a general shrug and they change the password.
The attitude of those in the oil industry does not help this at all. I work in oil and talking politics to those I work with just makes me sad. There is a lot of anti-government hate. People are still resentful about the National Energy Program and that ended 30+ years ago. Its kind of a mess.
I think its the notion that all the provinces are competing against each other, and that under any oil program some provinces will win and other lose that really holds us back. No one can see that we can all win by collectively working together. It doesn't help at all that there are now moneyed corporate interested involved that are not interested in seeing any changes take place.
I assume he's referring to platform lock-in ala Microsoft Office, AWS, Oracle, or Apple's walled garden driving future decisions. So the first software decision is a down payment until your technical debt hole is so big you can't do anything but throw money down it.
This sounds A LOT like myself. I've never really considered going to a psychiatrist or neurologist, I thought my experience was somewhat normal. Do you mind elaborating some about what sort of medication/condition are you talking about?
I've been editing to fill in some more details. I'd rather not discuss my experiences with medications specifically here, but feel free to shoot me an e-mail (see profile) and I'll be happy to answer any questions I can.
If you want a possibly contrasting experience, feel free to email me (see profile), I was diagnosed with ADHD-C within the past year after many years of "this is just how I work, there's nothing that can be done".
A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.