
The Threat to Americas Electrical Grid Is Bigger Than Imagined - weatherlight
http://foreignpolicy.com/2016/07/31/the-threat-to-americas-electrical-grid-is-much-bigger-than-you-can-possibly-imagine-cyberwar-squirrels-rodents-hackers/
======
doug1001
Ted Koppel (retired new anchor) began research for a book on this subject
after retiring; the book, titled "Lights Out" was published late last year, i
believe. Two things about this book were remarkable to me--one was the
exhaustive research; the second was a remark from Koppel in the book (i don't
recall the exact words he used nor where in the book it is) that it is
difficult to imagine a better design for a Power Grid whose primary criterion
was maximum exposure to catastrophic attack than the US Power Grid (used
collectively to refer to three geographic grids).

~~~
pp19dd
The critical piece of evidence presented in his book was the blackout of 2003.
Ted Koppel's version was short and implied that the power grid system is so
vulnerable that all it took was some overgrown trees to bring the northeast
down to its knees for days. He then spent rest of the book talking about SCADA
glitches and computing vulnerabilities and disaster preparations (focusing
namely on the Mormon community, which is always prepared for a disaster.)

Problem I had with the book is that he either misunderstood or misrepresented
that triggering event. Yes, there were overgrown trees, high summer demand and
temperatures that made power lines heat up and stretch. Wind speed dropped
from 5 to 2 knots and cooled the line less than it normally did in that place.
Cables stretched more than usual and grounded. These are all true facts.

However, that's not what brought the power grid down that day. Here's what
did.

At 12:40 pm, an engineer working for MISO (transmission company, sort of like
a traffic regulator) switched a regulation system to manual mode to make a
correction. Note that the correction was made because the system had an errant
piece of information regularly enough, which through a phone call the operator
knew was false and corrected.

The regulation system was called a "state estimator." It basically takes tons
of factors, loads, schedules, downtimes, capacities, etc and optimizes power
transmission. Like intelligent traffic control. Of course, all systems have
input garbage from time to time, and he corrected this thing.

At 1:30 PM that engineer went to lunch, forgetting to re-enable the automated
mode.

At 2:02 PM, a 345k-volt line tripped after touching a tree and started this
cascading reaction resulting in a system blackout for a week. Other operators
looking at the system readouts didn't know that the state estimator was now
sending a false picture, being frozen in manual mode.

By the time the unnamed engineer (from the NERC report) returned, it was too
late. Other regulators, operators and power plants had been making uninformed
decisions, preparations and adjustments for the power grid based on outdated,
non-realtime information. However, the blackout didn't happen instantly -- it
took four hours for a cascading chain reaction to trip every safety device in
the system.

Lets not even go into SCADA vulnerabilities. Here's some positive outcomes
from the blackout: investigators made a staggering list of 46 recommendations
-- none addressing tree trimming because vegetation events happened all the
time. Those were routine. They did focus on computer security, even though
there were no computer faults in this incident.

Secondly, during the blackout the Department of Energy ordered state of
Connecticut to energize one of their power plants despite their objections.
They complied, and that restored power to metropolitan areas. This action is
now a precedent for any future outage events: the legal framework for
prioritizing power restoration exists and is in place.

SCADA systems show up on HN often enough to be a point of ridicule. Good news
is that the outdated SCADA systems are getting replaced with synchrophasors
for the most critical power integration functions. Since 2003, nearly 1700
phasors were added to the national grid, and they're well on their way to be
fitted in the roughly 7,000 power plants in the U.S. What it will do over
SCADA is improve recovery times of an unstable power grid segment.

Edit: derp, northeast, not nw. Also
[http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/...](http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/BlackoutFinal-
Web.pdf)

~~~
doug1001
needless to say, i lack the domain knowledge to critically evaluate the
conclusions in Koppel's book, so always welcome to hear such evaluation from
someone who does.

------
munin
Everything that has an off switch has an on switch. Attacks against the power
grid are weak because, in almost every scenario you can conjure, the
remediation is just disconnecting the network and flipping the on switch.

The things to look for are asymmetries. What is something that is very easy to
do but very hard to un-do? What is a very important thing that would impact
lots of people that is very easy to do or set in motion, but much harder to
un-do or stop?

Look for those and you'll find actual problems. There are far fewer of them
and they are difficult (requiring creativity and domain expertise) to find,
which is why no one talks about them...

~~~
bagrow
Restarting a down grid is nontrivial. Small power stations are used to
kickstart the larger stations. Think of the starter motor in a car. Now do all
of this while precisely maintaining 60 Hz (in the US) everywhere.

~~~
ridgeguy
This is called a "black start". [1] Definitely nontrivial.

[1]
[https://en.wikipedia.org/wiki/Black_start](https://en.wikipedia.org/wiki/Black_start)

------
danielvf
The US has actually has three independent power grids: East, West, and
unsurprisingly, Texas.

~~~
vibrio
Some would say the US has two grids and Texas had one.

------
Animats
With all the money we spend on "homeland security", there really should be a
reserve supply of large transformers, just in case.

~~~
shorttime
A lot of power plants do, at least the ones I've worked at. I wouldn't be
surprised if these substations don't. Inspection teams come in all the time
and practically force companies to do things like obtaining a spare.

------
msisk6
I work for a grid operator here in the US.

There's certainly a lot of FUD in that article, but it gets one thing right:
critters in the equipment are a major source of failure.

Last year we had a power glitch at our office when a snake climbed up a power
pole and shorted across two phases. The snake didn't survive and folks learned
to make sure their desktops are plugged into the outlets served by the
building UPS.

------
vibrio
Today I learned that my dog is doing her part to improve National Security.

------
nxzero
Key lesson is that if you're the Russians, train squirrels to attack the power
grid, deploy them, and then deny, deny, deny.

~~~
ilaksh
In all seriousness, most of the squirrels are already working for the
Russians.

