
Senate passes bill to decrease grid digitization, move toward manual control - howard941
https://www.utilitydive.com/news/senate-passes-cybersecurity-bill-to-decrease-grid-digitization-move-toward/557959/
======
kevan
There's definitely some advantages to using age-old practices. It's easy to
explain how trust and verification works in a system when 2 people have
separate physical keys and you need both to perform an action. It's a lot
harder to explain things like modern crypto to most people. Until a critical
mass of the decision-making chain understand the tech we may be better off not
using it for mission-critical things.

Alternatively, this could also be a misguided effort that holds back smart-
grid tech in America by a decade. It's really hard to tell.

~~~
hinkley
Nearly this exact scenario has been bugging me for a couple months. I started
thinking about it due to deployment issues.

Why _don 't_ we have a proverbial two key system for certain activities like
deploying to production or deleting user accounts, or backups?

We make do with things like pull requests, but everywhere we've ever had
retros, I've run into situations where someone did something very stupid,
somebody else rubber stamped it, and now we have a mess. Approvals on an
action do not approximate the ritual sobriety that is embodied in two
independent humans deciding to turn a key.

There are some people for whom these two actions can be considered equivalent,
but we tend to bag on them for their meticulousness.

I think I'm wanting a tool where I type "deploy 1.0.12345 to production" and
it sits there waiting until someone else types "deploy 1.0.12354 to
production", and then it stops us because one of us transposed some digits.

~~~
eximius
Someone did release a Pam module? That required confirmation before someone
sudo-ed. I thought it was Stripe but I can't find it right now.

~~~
stouset
Hey, that was me! See the sister comment.

It's actually a plugin for sudo (surprisingly yes, sudo has plugin
capabilities[1]) and not PAM. I had originally developed it as a PAM module,
but the sudo plugin API allows for the neat trick where the TTY is mirrored.

[1]
[https://linux.die.net/man/8/sudo_plugin](https://linux.die.net/man/8/sudo_plugin)

------
rossdavidh
As a person with two electrical engineering degrees and 15 years' experience
as a professional programmer, I am nearly speechless at such a rational,
foresighted, and prudent action coming from our government, in regards
technology. It's like some bizarro-world where we have leaders who use the
power of government to encourage industry to be careful and secure. I will
wake up soon, and realize this was all a dream.

~~~
ip26
I suppose a captured regulatory body is right twice a day

~~~
craftinator
This put a genuine smile on my face =p

------
tootie
"bill that will study ways to replace automated systems with low-tech
redundancies to protect the country's electric grid from hackers."

The headline seems to be inferring more than the content is saying. The intent
is not to rollback digitized systems, it's to decrease dependency on
vulnerable systems. And the bill is only to commission a study on what systems
could benefit from manual fallback, nothing is slated to be implemented yet.

~~~
cududa
Thank you for the only substantive comment here!

------
userbinator
Good to see some more "common-sense" in contrast to the "Internet everything"
trend that's taken hold in some areas of the industry. Some of the control
systems in the national grid have been in use for over a century, and are
likely to continue working indefinitely if kept maintained.

------
avip
We need smart-gridness to reduce our dependency on idle fossil-burning power
plants as a safety net to balance the now much less predictable grid. Strictly
ecologically speaking, this seems problematic.

~~~
probablypower
This is an important point.

Increasing renewable penetration into a power system leads to:

-> reduced inertia

-> increasing Rate of Change of Frequency (RoCoF)

-> reduced time to react

If you rely entirely on manual systems to react to frequency changes on a grid
with high renewable energy penetration (e.g. the Irish grid), you'd really
struggle to avoid daily blackouts without decommissioning a bunch of wind
farms.

It is by using such automated control schemes that countries like Ireland are
able to push the limit on wind power penetration.

If I was being truly cynical, I would think that a push towards mandating
manual control (rather than mandating that it is available as a fallback) is
actually a way to limit growth of the renewable energy sector.

~~~
wongarsu
While a smart grid is incredibly useful to solve this, is it strictly
necessary?

For example you could build an array of flywheels that charges when the
frequency is above 50Hz and discharges when the frequency is below 50Hz.

No network connection necessary, and if you want to you can build it entirely
analog. If all you really want is inertia and reaction time it could even be
as simple as dumb three phase motors with weights.

~~~
blattimwind
> For example you could build an array of flywheels that charges when the
> frequency is above 50Hz and discharges when the frequency is below 50Hz.

All rotary field machines (power plant generators, phase shifters, motors)
work this way and contribute stability to the grid.

These basically add inertia (i.e. dampen frequency changes), but they do not
regulate frequency. That's a fundamental difference.

~~~
wongarsu
A simple motor or generator simply add inertia, but I see no reason why you
can't use a flywheel to deliberatly regulate frequency, for example by adding
a continuously variable transmission.

But really inertia is all you need because all we are trying to do here is
replacing the lost inertia from replacing spinning generators with solar. The
recovered inertia gives human operators the time to make phone calls to
increase energy production, spinning up a pumped-hydro plant or whatever is
available.

~~~
snowwindwaves
Who is going to pay to build and operate this flywheel inertia plant _that
generates no power_? Sure it is possible but if you are going to spend that
kind of money you can do a lot better.

~~~
g_sch
It's been done already! Beacon Power operates a 20 MW / 5 MWh flywheel plant
in Stephenstown, NY that does nothing but store grid energy and release it
when needed. And ancillary services such as spinning reserve carry a price on
the electricity market, so there is definitely an opportunity there.

I'm not sure if frequency regulation is currently considered an ancillary
service in electricity markets right now though - from what I know about NY
state, only 10- and 30-minute reserve are priced.

------
tantalor
The opposite of "digital" is "analog", not "manual".

The old phone networks were analog and vulnerable to cereal box whistles.

~~~
wtdata
Analog can still be automated. The right word here is indeed manual since it
needs human intervention.

------
beenBoutIT
Anything that the public depends on for survival could be a great candidate
for this type of restriction. Think of the hundreds of lives that would have
been saved had this type of law been in place for airlines and their aircraft.
A guy aware of the risks running himself into a tree in a Tesla is one thing,
an unaware set of pilots with hundreds of passengers on a public aircraft is
another.

~~~
incompatible
Some accidents caused by automation failures would have been avoided, but how
many others would have been caused by human failures?

~~~
beenBoutIT
If the automation is actively saving lives by correcting human failures then
they need to log those events silently and forward them to whoever manages the
pilots.

------
e2le
Why not simply stop networking these systems together and/or connecting them
to the internet?

~~~
colechristensen
It doesn't stop attacks.

Stuxnet successfully attacked air gapped systems.

~~~
simonh
It does stop attacks. It just doesn’t stop all attacks. Just because a
security measure, by itself, isn’t perfect doesn’t mean it isn’t worth
implementing.

------
dreamcompiler
I have mixed feelings about this. In some ways it's a good idea but the
demonization of "digital" is misguided. Some of us know how to build secure
digital hardware and software but there are no incentives for companies to
insist upon such designs, and there are many active disincentives for doing
so. The incentive structure needs to be changed to fix this, and then the
technology will follow. Demonizing a technology is short-sighted and
dangerous.

------
threezero
For some reason this reminded me about the still-unsolved San Jose substation
physical attack. The shooters did a lot of damage but didn’t interrupt much
power.

[https://en.wikipedia.org/wiki/Metcalf_sniper_attack](https://en.wikipedia.org/wiki/Metcalf_sniper_attack)

------
inlined
There’s a reason our nuclear system still operates on floppy disks.

~~~
minton
Do you have a source on this?

~~~
lucasmullens
Google it, it seems to be a generally known fact:
[https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-
nucl...](https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-
nuclear/index.html)

------
droithomme
Very good news.

Mandate paper ballots next please.

~~~
Whatarethese
Punch cards. Scantrons. Stone Age shit for elections.

------
CharlesMerriam2
This is an important step in safe guarding our collective experience of power
plants.

Consider that the way we currently interact with power plants: "Someone needs
to stay behind and manually make the boilers explode! I'll do it! Tell Laura I
love her." pales in drama to "OK, let's see, Power / Input / Boiler / Max,
confirm, confirm, confirm. OK, let's run!"

------
Corrado
This reminds me of the 2004 Battlestar Galactica reboot. All of the ships in
the fleet are modern and get hacked and taken over by an outside force. The
"Galactica" is being decommissioned because it's old and it's systems are
linked in with the rest of the fleet. Hence it is able to withstand the attack
and save humanity.

------
Avamander
Instead of fixing the laws and forcing proper auditing there should rather be
regression in scalability, response and monitoring?

------
tlear
This is very good news. Maybe few years late but better now. Current situation
is a disaster waiting to happen

Stuxney penetrated a heavily guarded, monitored, secured with air gap
facility. US power grid systems are child’s play in comparison

------
vinay_ys
It is quite alarming if we need to move away from digital connected tech for
electrical grid monitoring and management. What about banks? What about
hospitals? storage facilities? record keeping? These are the core
underpinnings of a functioning society. We are already using digital-only
systems for these. If these can be disrupted or corrupted, then society will
come falling apart quickly. If we truly cannot secure a digital system for
electrical grid and hence we should move away from digital, then we should be
doing the same for these other critical systems as well. Is this even
practical now? Shouldn't we be going down the path of inventing chips,
systems, algorithms, protocols and proof methods to secure these systems?

~~~
asperous
There's probably an ideal system down the route of more high tech, but in the
real world they would never be implemented perfectly. As complexity increases
so does room for security issues.

I don't see the problem with striking a balance with technology and putting in
low-tech solutions for security reasons. So yeah I think many of those systems
should implement similar measures.

------
jmartrican
US Gov taking steps to prevent the Butlerian Jihad before it even begins.

------
spacemanmatt
We are capable of protecting our assets. We just have an administration that
prefers to sacrifice efficiency to lock in a dying energy industry.

------
buboard
Just hope this doesn't snowball

This is quite an indictment of the security community.

------
Havoc
Good thinking security wise but could be a serious setback for efficiency

~~~
jacquesm
Efficiency does not trump security for such installations.

~~~
Nasrudith
It may be an unpopular opinion but I disagree - security is a cognitively
biased function owing to historical roots but it actually seldom comes up
which fails to justify its costs and there is little incentive to gain from
attacking it.

Even if a bad actor did bring down the grid all it would do to concrete
capabilities is GDP damage that they could easily get credit for and still
leave a pissed off superpower able to make an example. Even actual rival
status like say China were to do it would be shooting themselves in the foot
such that something stupid and self destructive that if the PRC decided to
simply start bombing Beijing it would do less damage.

Besides for decades a few tens or hundreds of people could go out, buy semi
automatic rifles and start casually shooting substations and walking or
driving away until someone arrests them. That would cripple power
infastructure far worse than any hacker could as it would take longer to
replace and be a distribuited issue to fix.

Given actual frequency even with lax security and pessimistic assumptions
efficiency is a likely winner. Logistics and not superweapons win wars.

------
Whatarethese
This seems like a terrible idea. Computer controlled systems lead to
optimization and smarter decisions.

~~~
rubidium
Not always in my experience. It’s balanced with “Out of sight out of mind”.
There’s a benefit to making certain critical decisions manual, as it forces
some review and awareness.

Plus the current track records of internet security for physical control
systems is terrible.

------
liveoneggs
should have used erlang?

------
tomohawk
This is a huge indictment of the IT industry. We need to kick the shiny rock
mentality, and actually engineer systems that are safe and secure.

Just look at how some companies interview and select people. They do so on the
basis of cleverness, not carefulness and attention to detail.

~~~
CobrastanJorji
Even if software were perfect, this makes sense as a defense-oriented
decision. It's harder to hack a physical switch. Air gaps can be bridged.

There are also plenty of reasons to automate things. It allows for for faster
reactions to problems and interesting new ways to reroute power. Both
approaches have pros and cons, and there are good reasons for backing either
approach.

Russia attacked Ukraine in a new way, and we responded by trying to become
more well-defended against that new attack. Much as I agree that coding
interviews are problematic, I fail to see how your point follows.

~~~
throwawayjava
It is possible to write safe and secure software. It's expensive, but it's
possible.

Digitizing the grid has enormous upside, to the tune of billions in savings
and improved resiliency/response to weather related outages. It we could do it
safely and securely it'd be a no-brainer.

We're just trading a devil we know for a (preventable) devil we don't.

BTW: digitized grids aren't even necessarily more vulnerable. In a complex
system, the increased latency and miscommunication opportunities introduced by
human operators are _also_ a potential attack vector...

~~~
rrix2
Digitizing the grid also opens up efficiency gains which we sorely need. Smart
water heaters which run when we have daytime solar surplus, etc.

~~~
gamedori
Why not send a price signal separate from the grid, or use AC frequency
deviation as a price signal?

~~~
cogman10
The neat thing is that already basically exists in the form of voltage.

If you've ever watched your voltage, you'd noticed that it isn't a perfect 110
or 220. It is often higher or lower. When it is higher, there is a local
surplus, when it is lower, there is a high load.

We could do this today. We might not have current pricing, but we do have load
vs production information.

~~~
cesarb
> When it is higher, there is a local surplus, when it is lower, there is a
> high load.

Or perhaps the voltage got too low, and an on-load tap changer in one of the
transformers increased the output voltage. Voltage does not necessarily follow
the load. AFAIK, the thing generators themselves use as the main feedback
signal is not voltage, but frequency; but it's not a useful signal for
consumers, given that generators are much stronger at keeping the frequency at
its nominal value.

~~~
cogman10
Frequency doesn't change with load.

Load causes the voltage to drop (that's what's happening when a "brown out" is
triggered). Some loads cause the current to lead or lag the voltage wave
(Inductive vs Capacitive loads, most are Inductive, particularly with heavy
duty equipment). But that isn't changing the frequency but rather the phase of
the current. This is all tied up with a number referred to the "Power factor"
(see
[https://en.wikipedia.org/wiki/Power_factor](https://en.wikipedia.org/wiki/Power_factor)
). essentially, the farther shifted current is from voltage, the more work is
done by the power plants essentially heating grid wires (rather than doing
something useful)

So, power grids will do 2 things. First, they'll work to keep the current and
voltage phase in sync. They do this by adding extra capacitors/inductors.

Second, they work to maintain the voltage of their tie in to to the grid.

Generally speaking, the type of power plant matters as well. Base load plants
will simply dump onto the grid at a constant rate (without really caring about
what the voltage is) while peaker and load following plants will attempt to
vary output relative to their voltage to try and keep the grid voltages
stable.

You are correct, the voltage variance can be misleading at the customer level
if the transformer is actively adjusting it's voltage ratio. I didn't consider
that.

------
2_listerine_pls
Electronics also need to be protected against EM weapons.

------
nwrk
Isn't this great success of 'Russia/China' so-called hacking ?

