
California power outage triggers chaos in science labs - McKayDavis
https://www.nature.com/articles/d41586-019-03086-2
======
TeMPOraL
Reminds me of a story a certain physics PhD told me about certain very known
university in Poland.

One day some years ago, the university administration thought, "why are we
wasting money keeping power on, running lights and aircon and God knows what,
when the buildings are vacant?". They soon turned that thought into action,
and when a long holiday break came, they just shut off the power to the
buildings as soon as everyone left for the day. About a week later, the PhD I
mentioned and his co-workers came back to the lab, saw some droplets on the
glass, and almost had a heart attack.

Turns out, after a week of being without aircon to dry and warm it, the air
was just about to cross the dew point threshold, and if it did, it would kill
the optics on a million-dollar research laser they recently got installed.

I hear that since then, they're a little bit less pound-foolish.

~~~
24gttghh
To be fair, this particular power outage in California isn't to save money.
It's to prevent forest fires. Also, how important can these experiments be (in
CA) if they don't have sufficient backup power at a university??

Edit: If a Uni has a multi-million dollar laser, it should have a few thousand
dollar backup generator!

~~~
dreamcompiler
I think it's to prevent _liability_ from forest fires. The difference is
subtle but important.

~~~
dragonwriter
No, it's to prevent forest fires, which is why the policy was blessed by the
federal judge overseeing their felony sentence and probation for the crimes
they were convicted of in the 2010 San Bruno gas explosion (who is involved in
the fire safety issue because PG&E’s role in last year's fires was charged as
a parole violation), even though he noted that the reason it was necessary is
because PG&E underfunded safety maintenance while issuing massive dividends.

~~~
salawat
May be off-topic, but...

What exactly is the usefulness of a felony as applied to a corporation?

As applied to an individual, it is capable of ruining your life through
privation of constitutional rights (firearm ownership, right to vote,
permanent mark on background checks make it increasingly unlikely you'll ever
hold any sensitive position, etc...)

What does it mean for a corporation to be a felon? It sure seems like the
concept is discontinuous in terms of scale. Obviously, there's an extra layer
of court oversight that's been mandated, so some autonomy is lost. Still
doesn't seem though like the the punishment for a corporation is concommittant
with the general treatment of an individual having committed the same relative
severity of infraction.

In fact, I'm surprised that anyone even _wants_ to do business with anyone
involved with the company during the period the felony was handed dowm.

The more I think about it, it seems like all the "felon" label does is act as
a signpost for discrimination being okay.

Anyway. Brain dropping. My apologies for the tangent.

------
whyenot
My university is not a top tier research university like UC Berkeley by any
means, but our co-generation plant is able to power the whole campus in the
case of a power cut. Even when the central plant gets knocked out (a squirrel
did that last year), individual buildings have backup generators. The building
where I work has three of them and all of our -80s are on back up power or
could be quickly shifted over. That is several layers of redundancy. We have
done a lot of planning for events like this. I'm a little surprised that a
more research focused (and less resource starved) institution like Berkeley
wasn't more prepared.

~~~
joshvm
It depends a lot on individual departments. Most astro groups I've been in
have had very competent sysadmins who aggressively lobby for fault tolerance
and assume that the university proper is incompetent (the very best kind of
angry greybeards). So all the server rooms have enough UPS capacity to
gracefully kill servers, data is properly backed up and so on.

At some places like MSSL we even had an onsite diesel generator becuase there
were flight critical experiments going on all the time.

Many other faculties don't need that level of resilience so they don't bother.
I'm not sure what the situation is for wider engineering or bio/chem. I
imagine there are serious health and safety concerns for things like fume
hoods that must be able to evacuate gases in the event of a power failure.

------
twblalock
I'm glad I live in the city of Santa Clara, where I am served by Silicon
Valley Power, one of few exceptions to the PG&E monopoly in northern
California.

Silicon Valley Power serves many homes in addition to large companies
including Intel and Nvidia, and they do so with lower prices and better
quality of service than PG&E. I will be very unhappy going back to PG&E if I
move to another city.

Electrical utilities don't need to be a government-granted monopoly like PG&E,
which combines the worst aspects of government-run and private-sector
business. Silicon Valley Power proves there are better ways to run electrical
utilities.

~~~
kerkeslager
> Electrical utilities don't need to be a government-granted monopoly like
> PG&E, which combines the worst aspects of government-run and private-sector
> business. Silicon Valley Power proves there are better ways to run
> electrical utilities.

Is Silicon Valley Power _not_ a government-granted monopoly? How does their
business model differ?

~~~
twblalock
Silicon Valley Power is a nonprofit operated by the city government. It's a
little island in the middle of the sea of the PG&E monopoly. I wouldn't
consider it a monopoly in comparison to the massive scale of PG&E, even though
it is operated by a city government.

Statewide utilities are bloated and inefficient, and they abuse their monopoly
power. PG&E acts like the only choices are to cause fires or turn off power
when it gets windy -- it faces no pressure to find a better solution because
there is no competition.

Small-to-medium municipal utilities that serve smaller areas can serve their
customers much better, as Silicon Valley Power demonstrates -- if people don't
like it they can move to a neighboring city. In contrast, PG&E is almost
impossible to avoid.

California should give more cities and counties the ability to experiment with
this model. Competition between municipal utilities is better than granting a
single company a monopoly on a utility millions of people depend on.

~~~
sqldba
> Statewide utilities are bloated and inefficient, and they abuse their
> monopoly power.

This seems overly general. There's plenty of state-wide utilities in the rest
the world which are fantastic in every regard including cost.

~~~
Spooky23
Utilities with regulated monopolies need to be on a tight leash from a
regulatory perspective. They trade operational freedom for a reliable return
on assets.

This changed in the 80s and 90s with deregulation. Some of them even turned
into growth stocks or did other financial fuckery that made investors money at
the cost of rotting out the business.

End of the day, local municipal utilities or co-ops almost always deliver a
better level of service at a lower cost. Smaller scale makes field service
more efficient and reduces overheads like executive compensation. One
municipal utility I’m familiar with has a civil service scale for all staff —
the head guy makes $140k.

------
noodlesUK
PG&E has been doing blackouts and brownouts more than basically any other
electric company I’ve ever experienced anywhere in the world since I was a
little kid. These blackouts may be bigger, but if you’re an institution the
size of a university, you should have significant disaster preparedness
infrastructure. I mean US universities run police forces! I’m sure they could
find a way of designing redundant power into their systems. They might even
save money in the long run, and wouldn’t need to worry about campus hospitals
or public safety going down in an outage (I’m sure some of that is already
taken care of, but why not go all the way).

~~~
zelon88
EDIT: I screwed this all up and it's wrong. I tried. I'm sorry. Leaving it up
so someone else can give it a go. And there's already a bunch of comments.

Washington University uses about 376,394 kWh/year. [1] That's about 7,841.54
kWh/week. Washington pays about 13.1 cents per kWh for power [2]. Running on
the electric grid alone this would cost $598.59 per week. This isn't what they
pay though, because they offset much of this usage with renewable energy like
solar panels.

That would take a 50kw generator to power. Assuming they don't power all
systems during a blackout they could probably get away with a 25kw generator.
For argument, let's assume a 50kw generator full load would probably use about
4.0 gallons of diesel fuel per hour. [3]

At $3.64/gallon for diesel [4], at 4.0gallons/day, for one week the
university's fuel cost would be $2,446.08.

To recap, the university would spend $2,446.08 to generate electricity they
could purchase for $598.59 from the electric company during a 7 day blackout
at full capacity.

[1] [https://www.electricchoice.com/blog/25-of-the-most-energy-
ef...](https://www.electricchoice.com/blog/25-of-the-most-energy-efficient-
colleges/)

[2] [https://www.bls.gov/regions/mid-atlantic/news-
release/averag...](https://www.bls.gov/regions/mid-atlantic/news-
release/averageenergyprices_washingtondc.htm)

[3]
[https://www.dieselserviceandsupply.com/Diesel_Fuel_Consumpti...](https://www.dieselserviceandsupply.com/Diesel_Fuel_Consumption.aspx)

[4]
[https://www.eia.gov/petroleum/gasdiesel/](https://www.eia.gov/petroleum/gasdiesel/)

~~~
jumpingmice
How can a university draw only ~50kW? That’s not even one rack of computers. I
think you missed a large factor somewhere.

UCB has a 26MW gas turbine.

~~~
skunkworker
That's one and a half industrial combi ovens (each takes 480V 3 phase 38kw
power). Those numbers seem so low.

~~~
duckymcduckface
Cause op didn't read citation 1 where it says that the University saved that
much power by switching to better light bulbs

~~~
zelon88
Yeah, I'm sorry. I edited it with a disclaimer as soon as I realized.

You're welcome to continue retracing my steps to correct me though.

------
quotemstr
It's fascinating how these PG&E threads are full of recriminations, but not
solutions. The power went out for 2.6 million people (on 800k accounts). No
natural disaster or unanticipated software bug caused this problem. The
situation is purely a human coordination failure. The state is doing nothing
except issuing sternly worded statements when it could be changing policy.
This odd mood of angry, passive resignation is disturbing.

This whole situation represents a loss of "social technology". Just as
material technology is a bag of techniques for organizing matter into useful
configurations, social technology is a bag of techniques for organizing people
for a useful purpose. When a society loses a technology, be it material or
social, it loses a capability. It becomes unable to do what used to come
easily to it. For 120 years, California was able to keep the lights on. Now it
can't. How is anyone supposed to not see this situation as a kind of
foreboding regression?

This new power grid unreliability is far from the only example of our
struggling to do something that came easily to us 20, 30, or 40 years ago.
We've definitely lost something, although it's hard to pin down exactly what.

~~~
mullingitover
I don't understand this comment. The grid is as reliable as it's ever been,
but PG&E has been _strongly_ incentivized to prevent wildfires. They're
reacting to that incentive by preemptively shutting down the grid. There's no
way of seeing the future where they didn't do that and whole cities burned to
the ground, but based on recent history it's very likely that that's the
future we're avoiding.

~~~
quotemstr
The reliability of the grid is a combination of the _technical_ reliability of
the grid and the _social_ ability to reliably coordinate the resources needed
to operate the grid. You need both systems to work in order for electricity to
flow. As you point out, there's nothing wrong with the grid on a technical
level. But the social element failed, so electricity stopped flowing. When the
electricity stops, it doesn't really matter whether the power is off because
of a technical or a human problem. The lights go off either way.

The wind that shut grid blew at 10MPH. (Granted, it was forecast at up to
45MPH.) Did we not see winds of this strength during the 120 years* that we
didn't shut down the grid over human coordination failures? If PG&E had to
shut down the grid to avoid fire in light to moderate winds, it was because
the people operating the grid were unable to coordinate the resources needed
to operate safely under these conditions. What we've seen here is a breakdown
of human systems that used to work reliably. We used to be able to operate an
electricity grid reliably in moderately windy conditions. Now we can't. That
should worry everyone.

* Yes, I know about Enron. That we've had _two_ grid failures in 20 years due to human-coordination breakdown is a huge red flag.

Edit: phrasing, note that the winds were forecast to be stronger than what we
ended up seeing

~~~
mullingitover
Where are you getting this 10mph number? There's a wind advisory in effect
with gusts peaking between 35-45mph this morning[1].

> NWS forecasts call for winds to be slightly weaker Thursday throughout the
> Sacramento Valley than they were Wednesday, but a wind advisory remains in
> effect through 6 p.m. as gusts between 40 and 55 mph remain possible.

[1]
[https://www.sacbee.com/news/california/fires/article23595593...](https://www.sacbee.com/news/california/fires/article235955937.html)

~~~
quotemstr
I was reading [https://www.cnbc.com/2019/10/09/pge-cuts-power-to-
millions-i...](https://www.cnbc.com/2019/10/09/pge-cuts-power-to-millions-in-
california-amid-hot-windy-weather.html), particularly this bit: " Wara said a
forecast Tuesday night showed winds in Mt. Tamalpais, a nearby mountain, were
less than 10 mph (16 kph). “There is zero wind where I live,” Wara said. “PG&E
is required to shut down based on observed conditions, not forecast
conditions.”"

The article goes on to mention forecasts of 35-45mph gusts. I'll edit my
original post. My point stands: whether we're talking about 10mph winds or
45mph winds, we're talking about weather within the normal range of variation
and conditions that electricity grids elsewhere handle admirably.

~~~
danans
> within the normal range of variation and conditions that electricity grids
> elsewhere handle admirably.

California has very different conditions than most other places in the US. It
gets no rain for 9 months a year followed by a wet winter which results in a
lot of vegetation growth. And that pattern is getting more extreme. You can't
compare that to a wetter place, or even to a much dryer place with little
vegetation.

------
ChuckMcM
This is such a screw up. Personally I think this is PG&E trying to negotiate
with the State to have it indemnify them against wildfires so that they
neither have to pay to maintain their infrastructure, nor pay out large damage
claims when that infrastructure burns down a town or two.

I really wish the PUC wasn't captured, otherwise we might be able to fix this
problem.

~~~
networkimprov
100 years of US govt forest policy provided the fuel, PG&E provided the
sparks.

Edit: this is not an anti-government rant. There is far more vegetation in the
wildlands of California than is natural due to forest fire-fighting policy
dating back 100 years.

~~~
azinman2
Can you explain more?

~~~
mdorazio
Probably that many places have pursued policies of preventing forest fires
entirely when they are actually a natural part of ecological development. If
you keep preventing forest fires, you end up with a lot more potential fuel
buildup than if you let them burn once in a while.

I'm sure someone will point out that CA does controlled burns on occasion, but
it's not the same thing.

Edit: here's some reading on the topic:
[https://www.npr.org/2018/09/27/649649316/fire-ecologists-
say...](https://www.npr.org/2018/09/27/649649316/fire-ecologists-say-more-
fires-should-be-left-to-burn-so-why-arent-they)

------
ethanpil
California has the 5th highest rate per kWh[1]. (#2 excluding Alaske and
Hawaii)

Just like our sales tax rates, income tax rates, our gas prices (due to
taxes), sanitation costs, ad nauseum. We pay excruciatingly high rates for
everything, and get mindbogglingly low returns for the money.

CA gov. and utilities are a textbook case in poor management and government
out of control.

Where does the money go?

[1] [https://www.chooseenergy.com/electricity-rates-by-
state/](https://www.chooseenergy.com/electricity-rates-by-state/)

~~~
bontaq
...PG&E is a corporation. If anything, the CA gov should make it government
owned and remove the profit motive.

~~~
ethanpil
It's barely a corporation. Just another monolithic utility with government
mandated monopoly and no incentive to provide service or value.

------
mikorym
Not to be insensitive, but for a South African this is a kind of LOL moment.
We have had power outages mess with experiments and universities since at
least the early 2000's. That of course and the arson that burned down
auditoriums.

But in terms of experiments, if you want your MSc or PhD then you better make
sure that you can go a couple of days without power to the grid.

If you are in a rural area, especially a previously "white" area as the
pre-1994 NP government called it, then you need to have your own > 5000 L
water tank. The town where I went to primary school had 9 days without water
or electricity a year or two ago. Probably half of the town (it's a small town
though) never have water between 9PM and the next morning since they simply
turn of the pumps (whether there is electricity or not). The government
officials also steal water to go sell for $1 per 100 L to areas where there is
no running water.

You can think of it as a kind of osmosis between previously affluent areas and
areas that had been mud huts. If you think of it that way it doesn't sound
that bad, actually, but I think the net position is not the issue, the real
issue to me is the corruption and apathy within the government and the rate of
improvement. Africa should be ahead of China in economic growth if you look at
the youth percentage and potentially massive workforce.

If you want to completely redesign an entire country's power grid and make it
100% solar/wind, then South Africa is the perfect candidate. That is if you
could forget about politics for a moment.

------
egdod
Power outages happen sometimes in the normal course. They are not a new thing,
and the fact that this one is planned doesn’t make it worse.

If you have specimens that you cannot afford to have warming up, it’s
inexcusable that you don’t already have a backup power source.

~~~
alloutblitz85
Lol what are you on? This was one days notice on a week's notice. Due largely
in part because they neglected to maintain their infrastructure over the years
so they can shave off operating costs - for better quarterly reports.

This is not normal.

~~~
egdod
You know what can provide 8 days of backup power? A generator (along with
periodic trips to the gas station). Which you should already have if your
power needs are this critical.

------
claudeganon
I wonder at what point California will just nationalize their grid/energy
infrastructure. These “markets” always end up as some private monopoly with
all the corresponding terrible incentives around upgrades and maintenance. If
it was actually controlled by the state, it could be mandated to build out
renewable-centered, decentralized systems as well.

~~~
chrisco255
Well, in order to nationalize their power they would first need to be a
nation. To be honest I don't know why one giant utility for an entire state is
a desirable thing. In most places I've lived in, each community runs their own
utility company. I've had cooperatives, city owned utilities, and private
utilities. The best experience by far was the co-op. Since the customers owned
the utility, we would get the profits back as credits on our bills usually
during the middle of summer when power usage is highest in Florida.

~~~
CameronNemo
>in order to nationalize their power they would first need to be a nation

A nation is a social construct that California could certainly fit into. Hell,
Jerry Brown called it one. Examples of nations: Palestine, the Kurdish nation,
Tibet.

The legal construct is called statehood. States can bind themselves into
larger legal entities by giving up some of their own rights. Examples include:
the US federal government, the European Union, and the United Kingdom.

~~~
chrisco255
Sorry, no. A nation is a legal construct, and California is not a sovereign
nation. If it's a social construct then I'm founding my own nation tomorrow.

I don't know anything about Jerry Brown but he doesn't sound very bright.

~~~
nostrademons
A nation is a group of people with a common culture, identity, and usually
language. A state is a legal construct implying self-governance and monopoly
of force.

"Nation" has become colloquially synonymous with "nation-state", which lead to
your confusion. But the legal aspect is from the "state" part, not the
"nation" part. There have been historical states (eg. the Holy Roman Empire,
Hapsburg Austrian Empire, Ottoman empire) that were not nations, and there are
present day nations (the grandparent mentioned some: Palestine, Kurdistan,
Tibet, as well as Basques, Zulus, and others) that are not states. Some
present day nation-states (eg. Germany & Italy) were nations since medieval
times but did not achieve unified statehood until quite recently (1871). The
U.S. started as a state (technically, 13 states and one federated government)
but did not really become a nation until after the Civil War.

If you can convince other people to identify with you, you absolutely can
found your own nation. It won't be a state, though, so while might have your
own customs good luck on having your own laws.

~~~
sseveran
The US was most definitely a nation before the Civil War. And the Civil War
was waged to prevent people that identified with each other from founding
their own nation.

------
atonse
Maybe these kinds of things will push us more towards decentralized power
generation like solar and wind, and bigger lab-wide or campus-wide battery
battery backups?

~~~
mc32
No, it's going to lead to even more regulation which in turn will constrain
what PG&E can do even further. At this point, it should just become a state
owned and operated enterprise and drop the dressing of being private.

Once it does that, we'll come to appreciate their service as much as we
appreciate the schedulekeeping of the MUNI.

~~~
baddox
PG&E has a government-granted monopoly to provide its utilities to its
coverage region. That makes it indistinguishable from a state-owner
enterprise, except that there are probably even more people reducing
efficiency by extracting value for themselves.

~~~
SamReidHughes
Extraction of value means increased efficiency is incentivized. Obvious
caveats about cutting corners with brush removal but also the shared
responsibility of homes built in dangerous places apply.

~~~
baddox
Where would that incentive come from? Not from the threat of competitors,
because no competitors are allowed to exist. There would perhaps be some
slight incentive due to the slight elasticity of demand for electricity (i.e.
people can probably reduce their energy usage in response to higher prices),
but I imagine that elasticity is quite low indeed.

It’s more likely that the more significant incentives would be other things,
like, oh, not maintaining equipment and eventually causing wildfires.

~~~
SamReidHughes
Increased efficiency increases profits.

~~~
baddox
Increasing efficiency requires significant investment, which decreases profits
_now_. And why do that now, when you have no competitors?

~~~
SamReidHughes
How do they have shareholders?

------
legulere
Meanwhile in Germany the average duration of blackouts in 2017 was around 20
minutes per customer, in quest where it was pretty high because of storms. The
frequency was 0.28 in 2017, which means you can expect every three years an
outage.

Source:
[https://www.vde.com/de/fnn/arbeitsgebiete/versorgungsqualita...](https://www.vde.com/de/fnn/arbeitsgebiete/versorgungsqualitaet/versorgungszuverlaessigkeit/stoerungsstatistik-2017)

~~~
Tepix
And we pay extremely high prices for electricity.. near $.30 per kWh

~~~
mschuster91
That's because consumers subsidize the big smelters and other industrial
customers who don't pay the renewables-buildout tax (EEG Umlage). The
generalmtax load on energy is immense.

------
cwhittle
All of these big research universities in CA are on major and active fault
lines. My old lab at Cal sat mere hundreds of feet from the Hayward fault that
runs through campus. This is a proactive power shutdown when you know it's
coming. If they can't an event with advanced warning, what will happen to
hundreds of millions of dollars of frozen samples and research instruments
when the unexpected earthquake hits any of these universities, not only
disrupting power, but also disrupting structures? Incredibly foolish not to
have at least reliable power backups and containments irrespective of what PGE
is doing.

------
StanislavPetrov
We're going to spend far in excess of 700 billion dollars to maintain our
global military empire this year alone (not counting legacy costs for veterans
and countless other huge, related expenses) while millions of Americans sit in
the dark without power because we outsource our control of our power grid and
other critical infrastructure to for-profit corporations. Hopefully some of
those people sitting in the dark will give that some thought while the lights
are out.

------
secabeen
This isn't the best sourced article, the sources here are all random people
complaining on twitter. It would be much more interesting to hear from an
informed spokesperson from UC Berkeley, with actual data on the percentage of
STEM research buildings without generators, etc. etc.

Even then, one of the sources did describe that they have an e-power plug in
their lab, for the -80 freezer. That's pretty much how it's supposed to be. If
they have additional important fridges, those should also be plugged into
e-power.

I'm sure it's possible there are some labs at Berkeley that may be compromised
in backup power, there are a lot of old buildings and infrastructure there.
That still doesn't mean that that experience is common, just that the people
impacted are complaining loudly. Most research labs may be totally fine, and
just waiting for things to come back.

------
pvaldes
Curiously, a similar problem happened last week in the Institute of Natural
Products and Agrobiology from the Consejo Superior de Investigaciones
Científicas in Tenerife.

Temperature of ultrafrozen samples raised to -75 Celsius and years of hard
work and a collection of DNA reference samples for agriculture were about to
vanish

------
dannykwells
Berkeley, my post doc alma mater, really has fallen on hard times. I get that
PGE caused this, but UCB should have reliable back up power for their labs.
Full stop. You can see why they're fighting so hard in the CRISPR case.
They're desperate to raise dough.

------
Mathnerd314
Here on page 682 (10.3.1D.19) "High-value specimen refrigerators, freezers,
etc." are listed as "optional" for emergency power:
[https://www.orf.od.nih.gov/TechnicalResources/Documents/DRM/...](https://www.orf.od.nih.gov/TechnicalResources/Documents/DRM/DRM1.4042419.pdf)

Clearly it's a big inconvenience but they're not in danger of losing any
grants.

------
fyfy18
I think it is interesting to compare PG&E to how the electricity grid is owned
and run in the UK. Many people here are saying they should be nationalised,
well the UK did the opposite 30 years ago and it seems to be working fine.

In the UK there is National Grid Electricity Transmission plc, which as the
name suggests is responsible for the transmission infrastructure across the
country. They are a publicly traded company, and the parent company owns
electricity & gas distribution in the UK, and some in the US. There are then
smaller companies who are responsible for maintaining more local
infrastructure (as I understand it's still owned by National Grid, but they
are responsible for fixing and upgrading it).

As a consumer you can choose to buy electricity from a long list of companies,
and easily switch between them. All that happens is a different company
collects your meter reading and they send you a bill. A lot of these companies
also own generation facilities (e.g. SSE, who are also publically traded)
while some are just resellers (e.g. So Energy).

~~~
perl4ever
I'm in NY, and the gas & electric company is National Grid, and they do give
you the option of choosing a supplier.

I've always thought that there must be hidden pitfalls of selecting a
different supplier, since National Grid would have the ability and incentive
to tilt the playing field.

------
gwbas1c
This is something I don't understand...

Portable generators are so inexpensive that I'd think an institution like this
could just keep some in a shed or garage, and shlep cords through windows for
extended outages. I bought a top-of-the-line tiny Honda Inverter generator so
I could keep my wife's breast milk frozen in case of an extended outage, but
even something half the cost would be good enough to run a refrigerator.

But, even then, if you plan for it, backup generators and Solar + Battery are
very affordable!

I recently installed a Powerwall purely for backup. Because I have solar, it's
slightly more expensive than a standby generator, but with no maintenance or
fuel I suspect it'll be a wash within a few years. (Backup generators need
expensive maintenance every year.)

Even Generac bought a Solar + Battery company because they know they can't
compete, and their solution is nicer than my Powerwall! (The Generac solution
includes a critical loads panel inside of it, unlike Powerwall where the
critical loads panel is outside.)

------
dmode
I have no idea why this shutdown was initiated. It had been much warmer and
drier earlier in the summer. It is actually quite pleasant and cold these
days, with morning dew. And no winds. I have no idea why these conditions
warrant power shutoff

~~~
aodin
The intensity of these blackouts is driven not just by weather, but by a major
change in risk assessment by the utilities. PG&E, the utility that services
most of northern California, was sued by a number of insurance entities after
it was found that their equipment was the cause of several major wildfires
over the last few years. Most notably the 2018 Camp Fire, which was the
deadliest and most expensive wildfire in California history.

But most importantly, it's what didn't happen: the state of California did not
bail out or grant PG&E immunity from these lawsuits. PG&E settled with the
insurance entities and has since filed for bankruptcy protection, since the
possible liability is larger than their market cap.

The blackouts are a way to minimize further risk and liability. But I would
also guess that the intensity of blackouts is a not so subtle middle finger by
the utilities to the state government for its resigned treatment of these
issues.

------
lawrenceyan
I go to UC Berkeley. My midterms were postponed so I feel like even though I
should be complaining, I really can't. PG&E basically gave me a week extra to
study.

------
dekhn
all the SREs on the thread are wondering why scientists don't automatically
mail their complex samples to their friends across the country as a backup.

and the answer is.... individual PIs optimize for the easy case (that power
stays on), not the disaster recovery scenario. There is no SRE thinking at the
individual PI level.

------
bavcyc
Rather than answer several individual posts, I'll be lazy and do a larger post
which involves a lot of simplification and hand waving. I apologize in advance
for any errors, but this seemed like a good way to handle insomnia instead of
my normal lurking. If something is unclear or you think wrong, let me know.

Power Companies circa 1900 there were lots of power companies, I've seen
pictures of urban areas where there were multiple circuits run on poles, how
many companies tried to serve a certain area, I do not know. Through lobbying
investor owned utilities (IOU), or in most cases a single IOU gained the
rights to serve an area exclusive. IOUs concentrated, for the most part, on
serving dense load concentrations. As such the US Govt implemented the Rural
Electrification Act, so low density areas could be served. The 2 types of
electric utilities are public power (municipal, REA, RUS, PPD, UD, etc) and
for profit (IOU's typically). In most cases, the IOUs give something to the
government in exchange for the IOU providing service exclusively, it might be
a tax or it might be free street lights. Municipals tend to subsidize the
local government in some way, either through returned dollars or free power
(street lights, buildings, traffic lights). IOUs typically have a defined rate
of return and are supervised by a governing entity of some sort.

Vegetation Management The recent SERC compliance meeting had a good
presentation on vegetation management from the utility perspective and another
on enforcement trends. In my opinion a lot of the issues in the W US is the
result of the policy 'no fire is good', the sand pile game I think illustrates
the issue where the longer sand keeps from falling results in a larger
collapses (see Yellowstone fire). Since the late 1980's the issues with not
burning has been known, I had an ecology class where if I recall correctly
that was discussed for a couple of days. Tree trimming and clearing out
undergrowth is done on a regular basis when the utility has an easement, but
especially in urban areas folks tend to plant trees too close to power lines
or even worse encroach on the easement with buildings. Most utilities patrol
transmission lines at least once a year if not twice or monthly, sometimes
this is aerial and other times it is feet on the ground walking the line. As
an aside, the NESC governs clearance of electric lines to stuff and how stuff
should be built; the RUS has publications on line design if you want to read
about it.

Distribute Generation The electric grid in the US is divided into 3 areas,
Eastern Interconnect, Texas and Western Interconnect; they all function
essentially the same. If I have a generator connected to the grid, it has to
synchronize to the grid before closing the breaker. If it is done correctly
then there is very little mechanical stress on the generator, if done
incorrectly then there is a large amount of mechanical stress on the
generator. One mis-operation I know about involved the A and B phases being
swapped during a re-wind, when the generator was closed in at commissioning it
had a large bang/clunk and the breaker opened immediately. The generator then
had to be examined, i.e. taken back apart, to figure out what went wrong and
if it could be put back into service.

If I have a generator partially supplying a facility (this can save a lot of
money for an entity) and a fault happens on the grid then my goal is to
protect the generator, so the generator will either shut off or island the
facility while shedding load above the generator's capacity. This happens very
quickly. One instance I know of, the urban area was supplied by transmission
(aka remote generation), a 30 MW generator was the closest source, the utility
had a fault because equipment misoperated and the generator was suddenly
trying to supply all the power to that fault such that the generator
protection operated and islanded the facility. It was no issue to close the
grid interconnect back in (once it was ensured it was safe to do so) but the
facility had to shed load to keep the generator running without causing
electrical issues to load and damaging the generator.

Once a facility is islanded and running on its own generation the phase angle
is a don't care until it is time to synchronize back to the grid. As long as
the facility can shed load to maintain frequency (there is a NERC standard on
Under Frequency Load Shed if you want to read about it) and not ruin equipment
by having a power quality issue. During dynamic studies for system stability,
it can be observed that a generator will diverge from the system frequency
phase angle but not trip off because it is isolated from the grid which
requires verification that isolation is happening and the protection scheme
will indeed work that way on the actual system.

My observation is that most facilities, data centers and other processing
facilities (refineries) tend to be the exception, concentrate on first costs
when designing their electric infrastructure. It is possible to design a
resilient system but it has a cost and it will not be utilized 100% until
something goes wrong. And if you are doing research then that can be an issue
as you may lose a large amount of data due to the power going out or possibly
being sensitive to transients on the system, e.g. a switching operation on the
transmission system affects the end user equipment. Even if you have redundant
systems (and/or power supplies) it is possible to have single point of
failures on your system. As well if you have enough local generation to supply
your load, it may be more economical to not run 100% of your generation as the
market price for electricity is cheaper than your cost of production (and
there are folks who don't like idle assets, not realizing the greater benefit
is not using it or only having to use it infrequently).

One other aspect of distributed generation is the automatic separation of the
DG when loss of voltage is detected on the grid side. Utilities do not want
voltage on their system if they have an outage due to worker safety (and other
reasons). Utility crews in hurricane areas will typically investigate if they
hear a generator running when the power is out to an area to ensure it is not
back feeding the distribution line. As a reminder keep your feet together if
you are near a downed power line and hop away, or even better don't go near
downed power lines.

------
mensetmanusman
People will now experience rolling black outs in California just like in South
Africa? Is this just because of the lawsuit?

~~~
eplanit
Yes (IMHO). PG&E was found responsible for the biggest/worst fire(s) in recent
history, and has since declared bankruptcy. They're giving us Californians a
big middle finger. I mean really, suppose a fire were to start on the Berkeley
campus -- that would not trigger a forest fire miles away.

~~~
SamReidHughes
It’s not the campus, it’s the transmission lines connecting to it.

------
proee
This event will increase the sales of Tesla Powerwalls.

~~~
twblalock
Pretty sure most of the people who wanted those already bought them. Sales
will increase on gas generators, not Powerwalls.

~~~
LeoPanthera
I can add my anecdata to your own: I will probably buy a Powerwall because of
this incident.

~~~
michaelt
This was a 5-day power outage, right? Is your house energy-efficient enough
that a Powerwall will run it for that long?

~~~
LeoPanthera
Our power was out for slightly less than 24 hours.

------
driverdan
If you have work that can be destroyed by a power outage you should plan
accordingly. Power outages are a normal thing and can happen at any time,
especially in areas with earthquake and fire risks. Not preparing is foolish.

~~~
Maxious
Be Elon:

> Tesla factory unlikely to lose power _even if most of Fremont does_ because
> of agreement with PG&E

[https://twitter.com/PBRStreetGang7/status/118178393541343641...](https://twitter.com/PBRStreetGang7/status/1181783935413436416)

( disclaimer: might be usual twitter humbug
[https://www.smh.com.au/business/markets/the-crowd-sourced-
so...](https://www.smh.com.au/business/markets/the-crowd-sourced-social-media-
swarm-that-is-betting-tesla-will-crash-and-burn-20190409-p51ck5.html) )

~~~
baddox
So, be corrupt? There’s no other way to describe a deal that a large company
can make with the electricity company to keep its electricity running when the
electricity company shuts off everyone else’s power.

~~~
core-questions
Why is this automatically corruption? They're no doubt paying for the
privilege.

~~~
baddox
Huh? The part where they pay doesn’t make it not be corruption.

~~~
quotemstr
If it's "corruption" to pay more for greater reliability, it's "corruption" to
pay more for a large sandwich than a small one. There's nothing wrong with the
quality of good or service varying according to how much you pay.

~~~
baddox
Sure, you could frame it that way: Tesla is paying for “increased”
reliability, and everyone else just gets their power shut off at a whim.

To me, there feels like a big difference between a reliability SLA and an
exemption from what is supposedly a vital emergency shutdown.

~~~
jjeaff
Presumably, if you are paying more, then part of that increased amount you are
paying is to make certain that the portion of the grid needed to keep your
factory going is properly maintained. Unlike the rest of the grid. Which would
apparently be too expensive (i.e. everyone on the grid would need to pay the
extra fees for increased uptime).

------
TomMckenny
Note on the blackout map, the county in the dead center has continuous power
and it has a municipal source.

Likewise the 2000 crisis where LA's DWP customers had power when PG&E did not.

Although I'm sure it makes me a "socialist" for pointing these things out, it
almost seems that having private monopolies in utilities is a bad idea

~~~
aodin
The issue at hand is dry brush and trees being blown across live wires and
sparking wild fires. The blackouts are targeted at the areas where this is
most likely to occur. Namely, not heavy urban areas.

Although not as widely reported, SoCal Edison is also doing rolling blackouts
in at-risk areas [https://www.latimes.com/california/story/2019-10-10/l-a-
face...](https://www.latimes.com/california/story/2019-10-10/l-a-faces-
critical-fire-danger-possible-power-outages-as-santa-ana-winds-buffet-
southern-california)

~~~
TomMckenny
The problem is lack of investment in the infrastructure. And that damage to
customers or economies is not a concern compared to lawsuits damaging the
bottom-line.

In particular, if you only take into account the revenue made on a given day
vs the likelihood of a lawsuit and nowhere is economic damage accounted for,
then you will turn the power off a lot more often and a lot more widely than
you should.

Also, since SoCal Edison is geographically adjacent to the DWP but Edison is
investor owned while DWP is municipal owned, I believe you are adding support
to my point.

~~~
aodin
Exactly, the question we should all be asking is how it got to this point. I
reckon you'll find a lot to blame with both the utilities and state
government, and the existing incentives for both.

However, to your point about LADWP, I think you're making an overeager
classification. It's not a matter of being "geographic adjacent", but being
liable for the infrastructure that would start wildfires in at-risk regions.
Take a look at the difference in coverage areas of Californias utilities:
[https://ww2.energy.ca.gov/maps/serviceareas/Electric_Service...](https://ww2.energy.ca.gov/maps/serviceareas/Electric_Service_Areas_Detail.pdf)

edit: it is tangential to this issue, but LADWP does not have a very glamorous
history, either.

~~~
inferiorhuman
_Exactly, the question we should all be asking is how it got to this point. I
reckon you 'll find a lot to blame with both the utilities and state
government, and the existing incentives for both. _

No this is solely PG&E's doing. When you write the regulations, you own the
blame. PG&E claims it's too expensive to underground its transmission lines
(which would prevent fires like these) at a bit over a million per mile.
Meanwhile they took blew their safety budget on executive bonuses. You want to
tell me that undergrounding around 100 miles of lines isn't going to reduce
the risk of fires? How about undergrounding 500 miles? Because in addition to
the safety budget, PG&E also spends around $400 million annually on stock
buybacks.

I don't particularly care if PG&E remains a private company, but there need to
be consequences like jail time for actions that go far beyond mere negligence.

~~~
aodin
You're arguing against a straw man. I agree with you: PG&E has acted terribly.
The attempted executive bonuses (currently being denied by a court) and stock
buybacks ($250MM in F'18, $395MM in F'17, and _$968MM_ in F'16) are especially
egregious. The company should no longer exist.

But PG&E are not the only offenders here. You seem to be arguing that there's
regulatory capture (correct me if I'm wrong), but the Public Utilities
Commission still exists, and they have the authority to regulate PG&E. If the
commission is neglecting its duties, then it should be restructured as well. A
president of the commission has been forced out before, and it should probably
be done again. If they neglected such an obvious case of greed and
mismanagement because of bribery (sorry, _lobbying_ ), that should be a
criminal case, too.

Public officials can throw PG&E under the bus all they want, but if systematic
changes aren't made, we'll be doing this song and dance again in a few years
with a different utility.

------
exabrial
I mean, don't live in a desert? Seems like an obvious conclusion

~~~
aodin
The dangerous combination that warrants blackouts is high winds, dry brush,
and exposed live wires.

While this combination is more likely in a desert, it's not true of all. For
instance, the Salt River Project that oversees electricity transmission in
Arizona switched to building underground lines in the 1970s.
[https://www.azcentral.com/story/news/local/asked-
answered/20...](https://www.azcentral.com/story/news/local/asked-
answered/2014/07/14/asked-answered-power-lines-above-below-ground/12571659/)

------
unethical_ban
Surely, PG&E will learn their lesson during this outage and send crews across
the state to trim brush along the transmission lines? Strengthen the towers?

Uncharacteristic edit: Why the downvotes? It must be because you took the
comment seriously and think the sentiment grossly naive, or heard the
skepticism and thought it unwarranted?

~~~
twblalock
They blew up an entire neighborhood in San Bruno ~9 years ago. They don't
learn.

