
It was raining in the data center - lgregg
https://medium.com/s/story/it-was-raining-in-the-data-center-9e1525c37cc3
======
Animats
And the article went on, and on, and on...

As for their humidity problem, the article says little about how their badly
designed chiller-less system got into a condensing situation. The original
article in the Register does.[1]

This was an HVAC configuration error. The facility used evaporative cooling,
which dumps heat into water by running air through a water spray or some wet
surface. This raises the humidity. They somehow got the system configured so
that it was recirculating air through the evaporative cooler, increasing the
humidity on each pass until it hit saturation. The problem only lasted for a
few hours until somebody figured out what was wrong.

They probably didn't have a union-trained stationary engineer on site. This
sort of thing is covered in the apprenticeship training for a stationary
engineer. Fans, dampers, and controls can all fail, which can create
situations like this. Stationary engineers deal with those problems. At large
data center scale, you may need one.

[1]
[http://www.theregister.co.uk/2013/06/08/facebook_cloud_versu...](http://www.theregister.co.uk/2013/06/08/facebook_cloud_versus_cloud/)

~~~
jiveturkey
it went on and on and on because the article isn’t actually about the cloud in
the data center.

~~~
thomasjudge
It's not really clear what this article is about

~~~
UnderProtest
It's a poem.

Edit: It's a mediocre poem.

------
mrkgnao
> like a thunderstorm formed from two opposing weather fronts, the physicality
> of outside Prineville air met the everywhere-ness of inside space in an
> impossible, non-Euclidean intersection

Not every word is a pretty one you can throw around, you know: what exactly
about this metaphorical meeting was "non-Euclidean"?

~~~
ibotty
The metric that connects data centers is not Euclidean. There are some names
of that metric but for that story the name "railroad metric" might fit best.

------
ucaetano
> Furthermore, much of this energy consumption is generated by coal — even
> ‘green companies’ will often buy carbon offset credits rather than invest in
> the energy storage required for 24/7 solar or wind power.

Plain bullshit. One of the reasons Facebook and Google have datacenters in
Oregon (FB at Prineville and Google at The Dalles) is the availability of
cheap renewable energy from the Columbia River.

And the companies don't buy carbon offset. The author has no idea how
electricity markets work. They buy renewable electricity contracts, for
example:

[http://www.datacenterdynamics.com/content-tracks/power-
cooli...](http://www.datacenterdynamics.com/content-tracks/power-
cooling/google-buys-up-62mw-of-dutch-wind-power/91896.article)

The author seems more preoccupied with the sesquipedalian loquaciousness than
with actual facts.

~~~
puzzle
I was at Google and they did buy offsets from 2007 until after I left. Not for
Oregon, sure, but definitely for places like the Carolinas (our good old
friends at Duke Energy...). It was only in 2017 that they covered 100% of
corporate/datacenter electricity with actual contracts.

------
curuinor
Of course the article isn't about the actual cloud in the actual data center.

Pipkin is a code artist, this is essentially a philosophical musing, but of
the continental kind, with importance given to phenomenology, with thought on
metaphilosophy and suchlike, with philosophy given as first science.

This kind of author specializes in deeper critique in that continental style.
Most such works are technically worthless, but that is not the point.

~~~
ucaetano
It isn't just technically worthless, it is technically wrong. I know that none
of it is even the point, but that's where the article loses itself: there is
no point but to make an appealing critique of nothing in particular.

It isn't even wrong.

------
AngryData
This seems like an egregious oversight to me, no one thought about controlling
the humidity levels in this carefully climate controlled server farm
beforehand? They used evaporation cooling to lower energy costs which ideally
would saturate the air up to near 100% humidity in order to work best, how
would you not know taking that air directly would result in water and humidity
problems? They likely had multiple engineers working on climate control by
itself, someone should have foreseen it.

------
nkrisc
The article reads as if it was as procedurally generated as the bizarre
graphics included.

------
CamTin
The much better (and probably worthy of a modern spiritual successor, which
this article is not) version of this theme is Neal Stephenson's excellent
"Mother Earth Mother Board" from a 1996 issue of /Wired/:

[https://www.wired.com/1996/12/ffglass/](https://www.wired.com/1996/12/ffglass/)

------
ghotli
Shitty title for what I felt was a really great read.

Actually, it honestly reads like a prequel to Asimov's short story, "The Last
Question". [1]

[1]
[http://www.multivax.com/last_question.html](http://www.multivax.com/last_question.html)

------
emodendroket
This article was a bit frustrating. It has a bunch of interesting threads it
never follows to the end or ties together.

------
davidw
I'm always amazed when I think of a big data center in Prineville. I'm
currently reading a book on the history of the area:
[https://amzn.to/2KLxqyM](https://amzn.to/2KLxqyM) , since I live down the
road in Bend. Pretty much your basic small western town, with native
americans, range wars, and a murderous group of vigilantes in its past and a
popular rodeo in this day and age.

Nice place to visit if you ever get the chance, on the way to the Painted
Hills for instance.

------
eftychis
Speechless (occurrence year 2011). Was it really just lack of appropriate
personnel or design flaw. I just can't get that information out of the
article. Someone cut corners.

Now, the article lacks focus. It expands unecessarily on every tangent (it
reached ARPANET -- drops mic). 2 or 3 pages could have been 2-3 paragraphs or
references. This article lacks an editor. (Or they are payed per page -- I
think it is word count actually.)

The author could state: they pick remote locations, because it is cheap and
away from publicity (power hungry) and less likely to become a target. Then
argue and clarify with some details. Then go to the destructive events
perhaps.

Like many others, I am not sure of the goal of the article to be honest.

------
hermitdev
This reminds me of something that happened in our on-site datacenter at a
former employer of mine. It wasn't raining inside, they were just doing
routine maintenance on the cooling system. Somehow, they managed to dump all
of the cooling water into the offices (which were not ours) in the 2-3
immediate floors below. Needless to say, there was an emergency shutdown as
the datacenter overheated. It was definitely an all-hands scenario as we got
ready for market open the next day. I think I crashed around 4am-5am after my
servers were back up and checked out.

~~~
walshemj
I had my small basement (shared with the print department) computer room
flooded in London - this was the second room the one before was flooded and
was repurposed as a printer room - it still had the flood marks on the wall.

------
rmason
Just noticed on the fiber map contained in the article that there isn't any
fiber in Michigan's upper peninsula.

How is that even remotely possible? It appears that it runs up to to Mackinac
City and then stops short of the bridge. The map doesn't even show the U.P.

I was at a tech conference once that had attendees states on their badges. A
lady came up to me and said how cute you've got a deer jumping over the state
of Michigan on your badge. Told her that's not a deer, it's part of the state
;<).

------
9935c101ab17a66
This article is very strange, and lacks a cohesive narrative. It meanders
willy-nilly. I stopped about half way through.

Also, the graphics are really bad, and completely distract from the already
questionable piece.

~~~
ucaetano
It is classic postmodern text: it is supposed to look well informed,
intelligent, but it lacks any coherent meaning. Simply fashionable nonsense.

------
blakesterz
They have a link to a study of the major internet routes in the US and the
name of the PDF is 'tubes_final.pdf'

That made me laugh.

"Long-haul fiber-optic cabling. Red squares mark where cables connect in
nodes, many in major population centers. University of Wisconsin and ACM
SIGCOMM, 2015.
[http://pages.cs.wisc.edu/~pb/tubes_final.pdf](http://pages.cs.wisc.edu/~pb/tubes_final.pdf)
"

~~~
felix_nagaand
Link is dead

~~~
lintroller
It's because of the quotation mark. If you remove it, the link works as
normal.

[http://pages.cs.wisc.edu/~pb/tubes_final.pdf](http://pages.cs.wisc.edu/~pb/tubes_final.pdf)

~~~
blakesterz
Fixed, thanks for catching that

------
ChuckMcM
TL;Dr version - author takes an article on data centers, an article on long
haul fiber, and an article on the structure of the Internet and puts it into a
blender.

It is kind of too bad since two of the three articles haven't had a lot of
good coverage. One being alternate cooling strategies for data centers, and
the other being disrupting the moats that long haul fiber imposes.

------
catchmeifyoucan
I didn't understand the graphics

------
matchagaucho
IIRC: to meet the "redundant fiber" requirement, one of the backbone carriers
needed some incentives to build out into the middle of a desert.

------
walshemj
Not sure the fact that cable runs next to train tracks - railway tracks are
built on the easiest routes.

In the UK there are canals trains tracks and motorway running next door as the
geography suits.

~~~
dgacmu
It definitely follows the tracks. Using existing tracks provided to major
advantages: the railroads already had the right-of-way, and it was relatively
affordable to lay the fiber by equipping a locomotive with a special trenching
attachment. (A fiber optic rail plow).

------
sAbakumoff
That's the longest article ever. What's TLDR? Facebook data center produced a
cloud within it?

------
paxys
Those graphics were...interesting.

~~~
phinnaeus
Are they from some sort of scientific simulation? As graphics, they look
terrible. I hope they have some sort of value outside aesthetics.

~~~
MrRadar
They reminded me very much of the artwork that accompanied 17776:
[https://www.sbnation.com/a/17776-football](https://www.sbnation.com/a/17776-football)

~~~
lgregg
that was strange

~~~
Avamander
.awesome you mean

------
jedberg
Is there some astroturfing happening with this link? It's at the top of HN
with 60 points, only two actual comments about the content, and I personally
couldn't find the point of the article.

Was is just a survey of interesting datacenter phenomena? I'm still not sure.

Edit: Amusingly this post had a bunch of points and now they're gone. Maybe
people don't find this comment interesting, or maybe the people who were
manipulating the vote didn't like getting called out. One shall never know I
suppose.

~~~
Moral_
Yeah this post reeks of vote manipulation. The article was strange and droned
on and on.

------
spraak
> You’ve reached the end of your free member preview for this month. Become a
> member now for $5/month to read this story and get unlimited access to all
> of the best stories on Medium.

Ugh. I didn't realise that Medium had become another paywall.

~~~
paulie_a
Not surprising. It became useless a while ago. It will be a shame the content
will be lost when they go under which probably won't be very long.

