
Shouting in the Datacenter (2008) [video] - bcaa7f3a8bbc
https://www.youtube.com/watch?v=tDacjrSCeq4
======
united893
Ah, it's a classic. And wow, Brendan Gregg looks so young. And video uploaded
by Bryan Cantrill himself. Damn it SUN had so many superstars yet it's Oracle
that survived.

~~~
mlacks
Brendan and Bryan in particular seem to be doing really well.

Brendan is senior performance architect at Netflix
[http://www.brendangregg.com/bio.html](http://www.brendangregg.com/bio.html)

Bryan is the CTO at Joyent, a cloud computing pioneer recently acquired by
Samsung. [https://www.joyent.com/about/management/bryan-
cantrill](https://www.joyent.com/about/management/bryan-cantrill)

~~~
sulam
As someone who is contemplating what I do after I am done with a full time,
“9-5” job, it’s amusing how much we as a culture equate “doing well” with a
job at a company.

Not a criticism of your statement, just an opportunity to point that out.

------
_trampeltier
It seems almost common to break HDDs just while testing the firealarm. There
are several incidents after a quick search. The latest big one was maybe this
April in the Nasdaq's Scandinavian Data Center.

[https://m.slashdot.org/story/339921](https://m.slashdot.org/story/339921)

~~~
DyslexicAtheist
nice paper on this:

Blue Note: How Intentional Acoustic Interference Damages Availability and
Integrity in Hard Disk Drives and Operating Systems [pdf]
[https://spqr.eecs.umich.edu/papers/bolton-blue-note-
IEEESSP-...](https://spqr.eecs.umich.edu/papers/bolton-blue-note-
IEEESSP-2018.pdf)

------
bjg
The making of video is arguably more interesting:
[https://youtu.be/lMPozJFC8g0](https://youtu.be/lMPozJFC8g0)

------
nisa
So, it's 2018 - this was filmed 2008 on Solaris. Where is this UI for Linux?
How can I reproduce the latency graphs in realtime on Linux?

~~~
kev009
The UIs were part of Fishworks which became the ZFS Storage Appliance. A more
fair question would be, why hasn't your storage appliance vendor (like Pure
Storage, EMC or whatever) implemented the UIs (and maybe they have?).

As others point out dynamic tracing facilities should be there with bcc but to
be honest you shouldn't even need dynamic tracing to create those graphs, just
disk I/O queues, service time and throughput. I know how to get that easily on
FreeBSD from libgeom(3), I'm sure it's not too hard on Linux either in one of
the pseudo-filesystems.

[libgeom(3)]
[https://www.freebsd.org/cgi/man.cgi?query=libgeom&sektion=3](https://www.freebsd.org/cgi/man.cgi?query=libgeom&sektion=3)

------
baruch
Did you know that your HDDs performance may drop if you have too many
correctable ecc errors?

We had servers that started an alert and a buzzer when ecc errors exceeded
some internal threshold. The buzzer was making the chassis vibrate at the
disks harmonious frequency and the performance dropped 30%.

After that I made every new hire to my team watch this video.

Now I deal almost exclusively with SSDs and this specific failure mode is no
longer an issue.

------
bcaa7f3a8bbc
related:
[https://news.ycombinator.com/item?id=17196238](https://news.ycombinator.com/item?id=17196238)

------
chris_wot
"Don't shout at your JBODs". Not something I'd ever considered, but good to
know!

------
7373737373
Use harddisks as microphones? Nice!

~~~
anonu
Thats funny you say that... there's a project I came across on the interwebs a
few years back (maybe it was on HN?) about a guy who turned old disks into
speakers by modulating the read head to generate different frequencies.

~~~
7373737373
There's also [https://github.com/fulldecent/system-bus-
radio](https://github.com/fulldecent/system-bus-radio) Sidechannels
everywhere!

------
raghavkhanna
tl;dr, Vibration due to shouting causes measurable impact on spinning disk
performance, increasing io latency. I guess this wouldn't apply to SSD's
though?

~~~
everyone
Are datacentres transitioning to SSDs ? Or have they already?

~~~
yjftsjthsd-h
SSDs are used in datacenters, though in my experience they haven't completely
replaced spinning rust. Typically I see flash as a (large) cache layer or for
more important stuff, with spinning rust for bulk storage.

------
mlosapio
Those were the days....

~~~
Annatar
Those were the days indeed.

