
An Inside Look at Facebook’s Approach to Automation and Human Work - riqbal
https://hbr.org/2015/06/an-inside-look-at-facebooks-approach-to-automation-and-human-work
======
fishnchips
I would not say that FB compares favourably to Google in terms of infra, and I
worked for both, in infra. Culture is way more relaxed though, especially if
you're not into cults.

~~~
ProAm
Can you speak more about this? Id be interested in some of the details (you
dont have to throw either under the bus, just curious about the nitty gritty
stuff)

~~~
nickpsecurity
I don't work for either but I do follow their projects a bit. I'd say you can
start by comparing things like Hadoop or HDFS to things like Google File
System and their Spanner-based F1 RDBMS (strong consistency + NoSQL scale).
There's clearly a difference in talent or applying it to solve toughest
problems at least. Both companies try to work around problems too hard to
solve but Google seems to solve the hard one's more often.

~~~
fishnchips
Pretty much that. FB has a strong culture of trying to use FOSS 1st, no matter
what. The problem is that at the scale FB operates this simply produces 2nd
rate solutions with a ton of duct tape. If you're into solving _hard_ problems
Google is a way better place to be. In FB it feels like you're solving _a lot_
of problems so it kind of feels way more productive but you quickly realise
that most of these are obvious self-inflicted wounds. There just isn't much
political will to invest in large non-customer-facing projects (like Google's
GFS and its successor, BigTable, Spanner etc.) and perhaps talent is spread a
bit too thin.

That being said Google inside too often feels like a cult and I don't miss
hearing the word 'googliness' at least 10 times a day.

~~~
walterbell
_> There just isn't much political will to invest in large non-customer-facing
projects_

How about OpenCompute? It was promoted by FB (even if inspired by Google) and
had so much impact on the industry that even HP is now having FoxConn build
HP-branded OpenCompute-style servers.

~~~
eitally
My company (Newisys, under Sanmina) helped with hardware refinements because
they were having cooling/airflow problems. They then contracted with Compal to
manufacture the servers and then us to do system assembly. Compal's QC was
horrifically bad -- we had about a 10% failure rate of servers they sent us.
I'm happy OpenCompute is starting to receive traction, but it's not because FB
is awesome at hardware. They've gotten a ton of both raw engineering and also
DFM assistance from the pros.

~~~
nickpsecurity
I'm very grateful for you confirming my intuition on how they're doing
hardware. It's so specialist and difficult I figured they had to be
outsourcing most of it to pro's. Not a negative on their part, though, as it's
what I'd do as well. There's just some holdovers in dicussions that think
these companies are all doing (rather than funding) hardware development.

EDIT to add: The Horus product is interesting. Formerly, I only knew about the
NUMAscale for AMD NUMA. Too bad competition rarely survives in this segment.
Things were more interesting in the MPP and DSM days.

------
nickpsecurity
The one technician for 25,000 servers part stands out the most to me. That's
quite an achievement. Seems like they're ahead of supercomputers in management
efficiency. Another along these lines is AOL's "lights out" datacenter [1]
that they ignore for long periods of time except for efficient, batch jobs of
maintenance.

[1] [http://www.zdnet.com/article/aol-shrinks-its-lights-out-
data...](http://www.zdnet.com/article/aol-shrinks-its-lights-out-data-
center/?_escaped_fragment_=#)!

------
akeck
My intuition is that there's a Moore's Law-like labor rule for IT labor due to
its high cost and the explosion of computing power:

"Every 18-24, an organization must be able to manage an order of magnitude
more servers (virtual and physical) with the same number of staff."

------
gumby
Unusually for the HBR there is a lot of interesting stuff in there. For
example having the compiler team mixed in with the front end development team.

~~~
tomnipotent
The compiler team wasn't mixed in. The front end team developed a compiler
within its own team over a multi-year period - this was in response to the
question "Do you still separate the teams working on innovations from the
teams maintaining today’s core business?"

~~~
gumby
That's what I meant. Sorry I wasn't clear.

------
drussell
Those having trouble reading, try copying the title into a web search and
going direct from there and/or using incognito mode.

------
r721
Mirror: [https://archive.is/57N99](https://archive.is/57N99)

------
s_kilk
The paywall is popping up with a message "You've hit your limit of 5 free
articles as an anonymous user this month.", despite the fact I haven't been
reading anything on HBR for as long as I can remember.

~~~
i0nutzb
Incognito mode does the job for me. Have you tried?

------
lnkmails
This blog post is interesting and it's good to see some numbers (one
technician per 25,000 servers). Also, it touches upon one interesting aspect -
"When do you automate a thing?". It would be interesting to know the actual
metrics they use. Shameless plug:
[http://stackstorm.com/](http://stackstorm.com/) is very similar to FBAR and
is helping customers reap the benefits of automation. We share the vision and
we are completely open source. Feedback is appreciated. (I am a full time
developer with StackStorm.)

~~~
epowell2015
Folks interested in this kind of approach may want to join the Event Driven
Automation meet-up in the Bay Area. Facebook is discussing FBAR and hosting an
upcoming one (already "sold out"). LinkedIn is speaking at the next one (in SF
6/18) [http://www.meetup.com/Auto-Remediation-and-Event-Driven-
Auto...](http://www.meetup.com/Auto-Remediation-and-Event-Driven-Automation/)

------
dingdingdang
Someone, OP maybe, post a mirror please.

