
How Facebook Ships Code - atularora
http://framethink.wordpress.com/2011/01/17/how-facebook-ships-code/
======
ajsharp
“most engineers are capable of writing bug-free code. it’s just that they
don’t have an incentive to do so at most companies. when there’s a QA
department, it’s easy to just throw it over to them to find the errors.”

Bull. Shit. This one line makes me think that this entire post is completely
hear-say. A 500 person engineering team that writes bug-free code on a system
as a large as facebook? Give me a break.

~~~
krobertson
Agreed. Anyone who works with FB apps/apis surely notices their lack of QA
leads to frequent and often times routine breakage.

~~~
lukeschlather
I have my Facebook set on Spanish to keep my Spanish language neurons firing
even when I'm surrounded by Gringos, and their localization definitely has
frequent bugs. There's been some issues for months with the birthday wall post
aggregation they've been testing in production. I've frequently seen stuff
like "So-and-so has written on so-and-so's muro for their birthday."

So yeah, lack of QA => lots of bugs.

~~~
sidupadhyay
Facebook's language translations are actually crowdsourced. It's funny that so
many people would approve errors like this. There is a brief blog post about
it: <http://blog.facebook.com/blog.php?post=10056937130>

The whole system is pretty interesting. They maintain a mapping of every token
on the site to every language and only user translators verify portions.

------
pak
So about 1000 people have unfettered access to all of the live data, but there
are absolutely no safeguards against them modifying it, copying it elsewhere,
or peeking at it? That's terrifying. I'm under the impression that Google has
more a stringent philosophy about this, and even then, one of the few SRE's
that they do give broad access to was caught stalking teens via GMail data.
Imagine the range of bad behavior an FB engineer can get away with after just
4 weeks of "Boot Camp".

Part of the reason I never made a real FB account was my general feeling in
2005 that securing a MySQL/PHP site against _internal_ abuse takes incredible
effort, and they wouldn't ever bother to do it. According to this article,
that's likely true. And the "everybody can modify anything anytime" philosophy
with little QA explains why, after 6 years, FB is still a gaping maw of
security holes.

~~~
holman
_after boot camp, all engineers get access to live DB (comes with standard
lecture about “with great power comes great responsibility” and a clear list
of “fire-able offenses”, e.g., sharing private user data)_

This is through a third-party, so I'm not shaking the finger at Facebook
outright yet, but the wording of that — _sharing private user data_ — is kind
of frightening. What about merely _accessing private user data_? Like you
mentioned, I know Google is super-paranoid about even simple data access.

And now someone from Facebook chimes in and replies to this and talks more
about their developer privacy safeguards and makes us all feel better. Go!

~~~
pollockmania
> And now someone from Facebook chimes in and replies to this and talks more
> about their developer privacy safeguards and makes us all feel better. Go!

Former Facebook dev here. That's all I have to say in response.

~~~
joe_the_user
Call me slow, but can someone explain to me what the above post mean?

~~~
Charuru
Since there was no denial we can reasonably assume there are no stringent
safeguards against engineers accessing user data.

------
kenjackson
I'm really surprised that all engineers have free access to the Facebook
database. That's really scary. I know at other organizations, with much less
sensitive data, this stuff was under lock and key.

I have a hard time believing many aren't abusing this, just out of human
nature ("did she mention me to any of her friends?").

~~~
adrianbye
what happens when one of those 1000 engineers decides to leak the FB database
to wikileaks?

~~~
prayag
Wikileaks exercises caution in what it leaks. I highly doubt wikileaks would
leak personal information of millions of users. Their aim is to open up
governments and public agencies, not people's personal lives.

~~~
hugh3
That was their story last week. This week their story is "We're going to leak
a bunch of folks' Swiss bank account details because some of them might be
avoiding taxes!"

~~~
ebaysucks
Yeah, that story confirms my suspicion that Assange is anti-authoritarian
rather than just anti-coercion.

I think he'll be losing some of his libertarian fans if he publishes
information of people who simply want to be more free.

------
ramanujam
Some inaccuracies pointed out by another FB engineer
[http://www.reddit.com/r/programming/comments/f3u0n/how_faceb...](http://www.reddit.com/r/programming/comments/f3u0n/how_facebook_ships_code/)

~~~
neilk
Copying the main points here, from Reddit user "epriest":

 _The article is has a few inaccuracies:_

 _There is mandatory code review for all changes (i.e., by one or more
engineers). I think the article is just saying that Zuck doesn't look at every
change personally._

 _People do not get called out for introducing bugs. They only get called out
if they ask for changes to go out with the release but aren't around to
support them in case something goes wrong (and haven't found someone to cover
for you)._

 _Getting blamed will NOT get you fired. We are extremely forgiving in this
respect, and most of the senior engineers have pushed at least one horrible
thing, myself included. As far as I know, no one has ever been fired for
making mistakes of this nature._

 _We have automated testing, including "push-blocking" tests which must pass
before the release goes out._

 _We absolutely do not believe "most engineers are capable of writing bug-free
code", much less that this is a reasonable notion to base a business upon. If
this is a real quote, it comes from a very junior engineer. It is empirically
untrue._

 _The nine push phases are not concentric. There are three concentric phases
(p1 = internal release, p2 = small external release, p3 = full external
release). The other six phases are auxiliary tiers like our internal tools,
video upload hosts, etc._

~~~
pak
OK, so according to him there's code review and some automated tests.

But that still doesn't address wide open access to the live DB by brand-new
devs, and the complete disregard for having a dedicated QA team.

Why QA? I would think FB engineers are more busy doing real work at work than
fooling around on their own site (heh, the irony) and finding out that Junior
Dev 39's change to the thingamabob broke chat for the umpteenth time in IE7.
And as many people have pointed out here, they are a _platform_ for many other
businesses now--there's no dogfooding that--and the API stinks in terms of
stability and documentation.

~~~
aamar
It's easy to dogfood an API. Not saying that FB does, but it's easy to.

~~~
pak
It depends. If your API was built underneath your own site, then it's easy.
FB's API doesn't seem to actually support anything on the main site, e.g., I
doubt their own "apps" actually run on the FB Apps API. And there's no
incentive for any of the engineers to build them like that, because they have
such lateral access to internal code and data.

------
ernestipark
This seems to fit how I view Facebook as a whole. I find some of their
technology extremely impressive such as their blazing fast search or HipHop.
However, a lot of their user experience is extremely flawed and there is a big
lack of consistency and intuitiveness. Examples: clicking your friend requests
on the right side of the home page pops something up that tells you to click
somewhere else, events are no longer notified so you have to manually check to
see if you received new ones, the distinction between adding a page to your
interests versus Like'ing it is not well defined, or if it is, many users are
unaware, making an event for a Facebook Page is very convoluted... etc etc. I
feel that issues like these are the reason why product managers and designers
exist. I think Facebook is starting to really fall behind in this regard as
more and more features start to clutter and complicate the site.

------
sanj
This is very close to how we develop at TripAdvisor, though we have a suite of
tests that get run during a branch merge and code review (of at least one more
engineer) of anything going into the livesite code base.

We also have very small QA team. They work to make sure that the buttons that
make us money aren't screwed up.

The key to this sort of uber-agile development is that you have very, very
talented engineers, a release process flexible enough to deal with errors, and
a management that buys into moving so fast that mistakes will happen.

I have to say that it is simultaneously exhilarating, humbling and a little
terrifying to work like this. Luckily, mostly the first two.

~~~
rhizome
I know all of this certainly has worked and does work for you, but the first
thing I thought about when reading "small QA team" and "the key...talented
engineers" is that QA is not very important to the company. A QA team that
only checks the money buttons tells me that traditional QA has been
incorporated into the developer positions themselves, which creates a conflict
of interest.

Couching my conclusion in the topic of this post, it would seem that Facebook
gives extraordinary freedom to developers, but also hangs everything around
the developers neck. Screwing up in this area sounds pretty fatal, and not an
aspect of a company I would want to work for. I'm kinda old, though.

~~~
brown9-2
_the first thing I thought about when reading "small QA team" and "the
key...talented engineers" is that QA is not very important to the company_

Couldn't the other corollary of this be that the company/development team has
invested heavily in automated testing?

------
intranation
Sounds like utter chaos for any seasoned professional. No QA? No real product
managers (or at least none with any teeth)? Design as optional resource? Ops
managing user engagement metrics? Where's the security audit?

Scary, even for a developer like myself.

~~~
brown9-2
Perhaps this is a good example of how much of that professional "seasoning" is
actually unnecessary or gets in the way of building successful products and
high-caliber responsible teams, i.e. "throwing over the wall to QA" etc.

edit: Can I ask why this is being downvoted? Because I am disagreeing with the
parent poster? I don't think I am raising poor points or somehow not
contributing to the discussion. Facebook seems to not do a lot of "standard"
things because they view them as counter to their goals of building their
product. I am merely stating that perhaps there is a lesson in there about the
value of what some of the convential wisdom in our industry dictates.

~~~
kenjackson
Upmodded... with that said, I do think that this is very industry dependent.
They are a free product that people don't necessarily "rely" on. If status
messages were severely delayed or dropped, it wouldn't be a huge deal
(assuming they fixed it when noticed).

There's very little that they can't fix after deployed, and they have no SLA
with consumers (advertisers may be different). In essence they have 500
million testers.

This is a bit harder to do with a database or compiler.

~~~
kemiller
App developers and other API users do in fact rely on them. Their
unreliability is part of what's convincing many sites to rethink that.

------
cduruk
Is the author of this piece an engineer working at Facebook? This part makes
me think not:

>I’m fascinated by the way Facebook operates. It’s a very unique environment,
not easily replicated (nor would their system work for all companies, even if
they tried). These are notes gathered from talking with many friends at
Facebook about how the company develops and release software.

So I'd take whatever is written here with a grain of salt. My communication
with friends working at Facebook yielded similar thoughts but nothing that
comes to what's written that implies a callous recklessness. I know for a fact
that they have some code-review tools and blocking tests.

Anyway, my point is that the author doesn't seem to be embedded too deeply in
the engineering at Facebook and his notes are, while not outright false,
definitely misleading.

------
flyt
More info directly from a Facebook Engineer about their testing process:
[http://www.quora.com/What-kind-of-automated-testing-does-
Fac...](http://www.quora.com/What-kind-of-automated-testing-does-Facebook-
do/answer/Steven-Grimm)

------
theletterd
It seems, from the article at least, that the're very much a culture of blame,
which I find quite surprising. I've never been a fan of the practices of
practices such as making people wear a hat if they break the build and the
like. It seems counter-productive to getting things built and shipped, and
just seems to make an unhappy workplace (in my mind, at least)

~~~
epriest
This part of the article is simply untrue. The culture is extremely forgiving
of mistakes.

(I don't speak for my employer, Facebook.)

------
patrickk
"If lots of [employees] are flocking to a new business unit, that's a good
sign that the opportunity is a good one. . . . If a business unit can't
attract people very easily, that's a good sign that it's a business Enron
shouldn't be in."

\- Jeff Skilling, former president of Enron

<http://www.gladwell.com/2002/2002_07_22_a_talent.htm>

Contrast that with the Facebook approach:

 _"Resourcing for projects is purely voluntary. -a PM lobbies group of
engineers, tries to get them excited about their ideas.

-Engineers decide which ones sound interesting to work on.

-Engineer talks to their manager, says “I’d like to work on these 5 things this week.”

-Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first.

-Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves._"

Good god. Does this strike anyone else as disturbing? Surely every single
piece of work being taken on should have the USERS needs and concerns as top
priority and not sexy stuff that can attract sufficient engineering interest?

What if there's important problems that are really bothering lots of users and
a PM can't get anybody interested (or no-one decides to take on the problem?)

Here's a complaint from a guy about maintaining a personal page and fan page:

<http://www.stevepavlina.com/blog/2011/01/leaving-facebook/>

Some quotes from that piece:

 _"As a programmer myself, I can’t fathom that it would take much technical
and design effort to address these issues, and Facebook is flooded with
complaints from users begging them to fix these headaches. From my perspective
as a Facebook user with a very active personal page and fan page, I can’t help
but get the impression that Facebook deliberately wants to make some basic
admin tasks (like blocking spammers) difficult or impossible in order to
compel you to spend more time on the site. There doesn’t seem to be any other
logical reason for these glaring design flaws that I can comprehend, other
than pure incompetence, and based on their success in other areas, it seems
more likely that these choices are deliberate."_

" _Surely someone on their team is aware of all the complaints and requests to
fix the broken elements. So why do they seem to ignore what appear to be such
glaring (and fixable) problems?_ "

" _I thought that Facebook would be an interesting place to share
inspirational messages and build more community around growth-oriented people.
But the current implementation of Facebook can’t handle the way I’ve been
trying to use it without creating more headaches than it’s worth, and their
momentum appears to be headed in the wrong direction for me to expect that
these problems would be fixed anytime soon._ "

 _"So I’ve crossed the threshold where Facebook’s value isn’t worth the hassle
to use it. I concluded that the best choice was to simply drop the service
altogether and invest my time elsewhere."_

Who will take on these issues?

Again from the Malcom Gladwell article linked above:

" _You might expect a C.E.O. to say that if a business unit can't attract
customers very easily that's a good sign it's a business the company shouldn't
be in. A company's business is supposed to be shaped in the direction that its
managers find most profitable. But at Enron the needs of the customers and the
shareholders were secondary to the needs of its stars._ "

Facebook should wake up in my opinion. They did really well to reach 600m
users (or whatever the figure is now) but if they want to stay there they
should get their priorities straight.

EDIT: Grammar

~~~
brown9-2
I think the comparison here to Enron is a bit of a stretch.

From what I know of Google, their internal staffing of projects is done much
the same way: you're assigned to certain teams when you first join the
company, but after that the various Product Owners try to recruit and
advertise within the company for engineers to join their team.

When I was interviewing there I recall seeing flyers posted in common areas
advertising different projects that were in need of engineers to come join
them ("Interested in _____? Google ___ could use your help!").

~~~
patrickk
_"I think the comparison here to Enron is a bit of a stretch."_

Could you clarify why this is? I'm not trying to imply anything, certainly not
that Facebook is Enron, just that (to me) they have similar attitudes to
assigning staff to work duties. When I read the Facebook quote, I immediately
thought of Skilling's quote.

It seems to me and many others here on HN that I've read over the last few
months have problems with the way Facebook operate e.g. those who need to use
their API to integrate with their own service, privacy concerns, spam, etc;
and Facebook doesn't seem to address these issues at all. That was my ultimate
point - how they address with these issues, and could the fact they are not
dealt with in a timely manner actually be a symptom of letting engineers pick
their own work?

~~~
lbrandy
_> Could you clarify why this is? I'm not trying to imply anything, certainly
not that Facebook is Enron, just that (to me) they have similar attitudes to
assigning staff to work duties. When I read the Facebook quote, I immediately
thought of Skilling's quote._

Let's see how I can put this. What you've just done is the business equivalent
of Godwin's law. It would be like comparing US customs to the Nazis because,
you know, both want to look at your papers.

What makes Enron Enron isn't how they did staffing, it's how they cooked their
books and defrauded people. Their name is synonymous with fraud and
corruption, and so invoking them in conversation about similar staffing
practices seems... misleading.

~~~
Supermighty
Cooking the books was the end result of an over-arching corporate culture.
Part of which started with how they did staffing.

If facebook is starting to go down this same road they might arrive at the
same end.

------
CWIZO
What does it mean to be "publicly shamed" in this context?

------
EGreg
Yeah, I wouldn't want to run my company like this:

re: surprise at lack of QA or automated unit tests — “most engineers are
capable of writing bug-free code. it’s just that they don’t have an incentive
to do so at most companies. when there’s a QA department, it’s easy to just
throw it over to them to find the errors.”

It does explain how facebook ships with bad documentation etc. -- well, now we
have enough knowledge to know how to compete with it :)

Kasparov often remarked how a good process is more important than the actual
participants. An average human and pretty good computer with a great system
won the championship against great computers and against great grandmasters.

Google believes in this, and their products are very well engineered, with
full documentation, videos etc. (although admittedly, many haven't taken off).
Yahoo definitely understands this. But these are the same companies that are
losing to facebook because of social.

If google and yahoo understood the dynamics of social, we would all be better
off.

------
tommi
Funny how this article has 33 main bullets points out of which 13 are or
contain corrections.

------
bootload
_"... What do you think? Would “developer-driven culture” work at your
company? ..."_

Hasn't everyone read, _"Microsoft Secrets"_ ~
[http://www.amazon.com/MICROSOFT-SECRETS-Powerful-Software-
Te...](http://www.amazon.com/MICROSOFT-SECRETS-Powerful-Software-
Technology/dp/0028740483)

...because what the author describes is pretty much the MS dev process (cf
_CH4 Defining products and development processes_ ). Reading JOS, _"How to be
a program manager"_ also shows how MS PM's worked in conjunction with
developers ~ <http://www.joelonsoftware.com/items/2009/03/09.html> MS sure
knows/knew a thing or two about organising large groups of people to produce
software & output product in a corporate setting.

------
krummas
The key point for me in this article was:

"resourcing for projects is purely voluntary."

As someone who works for an online gambling company where we are not even
allowed to use the product we are building (legal/trust issues i guess), it
would _rock_ to be able to have this kind of impact on features

------
anthony_franco
"no QA at all, zero"

As someone who develops on top of the Facebook Platform, I'm not surprised.
Huge, obvious bugs that affect many applications are released far too often.

------
jkuria
"very engineering driven culture. 'product managers are essentially useless
here.' is a quote from an engineer. engineers can modify specs mid-process,
re-order work projects, and inject new feature ideas anytime. during monthly
cross-team meetings, the engineers are the ones who present progress reports.
product marketing and product management attend these meetings, but if they
are particularly outspoken, there is actually feedback to the leadership that
'product spoke too much at the last meeting.' they really want engineers to
publicly own products and be the main point of contact for the things they
built."

This sounds like Mark Zuckerberg living his 'revenge of the nerds' dream in
his Facebook nirvana! It certainly doesn't sound like good management
practice!

~~~
maxawaytoolong
_It certainly doesn't sound like good management practice!_

Why not? I worked for silicon valley startups for 15 years and not once did I
meet a "product manager" who did anything worthwhile.

~~~
pzb
Product managers are very useful when you are not your own customer. Good
product managers are a proxy for your customers, providing you information on
the customer wants and needs. Good product managers work with the engineers to
prioritize projects; it is not a dictator position, but does help ensure that
the less sexy projects get done as well as the uber-cool sexy projects.

------
clojurerocks
Serious question. Why doesnt Facebook do any engineering talks anymore? Some
of the coolest talks on infrstructure from the past few years have been from
FB. And yet the only thing i can remember really hearing about their stack
recently is that they were using HBase. Which really came off as more about
marketing and hype about some products they were working on then truly geeky
stuff thats come out from FB in the past. Anybody know? I wonder if its
because Fb is no longer a startup? Which might the reason why we never hear
about Youtube or Twitter anymore either.

Oh my other comment was just meant in jest. But thanks for downvoting it and
killing all the worthless and meaningless HN karma points i had accrued!

~~~
daveman692
<http://www.facebook.com/Engineering?v=app_2392950137>

------
siddhant
Multiple mentions of "public shaming". Doesn't seem so good. A good engineer
is anyway going to have a sense of shame inside when he/she screws up. I
really don't think there's any need for "public shaming".

~~~
gaius
You don't get shamed for a bug in your code - you get shamed if you aren't
willing to go on-call to support your own code. That IMHO is a very healthy
attitude.

------
InclinedPlane
Estimated Joel Test score for facebook: about 5 out of 12.

<http://www.joelonsoftware.com/articles/fog0000000043.html>

~~~
nikolayav
As a Facebook engineer, I'd say the score is 10 or 11, depending on your
interpretation of the tools item.

------
clojurerocks
The more i read about facebook engineering the less impressed i am. It used to
be interesting to hear about their engineering challenges. Now its just scary
to hear anything about them. And i use it as a rulebook of how not to do youre
infrastructure. Im amazed facebook works at all frankly. I also think its one
of the ugliest sites ive ever used. Its what 5 years old now and they still
dont even have some basic functionality on it. Plus as another person said its
often broken.

------
kschua
"all engineers go through 4 to 6 week “Boot Camp” training where they learn
the Facebook system by fixing bugs"

I really like this part. Many new staffs prefer to come in and do the "sexy"
enhancements instead of the "mundane" support. This leads to their lack of
understanding of the system and poor design. I had always believe that new
starters should do support work for a while to gain an understanding of the
overall system.

------
didip
Given the current sad state of Facebook Platform, it must be because no one is
working on it.

Or it is labeled "unsexy" to work on it.

~~~
bjclark
Well, the lack of QA certainly explains a lot.

------
Aloisius
It sounds like they've institutionalized the herding of cats as a project
management system.

------
kemiller
Shocked... shocked I am that that buggy pile of crap has no QA.

------
kunjaan
I would like to know what are the other items in the 'clear list of “fire-able
offenses” , e.g., sharing private user data)"

------
keyle
This just looks like some sort of marketing propaganda to me.

