

Airbnb Tech Talk: Evan Priestley of Facebook - Move Fast and Break Things - clizzin
https://www.airbnb.com/meetups/25z4mzn4a-tech-talk-evan-priestley
Move Fast &#38; Break Things 
A practical set of step-by-step recipes for reproducing some of Facebook's biggest mistakes (in security, web stacks, polytheism, deployment, algorithms, photo faxing and other areas) delivered by an engineer who helped make them.&#60;p&#62;About Evan 
Evan Priestley worked as an infrastructure engineer at Facebook. He inflicted substantial and lasting harm, amassing a mountain of technical debt which the company will be paying down for decades to come.&#60;p&#62;Livestream 
If you are not able to join us in person, we'll be streaming Evan's talk at 7pm. Please visit the meetup page to get the link for the livestream. A recording will be posted to our Tech Talk page as well.&#60;p&#62;Dinner, prepared by Airbnb's Executive Chef, will be served at 6pm.
======
zaidf
If an engineer at Bank of America said something to this tune, it'd have
tremendous negative consequence as it should. Facebook is reaching a point
where breaking things should not be okay simply because it opens up the
possibility of breaking privacy settings and causing great damage.

Lots of people have pretty sensitive data within facebook with very granular
privacy settings. When I read this attitude from facebook, it scares the shit
out of me.

I'd love to hear more about why I shouldn't fear that their "Show this post to
only x group of people" feature will break and result in everyone being able
to see all my posts. What are your testing procedures for that and at what
point are those kind of bugs caught?

~~~
antics
Facebook is, by a _huge_ margin, the buggiest release software that I use
every day. When I look hard, I notice at least a dozen UI bugs a week. Yet,
Facebook is hugely successful -- so successful, in fact, that it's hard for me
to think of ways in which it could be more successful. Your comment implies
that there is some threshold that they will pass, at which point they will be
forced to adopt more stable sensibilities. There are 1 billion active users
[1], so I ask: if they haven't gotten to this point, when exactly will they?

I don't believe the lack of quality in Facebook code poses an existential
threat to Facebook, and I don't believe it ever will:

(1) Facebook code is not Google code, but it is not _bad_ code, it's just not
needlessly good code. If Google is consistently buggy, anyone can switch to
Bing. On the other hand, switching from Facebook is really hard. Their core
services are not easily replaceable. People are willing to put up with more.
Note that this is even true in the case of businesses: Facebook is _horrible_
to develop against, but developers don't have a choice.

As the network grows, this cost will be higher, which means Facebook can throw
its weight around more, not less. This is exactly opposite of the situation
you predict. The only thing that can really reverse this is if there is a
viable competitor, but the increased growth of Facebook makes this
increasingly unlikely. It is clear, in any event, that G+ is not quite there.

(2) By maintaining a lower bar, Facebook actually has an advantage. It is a
website, and can fix its product at will. Code can be deployed quickly and
rolled back accordingly, making failure cheap. This means they can concentrate
on things that really matter like user acquisition.

(3) Facebook does _not_ run nuclear reactors or bank systems. There is a high
incentive to switch from buggy bank systems. There is little incentive to
switch from buggy social network websites.

footnotes:

[1] whatever that means

~~~
zaidf
_(3) Facebook does not run nuclear reactors or bank systems. There is a high
incentive to switch from buggy bank systems. There is little incentive to
switch from buggy social network websites._

I'm arguing that facebook _wants_ to run something even more important than a
banking system. For example, Facebook wants me to share posts that are only
visible to me. Such a post could contain secrets that could have huge
consequences on my life and life of others' if they became public. It is
pretty clear from facebook's history that each year, they want you to share
_more_ of such private data.

They started with you being able to share your drunk party pics. They are
headed towards actually becoming something like PayPal. Really, facebook
should behave more like a bank than just-another-social-network if it wants
its users remain confident. In this respect, looking at their past success to
as proof that all is well is a mistake because I am certain facebook intends
to be much more than what it is five years from now.

------
casey
Evan is hilarious, check out all the copy on <http://phabricator.org/> for an
example.

However he should probably have been described as ex-Facebook, he left quite a
while ago.

~~~
benjaminwootton
This made me crack up the first time I read it!

'Facebook engineers rave about Phabricator, describing it with glowing terms
like "okay" and "mandatory".'

~~~
garindra
This made me crack up even more : "Written in PHP, so literally anyone can
contribute, even if they have no idea how to program."

------
BadassFractal
This is such a grey are that I think it's close to impossible to make a rule
about how much quality you should sacrifice for the sake of speed. Sometimes
you might get lucky and break something insignificant, sometimes you create
the IceBox Pro fiasco and be remembered at "that company". It's all calculated
risk / gambling.

On the other hand, perhaps someone will argue that all publicity is good
publicity.

------
pbiggar
I honestly think that "Move fast and break things" might be the singularly
most important software engineering mantra of our age.

Things break all the time in any company, but by understanding that frequent
breakages occur, and adapting your development process to that understanding,
you enable fast recovery and ultimately less breakage and greater security in
the long run.

~~~
jessedhillon
This is inane, frankly. There are people who code to a much higher standard
than this. People live and die by the code they write, so they can't accept
"things break all the time ..." and forget about writing correct code.

See for example: your car, your bank (not the website, the backend), the space
shuttle, the Fed, anything important. Some places cannot accept the supposed
fact of frequent outages and failure, and engineer for _that_ reality instead.

I don't know how you're concluding that the hackday coding methodology creates
more secure code, unless you are simply unaware of the industrial standards to
which other fields' programmers are held.

~~~
pbiggar
Of course I wasn't talking about banks and space shuttles, that would be
inane. Some places can't accept outages and failure, and naturally "move fast
and break things" is an unacceptable mantra for those places.

However, few companies are rated on the quality of their code. Rather, most
companies are rated on how well they serve their customers needs, including
the vast majority of companies discussed on Hacker News.

The way to delight your customers is to iterate quickly. The way to achieve
great software is also to iterate quickly. That is not a coincidence.
Willingness to withstand a little bit of breakage allows companies to move
fast. This contributes to an overall _higher_ level of quality, leading not to
frequent outages (Facebook does not have frequent outages and failure), but to
more robust product.

I am aware of how people build software in more traditional fields (my
background is in compilers and static analysis). I am also aware that
traditional software development techniques are being eclipsed by those who
are willing to throw off the shackles of the past. The "hackday coding
methodology" may not be useful to make the code for your car, but the
techniques used to build your car are not useful for building a successful
SaaS product.

~~~
jessedhillon
Is it still suitable to code loosely when people use your trivial SaaS as if
it were a vital component of their life? And if you encourage people to do so,
isn't it irresponsible to keep applying such a methodology?

~~~
pbiggar
I think you're missing my point. It's not coding loosely, it's optimizing
moving quickly. And by moving quickly you can actually have a safer and more
secure site than by, say, only releasing code every 3 months or something.

~~~
amirmc
This confuses the act of shipping code frequently with _what the code actually
does_. Both of those are important but don't really have any bearing on each
other.

fwiw, I disagree strongly that moving fast and breaking things _"might be the
singularly most important software engineering mantra of our age."_ Frankly,
this terrifies me. You already qualify it by carving out banks, space shuttles
(and presumably medical devices, transportation, etc). I suspect we could
discuss a list of things where it doesn't quite apply and in the end I would
have to add _"...for products that aren't critical to your users."_ to the end
of your sentence.

~~~
pbiggar
"Move fast and break things" is about moving fast, and only mentions "break
things" to indicate that its OK to break things. I agree that what the code
does is orthogonal.

I would carve out space shuttles, etc, as different, but they already are
carved out: they use special subsets of C with no memory allocation, run
extensive static analysis and proof software, because those things literally
cannot go wrong.

Anything that doesn't use that sort of thing has already committed to the idea
that it won't be foolproof. If you code in a scripting language, you are
already writing software which you implicitly acknowledge can go wrong.
Software has bugs!

~~~
amirmc
I guess my issue is specifically with 'break things' as it overgeneralises the
idea of experimentation (which is something I do agree with). The impression
that 'move fast, break things' leaves me with is that it's ok to break
_anything_ and that it's also (implicitly) not a big deal if they _stay
broken_ or somehow hurt a user (e.g by exposing, even temporarily, what was
once private).

Even though it's a wonderfully pithy 4 words, people can interpret them in
many different ways, most of which I'd argue are not good for the end-user.

 _"... you are already writing software which you implicitly acknowledge can
go wrong. Software has bugs!"_

Agreed, but shouldn't the _intent_ be to create things in a way that tries to
reduce the likelihood of damage while still allowing plenty of room for
experimentation? i.e take some care that what you're about to do/push isn't
going to do harm? If I follow the 'break things' mantra, I don't need to care.
That's what bothers me. It's so easy to hide behind when something goes wrong.

~~~
pbiggar
From what I've learnt of Facebook's attitude, it's more that you won't be
fired if you take the site down (at least, the first time). I don't believe
that they don't care when you break things, and that wasn't what I was
advocating.

------
fatjokes
Yup, Facebook's broken for me right now.

------
realrocker
I wish I could know, whether he tried moving slow and make things less
brittle.

