
Move Fast and Break Nothing - bpierre
http://zachholman.com/talk/move-fast-break-nothing/
======
mehrdada
The problem with the new motto is that it is uncontroversial. As Zuck himself
once mentioned at Startup School, corporate mottos are valuable only if they
are controversial, that is, "be honest" is not a valuable motto, because it is
obvious and uncontroversial, whereas "Move Fast and Break Things" is not so:
it says something about prioritization that not everyone agrees with, and
thereby communicates where the company stands on this issue. Everyone wants to
move fast as long as nothing gets broken. I agree that it is possible to
leverage technology to move faster while keeping things less broken, but it
does not change the fundamental dynamic that at some point, you will have to
trade _some_ stability for speed, and the new motto does not take a stance on
which one it prefers at all.

~~~
pm
Move Fast (and Break Things) is the unwritten motto of every mediocre dev shop
out there (breaking things being an unintended side effect of moving fast).
It's not controversial.

~~~
nickik
Its also the motto of many good companys. At least in my mind, move fast and
break things, means that you iterate quickly and move into testing fast, to
find the errors there.

This works of course better with exerimental APIs compared to things you have
to put on CD and ship.

~~~
seanp2k2
And in the "ships CDs" camp, we have Adobe and Microsoft, long held as
bastions of robust, stable, well-tested software.

~~~
godzilla82
There are other camps too. Like software consulting firms building software
for large enterprises like banks, insurance companies etc. The consequences of
breaking things are all written down in very exact legal terms. It is not at
all fun when a regression bug shows up.

------
aresant
I thoroughly enjoyed the visual design and layout of this site.

That he took the deck and turned it into vibrant and unique feeling content
made it feel more special than a normal blog post.

Not sure where I'm going with this but I liked the balance of medium.com-like-
focus + the design and style content of a NYT special feature.

I'm assessing my reaction from a web analysts perspective, just interesting
how much more engaged I was w/this from the first second than normal.

~~~
jgmmo
It felt similar to _why's stuff Like this 'Nobody Knows Shoes' book
[https://cloud.github.com/downloads/shoes/shoes/nks.pdf](https://cloud.github.com/downloads/shoes/shoes/nks.pdf)

------
tsunamifury
I'm surprised that few seem to understand that Zuckerberg chose a motto with a
deeper meaning than 'breaking products and code'. He meant break the systems,
calcified behaviors, and system-think that ends up killing a company over the
long term.

Move fast, and don't go in predictable directions based on existing corporate
momentum. The only way to disrupt yourself is to 'break things' i.e.
architectures, regular income, etc that may be holding you back from a better
opportunity.

~~~
hawkice
This is some deeply Talmudic reading of a fairly straight-forward description
of engineering deploy cycles. But why stop after just putting those words in
his mouth, when he could be retroactively understood to mean so much more!

Move fast! Run to all meetings! Take typing lessons so you move faster while
coding! Break things! Break your laptop to get a faster one from the company!
Move your laptop fast at the hard surface to break it faster and while moving
faster!

Some may say, if you keep running around with your laptop, you might actually
break it. I call that synergy.

------
kcorbitt
Tangential, but does anyone know what Github uses to match their css and html
classes in the integration suite? I often find that I don't clean up my css as
the free as I should because I'm scared of breaking another page by mistake,
so something that tells me "you're really not using this anywhere anymore"
would be great.

~~~
palcu
I don't know what Github uses, but where I work, we do screenshot testing. We
have PhantomCSS[0], which is a helper on top of CasperJs that visits pages,
takes screenshots and compares them. There are a lot of other tools, Huxley
and Selenium come to my mind, that do the same stuff. Even Facebook[1] tests
this way.

[0]:
[https://github.com/Huddle/PhantomCSS](https://github.com/Huddle/PhantomCSS)
[1]:
[https://www.youtube.com/watch?v=VkTCL6Nqm6Y](https://www.youtube.com/watch?v=VkTCL6Nqm6Y)

------
lifeisstillgood
This post just touches on so many of my personal "yes dammit yes" buttons I am
purring like the kitten in the gif (go have a look)

My new boss just gave a piece of advice that marries up very well with this -
clients want Stability, Performance and Features - in that order. Consumers
want features _Then_ Stability and performance if they like it.

So yes - hurry up and put the talk online please Zach - Itching to listen.

~~~
sanderjd
Woah, I _love_ your boss' advice! It is necessary to know your audience before
you can prioritize.

Also, I don't think consumers want features in the sense of "more", but in the
sense of "key". Maybe it is Key Features, Stability, Performance, More
Features, in that order.

~~~
couchand
In my experience clients and consumers really just care about Feature Lists;
power users maybe lean more towards Key Features. Performance and Stability
are only a concern when they've already become a problem.

~~~
sanderjd
Really? I don't think consumers have shown at all that they care about (or are
even aware of) feature lists. Even facebook is shifting toward a strategy of
cobbling together single-feature apps instead of building one big feature-
heavy platform.

------
Fuzzwah
As always, I enjoyed reading Zach's thoughts.

Spotted one tiny error:

[http://zachholman.com/images/talks/break-
nothing/anpp.jpg](http://zachholman.com/images/talks/break-nothing/anpp.jpg)

annp should be anpp in this image.

~~~
holman
Actually caught this last night and fixed it literally everywhere but the
important place: this image. :) Fixed! Thanks for pointing it out.

------
malkia
... Two code paths (old&new) - I've been doing this to some extent without
trying to get any insight, or make it a practice (of my own first), but often
the old code just stays before deploying the newer (day or two), or before
submission, and then it's killed (or #ifdef-out).

I liked the "empathy" bit though. Recently I've did the UCLA Extension TMP
courses, and one of the lecturers - Jorge Cherbosque, PhD talked exactly about
this and much more - here is the lecture (highly recommended)

[https://www.uclaextension.edu/tmp/Pages/79th/D1.aspx](https://www.uclaextension.edu/tmp/Pages/79th/D1.aspx)

Is there a video behind the talk?

------
aleem
Reading the title, I thought it would be a post about TDD. I don't do TDD, I
wish I had the time and resources and I am moving toward adding a more test-
driven approach. Adding unit tests to my key libs allows me to move fast
without breaking anything (for the key libs, our test suite is stupendously
comprehensive).

Of course, during new feature dev this luxury isn't always available. The next
best thing is to do a phased rollout where our users become our beta testers.
This works really well because users can switch to the non-beta version
through the menu bar very easily.

There are other strategies for moving fast without breaking stuff, but these
two provide easy, big wins.

------
rgarcia
> What do you consider good feedback? How can you promote understanding and
> positive approaches in your criticism of the code? How can you help the
> submitter learn and grow from this scenario? Unfortunately these questions
> don't get asked enough, which creates a self-perpetuating cycle of cynics
> and aggressive discussion.

I'd be really interested to hear how teams codify this. To the extent that
it's possible, I think feedback should be rooted in objective measures, e.g.
styleguide/lint violations, test failures, etc. All too often, though, there
are subjective things that come up: organization of code, interface design,
using framework X instead of framework Y. These are the criticisms that most
often lead to aggressive back and forths.

~~~
blakehaswell
But those are the most important and interesting discussions to have, so I’m
not convinced we should shy away from that sort of feedback.

That said, they’re also the sort of discussions which should often happen
before code is written. Decisions can be changed much more quickly at that
point, and people take it much less personally when you talk about how the
design of something can be improved when they haven’t spent the time to
implement it. :-)

------
hox
Twilio follows a similar model for large scale changes, but at a distributed
service level using a shadow proxy
([https://github.com/twilio/shadow](https://github.com/twilio/shadow)) rather
than having parallel code paths.

------
lifeisstillgood
What interests me in companies similar to GitHub is how the organisational
politics reflects the code (or coding process)

I assume if I knew more about fluid dynamics one could make some decent
correlations between blockages, eddies, and viscosity and middle management
sign offs, manual testing and IM channels.

I guess I am looking for justification that a flat collegiate structure
produces higher quality long term code.And some idea how to persuade people to
adopt such a format - open, free flowing discussion about code and a data
driven "prove it not say it with authority" seems to conflict with
hierarchies.

Or maybe I am just fed up with organisational politics. :-)

If a githuber is on could they comment on how design decisions and business
decisions are taken ?

------
k__
"Move fast with stablity"

That's why small companies have a chance.

The big ones get slow and focus on different things. They have to change their
maxims and their customers don't necessarily like that.

------
lnanek2
Really distasteful slogan. I think every engineer who has worked in corp.
America is used to management asking for mutually exclusive things due to
ignorance.

------
vdm
Comparing the results of requests processed against both the new and old
system is a pattern. [http://www.alwaysagileconsulting.com/application-
pattern-ver...](http://www.alwaysagileconsulting.com/application-pattern-
verify-branch-by-abstraction/)

It was also done in mainframes at the processor instruction level.

------
S_A_P
I think of "move fast and break things" in a different context that most
people(maybe). It doesn't say move fast and SHIP broken things.

To me it speaks to the fact that you shouldnt be afraid to completely change
something if its to improve the product. Eliminate technical debt. Dont code
around a workaround that doesnt always work.

------
hartzler
For me the best line was "You can have the best, most comprehensive test suite
in the world, but tests are still different from production.". Getting the
business logic working is often the easy part of distributed apps in
production environments with real users and unexpected data.

~~~
cpsaltis
While it is true that most tests are different from production, having a solid
test suite can go a long way in moving with startup speed but breaking stuff
as little as possible. We (as in me and a couple of others from our startup)
have written a howto on using Docker and Ansible to build your test suite a
few months ago. Strangely enough, it has almost the same title:

[http://blog.mist.io/post/82383668190/move-fast-and-dont-
brea...](http://blog.mist.io/post/82383668190/move-fast-and-dont-break-things-
testing-with)

------
mcguire
Personally, I think this is the most important piece of the article:

" _...I think the best way to get things done in a company isn 't to bash it
over your employee's heads every few hours, but to instead build an
environment that helps foster those effects._"

------
davidw
This pretty much sums it up:
[http://en.wikipedia.org/wiki/Project_management_triangle#.22...](http://en.wikipedia.org/wiki/Project_management_triangle#.22Pick_any_two.22)

------
RickHull
I agree that breaking a production user's experience is as verboten as it gets
for agile software development. That's why we have massive test suites and CI
infrastructure.

However, you do want to avoid "CI handcuffs", where either because CI is too
brittle or because you're simply integrating too many independent projects
which, in order to push a change requiring coordinating changes elsewhere,
requires massively outsized developer effort in order to push things through
with zero CI breakage. This is more of a problem in CI systems with binary
metrics like PASS/FAIL where every change that doesn't PASS is rejected.

It's too easy to "wedge" a system like this, where project X's change can't go
through without project Y being able to handle the change, and you end up
introducing multiple code paths in both projects on a solely temporary basis
just in order to keep CI green and happy. Rather than having a zero-tolerance
CI failure policy, developers should be allowed to break CI temporarily, so
long as they fix it in a timely manner (within an hour or two). Per-developer
breakage metrics, to the extent they are needed, should not be in terms of
breakage counts but instead breakage durations.

That is, outside of production, it's fine to break stuff and quickly fix it,
so long as you don't leave it broken. The big problems are where domain-siloed
developers break a zero-tolerance policy because it was necessary to relax it
"temporarily", and things stay broken because the policy stays relaxed and
cannot be reinstated without sirens going off. Then restoring the CI policy is
blocked for everyone by the one guy who knows the AIX quirks for debootstrap
or whatever.

Instead, breaking changes should be "allowed" where there is a window to fix
the error and still move forward. Only when the window closes without a fix
should the breaking change be rolled back (automatically). This line of
thinking lends itself to formal, automated policy, but this depends first on
judgment and cultural approval.

As it happens, the best CI system I'm aware of for truly distributed, multiple
project integration is OpenStack's Zuul: [https://github.com/openstack-
infra/zuul](https://github.com/openstack-infra/zuul) I'm not sure if it
accommodates my prescription above, but if not, they probably have a better
idea.

~~~
antimagic
I, uh, don't understand. No sane environment commits straight to prod. You
stage to an integration environment. The integration environment can break, no
problems. But you only migrate staged dev from the integration environment to
prod when all the test pass. Is that not how _everyone_ does things???

~~~
gknoy
Github does it differently. They have "prod" stuff that everyone sees, and
"candidate" things which are ALSO deployed from the same codebase (with
different enabling flags), and then can do long-term tests (and smaller pull
requests) to verify that the New Way (candidate) behaves identically to the
Old Way.

Slide 35 [0] of this presentation actually starts the discussion of this exact
thing, though Zach has talked about it before. Later, he shows a chart showing
the differences as the code in the parallel branch changed over about five
hours. (Wow, that's some fast iteration.)

0: [https://speakerdeck.com/holman/move-fast-and-break-
nothing?s...](https://speakerdeck.com/holman/move-fast-and-break-
nothing?slide=35)

------
robgibbons
You don't need to break things with good regression testing.

------
LeicaLatte
Nicely done

------
michaelochurch
Sorry to be That Guy but this is one of the reasons I'm really getting into
Haskell, the more I study it. Clojure's still stronger for exploratory work,
but Haskell's the only language where I feel comfortable that I'm breaking
nothing when I "move fast" (at least, on code I haven't touched in a while).

Ok, I'm totally That Guy so I'm just going to shut up.

~~~
scott_karana
I think you must have missed the point somewhere. Test coverage and metrics
are important not just to catch "programming" errors but also logical and
semantic errors. Haskell only partially helps with those other categories.

~~~
michaelochurch
Absolutely true.

I think that if you know how to use it, Haskell's type system can reduce your
bug count substantially. It makes a difference, spending 30% of your time
debugging as opposed to 70%. (These numbers are approximate and vary depending
on the problem you're working on.) You still need to test, though. That is
true. QuickCheck does a lot for that.

------
danso
Quite a bit OT, but I've never seen the transform-rotate property used as a
styling in text...probably something that can/will be used to death, but it
looks great here...gives the piece a very magazine feel (that, plus the
handplaced-layout of floating embed images)

~~~
untog
Unfortunately, it looks awful in Firefox.

~~~
couchand
What version? I'm on 32 on Ubuntu and it looks clean and crisp.

------
no_future
All this crap is easier said than done.

------
Animats
Does the article get any better after the cat video and the mention of
Cheetos™? I bailed at that point.

~~~
holman
It doesn't. The author goes on a long, ambiguous tirade about Skittles™ and
how they aren't as aerodynamic as once initially thought.

You did good by refusing to read the rest of the post and informing everyone
of that fact on Hacker News. Cheers!

~~~
trose
Fantastic response

