
Let a thousand flowers bloom, then rip 999 of them out - logic
http://www.gigamonkeys.com/flowers/
======
not-a-10xer
>Engineers’ effectiveness, on the other hand, is hard to measure. We don’t
even really know what makes people productive; thus we talk about 10x
engineers as though that’s a thing when even the studies that lead to the
notion of a 10x engineer pointed more strongly to the notion of a 10x office.

Management has a vested interest in pushing the meme that there are no 10xers:
it gives them an excuse for not paying these engineers what they're actually
worth. The rest of us just go along with it because it assuages the inadequacy
we naturally feel when we compare ourselves with someone like Linus Torvalds
or John Carmack.

Clearly there are 10xers, and some of them even post here: Walter Bright,
Bryan Cantrill, and Ted Unangst.

Edit:

I'm also not surprised Seibel would say something like this given his past
demonstration of lacking social awareness by titling his book of programmer
interviews "Coders at Work"\--a title that one of his interviewees, L. Peter
Deutsch, rightly objected to, much to Seibel's shock. Deutsch in 1984 co-
authored the seminal paper on building efficient virtual machines for dynamic
languages through just-in-time compilation and inline caching, techniques
which ultimately gave rise to the JVM. He deserves an immense amount of
respect for this and other accomplishments, and we do him and our profession a
great disservice by publicly referring to him as a "coder" or anything else
that elicits something less than awe from the general public.

Could you imagine doctors referring to some highly-regarded brain surgeon as a
"cutter" or "butcher?" No, instead they talk them up as "brilliant" or even
god-like, yet we do the equivalent with "coder" (and allow the media to do it
too) all the time.

~~~
stdbrouw
Albert Einstein and Richard Feynman were 10x physicists. Then there are
probably a couple more 9x physicists, many of them with Nobel prizes under
their belt. A lot more 5x physicists. A whole mass of 2x physicists. Another
whole mass of 0.5x physicists. I don't think anyone wants to seriously dispute
that there's a bell curve of aptitude in pretty much any human endeavor,
whether that aptitude came naturally or was nurtured due to hard work or both.

But the 10x gospel we see in software engineering is an entirely different
beast: the claim is that there's a small group of people who "really get it"
and that everyone else "just isn't made of the right stuff". It assumes both a
kind of determinism (you do not become good, you either are good or you're
not) and a hard dichotomy (there's no ordinary engineers, just bad ones and
great ones).

Of course management would prefer to make ordinary engineers better than to
let a couple of brilliant minds work their magic. But that's just common
sense:
[https://en.wikisource.org/wiki/Why_I_Never_Hire_Brilliant_Me...](https://en.wikisource.org/wiki/Why_I_Never_Hire_Brilliant_Men)

~~~
hitekker
Totally agree.

One thing I've noticed: the people who talk about 10x engineers almost
uniformly believe they're either apart of that elite cadre, somewhat close to
it, or will one day ascend to its ranks.

I believe that there are nodes in a network which have greater inherent value
than other nodes, but can only achieve 10x value when working with other nodes
of similar caliber in the context of the problem they are solving.

------
creyer
Nice article, but I can't agree with everything inside.

1\. Engineers are not computers, so saving 5 minutes a day will not increase
productivity by 1%. Most engineers work with passion, and don't look at the
clock. They are focused on delivering a certain task, and they will make their
best to deliver it, even with the interruptions, even if this will require
them to work 10 minutes more at the end of the day.

2\. The benefit of the tools is a clear win, but as these tools will not
change so much over time, after the initial boost of productivity, the effect
will no longer be noticeable. You can't increase performance each month
compared to the last month just by using the right tools. So I think the EE
team make sense to be created "on demand" and not as a permanent solution.
Time should be a parameter in the effectiveness model

3\. If you grow to a large scale enterprise, as the people are not computers
and reaching agreement between large number of individuals is not easy, the
bigger the number of people in the EE team the harder will be to approach
solutions and test them.

~~~
jules
Argument #1 does not really make sense, since by repeatedly applying the same
logic it implies that engineers will work an infinite number of minutes each
day. A variant of this argument that I've often heard is this:

"My decision to take the plane does not negatively impact the environment,
since the plane would fly with or without me in it."

~~~
creyer
The point I was trying to make is that for engineers you can't do a simple
math from minutes to productivity. The interruptions are big productivity
killers, but all engineers take voluntary breaks. A break helps productivity,
but there is a limit, above that it harms. The same with the interruptions,
some might help, too many will not.

~~~
zaphar
A break the engineer chooses is chosen and is unlikely to interrupt flow. An
interruption caused by the tooling is not chosen and is more likely to
interrupt flow.

Tooling cruft also increases the startup cost of some tasks. Where the amount
of pain you will experience trying to do it may inhibit your desire to dig in.

The author was generalizing he wasn't assuming perfect accuracy. But in an
engineering organization the size of Twitter the productivity gains of a
dedicated tooling team add up fast. Much faster than you might think.

------
wpietri
I mostly like this post, but there's something that troubles me about it.

One of the distinctions I learned from Lean literature was that bureaucracies
fall on a spectrum between _controlling_ and _supportive_. I had honestly only
ever experienced the former: rules and mandates to make me do what other
people valued. But I came to realize that there are other possibilities. For
example, when my team jointly makes a checklist we follow, that's a kind of
supportive bureaucracy. I could imagine scaling that up, but it's certainly
rare.

In this post is a hunger for power that makes me uncomfortable. Sure, any
short-term mess can be solved by bossing people around. But I've seen places
where centralized quality groups become strongly counterproductive. Every
mandate they issue is intended to clean up a mess. But as developers become
disempowered, they do less and less cleaning on their own.

Hopefully that won't happen at Twitter; there's a theme of wanting to build
trust. But there's also the theme of ripping stuff out, of controlling the
chaos, of whipping things into shape. That sounds much better to the person
holding the whip than to those getting whipped with it.

------
aaronbrethorst

        GOOOOAAAAALLLL! Fail Whale.
        GOOOOAAAAALLLL! Fail Whale.
    

Oh yeah, fail whale. I guess Twitter really has come a long way in the past
five years. Despite all of the shit they get for not building out their
product in a visually-engaging fashion, I have to say that I can't remember
the last time their product failed me. Props.

~~~
saosebastiao
That's true, but I can't help but feel like it was an extremely low bar to
hurdle. If "35.6 million tweets during the Brazil/Germany final" is to be
believed, that is only 5k tweets per second on average...maybe 50k peak due to
heterogeneous volume during a sporting event. Call me uninformed if that's
what it is, but that seems like something that should be easily handled by any
decently architected system on the JVM. I couldn't even count the number of
shit systems at Amazon that process that kind of volume with less than 4 9's
of availability.

~~~
aaronbrethorst
[http://highscalability.com/blog/2013/7/8/the-architecture-
tw...](http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-
to-deal-with-150m-active-users.html)

~~~
saosebastiao
Facebook is getting 43x the QPS of twitter, and they are doing it on MySQL.

[http://highscalability.com/blog/2010/11/4/facebook-
at-13-mil...](http://highscalability.com/blog/2010/11/4/facebook-
at-13-million-queries-per-second-recommends-minimiz.html)

~~~
random3
You're quoting a post which is 5 years old and comparing it with one that's
two years old.

------
bbrazil
> Once your engineering org gets to be a certain size the benefits you can
> obtain by investing in making all your engineers slightly more productive
> start to swamp the slight gains that one team might get from doing things
> their own, slightly different way.

I like to think of this idea as group vs. individual velocity. Yes, you
personally might be able to go faster by introducing new technologies - but
everyone else is slowed down by having to learn it. Knowing which few of these
technologies will be an overall win is the tricky bit.

------
hueving
This article repeats the assertion that there is no such thing as a 10x
engineer but doesn't site anything that shows such. It has become a popular
meme but I know counterexamples from people I have worked with that were
significantly more effective (10x minimum) engineers than others.

Does anyone have anything that shows they don't exist? I'm starting to think
it's something average engineers like to repeat to convince themselves that
there isn't anyone really 'better' than them. Sort of a reincarnation of the
condescending "there are no winners or losers" thing we do with childrens
sports.

~~~
InclinedPlane
I've come to dislike the idea of the "10x engineer" partly because it puts
software development too much in the realm of factory work or even craft-work,
which it is not. Fundamentally software creation is creative and inventive. It
is _not_ a matter of stamping out a certain number of widgets per day. It is
not a matter of churning through some number of bugs, lines of code, function
points, or what-have-you. It is design work, it is creation, it is art and
artisanry. To purport to measure developers against one another along some
linear measure is to miss the point entirely. It is the same as asking whether
Mark Twain and Walt Whitman are "10x writers", or whether fillet mignon is 10x
better than a PB&J. Not only are these things non-linear, they exhibit
complexity that is irreducible to such simple measures, which is why we try to
avoid the attempt (although somehow we refuse to learn that lesson when it
comes to movies, television, and video games, but I digress).

The top tier of developers are _far more_ than 10x more valuable than the
average developer. Not because they produce 10x more lines of code, or "crush"
10x as many bugs or sprint points, but because they build. better. systems.
Period. They write better code. They build the right stuff. They promote
better team cohesion. And they execute on difficult projects more effective.

And they don't just make one order of magnitude difference, often they make
_multiple_ orders of magnitude difference.

Here are three fairly common scenarios involving such "keystone" developers:

There's a large and crucial code base which is owned and maintained by an
entire team, but there is one developer who has a much different relationship
to the code than the other people on the team. They wrote most of the original
code, though it has grown much since then, and they know far more of it than
any other single person on the team. Tasks related to the code which would
require much more time and often the collaboration of multiple team members
can be done alone and with much less time by this person. If they leave the
project will live on, and it may even flourish, but the development velocity
and the capabilities of the team will be significantly diminished compared to
when the key developer was around.

There's a core feature on a very important product that is in development, and
there's one developer who seems to have an outsized impact on it. They've gone
the extra mile to make sure the feature is designed and implemented well,
they've put in the hard work to get the feature shipped with excellent quality
(both in the code behind the feature and in the way it works and feels to end-
users). They've made sure that the right work has gotten done and the most
important core aspects have shipped. And the end result is a phenomenal
product that attracts and retains users, engenders customer loyalty, and
ultimately results in a massive amount of revenue for the company.

There's a tiny team or a startup working on an ambitious product. Despite
their limited resources they produce something of significant value and
quality, something that normally would require a much vaster number of average
developers working for much longer to achieve a similar result. This leads to
runaway success.

These scenarios happen all the time in the industry, and they speak to a gulf
in capabilities between the most capable devs and the industry average. And
it's not necessarily about "productivity" it's about getting the right work
done, done well, and done efficiently. You put a top tier developer and a
middle of the road developer on the same task for a week or two and the
difference isn't that one has written a lot more code than the other, the
difference is that one developer has something that _works_ and something that
you wouldn't mind keeping around.

This is another thing. Good developers learn and build from their past
projects. Which means that over time the developer is getting better, and also
what they're able to deliver based solely on their own work grows
exponentially, because it builds and builds and builds, like compound
interest. Whereas mediocre developers are building foundations of sand, they
can only build so high before everything comes crashing back down and they
have to tear down and replace what they've built before with something a
little bit better. Mediocre developers tend to spend their time running in
place.

P.S. Another huge factor is culture, mentoring, leadership by example, and
work environment. Top tier developers raise everyone around them to a higher
level through their example and their interactions with others. Learning from
better developers is a hugely important way that devs get better at their job.
And merely getting into a mindset that getting better is possible and valuable
is important in its own right. Additionally, working with talented engineers,
people we look up to, people we take inspiration from, etc. are important
reasons why devs show up to work every day. When an "anchor" dev with a lot of
talent leaves a team or a company it can trigger an avalanche of talent
leaving the org as other talented engineers decide that the work environment
isn't as worthwhile as it once was and they leave as well, accelerating the
process until a lot of the talent has evaporated from the org. This is a very
common process in the industry because the people who tend to be the most
talented and the most intrinsically motivated tend to have the easiest time
finding other jobs and tend to be the most sensitive to the quality of the
work environment. Unless you're immersed in the work environment and sensitive
to these things you may not pick up how drastic these changes can be until
years down the road when the impact of a team that has lost all its talent
finds itself vastly less able to execute on important projects effectively.

And while I've talked about a lot of "soft" subjects such as "art" and "work
environment", these things absolutely have an impact on the bottom line.

~~~
dozzie
> [...] it puts software development too much in the realm of factory work or
> even craft-work, which it is not. Fundamentally software creation is
> creative and inventive.

I assure you, it _is_ craft. It's much less in the realms of art, science, and
invention, and much more in the realms of putting well-known patterns together
to build something, sometimes even something _new_.

I assume you have never seen a modern woodworker at work?

~~~
InclinedPlane
Craft and art exist on a continuum, there is no break between them. Often
people who are not or do not know artists imagine that the creation of art
entails some magical skill, some special step that is lacking in craft work or
engineering work. That some "quintessence" is injected into the process which
breathes a vitality of art into the work that would otherwise be absent. But
this is a false impression of art. the creation of art is precisely as you
say, putting together well-known patterns to build something, sometimes
something new.

Wood-working using the skills of working the lathe, the chisel, the rasp is
fundamentally the same sort of thing as painting using brush strokes. And both
skills can be used to create either a work of art or a work of craftsmanship.
It's a matter of skill and intent. Fundamentally though, creating a painting,
a poem, or a song is mostly craftsmanship, it's the application of skills to
build something in a familiar way.

There's no magic breath of life that elevates a crafted work into the realm of
art, rather it's the purpose it's made for and other factors which make that
distinction. Something that is considered "not art" would typically be
something that is completely unoriginal and has a purely practical purpose.

Art, on the other hand, tends to involve creativity and originality, it tends
to not just be for a practical purpose but also to be evocative, emotional,
and communicative.

You see, the method and skills of construction are not the differentiator
here. Both a EULA and a sonnet can use the same medium (written language) yet
while one is purely practical and ordinary the other communicates more than
mere facts and details, involves some degree of originality and creativity,
and engages the reader at a higher level.

Software dev is a very wide ranging field, so it is difficult to talk about it
holistically. However, in my experience it does not fit the description of
unoriginal rote work. Often there is creativity and originality, as well as
subtlety and fine artisanship. It is a new kind of thing which does not neatly
fall into either the manufacturing/engineering/"craft" or "art" buckets, and
for that reason it requires different ways of thinking about the work and the
workers.

The most important point here is that when you give craftsmen the task of
making 20 chairs, they will do so. Some with greater efficiency than others.
But this task is nonsensical when it comes to software, because it is only
ever necessary to make one of a thing in software, copies are free (or as
nearly so as makes no practical difference). Instead the equivalent task would
be to make 20 different chair _designs_. And now it becomes more clear that we
are not dealing with craftsmanship, we are dealing with design work. Creative
work. The closest analog to which is art. And how do you judge an order of
magnitude difference in design work? How do you objectively measure the
difference between an Eames lounge and plastic patio furniture? Objectively
you can't, but subjectively we are aware that the difference between middle of
the road design and top tier design is much more than a single order of
magnitude.

------
CamperBob2
Amusingly, that was Chairman Mao's approach as well.

~~~
zyx321
Yeah, from the title you'd expect an article about Chinese politics. Turns out
it's just an Effectiveness Engineer using fancy phrases without understanding
their historical significance.

~~~
task_queue
This happens so often in this community that it's cringeworthy.

When I got to the first mention of Twitter I hit the back button.

------
diminish
At first I expected an article on Twitter platform for 3rd parties where 1000
flowers bloom and 999 of them get ripped.

Twitter has a horrible, slow moving, bloated interface. I m facing countless
examples of disturbances every day.

Maybe a single platform based on PHP could be better, if Facebook is so
usable?

~~~
MrBra
Really, Facebook?

Yesterday I spent half an hour on my mobile just to find out where the "view
profile as it would look to someone else" page had been moved.

------
Mz
I strongly suspect the mathematical formula suggesting that 10,000 engineers
could be as productive as 45,000 with the right support is a case of "in
theory, but not reality."

When tree farming was invented, the initial crop was wonderful, but they soon
had to come up with a term meaning "forest death" for the sickly,
underdeveloped trees that followed. A monoculture of the same plants quickly
strips the soil of essential nutrients and if you cannot identify those
nutrients and aggressively resupply them, you will soon find that the trees
you wanted to grow and harvest will no longer thrive and somnetimes will no
longer survive. A forest can produce healthy trees for generations because of
the diversity of plants growing therein. Different plants use different
nutrients and some replenish the nutrients used by neighboring plants.

Obviously, software tools are not plants. But I have my suspicions that the
diversity of code he describes developed in part for reasons like "different
engineers happened to be experienced with different languages, tools, etc" but
there may be cases in there where it developed that way because it was the
best way to handle it. So when you start standardizing things, you may be
killing off things that are critical to the health of the codebase and the
company.

I can see value in having someone dedicated to serving the needs of the
engineers so they can be more productive. I can see a need to clean things up.
But complexity itself often has inherent value that cannot be replaced or
improved upon with a simpler solution. Sometimes, simplifying things amounts
to throwing the baby out with the bath water. I hope they don't wind up doing
that.

[https://en.m.wikipedia.org/wiki/Forest_dieback](https://en.m.wikipedia.org/wiki/Forest_dieback)

------
Rifu
That was an enjoyable read. As mentioned in the article, here's the link to
the same talk he gave at Facebook's @scale [0], though the sound gets
ridiculously out of sync after a bit.

[0][https://www.youtube.com/watch?v=8IyXcLFO9ns](https://www.youtube.com/watch?v=8IyXcLFO9ns)

------
retrogradeorbit
> Alex Roetter, now our SVP of Engineering, then an engineer, personally led
> the effort to convert what Scala code the ads team had already written into
> Java.

I'm having trouble understanding why on earth you would rewrite Scala code to
Java. The Scala code runs on the JVM and can interoperate with the Java, so
what is the point?

~~~
mseebach
That's cool if the Scala bits are mostly static bit, not so much if they see
active development.

When (say) 5% of your codebase is Scala and the rest is Java, and your
engineering team prefers working in Java, those 5% become stumbling blocks.
The context switching slows you down and increases the risk of introducing
errors.

Also, a tool chain that will support two languages in parallel, while
obviously not impossible, is necessarily more complex than only supporting
one.

------
jackgavigan
_> Twitter’s first line was written in 2006 by our now interim CEO, Jack
Dorsey._

Really? Not Noah Glass?

Also, no mention of Pivotal Labs role in helping Twitter get rid of the fail
whale...

~~~
sulam
What role did they have? Unless you mean the pivots that Twitter hired full
time? I worked at Twitter during the entire transition from the monorail to
the current architecture and there was ~0 Pivotal Labs involvement in that
process. I know they had a big impact on the earlier phase of Twitter but when
it comes to getting rid of the fail whale a large part of the credit goes to
Marius and Raffi, neither one of whom were Pivots and I can't think of a
single Pivot that built a service at Twitter that is in production today.

------
davedx
I'm really interested in the "standardization" part near the end. Is the end
goal to have Twitter building everything with Java, or everything with Scala,
or something like that? It does seem to go against the grain of "use the best
tool for the job".

------
im3w1l
> E = (eng - ee) * (1 + b * ee^s )

He posits that model and draws many conclusions from it. But how reasonable
is? I would expect the improvements in efficiency factor to decay much faster.
If everything is done exactly right, there is an optimal efficiency factor.
With no ee's there are deviations from that perfect process. As more ee's are
added, more deviations are fixed and you get closer to optimal efficiency.
These assumptions would mean that efficiency should converge as number of ee's
approach infinity. My model suggestion would be

E = (eng - ee) * (1 - b * s^(-ee) )

------
dunkelheit
The article discusses the kinds of problems that wildly successful teams of
1000-s of engineers have. As such it is kind of irrelevant for most of us -
not everybody will grow their team to 1000 engineers or assume the role of VP
Eng in a large organization. A more practical matter to consider, are these
problems worth worrying about from the start or is "let the thousand flowers
bloom" strategy part of the twitter's success?

------
throwaway4123
I was discussing this article with another ex-Twitter employee yesterday. I
quit in the middle of this monorepo clusterfuck, and my views are very
different - I believe most of what happened was a Battle of the Egos. I can
name names but its not worth the trouble, Twitter was a generous employer etc.
He remembers most of this as a Battle of the Languages ie. Scala vs Java vs
Ruby thing. Seibel has this 3rd version, having joined much later in 2013. The
list of ex-Twitter engineers is probably in the thousands at this point, so
you are going to hear a whole different set of narratives.Twitter is Roshomon
2.0. One thing is certain. The lack of profitability is very much due to
throwing lots of cash at lots of engineers who wrote and rewrote the same bits
until kingdom come, driving each other and the market nuts as to what all the
fuss was about.

------
franciscop
Once you get 4.000 Engineers in EE, it is worth considering creating some
tools specifically for EE. Let's call this team Effective and Efficient
Engineering, thus EEE.

------
nowarninglabel
The question that has been arising for me, is what amount of resources do you
devote to engineering effectiveness in a relatively small engineering org (~25
people)?

It seems like the presented formula would say to not devote any resources, but
there seems to be some good return on some effort put towards it. Right now,
I've just considered it some percentage of my job to work on.

~~~
JoshTriplett
If you have only 25 people, you have few enough that you can reasonably
discuss just about everything with the whole team. So, you can reasonably talk
about what kind of "engineering effectiveness" work you need, and its priority
relative to the other tasks on your list. How much time you spend on it then
depends on where the task falls on
[https://xkcd.com/1205/](https://xkcd.com/1205/) as well as how much time you
can spare from your current workloads.

------
wpeterson
Once upon a time, there were multiple repos and multiple tools for different
jobs.

Then Twitter grew to thousands of engineers, many of whom were from Google.
Now everything is being converted to enterprise Java in a single, monolithic
repo.

Is that an improvement or just bureaucratic standardization and cargo-culting
from engineers raised in the google3 monolith?

------
plesiv
Minor side-issue: 2014 WC final Germany/Argentina; Brazil/Germany was in semi-
finals.

------
lucio
I was constantly being reminded of Taylor, while reading the article. Is this
the scientific management of software engineering? It looks a little too
constraining for a rapid-changing discipline as SE.

------
presty
is it just me or this article points to a huge lack of technical leadership
inside twitter?

I wonder how much allowing things drag for so long cost the company? millions
in salaries?

------
anon4
Offtopic, but why is "monorepo" pluralised as "monorepi", while "repo" is
traditionally pluralised as "repos"?

~~~
thristian
I'm guessing the author is poking fun at the idea of having more than one
'monorepo' by giving it a deliberately silly plural.

~~~
pavlov
The plural of "bordello" is "bordelli", so there's that. (And that's all the
Italian I know.)

------
jordigh
I wish people would learn what the hundred flowers campaign was and not use
such dumb and insensitive titles for their blog posts. It's like calling it
"the final solution to the twitter problem".

~~~
Confusion
I wish people would learn that words and phrases can acquire meanings of their
own, independent of original meanings, origins and associations.

"Let a thousand flowers bloom" has an entirely clear and positive meaning for
the vast majority of English speakers.

~~~
btbuildem
Unless you know about "let a hundred flowers bloom", then it immediately
acquires a meaning at best sinister. Ignorance should not be an excuse for
callousness.

~~~
forrestthewoods
Actually it is an excuse. Callousness is about intent. Someone isn't being
callous if they have no idea.

