
The 10x software development gap - andymorris15
http://blog.contino.io/blog/the-10x-software-development-gap-why-do-some-organisations-win-and-others-lose-when-it-comes-to-software
======
gmarx
In my experience most of these problems are cultural and there is nothing to
be done once bad software dev culture takes root. The most common
manifestation is management asking for something reasonable but with
constraints that make it impossible. The developer who agrees is favored. The
one who warns is labeled as "negative". A year later when the project is in
disaster mode the manager never reviews what happened at the beginning because
he doesn't understand it.

It's essentially a Darwinian selection process for bad developers.

~~~
awinter-py
Very important point on good tech managers.

Strong engineering orgs hire managers who could do and have done most of what
their team does. Most TPMs at G, for example, have CS degrees.

Non-technical managers have the organizational leverage to undo any amount of
design & planning that skilled people underneath them set up. It's the brain
equivalent of having ADHD -- your executive control shuts down and that
sabotages long-term goals.

If you've never lived through this, watch the beginning season 3 of silicon
valley where their new CEO wants them to 'just build a box' and gets angry
when the conversation gets any more specific than that. (If you have lived
through this, don't watch it -- it's too real).

~~~
gmarx
Another point WRT Silicon Valley episode that has lessons for the real world.
Developer will do a much better job on a project they picked rather than one
that was assigned to them. This might be a lot of the advantage tiny, new
startups have over the giants.

------
maxander
I suspect there's an additional factor incidental to the "skills and talent"
issue- the ability of employees to _care_. I would wager that there are few
engineers who are really, truly passionate about Oracle or Salesforce, but for
some reason people still get excited about Google, Facebook, or the next Uber-
for-dogs startup. This means that the latter category get an entirely
different kind of value from the skills of their engineers.

If you're a good programmer, you can optimize for various different things-
getting something done quickly, writing pretty code, making the boss happy,
following every last implied detail of spec, or, importantly, _making the
software as useful as possible_. What you need is for an engineer to be able
to think, "well, bossman won't necessarily like it and it will take another
ten minutes, but I should write this in this specific way because I predict it
will prevent some problems if this code gets reused in certain contexts."
There are innumerable small things engineers can do solve problems that _might
happen_ , but no salary can incentivize them to do this (or even to look for
the opportunities to do so) since there's effectively no way to tell if
they're doing it- engineers who don't care will likely be optimizing for
making their bosses happy, so engineers who do care will even often seem
_worse_ according to any effectiveness metric the company employs. And if your
employees don't care, you quickly accumulate technical debt that weighs down
the company, even if everyone is happy and productive.

I don't particularly think there's a solution to this. A sad fact about
industrial economies is that a whole lot of boring things that no one cares
about _need to get done_. Even most of the applications that are exciting
_now_ will be boring in the future (someday, internet search will be an old-
hat, commoditized field everyone takes for granted, and Google will fall.)
Maybe someday company lifecycles will just be considered a fact of nature, and
everyone will simply expect the next Microsoft to roll over and expire when
its flagship product becomes irrelevant; it might make things _more_ sane.

~~~
braveo
I don't necessarily agree with software people solving problems that _might
happen_ unless they have a lot of experience in the domain.

Do the simple thing and if you're wrong then you have less work to throw away.

~~~
tonyedgecombe
I'm not sure it's about adding unneeded features, rather stuff like not
ignoring returned error codes or failing to write a unit test for something
that needs it.

------
js8
I talked about this on a lunch with a coworker few days ago. We work in a
traditional software company which sells legacy software to (big enterprise)
customers. We were frustrated that everything takes us so long to change.

I think the reason is that we have large customers who expect a certain level
of service. So even a trivial fix needs to be properly tested, documented,
packaged. Also, we don't have a really good insight into how our customers use
our software. We don't know what things we can break or deprecate, so we have
to be extra careful. Support calls from users cost additional money, and we
want to avoid those.

Companies that do not have external customers, but write software only for
their internal use (or have cloud platform), such as Facebook, Amazon or
Google, do not face these problems. They know who their customers are (guys
from the other cubicle), what features they use. The can take a risk of
deploying an improperly tested feature.

So yeah, they have an advantage. Whether this advantage will translate to the
only internal software development model surviving in the future remains to be
seen.

~~~
ryandrake
What? Facebook, Amazon and Google do not have external users?

"We need to be careful because customers use our software" is a pretty lame
excuse. Every company needs to properly test, document, package their
software. Some companies do it better than others.

~~~
js8
Where they have external paying users, they still operate the software for
them (i.e. provide it as a service), so they have access to all what these
customers are doing.

It's not a lame excuse, unfortunately, there is a very real barrier of trust
when you talk to a coworker (even from another group) and when you talk to a
customer. At the very least, you can agree on a common schedule.

IMHO it's not a coincidence that the blog listed those companies as more
"nimble".

~~~
tyingq
He also hints that his customers might have more power to push back on
changes. That's certainly the case for smaller software houses that sell to
big enterprise customers.

Facebook, Google, etc, are somewhat beholden to end users, but not in a way
that impedes them from rolling out what they want, when they want to.

~~~
ryandrake
Good point. The worst-performing companies I've worked for had one thing in
common: they were small and unable or unwilling to push back on toxic
customers.

~~~
tyingq
Pushing back doesn't always work. A good example...support for IE6, way past
the time when that was a good idea.

Your contact at "big mega corp" that pays for your software may have no
ability to change their internal policy that they've stuck with IE6 for years
after it was sunset. So, if you want the sale or renewal, you have to support
it.

------
aaron-lebo
_Why are digital leaders (Facebook, Google, Amazon and friends) ten times
better at rapidly delivering software than mainstream enterprise
organisations?_

That's the first sentence, hence the premise of the article. Don't you need to
provide some supporting evidence for those figures?

 _Experience, again, suggests not. The digital leaders themselves are
behemoths! Facebook has 1.19 billion active monthly users. Keeping them happy
is quite a task, yet they manage to innovate alongside keeping the lights on.
So how come enterprises are still so slow? Why do their technological efforts
always seem to cost much more than they should? Why does the software
development gap persist?_

Does not follow. Facebook has lots of users but they have a relatively small
workforce compared to traditional big companies. Over some threshold it's
probably easier to "satisfy" lots of users via network effects than it is to
please a small but critical community, anyway, but that's just conjecture.

The article is a continued series of unsupported assertions and "facts".

~~~
benjaminwootton
Anyone who has been around enterprise IT would probably say that this is at
least the right order of magnitude. A Spotify, Netflix, Google, Facebook can
fly compared to a traditional enterprise

~~~
walshemj
I think it depends on the enterprise I recall when I worked at a big telco in
the UK a stat that we delivered 80% better than the average - I though "Huh
company propaganda" \- but when I worked for other Large companies I was
shocked a how crap they where.

Then again my division was the direct heir to Tommy Flowers group and at the
time had more engineers than Google had employees.

------
tyingq
The problem, as defined in the article: "Why are digital leaders (Facebook,
Google, Amazon and friends) ten times better at rapidly delivering software
than mainstream enterprise organisations?"

Then, the conclusion is "skills and talent lie at the heart of this issue".

That's not been my experience. There's usually plenty of skill and talent at
mainstream enterprise organisations. It's other barriers: cultural,
organizational, funding, etc, that keep the talent hamstrung.

~~~
Joeri
Is the premise even true? Do facebook, google or twitter ship more major
features than traditional enterprise companies? That sounds off to me.

Even if it were true, it has less to do with talent than with organization.
Legacy organizations typically have legacy customers on legacy contracts that
dictate legacy development models. Just try to be agile or do continuous
deployment with a typical government contract. It can't be done.

~~~
zigzigzag
There's a lot of selection bias there. Google at least has failed and late
projects like any company. You just don't hear about them.

------
rb808
I've spent 5 weeks working with our offshore sysadmins to get Python3
installed on a dev box. Its a 30 second job, does anyone in a startup have
these problems? Big banks suck right now.

~~~
CoolGuySteve
Here was my experience at GS, I didn't last too long:

\- I was never able to get email search in outlook enabled on my computer. It
was a "security risk".

\- Every Monday, my Visual Studio install broke and I'd have to call support
in Bangalore to reinstall it.

\- Our "make" system's subtasks would segfault every 1/3 times so I always ran
it 3 times, sometimes 4.

\- The only way to commit code (after using an out of date CVS) was to use
this second in-house utility manually on every commit revision on every file.
The asshole who wrote that thing made MD.

\- Our 32bit system would run out of memory and take hours to compile because
Java talked to C# talked to C++ talked to Slang talked to javascript all in
the same address space.

~~~
qntty
What's MD?

~~~
jwoah12
Managing Director. Generally the highest non-executive role in Financial
Services firms.

------
pjc50
_enterprises also suffer from organisational and process legacy_

Indeed. It's possible to waste _huge_ amounts of time simply trying to decide
what to do and communicating with all the 'stakeholders'. If you want to
change the way your organisation builds software products you have to change
the organisation.

~~~
jrs235
Ding ding ding!

We've been experiencing a lot of entropy where I'm at. Everyone could feel it,
no one could understand it. I started looking for what might be happening and
what might fix it. I came across Lex Sisney and Organizational Physics. It
makes a lot of sense, at least to me.

P.S. I'm not affiliated with Lex. (Link to his Kindle book on Amazon:
[http://amzn.to/2iUh09B](http://amzn.to/2iUh09B) or find a lot of what's
covered in his book on his website for free:
[http://organizationalphysics.com](http://organizationalphysics.com) )

------
innocentoldguy
This is an ad, more than anything, and I don't think it even remotely
addresses the issues it brings up. I do have to say, it is refreshing to have
yet another empty slogan, _transform through delivery,_ to use at work.

~~~
benjaminwootton
I think that is unfair having just re-read it. (I'm the author.)

90% of the article is about our theory that large enterprise organisations
have a huge gulf when compared to digital leaders, and analysing why they are
in this place.

To break out of it, they need to do things in a completely different way, more
focussed on building their own capability than outsourcing work.

~~~
jordache
Again, you don't just "break out" and act like "digital leaders" just for the
sake of it.

Why does a fortune 50 company need to do that? What is the ROI on such effort?
What are their risks if their IT operation is not as cutting edge as Amazon's?
Those are the core questions, which your article does not even mention.

~~~
jordache
Ok just read through again.. you did allud to some of the rationale... my bad!

------
passivepinetree
Probably irrelevant, but I'm tired of seeing angled pictures of blurry code
that looks like minified javascript on every article about software
development from a business-focused source.

Can we get some better stock images?

~~~
zeroer
That would require someone on the stock image company's payroll who knows
something about code. That costs money. How much are you willing to pay for
articles with good stock photos?

------
philk10
Needs to read the Leprechauns of software engineering book - or listen/read
this interview with the author - [https://blog.fogcreek.com/10x-programmer-
and-other-myths-in-...](https://blog.fogcreek.com/10x-programmer-and-other-
myths-in-software-engineering-interview-with-laurent-bossavit/)

------
xigency
Not related to the article much -

The 10X developer concept is a bit flawed. If only because computer scientists
and students understand that a linear co-efficient is borderline meaningless
in measuring algorithmic performance, it seems to pander to the less-
technically-literate side of the engineering pool.

Another problem - the developer who accomplishes tenfold productivity of his
peers in one environment might be absolutely uncompetitive in another. For
example, what's the difference between tuning performance in a critical
backend path and building a fully functioning UI? A developer who is ten-times
better at one might be incapable of completing the other.

Instead, especially with the whole full-stack movement, wouldn't it be better
to look at qualifications, ability to learn, motivation, interest and
attitude? Even looking for a "ninja developer" might be better if it means
someone with an attitude to hack things together at any request. And looking
for a seasoned developer might be better for maintaining and rebuilding one of
those hacked-together platforms.

What's the difference between a 10X and X^2/10 developer? It's time for hirers
and SE bloggers to see X, log X, X log X, and X^2 developers, who are all out
there.

~~~
cel1ne
I don't find the distinction very useful either.

But I think that a 10x developer is one who knows all the ins and outs of the
tools in his or her belt.

Who will prefer a 10-year-old framework that served him well over an
opportunity to procrastinate by means of reinventing the wheel. (I'm looking
at you, JavaScript-land)

And who doesn't fall into the "tower of babel"-trap, which is the believe that
there's a universal language and a perfect way of writing a program or
structuring a system.

~~~
nine_k
To be fair, there are few, if any, 10-years-old JS frameworks that one can
still be productive with.

Well, there's jQuery, which is still fine for very small things (an animation
here, an AJAX call there), but is totally inadequate if you need to build a
rich UI of the kind commonly expected now. You'll spend less time learning
something like React or Vue and implementing things in it than you'd spend
fighting with the callback hell and manually crafted state machines.

~~~
cel1ne
I know and use React (and React-Native) daily, but, except the views which are
in JSX/es6/7, I like to write everything in java or kotlin which I can run in
the browser.

There are a gazillion very useful Java-libraries out there.

I just implemented a ListView with complex filtering, live text-matching and
sorting for an Android/React-native-app which has no problem displaying
100.000 items or more using Java-libraries.

React-Native's implementation on the other hand seems to be riddled with
problems, judging from the bug-tracker and the confused API.

------
acobster
Yawn. Paul Graham basically wrote this article twelve years ago. I'm no
fanboy, but at least he wasn't trying to coin _bland catchphrases_ by
_italicizing them._

------
ehutch79
Is this an article, or a sales pitch?

~~~
ryandrake
> "Posted by Benjamin Wootton on 05/01/17 15:20"

[...]

> "So what’s the solution? To reduce the software delivery gap, mainstream
> enterprises need to raise their game by transforming their own internal
> capability."

[...]

> "We founded Contino so that we could instead transform through delivery."

[...]

> "Benjamin Wootton is the Co-Founder and CTO, EMEA of Contino."

~~~
benjaminwootton
It's a corporate blog. Describing our view of the world and our approach and
experience in solving it. Pick a hundred blog posts and they will follow the
same format.

Of course it's partly marketing but it doesn't detract from the argument.

~~~
ryandrake
Just helping to answer the question. For what it's worth I found the non-
salespitchy content to be spot on. Agree with your points.

------
jheriko
this was an interesting read, but i have an similar, but slightly different
observation.

google and facebook also have mountains of technical debt and they don't
necessarily fix it. in particular some google projects i have had to build
have had the absolute worst kinds of dependencies and requirements to build -
from what i've heard from the inside they don't suffer so much from this
because they build elaborate systems to deal with it... that is not paying
back technical debt, it is creating the risk of more in the future to avoid
having to pay it at all.

aside from google some facebook projects i've seen also contain the very worst
of practices of the kind that invariably result in technical debt and
difficult maintenance work... but i know nothing about what they do

maybe it has the same effect, and maybe that is part of the trick, that its
more productive to live with technical debt if you do it right than it is to
pay it back

------
blacksmythe

      >> Why are digital leaders (Facebook, Google, Amazon and friends) ten times better at rapidly delivering software than mainstream enterprise organizations?
    

Because they pay significantly more, and are able to set up extremely narrow
hiring filters?

------
btilly
Maybe I'm in a bad mood, but this seems to be a poorly researched article that
throws around lots of catchphrases and unsourced claims in a way that speaks
to people's prejudices. Which means that people want to accept it, but
shouldn't blindly.

Let's work our way through it.

It starts with a claim that there is a 10x gap between software leaders and
the rest. Where does that figure come from? Is there any data backing it up?
If you dig in you'll find that the figure dates back to some poorly researched
studies in the 1970s and originally was about individual software developers.
There isn't any backing to this as a figure for organizations as a whole.

Next we have a reference to the innovator's dilemma that completely gets
Clayton Christensen's thesis wrong. His actual thesis is that companies are
very, very good at adopting improvements that they see as necessary to
improving their core competency. What companies are bad at is jumping on
inferior (by the measures their current market uses) technologies that are
about to destroy that market. He offers lots of examples and his follow up
book, _The Innovator 's Solution_, walks through the internal organizational
reasons for WHY it is so hard.

It is well worth understanding this thesis. However it has absolutely nothing
to do with building a better website.

The too big to innovate section had no flaws that I see.

The technological legacy and technical debt section comes closer to the mark -
then misses it entirely. The truth is that any organization has a culture,
core strengths, and core weaknesses that are part of the corporate DNA. These
matter and are very, very hard to change. These will determine what
organizations are good at. "Digital natives" don't simply win by not having to
spend on maintenance - my experience is that their ratio of maintenance work
to new development isn't different than anyone else's. They win by having
cultures, processes and decision making that work better for software.
Cultural elements like not trying to use schedule pressure to motivate
developers, processes that try to surface potential problems as quickly as
possible, and decision making that works hard to avoid a blame based culture.

Unfortunately you would learn none of that from this section. You would learn
more about actual core organizational problems from
[http://funnyshit.com.au/the_plan.html](http://funnyshit.com.au/the_plan.html)
and [http://www.tamingdata.com/wp-content/uploads/2010/07/tree-
sw...](http://www.tamingdata.com/wp-content/uploads/2010/07/tree-swing-
project-management-large.png). (Both of these highlight very, very real
problems. The Plan illustrates the consequences of information traveling
through an organization where each hears what they got in line with the bias
that they want, then communicates upwards the best sounding thing that fits
their understanding. It takes serious discipline to counteract this. And the
project management cartoon demonstrates _exactly_ how each role tends to
fail.)

Skills and talent would have been an excellent place to raise some of these
points. Would have been. Opportunity missed. Instead we get trite comments
about how talent perceives big companies, with no useful information on why
they are that way or what would be required to change.

Next, outsourcing. As a US based developer I like calls to reduce outsourcing,
but this one is sadly just wrong. All of the big "digital natives" have
outsourcing as a component of their development, and make it work. The actual
tradeoff is that outsourcing adds delays and creates cultural challenges, but
makes for cheaper development. Proactively address the challenges, outsource
things where development speed is not a priority, and outsourcing is a very
viable piece of your development. It should be part of the puzzle for any
large company.

And finally we come to their proposed solution. But if the real problems are
not understood, then there is no reason to believe that the solution will
work. In reality I would expect that the high performing team will come in,
perform its project well, generate resentment, get no traction, and never
figure out how the surrounding organization is sabotaging itself.

And now I'm going to need to read some Steve McConnell to get the bad taste of
this article out of my system.

------
altoz
tl;dr

Your software sucks because you don't have enough technical talent. + sales
pitch to develop technical talent at your company.

~~~
benjaminwootton
Sorry for the sales pitch at the end.

Hopefully it doesn't detract from the point that it is a massive gulf between
a Netflix and a traditional enterprise, which runs deep into their DNA.

The big boys simply have to overcome this over the next few years.

~~~
jordache
What you described is accurate.

However there needs to be an intent by the "enterprise" to achieve some end
goal, which is aligned with their overall business strategy, to operate their
IT more like the agile technical rockstar companies.

Such transformation is costly and disruptive, so all of this has to be
properly vetted in the context of their overall strategy.

