
Why Big Companies Keep Failing: The Stack Fallacy - walterclifford
http://techcrunch.com/2016/01/18/why-big-companies-keep-failing-the-stack-fallacy/
======
gregdoesit
From my experience the same (often random) reason that makes a company
succeeds, then becomes their DNA, and finally can make them fail.

I saw this happen with Skype where I worked a couple of years. The company
succeeded because of P2P: we grew with little infrastructure to reach 200M+
people. P2P became our DNA, rooted deep within (almost) every core component.

Then came the new wave of mobile messaging apps. We reacted... with a P2P
messaging solution. It was obvious this wasn't working - you sent a message to
someone from Skype for iPhone, and they got it... sometime.

We knew to have a chance against Whatsapp and other messaging apps we needed
server based messaging, so we built it.

It took 3 years. Yes, it took this long to get rid of the P2P code from just
the messaging components from the 20+ Skype products - we had 1,000+ engineers
and 50+ internal teams by the end which significantly slowed things down. When
we were done and popped the champagne - no one really cared.

And yes, the source code is still full of P2P references and workarounds to
this date.

~~~
im2w1l
Why was it hard to deliver messages to iphone using p2p? I mean I guess they
couldn't open ports and listen for messages, but that should have been true
for plenty of NATed systems too right?

~~~
gregdoesit
The problem was the avalibity when the sender was offline. You send a message
from your iPhone to your friend who has an Android. Your friend doesn't have
reception, and after sending the message you kill Skype b/c of the battery
drain (otherwise it is being part of the p2p network).

Your friend now gets reception and will get your message... between a minute
and a couple of hours. Most likely when you turn your phone back on, as the
p2p network was not designed to store and propagate offline messages.

Meanwhile someone at Whatsapp built a key value pair on a server and if you
had reception, you got the messages immediately and reliably. Oh, yeah,
because the read and unread message states in Skype also propagated via the
p2p network and also got took a while to arrive...

~~~
3pt14159
I'm really surprised that Skype didn't just build a separate system for
mobile. It seems even a simple evaluation of the facts (needing hundreds of
engineers to alter an already stable product) pushes things more to the server
side than the P2P side.

~~~
21p
hindsight is 20/20\. It would be quite easy for a team of developers and
product managers to get together to discuss "how we take what we have and
extend it to cover the new use case" rather than "what is the best approach
for the new use case".

~~~
firebones
But even before reuse, the "DNA" point can't be overlooked, though. Success
breeds a belief that "our differentiators" need to be considered and baked
into subsequent products. More often than not, those constraints lead to
situations where you're fitting a square peg into a round hole, as with Skype
and P2P messaging.

What saves some companies from this is a core DNA that can be interpreted in a
very liberal and malleable fashion. For instance, Amazon is about distribution
--be it products, bits, computing power. Facebook is about social connection
(not P2P or a particular narrowing of tech). Of course, maybe that's just
survivorship bias...no one keeps a good directory of the graveyard of
corporate DNA that was as similarly broad.

Now whether that is a function of articulating flexible DNA, or the
flexibility of the decision makers...I don't know. Either way, fixed mindsets
close down adaptivity to new opportunities.

------
anonymousguy
The solution to stack fallacy is simple but really counter-intuitive. All of
the mentioned examples, I mean every single one, indicate a business trying to
force its way into the higher level through business channels. For example,
when a business wants into a higher level they make it a business priority to
create a new product and attempt to drive the priorities of this next level
product through their business objectives. That is an epic fail.

It is important to instead concede that you don't know the needs of the
consumers in the higher level, and if you think you do it is because you are
guessing. The only way avoid the problem is to not attempt to move into the
higher level, at least not intentionally and not through business priorities.

This is extremely counter-intuitive because there are generally fewer expenses
and greater market frequency at each higher level, which means superior
revenue potential. Businesses exist to make money and to ignore moving up to
the higher level means denying this potential (vast) revenue source.

This doesn't mean you can't move into the higher level of the stack and be
really good at it. It just means you cannot do so both intentionally and as a
business objective.

The solution is to double-down on where you already are with what you are
already good at and focus on product quality of your existing products.
Continue to improve where you are already good. Improvements and enhancements
to existing products can gradually yield the next level as the improvements
progressively open new potential in an evolutionary fashion. While getting to
the next level this way is much slower it is also risk reduced and continues
to associate your brand with quality and consume satisfaction.

This will only work, though, if the goal is improving the current product and
not acquiring revenue in that desired higher level. Think evolution and not
revolution. It has to be a gradual, almost accidental, increase of capability
based on meeting current consumer needs.

~~~
PhilWright
I don't think this works in practice.

I cannot see Oracle slowly adding extra features to its database until it
becomes an actual CRM system. A CRM is not just a solution to a different
business problem but also a logically different layer from a development
perspective. You would have to build it as a product that sits on top of the
database product.

But as a CRM system it has no value to the end customer unless it has a
certain minimum level of functionality. So although you can add features
gradually and evolve your CRM offering you will not have a single customer
until you have actually added enough features that make it actually usable.
Nobody will buy your gradually evolving system until several years have passed
and it works for some actual customers. In which case you have already spend
alot of money and effort.

~~~
jodrellblank
_But as a CRM system it has no value to the end customer unless it has a
certain minimum level of functionality_

This sounds a familiar argument - roughly what you're saying is that an eye
cannot evolve, it has to be intelligently designed? :|

~~~
PhilWright
eh? I make no comparison between developing a minimum viable CRM product and
claiming irreducible complexity in biological systems.

~~~
jodrellblank
I made the comparison with the kind of argument, because you made the same
kind of argument.

 _Nobody will buy your gradually evolving system until several years have
passed and it works for some actual customer_

The parent was assuming people were already buying Oracle as a database (for
example), and continuing buying Oracle as a database, and gradually getting
CRM features coming with it.

So people will always be buying the gradually evolving system, and it will
always/increasingly usefully be working for some actual customer.

 _But as a CRM system it has no value to the end customer unless it has a
certain minimum level of functionality._

"There's a minimum crew requirement"

"What's the minimum crew requirement?"

"Oh, er, one, I suppose".

It has value as long as it has any value at all, there isn't a certain minimum
level of functionality unless you assume either 'absent' or 'broken' is
involved. By the time a single shipping useful feature rolls into a production
release, it has value to anyone who wants that function. Or, potential value,
at least. Like a proto-eye that has value as soon as it's a light/shadow
sensitive cell.

It wouldn't make sense to release one basic function as a CRM product - which
is why the parent is talking about 'evolving' an existing product up to a
higher stack layer, not trying to release a new product at a higher stack
layer all at once.

 _So although you can add features gradually and evolve your CRM offering you
will not have a single customer until you have actually added enough features
that make it actually usable_

Following the model where Oracle tries to build and launch a CRM product all
at once, or Google try to build and launch a social network all at once -
which is completely the opposite of the gradual improvement up to a higher
level which the parent was suggesting. You're arguing "you can't gradually
improve from where you are because that isn't releasing a finished product in
one go", no it isn't, that's the point.

------
adevine
I don't buy the arguments in this article. For example, the whole part about
why Google failed with Google+ is just wrong IMO. It wasn't that Google wasn't
capable of building a good social network. If anything, I (and most people I
know) preferred the design and interface of Google+. The problem was that
Facebook already had a huge head start, and all your friends were already
there. Facebook was "good enough", and there wasn't a big enough incentive to
want to switch to Google+.

If anything, large companies often miss out on new trends and changes in
business and technology, but it's not solely because building that one new
layer "up the stack" is so technically hard or different.

~~~
dasil003
Everyone you know thought Google+ was a better interface than Facebook? That
seems quite a strong anecdotal statement, but I have a hard time accepting it
at face value. Yes G+ had some nice aesthetics, mechanics, use of white space,
etc, but conceptually and organizationally it was a mess.

To chalk it all up to network effects is to let them off the hook too easily,
after all there is no reason you can't build other successful social networks
concurrently with Facebook, you have your Twitters, Instagrams, Snapchats,
etc.

The way I see G+ failing is because there was no soul or vision to the
product, it was driven by a simple fear of Facebook, and it's implementation
was a simple conglomeration of features that constituted a cool tech demo but
was not shaped by real users or a real use case (Google does this often, see
Wave, but at least in that case they were trying something novel). In short,
G+ was not true to Google's DNA.

I think my view still supports your main point though—stacks simply are not
the same for each company. Lower level stacks tend to be more similar, but at
the top they are serving unique market segments, so they are simply not
fungible.

~~~
goalieca
> but conceptually and organizationally it was a mess.

Facebook annoyed many users each time a new update came. The privacy settings
were constantly being reset. I do not believe facebook has a great UI to this
day. Google+ is not great but it does not offend me the same way.

~~~
TeMPOraL
A tangent.

> _The privacy settings were constantly being reset._

True. What privacy-minded Facebook critics often miss is that privacy settings
seem to have been constantly reset towards "more private" mode. Hell, at this
point Facebook is starting to get really annoying with those recurring
reminders about visibility of my post. I post all-public because I like it
that way, I hate the constant nagging about switching to more private mode.

~~~
tamana
Before a few years ago they were being reset to less private.

------
hyperpape
I don't want to be too dismissive because something about the article rang
true to me, but I don't know that I buy the whole central conceit that the
idea of a stack can apply as universally as this article needs it to.

Apple's networked services have often struggled. But are they really higher
level than the things Apple succeeds at? Asking whether enormous distributed
data stores are higher level than Mail.app just seems confused. It's
different, and it brings new challenges, but are they part of the same stack?
And is the data ingestion and sanitizing that Maps struggled with higher or
lower level than the client that was basically ok? You can multiply these
questions and I'm not sure you can get good answers.

~~~
Throwaway23412
I would argue that networking services, backend services, and apps are indeed
much higher level than the things Apple succeeds at (hardware, industrial
design, and operating system).

In fact, Steve Jobs never even wanted third-party apps on the iPhone
([http://www.theguardian.com/technology/appsblog/2011/oct/24/s...](http://www.theguardian.com/technology/appsblog/2011/oct/24/steve-
jobs-apps-iphone)). So, it's understandable why Apple has struggled with their
networking services, backend services, and apps: such concerns weren't even
remotely a part of Apple's DNA.

------
tn13
There was a brilliant essay by an Indian politician few years back after his
party lost the elections. Later in lecture he explain why political parties
and large companies have so much in common when it comes to failing.

His basic logic was that \- Success depends on processes \- Processes even
though might be thought of as abstract in reality are function of people at
top. \- Company gets successful because some bright guy is the rebel, he
questions status quo, persists and succeeds. \- As time goes by, the
rebellious ideas actually become conservative ideas. The rebel is now on top.
As his ideas fade he struggles to stay on top. \- He recruits people who see
the world through him, he builds processes that enforce that vision. \- This
makes it difficult for the truth to be visible to the top management. \- By
the time failure is visible it is hard to turn around the ship. \- IN SHORT:
Companies/Nations fail because someone at top did not know when to quit. \- In
the end that rebel turned conservative becomes bitter. He thinks the world
owed him something for what he achieved.

He explained who USSR examples. How a genetic scientist got promoted because
his fake research re-enforced something that Stalin had said long back and his
peers were scared to point out the fact because it might get perceived as
anti-Stalin.

I observed Blackberry very closely and it resonated to me so much. The
founders at one point blamed people for using iphone and not blackberry.

Best companies in the world are seem to be those where their top leaders quit
at their peak to make way for their successor.

~~~
dman
Would you agree with this generalization? - Best companies in the world keep
redefining the central problem they are attacking.

~~~
tn13
Yes and that redefinition very often comes from a drastic change in top
management. Ballmar was different from Gates and Satya is different from
Ballmar.

------
vonklaus
This article seems to have many correct pieces, but I don't think they
coslesce to prove the point, or at least, not entirely.

I don't think that manufacturing semiconductors are comparable to building
maps. Apple should have done a better job with maps, and even though they do
complex manufacturing, likely should have done worse at chip manufacturing.

Iirc they brought in 3rd parties to help with the chip fab, and certainly
spent more money building that core competency than maps.

I believe the author is correct that the issues is companies not fully
understanding, and consequently underestimate, what it takes to be successful
in a different arena putside their cc.

Google sees people as articles in a db. They dont understand people at all,
they dont understand design as it relates to people, and they didn't
understand that nobody needed another social network.

They probably underinvested (initially) in G+ and it was not a great product.
It didnt achieve critical mass quickly, and thus had no chance of growing as a
docial platform ever.

However, google is a lot more capable of creating something like this because
they have all the core conpetencies down.

I guess my takeaway is that the companies can in fact take these arenas, but
they underestimate the challenge. So to use a drug dealing analogy, they try
to start moving bricks amd kilos, instead of working their way up learning the
market pushing dimes and quarters.

They start too big, and when you fail big, you dont get the recovery of a
smaller failure which affords small relaunches and features.

Tldr big companies try to enter at the top, cant recover from huge public
failures, and either exit or buy in

~~~
anshublog
"I guess my takeaway is that the companies can in fact take these arenas, but
they underestimate the challenge."

Yes. And their core mistake is not understanding the real needs of the
customer because they make assumptions in a familiar territory.

------
libertymcateer
Apple is _not_ vertically integrated - Wikipedia entries to the contrary
notwithstanding. It is a grossly inaccurate statement. Up until very recently,
Apple didn't own a single factory - how can one possibly claim that they are
vertically integrated if they don't own their own means of production?

Apple is a fantastically successful software and industrial design company.
The vast majority of their production is outsourced. This is not vertical
integration.

Additionally, actually, Apple has tremendous amounts of hugely successful and
popular software.

Though I dig the underlying point of this article, that product management is
hard, I think the examples are less than good.

~~~
vacri
Where do you draw the line? If a company does own the factory that builds
their widgets, do you claim they're not vertically integrated because they
don't own the mines that get the materials, nor the transport network to get
it to and from the factory? At the other end, do they need to own the bank
with which they take payments from customers?

You don't need to own _all_ elements in a vertical stack to be vertically
integrated.

~~~
josu
Apple sells iphones. Who makes the iphones? Not Apple.

~~~
beambot
Apple designs iPhones. They manage the fabrication of parts very closely and
manage their assembly and QA more closely than most of their competitors
(like, big enough to make demands of Foxconn or cause them to build
specialized factories). Then they sell the phones directly to consumers. They
are (literally) on the Wikipedia page as an example of a vertically integrated
company:
[https://en.wikipedia.org/wiki/Vertical_integration](https://en.wikipedia.org/wiki/Vertical_integration)

I'm not saying you're wrong... but it doesn't look like you're right.

------
cturner
It's particularly funny when the stack fallacy is held by database providers,
because databases are the wrong tool for most of the jobs they get used for at
the moment.

Current usage of the database uses it as a loose, adhoc, difficult-to-
maintain, polling-based API between multiple applications.

The future perspective looks back on our time, shaking its head at the way
people use databases for everything in the same way that we shake our heads at
bloodletting.

Oracle's business model is (1) convincing people to use platforms they
shouldn't be using and then (2) selling the victims ongoing hacks and services
to work around the limitations of the model.

Amazon's software services won't be build on a database. They'll be built
using a decentralised messaging platform.

------
marshray
> “Can we compete with Intel or SAP?”

Well for one thing we know that Intel spends several $billion to open a new
semiconductor plant and has a dozen of them already.
[https://en.wikipedia.org/wiki/List_of_Intel_manufacturing_si...](https://en.wikipedia.org/wiki/List_of_Intel_manufacturing_sites)

Whereas SAP is, well, a lot of software. Which is something, but Intel needs
to make a lot of software too, and chip designs are in some ways a specialized
form of software.

So I think in some sense Intel is strictly more challenging to replicate than
SAP. (But this is probably just my misunderestimation talking. :-)

~~~
at5
You're actually quite right. Capital intensiveness is a barrier to entry.
That's why software businesses rise rapidly and fall rapidly at a greater
frequency than other types of businesses.

------
kazinator
This stack fallacy sounds very familiar. Oh, we will just have a few system
calls like open, close, read and write, some TTY and credentials related
stuff, a bit of signal handling, process control with fork, exect and wait ...
writing a shell language on top will practically just be a footnote.

~~~
anshublog
Yes many open source projects suffer from this too. We as engineers (even
former ones) often assume the what to build is "obvious". This is often not
true, if ever.

The best open source projects are ones where engineers are the end users
(customers).

~~~
marshray
Perhaps this is a feeling that, as experienced implementers, we have a big
head start on the hard parts.

Maybe this is another form of the "solution looking for a problem" phenomenon.

------
ap22213
An alternative (or maybe complimentary) theory is in Clayton M. Christensen's
innovator's dilemma. Big companies build enormous revenue bases on certain
types of technologies. Then they struggle to innovate because, by
transitioning, they eat away at their existing revenue streams.

~~~
HillRat
Actually, I don't remember that particular insight in _TID_ ; what you
describe is what I personally call "the Blockbuster effect" \-- the idea that
a company can quite rationally refuse to adopt a disruptive innovation because
its current financial structure is adapted to a particular size market with
particular revenues, costs and liabilities. In Blockbuster's case, the risk
was that aggressively taking on DVD-by-mail and later streaming would have
eaten into its revenue structure (late fees); render operationally worthless
its physical stores and thus sink their capital investment and liabilities
underwater; and shift the market to a lower-revenue level that would not
generate the cash flow necessary for Blockbuster's continued existence.

Of course, they _did_ go out of business while flailing about wildly, but to
my mind that simply underscores the importance of knowing how to gracefully
exit a market. Ironically, perpetually-beleaguered Yahoo isn't a bad model for
that.

------
anjc
I'd like some solid examples of what companies confirmed perceptions of
competitors were in the same vertical, versus the reality.

Because even the author references competency-based views of competitive
advantage, but for some reason ignores resource based views, and ignores the
fact that companies might be aware of their competences. That is to say, I'm
sure that large companies tend to _mostly_ be aware of what their competences
are based on the resources and knowledge that they have. If they don't have
marketing departments that have analyzed the ERP market, sales teams with ERP
training, tech departments with key HR, key knowledge etc etc, then I'm
certain they are very well aware of this.

Maybe some companies have had marketing missteps and have made poor strategic
and competitive decisions, however, but I really doubt that it's due to a lack
of introspection or simple analysis as described.

Also, IBM didn't "think nothing much" of the software layer. They
misunderstood the nature of power in the supply chain, and most importantly,
didn't solidify their position within the supply chain while they were
dominant.

~~~
anshublog
They think they know their competencies (database systems in one case,
virtualization in another) but often they misunderstand what customers really
value up the stack - in this example, Workday customers really don't care if
it runs Oracle or MySQL or non RDBMS at all. In AWS case, we don't care for
kvm vs VMware wars.

(I am the author of the TechCrunch post.)

------
oautholaf
A lot of the examples and counter-examples in the threads here are great, but
Microsoft in the Windows era is a great counter-example here: from operating
system to Office dominance. How did they crush Lotus and WordPerfect again?

------
fforflo
As a comment on TC says:

"What the article is referring to as stack fallacy is the work of Physics
Nobel Laureate Philip Anderson:
[https://web2.ph.utexas.edu/~wktse/Welcome_files/More_Is_Diff...](https://web2.ph.utexas.edu/~wktse/Welcome_files/More_Is_Different_Phil_Anderson.pdf)

Let's give credit where it due please."

------
dpflan
This article is funny - twisting together the ideas that a company specializes
in a product in a specific market and that other companies can use the
products / tools of other companies to develop their own unique products for a
specific market and market development and competition. The "up-the-stack"
company building something using products from "down-the-stack" has already
entered a market, gained market share, and specialized in a market in which
the "down-the-stack" company has no presence. Now, the "down-the-stack"
company sees an example of a successful product that uses their technology
that they know so well, but their company is not specialized for this product,
so it hubristically does low-hanging-fruit-snatching to try to enter the new
market. "Big companies keep failing" because they are not being innovative
based upon what's mentioned in this article; they see an easy out and enter a
market that already has an incumbent(s).

------
anshublog
I am the author. Happy to answer any questions about my post.

~~~
myth_buster
I think this could be turned into a book collating detailed case study in each
of those cases where they failed with perhaps cases where companies have made
it work like Google Fiber, Amazon's PAAS etc.

------
tokipin
Nice observation. Another way to put it is that induction is harder than
deduction.

A related factor is that larger companies tend to be more specialized
(formalized processes, specialists, focused teams/departments, and so on),
meaning they can be prematurely optimized with respect to new goals and poorly
equipped to conduct the necessary roaming.

------
a-robinson
The author claims that "Stack fallacy is the mistaken belief that it is
trivial to build the layer above yours", but then says that IBM was wrong when
they "happily allowed Microsoft to own the OS market".

Wasn't IBM a classic case of _not_ trying to build the layer above them on the
stack?

The Wikipedia page on IBM PC DOS even claims that their "radical break from
company tradition of in-house development was one of the key decisions that
made the IBM PC an industry standard".

~~~
Sammi
Oh IBM tried building it: [http://arstechnica.com/business/2013/11/half-an-
operating-sy...](http://arstechnica.com/business/2013/11/half-an-operating-
system-the-triumph-and-tragedy-of-os2/2/)

~~~
Qwertious
OS/2 was too little too late - by the time they realised what was going on,
Microsoft was already a new major player.

Didn't stop Microsoft from going out of their way to sabotage them, though.

~~~
yuhong
In fact, the history of 32-bit OS/2 is more complex than that, and involved MS
a lot more than this. It used to be my favorite topic.

------
shadowmint
Why do people keep reading TC?

Here let me make an article... wait wait... ah... "Big Companies FAIL" that
sounds like nice click bait. Now... hm, let's invent some stupid word to pad
it out how about the 'Stack Fallacy'. Programmers will dig the 'stack' part.
Yeah. Ship it!

Seriously, this article is content free.

People make products. Sometimes they work... sometimes they fail.

If you pretend you have some magical insight into why they fail or succesd
with gems of wisdom like:

    
    
        found it very difficult to succeed in what looks like a 
        “trivial to build” app  — social networks.
    

and:

    
    
        The stack fallacy provides insights into why companies keep
        failing at the obvious things —  things so close to their 
        reach that they can surely build. The answer may be that the 
        what is 100 times more important than the how.
    

Then... wow. I don't even know what to say.

Really? What you build is important?

No kidding.

Why is the top of the list this morning?

------
ogezi
A company should always focus on your strengths if not it'll be both
overstretched and unsuccessful. Great read.

~~~
justicezyx
I do not think that's the correct conclusion. The conclusion should be that
when venture into new areas, be sure to understand the market first.

~~~
2np
Or, just because you understand the underlying technology doesn't mean you
understand the market, and your previous experience will get in the way of
actually understanding the market.

If you want to go _down_ the tech stack, though, you might be successful
because you're _part of_ the market for that tech.

~~~
anshublog
Exactly. When you go down the stack like Amazon did with AWS, you know your
customer (often you). You also know the key pain points and gaps in the
market. This is especially so in "as a service" markets.

------
cbsmith
This seems more than a bit flawed. It presumes companies or anyone else think
these launches in to new markets are low risk. They are generally seen as
anything but. There is hope that leveraging existing strengths will improve
the odds, but only the idiots think they are certain to win.

------
mwnz
Do big companies really keep failing? I'm failing to see the evidence of that
assertion.

~~~
sawyer
Failing to gain traction upstream, not failing to keep the lights on.

------
annnnd
> The bottleneck for success often is not knowledge of the tools, but lack of
> understanding of the customer needs.

THIS! +1000! I would even leave out "often", or at least replace it with
"usually".

------
tuke
True enough, but there are also companies that were designed from the
beginning to have vertical integration and control much of their business from
beans to buildings. And of course I am thinking of Starbucks. (People don't
really understand the technological story of Starbucks, which has a lot to do
with their introduction of the vacuum packs to get their coffee across the
country, and overcoming the challenge of brewing coffee on passenger jets.)
Mostly for the better, they decided long ago to own as much of the stack as
they could.

------
ljw1001
Some insights perhaps, but the claim that this is "Why big companies keep
failing" is way overblown

~~~
anshublog
agreed. Its just hard to have title that says "Why some companies fail
repeatedly".

All of us are competing with "17 people under 30"... listicles.

------
peter303
The counter example would be Apple when they introduced music players and
music distribution. That was far away from their core business of PCs. The
smartphone was less of a stretch, because it was a potential extension of the
iTouch comunications computer.

------
dkarapetyan
Not so much stack as legacy. Have you seen legacy architectural decisions?
They're impossible to get rid of. It's surprising how much the initial
architecture can hinder change.

------
jackgavigan
_> Product management is the art of knowing what to build._

And in what order.

------
LCDninja
When reading the article I was reminded about Warren Buffets message re
identifying and staying within your "circle of competence."

------
shim2k
Peter Lynch writes about it in his book "One Up on Wall Street". He calls it
'Diworsification'.

------
a-dub
That's why when I build a company, I'm going to use a hash table.

