
Hidden costs of constantly shipping new things - michalbugno
https://www.mindtheproduct.com/the-hidden-costs-of-constantly-shipping-new-things/
======
digitallogic
Interesting! I especially enjoyed the insight into the evolution of the
company. Though with this insight I have slightly different conclusions than
the author:

1\. A bias to ship and a bias to ship _new_ things are not one and the same. A
lot of the problems, such as a failure to iterate on existing
products/feature, sound very much a product of the later, not the former. If
anything, the issue "Insufficient Iteration" is probably not correctable
without a bias to shipping.

2\. A bias to ship and a bias to ship things that impact your customers are
not one and the same. There is a note about an early shift to micro-services.
I can't speak to this organization, but, generally speaking, spending time on
internal engineering work to the detriment obvious missing features is a
common issue with early stage companies.

3\. A bias to ship and a bias to ship to the right customers. Specifically in
regard to the high value, high demand customers.

4\. A bias to ship and... idk what to call this? "The product specs were well
thought through, sometimes crafted for months." TBH, the problem with this one
feels like a lack of a bias to ship.

As described, I think the real culprit was a lack of or poor prioritization.
FWIW, I suspect the author and I may actually be in violent agreement as I did
find my self nodding with most of his lessons learned. Though I'd be careful
about letting too much hindsight bleed in (eg - do situations that'd be
improved by more decision documentation justify the effort of documenting all
product decisions, especially in the early phase when the product is rapidly
evolving)?

~~~
RandoHolmes
I think the problem is that the leaders in the company (technical or
otherwise) didn't recognize that software systems are like children: They have
phases.

The ship it constantly is probably ok when you're small, but at some point you
need to become more mature than that, both as an organization, and as an
approach to the software itself.

------
montagg
I work at a company that grew from a few hundred to a few thousand people in a
few years' time, and there was an identical mindset and identical problems
over that growth period. I started during that growth period, and 10 years
later, we're still digging ourselves out of those mistakes.

One reason for this was how heavily we prioritized first mover advantage. We
pushed extremely hard to get into new markets, and while strategically this
worked out extremely well for us, it meant that later development was
hamstrung by earlier decisions. Any iteration on old features almost always
meant rebuilding it entirely, and we've only just started to get to a place
where previous efforts to do this are now paying off in increased velocity.

I think there's an argument that this _could_ be good, since you're only
improving features when there's some other value you can ship. However, I
think it's short-sighted, because it means prioritizing small iterations has
an unnecessarily large cost, simply because that cost wasn't paid down slowly
over time. While I do think prioritizing new markets early on was a good call,
that mindset bit us in the ass in a big way when we tried to ship a very
flashy new feature later in our growth cycle that crashed and burned because
of tech debt, and it took us over a year to fix everything to release it. If
we had taken a mindset of building for the long term five years ago, which was
already years after we'd gotten to a comfortable place in revenue and
marketshare, we would probably have paid off most of the major tech debt by
now.

I honestly don't know how a company can get past the addiction to shipping, at
least not during its growth period. It can be strategically necessary, and I
imagine in only a two-year period at the early stages of a company, like
what's in this article, it's impossible to avoid. But leaders must have in the
back of their heads the idea that they will need to start tipping the scales
toward standardization, building to last, and building infrastructure as well
as features sooner than they'd probably like to avoid larger problems in the
future. And I suspect the only way leaders will really take that seriously is
if they've gone through it before.

EDIT: Clarified a few points.

~~~
jamil7
Apart from the one catastrophic failure you mention it sounds like, from a
business perspective this worked? You were able to extend your tech debt long
enough to start generating money and are now in a position to pay it back.
From an engineering perspective I agree it's a grind to pay all this back and
lose so much velocity. Are their business strategies that favour slower more
careful engineering practises? mission-critical systems come to mind but maybe
also nth-movers could be put in there? Where you know what you're building and
competing with but want to make the best version of X.

~~~
codemac
> Are their business strategies that favour slower more careful engineering
> practises?

This is a false dichotomy, and there is plenty of research (and experience)
that shows shipping higher quality software actually takes less time and
money.

~~~
LanguageGamer
I want to believe this. Do you have any citations?

My experience would suggest shipping higher quality software is a lot cheaper
in the long run, but has up-front costs that often times startups can't
afford.

~~~
sundbry
You can afford them if you don't have project managers or sales guys begging
for their demoable results of new shiny thing and promising it to others too
early and too often.

------
lbriner
This is kind of interesting but it sounds like the perpetual worry of the
perfectionist developer. Of course you end up with tech tech, you develop new
things when you could improve existing things, you deploy something that could
be iterated but you don't.

The truth is that the world is a complex place and you don't always know
whether you can keep your existing customers by improving what you already
have or get new customers to decrease the risk of declining income. It is hard
to measure what is acceptable tech debt and what is worth addressing.

In one sense, the proof is in the income. If you can help customers with an
80% OK product then you just need to live with the dodgy 20% bit.

I agree with the danger of big customers though! If you ever think something
is a way to avoid hard decisions, pivots and annoying some of your customers,
then it is too good to be true ;-)

~~~
michalbugno
I'm far from being a developer perfectionist, I wouldn't spend 10 years in
this company if I was one, trust me (btw we fired a lot of them along the
way).

The point of the article is not to convince anyone to slow down as much as
possible and work on bugs/etc. It's just that at some point (3, 4 years in?)
there comes a time that you just have to put more effort into the things I
described, otherwise it gets complicated _fast_.

Obviously there is a tradeoff, my opinion is that this tradeoff wasn't
correctly balanced (especially further down).

~~~
sreekotay
I think the article starts out sounding that way --- but reading all the way
through, it's about how to DO BETTER if you want to ship more often (the key
take aways from the article being: document, iterate/revisit, and be
intentional)

------
renewiltord
Every company suffers under the weight of the debt of being feature focused.
That's because everyone who didn't do that isn't alive to tell the tale.

------
trentnix
There's a lot of insight packed into this post, and really it transcends way
beyond the mindset of _constantly shipping new things_. This particular line
resonated with me:

 _Big customers certainly sounded good on paper, but they came with a cost
(complexity and a very hands-on relationship) that we weren’t really prepared
for, and that we didn’t manage to put boundaries on fast enough._

This is an issue I've dealt with more than once in my career. It seems to
occur especially when an organization values the _sale_ more than it values
the _business_. It's virtually impossible to successfully apply boundaries
after the customer has been trained that there are none. Irrespective of how
carefully you do it or how much you try to reassure your customer, it always
feels like you are taking value away. I've encountered this recently in an
organization that values customer happiness, which it calculates as an absence
of customer complaints, more than customer success. The squeaky wheels get all
the grease at the expense of just about everything else.

And the software inevitably has become a maze of settings and IF-THEN-ELSE
blocks and SWITCH statements, increasing the cognitive load required to make
what should be simple changes. When new features are added, the impulse is to
code defensively and build inevitable customization into everything up front.
And further down the spiral of technical debt and increasing complexity we
go...

------
Slartie
I always wonder why nobody appears to try a hybrid model between product and
project development when it comes to satisfying large customers which need
very specific customizations that often don't fit well into a product
development roadmap aiming at satisfying a more generic audience.

Like: don't just add weird features that make sense for just one customer, but
are eventually delivered to all customers, but instead maintain a "core
product" which provides a lot of generic, core functionality, but which by
itself isn't actually usable and just serves as basic building blocks to
construct the actual end products which are then built and packaged
individually for each (big) customer. Each of these "end products" is a
development project in itself, has its own team, its own codebase, but they
all draw from a core set of base functionality developed and maintained by
another dedicated team that is not directly in contact with any of the
customers (or at least not permanently).

This - at least in theory - prevents custom functionality for specific
customers from polluting the common core product that aims to please a general
audience, while still allowing for the necessary freedom to satisfy big
customers demanding custom functionality, but also offering big checks in
return.

~~~
pyromine
I mean you're describing what most Enterprise Resource Planning deployments
look like. There's a lot of enterprise software that follows this model

~~~
Slartie
You are right; probably I have to state the question in a more precise way,
which would be: why is it such that this model seems only to be used in huge
"enterprisey" contexts? The model of course depends on a certain modularity of
the software in question, but hasn't that been a core concept not just for the
biggest players, but also the smaller shops for a while now?

~~~
rahimnathwani
Implementation costs a lot, not just in coding, but in all the extra work that
goes into sales, gathering requirements, training etc. Whoever nominally pays
for all that work, it translates into a high price tag for the customer. Only
large enterprises will get enough value from the customisations for that high
price tag to be worthwhile.

All else being equal, a smaller company will spend a lower absolute amount to
work around imperfections, and get lower value from any product overall. So of
course they will not be willing to pay as high a price. Hence, for commodity
use cases, use commodity software rather than fully/semi-customised.

------
bleonard
When I was running a product team, I often talked about "nuance" being the
enemy. Every difference (or arrow in a flowchart) was a new piece of product
debt to have to think about later.

This article is in that direction. > … the difficulties started piling up
fast, and before we knew it we had to deal with a lot.

------
troelsSteegin
I think this is a solid critical reflection that at least in enterprise
relates well.

When it comes to your customers, I believe that you want to understand as much
as you can, as early as you can, about what they are actually trying to do and
about how they are actually faring with what you have sold them. I'd argue
that the success of your initial customers is more important than the features
for your next deal, and that feature investments should reflect that.

The problem, as the OP said, is in estimating the difference between the
incremental feature requirements of your initial customers and what your
strategic feature set will be. There may be no difference, and your strategy
is to ship features in order satisfy each new customer as you get it. Perhaps
you can design for that. Chances are though that your first customers are not
wholly repsentative of your strategic, target market.

So, in the early stage, you have to make choices about allocating dev for
tactical "now" or for strategic "later". In the early stage, if there is not
enough "now", there is no "later". The question is when to begin budget for
later. I'll argue that you are attempting to build a business, not a suspense
movie, and that you budget a fixed amount for "later" from day one. Plan to
succeed.

I agree with the recommendation of a strong Professional Services function as
an buffer for dev, but will note that culturally you have to work hard to not
silo. Your Support and ProServe teams become a primary channel of learning for
dev. As a business, though, you may have to give away ProServ hours.

All in all my biggest takeaway here is that you attempt to "hire" customers
strategically. As you journey together, will this customer want to go where
you will want to go? What time is it when your "Big Customer" sits on your
roadmap? ...

"Hire" customers strategically is easier said than done when there's salaries
on the line -- accordingly look at how you compensate Sales and budget away
from early stage big deals.

------
hn_throwaway_99
Also could be titled "Hidden costs of moving to microservices because the cool
kids are."

Very glad (most of) the cool kids have re-evaluated their stance on
microservices.

~~~
michalbugno
I have a plan to write a short article about specifically this topic at Base.
I personally fucked up a ton of things with microservices there :)

------
pkteison
What an awful article. Expensive conclusions are drawn with no apparent
consideration for their cost. The experience of a company which appears to
have successfully balanced complicated tradeoffs in choosing features is
discounted because there were bugs. And almost no examples or useful stories
about any detail, or suggestions on how to do it better other than to go
(much) slower and to spend (much) more or to not take rich customers money.

I suspect good arguments could be made for these things, but they need to
acknowledge and justify their cost.

~~~
michalbugno
Author here. Sorry you didn't like it :)

It's tough to suggest detailed solutions to a problem which is in general very
vague. Btw I hope the article shows my appreciation for what the company
achieved.

One small insight from Uzi (CEO + founder) when I discussed this with him
(after publishing article): Base should've focused on much less features but
with greater detail. It somehow confirms my guess that we didn't work on
existing features as much as we should, but we "spread too thin".

------
Covzire
Oh good, I was afraid this was going to be an intervention for me regarding my
steady stream of almost daily deliveries from Amazon/HD/Walmart/Ebay etc.

------
jmull
Hm... this article actually seems like a better argument to obsessively focus
on constantly shipping new things than the opposite.

~~~
coldcode
Being bought out for large money often covers up the nightmare that would have
been, and shifts it to someone else.

------
29athrowaway
[https://en.wikipedia.org/wiki/Feature_creep](https://en.wikipedia.org/wiki/Feature_creep)

[https://en.wikipedia.org/wiki/Scope_creep](https://en.wikipedia.org/wiki/Scope_creep)

~~~
arbitrage
That's not a particularly good way to encourage discussion.

What is it specifically about Feature & Scope creep that you would like to
say?

~~~
29athrowaway
I think this article is great and it does a good job characterizing what
happens to products that fall victim to feature creep.

