Interesting! I especially enjoyed the insight into the evolution of the company. Though with this insight I have slightly different conclusions than the author:
1. A bias to ship and a bias to ship new things are not one and the same. A lot of the problems, such as a failure to iterate on existing products/feature, sound very much a product of the later, not the former. If anything, the issue "Insufficient Iteration" is probably not correctable without a bias to shipping.
2. A bias to ship and a bias to ship things that impact your customers are not one and the same. There is a note about an early shift to micro-services. I can't speak to this organization, but, generally speaking, spending time on internal engineering work to the detriment obvious missing features is a common issue with early stage companies.
3. A bias to ship and a bias to ship to the right customers. Specifically in regard to the high value, high demand customers.
4. A bias to ship and... idk what to call this? "The product specs were well thought through, sometimes crafted for months." TBH, the problem with this one feels like a lack of a bias to ship.
As described, I think the real culprit was a lack of or poor prioritization. FWIW, I suspect the author and I may actually be in violent agreement as I did find my self nodding with most of his lessons learned. Though I'd be careful about letting too much hindsight bleed in (eg - do situations that'd be improved by more decision documentation justify the effort of documenting all product decisions, especially in the early phase when the product is rapidly evolving)?
I think the problem is that the leaders in the company (technical or otherwise) didn't recognize that software systems are like children: They have phases.
The ship it constantly is probably ok when you're small, but at some point you need to become more mature than that, both as an organization, and as an approach to the software itself.
I work at a company that grew from a few hundred to a few thousand people in a few years' time, and there was an identical mindset and identical problems over that growth period. I started during that growth period, and 10 years later, we're still digging ourselves out of those mistakes.
One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions. Any iteration on old features almost always meant rebuilding it entirely, and we've only just started to get to a place where previous efforts to do this are now paying off in increased velocity.
I think there's an argument that this could be good, since you're only improving features when there's some other value you can ship. However, I think it's short-sighted, because it means prioritizing small iterations has an unnecessarily large cost, simply because that cost wasn't paid down slowly over time. While I do think prioritizing new markets early on was a good call, that mindset bit us in the ass in a big way when we tried to ship a very flashy new feature later in our growth cycle that crashed and burned because of tech debt, and it took us over a year to fix everything to release it. If we had taken a mindset of building for the long term five years ago, which was already years after we'd gotten to a comfortable place in revenue and marketshare, we would probably have paid off most of the major tech debt by now.
I honestly don't know how a company can get past the addiction to shipping, at least not during its growth period. It can be strategically necessary, and I imagine in only a two-year period at the early stages of a company, like what's in this article, it's impossible to avoid. But leaders must have in the back of their heads the idea that they will need to start tipping the scales toward standardization, building to last, and building infrastructure as well as features sooner than they'd probably like to avoid larger problems in the future. And I suspect the only way leaders will really take that seriously is if they've gone through it before.
Apart from the one catastrophic failure you mention it sounds like, from a business perspective this worked? You were able to extend your tech debt long enough to start generating money and are now in a position to pay it back. From an engineering perspective I agree it's a grind to pay all this back and lose so much velocity. Are their business strategies that favour slower more careful engineering practises? mission-critical systems come to mind but maybe also nth-movers could be put in there? Where you know what you're building and competing with but want to make the best version of X.
Well, yes, I am saying from a business perspective this worked. but I also think we got lucky. Around five years ago, a little before the high profile failure, we were starting to lose marketshare, and though we've been able to claw most of it back, I think it's far more due to missteps of our competitors than our own performance. Had we invested in more sustainable development earlier, I think we would have been able to move faster to take advantage of new opportunities, diversify our revenue streams earlier (adding to them now is hamstrung in large part because of tech debt), and head off competitors faster.
In fact, once we started investing in that kind of development, we did do that with a successful new feature that we developed much more quickly than we could have in the past, and it's because of that success that leaders are finally starting to prioritize building standardized and reusable infrastructure.
EDIT: I think another way to say this is, once we knew our strengths and how we'd be successful in the future, we should have operationalized that and hardened it, so we'd have a sustainable advantage over competitors. Instead, we kept the "move fast, be flexible, and throw man-hours at problems" mentality too long, and it put us at risk.
Could there be a hybrid approach? Have teams that focus on being a first mover and shipping and then once there's a product on the market, you have a secondary team focusing on fixing the tech debt early while the other team continues to iterate
At my old company we had an innovation consultant come in and tell us that kind of approach was nearly impossible to manage in the longer term. Apparently teams dedicated to holding the fort grew bitter and had much lower retention than teams on greenfield work.
Paying back the debt, he said, usually requires too much knowledge of a system to cycle teams in and out consistently enough to ensure everyone gets a piece of the new versus the old to keep them engaged.
Whether or not that was backed up by hard data, I'm not sure. It was an interesting angle I hadn't previously thought of though.
Calling it “just holding the fort” would indeed be disadvantageous. If instead the teams would be empowered and rewarded for also keeping things running and improving on them it will attract the right kind of people which do grow the product and don’t get bitter. However, I have yet to meet an engineer who is comfortable maintaining a sunset product and in the mean time getting only flak about the performance of the old sunset application/feature or their own performance.
> Apparently teams dedicated to holding the fort grew bitter and had much lower retention
As someone who found themself in the "holding the fort" team, I can confirm that this analysis is at least anecdotally true, although the terms I preferred at the time for the two teams were Morlocks and Eloi.
“Apparently teams dedicated to holding the fort grew bitter and had much lower retention than teams on greenfield work.“
Very true. The good performers will be “rewarded” doing the things that are really important: keeping things going. And then you have low performers and interns doing the new stuff.
Worse than the fact that those holding the fort would leave, the best of those would leave (either to another company or just as often to the exciting new projects) leaving behind those that can't leave and very rapidly you end up with a not that great group which management doesn't notice or deal with.
Early on, I don't think so. You need all hands on deck to move fast. But I do think you need to start establishing those Good Habits way earlier than they might be immediately valuable. For example, getting good at product and technical documentation before you have a huge company. It's so much harder to establish those habits with more people, so you want to have that built in so new folks join, and doing that stuff right is just part of the culture. A little effort today saves a ton of effort to establish those habits later. You also end up with more flexibility: Person A doesn't need to constantly support Feature A because they built it and they're the only one who knows how it works, so you're getting more value out of their time.
In the long-term, I do think product-wide you always have teams at different levels of maturity. Some know more about the problems they're solving and how best to solve them, others know less. Some will move fast, some will build to last. The more a team knows about their problem, or the more a problem cuts across teams, the more I think they should build to last, so investment in those areas scales over a large group or important areas of the business.
Other people are also commenting, but IME, completely disassociating “making the think from “making the thing so that it can be supported” is a recipe for trouble.
There’s not an easy fix though, as most of the time I see orgs just punt at some point and allow the dev shop to write off all their tech debt by reassigning responsibility to an ops team.
> Had we invested in more sustainable development earlier, I think we would have been able to move faster to take advantage of new opportunities, diversify our revenue streams earlier (adding to them now is hamstrung in large part because of tech debt), and head off competitors faster.
While that's probably true, it's a difficult balance to know when exactly you should start moving back towards the better development practices. Too soon and you die, too late and you can no longer move because you're too inefficient.
> Are their business strategies that favour slower more careful engineering practises?
This is a false dichotomy, and there is plenty of research (and experience) that shows shipping higher quality software actually takes less time and money.
> This is a false dichotomy, and there is plenty of research (and experience) that shows shipping higher quality software actually takes less time and money.
I believe you when the business knows exactly what it's building. In the context of the article and the comment I was responding to, building high quality software wouldn't have helped them early on, thats kinda the whole point of tech debt and the discussion here. Knowing when is the right time to start changing the engineering culture and how easy (or hard) it is to do that.
I want to believe this. Do you have any citations?
My experience would suggest shipping higher quality software is a lot cheaper in the long run, but has up-front costs that often times startups can't afford.
You can afford them if you don't have project managers or sales guys begging for their demoable results of new shiny thing and promising it to others too early and too often.
My companies first 5 hires were junior developers. This cost a fair packets (because there were 5 of them!) and led to pretty low quality software (because they were all junior and didn't know what they were doing).
They could have shipped higher quality software quicker by hiring 2 senior developers for similar cost.
Startups bias toward hiring some less experienced developers by their nature, so they don't always know how to build quickly and solidly. The up front cost is either learning skills of abstraction from scratch, or hiring people who have.
These points aren't as opposed as you're making them seem. Throughput is better with careful planning and good design is an investment that pays off later.
Slapping it together pays off today but at the cost of acquiring a debt that costs lower average velocity and lower agility until its repaid.
A lot of small businesses make the decision to take on debt to move quickly at the beginning because the most important thing is initial growth and proving that you can actually make money.
I think that they're saying, in retrospect, they could have started making some payments on the debt a bit earlier when the business had scale and sustainability. And that in turn this would have avoided a lot of unpleasantness.
It's hard to change culture; you tend to keep doing what worked before even if it's not optimal anymore. Yesterday's adaptive behavior is today's unfortunate practice.
I worked at a small company that was always chasing the next potential sale. Potential customer mentions feature A and all of a sudden all development shifts towards developing that feature. The vast majority of the time the potential customer never became a customer. In the meantime competition focused on their own plan of features and our product wound up with features no one ever wanted and was missing critical functionality everybody demanded.
I've had the exact same experience. Throw in a large dose of "no repeat customers" due to us prioritizing Feature A for Potential Customer B over fixing existing gaps that affected actual customers, and it was a never-ending parade of haphazardly created, bug-riddled new features that never got improved.
Made every single day a death march and a company that survives purely on the sunk cost fallacy on the part of the investors. They've already dropped $MILLIONS into building this thing, and the CEO swears that this next big deal will be what finally pushes the company into the stratosphere. As miserable as it was to work there, I'm honestly impressed they're able to keep that zombie moving.
Yup, and the requesting "customer" always convinces your CEO with the same argument: "Just add this feature, sell it to us, then it's part of your product forever and you can sell it to many other customers! We're paying for your NRE!" and CEO falls for it!
The company I worked for didn't even need that incentive just the potential to make a sale! We added features just to complete bullet points on a marketing sheet even if we never made a sale of the feature.
The core issue, from what I've seen, is one of incentives. Many leaders at many companies are compensated on growth and short term factors. It's hard to find a leader that is willing to take a 5 year look, or even find a leader that is planning on being around 5+ years.
The leadership at my publicly traded company is a revolving door of short term hacks for promos followed by a quick pivot to the competition or another company. It often feels like the C-Suite at most companies thinks at most, 2 years out.
I just can't see many leaders thinking 5 years out.
I would add that when it’s not a revolving door it’s frequently someone who used to be technical and still fancies themselves as an elite coder. That in turn leads to low level decisions (how and not what) being made at the top. As a result, you’re chasing “cool” rather than “best long term decision” in service of the ego of someone who is already being appropriately compensated monetarily.
My previous employer prioritized shipping, won the market while having both more features and more technical debt than competitors, bought those competitors out as they started failing, and then migrated their clients over and threw their beautiful (but less featured) systems right into the garbage. To this day, I sometimes think back to how a particular competitor had the right schema for solving current problems, but they didn't even survive to the current day, which takes precedence.
> One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions.
I've seen so many developers over the years who view having to rewrite that module because it can't scale well enough anymore as a failure, but I call it a success. Software is meant to be tumultuous, things SHOULD be getting replaced.
In another post in this thread I mentioned that software systems are like children in that they have phases. Assuming that your company and software team is dedicated to fixing those issues (slowly, beside new features), then I think you're probably in a healthy spot despite the pain.
An observation is that business interests must ALWAYS trump technical interests when those business interests are critical. No business = no technical. But at some point when the company becomes stable that balance has to start shifting back towards the technical or the company itself becomes ridiculously inefficient.
This is kind of interesting but it sounds like the perpetual worry of the perfectionist developer. Of course you end up with tech tech, you develop new things when you could improve existing things, you deploy something that could be iterated but you don't.
The truth is that the world is a complex place and you don't always know whether you can keep your existing customers by improving what you already have or get new customers to decrease the risk of declining income. It is hard to measure what is acceptable tech debt and what is worth addressing.
In one sense, the proof is in the income. If you can help customers with an 80% OK product then you just need to live with the dodgy 20% bit.
I agree with the danger of big customers though! If you ever think something is a way to avoid hard decisions, pivots and annoying some of your customers, then it is too good to be true ;-)
I'm far from being a developer perfectionist, I wouldn't spend 10 years in this company if I was one, trust me (btw we fired a lot of them along the way).
The point of the article is not to convince anyone to slow down as much as possible and work on bugs/etc. It's just that at some point (3, 4 years in?) there comes a time that you just have to put more effort into the things I described, otherwise it gets complicated fast.
Obviously there is a tradeoff, my opinion is that this tradeoff wasn't correctly balanced (especially further down).
I think the article starts out sounding that way --- but reading all the way through, it's about how to DO BETTER if you want to ship more often (the key take aways from the article being: document, iterate/revisit, and be intentional)
Firing perfectionists is a strange signal. If the customers are sufficiently large, you should do everything possible to please them; beyond revenue, they are themselves an asset (sometimes a company is acquired for this only). In many industries it can take years to establish yourself as an approved vendor for a large org, and especially to a large enterprise company, this can be worth more than the underlying business. Not suggesting that’s the case for Base, but if you’re not gunning for an IPO, your exit strategy isn’t going to be one that necessarily resonates with a product or eng team.
I worked at a large web development company. The one thing which continually hamstrung the company was we had several large customers who were paying us a good chunk of money to maintain their web properties. They sucked up so much of our resources to keep them happy that other smaller customers suffered as a result.
The amount of churn we had was always the smaller clients who were still paying good money but felt neglected and would be open to changing vendors.
We went out of our way to mind the big clients and thought if we lost these few big clients it would so dramatically adverse the companies bottom line, we couldn't afford to lose them. This resulted in a situation where they were able to hold our company hostage, make unreasonable demands on time and energy and hamstrung our ability to keep our more loyal clients (who only bothered us on occasion) happy.
It was a good lesson there is a danger when you have a small company servicing a much larger company that you get into a position where your fear of losing them is a detriment to your other, more faithful clients.
There's a lot of insight packed into this post, and really it transcends way beyond the mindset of constantly shipping new things. This particular line resonated with me:
Big customers certainly sounded good on paper, but they came with a cost (complexity and a very hands-on relationship) that we weren’t really prepared for, and that we didn’t manage to put boundaries on fast enough.
This is an issue I've dealt with more than once in my career. It seems to occur especially when an organization values the sale more than it values the business. It's virtually impossible to successfully apply boundaries after the customer has been trained that there are none. Irrespective of how carefully you do it or how much you try to reassure your customer, it always feels like you are taking value away. I've encountered this recently in an organization that values customer happiness, which it calculates as an absence of customer complaints, more than customer success. The squeaky wheels get all the grease at the expense of just about everything else.
And the software inevitably has become a maze of settings and IF-THEN-ELSE blocks and SWITCH statements, increasing the cognitive load required to make what should be simple changes. When new features are added, the impulse is to code defensively and build inevitable customization into everything up front. And further down the spiral of technical debt and increasing complexity we go...
I always wonder why nobody appears to try a hybrid model between product and project development when it comes to satisfying large customers which need very specific customizations that often don't fit well into a product development roadmap aiming at satisfying a more generic audience.
Like: don't just add weird features that make sense for just one customer, but are eventually delivered to all customers, but instead maintain a "core product" which provides a lot of generic, core functionality, but which by itself isn't actually usable and just serves as basic building blocks to construct the actual end products which are then built and packaged individually for each (big) customer. Each of these "end products" is a development project in itself, has its own team, its own codebase, but they all draw from a core set of base functionality developed and maintained by another dedicated team that is not directly in contact with any of the customers (or at least not permanently).
This - at least in theory - prevents custom functionality for specific customers from polluting the common core product that aims to please a general audience, while still allowing for the necessary freedom to satisfy big customers demanding custom functionality, but also offering big checks in return.
Is there much written about the architectural models that enable this kind of customisation. This is highly relevant to my life right now, and I'm struggling to find much to read about how to build a core product that's extensible by third parties. The Apache web server, with its request lifecycle and chained handlers, is my mental model for that. But there must be other models too, and I'd like to see them described in the hopes that one leaps out as a good fit for our product and market.
You are right; probably I have to state the question in a more precise way, which would be: why is it such that this model seems only to be used in huge "enterprisey" contexts? The model of course depends on a certain modularity of the software in question, but hasn't that been a core concept not just for the biggest players, but also the smaller shops for a while now?
Implementation costs a lot, not just in coding, but in all the extra work that goes into sales, gathering requirements, training etc. Whoever nominally pays for all that work, it translates into a high price tag for the customer. Only large enterprises will get enough value from the customisations for that high price tag to be worthwhile.
All else being equal, a smaller company will spend a lower absolute amount to work around imperfections, and get lower value from any product overall. So of course they will not be willing to pay as high a price. Hence, for commodity use cases, use commodity software rather than fully/semi-customised.
When I was running a product team, I often talked about "nuance" being the enemy. Every difference (or arrow in a flowchart) was a new piece of product debt to have to think about later.
This article is in that direction.
> … the difficulties started piling up fast, and before we knew it we had to deal with a lot.
I think this is a solid critical reflection that at least in enterprise relates well.
When it comes to your customers, I believe that you want to understand as much as you can, as early as you can, about what they are actually trying to do and about how they are actually faring with what you have sold them. I'd argue that the success of your initial customers is more important than the features for your next deal, and that feature investments should reflect that.
The problem, as the OP said, is in estimating the difference between the incremental feature requirements of your initial customers and what your strategic feature set will be. There may be no difference, and your strategy is to ship features in order satisfy each new customer as you get it. Perhaps you can design for that. Chances are though that your first customers are not wholly repsentative of your strategic, target market.
So, in the early stage, you have to make choices about allocating dev for tactical "now" or for strategic "later". In the early stage, if there is not enough "now", there is no "later". The question is when to begin budget for later. I'll argue that you are attempting to build a business, not a suspense movie, and that you budget a fixed amount for "later" from day one. Plan to succeed.
I agree with the recommendation of a strong Professional Services function as an buffer for dev, but will note that culturally you have to work hard to not silo. Your Support and ProServe teams become a primary channel of learning for dev. As a business, though, you may have to give away ProServ hours.
All in all my biggest takeaway here is that you attempt to "hire" customers strategically. As you journey together, will this customer want to go where you will want to go? What time is it when your "Big Customer" sits on your roadmap? ...
"Hire" customers strategically is easier said than done when there's salaries on the line -- accordingly look at how you compensate Sales and budget away from early stage big deals.
What an awful article. Expensive conclusions are drawn with no apparent consideration for their cost. The experience of a company which appears to have successfully balanced complicated tradeoffs in choosing features is discounted because there were bugs. And almost no examples or useful stories about any detail, or suggestions on how to do it better other than to go (much) slower and to spend (much) more or to not take rich customers money.
I suspect good arguments could be made for these things, but they need to acknowledge and justify their cost.
It's tough to suggest detailed solutions to a problem which is in general very vague. Btw I hope the article shows my appreciation for what the company achieved.
One small insight from Uzi (CEO + founder) when I discussed this with him (after publishing article): Base should've focused on much less features but with greater detail. It somehow confirms my guess that we didn't work on existing features as much as we should, but we "spread too thin".
Oh good, I was afraid this was going to be an intervention for me regarding my steady stream of almost daily deliveries from Amazon/HD/Walmart/Ebay etc.
1. A bias to ship and a bias to ship new things are not one and the same. A lot of the problems, such as a failure to iterate on existing products/feature, sound very much a product of the later, not the former. If anything, the issue "Insufficient Iteration" is probably not correctable without a bias to shipping.
2. A bias to ship and a bias to ship things that impact your customers are not one and the same. There is a note about an early shift to micro-services. I can't speak to this organization, but, generally speaking, spending time on internal engineering work to the detriment obvious missing features is a common issue with early stage companies.
3. A bias to ship and a bias to ship to the right customers. Specifically in regard to the high value, high demand customers.
4. A bias to ship and... idk what to call this? "The product specs were well thought through, sometimes crafted for months." TBH, the problem with this one feels like a lack of a bias to ship.
As described, I think the real culprit was a lack of or poor prioritization. FWIW, I suspect the author and I may actually be in violent agreement as I did find my self nodding with most of his lessons learned. Though I'd be careful about letting too much hindsight bleed in (eg - do situations that'd be improved by more decision documentation justify the effort of documenting all product decisions, especially in the early phase when the product is rapidly evolving)?