Very quickly leads to a race to the bottom where executives settle on tried-and-true strategies for driving revenue: give Google & Facebook lots of money for paid leads, spend a lot on salespeople to convert potential customers, nickel & dime them for everything, work your engineers to the bone to deliver the features that salespeople promise, and flip the company before the accumulated technical, management, and oftentimes financial debt kills it. Ultimately, if you want durable success, you need to take some strategic risks and invest in projects where you don't know the outcome before starting. But that goes against nearly every emotional circuit in how we're wired, so few executives can do it.
But I don't know of any companies that were killed by technical debt.
Other examples might include tolerating an aggressive status-based "brogrammer" culture, institutionalizing long working hours, or instituting gatekeepers that can block launches to avoid specific issues that have bitten you in the past.
I've worked at two startups that were killed by technical debt, and founded one. The way it usually manifests is that the startup fails to get to product/market fit before the complexity of its codebase prevents any further progress. These failures usually happen early in the startup's lifecycle (before many people have heard about them) and are often chalked up to failure to find a market, but the reality is often that if they could've iterated in days instead of months they would've found a winning product. Frequently a competitor started a few years later comes in with a simpler approach, leveraging building blocks that have already gotten mainstream vetting, and takes the market. (For more prominent examples, there's projects like General Magic, Apple Copland, Windows Longhorn, Chandler, and Friendster.)
Here’s a good example. Many modern websites have task queues to perform background tasks. So, too, does Mediawiki. You may assume it uses RabbitMQ or another queueing software, but nope, it stores the queue in MySQL (acceptable) and by default it runs cronjobs at the end of requests (less acceptable.) As your wiki grows, these jobs take longer and longer and happen more and more. Suddenly you are wondering why pages hang and you always run out of PHP FPM workers.
Then you look at the stack traces and it makes more sense... so you go to configure cronjobs. Except, there’s not really a great fix. You can tune the parameters so that the queue effectively never progresses, but then you have to run the jobs manually - which is fine, but there’s no task running daemon or anything. Actual cronjobs work but are catastrophic for responsiveness. So... one of the recommended solutions is a shell script that loops and runs the runJobs.php file repeatedly. This solution works, but from an operations standpoint it’s not a wonderful solution. You probably want at least some visibility into what’s going on, and bash scripting is hardly the best platform for that kind of thing.
Of course, this is just one microcosm of Mediawiki, you can run into one technical debt related issue per day and you’d not run out for years.
> But I don't know of any companies that were killed by technical debt.
I would imagine the failures of those companies are attributed to other causes. Is it hard to imagine a company with a low bus factor being in an unrecoverable state if a key person leaves? The time it takes to get a new person up to speed could cause the business to cede ground to a competitor, just as a for instance.
I think a good analogy is that technical debt is like having a poor diet - if you don't fix it you end up with diabetes and other issues. Sure they're treatable, but it becomes harder and harder to continue innovating.
When all is said and done, you don't know for sure if that poor diet was what lead to a companies demise - but there's no way it helped them.
Ultimately, if the people in the org want to push the work to xyz instead I would let them. Maybe xyz is as good as they think, but probably everyone in that room will enter a slog project/feature.
When someone makes a threat your default behavior shouldn't be to back down.
What managers dislike is a lack of transparency. "Research" is very vague. How can anyone tell what you were researching, or what the result of it was?
A very clear spec doc is a useful tool - almost like writing an essay - to demonstrate what your research uncovered.
Research is in many ways naturally more unstructured than other dev tasks and people don’t get too much practice at doing or delegating it, so it can be fertile ground for organizational misunderstandings. Structure agreed upon up front can help all that.
I suggested formalizing and structure as words because new features/products/projects are a common enough occurrence that we have standard procedures for it.
Here's what your manager is worried about:
1. Will the job be done well?
2. How long will this take?
3. Are you on-task, or are you stuck, overwhelmed, going rogue?
1. I have a plan, and here is the plan.
2. Here is my progress in executing this plan.
3. I have made these findings so far. I anticipate
these difficulties and challenges.
1. Two sentences in standup. Think through what you are
going to say, read a prepared statement from a sticky
note if necessary.
2. A short email (<100 words) answering any questions
people have regarding your research. Use your own
judgment as to audience size and email frequency.
3. A document or wiki page that is the final work
product of your research.
As someone who has felt frustrated in the past in both roles at play here, I expect I will find myself sharing this comment with others soon and often. Thank you!
At the status meeting, talk about the stages you've accomplished so far ("I've completed scoping, looked at these three third-party components, etc").
When the research stage is done, write up a research report with what you did and the findings and present it to the team. This gives the rest of the team visibility into what you're doing. It's also very helpful months later when you question "why did we pick Foo?" because you can refer back and know exactly why you did.
Yes. This is the classic funded feasibility study / scoping study phase. The report deliverable can include a price estimate and also a risk estimate, e.g. expressed as the cost of additional rework, especially where there is a residual uncertainty (e.g. the use of a new component / framework, etc) at the end of the initial study. How companies manage their risk 'pots' is extremely telling.
In some situations there can be additional breakpoints, sometimes contractually, if third parties / subcontractors are involved.
But my best work was often done when me and others had weeks or months to play with things without interruption until we found an approach that really worked.
A thousand times, this.
Yep. Our research tasks either produce a document for the team to read and then discuss and/or user stories for the team to discuss, point and work on. Which happens depends on what the research is about.
Also, if someone is a manager/lead, make it a point to read, understand and ask questions about the produced document. Nothing demotivates more than producing something that isn't used.
Depending on how you count, I have somewhere around 30Y of paid coding experience. It took me until maybe 5 years ago to have discovered this for myself.
One thing I've noticed, though, is that if you create a culture around this, you'd must have guide rails in place. Otherwise, in the wrong company, or people will use the culture to present shoddy work with the expectation that other (senior or not) developers will challenge. These people exploit the culture by using shoddy implementations and challenges to offload hard parts of their projects. This is the mirror image of the illegitimate questions: illegitimate claims.
As a senior engineer, you should ALWAYS be willing to lead from the front with code deliverables, but your review-to-code ratio is so skewed that the number of times you can satisfy that kind of challenge is low compared to the number of times when you might offer (sometimes negative) input.
"I talked to person A, who said we need featured X,Y, and Z. Then I talked to person B, who said they need A, B, and C, but feature C and Y aren't compatible with each other, so I'll need to reconcile that".
"We'll need to use third-party products A, B, and C. B has the API we need, but I'm still waiting to hear back from the C's vendor to see if they support API call foo."
this has the added benefit of discouraging most developers from being willing to take over the task
This is very insightful - what if you first created a formal plan with a bunch of checkboxes so that your status audience can see a steady moving progress bar. I've been there and agree if you keep saying the same thing people start to get nervous.
Then maybe it's just that you aren't specific enough?
You aren't just researching, so tell them what you did, what you are doing and what you plan to do instead.
I had to implement SAML quickly on our software once. It's a subject that needs quite a bit of research and ideally not much implementation. So here a timeline of what I would have said if you asked me an update regularly (which is probably quite close to what happened because we needed that for a demo pretty quickly and they asked me quite a bit of update):
T + 5m: I'm still trying to figure out what is SAML.
T + 1h: I found what is SAML, but I am trying to understand how it works technically.
T + 2h: I'm looking for libraries to help me implements it because it's too complex to implements in house and not worth it
T + 4h: I found libraries, but I lost a few hours because it needs to be implemented as a login-module, not a library.
T + 5h: I've found one that has potential but our version of J2EE is too outdated and it may not works. I'll need to experiment with it.
I'm convinced that 100% of the time, this saves time. The problem is it's tough to quantify the time saved from mistakes or mid-course changes that you would have made without having this time, and it's not easy to get execs to give up two weeks of time from a senior engineer who will really do a good job at this.
I think your issue is that you're doing open ended research - if you don't specify upfront how much time you need to investigate something then no wonder you're being asked for updates after 2 days.
The Law School Scam (The Atlantic, 2014)
Consider that you spend a huge potion of your life on very well-worn paths that were cut by others. From the subjects you study in school to the business models you drive forward at work, everything has been done before by someone else and you are only doing it now because of a sort of survivor bias. The bad ideas and bad approaches have failed or been discarded and the better ideas have turned into Fortune 500 companies and academic departments. Walking those proven paths, you just have to do a reasonable job, follow the formula, and you're not likely to fail.
Now lets say you've struck out on your own and you want to do something new that no one's ever been successful at before. Now you're essentially guaranteed failure. There's no more formula to follow. There's no more boss or teacher to tell you you're not on the right track anymore. You can no longer just plug into an existing money-making machine and start spinning, you have to build one from scratch. The best strategy is to get out of failing situations fast before they swallow you up.
An analogy from skiing: When you're on the marked trails, everything has a name and difficulty rating so you know what to expect in advance. Hazards have been roped off and the trail is guaranteed to lead to a lodge or a lift. This is life within school and companies. Then you go off-piste and out of bounds. There's no longer any ropes or trail names or safety patrol, just a wild mountain. Some people have been through here before as you can see from the ski tracks in the snow. If you follow those tracks you might be ok. They will probably lead somewhere safe like a road where you can hitch-hike or a valley you can traverse back to the resort. But if you turn instead into the fresh snow where no-one has tracked before, you're in very dangerous territory. Who knows where this leads, hypothermia in a wilderness area, under an avalanche or very likely off a cliff. But it's a whole mountain of fresh snow just for you. This is entrepreneurship.
"A couple of months in the laboratory can frequently save a couple of hours in the library. "
"Weeks of coding saves hours of planning."
However, in a software engineering context, let’s say that you’re implementing a new library to do something in a new programming language. If other, more established languages had popular libraries that did something similar, I’d systematically catalog what those libraries are, read their user docs, maybe read their source a bit, and certainly try to develop an understanding of what I thought was the right approach.
In your own readme, including this information up front, along with why you chose to implement the way you did almost certainly advances the state of the practice.
At a minimum you’ll have a record of what influenced your thinking so that, if you come back in the future, you’ll know what other threads may have advanced in the meantime.
I’ve thought of this as a way to contribute to open source that isn’t writing code or docs after the fact. Like, imagine you’re a Racket user and you wish there were a library to do X: maybe it’s a worthy contribution just to start a github repository and document the libraries that do something similar, and what you see as good and bad about those approaches. Maybe someone with the bandwidth to write the software will see your documentation and find that it gives them a head start.
Though there's a fair bit of subtlety in using them. There's nothing quite as bad as finding a framework that does 95% of what you want it to and will save you years of development effort, building your architecture around it, then getting 80% into development and finding out that there's no way to make it do the remaining 5% reliably and its developers have abandoned it precisely because of that shortcoming.
Kaibeezy’s Unique Business Model Corollary: Don’t let that stop you.
So many people think they have to have the “killer” version in a category, or they collapse with disappointment when they find out someone else already had their amazing idea. Amateurs.
Existing solutions prove a market exists. Take the opportunity to see if the market is interested in your particular formulation. If you can carve out a niche and keep your overhead low, it could become viable or even quietly successful.
Our management just discounted the research, "we asked the wrong population" etc, and drove forward with their vision
Approx $100m later the project was quietly folded, senior managers who had been promoted during the process quietly moved to pastures new, everyone else given 6 weeks to find something else.
I saw the warning signs and left before the crunch, but in hindsight I had concerns that weren't answered from day 1 but I let my enthusiasm allow me to ignore them
A learning experience
Warning: Potential Security Risk Ahead
Firefox detected a potential security threat and did not continue to ottofeller.com. If you visit this site, attackers could try to steal information like your passwords, emails, or credit card details.
ottofeller.com uses an invalid security certificate. The certificate is not trusted because the issuer certificate is unknown. The server might not be sending the appropriate intermediate certificates. An additional root certificate may need to be imported. Error code: SEC_ERROR_UNKNOWN_ISSUER
But didn't name the issuer. Why?!
Likely either a fringe issuer or self-signed cert
The most valuable projects are always going to be a race. If all your competitors are spending time carefully researching the path they will take, there is an advantage to be gained by skipping research and jumping straight in. If the idea of your project has merit, then competitors will be fumbling to catch up to you, and if the idea was a dud, then you wasted time and money while your competitors sigh in relief and talk sideways about what a fool you were.
In these scenarios, the only glory to be gained is by the ones who skip the research and move forward boldly. Moving boldly for some is just dumb luck, and for others it’s intuition.
Businesses and startups look for people with intuition, capable of finding repeatable success through some unknown heuristics they have assembled together with their life experience; essentially this is research, but not deliberate research, instead it is a subconscious research.
These intuitive people do not need to spend weeks going out into the world and talking to people about a project or reading statistics and case studies. They could simply roll their eyes into the back of their head and draw from a deep well of wisdom to determine if a project is worth pursuing. They give you their answer and that’s it; it’s the final word. A business with one of these people in its ranks has a tremendous advantage in the speed at which it moves, unburdened by the need to do research or question the premise of its ideas.
Many people in the valley like to pitch themselves as being one of these people to proselytize a following of investors or employees, but almost all are frauds. You need a mind that has depth and breadth. A person who’s lived all their life in the valley or has simply hopped from one tech job to another is unlikely to ever be one of these people. It takes a very wide range of expensive experiences to become one.
But do put a price on your passion, more so if you plan to combine that with work. As the later can kill the former just as much as the other way around, get that balance right in a way that works for you.
What research can you do other than look at web traffic estimates of competitors? It'll be obvious who is big + who is small, but it doesn't help understand... "are these companies actually profitable? What's retention like?"
1. Cold email/linkedin message then speak to 100 potential customers. Attempt presales. You don't actually have to get presales, but that process should give you a good indication for willingness to pay / desired functionality, how people are currently working on the problem, and what language they use to describe their pain (quoting that back to customers is good marketing copy)
2. Build a landing page, run facebook / google ads, find out how much it costs to acquire an email address. Your real CAC will be higher of course, but this is a good starting point.
I'm a founder of https://www.saturncloud.io/. I started with #2, and found it cost 70 cents in advertising revenue to get a click, ~7$ for an email address, and ~$100-$200 to get someone to signup. When we started we were attempting a low touch sales model.
Since then we've started focusing on enterprise sales, because the willingness to pay is much higher. We did #1 when moving towards enterprise sales, but should have done that in the beginning.
Know that while competitors might have it right, they also might have it wrong.
Talk to people.
I thought this was a good idea, although after a while I noticed that a number of good ideas ended up getting rejected and decade+ later somebody else tried the same thing and managed to get a Science or Nature paper out of it.
> If you don’t make experiments before starting a project, then your whole project will be an experiment.
Remember, a few hours of trial and error can save you several minutes of looking at the README/instructions/manual.