As a general rule, successful programmers have been doing this forever:
- It works. You added 2 things. It doesn't work. Which thing broke it?
- It doesn't work. You changed 2 things. It works. Which thing fixed it?
- It works. Two people changed it. It doesn't work. Who broke it?
- One donut left. 2 programmers went into break room. No donuts left. Who ate it?
Now change "2" to "1" in all of the above examples. See how much easier?
- It works. You need to add 16 things. You add 1 thing at a time. 16 operations later you are good but the deadline has passed. Fired.
- It works. Add 16 things. It doesn't work. Remove 8 things. It works. You've eliminated 8 culprits. 4 operations later done. Delivered on time. Get a promotion.
- One donut left. 2 programmers went into the break room. No donuts left. Who ate it? Not any of the 30 programmers outside of the break room. Ask the 2 programmers who did it. Prisoner's dilemma. Fun had by all.
Having done this for a number of years now, I’d argue that your first point is based on a false premise. Adding each thing one at a time feels slower because you don’t get the frantic rush from doing 16 things at the same time on a tight deadline, but most of the time end-to-end it takes the same amount of time or less, because when something does break it’s super easy to look at the one thing you did and fix it without having to suck in the entire context of the 16 things.
If you’re doing all 16 at the same time and they’re not one liner trivial things, how do you even know where exactly the 8 from your divide-and-conquer strategy are? And can you actually back them out without further breaking things?
I like this, especially the first point. I would just like to point out that the "fired." part doesn't have to be a real threat, a perceived (or even subconscious) threat is sufficient to generate the effect.
That can be a horrible time sink as well though. Imagine if only changing one thing means, that you need a test cycle of 15 minutes or even if it is only 5 minutes. You will need 5 minutes per change then. If you know multiple things you still need to change ahead of time, you might also go ahead and fix them all, and then fix resulting or remaining issues after only running one test cycle.
I think it depends on the situation, what the best approach is. Is it a system, which is running in production and you are operating an "open heart"? Well, better only change one thing and check. Is is a newly developed thing, which you develop on your own dev machine? You can easily fix many things and many more will still remain for further iterations.
“In retrospect it is clear that without all-up testing the first manned lunar landing could not have taken place as early as 1969,” von Braun wrote. “It sounded reckless, but George Mueller’s reasoning was impeccable.”
I got so excited by your response, I figured what the heck and turned my original response into a code-generated, html-driven, svg-drawn comic strip and put in on the front page of my portfolio here: edweissman.com. (Hit the 2 big arrows to surf the other 18 comics. For me, this is way more fun than anything else I ever did on that page.)
You talked about having a positive mental attitude being really helpful in overcoming shortcomings. This implies you didn't start that way and you learned the behavior.
Can you explain the thought process you adopted? I sincerely struggle with having a positive mental attitude (see I did it again) but I have recognized it's a huge indicator of a better life (social, work, family etc)
I always had a positive mental attitude. Not really sure why. Difficult childhood? Last kid picked in kickball? Couldn't get a date? Knew nobody was ever going to give me a thing so I would have to do it myself? Who knows? Way too much to think about for an answer now, but that's OK because...
My problem wasn't that I didn't have a positive mental attitude.
My problem was that I didn't know I had a positive mental attitude.
I was a typical nerd who found solace in math and programming but somehow worked my way up into a dev / project management / digital rainmaker position. My customer wanted me to do something he knew I could do that no one else had been able to do (something boring sounding like "implementing a factory shop floor control system").
He became so exasperated at my pushback (I can't. It can't be done. It's too hard. etc) that eventually yelled at me something like, "You know the business! You know the tech! You're good with people! You have PMA! So what's the f*in problem?) I asked, "What is PMA?" He said "Positive Mental Attitude!"
That conversation changed my career (and my life). And he became my major mentor. (That's why I thought of him in that "Advice that changed your life" thread last week.
This may sound bizarre and counter-intuitive but I think the short answer to your question may be, "You have PMA when you decide you have PMA."
Over the years, I've witnessed countless counter-examples to this: people don't have PMA because they don't know they have PMA so they never "breakthrough" like they could.
I'm so glad I met that mentor. Thanks for reminding me.
Is there something that just generates the html for you?
Exactly. I rolled my own CMS just for this. No images, just html and superfast svg built from database parameters. I'll be launching a full webcomic only site by the end of the year. I hope to include source at some point. I'll keep you posted.
My corollary to this is that bugs are a sort-of conserved quantity. For every 1,000 real* bugs found, 1,000 bugfixes are required. Bug report in, bug fix out.
The longer you delay fixing bugs, the more concurrent issues are present in the codebase. So then when you add a feature, instead of either having 0 errors or diagnosing a bug in 1 new thing, you now have to diagnose n+1 bugs, where 'n' is the number of unresolved bugs.
The more this is allowed to fester, the worse this becomes, to the point of hopelessness. I've seen a codebase with ~1K open crash bugs, and hundreds of memory corruption bugs. Developers were making things worse and convincing themselves that they were fixing things because randomly the memory corruption was less bad on that test run.
*) To make things simple, just ignore false bug reports for now. Pretend all bugs are real, such as "makes the program crash".
Although I agree with the general view of the article, the rule "If It Can't Be Done Incrementally, Don't Do It" does not always apply. I once had to refactor a large application from non-currency-aware to currency-aware. This means every single database field and every single operation involving prices had to be touched. Of course, I did not randomly jump around the code and instead tried to do it systematically. But the whole refactoring took several weeks and had to be rolled out all at once.
Exception proving the rule. Yes, sometimes for something as deep as adding currency you do just need to fully refactor BUT I think you're still doing it in the spirit of "One Problem At a Time" You were adding currency, not adding currency AND updating the date library AND touching up the unit testing mocking AND adjusting that CSS for the checkout button (or whatever equivalent).
Hm. All your examples depent heavily on the circumstances. For instance, if you have a team of coders working on the same code base, it might be preferable to do all these things in parellel and role them out together. I just think that "Do things incrementally" is not so an important advice as the formulation "If It Can't Be Done Incrementally, Don't Do It" suggests. I would rather say: "Prefer to do things incrementally unless you have a good reason to do it differently."
I've come to realise that a lot of problems in software engineering are a result of either having one thing achieve two or more goals, or having two or more things to achieve one goal. The thing may be a function or a person.
If the thing is too big for one person, don't add a second person. Break the thing down until you have enough smaller things to assign to people exclusively.
This post actually doesn't resonate with me at all (and I've always respected Ben). I feel like I'm constantly solving a multitude of problems and actively look for opportunities where I can introduce new tech that hits on multiple disparate streams at once (I have neither the bandwidth as an org or the personal time to operate in a different manner), typically coupled with other new tech; even that is orthogonal to running dozens of work streams concurrently, which isn't exactly outside the gist of the article. Even getting a sufficiently complex product/program off the ground necessitates solving countless problems at once, otherwise the architecture has no chance of supporting growth. Perhaps I'm at the stage where advice like this doesn't apply, but looking back over my career, I can't really identify a point at which it did.
To change my phrasing -- If someone manages a large team, let's assume that they are responsible for all that the team does and can be viewed as a single entity, synonymous with a single engineer working at high bandwidth. That entity is absolutely not solving singular problems at once. Individual engineers within that team are not working in isolation from one another and they all have interconnect impact unless their delivery is absolutely rote. In order to move at a speed necessary for on-time results on complex problems, the advice proposed in the article must be violated.
I have the same reaction. Sometimes there's multiple problems that can all be solved at once, and it is like a force multiplier. Often it's from what you could call "outside the box" thinking, where you maybe take a totally different approach.
For example, there's a bug in the login system, again. Ok, we could probably fix it, but this is an old system built by outside contractors (aka "built cheaply, but expensive to maintain"). It's not the first bug, and won't be the last. There's some technical debt in the way it stores unsalted sha1 passwords, password resets work in a way that allows an attacker to effectively lock out accounts and do a DoS. Looking forward we have customers asking about MFA and SSO support that this system just can't do.
"Solve only one problem" philosophy says to just fix the bug. This is a case where replacing the system (whether buy or build) might make more sense. You have to be pragmatic about this of course, because if you can't live with the bug for however long it takes to replace the system, it isn't going to fly.
I'd still advocate some constraint, though: iteration 1 should fix the bug(s) and underlying tech debt, but shouldn't be adding anything new.
Some of the most productive work I do is this style because it's often only a bit more effort. I used "login system" only as an
easy to understand example, it's often more subtle, along the lines of: refactor a class so it has unit tests and fixes 3 other related but low priority bugs at the same time, instead of just patching. Instead of a day, it takes a week, but more value was delivered (other bugs fixed), there's significantly lower chances of bugs in the future (as future dev work happens), and we can reduce or eliminate manual QA on that component. The extra 4 days have saved dozens we'd spend over the next year if we dogmatically stick to "one thing at a time".
Doing things incrementally almost always feels more expensive. And sometimes it is. But it is always a net positive in the long run.
When you're starting those "replace X system" projects, what you're not anticipating is that half of those projects fail: they're either end up taking way longer than planned, or they don't get finished at all because they miss some critical requirements. The bigger the project, the more risky it is.
Your login system has two problems:
1. It has a specific bug
2. It has tech debt
The issue is that you're using current bugs as a justification to remove tech debt. Because, let me guess, you want to refactor, but you can't sell refactoring to the management.
Here's how management might look at your arguments:
> password resets work in a way that allows an attacker to effectively lock out accounts and do a DoS
Did it actually happen? I worked at a SaaS startup that had over 50K users, it had the same potential problem. It never happened. Side note: some security standards actually require this DoS vector to exist.
> Looking forward we have customers asking
And how far in the future are we looking? It might be years before it becomes a real problem.
You might be absolutely right in your desire replace the login system. Or you might be a perfectionist that's eager to solve problems that don't matter. The great thing about incremental process is that for a relatively small cost those risks go away.
> When you're starting those "replace X system" projects, what you're not anticipating is that half of those projects fail: they're either end up taking way longer than planned, or they don't get finished at all because they miss some critical requirements. The bigger the project, the more risky it is.
Totally agree! However, there are ways to do them successfully. I've done it many times.
There's no need to ever get into a position where you miss a critical requirement, let alone having it prevent completion. My favorite method is building replacement systems in parallel, leaving the existing stuff in place. Often there's a good opportunity to do a totally new feature this way first (maybe allowing you to increase your total addressable market), before you start adding existing features from the old system.
> Because, let me guess, you want to refactor, but you can't sell refactoring to the management.
Ugh. This statement grates at my soul.
First, let's deal with "want to refactor". Up front, as a technologist, I'll admit I do sometimes want to refactor bad code/systems/whatever purely because they're bad. However, as a professional, I know I need to justify that work.
This brings to the second part, "sell refactoring to the management". This is just not a conversation that I ever have.
If I am (or my team is) being slow due to constantly refactoring code, management is going to question what I'm doing and hold me accountable.
If we're slow and causing customer complaints because every time we touch code we add more bugs and start a break-fix cycle... management is going to question what I'm doing and hold me accountable.
What I do is look for spots where there's a provable return on investment (justification), and simply make that part of the next bug or feature we're working on. If you can take a system/component/whatever that is constantly getting bug fixes, spend slightly longer on the next fix (because you're refactoring) and then have fewer bugs after and/or be able to implement features faster, that's a win. Do it a couple times, and you'll never be questioned about this type of work.
The discussion around doing a large-scale replacement is entirely different, and the ways to justify it are using real numbers such as: potential to hit new market; time spent on support; time spent fixing (repeat) bugs; time it takes to implement changes; time spent fixing new bugs anytime someone modifies the system. Just like above, if you can't prove this, you shouldn't be doing it.
>> Looking forward we have customers asking
> And how far in the future are we looking? It might be years before it becomes a real problem.
Let me clarify my sentence: "Looking forward in the backlog, we currently have open customer requests.."
I agree with the sentiment I think you're getting at, which is you shouldn't build to unknown future requirements. You'll pretty much always get it wrong. I teach this to juniors with an analogy like: If you build a foundation for a skyscraper before you actually need a skyscraper, what you'll often find is it needs one more underground parking level than you thought and to be rotated by 15° -- so everything you built is not just wrong but is actively blocking you from building what you need.
> You might be absolutely right in your desire replace the login system. Or you might be a perfectionist that's eager to solve problems that don't matter.
Let me just be clear, because I wasn't originally, that my "login system" was a fictional example. The real examples I have require too much time to explain adequately.
> The great thing about incremental process is that for a relatively small cost those risks go away.
The point is sometimes a rewrite makes more sense because it solves several problems at once. This is where I take issue with the original article.
You hit on the reasons a full out rewrite fails: Not understanding all requirements. Doing a switch without backwards-compatibility. Going months/years without delivering value.
At least I think this is where we align again: The key to a successful rewrite is building incrementally, delivering value as quickly as possible.
I think the advice is fine if applied to one context at a time. You can paint one room in your house and replace the carpet in another "simultaneously" without too much risk, but doing both at once in the same room is asking for disaster.
It really depends on the congruency between orthogonal tasks. Painting walls and laying carpet are diametrically opposed and are mutually exclusive. There are other mixed contexts that don't attribute this trait, e.g. I can make updates to multiple disparate ad products in tandem which independently have impact on a central ad server while also introducing new caches to the ad server without exceeding the risk threshold of making such changes at once. This kind of thing happens every day.
How is a manager not synonymous with a high bandwidth engineer? If I told an engineer to execute on an immensely huge and complex problem (e.g. go build a new secondary market with tendrils into everything else the company is doing) by themselves and they did, how is that different than telling a senior manager to do the same? Really depends on the lens you're looking through.
Because a manager manages. Having weekly 1:1s with 8 direct reports is not “high bandwidth”. That’s normal bandwidth. 1 engineer doing all that on their own is incredibly high bandwidth.
Also, a high bandwidth engineer building all that is way more valuable than a manager building a team to do it, in terms of bottom line.
Sure managers are important and good ones produce results. But that’s not equivalent to a 10x engineer. That’s just the normal expectation for a manager.
To me this is just a matter of scope for the abstraction you named "problem", because ultimately we can only solve one problem at a time, and everything we achieve is achieved incrementally.
Whether you explicitly break up your "problem" into smaller problems by organizing your workflow that way, or implicitly break it up by simply biting off bits as you work seems to be the difference. The former being more planned, with the latter requiring greater mental tracking and ad lib task management.
The greatest disadvantage regardless of how you work is task switching. If you're doing push-ups and sit-ups, it's easier to do 50 each x2 than 1 set of 2 each x 50. So if you include switching tasks as a task, you're doing 3 exercises, not 2, and your work becomes 150 instead of 100.
I think viewing this within the domain of the count of problems one is attempting to solve at once isn't the proper perspective. I work on the basis of Risk Containment, which often ends up meaning "solve one problem at a time", or "develop list of problems and solve one problem at a time". But often there is risk choosing that approach.
Example: a couple of years back, I needed to do a new web app. I could have relied on my tried-and-true jQuery and HTML 5 skils. But with all I had read about Vue and the examples I saw led me to the conclusion I might do better timeline-wise (thus, reduced risk) by using the web app as a vector to learning Vue. I wasn't wrong, things went much faster once I get the hang of how to Vue. My client was so impressed with the work, he asked if I was just middle-manning a white-labeled tech solution from a third party. (Thanks bootstrap-vue!)
How would that have turned out if I did jQuery / HTML 5 then a rewrite into Vue to keep it "one problem at a time"? It might have taken 2.5-4x as long.
Anyway I look at the "solve one problem at a time" as a useful tool in the tool belt. But it is not an overriding principle. Risk Containment is that principle. A few years back, this was a peripheral tactic, even an afterthought. I've been working with some Risk Containment Ninjas for a couple of years, and the key lesson for me, the iterations are shorter in duration, the cost is less, and the idea evolves to a better end result. It forces you to think about every tiny piece of a project as driving home a MVP.
I kinda hear what you're saying, but if it's reimplementing existing business logic you can spend less brain power on solving a business problem and more on a technology problem.
I don't believe there's a 1:1 straight translation from (in this case) Ruby to Go without a lot of extra work in the form of re-evaluating features, flows, performance, etc; the problem they solved wasn't the business feature, and a programming language should never be a problem.
It depends on whether you're the one paying for someone's learning curve.
You'll be much better off if you learn the fundamentals then use a real project to polish your skills than trying to learn from scratch with a deadline looming.
Something has gone wrong if learning on a live project is putting deadlines at risk. I've learned the most working on live projects, but it was important that the things I worked on weren't crucial for meeting those deadlines. This has a lot to do with how tasks are broken down and how responsibilities and tasks are delegated and what support there is from the rest of the team to maintain an acceptable velocity. That ensures that you're not, say, getting stuck on something for a week longer than necessary.
As for "learning the fundamentals", it's incredibly difficult to know what you don't know or understand and what you need to know/understand until you're trying to solve real problems. It's so inefficient and unmotivating. I knew the absolute basics of Spring Boot when I started working as a Java developer, but it wasn't until I started working a job with Spring Boot that the real learning started and I felt the motivation and drive to learn everything I could.
Frankly, businesses need to adjust and learn to be okay with paying for learning curves, instead of expecting people to know and have experience with everything under the sun. There are long-term benefits that aren't easily quantifiable.
The _fundamentals_ of most new languages one might typically use take a pretty short time (weeks, not months) to learn, for experienced developers. Most significant real projects will need at least a couple of prototypes/POC's at the beginning. Thus a brand new project is a great time to learn a new language, you can make your beginner mistakes in code which will never ship.
This goes back to that "innovation tokens" thing though, a new language is one innovation and you don't want too many others at the same time.
> It depends on whether you're the one paying for someone's learning curve.
When I was consulting, I would occasionally make a time box deal with the client. One project sticks out to me: I thought their project was perfectly suited to Rails at the time, but the majority of my projects had been done in Django. “I’m going to try building this in Rails. When I hit a 40h timebox, I’m going to stop, look at progress so far with a critical eye, and assess whether we’re getting value from this or not. If I don’t think so, I’ll abandon the Rails project and start over for free with Django.”
As it turned out, the whole thing was done in about 30h with Rails and we didn’t need to have the rest of that conversation.
Any language I tried to pick up just to learn for is own sake never went anywhere. The stuff I just started running with a new project or job... Still use those today.
Regarding learning a new paradigm and having to build an application with it, I think it’s possible within a reasonable time.
My approach, ironically was the incremental approach mentioned in the article. So the first step was for me to view it as one problem - build app using X. Then the steps might have looked like
1. Learn ABC
2. Apply ABC
3. Refine ABC
I always know enough to build something. Eventually, I know enough to refine it.
Now the author said this is slow and I agree but at the end of what would be a learn-first-build-later approach is an actual, built system. You can now solve the next problem, enhancements.
To any new developers, I should mention that a good foundation of fundamentals is key to this kind of approach and I admit, there are much better ways to learn and build.
I have to agree with you. Since there are infinite things to learn and infinite projects to build, it's nice if you can combine it from time to time. Surely you shouldn't only do that, but it can be very exciting and fruitful in my experience.
This is presented as if this is sage advice that no one has heard before rather than tired cliches that only work in specific circumstances.
Maybe I'm just jaded. I cant count the times I've been handed projects where worked 'incrementally' meaning the original team mistook that for use all the time and budget yet not deliver the product. Or the times I've been handed projects where everything from the language to the os had changed without changing the timeline.
Absolutely only one one change at a time if you can. The real value comes in being able to take a situation with multiple changes and still make it work.
I agree, I feel absolutely blessed if I can identify new tasking that can be done truly incrementally without relying on highly orchestrated changes. Most of the time, my value comes from identifying how to organize multiple work streams that come together at once to solve complex problems against tight deadlines.
There is nothing more exstatic than the feeling of having solved 4 problems at once with one changeset - there is also nothing more devastating than having to figure out which of the 4 changes caused problem number 5.
At the team level I've liked the advice to only deploy one novel tech at a time. Similar to the article, if a problem happens between subsystem A and B, you'll struggle if A and B are too new to your team. Which subsystem is broken? Can A be patched to fix an unsolvable problem in B? You could end up with a very unusual config for B that would have been cleanly solved in A.
As long as you are paying very careful attention to what you are doing, and have a very clear idea of why you’re doing it.
Pausing between iterations on a thing can be an opportunity to sit back and collect the observations you need to know what to do in the next iteration.
But pausing between iterations of a thing simply because there is some other thing you want to start on and you just can’t wait to get started, even if it means pausing work in something that isn’t actually done yet, is how vaporware is made.
This post resonates a lot! I think finding a clear focus and going deep on it takes you places no one (even you) can anticipate in advance. It's how you can create something amazing.
I felt that personally working on eesel [1]. We kept a tight focus on the problem (making it easier to refer back to your work more easily) and it's become far more than we could've expected. Ironically, the classic "is this a company or a feature" question might actually be a validation of keeping scope.
The only caveat I'd add to the post is that you should try to solve only one problem, but you should spend time exploring before hand to discover the broader set of problems and prioritise carefully.
Ben Nadel, the king of selfie. Some time ago, I had a project where I searched a lot for ColdFusion, because I was new at this topic. I ended so often on Nadels website that I blocked his selfie section in ublock...
As a grad student, it seems like a pipe-dream. I've come to acknowledge that maybe I'm not the multitasker I thought I was but sadly, with tens of deadlines to juggle around, no respite for me.
I remember reading once that Steve Jobs only assigned his engineers one project at a time to optimize focus. A potentially apocryphal story because I later read his biography and didn't find any mention of this.
I personally have a tendency to cherry-pick easy tasks and procrastinate on the gnarlier ones. Single-threading projects would force me to confront those difficult tasks and "eat the frog".
Sounds like a variation on the "Choose boring technology" presentation. I agree with this idea, but there are far too many resume driven developers out there.
https://boringtechnology.club/
