> there are problems that are solvable by throwing a lot of human-hours at it (“Hard Work”), and problems that are not a function of raw work hours, but rather require dealing with ambiguity (“Difficult Problems”)
There's also the 3rd category of the hell of work that is easy, ambigous, but just uses up a ton of mental space to work through that mushy ambiguity. I'd solve difficult problems all day, can stomach doing hard work when its requirements are clear, but god keep me away from the hell of "death by a thousand papercuts" in the marshes of easy-but-mindspace/time-sucking quai-ambiguous problems...
Clarifying ambiguous requirements at least is rewarding when you achieve that clarity and help people understand what they actually need. But there's hellish work that is just unclarifiable before doing this, you just have to crawl to that mud of trivial but not-so-trivial-that-you-can-think-about-something-else-while-doing-them... 100% better to work a garden or serve in bar than that!
Ya after graduating from college, I spent 3 years moving furniture from 2001-2003 and consider that the epitome of hell work. In that time, on the paltry wage I received, I could have designed and built a machine to do the work for me. So I spent 3 years being gaslit that I should be grateful for that job and that it was my patriotic duty to be there since someone had to do it. The whole time coming up with an invention a day that would have made the job safer/easier/lucrative. That broke down my psyche to such a degree that I didn't fully recover until going through ego death and healing during the pandemic 20 years later. Now when I hear people say "nobody wants to work anymore", I think of successful people who didn't do what I did, and think to myself "damn straight they don't".
The yakk shaving of today's programming has become hell work. We need a better way, yesterday. But we're all too busy spinning our wheels doing hard work to make rent as the world burns.
Edit: I want to add that I miss the days of my youth and would trade if I could, even taking the good with the bad. Just because something is stupid doesn't mean that we have to let it fill our reality. Turn the guilt/shame into empathy and let go.
I always have to look up yak shaving when I read the term. I find it hilarious that Wikipedia seems to have two definitions for it, which are practically the exact opposite of each other.
So to use this great term to it's fullest extent, I would guess a lot of programmers think they're yak shaving when they're really just yak shaving.
Hahaha that's amazing, also I didn't know there were other spellings. I was thinking about yacc as I wrote yakk but I guess it's yak. Funny how associations work. And don't forget bike shedding and cargo culting!
I've solved so many of these uncertain problems by tossing coins. Once I make a decision, everyone comes out with reasons I'm wrong and just tells me what they want. Or, everyone accepts it and moves on and it is obvious nobody actually cares.
I think this comment gets to much negativity. Yes, this is often not a good approach. But it can be one tool in your toolbox that can get you unstuck when nothing else seems to work.
This 100%. I've learnt the hard way that the only way to get unstuck at this kind of hellish work is random action - I'm biased to over-analyzing the problem so get stuck because I can't pick an option. A bias towards action, ANY/RANDOM action if you can't decide is what seems to help here best!
As long as you don't ship your decision without review, and you're willing to change your mind if it's wrong, what's the problem? Like OP, I often find that people are reluctant to discuss anything that's ambiguous, until someone has made an attempt at implementing it, and presents it to them. Then, suddenly, every ambiguous choice that was decided wrongly (in someone's eye, anyway) in that attempt comes out of the wood work. It's an extraordinarily good way to spark discussions people don't want to have.
"Write one to throw away" is the greatest advice I've heard for our field. You will rarely fully explore the problem space by just talking about it ahead of time.
"Ignorance can be solved by random action"? I don't track.
If alternatives are actually equivalent, and that yields ambiguity in decision making, that's the worst time to go deep diving. "Resolve it and move on" is just a bias of mine.
Alternatives are almost never equivalent. If at decision time there is still ambiguity, it means that some criteria that separates the alternatives is not being considered.
You are suggesting to not search for that additional separating criteria and just use a coin flip to decide. Apparently you prefer a coworker that would flip a coin in order to move forward with a decision as soon as possible.
The other poster is saying the flipping a coin is the same as staying ignorant of that additional separating criteria. Apparently he'd prefer a coworker that wouldn't stay ignorant by choosing to flip a coin.
>> problems that are not a function of raw work hours, but rather require dealing with ambiguity (“Difficult Problems”)
> work that is easy, ambigous, but just uses up a ton of mental space to work through that mushy ambiguity
Am I missing something here? Because I read your third category as being exactly this second category. Or maybe it's halfway between categories one and two?
Solving ambiguity, and probably lots of it, is the difficulty. Easy but ambiguous is generally not a combination I see.
I frequently tell my manager/team that we have two kinds of problems: easy ones that just take a lot of work/time to slog through and difficult ones that may not be much actual work except for figuring it out.
Like, many hikes are just a whole lot of one-foot-after-another. But mountaineering is sometimes a short distance of don't-fuck-up-or-you-die.
"Easy but ambigous" sounds paradoxical, maybe the terms are not correct, but it is a very very different thing totally different than the other two.
Maybe there's a better way to explain this though: there's work that would be easy if it was clarified but it's not, and at the same time you can't just think hard and discuss and clarify and make it clear befor starting to work on it, so that you can set yourself to doing it, either in robot mode (while thinking of other things), or in regular problems solving mode; you just can't, that fog and ambiguity and unclarity is just "sticky", stays there until the problem is solved and sometimes even after, making it hard to even be sure that you've really solved it; there's no reward in tackling the hard problem of fully clarifying things because you can't succeed at it, but paradoxically there's no frustration either because you don't know it's a doable tasks either; you just have to muddle through, at every step of the way doubting yourself that you've chosen the right path since there's no clear way to measure stuff and to quantify your step; and you constantly have to interrupt yourself, going in "fishing trips" for information that is stuck in god knows whos brain because it was never documented, and never getting sure enough of what you discover to have trust that by documenting it yourself you make someone's life easier; and when you get to a "solution" you still keep the bitter taste in your mouth becase the systems have no adecquate testing tools and you can never be truly sure your solution will really 100% work; and if it seems to work for know, it's never sure it will keep working in the future, or that what it does will satisfy its operators...
To take your hiking analogy: it's basically a one-foot-after-another multi-day hike, but there's fog (and fog is a meteoreological phenomenon so you can't make go away), you're not really sure which of the destinations is the right one (or if many could qualify as adequate destinations), and there's a couple landmines hidden around, with very low probability of triggering them (1-5%) but yu still know they're there, and you have no water with you and you know that the water you'll find is 90% likely to give you disentery, and you might need to ask people for directions but it's hard to know whether they'll be right.
> I frequently tell my manager/team that we have two kinds of problems
...when enough managers/leads/architects/etc. mess things up in a row, you do end up with problems of the third kind :)
OMG this is so spot on. I've been putting together a business document without clear statement of what's needed, and I get the necessary info on what they want in increments, in the form of 'no, do it this way' afterwards, without any overview. It's just taken so much time, so much more than necessary, and worn me down badly.
Wow this thread is surprisingly helpful for understanding just what is so taxing about a current project. It's never difficult. It's never repetitive. It's an endless series of poorly specified requirements that only get clarified after implementation, and I don't have access to the right stakeholders to do anything about it.
Yes, this getting clarifications one drop at a time, or being unable to clarify things before doing them.
The KILLER extra topping for me is when you lack any kind of quick-iterative-feedback-loop too. Eg. maybe it's cloud infra done in a braindead way and any way to debug is to do a tens-of-minute-length rerun/redeploy etc.
I'm OK working incrementaly, and even with incremental ambiguity, but I need a FAST feedback loop to iterate.
Absolutely. Though sometimes quantity becomes quality. I had to take over a large project with countless moving pieces, none of which are particularly hard, but the vast surface area of the project makes it hard to hold all of it in my head at once.
> the vast surface area of the project makes it hard to hold all of it in my head at once
...yep, in general one of the root causes of having to do this kind of nasty work was someone (maybe even your past self) having failed badly at decoupling a distributed system ...add some "documenting it was supposed to be milestone 3" but then priorities got restructured and it never happened, then people left and others came, and voila!
I usually refer to this distinction as "tall" vs. "wide" problems. Tall problems require having at least one exceptional or specialized person to solve it, whereas wide problems can be cleared by most generalist problem-solvers. The time required to solve wide problems is simply one of aggregate bandwidth.
Unfortunately, I think society has a ton of people who can solve super tall problems working on super wide problems, because the largest (and most well-funded) organizations have the widest problems. In some cases they have super tall problems too, but past a certain size the lights stay on by solving the wide problems.
The way progress is made on tall and wide problems is different. On tall problems, you make progress "climbing", and don't move forward much until you reach the peak, and then it's behind you all at once. For wide problems, you can make a little headway "forward" in little increments, which is easier to track.
I think a lot of wide problems could be approached as tall problems, but the incentive structure in some places is set up to measure how much "forward" movement is made each day, so you're punished for "climbing" since people think that you're staying still instead of doing work.
> One of my favorite working habits is to have two projects on my plate at a time: one project that is “just” implementation work, and another that involves some ambiguous design. If things go as planned, the implementation work finishes around the same time the design work does, so I can start implementing the design, and pickup another ambiguous task. Ideally, this is a cycle that perpetuates itself. Since implementation tends towards Hard Work and design tends towards Difficult Problems, this habit provides a pleasant balance of the two.
That's a great idea!
In my case, I often have multiple projects going, but it is generally one "Still under development, and as-yet unreleased," and one or more that are in "maintenance mode."
I've been working on a fairly ambitious iOS app (about 40 screens, and two backends -that I wrote, and two other backends that someone else wrote), for the last couple of years. It isn't yet in the final phase, but we can see it from here.
I have a number of apps, actively on the App Store, that I keep up to date, and a whole gaggle of SPM modules. I keep all of them up to date.
Everything you open source has been top notch - I can't wait to see what your 'ambitious' project becomes. (In quotes for your ambitious is not a mere mortal's ;)
It will be pretty cool. It has a very narrow demographic, though, and it will be a free app, so I won't be announcing it here. The last thing it needs, is a couple of thousand curious geeks, signing up accounts that they'll never use.
I also will probably not be allowed to publish the source (but it uses quite a few of my SPM modules, and the BAOBAB server -modified, so there is still a fair bit of OSS involved).
I actually think this (changing hard work into difficult problems) can be a danger on the individual level: a dislike of "mere" hard work means that we often seek to transform it into an "interesting" difficult problem. The classic example here is spending a day automating a task that only takes a few minutes of boring low-level maintenance a week.
As someone in devops/sre/ops/etc., I think we do alot of easy work and most of our more ambitious work, i.e. automating tasks within our control or refactoring infrastructure as code is turning easy work into hard work. Often times in this world though we're dealing with someone else's code/software. A problem I've seen in many companies is keeping things up to date. Terraform Modules, Kuberenetes components, etc. There's a lot of hard work involved in this, and I do dream of trying to turn that hard work into hard problems of making it 10x easier to keep everything up to date; but I don't know that I've spent a lot of time trying to. I spend much more time automating, i.e. turning "easy work" into "hard work".
Sometimes avoiding the hard work is more trouble than it is worth, but turning hard work into hard problems can be difficult given the ambiguity and lack of control.
That is an excellent point. Creating solutions to hard work by solving a difficult problem is a great land of opportunity.
Something that comes immediately to mind is Hasura. They’ve taken the hard work problem of building CRUD GraphQL APIs and made it almost instant for their users by solving those difficult problems.
Various flavors of process automation. Lots of computer-based work out there that is hard for humans but simple for a computer. Data manipulation tasks that are rote, tedious, and error prone can be transformed into a single click or an overnight script.
I think there's a connection here with the mythic 10x developer as well.
Some developers are just so much slower at the Difficult Problems, so that the developers that are good at them seem like 10x (or 100x or infinitely) more productive than their peers.
But I don't think they have the same leverage on the Hard Work problems.
Every time developers are extremely productive, it's because they converted hard work into difficult problems. It's not only that they solve those problems faster, but also that most people won't even try to convert them.
It certainly does not work for every kind of work. But when that doesn't work, you don't have extremely productive developers.
There are problems that can be solved with a clever, elegant, minimal algorithm.
And there are problems that are tedious because there's a huge amount of not very well organised data, and you have to go through it case by case.
Especially true if you're trying to fully internationalise something.
Examples: verifying international addresses, dealing with sales taxes in various countries and jurisdictions, dealing with import/export codes. Etc.
There's nothing conceptually hard about these problems. But a complete solution is just a very long list of nested ifs, and there's nothing much anyone can do about that. (Except buy/hire an existing solution - if someone else has done the work.)
The tough thing about the problems that are just a long list of ifs is the shape of the data. It’s usually not clear till you’re very deep in the problem what the “right” data structures are. That’s problematic because those are the difficult things to change. I think it’s a very valuable skill to be able to sniff those out early.
Working on difficult problems that is so abstract, I find that
> Strategies that can collapse the higher dimensional Difficult Problem into a lower dimensional form of Hard Work reduce the cognitive costs to solving the problem
make it worse, abstraction leaks everywhere, every level of abstract seems exchangeable, applicable .. while it just doesn't work.
Abstraction is a malware of thought of breaking task down to Hard Work.
I like this distinction, and the bit about converting difficult problems into hard-work.
There's one thing that occurred to me in the context of agile + difficult problems. When I think about my sprints, the vast amount of my time is spend dominated on that one "difficult" problem, which on the surface does not feel difficult. The spirit of agile development is to _try_ to transform difficult problems into hard-work (in the way suggested in the article), but I feel like most people don't try very hard to do that.
We could improve the way we structure our work by allowing ourselves to be more honest about acknowledging this, as well as actually putting in the mental work to figure out if parts of the difficult problem can be converted to "hard work".
I don't think that's the spirit of agile development at all. Agile is about closed loops with fast feedback. It's about communicating quickly and reacting to new information. I don't see it as having much to do with "hard vs. difficult" at all.
It _can_. The idea of having prototypes that can be used immediately but very small in scope, not thinking about what you're going to be working on next week right now, making sure you're setting up systems that can give you fast feedback etc, are immensely helpful for difficult problems.
Just taking the examples that are mentioned here:
> Designing an architecture for a new, ambiguously scoped system.
The MVP is, can you take the top 2 most ambiguous risky part of a potential solution and script it out. Then present the operating characteristics. Anti-pattern: create a story to design an architecture.
> “Improve system performance”.
Profile various aspects where you have a hunch that things are bad. Investigate theory a, b, c to improve performance. Are they trivial fixes or need a large refactor. Build up buffers for the uncertainty but react when things become more certain. Anti-pattern: "timeboxed performance improvements"
> Coordinate a rollout of a backwards incompatible change across N dependent systems.
Sometimes you can't help but do a very large rollout, so alright. But one essence of agile development is to keep your rollouts as small as possible.
Just to be clear, I'm not saying these things are not difficult problems, but agile (with lower case a) development _can_ really help, if we want it to.
The idea of having prototypes that can be used immediately assumes you have hard work, not difficult problems. It doesn't try to transform anything into any other thing, it just fails to work on one of those cases.
As you quoted, it's a strategy for dealing with ambiguity risks, AKA solving the wrong problem. It assumes you have a lot of small problems to solve, and separates their risks so you can solve one at a time. It doesn't deal with big unseparable risks at all.
I was just talking to a friend about a similar mental model. The way I see it a task falls into one of three buckets: easy problems, hard problems, and stupid problems.
Easy problems are "hard work" and hard problems are an exact match for "difficult problems". Stupid problems are just difficult problems that are only being implemented because the business (marketing, sales, C Suite) and development teams suck at communicating.
Ambiguity driven development would make for a good book.
And not to quote pg's old crowd. The middleground approach of crafting little theories that will help you solve a larger problem through easy variation seems a good starting point.
I sold cars for a living and was basically top 0.1-0.2% at it nationwide.
Every now and then a coworker would say that selling cars was easy work. So I'd ask him if it's so easy, why didn't he promise to double his sales next month? Ohh.. because that would be hard.
A lot of things are easy to do casually and hard to do intensely at scale.
This blog post would work just the same if "difficult" and "hard" were swapped. Furthermore, the same can be said about "problems" and "work". Interesting? And so, in the trash it goes.
I can only speak to myself of course, but while there may be value in “hard work”, I find it really hard to do for any amount of time without wanting to quit being a software engineer.
There's also the 3rd category of the hell of work that is easy, ambigous, but just uses up a ton of mental space to work through that mushy ambiguity. I'd solve difficult problems all day, can stomach doing hard work when its requirements are clear, but god keep me away from the hell of "death by a thousand papercuts" in the marshes of easy-but-mindspace/time-sucking quai-ambiguous problems...
Clarifying ambiguous requirements at least is rewarding when you achieve that clarity and help people understand what they actually need. But there's hellish work that is just unclarifiable before doing this, you just have to crawl to that mud of trivial but not-so-trivial-that-you-can-think-about-something-else-while-doing-them... 100% better to work a garden or serve in bar than that!