Hacker News new | past | comments | ask | show | jobs | submit login
Three Laws of Software Complexity (maheshba.bitbucket.io)
268 points by r4um 57 days ago | hide | past | favorite | 138 comments



> In my career, I have taken a particular approach based on building new systems from scratch (before they succumb to the three laws), but this is a lot harder than it sounds

This seems like an infeasible solution in most if not all real world cases I've seen. Especially if you have a system with 15 years of legacy and some thousands of man-years of effort put into them.

Now, the obvious "don't let it get to that point" is a smug IT nerd answer I often see regarding this, but this seems like the kind of green-field startup thinking that doesn't apply to 90+% of software development. We have an existing system. This needs to be maintained and extended because it's our core business. We cannot just say "oh it's become too complex. Let's go and spend the next few years creating a new product, so we'll have a multi-year feature freeze".

The only feasible way I see to do anything even close to this is to split up the software into domain-like silos slowly and incrementally. At that point you can start considering rewriting silos as they become too complex/cumbersome/big. However, at that point you're already doing complex, lengthy and generally risky refactoring, and speaking of it being a write from scratch is very obviously not true.


I agree with you. Too many of us developers tend to forget that the customer only looks at the features. Clean code, unit tests, rigorous processes... they are only as valuable as the rate of features they make possible.

Complexity and technical debt are our problem, and the customer could not care less about how we solve it, or even whether we solve it. As long as we are struggling with an old piece of sh*t, that's good for us: it means the product is selling and the effort to maintain it still makes sense!

Selling without complexity and technical debt would be much better, I think we all agree on this, but it's a receipt whose success I have never personally witnessed.


While it is hard to generalise to an absolute I do largely agree with this sentiment.

The amount of times I have advised clients to not use something like Wordpress to just end up having to use a page builder like Divi hurts my soul.

At the end of the day the majority don’t care about things like carrying bloat/legacy code, rigorous testing patterns, using modern tech stacks or typesafe codebases. They simply want you to “make thing go brr”.

I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities.

I guess the main thing from a business perspective is that I’m getting paid but, at the same time, ouch, my feels.


> I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities.

I think part of the problem here is the "market for lemons" thing: The client can't tell the difference between the short-term-cheap-long-term-expensive option and the short-term-expensive-long-term-cheap option, so can't trust that the expensive option will actually save them money in the long term, rather than being a short-term-expensive-long-term-even-more-expensive option.

Consider getting quotes to re-plaster your house. One builder quotes you £2000, another £4000. Now, is the first builder giving you a fair price and the second builder charging you extra because you don't know what a fair price is? Or is the first builder a "cowboy" who's going to do a terrible job you're going to regret, and the second a craftsman who's going to do an excellent job you'll be happy you went with? Maybe you happen to know enough, but that's not where I was when we were renovating our house.

And consider that the client isn't sure they'll even need a website in 3 years; maybe this new venture will crash and burn well before then. Then we'll be doubly glad they didn't invest in a website with the equivalent of a 10-year guarantee.


> I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities.

Until more client gets burned with a cheap option that becomes expensive to maintain they won't care. Many clients (across the whole industry, I don't know your clients which could be very different from average) will not get burned as most problems will never get that complex. However most clients have a lot of small problems that are quick to solve meaning that most developers spend their time on the small minority of clients that have complex problems where this matters.


I agree with you though I'd state the cheapest upfront option. They tend to cost more in the long run. See Terry Pratchett's boot theory: https://en.m.wikipedia.org/wiki/Boots_theory


The problem with this theory is that the $50 boot manufacturers sell out to private equity when the founder wants to retire and then the PE firm jacks up the price to $75 and drops the quality below the $10 cardboard boots... so you're out even more money to have wet feet.


Customers care a lot about defects, and I've seen defects increase dramatically due to complexity and technical debt.


What gets forgotten is the thing you can't see.

Persistent refactoring is one of those things where the cost is clear and well known (expensive developer time) while the payoff is big, but very diffuse, very hard to explain and often gets attributed to something else - sometimes even by the developers who did it.

It's not the only thing like that. Plenty of companies skimp on cost centers and fail spectacularly as a result - sometimes without even ever realizing that skimping on the relevant cost centres was the reason why they failed.


Refactoring only matters if decision makers care about long term consequences and value. One thing that makes it difficult to make these kinds of decisions wisely is that no one knows how long a piece of software will live in production.

If I could know for sure that a piece of software would exist for 50 years in production I could make a better case for refactoring. Likewise, if I knew for sure that a piece of software would get completely replaced in 1 year I would not bother with refactoring.

Never in my career have I been able to accurately predict how long software would live in production. I have had throwaway code live for years and things I thought would last forever get replaced almost instantly.


Refactoring doesn't just benefit you in 50 years it can benefit within weeks.

I agree that it's a waste on throwaway code, throwaway code is common and what ends up being throwaway code is hard to predict, but I don't think it's impossible to get a sense of the likelihood of the code being tossed, and that sense can be used to tweak your refactoring budget.


> it can benefit within weeks.

But if I spend a month refactoring and only get a weeks worth a benefit before the project is scrapped it wasn't worth it. If I get a month back before the project is scrapped it wasn't worth it (I could have been working on something else instead). Unless/until we know how to quantify how much gain will be got back, and how long it will take to get that gain back and so we cannot do a proper economic analysis if it is worth it - but that is what we need.


Refactoring is an investment decision. One which, when it pays off, pays off in a nonobvious way. That was my original point.

A response of "but what if my investment decision doesnt pay off??" kind of suggests you might have missed that point.

That happens sometimes when you make investment decisions. You dont know how much you will make back or even if you will make back anything at all.

Some people dont invest their savings because they can't stand the uncertainty of not being able to quantify the payoff but I dont.


> Refactoring doesn't just benefit you in 50 years it can benefit within weeks.

Proving the value is the hard problem. It's almost impossible to prove the value of small changes today. Product wants things to do X and they decided the value tradeoff was good, not the engineers who are tasked with making it so.


>Proving the value is the hard problem

Or impossible, even.

Thats why I just do it. If somebody wants proof that my years of experience applied in a particular way will provide a quantifiable benefit that is impossible I will just avoid doing that thing.

Or more likely, move somewhere which can appreciate those non quantifiable benefits.


> Or more likely, move somewhere which can appreciate those non quantifiable benefits.

How can you find such a place? This definitely resonates with me.

There's so much value in doing things right, but it's impossible to prove. I can do things that make my team go faster, but there's no way to say that having a X% better tests resulted in us being able to release Y% more features.


Try to give off trustworthy vibes and pick customers/clients/employers who will trust you.

I'm not going to pretend it always worked for me, but it certainly got easier once I could demonstrate a wealth of experience.

There's a lot of luck, too.

In terms of refactoring, I have an iron rule that I never ask for permission. If I think it needs doing I do it. I don't justify it to product or ask for their opinion. Technical issues and priorities are on a need to know basis and they don't need to know. All the work gets baked into feature work and bugfixes.


Hum... If you are already refactoring, it's a very good evidence that you didn't throw the code away.


"customer could not care less". This perspective is too narrow, or even a fallacy, often repeated to justify abandoning some good practices.

Oh yeah customer cares. If I leave my car in a workshop, I care whether mechanics have one big mess there or not, I am looking for signs of a good routine, clever process, some smart choices they made for themselves. If they did a fix for me but have a mess, I will never go back to them. Most of customers do such choices.


In software, all that is hidden from the end customer. How do you look for the smart choices in code made by the team who implemented any of the software you download and install?


Oh believe me, I use a lot of corporate software where the flaws in development are obvious, without knowing any of the people involved


Doesn’t the fact you do use it mean the customer (the one who chooses the vendor) did not care, though?


That's obviously different because at the corp level there are politics involved in choosing one vendor over another, but at the same time wheverer I'm involved in the decision process I care to avoid these products as hell.


Choices of vendor are not binary. There are tradeoffs between available solutions in software. Software Quality is not a linear gradient.

If your solution is the best compromise, it is more likely to be selected.


Well at companies is much less compromise than politics. Over decaces I saw two main schemas:

- product must be purchased because its vendor is somehow connected to the upper management or the investor

- product must be purchased because the person who has chosen the product is at the moment perceived as having some kind of holy cow status and can't be criticized, so people around put that to the extreme

Happening at most of companies everywhere.


If you're in one place for more than a few years, you become the person making these choices, and you don't forget.


Poor choices during development reveal themselves in many ways: poor performance, awkward workflows, mandatory upgrades, one-way data conversions, inaccurate documentation, and so on.


> Poor choices during development reveal themselves in many ways: poor performance, awkward workflows, mandatory upgrades, one-way data conversions, inaccurate documentation, and so on.

Not necessarily. Some otherwise good systems might have one bug in each of those categories.

And, most other poorly designed and poorly executed systems just chug along fine without the customer realising anything.


Sure, fine, a well-designed system can have bugs and a giant ball of duct tape and twine can work great.

That's not the norm though and I hope we can all acknowledge that.


> Poor choices during development reveal themselves in many ways: ...

All the examples you provided miss the point. Revenues are the point; all the rest is ancillary at best.


I'm not sure what you mean. jagged-chisel asked how we look for smart choices in the software we download and install. I gave examples of how poor choices manifest for end users, the idea being that a lack of those issues suggests smart choices. I don't see how revenue is relevant to the question. Revenue for whom? And how did my examples miss the point exactly?


> Oh yeah customer cares. If I leave my car in a workshop, I care whether mechanics have one big mess there or not, I am looking for signs of a good routine, clever process, some smart choices they made for themselves. If they did a fix for me but have a mess, I will never go back to them.

In your car, you can tell, in software you cannot.

Add in the fact that most times even cowboy-coded systems are once off and will be replaced (and not maintained) down the line, and you really cannot tell if the system was written as a huge mess just waiting to be exploited, or as a well-crafted and resilient system, because the majority of systems are in-house only and face no adverserial threats other than the employees.


I respectfully disagree.

I can tell about bad development habits from using the software it because code smell (and process smell) stinks.

Besides, I was telling not about car repair per se, but about the process around car repairing, which is visible when you visit car workshop and talk to the guys, and is even more visible when you use the software.

Concrete examples:

- installation/upgrade process involving interactive scripts means that somebody thought dumbing down such process for a tiny percent of less-tech users is ok, disregarding the majority who automate everything (and interactive scripts are obviously harder to automate); it also can be a sign that vendor's decision maker (product owner) is disconnected from reality

- poor documentation; a distinct case here is when part of the product is an API and of course "there is" documentation, generated by Swagger - but the crucial part of it are some subtleties of arguments passed which are themselves undocumented, making the whole API simply unusable. The vendor has an excuse "but we have documentation" - heck yes, it's because your employees hate their superiors so they gave them something to f*ck off; and this is given to the customer. Very common practice actually, I can give you a list of vendors who notoriously do that

- painfully slow webapps are a sign of a process smell that no one thinks about the real world operations; sometimes it's not even about scalability, but bad choice/usage of libraries; i need to give here names, in rot13 so it's not googlable: Argfhvgr has a HR product for corporations where every single damn click on any active field in the browser - and you click it a lot because it is fscking calendar - triggers a series of round trips to the server, each taking over 2 seconds. The usability of this product is so braindead at so many levels that I can see a whole bunch of teams failing to stop this sh*t from being released & sold. Long story but a good example that just by using the product you can see how rotten the vendor must be


The customer would also be elated if they paid you nothing and they were your only customer.


A lot of what you're describing sounds like the innovator's dilemma. One of the things that makes legacy systems so difficult is that everything is there for a reason, but often the reason is to satisfy a small minority of customers. It's easy to look at a component and say, gee this feature is only used by 2% of customers, we should consider dropping it. But that 2% is someone at the company's customer and they will raise hell if you get rid of it. So naturally the system gets filled with all these special cases to handle niche customer use cases none of which can ever be removed. This is when a startup comes along and targets the high value basic use cases and drops most advanced functionality. Then they start acquiring bigger and bigger customers, complexity accretes to meet their needs and then eventually the startup is the legacy company and the cycle repeats.

The question as a developer is do I want to work on such a system? For me, the answer is a resounding no. If I agree to work on such a system, I will essentially be enslaved to the bad design decisions made before I joined the company. It will never be truly satisfying to work on that system. I am speaking from experience.

But that is why companies pay a salary. The established company mindset is not the startup mindset is not the hobby project mindset. It's up to you to figure out what is best for you at any time and that choice may change as life circumstances change.


Even systems with 15 years of legacy and eons of manhours behind them get replaced. These projects do not always succeed but I've seen it happen.

A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand.

That last part is also the most common reason why large system replacement projects fail, go over buddget, or are not even attempted.

Often the people controlling the money do not have the technical knowledge how much complexity the old system hides. The safe bet is to keep it going.


> Even systems with 15 years of legacy and eons of manhours behind them get replaced. These projects do not always succeed but I've seen it happen.

Yes. It can happen. If you can't do it with a 99.9% succes rate you cannot make it a general business practice.

> A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand.

Beyond my general disagreement is shouldn't need full parity (man our clients would be pissed off if we cut functionality "because it was easier to rebuild a system that was functioning quite well on your end", and they have guns!), I don't think this takes into full account how much can be hidden in years and years of commits and code. We have systems being maintained where the team responsible goes "look we have a million lines of SQL stored procs. We don't know what all of them do because most of them were written in the mists of time and there's no documentation, or the documentation is so obviously wrong we can ignore it entirely". This, in spite of all handwringing about how this would never have happened if people maintained proper documentation, will happen in any legacy application. We're talking about a hundred people working side by side over decades. These things will slip through, or 'best practices' change so much that things that were a good idea then are entirely unknown now.

Even something as non-intrusive as attempting to stranglevine this will take up a lot of time and effort, if it can be done correctly.


> We have systems being maintained where the team responsible goes "look we have a million lines of SQL stored procs. We don't know what all of them do because most of them were written in the mists of time and there's no documentation, or the documentation is so obviously wrong we can ignore it entirely".

Interesting - when thinking of systems that I've personally worked on which I could never imagine being replaced, the first one that came to mind was one with exactly the same problem: a massive cache of SQL stored procedures at the core that nobody seemed to fully understand. This particular company had even implemented stuff like HTTP auth token validation and all access control with SQL scripts without any centralized access control architecture. It was outright terrifying to touch the SQL.

When I started at the company the manager advertised they "really have no upper limit" on how much they compensate developers who stick with them for years.

I guess a big problem with implementing large parts of the application logic with SQL is that it's simply not a tool fit for the purpose. Incrementally replacing, or even reasoning about, logic splintered accross assorted SQL scripts is very difficult. So nobody has the full picture of what are all the things the monolith does.


> This seems like an infeasible solution in most if not all real world cases I've seen. Especially if you have a system with 15 years of legacy and some thousands of man-years of effort put into them. > Now, the obvious "don't let it get to that point" is a smug IT nerd answer I often see regarding this, but this seems like the kind of green-field startup thinking that doesn't apply to 90+% of software development.

I think that what often happens, is that the manifestation of an old system crumbling under the weight of its complexity is that a "new kid on the block" (startup, etc.) eventually takes over. As long as the new thing only forgoes the old cruft very few care about and add stuff many care about, there's a possible takeover.


Entropy is a good analogy for software complexity. In that it takes more energy to decrease entropy than to increase it—is fundamental in understanding why in nature, without external intervention, systems tend to evolve towards greater disorder. As a result, a single engineer can wreck such havoc on a system that it's essentially impossible to repair.

This seems to be the a key to the issue and it's a people problem.


The issues always enter when a codebase has to be extended in some way that was not part of the original design. You actually have to do ‘the big refactor’ right at that point of extension such that it is as if the new requirements were known up front when the system was originally designed. Otherwise you end up with a complicated mess very quickly.


What seems like an infeasible solution? The article doesn't describe the author's "particular approach".

You snipped:

> [...] a lot harder than it sounds – more on that in a different post.

I don't see a newer post in that blog as of now (May 29, 2024).


> What seems like an infeasible solution? The article doesn't describe the author's "particular approach".

The entire idea of "well let's rebuild this before it ever gets bad". That only works in the one tiny niche corner of greenfield development that nearly no one works in. To say "oh yeah just don't ever let software get complex" is like saying "well have you tried excising any oncogenic mutations before they cause cancer"?


I actually laughed out loud when I read that conclusion. Not only does it sound infeasible and unsustainable, it also sounds not a little arrogant and likely annoying for everyone else working with them.


Can you imagine saying to your product owner or whatever "oh yeah we're not gonna do anything new. We'll spend the next year or so rebuilding this service because the code looks ugly."


These are definitely not laws but consequences of bad engineering culture.

I’ve worked on well designed systems that stayed well designed. For example one I remember that was very solid: How? The team had enough support and autonomy from management to:

1) allow us to hire the right people (no secret sauce - just a mix of not too fast growth and a healthy mix of seniors and juniors)

2) not rush new features, and spend enough time on design and tech debt.

It probably helped that the system was a core e-commerce platform. If it descended into chaos it could have had a material negative impact on the top line.

When reading articles like this positing “laws” remember many people live in a bubble and may have only worked in dysfunctional orgs. There are tons of counterfactuals out there when the system and complexity was critical to the business. (And also tons of examples where despite that it still descended into chaos- but that isn’t a law)


Mostly agree, although I think "good" engineering culture is so incredibly rare that it feels strange to call it "good" and everything else "bad". It's more like "regular" engineering culture vs "exceptional" culture.

In my experience I think the larger any team gets the more likely the team is to hire "regular" engineers and the quality regresses to the mean. So it's only a small set of specialized teams with unusual luck and control over hiring that can become "exceptional".


> bad engineering culture

If 99% of projects end up in a way that's not a consequence of a bad engineering culture.


It could be argued that the entire software industry lacks the "engineering culture" of other fields.


To call any software monkey an engineer is laughable. Engineers are generally responsible for a system catastrophically failing. In IT it's seen as a natural consequence of progression. It'll take another 3-5 generations before we have the proper body of knowledge to even have an engineering culture.

Even then though I'm not sure it'll ever really improve. Even beyond the human element I feel like the incremental building approach cannot and will not lend itself to ordered systems. Unless we go back to the ecosystem of objects that Kay talked about back in the 80s we'll always create frankenmonsters, and even if we do: we just create another kind of frankenmonster.


The amount of bush-league mistakes I see because people truly don't know any better is shocking. But we've gone from a world where you had to very intimately understand the machine to do your job, to one where a six-month crash course of very specific web technology is enough to net six figures. What did we expect?


Sounds like your company was drowning in money and time.


They were well funded, but it’s not highly unusual to encounter culture like this.


Do you thing that was a coincidence, or do do you believe in effect and cause?


Don’t really believe in cause and effect there. It’s hard to get to PMF while having high quality code unless you have an abundance of money and time. I don’t even think it’s a good idea to do that.


It doesn't sound like you disagree with the thesis of this article as much as you think you do. Your argument basically boils down to "complexity can be controlled if you spend enough time and mental effort on doing so."

Which, I mean, is exactly the point of this article - software complexity is like entropy, it will always increase over time unless energy is spent to keep it low.

Most companies have no economic incentive to spend this energy, therefore most software becomes poorly built over time. It's not because the engineers were bad or because they had bad culture, it's simply because we live in a capitalist world.


> Most companies have no economic incentive to spend this energy, therefore most software becomes poorly built over time. It's not because the engineers were bad or because they had bad culture, it's simply because we live in a capitalist world.

My frustration is that there is economic incentive, but it's diffuse and difficult to measure. And in today's culture across a broad swath of contexts, benefits that can't be measured are benefits that aren't valued.

This isn't unique to software development, but it is something that seems to be plaguing American thinking as a whole these days. Public education is extremely valuable, but it's a large cost and increasing numbers of people are skeptical of the benefits. Fixing potholes is expensive but it's surely less than the diffuse cost of fixing flat tires and broken axles. Public transit is expensive but the reduction in traffic and QOL benefits are large and widespread. Money spent on at-risk youth typically pays for itself many times over, but years later and in a difficult-to correlate reduction in policing and prison needs.

More and more we narrow our focus on short-term expenses at our of long-term detriment and it's becoming more and more apparent as a long, slow-moving rot growing from within.


Yeah, on reflection I disagree with the premise these are "laws", they're more "tendencies".

I also don't agree complexity is necessarily a moat. Sometimes it drives customers away to simpler, less expensive solutions; it depends where the complexity is as to whether it creates real lock-in or just friction and annoyance.


> In my career, I have taken a particular approach based on building new systems from scratch

= Software consultants and free lancers. I don't like them because I am jealous that they get to do greenfield projects and make all the architectural decisions but never actually have to maintain their own creations (and therefore never really learn how to design a system for changeability and maintainability).

> What can we do about this state of affairs?

1st: be aware of these laws and that keeping a 20 year old system in production is a non-trivial task and to be proud of every additional year I can keep it running.

2nd: Instead of seeing it as a 'DoS attack on myself by previous developers', I try to see it as a neverending Sudoku that challenges and stimulates and pays me every day.


"2nd: Instead of seeing it as a 'DoS attack on myself by previous developers', I try to see it as a neverending Sudoku that challenges and stimulates and pays me every day."

This is how it is for me, I've had a great time with several business systems initiated by amateurs or mediocre developers that took hold and had been going for 5-10 years when I arrived. Yes, the WTF:s are everywhere, changing these systems is really hard and takes a lot of discipline and care, but every time you put something in production you'll also have made other parts of the application better, more robust and faster.

To me that is really rewarding, and not even noisy, butterfingered management can stop you from littering the commit log with lovely little improvements that incrementally turns the turd into a much more stable income for them, and possibly yourself (and if not you jump jobs after a while and get the raise then). I don't care whether they appreciate my work, first I care about practicing my craft well, second I care about how it is perceived by my peers in the craft.

Of course, this isn't a reason to make crappy software on purpose, but this kind of just happens when you on the cheap test an idea against a market and it flies really well and customers flow in.


> To me that is really rewarding

Agreed. Many years ago I spent two years on a greenfield project that got slowly but surely worse, which was very frustrating. Now I'm working on something that's seven years old and has been through a lot of hands, but every small improvement feels good, and when I need to do the occasional hack, I don't feel bad because on the whole it's better than it used to be.


Well said, I agree so much :)


Sudoku is such an apt metaphor, thank you!

If I can extend it a bit: sometimes when you start a new job, the board state you inherit is actually subtly invalid (i.e. it will lead to an unsolvable solution), and you have to figure out the smallest number of numbers to "erase" to end up with a valid board state.

You get to realistically use the eraser maybe once per year, because nobody likes erasers. So you just compensate for it with lots of notes!


Haha indeed, bosses do not like the eraser at all :)


I think it is important to discuss the notion of accidental complexity and essential complexity here. If your organization's strength is that you have world class engineering essential complexity is your friend: a problem domain with big essential complexity is really a moat: it keeps the barrier to entry into the market high. If there were less essential complexity in the world there would be much less money in software engineering and much less software engineer jobs would exist. Case in point: markets where barrier to entry regarding technical complexity become too low degrade into a race for the bottom. (like the flood of indie games that do not make money.) On the other hand accidental complexity is not our friend: if you maintain a system with too much accidental complexity there is a great risk that a smarter competitor could create something at least as good with less resources.


> a well-designed system is an unstable, ephemeral state; whereas a badly designed system is a stable, persistent state

It's kind of entropy: I wouldn't say that a badly designed system is "stable", it is just as unstable as a well-designed system and keep evolving, but there are many more ways to be "badly designed" than to be "well designed" so without actively fighting entropy, it'll be unstable but consistently "bad"


I think this is a consequence of near-term expedient decisions being taken, and not having the ability/time/confidence/courage to make major refactors when needed.

At TableCheck we have a Ruby monolith app we've been working on for 11 years, and TBH it feels better/cleaner today that it did 8 years ago. Granted we haven't had a lot of personnel turnover and spend a lot of time on maintenance tasks.


I would not say that it's just "entropy" in the sense that there are far more disordered states than there are ordered ones.

There are. But also in software, the article says that ordered states are "easy to change" and disordered ones "difficult to modify". So the disordered states have more "friction" i.e. from there it is harder to to safely move to an ordered state. But possible to step further into disorder and yet more friction.

I have observed this in cases where past a certain point people give up on refactoring and just add code to workaround - they just wrap the code that does most of what they want in a special case for their specific needs. Which in turn makes it harder to understand and refactor. After a while, it's a big pile of nested special cases and workarounds.

Fighting that tendency is harder. It is going uphill, against the path of least resistance.


> So the disordered states have more "friction" i.e. from there it is harder to to safely move to an ordered state. But possible to step further into disorder and yet more friction.

It's not possible to safely move further into disorder. It's not even all that safe to stay without change.

I agree that there's more friction, though...


I have a counter-take on the first law that I've seen two examples of within my (20 year) career:

> A system that inevitably degrades (to a significant degree) over time was badly architected. Most systems aren't badly architected but most systems that *succeed* are. Good architecture requires time (usually an unprofitable amount) & most successful systems prioritise velocity.

In summary: the demands of profit necessitate bad system design.

Of the two examples I've seen in my career: one was by a larger-than-needed/accidentally over-resourced team in a very large corp, who were operating on an undervalued background product with no real OKRs. The second was a government-funded contract (these often skew badly in the opposite direction due to corrupt tender processes but barring that there's often breathing room for quality if there's a will).


I understand software better from words and diagrams than reading the code.

Complexity is aided by having sturdy mental models - what you intuitively understand. And to see the truth clearly.

To have humility: my code is probably ugly to other people, but I understand my code faster than reading yours.

I can't pretend to know that the system I build would be better than yours.

Be wary of pride.


> Be wary of pride.

It's a good habit in and on itself, but there still are objective heuristics to evaluate software quality.

Simplicity for example: if to solve the exact same problem, with no (impacting) performance penalty, one solution is considerably simpler (meaning, more straightforward) than another, then there's a clear winner.

The amount of automated tests is another objective way of gauging software quality.


Thanks for your reply.

I think I am contradictory when it comes to software : I don't enjoy maintaining something that breaks all the time: dependencies, system upgrades, deployment scripts and things that aren't 100% working reliably every time.

So my ideal system is to run a simple binary against a file or SQLite database and have it work reliably everytime. Not a complicated micro service architecture with lots of indirection and keep things running on network.

But balancing this with my hobby of designing multithreaded software.


Life would be so boring if we made everything flawless, we must find ways to spice it up whenever we feel like it's going too well.

I have a friend like that: loves pure functional programming languages; day job? jack of all trades doing buses communication systems ¯\_(ツ)_/¯


> Simplicity for example: if to solve the exact same problem, with no (impacting) performance penalty, one solution is considerably simpler (meaning, more straightforward) than another, then there's a clear winner.

Not so clear: some people say `foreach` on arrays is simpler than single line map/reduce/comprehensions. Others say the opposite.

Some people say a monolith is simpler than microservices. Others say the opposite.

Some people say Python is simpler. Others say Java is simpler.

I mean, an earlier response to an earlier comment of mine on a different story was from someone who thinks ORMs are simpler than SQL. I think the opposite.

Until we can get everyone to agree on what 'simple' means, it's not that useful a metric.


I should have emphasized considerably, my bad: the goal was to cover the foreach vs. map type of issues, both being essentially equivalent, and more of a matter of style.

What I had in mind was things like removing 20/30 lines of literally useless code, or avoiding sophisticated OOP patterns when a few bare functions on ``struct``s are sufficient. I've saw both cases in practice (eh, I'm guilty of it myself as well): they're often the result of overthinking, or "trying to make code reusable."

For the micro-service vs. monolith, I don't think they are comparable as-is, with respect to complexity. Once the factual requirements are known, and once the usage of each pattern to fulfill those requirements is known, then we can compare. But before, it's kind of a "C++ vs. Rust" type of debate: what is the real situation?

Regarding ORMs vs. SQL, I tend to agree with you: we often can't pretend ORMs are perfect ("non-leaky abstraction") black box: in some use-cases, it's true, but as soon as you shy away from toys, you tend to have to understand both how SQL and the ORM work. Way more work than just dealing with SQL.

Same goes for dependencies in general. But they are also essentially mandatory dependencies (e.g. a TCP stack, disk driver, an HTTP lib, etc.).


First law implies that it is impossible to change a badly designed system into a well-designed system through incremental changes.

I disagree with this, it is certainly possible to improve the state of some system without starting from scratch.


Agreed.

For example, once you have a good grasp of the codebase, and an overview of the future requirements, you can perform low-risks, local refactorings, so as to ease implementing both current and later features/bugfixes/etc.

The requirements aren't systematic though. Meaning, as a dev, you're not always, at least explicitly, allowed to get a bird-eye view, nor to act on it.


I don’t think it does - it only says badly designed is a more stable state. You can roll a boulder uphill in very small steps, as long as you can also keep it from rolling back down (analogous with other devs in your team/new requirements to implement).


I think we can look into the notion from complexity theory of attractor states. If you want to make a change, you need to shift your system enough that it moves into another state.

In more normal words - the codebase will fight your changes. And that means that small incremental changes may not be enough, and you will need at least a few big leaps.


Not impossible, no. But it it moving uphill, in the opposite direction to the path of least resistance.


Not just uphill, but also through thunder and flames.



Very interesting article. I feel like these are more laws of culture than laws of software per-se. Said in other ways:

"It only takes a match to burn down a forest"

"One bad apple spoils the bunch"

Keeping a system well designed requires good engineering culture (as other people have said), and a great alignment across engineering, management, sales... the whole company.

In control systems, you typically handle these situations with some sort of feedback mechanism. Said another way: you need to "feel" the problem if you want to be able to adjust quickly enough.

It'd be interesting to know if this kind of "feedback loop" exists for human societies / companies. Or it is just what it is: it exists, but it has a very large delay, which causes the system to eventually collapse and restart...


> "It only takes a match to burn down a forest"

This one is cool to me because (as I understand it) a healthy forest experiences multiple natural minor burn events that clear out underbrush. The resulting ash help regrowth by acting as a fertilizer.

However, when people with bad policy get involved, then these minor burn events are prevented until the underbrush accumulates to such an extent that a single match can burn down the entire forest.

The analogy to refactoring, bad development methodology, and project failure really speaks for itself.


I think this was more comprehensively described by The (Eight) Laws of Software Evolution: https://en.wikipedia.org/wiki/Lehman%27s_laws_of_software_ev...

This subject always reminds me of something someone said, possibly Professor Alain April, "software is the only system where maintenance tends to degrade it."


>"software is the only system where maintenance tends to degrade it."

Someone hasn't met my shoddy electrician/handyman


I disagree with "a well-designed system is one that is easy to change over time".

To me this is the axis of "generic" vs "specialized". As systems get increasingly optimized (for anything: resource usage, bandwidth, latency, cost, etc.) they increasingly need to make choices that limit other potential choices.

This has nothing to do with being badly or well designed. A well designed system is one that does it job well, which in part includes being able to evolve it over time (or not). This might includes staying more generic, or becoming more specialized.


17 years ago I built a system to automatically consolidate and pre-processed data from various sources. It saved 20 man-hours a day of effort and downtime. Put in 12 hours of support over the 2 years it stayed under my management, while I took many hits about how "badly" it was designed and how other teams could improve it. Moved on from that role.

6 years after left the role, I got a call from a sysadmin. He told me the machine that hosted the system went down but they managed to recover the HDD and restore it into a VM. I was surprised because I thought they had decommissioned it long ago, when the vultures who wanted to pad their KPIs tried to write a better version. He said multiple efforts all failed and no one was able to replicate what that system does. The reason he gave me a call is because after they loaded the HDD into a VM and turned everything back on, the system came back online and was working perfectly without any effort on his part. He was astounded and just wanted to talk to the person who was able to build a system that literally wouldn't die.

It wasn't the most elegant or best designed system, but I designed it not to die and that's exactly what I got!


> In my career, I have taken a particular approach based on building new systems from scratch

This is often what I see junior programmers do. The author has some experience, much in research, and I’m curious what they practice here and how effective it has been.

I don’t see poor design as inevitable myself. It can be hard to battle against the tide of developers with good intentions and wrong incentives. All too often we are pushed to accept fast and cheap over good and well done.

Update: spelling/grammar.


This is a bizarre piece riddled with odd metaphors and attempts at sounding smarter than it is, e.g. through mathematical parlance.

>Let’s say that a system X is well-designed. Someone comes along and changes it – by definition, quickly and easily – to a different system X’. Now X’ either continues to be well-designed; in which case it can be quickly and easily modified again to a different system X’’; or it will enter a badly designed state and hence be difficult to modify.

???


Seems simply stated and correct to me. Nigel Tao expresses the same idea this way

> Software has a Peter Principle. If a piece of code is comprehensible, someone will extend it, so they can apply it to their own problem. If it’s incomprehensible, they’ll write their own code instead. Code tends to be extended to its level of incomprehensibility.


Yeah this is the main truth in the fine article.


Ok, I'm about to go read the article yet, but this seems insightful to me. A badly designed system tends to stay badly designed, while a well designed one is free to mutate. And that's inherent.

Personally, I have never thought about that.


> attempts at sounding smarter

It's possible that they are smarter, and use mathematical language as a matter of course in their thinking and speech.


Certainly possible, but given this is a $2 paragraph for a 2c concept, I'm not convinced that's what we're seeing.


You have a point there, he probably could have expressed his idea without introducing calculus notation.


The real bizarre piece here is your comment that considers X´ a trademarked math parlance... People like you baffle me not gonna lie.

Not only is the author correct about what he said, he expressed it just fine. get off your high horse.


An example of a large code base that has survived decades and undergone huge scope changes without running into these problems is the Linux kernel.

Initially it only ran on X86 PCs and most of the hardware it supports today didn't exist. Today it runs on many architectures from small embededded systems to super computers.

Some of the initial code was pretty rough but the modern stuff is pretty good and cleanups and refactors are done all the time. Indeed long term maintainability is one of the major considerations for maintainers deciding to accept or reject patches.

Why does this work for Linux? Because companies are used for funding (the majority of kernel developers now do it as part of their paid jobs) without giving them technical control of quality. The companies paying people to work on the kernel get a say in what is worked on (presumably things important to them) but not about how it is done.


These "laws" are poorly framed. Software "complexity" is a function of the natural or artificial intelligences observing and altering the software system, a corollary of- with enough eyes, all bugs are shallow.

The classification of "complexity" is a function of the observer, not the observee.


I liked this article. These follow from the basic law that all systems that grow or change eventually decay and die. Death is an inevitable part of life whether that be for animals, societies or even software.

This is good. Rebirth brings vibrancy and you cannot have birth if you don't have death.


Some systems have short and chaotic lives, and other systems grow and mature for a long time. What matters is not how long a system lived, but how well it served its purpose.

Memento mori, the inevitable entropy, constant decay, and disintegration. But also, vive ut vivas, live that you may live - there is an opposite force, often called negative entropy or "negentropy", but I prefer to think of as creativity. It's what organizes things, flowers, designs, buildings, systems.


indeed, thankyou


If these are the first three "fundamental" laws of software complexity, where does Gall's law fit? The zeroth law maybe?

Also, the second law: "complexity is a moat", seems to be contradicted by "worse is better".


I read this piece with great relief. It's easy for anyone to feel offended/frustrated when coworkers, outsourced teams or even customers twist and turn your system's original clean design, making it badly designed over time. I enjoyed reading the three laws because they make me feel part of an universal problem. Software systems are no different from physical matter: it decays. It's the damn entropy. Who of us would dare to fight against it?


Ah the three laws of software complexity.

> Rate limit for this resource has been exceeded

nods thoughtfully


The media is the message


Re: first law ... one thing I have been thinking a lot lately is just how much like gardening software is. But in gardening we are not afraid to trim the plants and put them in shape as we go. In software we end up just adding stuff on top, without regard for the overall design. There is this bias towards always adding rather than removing things, and the key to keep things in order is to remove things, to say no, etc.


I don't really agree with this take. Similar to knolling [0], refactoring the code you're working on and that around it should be a natural part of every development flow. If you see a design choice that no longer makes sense, restructure. If code is unused, remove. Etc, we have plenty of powerful tools at our disposal in modern times and always cleaning things up as you go is the only way you will ever keep a code base maintainable. There will never be some magical future where you have time to clean up and refactor those thorny bits, the time is now. Similar to gardening, if you're pruning roses and you see a weed taking root, just pull it out, it's right there. You won't get all the weeds this way and sure, it would be nice to have more time to deal with them, but surely it will help a little.

[0] https://youtu.be/s-CTkbHnpNQ?si=KYwllK4NJY1bjRa3


This "trimming" is not just about features, but more generally to simplify and minimise the codebase (less code = less bugs).

Such refactoring is an essential part of maintaining software, regardless of how old or bloated it may be.


Yes, absolutely


I think most of us go through a 'software development is like..' metaphor phase.

Back in my day it was via McConnell and the PragProg guys. Jeff Atwood covered it in a blog post from over 15 years ago:

https://blog.codinghorror.com/tending-your-software-garden/


The top comment of that article is too blatant


> the key to keep things in order is to remove things

As soon as you are ready to give up the income for those things you want to trim, please go on!


> When systems compete with each other for market share, delicacy goes out the window and designers often give the application everything it wants.

This rings false for established systems that are locked to their APIs.

The vendor of a large, complex set of APIs that large, complex applications depend on is not going to give the application users everything in a new API for fear they will migrate to another set of APIs.


How's about the 1 law of software complexity: software developers are not taught how to professionally communicate, so they over talk and ignore, misinform and mislead one another into a clusterfuck of complexity, over and over again. Never dawning on their minds that better communications would solve their repeating nightmare of complex weakly composed mud balls called their work.


What's a good way to learn how to professionally communicate?


I think the main thing we have control over is if we are part of a green field app making damn sure it is as dead simple as possible. Im currently working on a redesign of a medium sized app and the powers that be decided they wanted feature flags, which means keeping the old and new versions in the same app, and a giant separate branch for the in progress redesign. One of these would have been sufficient because they actually don’t intend to ever look at the old designs again. Not to mention three different models for the same data in the same domain that I immediately consolidated. Also, I hate feature flags and think they are the worst bandaid for design indecision and business people incompetence ever devised


I don't agree with the third law; complexity is bounded by failure.


At the root of all this is a philosophical problem that encompasses all we do: unsustainable growth. We're culturally obsessed with the concept of "more". More value. More money. More features. More, more, more. This is where it gets us: over the verge of ecological catastrophe. Unsustainable systems prone to breakage. Enshittification of everything. Planned obsolescence and yearly phone upgrades, natural resources be damned!

If we are to survive, we need to slow down and instead of making "more" make "better". Follow Unix philosophy. Embrace "good enough" and learn to stop.

Who am I kidding, we're doomed.


I have a law I figured out a few years ago.

Almost all programming jobs will be on too complex systems. If a system is well designed and not too complex, then they wont need to hire you!


I always find myself sitting down to read Out of the Tar Pit[0] at least a couple times per year. It has been—and continues to be—one of the most seminal texts in my career. I still remember the first time I read the following passage on complexity, and how it just turned on all the mental light bulbs:

>> Essential Complexity is inherent in, and the essence of, the problem (as seen by the users).

>> Accidental Complexity is all the rest — complexity with which the development team would not have to deal in the ideal world (e.g. complexity arising from performance issues and from suboptimal language and infrastructure).

>> Note that the definition of essential is deliberately more strict than common usage. Specifically when we use the term essential we will mean strictly essential to the users’ problem (as opposed to — perhaps — essential to some specific, implemented, system, or even — essential to software in general).

The best skill I've learned, and continued to practice and improve, is the ability to strip how we talk about problems we want to solve with software down to what's truly essential to the problem. Making a habit of doing so helps clarify the contours of the problem itself, and improves discussions around solving because the boundaries of what's truly essential become clear—and then everyone involved knows that every choice we make from that point is additional, accidental complexity we are adding to the problem ourselves.

Far too often I have seen even greenfield software quickly ratchet up the overall complexity because the people making choices don't take the time to really isolate the problem from the software—but instead frame the problem within the context of languages, frameworks, architecture, infrastructure, and so on, and then just start slinging code at the problem.

If you haven't yet read Into the Tar Pit, it truly changed the way I look at and think about software and problem complexity. You may find value in it, as well.

[0]: https://curtclifton.net/papers/MoseleyMarks06a.pdf


It's comments like this that are the reason I come to hacker news.

Now what do we do about it? What I'm doing about it is

http:/github.com/civboot/civboot and http:/github.com/civboot/civlua



Thanks!


This rings so true. I noticed a consistent level-up in my abilities once I started to seek the essence of the problem. I ask myself: “I start with this information. The desired output is X. What is the essence of the data transformation that takes me from the input to X?”

When I boil down the task to its nature as a data transformation, the solution flows from my understanding of the problem, and I’ve found that my choice of tools flows transitively from there pretty easily. The problem is “isolated” from the software as you said which makes it so much easier to reason about things.

I sadly have not gotten much traction when I try and advocate for this mindset in our industry.

As an aside: It reminds me of a funny point from secondary education. Did you take AP tests in high school? If you did, you might remember as I do a consistent refrain that teachers used to beat into students preparing for the tests: “Answer the question” Over and over we heard this ad nauseam until it became second nature, whether for AP English or AP Physics - and it was good advice! Because the number one mistake students make on those exams is not actually answering the question asked, which even when couched in the most wonderful prose, results in a failing mark.

I think software engineering is often pretty similar. Even the best, most sophisticated tools will not produce a working solution if you don’t understand the problem.


> I sadly have not gotten much traction when I try and advocate for this mindset in our industry.

Yeah, I know what you mean. It's become a bit of a primary signal for how I evaluate a company's engineering culture. I've been lucky to work with some fantastic people who really get it, and I've also struggled and suffered through working with those who do not.

> Even the best, most sophisticated tools will not produce a working solution if you don’t understand the problem.

I'm sure we've all seen some awful monstrosities—or created them ourselves—that we could call a technically working solution to a given problem ... but it doesn't mean anyone wants to work on it. Keeping complexity at bay requires finding the simplest solutions that isolate essential and accidental complexity. Simplicity is hard, and it requires doing this well, constantly. It is [ahem] essential to spend the time required to isolate the problem and articulate it clearly. If you can't isolate and articulate the problem without referencing your tech stack and tooling, or your explanation gets all muddy and convoluted, you haven't actually identified the essential complexity of a problem. You're still stuck in accidental complexity territory. And that's a horrible place to be designing and architecting your software from.

It's also critical to note that over the lifetime of any piece of software, as new things come up—new bugs, new features, etc—you have to keep re-engaging the same process, and evaluating/reflecting on how new things fit (or don't!) within your existing architecture/design. Failing to do so is what drives toward infinite complexity and endless "tech debt" in poorly designed software. Well-designed software isolates and encapsulates all the accidental complexity into its own space(s), leaving open avenues to adjust and expand the software. Well-designed interfaces allow you to refactor, reshape, and grow the internals of a problem domain in isolation from its callers. This requires discipline from a software team—and its leadership—to take the time necessary to adjust and refactor as priors change. Such work should always be moving the needle toward greater team velocity and individual productivity.

> Did you take AP tests in high school?

Yep, sure did! I definitely remember what you're describing here.


> If you can't isolate and articulate the problem without referencing your tech stack and tooling, or your explanation gets all muddy and convoluted, you haven't actually identified the essential complexity of a problem

100%. I don't have much to add but I've really enjoyed our discussion.

Have you seen this talk by Mike Acton? If you haven't, it might really resonate with you. https://www.youtube.com/watch?v=rX0ItVEVjHc


I haven’t seen that talk yet. I’ll check it out for sure. Thanks!



The solution is probably going to involve dropping our dangerous utopian ideals about how complexity and deviation from perfection is problems that must be solved by any means necessary.

The world is a complex place where nearly nothing fit into an simplistic vision of simplicity and virtually no other engineering discipline shy away from gradual improvements and complexity management the way the IT sector does.

There is plenty of examples of real world road, water and sewage infrastructure where the system as a whole have continuity dating back centuries where every problem occurring was fixed in place without anyone ever redesigning the system by wiping and redesigning, and this is a source of pride not shame for the people working with those infrastructures.

The sooner we go away from the idea that just one more redesign using X tools in just the right way width the right team will finally crate an system that don't need constant maintenance and refactoring to keep serve the needs of it's users.


That hurts a little too deep.

Elegant code and clean architecture are no match for the chaos of the human mind.


That hurts a little too deep.

Elegant code and clean architecture are no match for the chaos of the human mind.

Note how even the author thinks that nonsensically complex systems like Kubernetes and Zookeeper are the best possible designs.


I once wrote a small piece of software which did everything for everybody in constant time with no dependencies. It was very simple.


I'd like to report a bug in that software


"Rate limit exceeded." It appears another software law has been violated.


Software complexity is directly proportional to the number of developers that have, are, or will work on the code base.


When I read the title, I was expecting to find something akin to 3 metrics of software complexity from Ousterhout's Software Design Book.

the actual article was kinda disappointing.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: