I was put on a project as an architect recently. My conclusion is that it's over-complicated. Why? The problem that was put in front of the dev team was pretty complex. So if you ask the dev team, it's a complicated solution to a complicated problem. And they are somewhat right.
However, the problem didn't have to be this complex. If someone would've asked some questions about the 'why' of it all, they would've been able to come to that conclusion. But nobody did. So the team was working on an over-complicated system for an over-complicated problem.
So, while I agree with the article that one might be under-estimating the complexity of a problem, it might be wise to look into whether solving the complex problem is actually necessary and whether 99% can be achieved by solving a less complex problem in a less complex way and accepting the solution isn't perfect.
I have dealt with something like this before. To keep it short, a paper process was digitised without any thought, and kept running at great expense, again without any thought.
I went for the "why", and found that this whole thing that swallowed 45 man-hours a week could be reduced to a very quick daily cron job.
I did that. My manager shrugged. I went to work elsewhere. It showed me how weird large corporations can get, but also how important it is to question requirements.
Classic case of someone bringing some elaborate 'problem' to the IT department, demanding some elaborate solution that they've decided is the only way, only for the IT staff to simply ask, "What are you trying to do?" and only then arriving at a much simpler, sustainable solution.
No, its usually over complicated. Usually the ones creating the over complicated solution lack the skill set or unaware of the simple solution, or are scared of the simple solution because its unfamiliar territory.
For example, I was asked to first guide and finally take over a project that went south. It was a simple control system. The person doing the work was unfamiliar with PIDs, I tried to guide them towards that solution, but it was unfamiliar territory for them and they thought they could do better with their own solution.
They ended up creating a very complicated non-functional mess, and were insistent that it was impossible to solve simply. 10 lines of PID later it was under perfect control.
I think sometimes it’s overcomplicated because there’s no one with sufficient authority to challenge the original framing of the problem and requirements, and part of the requirements/problem are in the hands of people who literally don’t know what they’re doing and may have misaligned incentives.
For instance, I’m flying to the other side of the country with a team to do an experiment at a vendor’s location because the safety folks at my current location are inexperienced and are working with a contractor who has zero experience installing the appropriate safety equipment at my location and it has been stuck in discussion for about two years after the scope has increased an entire order of magnitude.
One person with sufficient authority and courage could have cut the Gordian knot a year ago and improved the efficiency of the entire project a good order of magnitude.
True. A lot of overcomplicated solutions start from good intentions.
Funny thing is, I found it much easier to challenge the original framing with upper management than with my peers.
- (me) Hey, if we sacrifice this feature, we can ship the whole thing twice a fast.
- (ceo) Fuck yeah, lets do this.
vs
- (me) Hey, this seems to be solving a lot of the problems we don't have today, how about we keep it simple and leave only what matters?
- (engineer) This is best practice, simpler solution is an anti-pattern.
- (engineer) Google is doing it this way. Do you think you're smarter than Google?
- (engineer) But I already built it, why change it to something inferior?
- (engineer) My solution is very flexible, it allows changing database to a toaser with a simple configuration. It has unit-tests and mocks. Are you saying all of that was for nothing?
And yeah, it's mostly misaligned incentives. Solving complex problems makes engineers happy. When we don't have complex problems, we invent them.
This strikes me as the most common issue, along with the issue of the person defining the problem and requirements being erratic. Often people end up using versatile-but-complicated solutions because they're afraid the requirements will shift too much.
> A proportional–integral–derivative controller (PID controller or three term controller) is a control loop feedback mechanism widely used in industrial control systems and a variety of other applications requiring continuously modulated control.
Think of it as smooth ramp up | ramp down response to change, pressure, volume in industrial process control systems.
while "defrost" quoted wikipedia, a PID controller is used to prevent oscillations in equipment that have changing load. Say a milling machine that slows a bit when the tool interfaces the workpiece, so a "dumb" controller would just increase spindle RPM, but now the spindle is overspeed when the tool is lifted from the surface - say to move to a different area - and so the dumb controller slow the spindle down. Either the spindle is moving too slow or too fast per the work's spec sheet, and you get a piece of metal launched into the innards of the machine somewhere.
The proportional part is similar to a "dumb" controller - spindle slow, speed up spindle. The Integral part will increase the rate at which the spindle changes speed the longer the spindle remains the wrong speed, and the derivative part of the controller attempts to prevent oscillations of spindle speed.
>I realized that many of the times you hear this phrase, it’s from someone who has not given the problem and all its subproblems very much thought.
The author's point being that if you did give "the problem and all its subproblems" much thought, you'd agree on their proposed solution.
But there's also absolutely a problem of "giving the problem and all its subproblems TOO much thought".
That's how over-complicated, over-engineered, second-system effect, everything-and-the-kitchen-sink designs, and architecture astronauts' designs, come to be...
Sometimes you're just giving the problem and all its subproblems too much though, and solving the simple case will get you a Pareto 80% of the way there - or even 100% of the way there, and all the other issues you want to tackle are remote eventualities and edge cases you can just ignore without issue...
>Do not judge a solution as “over-complicated” until you fully understand the problem domain and every sub-problem that the business needs to be solved.
That's another issue: every sub-problem that the business wants to be solved is seldom every sub-problem that the business needs to be solved.
Requirements get blown up with BS all the time, to the detriment of business value, delivery times, etc. Sometimes by the customer (who just throws in stuff they think they need, or they might need someday), oftentimes by the sales people or resident architecture astronauts...
So, solving at an appropriate design level "the problem domain and every sub-problem that the business needs to be solved" doesn't mean your design isn't over-complicated. It might be properly-complicated for those needs, over over-complicated compared to what it should actually be, if the real needs where considered (as opposed to mere "business wants" presented as "business needs").
This is definitely a pattern that I see. What's often overlooked after giving the problem and all subproblems much thought, every problem detail has a corresponding implementation. A simple solution is one that can somehow group problem details conceptually then design a de/composition with natural seams and loose coupling.
The difference in process I usually see is incrementally coding up a solution adding detail cases vs first mapping problem facets to concepts then factoring those concepts so the implementation has boundaries that follow the problem seams. Just because you can point to an aspect of the problem for every line of code that exists doesn't mean it's not over complicated.
"Over complicated" is either an assessment by someone who doesn't fully understand the problem OR an inability to see or even imagine that a simpler solution could exist. Simple doesn't mean easy (e.g. add an 'if' somewhere in current design). Understanding is not only about coverage. A thing that I recently encountered is decomposing a problem top-down, and implementing the solution top-down, which lower-levels getting parts of the problem at hand unnecessarily leaking into it. Once I suggested to think of decomposing the problem top-down, but built the solution bottom-up from that envisioning, all the while trying to name concepts of the lower-level things and keeping them as free of unnecessary details at each level, things cleaner separation of concerns with deliberate interface points.
My theory is that, as an industry, we have almost completely lost (or maybe dilluted?) the ability to architect complex systems. Everything is 90% third-party libraries and microservices duct-taped together without much thought.
Virtually every project I joined was missing an overarching design document. Where it existed, it was usually not updated in the entire lifespan of a system. Asks to estabilish a domain or a versioned API boundary around something particularly gnarly are usually met with blank stares or violent opposition.
It's weird because I'm not even that old, and this kind of planning or refactoring was definitely a part of the curriculum at my definitely-not-Ivy-League uni.
There seems to be almost like a fear of original thought in some shops. It is humility but taken to the absurd. It's one thing to admit you may be wrong, it's another entirely to refuse to consider the idea you might be able to do something correctly.
Even though NIH is a pretty toxic attitude, the opposite can be as well. You don't have to stray that far past hello world CRUD apps to find an areas where the available libraries are poorly suited, and the superior option is to roll something on your own.
That's precisely why we should stop thinking about "good" architecture. At the very best, we'll find the least worse architecture, and that's that.
But realistically, just a handful of the real world scenarios we all face fall into this category of "least worse arch is black magic".
I'd bet my house than 90% of the architectural problems we faced exist because of a poor process on understanding requirements or poor architectural design. In no small amount due to hese two things being very hard.
Scott Wlaschin mentioned this nicely (I think in "Domain models made Functional"). Paraphrasing: "You can have all the nicest tools and greatest libraries and frameworks and coding skills.. if garbage goes in, then garbage comes out".
Usually when I assess code as over complicated it is because the engineer is solving the problem indirectly. Very often I see people focus on the perceived constraints of a problem instead of solving the actual problem. These same people often get married to their approach and resist other ways because they have already invested so much time in their current approach. We never know less than we know today and to not change your approach based on new information will undoubtedly result in spaghetti code, I mean “over engineered code”.
It’s incredible how simple things can be if you think about them creatively enough.
I dunno, I often feel things get complicated when such an incredibly simple solution encounters a set of additional requirements.
They often tend to be sort of bolted on, rather than integrated neatly by rewriting the thing to a new simple solution that caters to both requirement sets.
The problem is more like Katamari Damacy-code than spaghetti code.
Personally I think it’s ok to reflexively question complexity on a vast majority of projects.
Most engineers get excited about engineering stuff, that’s a good thing but in my experience, a ton of premature optimization can happen before you get your product in front of users.
Personally I think working prototypes that you can throw it front of a non technical person should always be the initial state of a project if possible.
It’s much easier for folks to grok complexity when it’s tied to a functional prototype and discussions come up about its limitations.
The reason people (talking primarily about myself here…) over-complicate things is fear and their coping mechanisms to fear. Engineers like me enjoy making and solving puzzles. So when there is a task requiring social courage (such as publicly revealing a massive knowledge gap in order to ask a clear simple question), it is tempting to puzzlecraft the way around the simple hard task.
The leadership solution to overcomplication is not to say “you’re overcomplicating things” but to be a steadfast curious collaborator with them to encourage the person to see more clearly the challenge that they’ve obscured with needless puzzle-making. Easier said than done.
The crazy thing that brings hidden complexity is the waterfall model. “We have that thing that get fed to this one”. Ok, how do you report errors, in the upstream system? The downstream one? Do you have parameters with the downstream process? Now you need the user to understand how they influence the output.
My point being that when you put an arrow on a white board you absolutely and immediately have to be skeptical and assume that some data will have to move against the arrow, and that will drag some hidden complexity. Stuff like very far downstream passes in compilers will need the source file line numbers and some inversion system because they might need to report errors or emit debug symbols, and that can be complicated because the code might be unrecognizable at that point.
In the 1st edit, the author hits on something really crucial - the business has needs beyond just "the software works".
He lists some frontend considerations; a backend developer might have a list like: comprehensive observability, automated builds and deployments, infrastructure defined as code, developer function-as-a-service platforms, batch processing infrastructure, horizontal scalability, database recovery time/point objectives, compliance, and SLAs.
The truth is that these ARE requirements once a project gets to a certain scale, yet they have little to do with your core application domain. To a certain extent, it is inherently complicated.
But if you look at the "modern" cloud approach to these problems, it's full of additional complexity from the solutions themselves: k8s, ec2 spot market, vpcs, prometheus/grafana/jaeger, ELK, multiple environments, airflow, a database+replica+connectionpool for every microservice, terraform, jenkins, istio, lambda ... oy. It literally takes a team of people to manage the scaffolding to support the actual apps. Is that really the best we can do? I think there's room to radically simplify this scaffolding, while still meeting all of the advanced business requirements. In that sense, today's approach is over-complicated.
> “Why are we over-complicating this?” is a very easy thing to say after being introduced to a project. What’s hard is to reserve judgement until you’ve educated yourself on the entire problem domain, and noted all the considerations that were made by previous developers.
I think that this is the "money quote."
I wrote something to that effect a while back[0].
I've been at this stuff for a long time. I have definitely done my share of overenginering, and overcomplicating, but it's my experience that a lot of that, comes from underestimating the complexity of the problem domain, during the design phase, and going "Lucy and the Chocolate Factory,"[1] after starting implementation (when things start to come out of the woodwork).
The more experienced that I get, the more likely I am, to predict some of the issues, before implementation, and I have learned to develop in a modular, flexible manner, that allows me to modify things, down the road.
Right now, I am in the middle of a fairly major refactoring of an app, that is in "chocolate meltdown" mode, because of all the addons.
Rewriting the server interactions is a big, risky task, but we aren't shipping (one reason I don't like the MVP model), so it's entirely possible to get it right before our first real users get their hands on it.
My experience is exactly the opposite in almost all cases. Most software is much more complicated than it needs to be. Reads like the author is just butthurt at feedback they received about their work.
Except when it is not. I have been burned many times on that one. Trying to "simplify" code but in the end complexifing it even more after all the edge cases have been taken into account.
You can often simplify code at the surface level, such as removing dead code, but structural complexity is really hard to get rid of, often impossible.
If there is a simple solution to a "complex" problem, then it either was not actually complex or you've made a big mistake.
People are conflating "complex" with "difficult". Difficult problems can have simple solutions. Complex problems cannot have (correct) simple solutions.
What most people usually mean by "complex" software is that its Kolmogorov complexity [0] is high (even if they don't know what Kolmogorov complexity is). You can think of this like saying "it takes a lot of software to correctly describe the shape of the problem and its resulting solution".
This is different from "it's hard to figure out how to represent this problem and its solution in software". For a crude analogy, E = mc^2 is a very simple equation, but it was very hard for humans to figure out. In other words, it was extremely difficult — but it wasn't necessarily complex.
>What most people usually mean by "complex" software is that its Kolmogorov complexity
I disagree. What most people usually mean by "complex" in any context is some fuzzy not-really-defined thing they aren't really sure about. You're bonkers if you think most people's minds refer to a soft intuition for descriptive complexity when they say the expression "complex", even if the context is software
Having said that, I do agree with your original reply since you were thinking of that specific type of complexity
> I disagree. What most people usually mean by "complex" in any context is some fuzzy not-really-defined thing they aren't really sure about.
I didn't say that this is what people mean in "any context" but rather _specifically_ in software (see the quoted text in your post). I do agree that people are fuzzy about it; that's also why I also said that there is frequently a confusion between _complexity_ and _difficulty_.
Take all the source code. Compress it using your best compression algorithm. The larger the size of the resulting file, the higher the complexity.
Basically entropy.
Now, if you can write another program that does the same, but can be compressed to a much smaller file, it is likely to have a more elegant approach. (Especially if both use the same language and style guide).
Another factor to consider, though, is the compression factor of a program. The more a program is compressed, the more repetitive it's likely to be. Such code tends to have less real complexity, even if it may look complex at first glance.
I usually find myself being accused of making things over-complicated while I'm explicitly trying to simplify an already over-complicated system. Quite often there's some elegant unifying concept, one capable of expressing all the desired complexity, that's been overlooked while everyone's kept busy piling on new feature requests. But solving the root problem, no matter how simple compared to the existing pile of unmanageable code, is always more complex than just adding the next shiny button, so no one ever wants to do it, even if means never having to manually add another shiny button ever again.
It’s funny because right now I am tearing down a system which performed well for small data but was unacceptably slow for large data sets. The earlier version sent at least 10x the data to the front end for features that were nice but not as nice as fast is nice. It went from multiple server requests to one modest sized lump of data and most of the work was the demolition of the old and careful porting of small features that still make sense with the simpler product.
Performance can be orthogonal to complexity, though. Sometimes I've seen 10x gains in performance by rewriting using 90% fewer lines of code.
Sometimes, I need to add a lot of extra code to gain 30% performance.
Well, I suppose it is a bi-modal distribution:
Level 1: Naive implementation of a problem, using little code. Often full of bugs or uncovered edge cases, or very slow.
Level 2: Most bugs and edge cases are covered, but the code base is enormous and still very inefficient.
Level 3: The code is successfully refactored. An elegant solution was found, that removes a lot of repetitive code, while improving quality and performance (often by a lot).
Level 4: The code is tuned to near-perfection. Most of the code is dedicated to hardcore optimizations, making it really hard to understand to a novice. Also, this level risks introducing bugs and vulnerabilities that are very hard to find.
In web-based client-server systems one major factor for performance is the number of client-server round trips required to do a task.
There is the extreme case of the app that does 3000 requests to boot up that boots up in 40 seconds over a LAN and takes 40 minutes over the WAN. From one viewpoint the code that does all the round trips might be straightforward and the process of packing up your data so the app can get it all done in one request involves more hard thinking. Something that might be just "1 more request" for the first system may involve more rethinking of the one big request in the second system.
On the other hand, the second system is automatically much less error prone and you never need to think about it doing http requests when it is in a partially initialized state.
Questioning complexity is a good practice because complexity slows everyone down and makes it harder for people to learn or understand.
Sometimes even if a complex solution has benefits you have to weigh what the cost is. Will it be harder to maintain this code/process? Will someone new coming in ever be able to understand it without extensive training?
Sometimes complex problems require complex solutions, but it should always be a last resort after exhausting how it might be done simpler.
The missing piece in this post is that it places vague requirements on a critic to "learn the domain" before criticizing, but no requirements on the author of the complicated solution to explain and justify why the complexity is needed.
If you have a complicated solution and can't explain or justify that complexity, you can't necessarily expect others to just respect it and take your word that it's necessary. They might have their own opinion. And maybe their opinion is wrong! But if everybody's flying by their own personal tacit intuition and can't communicate about the actual technical issues, you're going to get disagreements. You're going to get a stand-off of everyone throwing their feelings at the problem, and the HIPPO will decide what is ultimately done.
In my experience, "overcomplicated" may mean anything between "it requires novel research to come up with a solution" and "I googled for five minutes and didn't find a paid service that does it".
Too many times I have been the guy in the top left quadrant of the chart in the post. I eventually came to the realization that I have often been disrespectful to the intelligence and capacity of my colleagues. I am lucky that I get to work with smart people, and I have to remind myself that these smart people know what they are doing and don't want to do more work than necessary. This has helped me set my tone and ask better questions.
There's a joke where I ask people "Do you think if they put a bunch of ex-football players in a room and made them invent the internet, would it be this complicated?". I think the most realistic answer is that they would become renowned computer scientists and engineers and invent something just as complicated but different.
I think your thesis is better posited as Complexity != Complicatedness i.e. a complex system can be uncomplicated (eg. iPhone), and a simplex system can become very complicated (eg. the US tax code).
IMHO Complexity is a function of the number of subsystems; while complicatedness is the function of the nature of those interactions.
I have that feeling a lot in ticket complexity estimations - something needing to be done in a tonne of places for example doesn't make it complex: it can be trivial and yet tedious, long, and time-consuming.
(Which leads straight back to the worst part of it - why do we pretend complexity is what we want to estimate anyway? We all know it's time the managers at least are interested in. We don't have 'an appetite' for x amount of complexity in a two week period, we have time for a certain amount of stuff. Calling it complexity is just pretending not to be micro-managing time; we should estimate time (and call it that) but have an expectation that the estimates will often be way off and a culture that that's fine.)
"It’s a trap in software engineering to hastily assume a problem is simple that in reality is much more complex."
Very true, and also often, in my experience, the cause of over-complicated implementations, by the time the previously-unacknowledged complexities could no longer be ignored.
Sometimes there are simple solutions to complex problems, but they require a lot of time with trial and error, clever people and a willingness to try something new.
However, the problem didn't have to be this complex. If someone would've asked some questions about the 'why' of it all, they would've been able to come to that conclusion. But nobody did. So the team was working on an over-complicated system for an over-complicated problem.
So, while I agree with the article that one might be under-estimating the complexity of a problem, it might be wise to look into whether solving the complex problem is actually necessary and whether 99% can be achieved by solving a less complex problem in a less complex way and accepting the solution isn't perfect.