
How to Accept Over-Engineering for What It Really Is - fagnerbrack
https://medium.com/@fagnerbrack/how-to-accept-over-engineering-for-what-it-really-is-6fca9a919263
======
mannykannot
I suspect that quite a bit of the over-engineering I have seen was driven by
the personal agendas of some of the developers: turning a boring, run-of-the-
mill project into something that feels like you are working on the leading
edge of technology; "I have learned all this stuff, now I am going to show I
can use it"; the misapplication of design principles for large projects to
small ones; resume padding. It is not done by those programmers who struggle
to get anything done or by those who always do the least they can get away
with out of laziness; it requires a certain level of competence and is driven
by ambition.

These motivations seem to be behind many of the almost constant calls for
rewrites, but rewrites with no better justification than this rarely turn out
to be a good use of resources. It is particularly galling to see a rewrite
that is no better than the original.

~~~
taftster
I agree with you. Only the very best developers that I have worked with
advocate for principles like: no rewrites, refactor only; less is more;
minimal viable product with customer feedback loops; reducing complexity; a/b
testing, etc. My best work has been alongside product managers that push back
on new features. I have never seen a rewrite that was functionally better than
the original.

I honestly don't have any experience working with a team that produces a top-
shelf title, like a popular web browser, spreadsheet, or calendar application.
I've only been involved working on internal facing "intranet" business
applications, because that's where the majority of the jobs are.

That being said, I wonder if what you're describing is worse for internal
applications, simply because teams don't really "listen" to their users
(employees)? I don't think I've ever seen a nice intranet or internal payroll
application, for example. My current company, which is a very big F500
company, has one of the worst payroll and HR apps considering its size.
Intranet applications tend to accumulate every departments special hidden
feature without anyone keeping it in check.

Looking at Apple I feel that most of their customer facing applications and
operating systems are getting worse not better. They are getting too "clever"
with their user interfaces, relying on tricky unintuitive user interface
gestures and hidden button clicking. I wonder if the symptoms you describe
manifest themselves at Apple as UI "cleverness."

~~~
christophilus
I've over-engineered plenty of software in my time. Most of the time, it was
because of boredom with yet another crud app.

An anecdote about rewriting:

I've also been involved in 4 successful rewrites. The decision to scrap/start
over is rarely a good one. In my case, my team had inherited a bizarre suite
of apps that combined VB6/C++. It was full of memory leaks, and had never been
successfully deployed in the 3-4 years of its existence. We rewrote the entire
suite in a year using .NET (1.x at the time). We (developers) went onsite to a
bunch of our clients and watched them work, figured out what the system really
needed to do, and made it do exactly that. The result was the first successful
deployment the company had had in 4 years. It also sold itself in sales
meetings, because it had been designed by watching users, rather than by
guesswork.

So rewrites can be done. The question is: how bad is the underlying system?
And how much easier is the problem to solve in technology X vs Y?

In our case, the underlying system was atrocious, undocumented, untested,
written by a COBOL programmer who did not understand VB or C++ or OOP. And the
.NET WinForms stack was vastly more productive than the C++ (and even the VB6)
stacks we were replacing. The decision to rewrite was pretty easy.

~~~
jtrtoo
This seems successful because It's less a rewrite and more of a build a
different solution that actually fits the need.

Which is a good reason to dump something that isn't appropriate any longer.
Or, worse, never was appropriate. :-)

------
contingencies
I believe most experienced software types would agree that maintaining systems
over time is a rather different skillset to building them in the first place.
'Fire and forget' is a great short-term commercial strategy, but usually a
terrible operational one.

Certain (usually early) stages of a project may be time or resource rich or
permit wide-ranging changes with minimal risk (client impact, downtime, live
failover requirements, carefully engineered change-overs, etc.). At these
times, an experienced person might opt to utilize resources to 'over-engineer'
for contemporary requirements in the knowledge that it is a strong and well-
reasoned long term play, while to the less experienced, non-technical project
managers or stakeholders this may seem questionable.

Ultimately, every project is different but there is a 'sliding window' of
project resources (particularly time and the availability of low-risk
incremental change) and requirements (expanding, shifting, contracting).

A famous quote from a famous figure on migration: _Those change-over things
are really severe. Really severe problems._ \- Joseph Henry Condon, Bell
Labs.[0]

Perhaps ultimately, over-engineering may be inefficient but efficient if
considered in light of the alternatives: by parable, eventually including a
rewrite in javascript that also reads email.

[0]
[https://github.com/globalcitizen/taoup](https://github.com/globalcitizen/taoup)

~~~
ChemicalWarfare
Speaking of the long term.

The thing is the flavor of over-engineering JavaScript suffers from - bringing
in frameworks/tech the project can be done without is bound to backfire at
some point when (and in a ridiculously fast moving ecosystem like js we're
talking 1-2 years) you can't find engineers with the skillet needed for the
the "old tech"-based projects.

~~~
ams6110
That's why when it comes to javascript I don't use anything but bare JS and
jQuery. There is too much churn in everything else.

~~~
arvinsim
But then you are back to the problem where companies require new hires to know
React, AngularJS, Karma, Mocha, NodeJS, etc..

------
knocte
I mainly disagree with the article.

Most over-engineered monstruosities I've seen in my career don't come from
engineers trying to uber-optimize their solutions to a sick level, but from
noob/unexperienced/dumb programmers trying to get something done.

Things like: methods with just one line of code (to call another method),
classes that seem to be there just to satisfy some dubious OOP pattern, too
many levels of error handling which can be simplified by just removing them
and catch the exception in a single place of the program (at least in
exception-based languages), too much 'public' elements in a class (justified
by the programmer thinking: "just in case" someone needs to access them in the
future), excessive interface use due to the (clueless)need of pedantic levels
of unit testing, NIH-retards creating frameworks/ORMs/libraries for every
single component their software needs (I even saw once a tech lead that
replaced all uses of hashtables in the company by his own implementation
which, hey, let you look for keys in a case-insensitive way!), choosing low-
level languages to implement non-performance-critical software, and a big
etcetera.

~~~
rejschaap
What you describe is not really over-engineering, at least not according to
the definitions he quotes. I would just call it bad engineering instead.

~~~
knocte
Thing is, in my book, bad engineers are the ones that cannot get anything done
(or barely do) because they are not talented/intelligent enough. These are
harmless.

But what about the ones that get things done (at least their managers perceive
so) but creating these enormous turds? They think they have a clue, but are
just barely starting to scratch the surface, and they can harm any project a
lot. For me, overengineering is not excessive optimization or quality, but
excessive unnecessary complexity (when there could be an alternative, much
simpler, implementation, which still complies with the requirements imposed).

~~~
xg15
> _For me, overengineering is [...] excessive unnecessary complexity (when
> there could be an alternative, much simpler, implementation, which still
> complies with the requirements imposed)._

I think the article relates to that though. I agree there are cases in which
the complexity of objectively uneeded and will never be useful. (Reinventing
the wheel, bad design, the aforementioned misapplied OOP principles and unit
boundaries) but what the article stresses IMO is that it's not always easy to
see if the simple implementation will comply with the requirements in the
future as well. Some complexity could be justified if the speculation on the
future is backed by facts.

------
dansingerman
My take away from this is you need to ask 'why' as often as 'what' and 'how'
when developing software.

In my experience many software developers completely fail to challenge the
'why' of what they're being asked to do, and jump right to design/coding.

To be fair though, not all software developers work in an environment where
they can, or feel they can, ask or challenge the 'why', but the software will
only benefit if they do.

~~~
js2
Why? is also how you resolve the XY problem[1], and asking it five times[2] is
the basis of a root cause analysis, and as any parent will tell you[3] is part
of raising children, and is also what you should seek to put into a commit
message[4].

Why? is almost always a good question to ask.

1\. [http://meta.stackexchange.com/questions/66377/what-is-the-
xy...](http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem)

2\.
[https://en.m.wikipedia.org/wiki/5_Whys](https://en.m.wikipedia.org/wiki/5_Whys)

3\.
[https://m.youtube.com/watch?v=sahSAMj8OIY](https://m.youtube.com/watch?v=sahSAMj8OIY)

4\. [https://github.com/erlang/otp/wiki/writing-good-commit-
messa...](https://github.com/erlang/otp/wiki/writing-good-commit-messages)

------
qznc
Ultimately, software design is a prediction of the future. You make those
parts flexible where you will need it. You make other parts simple and
opinionated where you need no flexibility. Like all predictions we are often
wrong, which leads to either over-engineering or lack of features.

~~~
jasim
And it is incredibly hard to predict the future. So I just try to engineer
things forcefully for only today's needs. And instead of future-proofing, I
try to make it easily comprehendible. Make their intentions as explicit as
possible. Comprehension trumps everything. Because before we can actually
change code by typing it in, we need to be able to hold it in our head and
mend it there. And if you can clearly see in your mind's eye what is currently
there, you can see what it can become.

The above applies only when we have to 'predict' the future. Oftentimes we
know the future already - there is going to be ton of traffic, we are going to
need an API for this use case, we are actually going to stay in business etc.

~~~
qznc
> ton of traffic

This comes with assumptions. Depending on the magnitude of the traffic you
need very different architectures. For example, see Pokemon Go
[https://cloudplatform.googleblog.com/2016/09/bringing-
Pokemo...](https://cloudplatform.googleblog.com/2016/09/bringing-Pokemon-GO-
to-life-on-Google-Cloud.html)

> an API

How flexible should the API be? For example, do you need multiple
representations (JSON, YAML, XML, etc)? Maybe it makes sense to build it
representation-agnostic, but maybe it is over engineering.

~~~
0xCMP
This is true, but unless you understand the problem then (which you might not
since not all of us work at 50x expected load) then how do you architect it
properly while also not wasting time?

------
CodeWriter23
The year was 1991. My friend and I were partying and playing darts. 301, to be
precise. At a certain point, I noted a subtraction error in my friend's score.
I pointed it out so we checked everything and found I made some subtraction
errors too. We were pretty wasted so this was no surprise.

I had a computer in the garage on my workbench. So I decided to solve this
problem by writing some code. About an hour and some number of shots later,
bullshitting with my friend the whole time, I had a working scoreboard
program.

We continued playing darts. We were both amused that even though I was too
wasted to subtract, I could still write code.

We probably should have just switched to Cricket.

~~~
gohrt
how do you know your program was correct?

~~~
CodeWriter23
That's a fair question. Once my ability to subtract was restored (after a
nasty hangover) I checked my work.

------
raz32dust
The bottom line is that whether something is "over-engineered" or not can
mostly only be confirmed in hindsight.

With experience and instinct, some people are successful more often than not
in predicting what level of engineering is sufficient, and they go on to
become great engineers and leaders. But I am not wholly sure if this is more
skill or more luck, though I am sure there is some skill.

With software, it is especially hard. Especially as you go deeper and deeper
into the components. At the UI level, it might seem easy to say what is the
amount of flexibility you need. When designing a common utility or
infrastructure piece to be used by everyone in a large company, it is much
less obvious. But I think the hardest layer is the mid-tier, which has to deal
with the backend and the frontend, and support all kinds of changes and
pluggability.

~~~
wvenable
Not fearing change and being able to handle change is the skill required; and
it can be learned from experience. An over-engineer will attempt to build a
system that is too flexible and too complicated in order to mitigate all
possible changes. Similarly, an under-engineered project will become of
patchwork of impossible to maintain code when the changes roll in.

The skill is knowing how to evolve a project -- create new abstractions and
layers -- in the face of change. To fundamentally alter the product and keep
it working all at the same time.

If you under-engineer a project and you don't know how to do this eventually
you'll be faced with the dreaded re-write. Throughout history how many
companies and products have failed because of that. If you over-engineer you
can suffer with a huge amount weight you don't need. And even then you still
might get it wrong and be stuck with an architecture that doesn't work given
new requirements -- which might require an even more difficult re-write.

So many engineers can't bring themselves to remove code or change their
architecture; so they try and get it right the first time.

~~~
mmazing
> Not fearing change, being able to handle change is the skill required

Agree x1000, I'm currently dealing with a situation stemming from this sort of
thing (not being able to handle change, in software design, and
philosophically with the architecture).

I also keep using the term "evolve" with respects to how to develop a good
codebase. You can use design pressure to drive abstraction and logical
components of a codebase. If you try to figure all that out from the start,
you're going to have a bad time.

------
smegel
> Over-Engineering is when someone decides to build more than what is really
> necessary based on speculation

This is so wrong, in the context of software at least.

Over-engineered software has layers of abstraction that add nothing to the
function or form of the product the costumer sees - it is over-engineering for
the sake of ideological notions about design patterns etc.

As for functionality delivered to the client, software very, very rarely gets
over-engineered - in reality it almost always gets delivered late and with
core features still in development. The idea that there is all this software
being delivered on time that not only meets 100% of all the original
requirements with exacting quality and perfection, but is somehow delivering
in gross excess to the point of giving customers more than they need is pure
fiction.

~~~
hinkley
I don't remember who it was. Might have been Michael Feathers, might have been
Jeff Atwood, could have even been Slava Pestov.

One of them said words to the effect, "people over-engineer because they're so
afraid of making a bad decision that they try not to make any decision at
all." Make a decision. If it's wrong, own it. Develop OTHER coping mechanisms
for being wrong. It's good for the ego, probably good for the soul.

The fights I have gotten into with myopic people over the years have largely
come down to them putting a bunch of energy into what should have been
reversible decisions, and treating irreversible decisions in a cavalier manner
(eg, trying to 'get it over with').

Learn to tell the difference. Be bold with the reversible ones. Put off the
irreversible ones if you actually can - but know that deciding to do nothing
IS often a decision. Like deciding to add security or localization 'later'.

~~~
gaius
_" people over-engineer because they're so afraid of making a bad decision
that they try not to make any decision at all."_

This goes for refactoring and TDD too.

~~~
zem
how does that go for refactoring? when I refactor code I'm typically _making_
a decision that we have now pinned down the functionality of the code, and we
are going to make the structure reflect that functionality clearly and
directly while stripping off incidental complexity that came from exploring
the design space.

------
tzmudzin
I agree this may be subjective, but aren't we crossing the line when the 'how'
becomes more important than the 'what' (i.e. the purpose of the code?)

Following this principle code example 2 (i.e. Javascript after deduplication)
would be the sweet spot. It fulfills the requirements without in the most
obvious way, at a good level of abstraction (the requirements did not make any
of the three pictures specific -- defining a separate function for each
actually goes beyond the requirements, while obscuring the higher-level goal).

All of the subsequent examples focus on using the features of the platform
rather than clearly showing the purpose of the code, i.e. unnecessarily
obfuscate it.

~~~
Mathnerd314
There is a good argument for using $('.image').on('click', clickImage);
instead of attaching three click handlers clickImage[ABC] and hardcoding those
in HTML, which probably explains why jQuery is so popular.

OTOH I agree that using jQuery for animation doesn't add anything,
particularly now that CSS Animation is omnipresent. My initial though on the
problem was "Can I do this in pure CSS?", not to go to javascript. And since
it turns out you can't, the logical next question is why the desired behavior
is actually desired, instead of e.g. just making the image a thumbnail and a
clickable link to the full-size image. So really the entire question is an
example of overengineering, requiring an absurd amount of technology for a
feature you aren't even sure you want.

------
johnrob
The real downside of over-engineering is that time expenditure is non-
refundable. If you build a lightweight v1, you still have the option to invest
additional time into something more "robust". If you choose the opposite, you
don't have the option to get your time back by making the product less
engineered.

Even worse, if your under engineered product is causing problems, you'll
almost certainly notice it (rather quickly too). If you over engineer, you may
never realize it.

~~~
tigershark
Nope, for example if an application is designed without dependency injection
because it is more "lightweight" it is a pain to introduce it later. If you
designed it from the start to have clear dependencies and with an IOC
container then you'll have a much easier time in changing it when there are
new requirements. As you can see the boundary between good engineering and
over engineering is pretty thin. And it obviously depends from the context. In
a toy application probably you can survive without a container. In a real
world project you are going to kill yourself without it (been there, seen
that).

~~~
xupybd
Yes but dependency injection only makes sense at a certain scale. I've yet to
see it provide more than it cost.

~~~
adrianratnapala
I don't see this.

Designing for dependency injection is mostly about passing objects around as a
parameters instead of using globals / singletons.

It's in big code bases where the tempatation to use a singleton is greatest,
because no one wants to refactor all the function signatures to add the extra
parameter.

~~~
taeric
Making every part of your system an injected thing can lead to massive
configuration points where there is only one possible choice for much of what
is configured. That is annoying and I fail to see how it really helps
extensibility.

Oddly, I think the problem is often lack of a global namespace. Consider how
you pull in the "parseFoo" method. In languages with a global namespace, you
just use it. You can put a declaration saying the method has to exist, but no
need to import it from anywhere. That is a build time concern, not a write
time one.

Even better, changing the implementation does not require changing the user
site, at all.

Namespaced languages have to use factories and abstract factories to get the
same flexibility. (Or dependency injection.) I don't even think they're bad
ideas, just find it interesting how much effort it takes to change which
method I'm actually calling.

~~~
adrianratnapala
I agree with you about fractories often being stupid replacements for free-
functions. I am thinking of more dynamic things: e.g. a handle to a network
connection should be passed around instead of singletoned.

But even if dependency injection has all the costs you claim for it. Those
costs are at their smallest when the system is small. So I still say xypud is
wrong: you should do injection by default and when it gets too cumbersome
_then_ you think about how to streamline the interfaces.

~~~
taeric
To be honest...I have not thought hard on this. I expected my post to get down
voted, as I figured i was taking a frowned upon view. To be more fair, I
suspect I am wrong.

In practice, I agree with your point. I'd rather do some small costs up front.
I'm curious to explore the designs more. I think in systems where you have
independent programs working together, a lot of this solves itself. The trend
today is to make things one big program. (Not a new trend.)

------
erikb
Why do we never talk about how this over-engineering stuff relates to skill
development? It's the same, no matter if you look at soccer stars, chess, or
programming.

In the beginning people are clueless and just try to make things change and
take a horrible amount of time for it. Then they start to develop some
patterns. Then they see patterns from other people and choose one as their one
true hammer that will hammer all the nails. Then he spents an awful time
arguing why X and Y are nails. Then he sees that not all methods apply all the
time and starts to balance them with each other. And finally if he gets that
far, he will start to develop his real own, efficient methods.

So where is the over-engineer in this? Not in the psychopath section but in
the hammer-and-nail section. He developed far enough to be effecient in some
ways but still waists quite some time on trying to make all the problems to
apply to his solution. That's normal. That's also about the skill level that
you SHOULD have when you leave a university type of education. In soccer
that's the point where the usual children club stuff ends and they start to
compete in serious competitions, maybe one age level below the one that gets
public attention. You want to put people on this skill level against real
tasks to realise that not everything is a nail.

So instead of always blaming people for being in this level we should
appreciate it. Our university education seems to work for that kind of job
quite well. And then we should continue to put these people in the right
environment to develop further and provide value to their teams despite the
things that they still lack.

------
kev009
I'm trying to think of examples of overengineering in software and am having
trouble from my own experience and extensive reading.

I can think of languages like C++ that facilitate what I would use a more
derogatory term for like autoerotica. The site's example, taking Javascript
seriously, almost always falls into this category.

In something like an automobile, using boxed steel everywhere would make the
thing sturdy as hell at the expense of fuel economy and less objective things
like aesthetics. A bridge design might be too expensive, or a material too
heavy to facilitate the architectural vision. Etc.

In software this term is used where the software is actually under-engineered,
like a shoddy deployment without any consideration of consequences. Technology
like a message queue or distributed database are the victims of poor
understanding or oversold claim. Almost always the real problem is clueless
management and a symptom of fad driven architecture.

So going back to the materials example.. What software wouldn't benefit from
end to end back pressure? Circuit breaker fault management? Memory safety?
Formal proof? None of these things should have any bearing on the user, and
there are a lot of companies could afford to figure out how to make one or
more of these things have minimal effect on front side of SDLC with very
positive effects on the long tail. Software is almost to fluid and inexpensive
to be over-engineered.

~~~
skookum
In the cases I've encountered it, the term over-engineering in the context of
software development is more often used to refer to building in allowances for
future functionality rather than for making the software more robust. Although
I have seen the latter as well - for example massively increasing the
complexity of some part of a distributed system to eke out a theoretical eight
9 of availability.

The concept of over-engineering has been co-opted more recently by Agile
zealots as a major bogeyman to the point where I've seen people trot it out to
argue against structuring code to account for needs that are several weeks out
because "It's not in THIS sprint! YAGNI!", with the predictable result that
several weeks later it creates more work to restructure the code and when
invariably there isn't time for that, leaves a worse architecture in the long-
run as the needed functionality is bolted on.

~~~
adrianratnapala
In my experience YAGNI-abuse happens when people apply it at the wrong level.

For example, people might ignore some rule of the architecture in a rush to
produce some feature. This sounds pragmatic: the feature worked, so that
archtectural rule must be overengineering right?

Then you ask why this particular feature was needed _now_ at this point the
project timeline, and why we had to throw aways fripperies like error
detection to get it. Then I rarely hear clear answers.

------
jondubois
Over-engineering just means implementing something that is more
advanced/complex than what was needed to get the job done (in the short term).

It's not necessarily a bad thing in a technological sense, but it's bad for
business.

If people can build an extremely powerful, complex system which works well and
is stable, then that's great.

Some engineers will stay away from any system which they regard as 'complex'
or which doesn't strictly adhere to their technical philosophies about
simplicity.

Personally, I think that if enough engineers invest their energy into building
a highly complex system, I think eventually they can make it work.

You can just look at operating systems; they are extremely complex but so much
human effort has been invested in them that they have become quite stable and
have become essential to the functioning of our society.

This is how I feel about newer systems like Docker and Kubernetes - I think
that together, they are extremely complex. When Docker started out, I thought
it was crazy, I didn't see the utility value; but enough fools got involved
that it eventually grew into something meaningful. I think that the offerings
we have now with Docker, Kubernetes, Swarm, Mesos, etc... have become
extremely compelling.

I think often it's regarded badly in engineering when you have a single
component that does a lot of different things (often, that's what people often
call over-engineering) - But look at the general-purpose CPU; we didn't need
to invent it; people could have been satisfied with building specialized
electronics for every kind of machine that just did one job well (no
programming needed) but this would have slowed down innovation (because
manufacturing circuit boards is expensive and time consuming).

Making generic components that do a lot of different things CAN be extremely
useful when you look at the big, long-term picture. It just has to be done
well and needs to be given the time to stabilize.

------
partycoder
An over-engineered system is a system where the complexity of the requirements
is not proportional to the complexity of the solution.

The part that is counter-intuitive, is that a rather simple idea can sometimes
have very complex requirements.

~~~
fagnerbrack
Idea: As a writer, I want a platform to write a story that allows me to focus
on the content instead of the formatting

Solution 1: Build the platform using JavaScript. Do user research to come up
with the best design. Hire a marketing team to receive a lot of attention and
consecutively feedback on the platform. Hire data scientists to analyze the
site traffic and decide which feature to add or not... in the end, through
small cycles of feedback, continuously improve the platform and spend millions
of dollars to come up with something very good after some time

Solution 2: Use medium.com, because sometimes a rather complex idea can have a
very simple solution.

~~~
partycoder
Solution 1 is a committed solution.

Solution 2 is an uncommitted solution.

You can always go from 2 to 1 at a low cost.

------
fuzzfactor
For motorists, over-engineering is how the engines worth buying are those
which are expected to last 10 years even though they are only warranted for
five years.

It's the way many bridges having a posted weight limit of 10tons can handle a
12ton truck without incident.

If you go to all the trouble of actually engineering something to begin with,
might as well include some effort into making it stand the test of time while
performing beyond expectations. One of the most fundamental features of true
engineering is a "margin of safety".

Traditionally this is what increases the chances of your product having that
most useful feature; "worth buying".

------
ofir_geller
The specs are the part I found most interesting. 1) First the specs say 3
images, then they say 2. if you are dealing with a manager that can change the
specs like that you might want to over engineer just a bit to cover 'arbitrary
number of images'. yes I know this is probable just an error by the writer of
the article, but it demonstrates something that happens in the wild. 2)
Communication is the missing piece in this story. when a developer is given a
task they should ask questions about future requirements. "Is it always going
to be 3 images?", "will the images always grow by the same ratio?" and so on.
the way to not solve imaginary problems is to talk with the stakeholders.

~~~
fagnerbrack
Good catch! That was a typo. And very good remarks on how to relate that to
the real world, definitely agree with that!

(Btw I have fixed it :D)

------
richie5um
I think 'over-engineering' is too broadly used. Clearly some solutions are
over-engineering, but most are built by skilled and experienced engineers who
know that requirements change / evolve. In this (common) scenario, not
predicting it - and creating the necessary 'over-engineered' solution - will
lead to a failed project. Alternatively, predicting the wrong thing to 'over-
engineer' is equally problematic.

Without these gambles, the project will often fail too.

It's a really rather difficult problem.

------
analog31
Managers are always suspicious that engineers are doing the wrong things.
Living under this cloud is just part of being an engineer.

Granted, my experience is more with hardware than with software, but getting a
design "just right" can be costly in analysis and testing, and there's a
palpable concern that if you aim low rather than high, a substandard product
will get pushed out the door. It can be exponentially more costly to correct a
weak design late in a project than to over-design it early: Subsystems and
schedules become more inter-dependent, the closer you get to launch.

I can give you a good design in six months, or a bad design in twelve. ;-)

------
rdlecler1
The take home point I had from this article was that the requirements of will
determine what you need to build. If this is a startup with minimal resources,
the requirement is that you need to create something fast to see if you can
solve a problem. At that point you might take those learnings, throw away the
code and build a more extensible and robust version 2. The problem I often see
is that people over engineer based on a set of requirements which assumes that
the problem you are trying to solve is the right problem to solve.

------
andrewprock
In my experience, over-engineering is driven by a fear of refactoring. For
whatever reason, refactoring is a process that is both culturally and
technically undernourished.

------
DenisM
Two take-aways for me:

1\. Make sure engineers are engrossed in the business case. This sets proper
scope.

2\. Don't generalize something ahead of time, only _after_ you do the same
thing twice. The architecture should support easy refactoring, and there
should be certain discipline about actually _doing_ the refactoring the second
time around.

There are many challenges to overcome with both of these, but it will be worth
it.

------
Ericson2314
In my experience we live in an era of near-insurmountable technical debt, and
gratuitous complexity as the quick-and-dirty solution is always slap on
another leaky abstraction.

Under-engineering is the problem for anything that's supposed to last.

------
stevesun21
My definition of Over engineerkng is that you try to cover many edge cases
which are not belong to the current system scope, or you try to concern system
performance improvement cases which are uncertain.

------
iagooar
Let me tell you a story about companies that often confuse thinking about the
best solution with over-engineering...

You get hired by this startup with great ideas. They have really hit their
niche nicely and are getting more and more customers. You build a great
throwaway prototype and customers and the mighty CEO love it. You ask when you
are going to rebuild the prototype, since it's a bit of a mess... The hipster
CTO says that there is no need, that you are the greatest dev ever and that he
will get a tattoo with your name.

The first change request comes in. Easy. It's still straight-forward to make
some quick and dirty changes here and there. You call it a day and your
hipster CTO throws a party to celebrate the amazing job.

After change request 10, you feel like slowing down drastically. The mental
model you could fit in your brain until now, is too complex. You can't touch
all the involved parts of your application anymore or it would cost you a lot
of time. You get away with some if/else statements to fix the edge cases. You
feel happy again, you did it! You call it a day, your hipster CTO pats your
back and gives out some beers.

After change request 20, the system is a mess. Different layers of the
application talk directly to each other, thus creating painful coupling. Your
abstractions are either non-existent or all in the wrong places. You have no
CI, because that would have been over-engineering the test suite. Your local
tests take half an hour, because making them run in parallel would have been
over-engineering. The time for documenting the project was spent hunting
quirks somewhere in the darkest places of your application. You don't even try
to touch some of the code you wrote just a couple of months ago. "There be
dragons" is, next to "TODO", the most used comment you find in the codebase.

So after many days, the job isn't done yet. The mighty CEO gets nervous and
starts running all over the place, giving you some "leadership". He says that
there is a deadline not to be missed or the universe will collapse. The
hipster CTO even has to skip his yoga-classes to meet the deadline. After many
extra hours and even more energy drinks, you get the change request done. The
hipster CTO just smiles at you, but no beer this time.

There is 10 new bugs but you don't know about them yet. Soon you will, because
people will start complaining about that last change you did and the bugs you
introduced. By this time, you are sick and tired of it all. You hate yourself.
You feel guilty. Problem is, you don't even have the time to think about how
tired you are. Request 30 is going to arrive soon...

But hey!, don't forget that you HAD TO ship it quick, not think about solving
the problem! Abstractions? Over-engineering! Designing for resilience? Over-
engineering! Small, independent services? Over-engineering! Staging system?
Over-engineering! All these thoughts come to your mind everytime. Yet, you are
so tired, you fall asleep and try to forget it all...

WAKE UP! THERE IS A BUG IN PRODUCTION!!!

You need to rush. In the middle of the night, your messy apartment gets
illuminated by the blueish light of your laptop. You don't even care anymore.

The End

\---

Appendix

By change request 40, your soul committed suicide.

By change request 50, the company was spending 80% of their time doing bug
fixes and keeping their self-build house of cards running.

By change request 60, there was this tiny competitor who gathered together a
nice little team of skilled and motivated engineers. The CTO would give them
time to think about how to solve the problems best. Soon they entered the
market at an amazing pace. All competition went out of business. You lost your
job. You have no motivation. You feel lonely.

By change request 70, which never actually happened, that tiny competitor sees
you as a valuable asset to their company. Since they are growing, they need
people like you, with experience and skills. You still feel like the worst
developer ever, but you take the job. People respect you. The software was
designed following best practices. It is stable, people iterate quick. You can
take your time to think about the best solution. People share their ideas and
with each iteration they make it better.

Finally, after all, you are happy again...

~~~
Udik
> The software was designed following best practices. It is stable, people
> iterate quick. You can take your time to think about the best solution.

Except that the messy project you were working on before sounded like a one
man project. And at every new change request you delivered, working by
yourself, a pretty big chunk of value.

Now, in the nicely engineered project, you work in a team. The team is
iterating in sprints that require a number of developers two or more weeks of
work to deliver the tiniest feature. Yes, iteration is quick, but it doesn't
really deliver much value. Every time the new CEO requests something even
slightly different from the exact functionality that was developed so far, the
project architect's answer is that this will not fit in the current set of
features, and it might be possible, if ever, after another series of sprints
that will take multiple months to complete.

And the funny thing is that, while in your one man project you were considered
a lone cowboy working on a relatively simple system, the architect of the
current project (that delivers more or less the same functionalities but
requires 5 or 6 times the time and resources to maintain) is considered a
demi-god "because the project is so complex". Because of course the CEO
measures the complexity of the project not but what it delivers, but by the
amount of resources consumed by it. And while the CEO of the previous company
was annoyed when a new feature required more than a week to implement all by
yourself, now whenever the new architect says "we need more resources for
that", this actually _increases_ his power and consideration in the company,
as it is taken as a sign of the complexity of the project.

------
Frqy3
> more robust or complicated

In my experience (mostly in networks) these are two different things with a
negative correlation between them.

------
hiou
And within all of that nothing is included in the code to handle double clicks
or event bubbling.

------
Gravityloss
Are the DC-3, B-52 or W124 overengineered? I fear the gentleman is misusing
the term.

------
basicplus2
I tend to associate the term "over engineering" as a positive term..

One can't lump safety in with poor engineering.

seems to me the article is talking about something akin to "wrong engineering"

------
optionalparens
Over-engineering is a tough thing to discuss and there's never going to be a
suitable answer anyone agrees upon. It is also a huge topic with a scope far
beyond this textbox or a medium article.

Anyway, among many of the big issues here are so many things are contextual -
the problem domain, stakeholders, programmer abilities, business climate, etc.
It's been my experience that you cannot apply judgement to a solution without
understanding as many of the variables involved as possible. Shifting these
variables even slightly can change everything.

For example, say I build a feature A at company Y and then I change jobs next
week and to company Z which is much smaller. Company Z wants the same feature
I just built at company X, but of course I can't steal it outright. Building
the same thing at company Z requires different engineering decisions, given
the different climate and probably different development team, programming
language, and so on hypothetically. Even if I could just copy the solution, at
company Y it might be perfect, while at company Z it might be considered over-
engineered. It's a combination of both objective and subjective judgements.

As for the concrete implementations of features, I often am driven crazy by
both over and under engineering. I worked with several people in positions of
power above or horizontally to me that would routinely shoot down many well-
engineered features and implementations by other people because they were,
"Too complicated." I began to realize this was often shorthand in most cases
for, "I know nothing about this, my ego or political stance is threatened, I
lack talent, and/or I am simply a moron." Of course I also hit plenty of
actually over-engineered things that were indeed exactly this, but in many of
these cases it was obvious. Where it is less clear is anything of decent
scope, challenge, subjectivity, or other hard to quantify measure. I think
people can look at something like the FizzBuzz and see that the parody
versions using OO 10 design patterns with abstract factories are obviously
over-engineered, while it gets much harder in the context of a big project.

And this leads to another problem. Complexity is not over-engineering, nor
vice-versa always. Nor is scope, or code size, or any of these related
concepts that come to mind. A very small amount of code can be incredibly
complex, a large amount of code can be simple, and so on. Either can be over
or under engineered. Just look at math - some of the most useful tools are
very difficult to understand, while others are incredibly simple with tons of
layers, nuances, and so on. Programming is not that much different in this
regard. We've all seen things done in a tiny amount of code we would have
never thought of and maybe can't even understand. Likewise, we've encountered
huge amount of code that someone complains about but is written clearly,
thoughtfully, and cleanly. These's a good related talk on some of these ideas
by Rich Hickey if I recall.

Another issue is that it is hard to quantify both knowing what and when
justifies substantial engineering efforts. As I mentioned, context plays a big
part, as does intuition and experience (in relation to ability, not just pure
years). These are all nearly impossible to quantify, and yet must play a large
role in cases where large engineering decision making. A lot of people see
things they don't understand or are new to them and just throw out the over-
engineered card. Likewise, a lot of people love pulling out their favorite
tool (ex: SQL, Framework of choice) and try to use it as a golden hammer.
Anything else is "over-engineered." These things can lead to over-engineering
or at the very least, poor engineering.

Over-engineering is also something that is relative. It is really hard
sometimes to work with someone else and to get them to understand especially
those hard to quantify things such as intuition. Programming can also often
highly be about ego and ownership. People will not hesitate to point at
something not written by them as over-engineered because it was not done how
they would do it.

Game programming for me again comes to mind as an area where this is a
constant pain. The inexperienced game programmer will both under and over
engineer things like you've never seen. First, they'll try to use their
favorite language to write a game. They'll then do things when they learn a
technique like OO programming and try to apply it everywhere, to everything.
"A monster? Obviously I should make a monster base class, then subclass this
for an Orc, then subclass it again for an Elite Orc." Everything must be
designed around the tools they know instead of the problem. The minute this
person does enough in their career to get any real authority or a real job,
they take this unchecked mentality and try to impose it on the people around
them. I see the same thing in web application and desktop programming. Someone
learns lambdas, functors, currying, design patterns, IOC, monads, etc. and
they must apply them to any situation that triggers this recognition. The
"engineering" then comes from the solution and tools, perhaps the XY problem
in many cases. Usually these people also fail to consider the aforementioned
contextual concerns.

I know in game programming this happened often where some programmer would
come in and replace some lengthy looking code that made heavy use of arrays
and manual for loops. They'd then go in and replace it all with vectors,
iterators, and all kinds of heavier stuff. They never considered that the code
was engineered that way for performance or for portability or backward
compatibility or some other reason. Moreover, they didn't actually know that
the seemingly "simpler" solution was actually making things as a whole more
difficult by pushing out these concerns to be fixed or handled elsewhere (ex:
making up performance somewhere else).

On a personal note, I am often annoyed by the over-engineered finger pointing.
There are so many things like XML messes and 200 steps to setup something for
a simple purpose that are easy enough to agree on as over-engineered. Most of
these come down to asking yourself, "What problem am I trying to solve" and
perhaps repeating those steps aloud. If you feel ridiculous saying them,
especially to another person, something might be wrong. But if it turns out
that the problem you are trying to solve really does require something that is
not a sound-bite, the problem explanation will also often have to be long and
cannot be skirted around without ignoring the problem.

Too often I see things in this industry thrown out there to "fail hard," to be
an MVP, to test the market, to disrupt, to fill a gap. That's all fine, but
make sure you are solving the problem and not creating new ones like security,
performance, misleading people, defrauding investors, or worse because you
thought engineering a proper solution is too hard or takes too long. Find the
sweet spot and make informed decisions. I recommend another talk about Rich
Hickey (sorry two by the same person, just memorable talks) about Hammock-
driven development and thinking before acting. If more people did that, it
could swing either way and maybe we would have less short-term "stuff" but
better long-term solutions, less over-engineered messes, and things that
actually are worth building on top of as bases. As it stands now, this
industry is littered with poorly engineered solutions we were stuck with
because of both over and under engineering.

------
justinlardinois
> A famous comic using the attempt of building a tree as the analogy of what
> is wrong in each part of the traditional development lifecycle

The project in the comic actually appears to be building a _tree swing_. I'm
sure this was just an oversight on the author's part but if anything it
reinforces the point the comic is making.

~~~
fagnerbrack
Author here. You are right, I did not know the name of the thing because
English is not my primary language. Thanks for teaching me that, I have edited
the picture caption to show the correct description of the image. Thanks!

~~~
zachrose
I've never heard of a "tree swing" before, or any swing named after the thing
it's hanging from.

I would call the thing in the last panel a "tire swing." (Native English
speaker, FWIW.)

~~~
justinlardinois
Yeah, it's worth noting that "tree swing" isn't a common idiom; I only meant a
swing that's set up on a tree. I just didn't want to call it a "tire swing"
though because it only took that form in one panel. Maybe "rope swing" would
be the best name for it?

~~~
inimino
It's just a swing.

~~~
taneq
So you're saying the term 'tree swing' is overengineered? ;)

------
idlewords
This engaging article on overengineering is 2.7 MB in size, and takes over a
hundred web requests to render.

~~~
9935c101ab17a66
So? The author didn't 'engineer' medium in order to distribute this post.
Maybe he was already familiar with the site, or maybe he finds it the easiest
to use?

~~~
megablast
Oh, so there is a good reason to accept over engineering.

~~~
markcerqueira
This argument makes no sense?

~~~
logicallee
The argument makes perfect sense. Nobody can use the theoretical swiss army
knife pictured in the article - it's too wide, and ridiculous.

In the graphicon of over-engineering taking over users and enveloping them,
pushing design out of the way, it implies that the result is hostile to users,
and pushes design out of the way.

But where we read this is an example of the person selecting the "knife" in
question out of every single blogging platform on the planet. So, the swiss
army knife pictured is inaccurate, it's a straw man. Secondly, the author
clearly finds the design appropriate enough to use, and a great match for
their user stories.

So whereas _the_ central thesis in the article says:

> "small, elegant, and successful systems tend to be plagued with feature
> creep due to inflated expectations".

The author is _not_ using a "small, elegant, succcessful system".

Perhaps the swiss army knife pictured is the best knife on the planet, and not
one of the worst. This fundamentally undercuts the point the author is making.

~~~
markcerqueira
I believe the comment I was replying to was saying one should not use Medium
because it is an "over-engineered blogging platform." I was not arguing for
creating or using theoretical swiss army knifes.

