
Modern Software Over-Engineering Mistakes - rbanffy
https://medium.com/@rdsubhas/10-modern-software-engineering-mistakes-bc67fbef4fc8#.txrola3hf
======
sidlls
He missed at least one: functions should be "small" and broken up for
"testability."

No, functions should perform one task and perform it well. Not every
conditional test inside the function needs to be its own function and have its
own unit test. Sometimes a function that does just one thing and one thing
well has more than one step to do that thing, and those steps don't
necessarily belong in their own functions. Testability and composability are
important, but that has to be balanced against locality of reference and
context. When I see a colleague write in a code review that pieces of a
function should be factored out of a larger function "just in case they'll be
reused" I step on it hard. This is related to "over generalization" but not
exactly the same.

~~~
20years
Yes, yes! Having to locate and dig through 10+ different smaller functions
that are called within another function is much harder to troubleshoot and
work with. Especially when 95% of those are not re-used anywhere else.

I know it may not be the popular belief but I would much rather work with a
larger well written 50 line function that does what it is supposed to do vs
having to navigate around a bunch of smaller 5 line functions that are never
re-used anywhere else.

~~~
potatoyogurt
If it's done right, though, you shouldn't have to navigate around all the
smaller 5 line functions, at least not unless you're debugging. Each of the
submethods should clearly describe what it does, and do that without any
unexpected side-effects or implicit state changes. Effectively, what this does
is raise the level of abstraction of the code, so that your 50-line function
becomes a 10-line function, and you only need to read those 10 lines to
understand what's going on. Sometimes when you're debugging, you might need to
dip into these subfunctions, but when you do, you should be able to think
through them as mostly independent units that just need to do what their name
says they will do. This ideally limits the problem of having to load
everything that your code is doing into your head all at once in order to
reason through it. When you're just reading through the code normally, you
should be able to gloss over the details of how the submethods work, just as
you probably wouldn't feel compelled to dig into the internals of how a
library call like Collections.sort works.

The problem is that method decomposition is not always done well, and when
there is unexpected behavior or implicit state changes, it can make bad code
even harder to navigate. I'd also rather read a well-written 50 line function
than a poorly decomposed cluster of 10-line functions. With proper care,
though, method decomposition is super helpful.

~~~
sbov
Part of the problem is that the abstraction of "a sorted list" will never
change. But the abstraction of "twiddle my foobar" can change on a weekly
basis based upon business needs. Because of this the abstractions we write
will never be as strong or enduring as those found in the standard library.

~~~
potatoyogurt
Certainly true, but that's not a reason to give up on creating those
abstractions, like the post I was replying to was suggesting. It's a reason to
be judicious about when it's appropriate to create these abstractions, and in
particular about when they should be shared. An abstraction that's used in
just one or two places is still really easy to change. A bad abstraction used
in a dozen places is a pain the ass to clean up. It's not an easy problem, and
it ultimately takes some combination of experience and careful mentoring to
learn to identify good and bad abstractions before they're built.

------
ams6110
Back in the days when I worked on a largish "classic" ASP site (each page is a
file full of mixed HTML/VBScript) the senior developer insisted that the best
approach for developing anything new was to copy a page that did something
similar and then change it to fit the requirements.

There was no code re-use.

Sounds insane by today's practices but in reality it worked well more often
than not. Business functional changes almost always applied to just one or a
small number of pages. You could change those pages with impunity and be
pretty confident that you would not break anything in any of the hundreds of
other pages in the site.

Rarely this approach caused more work when a change did affect dozens of
pages. But on balance it made most changes much easier to implement and test.

~~~
tluyben2
Yep, that is one of the reasons PHP and ASP got such a bad name. Projects like
those take a few greps and minutes to get into and get you to productively
make changes without much risk. And, outside HN, it is still quite common
because of it.

~~~
rbanffy
> without much risk.

The trick is to strike the right balance between repeating code and testing
it. I've seen codebases become unmaintainable piles of almost repeating code
that was never tested beyond a developer opening the page and checking the
behavior manually.

To prevent such messes is one key responsibility of a developer.

------
jacques_chester
Those who forget history are doomed to be dogmatic software developers.

A lot of stuff we take for granted are either accidents of history, or
powerful counter-reactions to the accidents of history.

There is a practice, and it turns out to be bad. Mild discussion of the
virtues and vices would, in a world composed of Asimovian robots, be
sufficient to update the practice to something better.

But that's not how humans work! Typically an existing practice is only
overturned by the loudest, most persuasive, most energetic voices. And they
_have_ to be. Humans don't come to the middle by being shown the middle. They
come to the middle after being shown the _other fringe_.

So a generation changes its mind and moves closer to the new practice.
Eventually, that is all the following generation has ever heard of. The
original writing transforms its role from mind-shifting advocacy to the
undecided to being Holy Writ. The historical context, and with it the chance
to understand the middle way that had to be obscured to find the middle way,
is lost.

My previous role was as an engineer teaching other engineers an XP-flavoured
style of engineering. I often referred to our practices as "dogma", because we
_are_ dogmatic. But if we aren't, less learning takes place. Dogma is most
instructional when someone later finds its limits.

When I was learning to coach weightlifters, I was told something that has
always stuck with me: "As a coach, you will tell trainees a series of
increasingly accurate lies". You can't start with nuance. In the beginning, it
won't work.

~~~
nradov
You can start with nuance and openly acknowledge the lie. This doesn't inhibit
learning. Every modern textbook on Newtonian physics tells students that it is
effectively a "lie", in that it's only an approximation that works reasonably
well in most real world cases. But you have to start there before learning
general relativity and quantum physics.

~~~
jacques_chester
It depends. Each case is different, the safest place is the dogma.

~~~
TeMPOraL
You may have more experience than me, but at this point in my life, I find
myself disagreeing with this. In fact, one of the biggest problems I had with
education is teaching dogmas. On the other hand, when I did teaching, tutoring
and lecturing, I always tried to make it clear and explicit that what I'm
telling is a practical simplification, that it has limits here and there, but
_within these particular constraints_ it's a good approximation. And the
feedback I got was always that it made things much clearer to people - people
felt it makes sense, because it had context.

~~~
jacques_chester
I guess I am doing a bad job of explaining.

What I taught was a way of working. I didn't deviate from the practices,
because the principles are easy to state but hard to truly grok.

Going back to what I said earlier, this is the difference between
weightlifting drills for various parts of the movement, versus discussions of
physiology, anatomy, anthropometry or physics.

You start with the drills.

~~~
TeMPOraL
Thanks for the clarification. So it's something like, first learn to do
something in a decent way, and only then - when you're familiar with the
subject matter - start thinking from first principles?

~~~
jacques_chester
Sort of. It's mostly about instilling habits of thought -- eg asking "how do
we test this?" before touching a keyboard.

Learning when and why to break the rule takes longer. It helps to first go
through a bunch of concrete examples.

------
caseymarquis
I think working on a large project in c89 made me better at writing in other
languages.

When you strip all your useful tools and concepts away, you're forced to
rethink how you can organize with just data types and functions. Surprisingly,
you can do pretty well with just these.

It's the sort of thing that helps with recognizing when you're looking at
FizzBuzz and when you actually need to use a generic factory.

~~~
firasd
When I started working with PHP over a decade ago I had some experience with
Java so I thought it was funny that in PHP everything is an 'array'. Now that
I'm working with JS and Elixir and getting into functional programming I'm
realizing, that hey, maybe that _is_ the right way to go. Putting everything
into structures like maps helps blur the line between code and data.

~~~
cygned
I have quite some background in OOP and switched to functional programming a
few years ago, now working a lot with Clojure, where you actually have the
"code is data" paradigm. And from my experience, it's way easier to model data
with three simple structures; vectors (= arrays), maps and sets. There are
other things, too, but most of the time these three types are sufficient. It
requires some discipline in the beginning but it can give you a lot of
benefits.

~~~
Roboprog
You are going to be lectured at length by the (OOP) static types crowd. And
they have a point about using explicit types, _when_ it's easy.

That said, I wish the "strong types" crowd would take a look at all of the
temporal coupling that their OOP designs are causing, with their update-at-
will practices. Now that we have working garbage collector software plus
adequate hardware support, why not make use of that?

FP needs to become a more widely used practice.

~~~
jghn
Some of the largest "strong types" advocates are FP folks

~~~
Roboprog
Some are, some aren't :-)

Some are just lost if autocomplete in the IDE doesn't enter stuff for them,
and the more verbosely the IDE spews, the better, cuz it looks like "work".

------
firasd
I like this part: “Duplication is sometimes essential for the right
abstraction. Because only when we see many parts of the system share “similar”
code, a better shared abstraction emerges. The Quality of Abstraction is in
the weakest link. Duplication exposes many use cases and makes boundaries
clearer.”

I just did this yesterday... copy/pasted some code from one function to
another function, tested that the new function works and moved on, and then
when I went back to work more on it, I wrote more generic code that can be
called by either function. Don't have to overthink things before writing the
first function.

~~~
crdoconnor
I tend to see this when people try to pre-emptively abstract. Not "we seem to
be duplicating 3+ blocks of code here" but "we _might_ want to do this again
at some point so we need an abstraction to make it easier next time".

The former almost always works while the latter almost always fails.

The former gives a little dopamine hit and a boost to the ego because that's
being "an architect" while the latter feels more like being a janitor. Ironic
really.

~~~
cygned
Interestingly, the second approach is often recommended in functional
languages, although in a slightly different way; compose complex/complicated
logic of small pieces. You end up with a lot more functions but it allows for
code reuse.

I am note quite sure, though, if we can already talk about an abstraction
then.

~~~
AstralStorm
There is no code reuse of small non-generic functions in practice, just
duplication and thus ravioli code. And to make these small functions generic
costs time.

What may make sense is to make something similar to lemma in proofs. This is
strictly for ease of debugging or readability.

------
partycoder
Point #7 are called non-functional requirements. Those are often implicit from
non-technical people. Nobody will say: "I don't want to be hacked" or "i don't
want this system to slow down and die"... This guy says "we don't need NFRs,
focus in the functional requirements"... well, that's exactly the cause behind
most engineers that get fired for technical reasons: causing one or more
serious incidents by not caring about those requirements.

Point #2... seriously? Keep your code consistent, and if you identify
opportunities for reuse for things that are related then do it. Copying and
pasting code, which is what this guy is advocating for, is not good.

~~~
mathgenius
> Copying and pasting code, which is what this guy is advocating for, is not
> good.

Well obviously this is why we have variables and functions, etc. And hopefully
this helps enforce DRY ("don't repeat yourself.")

But, reusing a component leads to coupled code and that can also spell
disaster. Sometimes it really is better just to copy stuff. Maybe this is a
matter of taste; some people like to have the perfect design and if that makes
them a happy coder then i guess you should let them do that. I find that, more
often than not, the "perfect design" is not worth sweating over. My style is
tracer-bullet: get it working today, write some tests maybe, and then refactor
it.

~~~
taeric
I like the term "tracer-bullet." You should get that written up as a blog, if
it isn't already.

Reminds me of an anecdote about getting people onto the moon. When that was
their goal, the very first thing Nasa did was make sure they could hit the
moon with a rocket. Then iterated from there.

~~~
TeMPOraL
> _I like the term "tracer-bullet."_

The term is actually from the book Pragmatic Programmer (a good book, though
I'd say mostly for beginner-to-intermediate programmers, since a lot of
software blogs are essentially just repeating the contents of this book ad
nauseam).

I tend to do that too - get something working fast, and iterate over it.
Focusing on perfect abstractions makes no sense if then you find out you're
aiming in the wrong direction.

~~~
taeric
Embarrassingly, I should have known that. I just checked to make sure that
wasn't where I heard the "just hit the moon" quote.

I find I have been terrible at obsessing over abstractions I could be proud
of. I try to instead keep my eyes on solutions I can talk to.

------
DenisM
The most interesting takeaway for me is this: [...] The best quality of a
Design today is how well it can be undesigned [...]

Aiming for design that's easy to refactor and/or replace bolsters application
longevity at the expense of code longevity. I like that. It's like the
ecosystem longevity is achieved at the expense of the longevity of individual
organisms...

Now then, who has ideas, or best practices or even just anecdotes? I'm eager
to hear those!

For my part, I follow design philosophy that revolves around persistent data
structures. My code is supposed to be just an accessory to the data. I don't
think I'm explaining this well, it's just where I put all my focus on. Another
principle is to try capturing users intent rather than the outcome of users
actions. This way I can redo the code that detived outcome from actions and
fix some the earlier erroneous behavior.

~~~
firasd
_My code is supposed to be just an accessory to the data._

Fred Brooks: "Show me your flowcharts and conceal your tables, and I shall
continue to be mystified. Show me your tables, and I won’t usually need your
flowcharts; they’ll be obvious."

------
hliyan
Of all the 'maintainable' design pattern use I've seen over the past decade, I
have never seen one prevent a continuously maintained code base being
rewritten periodically. All that work is thrown away just as a simple to-spec
implementation would have been.

It seems that just as in literature, in software engineering too, the essence
of writing is _rewriting_...

~~~
DenisM
Accepting what you said, there is still one pattern that's helpful - writing
code that's easy to rip out later. It's covered in the article as well:

[...] The best quality of a Design today is how well it can be undesigned.
[...]

~~~
hliyan
Agreed! Piecemeal rewrites instead of big-bang rewrites...

------
edejong
Point #8 sounds great: reuse OSS, don't reinvent the wheel. However, this only
applies to software that is maintained, now and in the future. Relying on
obscure code-bases without knowledge on its inner workings is going to bite
you down the line.

Examples? Grunt, Bower, Hibernate, Apache Commons. How long did it take the
Apache Commons project to properly add generics to its libraries? How is
backward compatibility and developer availability holding back projects down
the line?

Additionally, in order to use a library, you need to have a good knowledge of
the problem it tries to tackle. By overly relying on open source software, you
might blunt the competitive advantages of your business. It's an example of
exorbitantly relying on abstraction.

~~~
dom0
I like to think of this as a _dependency cost_. Adding a library may "relieve"
implementation cost, but adds a dependency cost (ie. finding+evaluating
dependencies, integrating and maintaining the dependency, testing the
dependency and so on).

This cost varies a lot, for example a library to left pad a string may be
easily replaced, but if you choose a library to implement some on-disk format,
then that can cement things in. If the library becomes unsupported one very
well may end up maintaining it alone, for example.

In the more general case more issues are attached: choosing a deployment
stack, a documentation or translation tool etc. can result in decisions that
are essentially set in stone. These also tend to be composed from multiple
tools, leaving more wiggle room. Deployment for example might be done with AWS
and Ansible on the higher levels, but use in-house tooling for figuring out
the details.

Similar arguments apply to code de-duplication and DRY: de-duplicating code
isn't free, and not always the right thing to do.

See also: [http://blog.liw.fi/posts/dependency-
cost/](http://blog.liw.fi/posts/dependency-cost/)

~~~
collyw
This varies a lot. Python generally has good libraries. JavaScript (in my
experience) generally has far more fragile libraries and its a lot faster
moving, so the dependency cost is more likely to bite you. Usually there are a
few libraries doing the same things, so go for the mature ones.

------
iabacu
This is a great article, thank you very much for writing it.

I pretty much agree with all points, except number 9 could probably be "Not
challenging the Status Quo" instead of "Following the Status Quo".

Breaking the status quo just for the sake of it would be a mistake, while
healthy challenge of status quo with open mindedness to accept to not change
anything is probably a better direction.

> Areas of code that don’t see commits for a long time are smells. We are
> expected to keep every part of the system churning.

Or it could be an indicator of mature features that were well designed and
implemented to minimize future headache: they just work and have very few
stable dependencies, if any.

------
clifanatic
One of the (really sad) reasons I suspect is behind a lot of these practices -
like pointless wrappers and pointless genericization - is that nobody will pay
you to "understand something". You have to produce something, even if you
don't quite understand what it is you're supposed to be producing. Sure, you
can spend an hour reading a horribly written-by-committee "requirements
document", but you had better be producing something that looks like a program
by the end of the day. Since admitting that you don't quite yet "get" the DB
layer is a fast-track to the unemployment line, some developers have learned
to buy time by creating meaningless abstraction layers while they're trying to
figure out the inner details of OAuth or Cassandra or whatever else was
supposed to be "saving us time".

------
lordnacho
Those things are not necessarily all that terrible. I can see how overeager
coders might do any of those things in anticipation of some as yet unseen
requirement. And it's not always clear why a code base evolved the way it did.

Also let's not forget the opposite. I've worked in places where everyone just
wrote their own spaghetti, no concept of version control, and every time
there's a small change it takes ages for the only coder who wrote it to
untangle and modify it. Basically a steaming pile of turd, used to invest real
money in the real market. The worst part about it is when you call them out
it's YOU who doesn't understand the requirements.

~~~
ThomPete
The thing is that just because it's spaghetti code doesn't mean it's not
successful and thus it can be fixed.

Premature optimization is the developers version of pixel perfection. They are
both the enemy of getting anything done.

~~~
tzakrajs
Spaghetti code is so frustrating to refactor, that you sometimes may as well
rewrite to maintain your sanity.

~~~
ThomPete
Sure but then you do that. Its better than having spent an insane smount of
time on something no one ends using.

~~~
AstralStorm
Write a serious test suite before doing so. Otherwise you will miss a bug
31337 fixed in 19xx and will be sad later.

------
gtirloni
Besides agreeing with the article on many points, I thought the examples were
gold. Too often articles about best practices are too abstract.

I would love to read more practical articles like this. Any URLs people could
share?

------
quickben
11\. You need Big Data.

~~~
KevinEldon
Or "You have Big Data". Paraphrasing Greg Young's idea, "If you can put it on
a Micro SD card that you can buy on Amazon, you don't have big data". So if
your data fits in 128GB ($37.77 on Amazon right now) you don't need big data
solutions.

~~~
kabes
I would say that the definition of what is big largely depends on the problem
you try to solve. If that problem is finding keywords in text files, then your
definition sounds about right. For other problems even a couple of KB might be
big. To me, big is when your dataset is too big to solve your problem in
reasonable time on one machine.

~~~
firasd
Right. I'm not comfortable with this idea of big data being dependent on the
actual file size of the data. There really are problems where doing stuff like
reading files or using a relational database break down and you need something
else that's more specialized in solving a problem even if the dataset is just
a couple GBs. (So in my mind big data is a reference to specific tools or
approaches like map-reduce, etc.)

------
partycoder
Here there is a more mature list anyways: [https://en.wikipedia.org/wiki/Anti-
pattern#Software_engineer...](https://en.wikipedia.org/wiki/Anti-
pattern#Software_engineering)

------
smegel
Not modern software, just bad software, written by Java developers.

Not everyone thinks or codes like this, and those that don't are not all
greybeards.

------
tbrownaw
_1\. Engineering is more clever than Business_

...I don't get it? This sounds like it should be saying we try to "plan ahead"
too much, but then the description seems to say we don't do enough.

 _2\. Reusable Business Functionality_

I think this is arguing against doing too much design work up front? From what
I've seen, incrementally growing a system tends towards the opposite problem
unless I'm _aggressively_ looking for refactoring opportunities.

 _3\. Everything is Generic_

This isn't a case of too much vs enough, it's a case of correct vs incorrect.
If you can guess how the business requirements are likely to change, making
things generic _in the right way_ will make that change much easier to do.

 _4\. Shallow Wrappers_

Yeah. Unless you have actual advanced knowledge that you'll need to switch out
a particular library, this should be done on-demand as a refactoring step
before such monkeying around. Except for things where you need a seam to make
testing easier.

 _5\. Applying Quality like a Tool_ and _5.1. Sandwich Layers_

...does anyone actually think this way?

 _6\. Overzealous Adopter Syndrome_

Maybe, but keep in mind these can also be used to clarify _intent_ or to
intentionally constrain future changes.

 _7. <X>–ity_

The examples look like things where pursuit of whichever <X>-ity _didn 't
actually work_, rather than cases where it wasn't needed.

 _8\. In House “Inventions”_

These tend to be a result of either very old systems that date back to before
an appropriate reusable version became available, or organically grown systems
that had parts gradually come to resemble some existing reusable thing (that
initially would have been overkill and more trouble to use than it was worth).

 _9\. Following the Status Quo_

Or in other words, "don't fix what ain't broken" isn't actually good? How is
this "over-engineering"?

 _10\. Bad Estimation_

How does this fit the theme? I thought the standard way to improve estimates
was to put _more_ thought and detail into them, which means the problem here
is actually _under_ -engineering (well, that and general noobishness).

.

.

Edit to add:

 _Important Note: Some points below like “Don’t abuse generics” are being
misunderstood as “Don’t use generics at all”, “Don’t create unnecessary
wrappers” as “Don’t create wrappers at all”, etc. I’m only discussing over-
engineering and not advocating cowboy coding._

So... if you disagree you're wrong and misunderstanding the article? If it's
that misunderstood, it's the article's fault for failing to communicate
effectively.

~~~
TeMPOraL
> _...I don 't get it? This sounds like it should be saying we try to "plan
> ahead" too much, but then the description seems to say we don't do enough._

My view may be biased by my experience, but I understand it that - no matter
how beautiful logical structure you invent that makes all the requirements fit
perfectly, the business will quickly come up with a new case that doesn't
really fit anything. Business requirements are unpredictable because _they
make no fucking sense_ \- they're combination of what managers think customers
need, what marketing thinks it needs, what future visions your boss has that
he didn't tell you (or that he doesn't himself even understand yet), all with
a sprinkle of peoples' moods and the subtle influence of phases of the Moon.

See also the motto of American Army - "If we don't know what we're doing, the
enemy certainly can't anticipate our future actions!" ;).

~~~
tbrownaw
This is only the case if the business is incompetent and insane, and also not
communicating effectively with development.

Or maybe things can _look_ that way, if the devs don't understand the
business.

The business is doing something to make money. The requirements will somehow
work to support that something, or will be something that your business
contact _thinks_ will support that something. If they don't make sense, that
means your mental model of the business is different than your business
contact's mental model. And unless your company is unbelievably dysfunctional,
that's an opportunity to talk and reconcile those models.

------
pastaking
Finally, some real talk!

~~~
edejong
Sorry, please comment on the article. No 'me too!' comments. If you like the
article, there is the up-vote button.

------
haalcion3
This is a fairly well-written post with some good ideas and some
generalizations that are going to get somebody in trouble if they follow all
of them, e.g.

> TL;DR — Duplication is better than the wrong abstraction

Woah, horsey, hold on a moment!

While it's true that an abstraction can get you into trouble, that's not
always true.

Over my many years I've heard a number of people say: "We have 100 copies of
the same site, but slightly altered for each client, and we don't have time to
go back and refactor them, we can't upgrade them, and we're three major
versions behind. Want to come work for us? (Silence.)"

I've only heard one person say, "We refactored X and it bit us in the ass,
because the developer didn't check when he/she was altering it and
accidentally changed behavior for everything."

~~~
firasd
The trade-off isn't about the dangers of not refactoring v.s. refactoring. The
trade-off is about time plus the aspect this article hammers, which is lack of
knowledge about the future--i.e. if you made a multi-purpose multi-client
framework to start with you might still be building things and rewriting them
to fit them into even more unforeseen situations instead of having live code
running for 100 clients.

~~~
haalcion3
Two things:

* It's vs. not v.s.

* If you want to copy a site 100 times, go for it. But I know very few that ever thought that was a great idea, and each time it was a very specific case. Yes there are client-specific features and multi-tenant sites, but that's not what he said. He said "copy vs. abstraction" which is the opposite of refactoring.

As an aside, I'm really having trouble understanding how people in the HN
community could be thinking I'm wrong on this.

I think I need to go to a forum that's more grown-up if this is how things are
here now.

------
0xAA55
This only really applies to web-developers.... but I guess that is what is
meant by "modern software"

~~~
DougWebb
I'm not sure about it applying only to web-developers. It might be more
focused on application-developers, where business requirements hit the UI.
Developers working on services, libraries, frameworks, and platforms have
different engineering needs where some of these guidelines don't apply (and
others might).

------
cheez
This is great. TL;DR of the whole thing? Don't do (almost) anything
unnecessary ahead of time.

I have a small bone to pick with 4) Shallow wrappers. My design process
involves ignoring existing solutions for problems and then finding things to
match my desired design. Often, this requires absolutely no wrappers,
sometimes it requires shallow wrappers, sometimes more involved wrappers,
sometimes implementing it all myself.

I do agree that you should not blindly wrap anything.

All in all, a great article. I think this should be required reading for
aspiring designers/architects.

