
Ask HN: Are engineers biased to over-engineer solutions? - afco
I just read the (fantastic) article detailing the history of Twitter&#x27;s infrastructure at scale.
https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13441932<p>And though I appreciated it, I&#x27;m left wondering whether they really needed to invest all that money and effort? Or were their engineers unconsciously padding their resumes on someone else&#x27;s dime? After all its difficult to gain such experience or knowledge otherwise.<p>Maybe in Twitter&#x27;s case it made sense to exert that much control over their service. Especially at the time they were founded.<p>But in today&#x27;s &quot;cloud first&quot; world, it doesn&#x27;t seem to make sense for any company to roll their own infrastructure [1]. To me it feels like they may have over-engineered their solution. It would be good to know the business motives behind the decision and for the continued use.<p>I&#x27;m afraid as engineers we&#x27;re biased to &quot;roll our own&quot; solutions, if not for the experience then out of a hubristic thought that what we make will be better than whats existing.<p>Do you find engineers and tech companies in general (especially in the Bay Area) tend toward this?<p>Should we stop it? and if so, how?<p>[1] I don&#x27;t mean to focus exclusively on infrastructure or Twitter. I see the same with Uber or any number of tech companies who feel they need to roll their own services like the Big Four.
======
lcw
I'm going to over simplify this, but I really have noticed two types of
developers that I have worked with and/or managed. There are engineers who
favor the science, engineering and technology aspects of what we do. They tend
to love learning new algorithms, building libraries that do complex things or
just simply trying a new programming language as opposed to building a video
game or the next popular app.

Then there are the developers that see the code and job more as a tool to
build something and enjoy the process of completing a project more then the
technologies used to do it. They tend to build video games or fun apps.

Both obviously like technology, and have to dabble in both sides of this
simplified equation, but they favor one over the other.

I find that the engineers that like the science, engineering, and tech side of
programming are the people predisposed to over engineering. On the other hand
I find the people who enjoy finishing projects predisposed to not making
something extensible enough.

I really think there is a good mix of both of these types of programmers in
every dev environment of any size.

~~~
xsmasher
That "extensible enough" is a good phrase to remember; part of the job is
figuring our what parts of the problem are likely to change in the future, and
what parts are likely to stay the same, and exposing them accordingly.

This can depend highly on the problem space; as a game developer, I can
consider "number of seconds in a day" to be a fixed value that will never
change never. In other fields, that assumption could get your fired.

~~~
ViliusVilius
>"number of seconds in a day"

That's where you're wrong, kiddo.

------
jjirsa
> it doesn't seem to make sense for any company to roll their own
> infrastructure

Says someone who hasn't paid AWS a million a month before.

You roll your own infrastructure not to pad your resume, but because the capex
spend provides you with a lower opex, and presumably you have capital to
spend.

Cloud charges a premium for elasticity. Not everyone needs elasticity.
Capacity planning is a skill, you can get better at it over time, and other
technologies (VMs, containers, whatever) can help bin pack for better
operating cost than you'll ever see in the cloud.

------
xsmasher
One sign of maturity in a developer is a willingness to ignore technically
interesting challenges in favor of solving less glamorous, but more important,
problems.

"Important" problems being the ones with the most impact on the users and the
highest ROI for developer time.

Often a kludge (that may stand for years) is better that finding the "real"
and/or "best" solution to a problem. In short - real artists ship.

------
sokoloff
I know for a fact that RDD (Resume Driven Development) happens. I've seen it
firsthand. In the first few years of my career, I'm sure I did it first-
person.

What's not clear is how much of it is pure RDD out of selfishness vs how much
is BDD (Blog Driven Development) where a new technique or technology is
adopted because someone influential wrote a blog post about it vs how much is
simply the someone else with talent has tastes different than my own and while
I've have built it in technology X, they chose to build it in hot new
technology Y, but all I can be sure of is that I could have made it work in X
not that Y is a poor choice necessarily.

As I've gotten more experienced, I'm much more tolerant of differences,
provided that the company mechanisms are robust enough to keep everyone
aligned with the customer outcome. Undirected wandering is bad; meaningless
differences in tech choices are null.

------
nrjdhsbsid
The only company's architecture I've respected consistently over the years is
google. For all the interesting stuff they do their work seems very practical
and down to earth.

Simple choice of languages. Really solid core libraries. Emphasis on speed and
simplicity for the most part.

The simplicity of their search page is a prime example of the development
model they respect.

------
IndianAstronaut
As a data warehouse engineer, the one area I see the most over engineering is
in analytics platforms. Data layers on top of data layers, complex ETL
processes, complex front end code written in the hippest new JS library, etc.

Really what you need are a few talented analysts, and tools like R and Shiny
to quickly whip up analysis and dashboards.

------
wikwocket
I think many of us do enjoy complexity (when it works), and have a bias for
complex systems that seem cool. After all, one of the reasons many of us are
in engineering is that we aren't afraid of complexity, and indeed have an
aptitude for it.

I'd like to say more about how we can recognize and manage this bias, but
first I want to finish my general purpose AI-driven comment reply engine. I
just need to finish migrating the front-end from Angular 2 to React and the
backend from Ruby to Go, and then it should be pretty streamlined.

------
gozur88
>But in today's "cloud first" world, it doesn't seem to make sense for any
company to roll their own infrastructure [1]. To me it feels like they may
have over-engineered their solution.

It's possible you're just incorrect. Hard to tell from the outside.

And it can very much make sense to roll your own services, for a variety of
reasons from service guarantees to cost.

------
sokoloff
Having had a senior tech position at a large/medium e-commerce company for 13+
years, I can sympathize with the "that tech that I'd pick today wasn't
available when we had to pick."

I also recognize that the Big Four have scale problems that are larger than
the ones my teams have faced and tackled and so Google may very well need
BigTable, Facebook may need HipHop, etc.

~~~
tracker1
I have to agree with the assessment that a lot of tech may not exist when
someone starts out. Also, it's not always possible to _know_ you'll need
something until you need it. BigTable/Cassandra and the like are definitely
geared towards when you need more scale than smaller/singular systems can keep
up with. The vast majority of applications don't need it, and would probably
be better served with something simple to develop against or easier to manage
from an IT perspective.

On the other side, sometimes people spend too much time trying to integrate
certain frameworks into a codebase when a much simpler design will do. That,
or they will embrace a lot of one-off technologies, and wind up with
potentially more environment diversity than is probably reasonable. Oh, that
process runs in AWS, this other one is internal, and this one over here is on
Azure... since we have azure credits to burn. Not that any given cloud is bad,
but trying to be too diverse in a smaller team environment will lead to
fragmentation and a lot of split knowledge that doesn't replicate easily.

It's worth noting that I've been guilty of a lot of the things I mention. It
comes from experience. Sometimes much simpler is so much better.

------
dicroce
Sometimes you build your own X not because you can build a better X in
general, but because you can build a better X for you specifically. Whether
you should build your own X is largely a function of how core X is to your
business. If X isn't that important, use off the shelf. If X is important,
build it.

------
aaronhoffman
Fwiw, I tend to see little discussion of ROI, or weighing opportunities and
risks, mostly just bikeshedding
[https://en.m.wiktionary.org/wiki/bikeshedding](https://en.m.wiktionary.org/wiki/bikeshedding)

------
tabeth
I've been wondering this myself.

Personally I want to have a backend and then a SPA for each app. On one hand
that sounds like overengineering, you could just serve everything from the
backend. On the other hand, separation of the API from the services could
actually be a simplification.

------
mohamedattahri
At the risk of stating the obvious, I believe engineers are first and foremost
biased by the incentives put in place by their organizations. These incentives
are often non-financial, and it's precisely why culture matters.

------
buzzybee
It's a problem with a fractal shape. It's not really specific to microservices
or SV big tech. To elaborate:

1\. An individual software developer might bias towards feature delivery,
performance optimizations, extension hooks, tests and static analysis, or some
mix of those. A more experienced developer, all things equal, is more likely
to strike an appropriate balance for the problem they're working on. A wrong
balance might be overengineered because it took too long for the results. A
wrong balance that is underengineered, on the other hand, might fail to
deliver altogether by making critical-failure assumptions that go untested
until it's too late.

2\. A team of developers holds all the same biases as the individuals, only
now they have to collaborate. This produces a team-internal political
structure that enacts some compromise depending on management's goals. The
compromise may not be good engineering, and it is likely to overdo things in
at least some aspect. Managers often seek solutions that tame development and
allow it to proceed at some predictable rate of change, or to report a metric
of progress with person-hour linear scaling properties.

3\. Multiple teams of developers across a larger organization also have to
engage in political compromise, and at this scale, they are beholden to build
and share substantial infrastructure, because management tends to act
competitively, and infrastructure projects are politically useful.

4\. Across different organizations, shared formats and common data interchange
is desirable, and the chiefs of each organization have to satisfy their
internal political structure while simultaneously coming to an agreement with
the relevant standards groups. This causes the standards to bias towards a
kitchen-sink featureset.

These various factors all contribute to the "enterprise" mindset, and unless
you have a visionary of the Steve Jobs type at the helm, who will happily come
down from the heavens to throw lightning bolts at hapless managers and force
them into compromise for the good of the company, it's more likely to appear
as your org gets bigger: In the default case, nobody is incentivized to take a
risk on simplification or bold, radical changes. Incremental accretion of code
looks good for everyone, and allows sub-par developers to blend into the
background. The middle management fights localized battles over headcount and
budget, and flashy projects that covertly create large engineering
dependencies give them political leverage. When a new standards proposal comes
up, everyone rushes to get their fingers in it. So the systems that get built,
in the end, tend to express everything while doing almost nothing. Every one
of these companies has layers of crushed dreams sitting in their repository.

Because Twitter is funded at the assumed ambition level of "become Google" or
"become Facebook" their organization was staffed up in anticipation of
achieving that scale. That inherently creates the kind of conflicts resulting
in enterprisey code. The code reflects the organization, and the organization
is big, so the code is big.

At the other end of the scale, where individual developers do whatever they
want as they think of it, without business interest, none of these scale
concerns arise. The code can be direct and to the point. Where it has to solve
larger problems, it necessarily leverages the open-source ecosystem, creating
a "bottom-up" pull where the developer is more likely to submit to the
restrictions of the ecosystem and in turn extend that ecosystem outwards, than
to attempt a bold infrastructure project; as such, off-the-shelf options,
including cloud deployments, become more and more attractive as your
organization gets smaller, and for solo developers, they approach "no-
brainer". But, being based on self-absorbed whims, the code is much less
likely to be of value to anyone else - it doesn't answer to clients, or future
maintainers, or possibly even the original developer. It's less likely to have
coherent design or documentation, even if the developer aims for their best
try at quality, because there isn't any real conversation around it, any
probing of biases or testing against the real world. A solo coder is most
likely to be guilty of coding up a fantasy, which can be good if they happen
to be making a game, but is less viable for a lot of other lines of software.

So the best results tend to have a pattern of: Small, experienced teams - more
eyes on the design, appropriate challenge and dialogue about process and
features, but nobody getting lost in the crowd or submerged in political
maneuvers. Projects with some business interest, but relatively lower pressure
from the top, so that feature accretion and testing is guided by well-defined
business needs, not deadline pressure, bells-and-whistles, or whimsy.
Management that can define a healthy lifecycle for the software and allow it
some time to mature, but also be deprecated and incrementally replaced as the
surrounding software ecosystem changes.

When projects fall outside of those boundaries, bad software - of the over or
under-engineered sort, or the non-valuable sort, or the under-maintained sort
- tends to get made.

