
Invented here syndrome - Rexxar
http://mortoray.com/2015/02/25/invented-here-syndrome
======
jsankey
There's a lot of truth in this post but this line of thinking also plays into
our own prejudices. I do often look at third party code and think it's pretty
crappy, but then I need to remind myself that my own code is far from perfect!

More importantly, though, we tend to underestimate both how long it will take
to implement something ourselves and _especially_ how much it costs to
maintain over time. If you can use third party code from a popular and active
project you get a huge amount of testing, bug fixing and other maintenance
from the community over time. If you implement in house then you've just given
yourself an extra burden for the life of your project.

~~~
dreamfactory2
> If you implement in house then you've just given yourself an extra burden
> for the life of your project.

If you are doing this work for an organisation, you can also argue that it is
professionally irresponsible. You are certainly unlikely to be there forever
to pick up the tab for that. Coding in house is a toxic line of thought
endemic in mid-level developers who haven't yet had exposure to full lifecycle
costs of custom code.

(And ironically the problem of so many crappy libraries is precisely due to
people succumbing to NIH rather than uniting around a standard solution.)

~~~
nirvdrum
While I agree it'd be nice for their to be unification, rather than a
fragmented set of varying libraries, the one size fits all solution tends to
be a maintenance nightmare. I guess we just have wildly different experiences,
but mine is such that adding a third party dependency is typically a major
liability and often originates from less experienced developers.

Adding a dependency is a burden. You don't have direct control over the
evolution of that code. You have to keep track of the changes. If you're
lucky, the library will have a changelog, but that's sorely lacking in most
projects. Blindly upgrading will bite you, so if you have a product that can't
suffer downtime, you really need audit that code for every upgrade. Good luck
if you've determined that the version you have is sufficient, so you've locked
it down, but then a security issue is found . . . winding back patches like
that can be an exercise in futility. Oh, and then there's the wonderful world
of transitive dependencies.

NIH for everything isn't ideal, naturally. But pushing everything out to 3rd
party libs has its own set of problems that in many cases are worse. In my
experience, a more experienced developer knows when to make the trade-off and
solve the problem in-house with a much smaller body of code that's far easier
to maintain.

~~~
dreamfactory2
> a more experienced developer knows when to make the trade-off and solve the
> problem in-house with a much smaller body of code that's far easier to
> maintain

Indeed, but this is hopefully obvious to all of us. My point is that the cost
of maintenance is vastly higher than is ever really understood. Most code is
written in the context of delivering a project, and here the highest cost is
considered to be getting functional code out of the door (even the first set
of bugs is often overlooked).

I'd propose the counterintuitive and unpopular view that the cost of
development is the tiniest fraction of the lifecycle cost of lines of code
when you factor in things like bugs, keeping up with security, adapting to
environment changes, modifications due to new requirements, and probably most
importantly the multiplying effect of adding new moving parts which makes each
piece of code in a solution more expensive to write and maintain than the
last.

I'm perfectly aware that this will seem ridiculous and in some ways it is a
little aggressive to make the point, but as evidence I would consider how many
work years are spent on libraries which contain a tiny part of the
functionality that is contained in useful software.

Therefore the trade-off, while being about the right things, is very rarely
correctly sized.

------
dirtyaura
I don't know. Based on my experience I firmly believe that between not-
invented-here vs invented-here syndromes, many more projects fail because they
try to do too much themselves. Especially in startups. For anything that is
not the laser-focused core offering of your product, pick a good enough
library and be done with it.

Invariably, most problems are such that you can get 70% of the implementation
done (the fun part) with a reasonable amount of work, but the last 30% (all
the little details that you didn't thought of) take +70% of the time. And you
also need battle-test it. For example, React is interesting, because it is
battle-tested by Facebook, not just because of the approach it took to the
rendering.

Good candidates for underestimating the amount of work:

Anything with i18n and l10n (dates, money).

Anything with threads (futures and promises library).

Anything with small errors causing cumulatively bigger errors (scientific
calculation and money).

Anything integrating with the real-world outside of your app (sending email)

Anything with charts.

------
stormbrew
I think the real problem is actually the sunk cost fallacy, where often by the
time you discover the integration wasn't worth the effort put into it you feel
that you should just move on and leave it be. But in practice, something
that's a pain to integrate is usually also a pain to maintain that
integration.

In the end you can't really know if something is going to save time unless you
actually try it most of the time. But if you try it and it becomes clear it's
going to be a drain it might be time to throw it away and do it better.

Of course, there are a few vultures out there that hang on the entire industry
because the task they perform is so complex almost any attempt at reinventing
their wheel will fail in some non-obvious way. See for example ICU for unicode
text processing.

------
CGamesPlay
I've done this a few times in recent memory and on at least two of them it's
bitten me already. In one instance I wanted to use an external service to
handle email drip campaigns. It turns out that introducing an external service
added a lot of synchronization problems I was not prepared for, and I'm still
dealing with those today. In another instance, I wanted to use a workflow
management system. The DSL for job specification ended up causing more
headaches than it resolved and the tradeoff for the additional monitoring
capability it was adding wasn't worth it. I try to think harder about the
specific parts that I do want to write and which parts I will really gain from
by outsourcing.

~~~
sbov
Email, deliverability in particular, is a rabbit hole.

Its easy enough to get it set up and "basically" working, but not. Basically,
you're using in your favorite language's mail sending library to send an email
out.

The next level is figuring out all the standards surrounding email that those
third party systems know and implement for you. This will help deliverability,
to a point.

Beyond that, once you hit enough volume, there's best practices should have
always been doing but you learn from experience and need to abide by or else
you're not getting anything through even though you do all the previously
mentioned stuff.

If I were to develop an app with low amount of sent email, I would be okay
with doing the above myself. But I'm also familiar with the process by now.

Beyond that, theres continuous monitoring of deliverability and reputation,
tweaking your sending algorithms (you don't just send email as fast as you can
generate them, right?), keeping up with how email providers are changing their
rules over time, maintaining your relationship with people who work at major
email providers, and more I probably never had to get into.

You can easily have a team of people dedicated to ensuring your email doesn't
just end up in the spam folder, or being rejected outright.

~~~
lost-theory
Agreed. That's why it might make sense to fight NIH on the general problem of
sending emails, but to give into NIH when deciding to use an "external service
to handle email drip campaigns" vs. building it in-house. That type of problem
is much more specific to your needs and attempting to outsource it could cost
you more time and complexity than it saves.

Using an ESP for delivering email is a no brainer in most cases.

------
Bahamut
I've seen more junior developers fall prey to this because they were not
comfortable writing the functionality. A good engineer balances the two needs
and gets it done right. Sometimes that involves contributing to the third
party libraries themselves.

------
stewbrew
I'm not sure the authors understands what the NIH syndrome
([http://en.wikipedia.org/wiki/Not_invented_here](http://en.wikipedia.org/wiki/Not_invented_here))
is about. It is about writing everything yourself. IMHO the following
paragraphe is highly symptomatic for that:

    
    
        A great mass of libraries, modules, code, and other 
        software products, available on the net are either 
        crappy in their own right, or simply not suitable to our 
        project. Forcing things to work together can be costlier 
        than simply writing the required code on our own.
    

In the context of software development, NIH syndrome doesn't mean that you use
libraries not invented here but that you re-implement everything on your own.

~~~
wmt
I think the author understands the NIH syndrome just fine, and starts the
whole post describing the IH syndrome as the extreme opposite of NIH pretty
well, where one is ready bend over backwards to use a subset of a 3rd party
library, even if it might be easier and quicker to implement the desired
functionality yourself.

    
    
        Are you afraid to write code? Does the thought 
        linger in your brain that somewhere out there 
        somebody has already done this? Do you find yourself 
        trapped in an analysis cycle where nothing is getting   
        done? Is your product mutating to accommodate third 
        party components? If yes, then perhaps you are 
        suffering from invented-here syndrome.

~~~
stewbrew
Could be I got lost between thesis and anti-thesis.

------
pjungwir
I can relate to this. I researched a bunch of Ruby gems for "feature flags" a
while back and didn't find any that worked for our needs, so I wrote the code
myself, and it was super easy. On the other hand I'll choose Devise over DIY
authentication every time. . . . :-)

------
angersock
My general heuristic goes on the following questions:

-> _Is this project going to outlast my time on it?_

-> _Is the problem I 'm solving something that can be done in less than a couple pages of code with comments?_

-> _Is there a minimal open-source library out there that solves this problem for my language?_

-> _How many variants of this problem am I going to face?_

-> _Are other engineers than me going to be working on it concurrently?_

-> _If there is a library already out there, does it have good documentation?_

Those usually guide me pretty well, and in the event where I end up writing my
own code, I look to either contribute to an existing library (yay pull
requests) or to publish my library for other folks to use.

------
krylon
One advantage of rolling your own is that you only need to consider your (your
own, your employer's, ...) specific needs, and sometimes that added
flexibility is well worth the effort.

Especially once those needs change, you can simply change your solution rather
than beg some company or open source project to implement that change for you
or try to get some patch accepted or maintaining it yourself. _Sometimes_ that
is a big advantage.

On the other hand, if your hand-rolled library/application/service/whatever is
built by a single person and that person leaves the company, you might be
royally screwed. About a year ago I had the, um, honor of taking over
maintenance of a bespoke application developed in-house (by a single guy) to
make up for some shortcomings in our ERP system (which can, in theory, be
customized, but 1. it has to be customized in Visual Basic Script, and 2. the
vendor makes no guarantees whatsoever that your customizations will still work
after the next upgrade).

It has _mostly_ worked out OK, because the guy that left coded in a very
readable style, but when I deployed an updated version of the application, the
accounting department complained that things stopped working which _used_ to
work. But when I looked at the source code, they should not have worked at
all, because the code was not there. So probably the version the accountants
had been using before my update was not built from the version of the source
code I was given (not to mention that the code was not maintained using any
kind of versioning system).

TL;DR - in the case of small-ish applications (depending on how
unusual/special your needs are and what amount of resources you can afford to
invest) I believe there can be big advantages in rolling your own, but you
need to be aware of the risks. I've seen a few cases where companies had kept
using bespoke applications for ten years and longer, even though they were
clearly broken / out of sync with the company's needs, because nobody was left
to maintain them.

EDIT: Just to be clear, I am thinking more at the level of (small)
applications than libraries. With libraries, I prefer to use existing
solutions, for a lot of reasons, many of which other participants have already
pointed out.

------
chowells
I've seen this happen so many times. "We can't use X because it doesn't come
with library Y!" And then they point to their favorite alternative to X, and
note that it does have library Y, and ignore just how _utterly terrible_ that
library is.

Sometimes it's better to rewrite a library yourself than use an existing one
that gets in the way as often as it helps.

~~~
this_user
> Sometimes it's better to rewrite a library yourself than use an existing one
> that gets in the way as often as it helps.

Sometimes, but not most of the time. Most of the time your own library will be
just as terrible as the one you were trying to replace. This is especially
true when it is done by someone who is not good at designing APIs. In then end
you then get something that is barely documented, slow and full of bugs. I
would know, I have removed more than my fair share of crappy inhouse libraries
from projects.

------
IgorPartola
Or maybe different people have different strengths? Look, I like to tinker at
all levels of the stack, but I know what it takes to, say, write a correct web
server. I am not going to write my own unless I must (did this at least once).

Some people prefer to re-use existing code and are good at integrating
existing pieces. Others are good at writing purpose built stuff from scratch
quickly. And even that is spread across technologies. I can write an async
task manage, but can't easily create a CSS grid system. Just don't get stuck
in the "built here" or "not built here" camp and you'll be fine.

------
_rpd
Additional bonus for rolling your own: not having to deal with restrictive,
ambiguous or outright bizarre licensing terms and, in the case of SAAS, that
the service or its best API will become unavailable with minimal warning.

------
JamesBarney
I haven't yet worked in a shop where I thought they should have spent less
time buying products, and more time inventing things. Obviously there have
been some library choices that have been regrettable but there is no shop I've
been to that has a general cultural problem of not building enough things.
(Background enterprise)

But when I have seen someones make an error in judgement, I have noticed that
they realize it much more quickly when integrating software then when building
it themselves. I suspect because of the attachment that quickly grows for
one's own code.

------
dicroce
My company so prefers outside solutions that sometimes we engineers joke that
the best way to get your code used is to open source it and let management
think it's from outside the company!

------
untog
Some well structured projects actually _do_ allow you to use only part of the
library - in JS-land you can:

    
    
        require('library/just/this.js');
    

Of course, it takes effort to make that work, but I'd still try to do it,
because if something is confusing me about almost any library then the answer
is a quick Google search away. If I wrote it myself I'm stuck.

------
lisa_henderson
I briefly worked with a startup that has built their whole system using Python
(though they use things like ElasticSearch, so they also run the JVM). The
programmers at this startup are very intelligent, and very thoughtful. There
are 4 main programmers. Their background is a mix: some of them were initially
scientists, some were computer science engineers. They are all in their late
20s and early 30s. The startup began in 2011.

Because they are so thoughtful and talented, whenever they see a flaw in a 3rd
party library, they give up on that library, and they try to write their own.

So for instance:

They tried using Chef for configuration, but they didn't like Ruby. Then they
tried Ansible, but they found it immature. The tried Puppet, but didn't like
that it had a central daemon. So they started writing their build/config
system, which they released as open source. Their system has some unusual
ideas, a heavy reliance on bash, and an emphasis on supporting a polyglot
system. However, this took a lot of time that could have been directed at the
specific needs of the startup.

They decided they would need a system of dependency injection. In Python it is
common to configure some variables at the level of the module, but they felt
this lead to tight coupling. They did not like any existing system for
dependency injection for Python, so they wrote their own. They released this
as open source. Their DI system borrows a lot of ideas from Angular. It is
clever and they have long debates about how to make it better. But this also
took time that could have gone to the specific needs of the startup.

They needed a framework that would allow them to serve a website out over
Nginx. They reviewed all of the existing Python web/application servers. They
found flaws in all of them: Zope, Pylons, Flask, etc. They decided to build
their own. They based their work on Flask but they added a lot of their own
unique ideas. They released it as open source. It has been a big effort and it
is still missing certain features. For instance, they started off assuming
that all of the pages of their site would require a login. When they
eventually had to add in some pages that did not require a login, they had to
re-work the way they were handling security and authorization. All of this has
taken time away from the specific needs of the startup. And, of course, their
software may have security flaws which they have not yet discovered. Although
it is open source, I don't believe anyone else is using their software.

They decided they needed a way to enforce contracts on the structure of the
documents moving through their system, and they wanted a system that could
work almost exactly the same in Python and in Angular. So they built their own
library for validating the structure of documents. They built out a whole
vocabulary for the "types" in their system, using types which were generic
enough that they could work in both Python and Javascript. It is an
interesting system, but it takes at least 25% of the time of the most senior
engineer. If they did not write their own system, he would have at least an
extra 10 hours a week to put toward tasks that are specific to the needs of
the startup.

They wanted to store JSON but they did not like any of the JSON document
stores that were available in 2011/2012\. They love PostGres. This was before
PostGres had a JSON column type. So they decided they would store JSON in
PostGres as a string in a TEXT column. They wrote a library that serializes
and deserializes the string. They built this as a layer on top of SqlAlchemy.
Their code is very clever: you can query the database with their ORM and you
always get back objects, you never have to think about JSON as a string. In
some ways, they have reinvented the idea of an "object database" that was
popular in the late 1980s and early to mid 1990s. These are clever ideas,
though it took time away from the actual needs of the startup.

I could go on. They wrote their own cache system, their own system for
transforming data structures, etc.

They are thoughtful. When adding a new feature to one of their libraries, they
have respectful debates that sometimes last 2 or 3 or even 4 hours. They
release everything as open source. They are trying to help the Python
community. But it adds up to a huge tax on their time, and it takes them away
from the needs of the startup. The founder/CEO is not technical, and I think
they often wonder why things take so much time.

With startups, it is especially true that "the perfect is the enemy of the
good." Their perfectionist streak takes them away from the needs of the
company. They pursue an ideal of absolute brilliance, though they often don't
have the time they need to make their libraries as good as what they dream of.

Having worked with them, I've gained a new respect for the idea that startups
should, as much as possible, just use what libraries they can find. All
libraries have their quirks, but it is an illusion to think it is cost
effective to spend your time re-inventing what's out there.

~~~
Leszek
Do you have links for any of these released systems?

------
AnimalMuppet
Here's an analogy: McDonald's Big Macs. Is the code in question part of the
Special Sauce(TM), or is it just something like the buns? McDonald's makes the
Special Sauce, but I bet they just buy the buns.

If it's not part of your special sauce, if you can buy it, do so. Spend your
time on the special sauce.

