
I bet you over-engineered your startup - hamax
http://www.zemanta.com/fruitblog/i-bet-you-over-engineered-your-startup/
======
debacle
Of the three startups I've worked with, two of the three were ridiculously
over-engineered monstrosities that were way over time budget. It was clear
that the CIO/CTO wanted to do cool fun stuff and not build a marketable
product.

The other was cobbled together with completely shit code, was constantly
breaking on releases, and was glued together with perl scripts. They're now
publicly traded.

~~~
lifeisstillgood
Can you name names? Specifically the publically traded one ?

~~~
55pts
Here's my guess: Synacor. Took me only a few minutes of googling.

~~~
danneu
Can we refrain from punishing people for sharing their insights, especially
when they explicitly opt to not share those details?

~~~
55pts
You're right. I tried to remove it, but I can't edit the comment anymore.

------
xd
From my own experience, developers, particularly inexperienced developers,
when learning something new have an insatiable need to implement their new
found knowledge .. no matter if it's a good fit or not for the problem at hand
(myself included .. I remember abusing the hell out of recursion for example).

One of the worst examples I've seen is the functional programming paradigm
being crowbarred out of PHP .. why.

~~~
mootothemax
_One of the worst examples I've seen is the functional programming paradigm
being crowbarred out of PHP .. why._

Most likely related to the Innter-platform effect: _...the tendency of
software architects to create a system so customizable as to become a replica,
and often a poor replica, of the software development platform they are
using._

<http://en.wikipedia.org/wiki/Inner-platform_effect>

One of the classic PHP examples being the Smarty templating system:

 _Smarty has been criticized for replicating features that PHP does natively
very well, which leads to inefficiency_ :

<http://en.wikipedia.org/wiki/Smarty>

~~~
anthonyb
Hey bill - looks like you've been hellbanned. Something that you said on
<http://news.ycombinator.com/item?id=4213223> ticked someone off. Maybe talk
to info@ycombinator.com and get it sorted out.

PHP still sucks though ;)

~~~
RegEx
Wow, I've seen much worse snark on HN get all the upvotes.

~~~
anthonyb
Yep. Evaporation in action... :(

------
programminggeek
Many devs tend to over-engineer things whether they are at a startup or not
simply because they like building things, that's why they are software
developers. I don't think this phenomenon is much more complicated than that.

~~~
eragnew
'Over-engineering' is a term I'm starting to pay a lot more attention to :)

~~~
dj_axl
'Over-architected' and 'over-optimized' are the terms I prefer. Whatever
term(s) is(are) used, criticism should emphasize 'non-pragmatically-built',
instead of emphasizing 'startups don't need any design, engineering, or
optimization, whatsoever'.

~~~
eragnew
> _over-optimized_

That does seem like a better term for it. Or at least an equal term for it :)

------
tedsuo
Sorry, but I think this is really bad advice, it goes about it from the wrong
direction. Saying "monolithic" or "services" like one is good and one is bad,
or one is complicated and one is simple, is kind of silly.

For example... which is simpler, writing your own search indexing tool in ruby
on rails, or installing solr as a service? MySQL is also service, for some
reason people tend to forget that. Conversely, if your processes aren't yet
resource hogs, why not just let them remain general purpose workers? If you
are constantly fiddling with multiple services to make any changes to your
app, then yes, you have probably made a bad choice somewhere. But a
HAproxy/nginx/rails/memcache/mysql/solr stack is already six services, and not
really so complicated to work with. When you write your own services, you
should aspire to that level of simplicity.

At the end of the day, the shortest path will be wherever it will be. It's
your job as a developer to weigh the pro's and cons on a case by case basis.
The hard part is to test drive everything so that you can change it later, and
constantly evaluate what choices each decision you make is removing from the
table (painting yourself into a corner if you are not careful).

Another way of putting it: if you are picking your architecture before you
begin, based on some kind of generalized principle, you are already over-
engineering.

------
nemesisj
This is a really weird article.

The solution is not to say "Services Suck" or "Monolithic yay!", it's to
realise that you should start with a monolithic app built in a good framework,
then see how your app's internals are used/accessed as you grow THEN split out
to logical well designed services.

Or in other words: Premature optimisation is the root of all evil.

~~~
swah
More or less what the article said:

> Start merging services until you come to the bare essentials that have to be
> discrete.

He's giving an heuristic for finding the right granularity of services - that
is what is up for criticism, IIUC.

I totally agree with you, start "monolithic", SQL + a lightweight Python web
framework for most simple apps.

~~~
z92
> Start merging services until you come to the bare essentials that have to be
> discrete.

I thing GP advised to: start with a monolithic system and then split up the
services as your system grows and as you feel the need.

That sounds more solid advice.

------
codebeaker
After reading the "What Happens" section of the OP's article I can see that
he's made the classical mistake of making many things do one small thing, but
they're not independent.

The message queue to notify component X of changes to data in Y is endemic of
badly designed systems; if system X cares about changes to data in Y it should
be designed that (at scale) it caches the data for a suitably short time,
otherwise reads-through to the canonical source.

This is a common anti-pattern, and I've seen it built by smart teams at epic
scale (millions of uniques per day) and it is still un-manageable.

Feature toggles, hard and soft-failing, together with a baked-in assumption
that APIs are asynchronous (that is, unreliable) at as many levels as is
feasible is a good architectural move. (And does not necessitate an abundance
of architecture over feature code)

Loosely coupled components that expect their counterparts to respond slowly,
or not at all are easy to implement and even easier to test. (HTTP, if one
wants to use HTTP as the transport medium, learned this, and offers 201 and
202 for CREATED and ACCEPTED).

In my own projects (and I work mostly on near-realtime billing APIs) we bake
this assumption (and others) into every transaction, as try to be restful, and
transmit the state, and a URL which can be used to get the canonical
representation of any resource, at any given moment, and objects in all parts
of the system are stateful, relying on the handshakes (accepted/created) (404,
406, 409) to avoid race conditions and to make sure our systems can handle
downtime of any component (internal or external)

As a result, we have lightning quick tests, we are very confident in the
system's ability to perform, and we have read-through caches which respect the
transport medium's headers.

I suspect the OP is right, many do over-engineer the startup, but remember
many startups appear to have an abundance of developer potential, until they
don't. (Usually through bad design, not over engineering, the two ought not to
be confused)

Perhaps I am biased by having seen it happening in a large company, completely
unscalable architecture, and at a point as many "architects" as they had
developers, desperately trying to keep the wheels turning.

~~~
aaronblohowiak
You sound like you'd be interested in CQRS.

------
efsavage
Scalability is about more than operations/second or any single metric, it's
about making a service/system that can survive at scale, both technically and
logistically. A server that goes 10x faster than it's competitor but cannot
work with other servers to go to 100x is not a scalable solution, whether that
reason is because of some low level networking limitation or a high level
programming or administration limitation.

More semantics, but over-engineering is about more than making things
complicated. If a tool doesn't do what it was intended to do, or if you've
ignored the proper level of complexity, it's because it was _badly_
engineered, not over-engineered. _Good_ engineering will "over-engineer"
things as much as possible within the constraints of the solution.

And if you keep that in mind, then you really can't over-engineer things or
make them too scalable.

~~~
jacques_chester
You're sorta kinda conflating two distinct axes of scalability: vertical and
horizontal.

A system which is vertically scalable is one which will run faster (for a
given definition of "faster") on chunkier hardware.

A system which is horizontally scalable in theory becomes faster by adding
more independent hardware. In this age of spinning up anonymous VPSes by the
handful that's an attractive quality.

However, horizontal scalability levies a very heavy architectural tax. Nobody
has produced a convincing platform that successfully abstracts away the many,
_many_ moving parts and oversight that horizontal scaling requires in the same
way that an operating system can abstract away a lot of the complexities of
vertical scaling.

So what happens is that you spend less time thinking about the _problem
domain_ and more thinking about the _solution domain_.

And which one does the user care about, again?

~~~
efsavage
My point was not clearly made.

I'm not talking about vertical or horizontal, I would put them both in the
"technical" category of scaling. I meant to say that scalability is not
something that can simply grow to meet a certain load, it is a concept that
your system "can survive at scale" in whatever form that scale takes. It
encompasses many things like having more servers, bigger servers, more people
working on the servers, more users, longer sessions, more activity per user,
supports more features, and so on.

It's a semantic point, but it seemed like the author had a very narrow idea of
the terms he was using, and was complaining more about his own definitions
than the concepts themselves.

And the user cares about all of them, indirectly. They all matter because
failing at any one of them are reasons to use something else, whether it's
simply slow performance or because your high performance system is so brittle
you can't evolve, or downtimes because your high performance, quickly evolving
system requires more admins than you can afford, and so on.

~~~
jacques_chester
All well and good.

But you can go a long way -- a _very_ long way these days -- with vertical
scaling alone. As Stack Overflow have pretty convincingly demonstrated.

------
markokrajnc
You can do it Microsoft way: build Windows as a GUI for DOS, capture many
users, earn a lot of money, then pay best developers to develop Windows NT and
merge it into a new OS. (user experience first, architecture later).

But you can do it the Apple way: make it good from inside out - including
architecture, wait loooooooong looooooooong until users recognize all this,
then get maaaany users and earn lot of money. (Hopefully you have survived
until then.)

If you have luck, you can have both: good architecture inside and very good
user experience...

But anyway I agree with the author, that many many solutions are over-
engineered instead of just simple...

~~~
mjwalshe
what was good about OS9 :-)OK it gave Amiga users a good laugh back in the
day.

NT was developed by ex VAX Guys from DEC

~~~
cygwin98
I believe you was talking about OS X.

~~~
mjwalshe
Nope the poster was comparing Apple Developing Macs with Ms developing NT when
MS started with NT Apple was on OS9 and stayed that way for years.

~~~
batista
> _Nope the poster was comparing Apple Developing Macs with Ms developing NT
> when MS started with NT Apple was on OS9 and stayed that way for years._

Nope, the poster was comparing MS from _one era_ (when they developed NT) to
Apple from _another era_ (when they developed OS X).

That is, contrasting those two transitions as different ways of building
something.

~~~
mjwalshe
oh so comapring two totaly diferent era's then?

~~~
batista
Yes, the comparison is not about some era (what each company did at the same
specified time) but about the methodology (how each company handled a specific
transition to a different OS).

------
btilly
Here is my simpler version: function calls are the fastest RPCs. They are
faster to run, faster to write, and faster to debug if something goes wrong.

By all means conceive of your application as a bunch of well-engineered little
pieces. But for now, write them all as modules that you're making method calls
to. Should you some day want to, you're free to replace one by a facade around
a remote service. But for now it works and you can move on.

~~~
nix
_you're free to replace one by a facade around a remote service._

RPC rarely lives up to this promise. Starting simple is certainly a good
approach, but don't fool yourself about how hard it's going to be when you
need to deal with latency and partial failure later.

~~~
btilly
Trust me, I'm fully aware of all of the trouble with getting RPCs to work
properly. I still think that it is easier to do that work after you have the
original non-rpc version as a reference.

~~~
aaronblohowiak
Yes, but making those module boundaries loosely coupled and sharing
information only through their interfaces will go a LONG way towards enabling
you to remote the call at a later date. The biggest mistakes that prevent
remoting a module have to do with requiring context that is not part of the
interface of the method call (or having a very chatty back-and-forth, but that
is less common.)

------
shreyas056
And all that matters is the end user experience. Users don't care how it works
as long as it works(according to their expectations).

~~~
icebraining
I'm sorry, but that's a platitude.

~~~
ams6110
It's also true.

------
andraz
My biggest grief is that most of the engineering meetings consist of
discussions of how to get a piece of data from one system to another.

Instead of working on things that matter to the user.

~~~
dspillett
If you could solve that problem (hint: it is _far_ from being only a technical
problem) in the general case, you could be a rich man!

Data always being somewhere else or where it is wanted but not in the right
structure to allow practical use, is something we are exposed to a lot by our
clients.

------
SeoxyS
This isn't so much the symptom of a distributed architecture as it is one of a
badly designed distributed architecture.

Distributed architectures are the only way to go once you reach a certain size
both in terms of scale and in terms of team size. You can certainly make do
without it (Wikipedia) but you'll have a much more robust product with it
(Netflix).

The trick is always using the design appropriate for the current needs. It's
good to think ahead, but it mustn't come at the expense of the present.

At the beginning—which is the case for most startups, since few make it to the
later stages—it's often a good idea to go with a monolithic codebase based on
a lean framework. As you grow, you're going to want to start adding components
like a message queue for async work, rethinking your data store for scale,
etc. As you grow even further, you're going to want to transition to a
distributed architecture. I don't know what comes next… I haven't gotten there
yet. But I'm sure as you grow even further, your needs are going to change yet
again.

------
calinet6
Joke's on you, I don't even _have_ a startup.

------
hessenwolf
Yes. We spent ages before product launch devicing a really solid service that
nobody wants, and now make a stream of money from a side-shoot of our main
product. Oh well.

------
gizzlon
I'm all for simplifying, but what exactly does he mean by "monolithic
architecture" ? Not even sure I get his overall point, the rant seems to go in
many different directions.

That phrase "monolithic architecture" makes me think of _one-huge-Java-
project_ and that's not exactly "simple" in my mind. Probably not what he
meant though..

Generally, I guess separating things is more work and can actually lead to
less flexibility when situations arise that you didn't plan for ( _"maybe
bloggers should be able to advertise too"_ ). OTOH, _not_ separating can lead
to entanglement, where you can't really change anything because other things
depend on this or obvious and subtle ways.

(edit: removed dumb example ;)

~~~
dools
I think this is basically the context for this article:
<https://speakerdeck.com/u/kennethreitz/p/flasky-goodness>

------
sandfox
This seems a bit straw man. "Distributed systems suck because Developer B has
to wait for Developer A to add stuff to System A so System B can have a new
feature". What you have there is a totally different problem that is unrelated
to being distributed or not. Replace the word System with Module and imagine
they are in the same codebase, still have the same problem. This post smacks,
distributed system are hard and gave us a new set of problems that seem hard,
runaway!

------
rexreed
Side note: I _love_ the fact that there is a COBOL manual in the top graphic
in this article talking about over-engineering a software business.

~~~
ams6110
You are saying COBOL software tends to be over-engineered? Compared to what?

~~~
rexreed
I'm just saying that I love the photo ;)

------
creamyhorror
Some counterpoints and cases, anyone? I'm sure many of us would like to hear
them. (I'm not at all qualified, just starting out myself.)

~~~
michaelt
The article says you should merge all your databases into one, to avoid
setting up a service-api-notification-message-queue mess.

While sharing a database lets you develop the initial system quickly, you'll
have problems later on because you've made no distinction between your
interface (which other people code against and you commit to not changing too
often) and your internals (which you may want to refactor from time to time).

So either you make schema changes at will - in which case other developers do
too, and you're spending all your time fixing that instead of developing new
stuff - or you rarely make schema changes and do it with advanced warning and
approval, in which case the pace of development slows to a crawl because other
people are too busy to support the changes you want to make.

With well defined interfaces, only the interfaces have to evolve at a snail's
pace; the internals can change as fast as you like, as long as you do it
without breaking your interfaces.

Example: If you run a amazon style computerized warehouse and an amazon style
shopping website; you want to know if an item is in stock. If the website just
goes directly to the warehouse's database tables, the warehouse schema can't
change without worrying about breaking the website. A nice simple how-many-in-
stock web service would be a lot easier to maintain.

~~~
andraz
> A nice simple how-many-in-stock web service would be a lot easier to
> maintain.

... yes, _after_ you are doing millions of transactions per month. But until
you've actually built a business, there's no point in setting up a web
service.

If there's a web service, there has to be a team to maintain it. This means
you need to be big enough to have one team per web service.

~~~
michaelt
Absolutely - I'm sure you've heard of the idea of technical debt [1,2] and
sometimes it makes sense to build up technical debt to get a system off the
ground, then worry about maintainability, documentation and whatnot in your
copious free time later on.

[1] [http://www.codinghorror.com/blog/2009/02/paying-down-your-
te...](http://www.codinghorror.com/blog/2009/02/paying-down-your-technical-
debt.html) [2] <http://en.wikipedia.org/wiki/Technical_debt>

------
sneak
Agree, but trotting out Google and Twitter's most painful and unattended-to
shortcomings, while touchstones, does not support your original thesis.

Furthermore, Google and Twitter aren't startups.

~~~
Swizec
Furthermore read onward for some examples from Zemanta itself, which afaik is
a startup.

I could also get into the experience I had when using Google App Engine for my
startup a few years ago. Horribly over-engineered architecture, nothing
worked, had a whole bunch of trouble and everything went all manner of bad to
worse very quickly.

But hey, we had awesome scalability! Until we got 200 users and everything
started falling apart because of the overhead of keeping all the different
parts of the system communicating.

PS: the "saving is taking too long" example is actually aimed at Buffer not
Twitter or Google :)

------
jrs235
Isn't a philosophy like 37signals the answer to this? Simplify the software,
reduce the number of "features", resist adding and adding and adding to the
application?

Simplifying the software will allow you to simplify the architecture, no?

The stand still and dilemma for Developer A and B stems from a failure to plan
and having a road map. I, of course, am assuming both devs work internally at
the company.

------
hippich
This is bad advice for people who care. Granted, for businessmen it is most
import to cash check, but even quick and dirty project slapped together will
require rewrite eventually if it take off. And it most likely will be painful
and have all kind of subtle bugs.

~~~
z92
> will require rewrite eventually if it take off

The general advice is that once it takes off, you will have the money and can
hire the resources to rewrite it.

And if it doesn't take off, which is most frequently the case, you lose less.

------
eragnew
Exactly what I needed to read this morning. Thank you.

------
PaulHoule
cogs bad...

------
mjwalshe
Thats not over engineering just crap design and using the latest toys for the
sake of it.

Overengineering is the Victorian egineers way of desiging as they did not have
the indepth knowledge we have and so have massive margins of safety.

Of course that means that some of the pre WW2 London Underground tains lasted
longer than the ones bought in the 70's.

