
Simple Systems Have Less Downtime - gk1
https://www.gkogan.co/blog/simple-systems/?r=0
======
papito
Instagram was what, 12 employees when they got sold for a gazillion dollars?
They all could fit into a van. Because they kept their system simple. It was
(and still is) a monolith. Now imagine that they decided to go the
microservices way. Multiply that team size by 10 at least.

Don't solve problems you don't have.

~~~
nickbauman
But microservices are an example of simpler systems. Each microservice does
far less than the whole monolith does. You can read all the code in ~15
minutes.

I've worked at companies that have monoliths that are 50x more difficult to
work on because of the size. Some of them millions of lines of code. Nobody
really knows how they work anymore.

~~~
sgift
> But microservices are an example of simpler systems. Each microservice does
> far less than the whole monolith does. You can read all the code in ~15
> minutes.

Microservice usually means distributed system. A distributed system is more
complex than a non-distributed system since it has to do everything the non-
distributed system has to do and, additionally, handle all the distributed
problems. Microservices just hide the complexity in places where people don't
see them if they take a cursory look over the code, e.g. what is a function
call in a monolith can be a call to a completely different machine in a
microservice architecture. Often they look the same on the outside, but behave
very differently.

The hierarchy of simplicity is: Monolith > multithreaded[1] monolith >
distributed system. If you can get away with a simpler one it will save you
from many headaches.

> I've worked at companies that have monoliths that are 50x more difficult to
> work on because of the size. Some of them millions of lines of code. Nobody
> really knows how they work anymore.

That is a bad architecture, not something inherent to a "monolith". There's
probably also a wording problem here. A monolith can be build out of many
components. Libraries were a thing long before microservices reared their ugly
heads. What you describe sounds more like a spaghetti architecture where all
millions of lines are in one big repository and every part can call every
other part. Unfortunately, microservices are not immune from this problem.

[1] or whatever you want to call "uses more than one core/cpu"

~~~
jayd16
>Microservice usually means distributed system. A distributed system is more
complex than a non-distributed system since it has to do everything the non-
distributed system has to do and, additionally, handle all the distributed
problems. Microservices just hide the complexity in places where people don't
see them if they take a cursory look over the code, e.g. what is a function
call in a monolith can be a call to a completely different machine in a
microservice architecture. Often they look the same on the outside, but behave
very differently.

This is a false assumption. Some problems are distributed. Sometimes you'll
have an external data store or you'll need to deal with distribution across
instances of the monolith. You really run into pain when you build a
distributed system in your single monolithic code base and your monolithic
abstractions start falling apart.

In my experience you end up solving these problems eventually, monolith or
not. You might as well embrace the idea that you're deploying multiple
services and some form of cross service communication. You don't need to go
crazy with it though.

~~~
marcosdumay
What assumption are you talking about?

Anyway, if your problem requires a distributed system, congratulations, you'll
have to go to the top of that complexity hierarchy, and will have to solve all
the problems that come with it.

That doesn't change anything about there being more problems. You just don't
have any other option.

------
rsync
I gave a talk on this subject at CU last year and have, one way or another,
spent my entire professional life thinking about this topic.

I agree wholeheartedly that simple systems have less downtime.

I would like to add a line from the talk that I give:

Simple systems fail in boring ways. Complex systems fail in fascinating,
unexpected ways.

rsync.net storage arrays _typically_ have multi-hundred day uptimes. But
across our entire network, our actual aggregate uptime is unimpressive -
perhaps 99.99% ? Our SLA dictates 99.95.

 _But when they do fail_ they fail in very, very boring ways. There are
usually zero decisions to make in response to a failure.

~~~
carterehsmith
"simple systems have less downtime"

That is not much of an insight. It is as insightful as saying "water is kinda
wet.". Well... sure it is.

What we need to deal with, is not "make a simplest system".

Rather, we need to deal with: "build a system that does A, B, C, ... and so
on". Now, if you can do all of the above and make it simple... awesome. But if
you cannot do all of the above, but the system is simple.... that is useless.

~~~
rsync
"Rather, we need to deal with: "build a system that does A, B, C, ... and so
on". Now, if you can do all of the above and make it simple... awesome."

I agree with this. It is a good point.

Some systems and protocols have very difficult requirements to meet and
different constituencies driving those requirements - it's not always possible
to implement simple and elegant solutions.

------
hyperion2010
The only language that I have worked with that realizes that in the real world
simple is _always_ a lie, is common lisp. It is the only language that has
actually embraced the fact that its designers/committee were not geniuses and
provided the tools for dealing with the complexity of the system. When unix
tools fail, hope that you are on a system where it is possible to get the
symbols and/or the source code, and even then good luck fixing things in situ.
Most existing systems do not empower the consumer of code to do anything if it
breaks. CL remains almost completely alone in this, because the debugger is
part of the standard. Most of my code is written in python, and I can tell you
for a fact that when python code really fails, e.g. in a context with
threading, you might as well burn the whole thing to the ground, it will take
days to resolve the issue, roll back the code and start over. The fact that
people accept this as a matter of course is pure insanity, or a sign that most
programmers are in an abusive relationship with their runtime environment.

~~~
jdc
Sure lisp is a wonderful language, but let's not pretend there's no difference
between incidental and accidental complexity.

~~~
thesuperbigfrog
Fred Brooks called it "essential complexity" and "accidental complexity"[1].

Essential complexity is inherent to problem being solved and nothing can
remove it.

Accidental complexity is introduced by programmers as they build solutions to
the problem.

Lisp is nice because eliminates a lot of the accidental complexity through
minimal syntax and lists as a near-universal data structure.

1: [http://worrydream.com/refs/Brooks-
NoSilverBullet.pdf](http://worrydream.com/refs/Brooks-NoSilverBullet.pdf)

~~~
jerf
One of the traditional criticisms of Lisp, though, is that it lets programmers
re-introduce a whole lot of accidental complexity in their Lisp code, and,
worse, everyone introduces a completely _different_ set of accidental
complexities into their code.

~~~
thesuperbigfrog
That problem is not limited to Lisp.

As a programming language, Common Lisp is large enough and multi-paradigm
enough to allow for elegant solutions to problems. It does require some
experience with the language and some wisdom and discipline to know what
pieces to select and how to best use them.

However, like all large, multi-paradigm programming languages that have been
around for a while (I'm looking at you C++), programmers tend to carve out
their own subsets of the language which are not always as well understood by
those who come after them, particularly as the language continues to evolve
and grow.

There is also the problem where programmers try to be too clever and push the
language to its limits or use too many language features when a simpler
solution would do. All too often we are the creators of our own problems by
over-thinking, over-designing, or misusing the tools at hand.

~~~
jcelerier
> programmers tend to carve out their own subsets of the language which are
> not always as well understood by those who come after them

if the language is not powerful enough to allow for that people will
inevitably add preprocessors, code generators, etc... to do the things they
want.

~~~
thesuperbigfrog
>> if the language is not powerful enough people will inevitably add
preprocessors, code generators, etc... to do the things they want.

This is definitely true and it adds to the accidental complexity of the
system, usually to save programmer time or implement layers of abstraction for
convenience.

Common Lisp and C++ have both incorporated preprocessors and code generators
through Common Lisp macros and C++ template metaprogramming and C++
preprocessor / macros. These features give the programmer metaprogramming
powers, enable domain-specific language creation, implement sophisticated
generics, etc.

They are powerful language facilities that need to be used wisely and
judiciously or they can add exponential complexity and make the system much
more difficult to understand, troubleshoot, and maintain.

------
throw0101a
See Gall's Law:

> _A complex system that works is invariably found to have evolved from a
> simple system that worked. A complex system designed from scratch never
> works and cannot be patched up to make it work. You have to start over with
> a working simple system.[9]_

* [https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law](https://en.wikipedia.org/wiki/John_Gall_\(author\)#Gall's_law)

~~~
klysm
I find it mildly annoying when people state such nuggets of knowledge as
“laws”. Sure, it’s generally true that you can’t go for the most sophisticated
and powerful system at the first go, but sometimes the minimal working system
is complex no? Anyway, it’s certainly not a “law” of the universe.

~~~
mdonahoe
Klysm's Law:

Nuggets of knowledge are not laws. Referring to them as such annoys Klysm.

~~~
Smoosh
An aphorism being called a law is called a klysmism.

------
carlmr
>For example, an analytics dashboard built with a no-code analytics tool like
Looker is likely to have more qualified people to fix it than one built with a
patchwork of custom scripts and APIs. Nobody should have to pull data
scientists or product developers away from their work to fix a bar chart.

I don't have experience with Looker, but the unseen complexity of no-code
tools often leads to very complex systems with interactions that are hard to
understand. A dashboard may not have that many interactions with other tools,
but the black box nature of these kinds of tools usually leads to significant
downtimes when there's a small detail not working as expected, or where you
need a small customization that wasn't foreseen by the no-code API creators.

~~~
buremba
That's true for drag&drop tools that provide a user interface as an
alternative but Looker or similar tools introduce their own language that
limits your capability by sandboxing the environment but still lets you
programmatically define your models. See LookML for Looker
([https://docs.looker.com/data-modeling/learning-
lookml/what-i...](https://docs.looker.com/data-modeling/learning-lookml/what-
is-lookml)) or Jsonnet for Rakam.
([https://docs.rakam.io/docs/compose](https://docs.rakam.io/docs/compose))

P.S: I'm affiliated with the company Rakam.

~~~
jbott
I think that’s missing the point he was making. No matter what front-end
representation of a system you get, the implementation backing a no-code
system will have edge cases around interactions both internally and with
third-party systems. The fact that you’re hiding a system behind a veneer of
“no code” doesn’t hide the fact that code is behind the actions that are
taking place, it just means you can’t introspect it (even if this “no code” is
in fact code).

I think a better point would be to make that that’s not a bad thing, sometimes
abstractions provided in “no code” systems can greatly simplify a solution.
But even the most well-designed solution will fail to some essential
complexity of the root-problem the original authors didn’t understand. Without
debugging tools (or privileged access to the backend of the implementing
system) it’s difficult or impossible to understand what went wrong.

------
pkolaczk
I like the ship analogy from the article. The fact it is so robust is not
because the system as a whole is simple, but because its components have well
defined, narrow responsibilities and there are abstract interfaces between
them, which hide a lot of complexity. You don't need to understand the
internals of the pump to explain what it does, even though the actual internal
technology may be quite complex and clever. Also you don't need to understand
the internals of the pump to design a rudder. This is a property that many IT
systems lack. Insufficient abstraction leads to a network of components where
you can't reason about anything without understanding all of it. And such
systems often break in weird ways.

~~~
notyourday
A startup is not a container ship going between port A and port B where
everything is known.

A startup is a new destroyer that has been floated, did not have sea trials
and went to war with a crew that might have a couple of people that used to a
drive container ship but mostly staffed with kids that thought it was cool to
play with a destroyer. Oh, and 3/4 of the systems are still at best have been
drawn on a napkin and most of the rest came from a salvage yard. Oh and if you
complete your next mission you may get money you can spend on some of the
systems but if you do get that money you would be expected to do a lot more
runs, quicker.

~~~
collyw
Your startup is probably just a variation of CRUD.

~~~
notyourday
The reason why we consider shipping to be a simple and solved problem is that
in 1956 McLean had a brilliant idea that to ship efficiently and well
everything imaginable would be put into identical containers with no
variations.

That's why it is possible to have a container ship with a crew of 5-6 people
pilot the load. Containers aren't sort of identical. They are completely
identical and completely standard ( several standard sizes ). They have the
weight distributed in a specific way and they are stacked on a ship in a
specific way depending on the weight of every container.

The startup equivalent of a container ship in simplicity is a startup sorting
a pile containing A4 paper, standard business envelopes and hang folders at a
daily rate of 500 items.

------
csomar
The author, probably, picked the worst example. Here is a pic:
[https://iro.nl/app/uploads/2018/12/P-67-onboard-the-BOKA-
Van...](https://iro.nl/app/uploads/2018/12/P-67-onboard-the-BOKA-
Vanguard-1000x750-c-default.jpg)

No, the containership is not a simple system. It's very sophisticated one. It
takes massive engineering effort (literally historical effort and knowledge)
to build, massive resources for the material and outsourcing, very complex
(like the pic) infrastructure in case of repair, satellites to guide the ship
and check the weather, operators from all around the world to track the
containers, massive ports to load-unload the shipments, etc...

The operation of guiding the containership through the sea is a simple one but
only once you have everything of the mentioned above in order.

If you are building software (a full SaaS solution), you are not in the
business of guiding the containership. You are in the business of building
either the containership alone; or the whole related infrastructure (or parts
of it). That's a huge undertaking that requires massive engineering efforts.
It's not a simple task.

If your task is a simple team that guide the container, you might make money
for a while until everyone else figures out that you have no edge there and
they can do your job for much less.

The Instagram/Tinder examples are a bad one. It ignores the fact that to find
the right (addictive/highly viral) app, you'll need (again) massive
psychological knowledge and expertise into how primates brains work. We don't
have that, so engineers are brute-forcing by making too many apps until one
hits the jackpot.

~~~
gk1
Hey, author here.

My point is that simple-to-understand systems--not simple as in primitive--
have less downtime, not that we shouldn't ever have complex systems. A
container ship like the one in my article can be drydocked ("in the shop") for
repairs and back on the water in less than two weeks. A nuclear-powered
aircraft carrier cannot. So be mindful when you're building an aircraft
carrier when a container ship would've sufficed.

By the way, most of the "complex" stuff in the photo--which isn't a container
ship--is scaffolding and piping.

Source: Got a naval architecture degree and a marine engineering license,
operated a steamship for six months at sea, and designed ships for the US Navy
for three years before abandoning ship to work with startups.

~~~
btilly
An interesting thing to contrast that with is the Boeing disaster. Where they
decided to make the steering actually more complex, then hide it from the
pilot with software to make it look like the old, simple thing.

The result was a system that worked fine until you encountered bugs in the
software, at which point there was no helping that several hundred people were
about to die.

------
emerongi
The author is not arguing for simplicity per se, but about being able to
insert a human into a system that is normally automated. The example about the
ship's steering system is perfect, actually. The system is not "simple" (if I
were on a ship and the steering failed, I would be clueless), but it provides
plenty of interjection points where a knowledgeable human can step in and
either debug or fix the issue. It's those points that should be relatively
simple - the whole system itself can be as complex as needed.

Your system should be testable, debuggable and it should allow a human to step
in and take control if needed. An open system is generally better than a
closed one.

~~~
tialaramex
Humans in the loop are likely to be a source of trouble, and you will need to
measure to figure out if they provide any overall benefit other than giving
you somebody to blame when that trouble happens. You need to ensure that the
"human in the loop" scenario is obviously worse than automation or else
humans, always over-confident of their abilities, will put themselves in the
loop when it'd be far safer not.

A good example of how to do this would be London's Docklands Light Railway. A
DLR train is capable of autonomously going from one station to another.
Trained operators are aboard every DLR train, and they have a key to a panel
in the front of the train which reveals simple controls for manual operation.
But very deliberately the maximum speed of the train with an operator at the
controls is significantly lower than maximum speed under automation (there are
two modes in fact, driving with the train still enforcing rules about where it
can safely go which is a bit slower, and driving entirely without automation
helping which is much slower). This underscores that manually driving the
train is the way to sidestep a temporary problem, not a good idea in itself.

The DLR has had several incidents in which trains collided, it will come as no
surprise that they involved manually controlled trains. Humans are very
flexible but mostly worse as part of your safety system, don't use humans if
you can help it.

~~~
tossing459
I think what he meants by human and users is "programmers" and layers of
abstractions in your code. The ability to change and go deeper when you need
it.

But you example works nicely as well, you should avoid having to work at
lowers abstractions, and limit the work there as well. "changing the core to
adapt to new features rather than adding to the core"

------
DecoPerson
I like to “embrace complexity.”

Any line of code, especially one that introduces a new concept, like a class
or a method, is in “addition” to the root problem we’re trying to solve.

The closer the code/data/org-processes/whatever matches the fundamental
problems we’re trying to solve, the less complexity we introduce.

However, you cannot “remove” complexity! A common mistake I see in software
companies is using a single Jira ticket for one or more reports, one or more
conclusions, and one or more action items.

For example, if we get reports A, B and C, they may all go into a single Jira.
Oh, but C is a totally different thing! It’s not a “duplicate”! Ok now someone
needs to extract the info from a comment into a new Jira (which no one will...
and it’d be a mess if they tried).

Now a bunch of discussions are happening in the comments of this mega-Jira. 3
conclusions come out of these discussions — perhaps what the cause is and what
should be done about it.

Then let’s say there’s two action items, one is a temp fix and one is a long
term fix. Unfortunately, I usually see people is the same Jira for both tasks
(“Assign this back to me when the temp fix is in so I can do the real fix”).

But splitting Jiras up is frustrating. The actual splitting is frustrating to
set up, and it’s a pain to browse them.

So most Jira workflows remove fundamental complexity (reports vs
discussions/conclusions vs tasks) and introduce extraneous complexity (tonnes
of fields and workflows that don’t necessarily apply).

One should “embrace” the fundamental complexity of the problem(s) they’re
dealing with.

~~~
jyounker
My favorite variants of this phenomena are "let's just add a column instead of
creating a new table" and "let's add a boolean argument instead of creating a
new function."

~~~
Minor49er
Or worse: "we built a table for arbitrary key-value pairings, but now the
value needs to be an array or object, so we'll store it as a JSON string"

~~~
ansgri
I usually do the inverse: who knows what additional data we'll want to store
for each row, let's put there a jsonb column, store anything not clearly
critical there, and add columns or tables as these jsons get populated and
used.

------
monus
Usually, being easy is confused with being simple. Easy to use systems usually
tend to be very complex but they'd only hide it from the average user. Once
you go one step ahead, do something outside of quick start, you find yourself
helpless. In that sense, the article is on point. Not only the management of
the ship is simple, machinery behind the scenes is simple as well.

This applies to programming languages, too. Python is easy but when you do
advanced stuff, often without realizing, you start to grok many complex
details to make it work or make it work in a performant way. On the other
hand, Go (and Rust supposedly, but I don't have experience) is simple. It
doesn't let you do many things and that results in fewer ways of doing a
specific task but when you do it, you feel safer because in the end, it does
look and feel simple although you spent 1+ hour than you'd normally spend
compared to coding in Python (this assumes you're doing some advanced stuff,
not quick start).

~~~
tsimionescu
Go is actually a pretty complex language, more so than Java for example, in my
opinion. It has all sorts of primitives that you need to get used to, special
rules for how the built-in types work, there are all sorts of rules to see
what creates a copy of a struct and what doesn't, pointers have at least three
completely different use-cases (optionality, mutability, and avoiding copies),
several rules for what can and can't be addressed etc. And this is all before
looking at the standard library.

~~~
monus
I think concrete things you listed like the case for pointers is not much
different than Java. While why you use a pointer changes, it's simply pointer
in the end, just like Java.

I'd say Go is able to put so much meaning to those fundamentals because 1.
they have to, there isn't many other mechanisms 2. they can because it's still
early in the evolution of the language. After some adoption and wildly
different use-cases that users wants to be addressed, those patterns start to
disappear and people end up with lowest common denominators, pointers being
just pointers in this case.

------
blacksqr
Every new programming style epiphany is an ad hoc, intuitively-specified, hazy
restatement of the Unix Way.

~~~
mhd
If Unix is your yardstick for simplicity, that just shows how far we've come…

~~~
ailideex
Linux is incredibly simple compared to Windows once you actually your computer
as a computer instead of as a really poorly made gaming console.

~~~
vntok
Try talking Grandma over the phone to help her share her photo directory with
Aunt Jeanine, on the same PC. No command line allowed, and you know she can
only focus on that type of task for 4, maybe 5 minutes top.

~~~
mhd
That wouldn't work reliably under Windows, either, as nobody knows anything
about files and directories generically anymore.

(That's the reason why first the Windows explorer was suddenly sufficent
enough and then almost completely disregarded)

A public folder on the same PC is way harder to explain than just telling your
photo app to share it with your resident spying company/cloud provider,
despite how wasteful the round trip is.

Not that that has anything to do with a discussion about the "unix way" as a
special case or even superset of IDEs.

------
obscura
There's way too much simplification and generalisation here, IMHO.

"Choose tools that are simple to operate over those that promise the most
features."

Doing this very much depends on your requirements, doesn't it? What good is a
simple-to-operate tool if it doesn't do what you need it to do? Sure, maybe
you can simplify your requirements; then again, maybe not.

The example of troubleshooting a whitepaper form very much depends on the
design of the system. Maybe there's a reason for having multiple forms. If
they all shared common architecture and the problem lies in what they share,
it's not necessarily more difficult to troubleshoot many as opposed to just
one.

Moving from Marketo to HubSpot is good and well - for now. What about years
from now? How did they end up with such complexity with the Marketo solution?
Both solutions and requirements evolve. Down the line, you could end up with
the same difficulties with HubSpot. I think it depends in part on how the
organisation handles change.

Lastly, I agree with the sentiment of Mateusz Górski, who commented on the
page: is there any hard proof beyond anecdote to back up the article? If you
expand the article's concept of a system to beyond just software, all you need
is a hardware fault in a third-party hosting company with lousy support for
your simple piece of software to be down for weeks.

~~~
couchand
> all you need is a hardware fault in a third-party hosting company with lousy
> support for your simple piece of software to be down for weeks.

I suspect the author would reply that you can just replace the hosting
provider. Since the interface and division of responsibilities is clear, one
host is completely replaceable with another.

------
downerending
As a sub-case of this, almost every HA (high availability) system I've ever
seen has been less reliable than the original system was (without HA). It
sounds like a good idea, but the extra complexity kills it.

One non-software system was a rather expensive UPS/generator. It was meant to
trip on a power loss and provide X minutes of stable power. In reality, it was
incredibly sensitive to minute power fluctuations and would trip and then
immediately fail, dropping all supplied power in an instant. The system was
dramatically more reliable with this unit simply disabled.

~~~
thijsvandien
At least for the relatively small businesses that I deal with, I basically
always recommend to save the effort that would go into building a 'highly
available' system, and instead spend it on a scenario for keeping the business
running when tech fails—which it will to some extent at some point, no matter
what. I suppose that line of thinking would do wonders for much larger
businesses as well. When a system becomes too important, rather than trying to
make it even more reliable, try making it less important first.

~~~
downerending
Indeed. It shocks me when I see boo-coo bucks being spent on all sorts of
bells and whistles, and then ask about their backup strategy. _Oh, all of our
drives are RAID..._

------
wonderwonder
I wrote my first deployed side project 5 years ago. It has been with the
customer for the whole time and worked flawlessly while doing some fairly
complex things, managing multiple processes for the customer. When I wrote it
I had never done any web development and had not coded for 4+ years. C++
before that.

Due to my lack of web development knowledge and massive time crunch, I built
it with no framework, zero best practices and the most basic code I could
build. The code is atrocious, and would not scale at all but it does what it
is supposed to do and works because I did everything in the most basic way,
there are no gotchas. Straight html, basic javascript and php on the backend.
It interacts with google docs as well. Its almost to dumb to break.

Sometimes we really do complicate things with all of our new fancy frameworks,
microservices, states, etc.

With that said, it would be impossible for anyone else to maintain and if
anyone posted the code on the web I would probably be unemployed. But it
works, very difficult to insert new features though.

------
_wldu
This tactic is also used to criticize/de-legitimize existing, working
solutions when a new manager wants his old, familiar environment. _" What
Bob's team built is far too complex. If we replace it with the system I used
at my last company, it will be much simpler."_ If I had a dollar for every
time I've seen this type of thing, I'd be rich. So be careful you are not just
playing corporate politics and changing solutions every time a new CXO joins.

~~~
gk1
Totally. Funny enough, marketing platforms like those mentioned in the essay
are usually the first things to get replaced when a new VP or CMO joins. Too
often, unfortunately, that decision is made not because it's the best tool for
the company but because it's the most familiar one.

------
vezycash
Using the company this author helped to migrate to hubspot as an example.

I'm not sure if the simpler solution - hubspot existed when the company
started hacking solutions together.

Also, the move to hobspot was relatively easy because the business processes
and workflows were already well known and defined.

Starting from scratch, fighting the fires as they appear, a bit of patchwork
seems unavoidable.

Everything is obvious in hindsight.

I guess the real lesson here:

Simplify once in a while. Like Facebook just did with messenger.

~~~
notyourday
> I'm not sure if the simpler solution - hubspot existed when the company
> started hacking solutions together.

Author's value proposition is "I will help you outsource your complicated
marketing tools". If you reject the gospel that your marketing tools are
complicated, author's service would be not needed.

~~~
gk1
Not the intention at all. My job is to parachute into a startup facing some
problem, fix the problem quickly, and get out. I have no business being at a
startup that's running perfectly, or creating solutions that will require me
to stick around.

------
busterarm
I built a static jekyll site using wordpress as a backend for the content
editors (jekyll build reads directly from the database) hosted on a handful of
AWS VMs running NGINX and loadbalanced by HAProxy. The build system is
automated in CircleCI. WordPress is hosted privately on a hosted WordPress
provider.

There's an ELK stack + plugins for logs and alerting and saltstack to trigger
some basic automated sysadmin tasks.

I left 3 years ago and the entire infrastructure has run on autopilot with
zero downtime. Content editors and web developers still add to the content
constantly. There is some dynamic content on the site handled by JavaScript
and a few light APIs that I built to support the site. There's over 5000 pages
of content. This simple infrastructure powers a website with a low-5 digit
alexa rank for a business bringing in about a billion dollars a year.

The entire infrastructure bill comes in under $100/mo and they no longer need
to employ any backend engineers. We negotiated a contract agreement for me to
support the site if they need it. They haven't.

------
sebringj
In terms of the little guy having a saas, this is true for me...my saas
basically is relying on s3 for both file storage and a json file per account
to list out users allowed in the system. The only time it goes down is when s3
goes down or the occasional container restart.

~~~
dhbradshaw
I'd be interested in hearing more details here.

Are you saying that you use s3 as your only data store?

~~~
sebringj
Yes...think of it as layers... or manifests that point to another set of
manifests global (list of users) -> accounts (have lists of users in them and
role), content -> list of json files. I use a combination of json logic or
simply listing out keys in a specific bucket...the speed of access doesn't
seem to be something that bothers anyone, it started out as an experiment of
less is more and because the app is not heavy on db searching etc rather only
specific to account. I use oauth2 instead of relying on my own infrastructure
and let people send out invites where they would use an email only but it must
support the oauth 2 providers i already have in place. Basically, doesn't go
down because it relies heavily on other things working that has teams of
people working on them with gigantic resources. Don't get me wrong...I'm not
saying it will scale to a million users, my clientele is mainly corporate that
have a good amount per account sale but if it needs scaling, at least i didn't
waste time and money on it during the interim and can always db-back it or
dynamo it later.

------
einpoklum
> Simple systems may have less downtime

* Yeah, but often they do less / are less effective during uptime.

* "simple" is an inspecific concept. Is the implementation simple? The API? The hardware? The requirements?

* "simple" is a vague concept. Is a 1000-line C program more complex than a one-liner script in a fancy higher-level scripting language? Yes, if you ignore the 1,000,000 line abstract virtual machine, JIT compiler, interpreter, large standard library and what-not. Otherwise - maybe so, maybe no.

> Modifications before additions

I actually agree, but:

* Managers don't like it.

* Sometime this downtime happens before there's ever been any uptime...

* Tomorrow you need something else, and the changes for that require re-engineering the parts you already re-engineered to best accommodate the previous addition.

------
praptak
This holds up to a point. The systems that require really little downtime get
progressively more complex.

Need to handle machine failures? Push a bugfix without causing downtime?
Maintain a single state across the redundant machines? Handle load spikes?
Ones created on purpose to DOS you?

All of this requires additional mechanisms, which can themselves cause
failures.

------
uk_programmer
This feels like it is stating the obvious. However it is easily forgotten.

A lot of Software Engineers / Developers in my experience tend to over
engineer projects or they are mandated they do so by so their dev leads. It is
easy to get caught up into abstracting almost everything away almost for the
sake of it while it provides no real benefit.

~~~
hnick
Don't forget the old classic Resume Driven Development.

------
hinkley
Simple systems also benefit from Amdahl's Law, but with humans instead of
CPUs. This is an observation I've been trying to sell others on for years and
having only modest luck. When I can show them, they get it, but you have to
push and push to get people there if words fail.

When the shit hits the fan, there's an 80% chance that your most senior
members will find the problem. They may find it quicker in the simple system,
even, and others might get there first. But that last 20% is a very long tail,
and even longer in a complex system.

In a complex system, many of your members cannot participate in the triage
process, because they don't know enough to know what's relevant. They slow
down the people trying to work the problem trying to learn new things (good)
or offering low-probability scenarios (bad).

There is no 'All Hands on Deck' scenario for the complex system. To be
responsive, you have to kick some or even a bunch of people out of the room,
and once they leave they can't really participate.

The first phase of a triage is getting a tight repro case. Many avenues are
blocked until that happens. And some people have a knack for repro cases but
aren't so great at debugging. With more people you get a higher quality repro,
which cuts a lot of time off the rest of the process.

Part of debugging is the cost of the verification versus the likelihood.
Simple Systems afford the opportunity for people to test out unlikely but
plausible scenarios that are in the long tail, without distracting from the
more 'boring' checks already being done.

And when bugs are identified prior to deployment, a simple system means you
can hand the repro case to the responsible party and expect/demand that they
get their fix on the first try.

------
geggam
Think this article is a different variation on the same theme.

[https://mcfunley.com/choose-boring-technology](https://mcfunley.com/choose-
boring-technology)

------
numlock86
> Modifications before additions.

> When new requirements come up, the tendency is to add layers on top of the
> existing system—by way of additional steps or integrations. Instead, see if
> the system’s core can be modified to meet the new requirements.

I like the mention of this as it is - at least to me - really counter-
intuitive.

------
dbg31415
"[The] more simple any thing is, the less liable it is to be disordered, and
the easier repaired when disordered..."

\- Thomas Paine, Common Sense, 1776

~~~
gk1
Perfect. Added it to the essay. Thank you!

~~~
dbg31415
Took a few years, but glad that Government degree finally came in useful! (=

------
buboard
the article is self-refuting in a way, although the message makes sense. The
author describes simplicity as "fewer levels of abstraction, and each of them
well understood" , like a ship rudder (which is a good definition). But no-
code tools etc are anything but well understood, and usually introduce
hundreds of levels of abstraction (due to their genericality). The same goes
for frameworks that introduce tons of abstractions in order to cover cases
that are irrelevant to the problem at task. Ideally, the chain of scripts that
do one thing and can be read in 10 minutes is actually simpler and a much
better way to fix the ship.

------
matchagaucho
Declarative coding tools are to blame here.

This is a well written article, but the metaphor is a little stretched.

Marketo and Salesforce are the "ship builders". Marketers are the crew
operating the ship.

The declarative tools in enterprise apps provide Marketers with conditional
branch and execution capabilities without code structure, so the average
Marketer just keeps adding layers of complexity. Whereas a Developer would
continually refactor and make old code obsolete.

The crew operating actual ships do not have anywhere _near_ this level of
access to the ships configuration, hence more reliability in the system.

------
ddevault
This article misses one crucial point in suggesting Looker over "a patchwork
of custom scripts and APIs". The complexity doesn't go away because you
offload it to Looker. When Looker goes down, your application is still in
trouble. It's still your responsibility when your system goes down, even if
you can point the finger at some third party service. Your system is still
down! That being said, don't use Looker in the first place. _Do not spy on
people_.

~~~
gk1
Several people pointed out the fault in using Looker as an example. I very
much welcome suggestions for a different product to plug there!

------
ThePhysicist
I don’t know, most systems like the container ship he/she described just seem
simple but are actually pretty intricate inside. The logic behind them might
be easy to understand, but the devil is in the details. A combustion engine is
also simple conceptually: make stuff explode in a closed chamber to move a
rod, convert that the movement to a rotation of a shaft. Super simple, yet I
doubt I could debug many problems my car engine has with such basic knowledge.

~~~
XorNot
This seems like an oddly appropriate metaphor though - engines aren't super-
complex once you know what everything does, but the biggest problem in car
mechanics is just getting access to the specific part you need to measure or
look at.

------
thinkloop
Reminds me a bit of home automation (which I enjoy). You manage a plethora of
battery powered devices that need constant connection to finiky wifi with
continuous access to your Linux server hub so that push and pull of data
remains in tact so that you can have AI actuate a function through a "remote"
\- all to replace a light switch. Sometimes I wonder where this will end up.
Might need 50 years before reliability and simplicity are up to par.

------
vasergen
The problem is that for different people 'simple' has different meaning.

------
awinter-py
> features don't justify complexity

Such a good line. I've seen companies killed by hard-to-follow arguments
advocating for technical complexity which serves feature goals which serve
business goals.

'Good strategy is simple' from some strategy book is sometimes right;
arguments that hold business goals hostage to technical complexity are
trouble.

------
stormdennis
His website had this annoying message about subscribing for updates that has
no obvious way of getting rid of it excepting by clicking the accept button. I
guess that's simpler than having an additional "no thanks" button. The article
was a formulaic retelling of well known wisdom too.

~~~
gk1
Hey, there is an X in the corner to close it. The subscription slider is made
by Mailchimp so I don't have much control over the size of that X,
unfortunately.

------
z3t4
Sometimes the complexity is there for a reason. But if you can just start
over, those reasons was not that important!? So don't hang on to functionality
because it would be nice to have, you will have to pay the price for those
features over and over.

~~~
collyw
Sometimes, but a lot of devs fetishise complex stuff. I disagree wit a lot of
my tech leads decisions because he always wants to do the "most correct" way,
whereas I want to keep things as simple as possible with the aim of keeping
our code maintainable. For example, he wants to a add a graph database to the
tech stack, while I realise that its one more thing to manage and potentially
break and mongodb (that we already have) will be probably be good enough.

~~~
z3t4
Things only get complexed if you mix them, like cords in a box. So you might
be able to keep it simple as long as you do not have code that depends on both
the databases, and that the two databases don't depend on each other.

------
aschatten
Everything seems logical and simple. But the examples lack details and
realism. Take for example this quote:

> In the end, the system I put in place had 97% fewer processes (from 629 to
> 20) while providing all the same capabilities. A bug that was found a few
> days later got resolved in four minutes.

It‘s called refactoring. It‘s a constant process, that none likes to allocate
time to, because it does not move metric immediately. Especially in marketing
with a very fast changing landscape you go from 20 to 600+ very fast.

------
Papric0re
I share the idea of the author, but I miss actual talk about how we can make
things better. Because his arguments are also true when it comes to security.

What are alternatives to commonly used complex systems, like
Docker/Kubernetes, Active Directory, Remote Access or just shared folder
structures? How to keep the functionality of centralized management while
making it simple? Would be happy to hear your thoughts on this.

------
Taek
It's similar to optimization in that it follows a U-curve. Up to a point,
simpler things have more performance, and simpler things have less downtime.

But after a certain point, it's no longer enough to make it simpler. You have
to get a lot more involved again, and this time the complexity has to be
intelligently focused around the end goal (be it performance or uptime).

------
rantwasp
plug for my all time favorite paper in the subject
[https://web.mit.edu/2.75/resources/random/How%20Complex%20Sy...](https://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf)

5 pages of pure awesomeness

------
speedgoose
It's important to find a compromise. The simplest system at my work would be
something like a php/mysql backend hosted on a old fashion shared server using
ftp.

Good luck attracting good developers with that. We use something over
engineered but we have fun. The customers don't care but we do.

~~~
somurzakov
There are a lot of companies where Senior PHP devs are paid more than Senior
Java devs, because it is so hard to find a brilliant PHP dev

------
vasco
A bunch of paragraphs to type out KISS.

------
xiaodai
Do they also have less people using it?

------
je42
I guess the main question to be asked is should be:

is the complexity appropriate for the goal (i.e. Everything should be made as
simple as possible, but no simpler. )

Hence, often a required step to fix complexity is to change the goal, in such
a way that it can be implemented with less complexity.

------
gdm85
> In the end, the system I put in place had 97% fewer processes (from 629 to
> 20) while providing all the same capabilities.

I am suspicious of this statement. Is the author now supporting the company
and defacto replacing the guy that left?

------
dilap
I totally agree when designing man-made, synthetic systems. But it is
interesting to consider that the most reliable, self-healing systems that
exist -- animals! -- are incredibly complicated...

~~~
rantwasp
there is a difference between complex and complicated. Also consider that
systems where complexity serves a role are not complicated. Also: simple
things that are aggregated into complex things through a hierarchy that is
also simple can be reasoned about with failure happening subsystem level (and
can be easily mitigated - think supervision trees in erlang)

------
Koshkin
Surprised nobody has yet mentioned Einstein’s quote.

------
deckarep
My microservice that returns “Hello World” via HTTP has been running solid for
5+ years now with zero downtime.

------
pm2222
`Uptime` of various __*x systems comes to mind. Quality of code also matters.

------
unnouinceput
Very good article. Please also preach this article to those who praise
Kubernetes right and left (remember, Kubernetes it's the new blockchain hype;
and just like blockchain you don't really need it unless you're a
cryptocurrency; which 99% of companies who use Kubernetes aren't).

------
abinaya_codes
I love simple things, thank you for sharing!

------
winrid
I love simplifying things.

~~~
nineteen999
I'm not sure I follow, could you break that down for me?

~~~
hyfgfh
KISS

~~~
ArcMex
Keep It Simple Stupid

~~~
lincpa
Keep it Simple and Unified.

[https://github.com/linpengcheng/PurefunctionPipelineDataflow](https://github.com/linpengcheng/PurefunctionPipelineDataflow)

------
pirate_dev
duh

------
harrisreynolds
This is a very important idea. I like the shipping container metaphor. I have
often used the idea of a 2x4 (as in lumber) in building houses. The humble 2x4
is such a simple product, but when put together with nails and screws becomes
the cornerstone of a complex structure.

It is simple and it just works for its intended purpose.

I am working hard to apply this to the design of Webase [1] which is a #nocode
platform that is inherently complex. But I really believe that if we get the
design right it will support a large number of sophisticated use-cases.

Thank you for sharing!

[1] [https://www.webase.com](https://www.webase.com)

~~~
dang
You're overdoing these links. That's no doubt why you're getting downvoted and
flagged.

It's fine on HN to post your own work (1) in places where it's relevant, as
long as (2) you only do it occasionally and (3) are also participating in the
community in the intended ways, submitting interesting articles and having
curious conversation. But when users break these rules of thumb (and you've
been breaking all three of them!) it crosses into spamming. The community is
very aware of that and really doesn't like it, so it's not in your interest.

Doing this by hijacking top comments and threads about other people's projects
is particularly not ok.

~~~
harrisreynolds
Thanks for the feedback. So far I have been getting positive feedback which is
why I have continued to do this.

I have been trying to add to the overall value of HN with insight in general
and only including a link to my site if it is relevant.

But I hear you and will adjust accordingly.

I do sincerely appreciate the feedback as I have been steadily gaining karma
and then out of no-where it when backwards today.

~~~
dang
It's a clear indication from the community that you're overdoing the
promotional links. The fact that you haven't really been participating in any
other way accentuates that impression. Readers check these things.

