
Boring Systems Build Badass Businesses - samscully
https://devopsu.com/blog/boring-systems-build-badass-businesses/
======
onion2k
In the stated examples there's are no benefits to the additional complexity.
No one would argue that complexity _for the sake of it_ is a good idea. That'd
be insane. If Alice's restaurant could handle 5,000,000 covers a night with
only 1 member of staff while Zola's restaurant could only handle 10,000 then
you'd have a more realistic scenario to compare with the SaaS industry. The
benefit of "complexity" is that you are able to do more things with less work.

The ideal is to build powerful systems from small, simple processes - if any
single stage is trivial then anyone can understand it, fix it, modify it, and
so on. With many little processes working together you can do amazing things.
A good example in software is a build process - a good system can lint code,
test it, uglify it, minify it, push it to version control, watch for changes,
reload an output mechanism, clean up a distribution, and push it to a live
server if it's working all from a single command. That's very 'complex', but
really it's just a set of very simple linked processes.

~~~
ryanjshaw
I think the restaurants were a poor analogy. Much better for me was the rule
right at the end:

> Innovate on your _core product_ , not on your _plumbing_ (this rule is
> extremely tempting for developers to break - see next rule)

I was first exposed to this during my final year university project in 2004.
We had a sponsor who wanted to build a marketplace for tourism operators and
B&Bs. I was the only real developer on the team of 4. I worked really hard
with little sleep, producing a really great caching layer and a very
impressive WYSIWYG editor. But when the project deadline arrived you couldn't
even book a room on the site.

I still struggle with staying focused; the best technique I've discovered to
deal with this is to log ALL work around a project, prioritise the tasks and
stick to only working on logged tasks with the highest priorities assigned
rather than do whatever I feel like.

(It took me even longer to realise what a business opportunity I had lost.)

~~~
akshatpradhan
He should have used the analogy of Alice making an automated wait staff (e.g.
waiter drones) for the restaurant. Not an amazing electrical system. I wonder
how that kind of story would fair in his scenario.

------
bsaul
I can easily see how this post could be mis interpreted, so i'll add my
personnal experience :

I had the occasion of building the same system in two different companies :
one was a start up, the other a huge company.

For the start up i could choose the tech i wanted, and decided to go for
python + app engine + backbone ( new at that time ). Those tech were "hype"
yet not absolutely brand new. I took some risks by choosing them but thought
it could be worth it.

For the big company i had to go with java spring mvc + sencha, they didn't
want to hear of any new tech that would be different from what they were used
to. They deployed it in their own infra.

Now, the start up project took 3 man month, the big company took more than 7,
and a year total before being deployed. The startup only paid an intern to
maintain the software,and almost nothing in infrastructure fees. The big
company outsourced maintenance to a software service company that proved
unable to do even the most basic troubleshooting.

I designed and coded the two systems, and i wasn't a guru of any specific
tech, so it's not an issue with the people. Sometimes, under the right
circumstance new technologies _are_ way better.

~~~
sgeisenh
The arguments of most of these types of posts can be hand-waved with "use the
right tool for the job."

But honestly, plumbing and electrical infrastructure is a terrible metaphor
for a development framework. There is no situation in which a handyman would
be praised for advocating complex systems. They complicate every single layer
of the build.

A sizable chunk of the "hype" tech facilitates faster, more efficient
development at fairly low cost (maybe sacrificing some scalability or
stability).

It would be reticent of me not to mention how much the deluge of JavaScript
frameworks pisses me off. It is overwhelming trying to keep up.

~~~
binocarlos
You can always ignore them - trying to keep up with every new javascript
framework (where 80% wont be around in 2 years) is gonna piss anyone off : )

------
jasonkester
Well said. I can't tell you how nice it is to have software in production on a
boring stack. It gives you freedom to _do other things_.

I can (and often do) go entire months without touching the codebase of my main
rent-paying products. It means I can, among other things, pick up a full-time
development gig to sock away some extra runway, take off and go backpacking
around the world, or better still, _build yet another rent-paying product_
without having to spend a significant amount of time keeping the old stuff
alive.

It seems like on a lot of stacks, keeping the server alive, patched and
serving webpages is a part-time job in itself. In my world, that's Windows
Update's job. Big New Releases come and go, but they're all 100% backwards
compatible, so when you get around to upgrading it's just a few minutes of
point and clicking with nothing broken.

I see it as analogous to Compound Interest, but to productivity. The less
effort you need to spend on maintenance, the more pace you can keep going
forward.

~~~
copperx
What is your to-go boring stack, if I may ask?

------
gordaco
This is why Java is used widely. It _works_. It works _well_. And this is also
why Java is great for huge systems (not in terms of users, disk space or
bandwith, but in terms of code size). The same can be said about a lot of
"old" technologies, and certainly about almost every industry standard out
there.

On the other hand, once in a while the Alice/Albert bet happens to win; be it
because the new system is really better (as in: easy to maintain, or really
being capable of managing higher amounts of workload), for non-technological
reasons (Alice/Albert just happen to have a great idea), or just because of
luck. Over time their technology may even become the new industry standard.
The problem here is that it's the Alices/Alberts from the world who make it
progress by trying new things (and failing often), but we're afraid of
failure.

So, yes, it's completely natural that corps resort to Java or C#, while
startups use Scala or Ruby.

For all of you doing startups in shiny new technologies: this means that even
failure has a bright side, since even in that case you've put your grain of
sand to make the technology more mature.

~~~
Retric
It's not just a question of language or platform but also culture.

Java is probably the #1 culprit for building huge systems to solve simple
problems. The goal is small, simple, fast, cheap, and 'enterprise' solutions
are rarely a good fit for new companies. Now, if your United Airlines and you
have complex problems involving hundreds of systems yea your in a world of
pain and Java is a great option for dealing with complexity.

That said, their are plenty of rock solid Java programs that avoid the bloat
there just rarely websites.

~~~
pjmlp
> Java is probably the #1 culprit for building huge systems to solve simple
> problems.

Don't blame the language for the enterprise culture.

Before Java existed I have seen similar complex systems done in C, C++,
Clipper, VB and quite a few other languages.

Just to cite a concrete example, before J2EE existed, we had CORBA and DCOM
systems, plugged with distributed transactions across multiple OS systems.
Great debugging experiences.

~~~
Retric
The culture, tools, and language evolve together.

How else do you explain things like:

InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonPainter

JDBC is perhaps a better example. It's short and simple but designed around
raw SQL which ends up creating a huge mess vs something like LINQ. Sure, there
are hundreds of 3rd party solutions to this problem just pick one except
external library's might pick something else...

~~~
pjmlp
Easy. As they have on a few CERN offices "You can write FORTRAN in any
language".

I have seen
InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonPainter
written in multiple enterprise languages.

------
noonespecial
The fun part is that you can be either Alice or Zola with nothing but perl.

The moral of the story might just be "stop trying to be clever and start
trying to be done", with all of the usual yaddayadda about preoptimized yak
razors.

~~~
thybag
This is pretty much my point of view. How new or buzz-wordy a bit of
technology is, isn't really the issue. It's whether or not the thing in
question will allow you to end up with a simpler solution.

Simplicity and supportability come hand in hand for me. I'd much rather
debug/fix a really simple app in a language I don't know, than a really
complex one in a language I do.

------
venomsnake
A simple rule - you should always remove complexity from a project and never
adding it. It builds on its own so any tech you add must remove some
complexity from the current project.

Warning sings for tech that brings more complexity than it is usually worth -
extensive xml configs, hiding of executable code, stack traces more than 240
levels deep.

Current favorite offender - GWT - I just love when something blows up in the
javascript and it just tells you - well signature object object object is not
what I expect in javascript apply. And you have no idea where exactly it was
generated.

So it is a KISS - the project must be of the least possible complexity to
solve the problem.

------
karterk
Boring system themselves do not ALWAYS build badass businesses. It's knowing
when to stick to boring systems vs taking the chance on something new. A lot
of systems start off as someone's side-project. It's a calculated risk when
you pick something that brings different things to the table.

------
benjvi
This post highlights something that can be a problem with the contracting of
workers. Namely, that Albert will be more in demand than Zip, despite having
built an inferior system. The failure of the business, in real life, is
probably not attributable to him - there are many other variables that one
could point the finger at (low demand, location, infrastructure, sourcing
prices, etc..). And, the manager will often not understand what truly
constitutes a "best-practice", maintainable solution. So, by default, he
probably ends up being paid more, and is seen to be more important and
accomplished as well.

So, where is the incentive for the handyman to act like Albert? And how do you
identify these people?

~~~
twic
Reminds me of Fred and Jane:

[http://groups.csail.mit.edu/mac/ftpdir/users/mrb/Hacks/Plans...](http://groups.csail.mit.edu/mac/ftpdir/users/mrb/Hacks/Plans/plan.brilliance-
v-brute-force)

 _Jane came to us with a great reputation. We thought she was going to be as
brilliant as Fred. But she hasn 't really proved herself yet. We've given her
a few problems that we thought were going to be really tough, but when she
finished it turned out they weren't really difficult at all. Most of them
turned out pretty simple._

------
mijustin
The maintenance aspect is huge. I've been able to observe how fancy, complex
systems fare over a long period of time (as opposed to simple systems): in
almost every case the "cool" complex system required way more maintenance.
There are just more things that can break.

Unfortunately, we don't normally record the "long tail" cost of a feature. We
build and deploy, but don't keep an eye on how much time it takes to maintain
that feature.

------
qwerta
Unproven innovative technologies are not necessary bad. They can give you edge
over competition. The real problem is to restrain yourself when applying them,
and have a fallback plan.

~~~
huherto
I liked the usage of the word restrain. I try to introduce one or two new
things on projects. Too many new technologies and it you will increase the
risk of the project, and you will spend more time learning than doing.

------
krisgenre
"There are many ways to achieve developer happiness, but making your core
business products a playground for developers seeking novelty is the path to
hell."

Excellent point. This also applies to programmers who'd like write everything
themselves so that they can learn more in the process. My current job involves
maintaining an application that has everything written in-house - logging,
html templating, url mapping, validation, form bean binding, scheduling and
what not! - all this is possible with just using slf4j, Freemarker, Spring and
a bunch of other lightweight libs. Some of the stuff is good and so it makes
me think the only reason would have been to become more proficient in OOPs and
Java.

------
noelwelsh
The problem is, simplicity is not an objective measure. Take monads, for
example. To most developers these are a foreign and possibly scary concept.
Once you understand them, however, they seem ridiculously simple. This is one
of the problems with monad tutorials -- they are so simple there is almost
nothing there. I know I spent a long time trying to find a "deep" concept when
learning monads, before I realised there isn't one.

Building a system with monads, if you understand them, is simple. You can
write together components easily, and have concurrency, logging, error
handling and more all nicely abstracted away. But is this a simple system? It
depends entirely on your background.

------
mathattack
This is an argument for buy vs build. As others have stated, the question is
whether complexity is worth it. My bias is we tend to underestimate the
complexity of small additions, and overestimate the benefit of having control
over a system. The implication is too much complexity in things we build
ourselves. Sometimes the industry standard solutions aren't appropriate, but
it all depends on what a company wants to focus on.

And I'm not sure of the reference for Zola's restaurant, but I like the
Guthrie inspired complexity of Alice's restaurant.

------
nadam
Of course if you are working on a boring problem it is a mistake to try to
make it more interesting by incorporating interesting tools. This is a common
problem in web development for a lot of people. On the other hand if the
problem you want to solve is interesting and hard then probably you will not
go far with the boring standard solutions. (see: Oculus Rift).

Summary: if you don't want to be bored, choose interesting problems, not just
interesting tools.

------
dscrd
Even though it's still hip, this is why Go works in capable hands. As a new
programming language / platform, it's academically quite boring. In fact,
that's the number one criticism of its detractors.

~~~
sergiotapia
That's what I love about it. No surprises and rather straightforward. Plus
it's the first language I feel comfortable using a plain text editor to work
with. I use Sublime Text.

~~~
poolpool
I wouldn't call sublime text a plain editor.

~~~
estebank
Plain text[1] editor, not a plain[2] editor.

[1]:
[http://en.wikipedia.org/wiki/Plain_text](http://en.wikipedia.org/wiki/Plain_text)
[2]: "not decorated or elaborate; simple or ordinary in character"
[https://www.google.com/search?q=plain](https://www.google.com/search?q=plain)

------
jesstaa
The sad thing about our industry is that the boring systems with the greatest
support are also often the most complicated and hardest to deal with.

So the choice ends up being,

* Go with the system you can get some kind of possibly useless support for, but your people will have a hard time dealing with on a daily basis.

or

* Go with something your people can understand that might not have wide external support.

------
nsfyn55
The cases presented in this article are contrived.

1) There are room for both boring and cutting edge technology in any business.
Albert didn't drop the ball by choosing cutting edge tech. Albert exhibited
poor risk management skills. Alice wouldn't be complaining if Albert took a
controlled risk and installed a next generation flash fryer that gave a clear
competitive advantage over Zola in terms of personnel and order-to-delivery
time.

2) Good ideas require both Albert and Zip. Zip keeps the lights on and the
costs down for all the mundane BS required to run a business. Albert is the
disrupter. He is the reason starting the business was a good idea. He is an
iconoclast that looks at the state of the world and said "I can do this
better".

The title of this article should be Boring Systems Build Benign Businesses

~~~
shitgoose
Albert doesn't start the business. He is the plumber on payroll and should
behave as such. Alice and Zola start the business and innovate in the realm of
cooking, not plumbing.

~~~
nsfyn55
I didn't say Albert started the business. I said that Albert was the reason
starting the business was a good idea. What I did say is that either Alice or
Zola would be better served by having elements of both Albert and Zip.

I would consider hard to find a decent software engineer that would introduce
20 unproven tools to for something allegorically related to "plumbing" like
say your web server.

Additionally, a lot of this is based on perspective. If I suggested standing
up a web infrastructure on "nginx" am I an Albert or a Zip. I mean "Apache"
has a much longer track record...right? By this article's logic we should
write everything in FORTRAN and use single-tasking operating systems. Those
have been around since the 50's ... rock solid.

Net out is "proven track records" and "best practices" are convenient stop
gaps for ignorance. They are a way for people that don't know any better to
manage risk. "I don't get it ... but it worked for her". Being competent and
knowledgable, having good risk management skills, and understanding the
problem will serve an organization much better than being conservative in your
choice of tools.

------
dsirijus
"Boring is where the money is". ~ some HN dude.

------
maximgsaini
Why add the complexity of a car when you can simply walk?! The car will break
down, you will have to waste days taking it to the service station, you will
have to get a driver's license, you will get tickets, you may kill someone and
get in trouble, you can't drink if you will be driving. Why add so much
complexity to your life?

The idea that complexity in itself is bad is flawed. Sometimes innovation does
require complexity. Complexity for the sake of complexity is bad.

>"But [some new unproven system] is really cool! Even [some big company] uses
it!"

A company I know uses a big/buggy oil pipeline leak detection software. It is
very complex and very buggy. Tech support has to be called in every few
months. But they still use it. Why? Because it will detect oil leaks much
faster. Potentially saving them millions in case of something bad. Should we
stop innovation because we are scared of 'complexity'? I wouldn't suggest
using a system because it is 'really cool' and a 'big company uses it'. But
why do they use it and why is it 'cool'? Can it make you more money? Those are
the questions worth asking.

>"Innovate on your core product, not on your plumbing "

Every bit of complexity deployed to make more money is good. Can you tell and
prove how it will make money?

Every bit of complexity added because it is 'cool' is flawed! If plumbing can
make me more money, then hell yeh it requires some investment. Every situation
is different.

------
weixiyen
I wouldn't say that boring solutions are always the best, unless they
satisfied the conditions below.

Here's what is REALLY important:

A) How fast can you get your first product in front of customers?

B) How often can you measure and iterate on that and get the new version in
front of customers.

You should pick the best solution that optimizes for A & B. Both are really
important because they will help you discover the actual thing you need to be
building.

~~~
collyw
I am involved in an ongoing discussion at work. The guys implementing the
project want to go with a NoSQL / Angular / Node solution. I see a relational
database as being a better fit.

"Its so fast, even running on my laptop" "you only have 3Gb of data just now"
"Its so quick to develop, just plug Angular straight into elsaticsearch, no
schema,..."

The thing is, I just see lots of problems stored up for the future. Sure you
have to define a schema up front if you go for a relational database, but
changing it isn't such a big deal, not until you are in production with live
data. Sure you might save time up front, but you are essentially cutting
corners to achieve this, and storing work for the future.

------
mooreds
I run/work in a small shop supporting a real estate brokerage with custom
software. For every new core project, I ask if the problem can be solved in
java. Not because it is the best language, or the one I am most familiar with,
but because it is the language that almost all the other systems are run in.
Ditto for databases--projects should use mysql unless there is a very
compelling reason not to. And we use one data processing tool--pentaho kettle.

Now, we've actually had some other languages 'sneak in'\--perl, bash, python,
javascript. But they were for closely scoped projects and, in a couple of
cases, I asked for a prototype in java and the other language first.

It is hard to do this, because I read hacker news and am interested in keeping
my skills up to date with the latest and greatest interesting technologies. I
and my team have other alternatives to explore (hackfests, not-work side
projects), which makes it easier. I'm doing the right thing for this company,
and that's the right thing to do.

------
porker
I was planning to architect a new system using SOA (or the newly-hyped
microservices), then realised the complexity it'd bring.

Each individual system is easy to debug, test etc. Debugging what's wrong when
the output isn't what you expect - much harder.

Not sure what's the boring (vs familiar) option here, but the old adage "All
that gleams isn't gold" is true...

------
robinwarren
I get the rather bluntly hammered home moral to use safe reliable tech but
that misses a lot of subtlety.

I think the moral of the story is to load test before you dump a bunch of
customers onto your system. Regardless of the tech you use you can easily fail
in this regard. And secondly not to value people for the effort they put in
but the results they achieve.

------
inthewoods
This post spoke to me because I'm in the middle of an interesting decision -
for our public/marketing website, should we go with Wordpress (something a lot
of people know, etc) or a static site generator (fill in your favorite - Harp,
Docpad, etc.)? The argument for going with the static site is that we'll have
a much faster site (it will be static) that will likely be easier to customize
(we don't need a lot of what Wordpress offers) but the potential downside is
that most developers don't know the static tools so if I hire someone new, I'm
likely training them. Now, I don't think training will be that hard if you get
someone with a decent background, but you get the idea.

What would you choose? Safe and stable Wordpress with more customization
effort, or the static site generator idea with less installed based of
developers?

~~~
stesch
Wordpress, safe, and stable. Wow. Never heard this together.

~~~
jodrellblank
You can have safe and stable Wordpress, you just can't have it cheap:

[http://wpengine.com/](http://wpengine.com/)

( patio11 post about WPEngine - [http://www.kalzumeus.com/2012/02/09/why-i-
dont-host-my-own-b...](http://www.kalzumeus.com/2012/02/09/why-i-dont-host-my-
own-blog-anymore/) )

------
chasing
Well, I mean, it's all about understanding both the tools and the needs and
selecting the tool that fits the need. Some "restaurants" have exotic needs
that good ol' Zip might not be able to satisfy using his system. Or Albert
might have ways to do things that are way cheaper -- require fewer resources,
less time, etc -- but have the drawback of using newer tools that might become
abandoned, have low developer numbers, etc.

But, as a developer, this is why you have a conversation with your client and
understand what their needs are. So you can understand these trade-offs and
make the best possible recommendation. Neither Alice's way nor Zola's way is
the Way Things Should Work 100% of the Time.

------
goombastic
I think this is the precise reason why every industry and function has a
process framework. Working around processes/functions and its value maps while
creating a solution is the best way to not just meet customer expectations but
also ensure that your products play well with other products a buyer might
have.

Processes and sub process maps like Procure 2 Pay, Order 2 Cash, etc, are
there for a reason. They tend to make life simpler for buyers making a choice
and also help ensure that your product doesn't have process blind spots that
will kill it.

The big guys in the ERP space have perfected this approach and it's something
a lot of business oriented startups don't seem to consider.

------
dkrich
A better analogy would be Alice doing a bunch of research up-front to think
that she needs all kinds of cool features before opening and then hires the
plumber to enable them, rather than just hiring him because he knows how to
build complex systems. The latter really isn't too realistic while the former
happens every day in many, many companies.

This overbuilding is usually borne from a lack of understanding of what is
really needed and in lieu of learning through doing and reacting, people try
to build up-front in anticipation, putting a lot of importance on things that
ultimately turn out to be completely trivial.

------
frankwiles
Couldn't agree more. The pre-optimization of cool might be a good name for it.

------
_random_
And then there is outsourcing to the cheapest bidder overseas...

~~~
pjmlp
With guaranteed quality delivery on time. :)

------
jesalg
Great analogy, this can't be said enough.

Although there has to be a distinction made between using the latest
technology and how you choose to architect and design a product with that
technology.

I've seen super complicated web architectures with something as widely used as
PHP. So it's not so much about what tools you use but rather how you use them.
Choose the right tool for the job while keeping the business constraints in
mind.

------
borplk
From the title I was expecting this to be about the area of the businesses,
perhaps suggesting to solve boring and real problems for real people who pay
real money instead of, as is fashionable today, building "buisnesses" for
sharing your crap to another crap and liking and commenting and following this
and that whilst being fed with advertisements.

------
midas007
There might be another way to look at it in terms of overall risk appetite:

\- when maintaining a traditional venture, innovate more on tech to attract
more talent and interest. (Microsoft Azure folks, Basecamp (formerly
37signals))

\- when innovating in something game changing, don't take huge risks on
plumbing. (Facebook, Craigslist)

------
timc3
Simply put: Choose your battles wisely, and the ground you do it on even more
so.

Though one can gain serious competitive advantage by using something new to
compete against established players or use something well tested in an
innovative and new way.

------
mark_sz
Boring stack: PHP+MySQL ?

~~~
a3voices
I've always wondered how to scale this once you have thousands of active
users?

~~~
acrooks
[http://hhvm.com/](http://hhvm.com/) seems to work pretty well for Facebook.

------
flylib
what was up with the Github and 37signals references? They both use Ruby on
Rails which could be considered a niche technology, 37signals even admits to
use the most bleeding edge version live in production before they even put out
a beta of the version to the public so if anything referencing them is
actually hurting the article

------
menubar
Bad biscuits make the baker broke, bro.

