
In Pursuit of Production Minimalism - grey-area
https://brandur.org/minimalism
======
peterwwillis
Fwiw, the SR-71 was not engineering excellence. It was slapped together to do
what needed to be done. It literally leaked jet fuel onto the ground when it
wasn't in the air, in order to allow the metal body panels to literally
stretch and close gaps in its skin when it was in flight (because physics).
The air frame couldn't handle a takeoff with a full fuel tank, so it had to be
filled up right after taking off. 90% of the parts manufactured didn't work.
It was difficult to maintain and eventually developed a "speedy" one-week
turn-around time to get it to fly again after a mission, assuming a base had
the specialized tools and support staff to do so.

The SR-71 is an example of how the sheer will to get something done can indeed
accomplish great feats, if at great cost and complexity. It's also an example
of how innovation is no guarantee of success; since the Blackbird was retired,
we still have no airplane as fast as one built in the 1960's.

A great plane is just what it needs to be. It may not be the most minimal it
could be, but it will be the most effective. Another example, the A-10
Warthog, is not "minimal", but it is an amazing plane.

I think to get the best result based on the desired outcome, you have to
pursue craftsmanship and excellence through continuous education, practice,
and improvement, and not worry how it looks so much as how well it works.
Complexity is a sin, except when it is necessary.

~~~
fermienrico
I couldn't disagree with you more. I am presuming you have neither read the
history of SR-71, nor have a good understanding of the trade of engineering. I
am ex-Lockeed employee and even though SR-71 was a distant historical project
when I was working there, there were internal discussions about the
engineering excellence behind the plane's performance. I worked in stress
analysis team where we frequently joked about how we don't need to deal with
thermal expansion like in the SR-71.

You can choose to call it "Slapped together" and brush over the sheer
ingenuity of the technologies that made SR-71 successful, or you could try to
read up on the insane amounts of __engineering excellence__ that took to make
it achieve its mission as a recon aircraft. Its mission wasn't to be a
commercial aircraft, its mission was to go as high, far and fast as possible -
Reconnaissance and to return safely by literally outrunning missiles.
Therefore, refueling and leaking fuel are appropriate trade-offs.

Skunkworks engineers were perfectly aware of the trade offs you mention - that
is integral to __any__ engineering discipline from semiconductor to aviation.
That __does not__ imply the lack of engineering excellence.

Leaking fuel is __the__ most popular fact about the SR-71. But you omit or you
are unaware of the ingenuity that went into making the Ramjet engine, material
science, hypersonic regime that was completely new and nothing had flown at
that speeds before, etc...there is just so much to say, I am having a hard
time coherently replying :-) I apologize if my response was a bit harsh - I am
just passionate about this topic!

~~~
peterwwillis
If we're defining "engineering excellence" as the craziest, most amazing thing
you can possibly build, then the SR-71 absolutely qualifies. So do top fuel
dragsters (though obviously the SR-71 was way more difficult to achieve).

What I meant by "not engineering excellence" was in relation to the original
post, comparing how to manage production servers with the SR-71. Nobody should
regularly operate their servers like an SR-71 or top fuel dragster. It's great
to fly really really fast, but most people should probably not be doing that,
because it requires making exceptions like leaking fuel. You actually want to
run your server like a big slow complicated comfortable redundant jumbo jet on
autopilot.

Hopefully you get my meaning. I apologize if I made their accomplishments seem
minor. I meant no insult against the Skunkworks engineers or the SR-71, and
your response was not harsh at all.

~~~
Animats
Yes, the SR-71 was a remarkable achievement. Ben Rich's book about the Skunk
Works describes the project. It's amazing that thing ever worked. It's not,
however, engineering minimalism. It's money is no object R&D. There are many
exotic technologies used in the SR-71 never seen before in aviation and seldom
used again. Large parts made of Invar, a zero expansion alloy normally used
only for watch springs, for example. The price of all this was a high-cost,
high-maintenance aircraft.

A good plane to look at for production minimalism is the Boeing 747. A huge
aircraft even today, it was designed as a commercial product to be produced in
quantity and used intensively over a long life. About 1500 were built and many
are still flying after 40 years. It's reliable, maintainable, safe, and
profitable.

It's not exotic. There are some new things in it, but not that many. It's
mostly upgrades to technologies known at the time. Better flight controls,
better navigation, better autopilot, better hydraulics, better tires, better
engines, etc. But little that hadn't flown before at smaller scale. The
emphasis was on eliminating the weak points during design. The B-747 has four
hydraulic systems, for redundancy.

It's worthwhile for people in software to read the pilot's manual for a large
commercial jetliner.[1] They're not that complicated. If operating your system
is more complicated than a jetliner, you're probably doing something wrong.

[1] [http://www.aviationforall.com/wp-
content/uploads/2016/09/AOM...](http://www.aviationforall.com/wp-
content/uploads/2016/09/AOM_727_200_Sim_Copy.pdf)

------
toast0
I agree with the thrust, but these need more nuance:

> Avoid custom technology. Software that you write is software that you have
> to maintain. Forever. Don’t succumb to NIH when there’s a well supported
> public solution that fits just as well (or even almost as well).

Software you run is software you have to maintain; until you can burn it in
the fires of Mordor. If there's a public solution that is a good fit for your
problem, and was built along the same lines you would build it, then that's
great; but if it's a great fit for the problem and also pulls in 70
dependencies and updates three times a week, that's probably not worth the
integration mess.

> Use services. Software that you install is software that you have to
> operate. From the moment it’s activated, someone will be taking regular time
> out of their schedule to perform maintenance, troubleshoot problems, and
> install upgrades. Don’t succumb to NHH (not hosted here) when there’s a
> public service available that will do the job better.

You don't need to host everything, but you better have a backup plan when the
service you depend on decides that they're no longer going to provide the
service. Chances are, you're going to need to debug the service too, sooner or
later, and it's harder to debug a black box. That said, it's great when you
can lean on a rock solid service to do things that are important, but not a
'core competency'

~~~
iamleppert
The whole fallacy of slopping together a bunch of open source stuff and not
bothering to understand any of it is a house of cards waiting to fall down.

It's far better if you have the time and resources to develop your own
software from as low a level as possible. Each layer of abstraction that you
can shed is an opportunity to tailor your solution more closely to your
problem and to have expertise in-house. Big tech companies know this and it's
why they do a lot of stuff in-house. The key is in knowing what to develop in-
house and what to punt on, and when.

~~~
dsjoerg
That's why I fab my own chips!

~~~
iamleppert
If I had unlimited time and resources that’s exactly what I’d do. Big
companies often have both so that’s exactly what they do.

------
simonw
The rule of thumb that I go by is that in order to add a new technology to a
stack it can't just represent an incremental improvement: it needs to be a 2x
improvement in productivity or (even better) it needs to make it possible to
build something that isn't possible to build without it.

The cost of introducing new technology (in terms of training, tooling and now-
you-have-two) is so high that it's just not worth it if it will only
incrementally improve how things work at the moment.

------
mercwear
This article mentions Clarence "Kelly" Johnson. If you don't know who he is
and care to learn more about one of the best engineers that ever lived (SR-71
is his work), check out his book: [https://www.amazon.com/Kelly-More-Than-
Share-All/dp/08747449...](https://www.amazon.com/Kelly-More-Than-Share-
All/dp/0874744911)

His second in command at Lockheed, a man named Ben Rich also wrote a very good
book: [https://www.amazon.com/Skunk-Works-Personal-Memoir-
Lockheed/...](https://www.amazon.com/Skunk-Works-Personal-Memoir-
Lockheed/dp/0316743003/ref=pd_lpo_sbs_14_t_0?_encoding=UTF8&psc=1&refRID=24ZEA753MP8VAPZAA7KC)

~~~
ThenAsNow
> (SR-71 is his work)

One unfortunate trait of the original Lockheed Skunkworks culture was to make
Kelly Johnson (brilliant as he was) _the only face_ of the organization. As
just one such example, many know of the clever inlet system on the A-12/SR-71,
but it, like many things Skunkworks in the pre-Rich era, is incorrectly
ascribed to Johnson. The real person who deserves the credit as the chief for
that system is David H. Campbell, but how many people know his name?

~~~
theoh
The form of the T-38 was apparently determined by Lee Begin at Northrop. Who,
you ask? Well, good luck finding information about him on the Internet.

------
maxxxxx
These are not bad. But "Use services" seems like a recipe for long term pain.
It's an external dependency you can't control. I would prefer to use the
external service to learn what you really need and then try to insource it
again.

------
castlegloom
Great article, classic comment section:

Half of people arguing about the plane analogy

Other half arguing about jQuery

------
zerotolerance
Minimizing what code you own is a naive optimization. When you consider the
liabilities that third-party library and service dependencies introduce it
becomes quite attractive to build your own minimal systems. I'm not sure when
programmers became so averse to writing simple code. But leftpad was just the
beginning.

~~~
robertAngst
>I'm not sure when programmers became so averse to writing simple code.

I know everyone likes to spend a day writing a date/time library, but when you
need to develop a full stack app. You just dont have time to do everything
from scratch.

These are decisions you have to make.

~~~
paulddraper
> date/time library

> leftpad

Yes, there is a difference between these.

~~~
dsr_
There's also a difference between "we have a dependency on this versioned
library that we have archived and deployed" and "we have a dependency on this
library which we refer to dynamically at run-time".

But the second one gets you closer to the mythical perfection called
"serverless" because it uses so many different servers you can't keep track of
them all.

------
1996
The list of suggestions do not match the headlines.

Introducing new technologies to replace old ones, preferring services to local
hosting - I just do not agree!

I use the oldest technology I can get away with, and host everything on
dedicated servers as it is often more performant and cheaper.

Simplicity needs bounds. The suggestion of reuse also needs that: If you end
up storing binary content on SQL to avoid having to roll a file hosting, you
are taking the simplicity mantra too far.

~~~
whitepoplar
I really enjoy old, stable tools that are maintained and made incrementally
better, day by day. Postgres is one example, Elixir is another (which doesn't
re-invent the Erlang VM, but builds on top of it).

I also really like hosted cloud services, but they need to be replaceable with
little to no effort. For instance, I don't want a new NoSQL platform, I want
high-availability, managed, on-the-fly-scalable Postgres.

~~~
abraae
Unfashionable I know but I feel strongly like this about Java.

We chose Java maybe 2 decades ago when it was newly minted.

Some decent swathes of code written back then are still running in production.

Its enormously satisfying looking at anything in your stack which has really
stood the test of time.

I harbour similar feelings about the Oracle DB but I'm a little more
conflicted there.

------
pdog
Looking at the SR-71, despite all its engineering "flaws" (documented
excellently in a sibling comment:
[https://news.ycombinator.com/item?id=17675996](https://news.ycombinator.com/item?id=17675996)),
I wonder...

Have we really progressed technologically, as in, _doing the impossible_ ,
since the late 1960s?

The Blackbird is still the fastest jet in the world more than fifty years
later.

I'm not even sure we could return to the Moon successfully on the first try if
we tried.

~~~
skybrian
Making a plane faster is only one dimension for optimization and it's not
actually a goal in itself. It sacrifices other goals. Nobody has done better
because they are optimizing for something else.

The purpose of the SR-71 was reconnaissance. Do you really think the U.S.
military hasn't gotten better at that, using drones and/or satellites?

~~~
varjag
SR-71 was meant to be flying over USSR and China. Few existing USAF drones
would have survived 20 minutes there even in 1960s.

But sat reconnaissance have certainly got a lot better to the point of
rendering the bird obsolete.

------
skybrian
It's interesting how often good-sounding advice is contradictory.

The article advocates standardization. Using fewer programming languages means
fewer toolchains to maintain and less fragmentation of engineering knowledge.

On the other hand, the frequent advice that we should "use the correct tool
for the job" suggests being familiar with a lot of different, specialist
tools. More programming languages?

I guess it all depends. Knowing when to make a move versus when to do the
opposite comes down to taste and experience.

------
mark_l_watson
I really like managed app hosting like AppEngine and Heroku. I have spent
decades doing what we now call devops, or ‘you build it then you own it.’

Now I like to save my time for other things. AppEngine wins on cost while
Heroku wins in that if you use standard things like Postgres, Kafka, etc. it
is not too bad moving to your own dedicated services.

Edit: I also like the article’s advice on minimizing the number of components
in systems. Using Postgres as a ‘Swiss Army Knife’ for information storage is
a good start.

------
stockkid
> Use services. Software that you install is software that you have to
> operate.

On the other hand, a software that you use as a service is a software you do
not control. Therefore whether to use services depends on how mission critical
the software/functionality is to the whole system.

------
hirundo
If you remove everything that doesn't break the widget then everything left is
functional. Pursing minimalism equals pursuing function. Therefore engineering
equals applying minimalism to technology.

~~~
nine_k
That "doesn't break" needs an "under which conditions" clause. A lot of
complexity arises from having to handle the imperfections of the environment.

But I agree with the general idea. Good design usually arises after a worse,
messy and more complex early design is produced, analyzed and _simplified
away_ , often radically. Coming to simplicity by a shorter route is very rare.

~~~
chrisweekly
Yes. "Perfection is achieved, not when there is nothing left to add, but when
there is nothing left to take away." \- St Aubrey(?)

~~~
nine_k
Antoine de St Exupery; quoted in the article :)

------
mLuby
Fantastic suggestions.

Not only does having fewer technologies in your stack mean there're fewer to
maintain, it also means they're used more so your devs are more expert at
them.

------
contingencies
Some good architecture related quotes here. Added to
[http://github.com/globalcitizen/taoup](http://github.com/globalcitizen/taoup)

------
tharakam
"It seems that perfection is attained not when there is nothing more to add,
but when there is nothing more to remove." \- Antoine de Saint

~~~
jonahx
Though less literally accurate, I prefer the translation:

> Perfection is achieved, not when there is nothing more to add, but when
> there is nothing left to take away.

------
edpichler
Great article. I always tell to my clients: Software projects are cheap, the
expensive is the never endless Maintenance.

