
Anatomy of a Modern Production Stack - jbeda
http://www.eightypercent.net/post/layers-in-the-stack.html
======
gcr
This is an interesting portrait of _container_ -based production stacks, but
I'm sure most companies aren't using containers.

This also doesn't mention anything about the application you're serving: HTTP
servers, databases; nor does it mention the vehicle you use to push changes to
your application (source control? ssh to host? ssh to each guest container,
good heavens?), control distribution upgrades (how would you upgrade your
kernel? how would you upgrade apache?), and so on.

What do other kinds of stacks that aren't on the "Container Bandwagon" look
like? Just "ordinary" provisioning of virtual machines?

With so many problems, I don't think "Container" can be considered as
equivalent to "Modern" quite yet.

~~~
squiguy7
I'm no expert, but I would imagine environments without containers probably
use some sort of provisioning software like Chef or Ansible. If your
application is big enough, you cannot manually deploy to each one as you
described.

This article seems more like a container-centric description since there is no
mention of build servers, applications, databases, etc.

~~~
gcr
Exactly. It would be really fun to explore those facets of a production
pipeline.

I'm sure that's been written before, but I can't think of anything off the top
of my head...

------
jryan49
It's articles like this that make me feel like containers are the new Java EE,
factory-factory-builders, dependency injection, etc. I mean all this makes
sense in some context, but whatever happened to right tool for the right job?
Just having all of these things for no reason doesn't make you "modern".

~~~
est
in other words, how to emulate badly working scripting language in a XML based
configuration container for a web project, yo know, with flexibility.

------
choppaface
This is definitely a visionary post and the title is a little misleading. A
more useful title might be: A Vision for a Container-based Production Stack
(based on experiences at Google). I think the post covers the scope of
container-related problems very well, but I definitely know of a lot of
production stacks that work well and don't look at all like the post describes
:)

One consideration missing from the post: one day we could very well have a ton
of opaque/sealed/closed-source container images provided as SaaS. For example,
Oracle could provide their products in containers which you would then use on
your own hardware or your own cloud provider. This sort of product would give
CIOs a bit more control over OPEX/CAPEX and security.

The current post assumes a single central registry. IMO it's more likely
containers drive a distributed ecosystem of registries, and container-oriented
stacks will need to deal with this.

~~~
jbeda
[Author here]

I'd _love_ to see a decentralized image naming and distribution system. I'm
pushing hard for this to happen as part of the OCI. See
[https://groups.google.com/a/opencontainers.org/forum/#!topic...](https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/OqnUp4jOacs).

------
draw_down
I noticed how much of this focuses on containers and the problems that follow
specifically from using containers. I thought the reason for using containers
was because it makes shit simple.

~~~
technion
I'm always thinking about a Shellshock type scenario.

Instead of "use x tool to upgrade bash on every server", it becomes "upgrade
bash on every server", followed by "rebuild each of my several types of
containers" then "redeploy every single container".

~~~
aiiane
Ideally your containers don't depend on bash, and instead are just running
their app's process. So then any given scenario is generally only _either_
upgrade bash on every server (if the problem if lower in the stack than the
container) _or_ rebuild and redeploy every container (if the problem is in a
library used at the app level).

------
lqdc13
It is important to not go overboard with these things. If you are building an
app that probably won't have 10^6 requests a second and won't need to scale to
the moon, this is overkill. All of such things add a lot of extra latency and
are CPU hungry. Logstash and elastic search are big offenders here.

------
therealmarv
what I'm worried about and I think is frightening is that people may start to
think that MODERN==CONTAINER which is wrong. It depends really. Do not use app
containers just for the sake of being modern. If you really want a production
ready stack first inform yourself deeply how to do normal Dev & normal Ops
right, understand the concepts and get to know how to make a single server and
your application bullet proof for production usage before ever thinking of
adding the complexity layer of app containers on top or underneath of this.
Chance is pretty big that you do not need app containers because your use case
is too small for justifying this "modern" layer of abstraction which will give
you also new sort of "modern" problems for sure at some stage. Don't get me
wrong... containers can be great but not everybody needs them.

------
jbeda
Author here.

Lots of comments on the term "modern" and the fact that this is container
based.

While there are obviously other ways to skin this cat, for me, I want the
following out of my production stack:

* Self healing and self managing. If a machine fails, I don't want to have to think about it. The system should just work.

* Supports microservices. The idea of breaking your app into smaller components (regardless of the name) can help you to scale your engineering _organization_ by keeping the dev team for each µs small enough that a 2 pizza team can own it.

* Efficient. I want a stack that doesn't require a lot of hand holding to make sure I'm not wasting a ton of resources.

* Debuggable. Complex applications can be hard to debug. Having good strategies for application specific monitoring and log collection/aggregation can really help to provide insights into the stack.

------
chrismarlow9
Beware the pre-optimization... it's a time suck...

------
joelberman
These articles always remind me of William Gibson, who said "The future is
here, it's just not evenly distributed yet."

I am sure this is in our the future.

------
mcms
I wonder why nobody considers Ceph[[http://ceph.com/](http://ceph.com/)] as a
proper storage system for micro-service stacks. Maybe because it is mostly
considered an infrastructure solution.

Migrating data between hosts and containers for stateful services, data
snapshots and file-system isolation can be easily solved for example with Ceph
RBDs.

------
dkarapetyan
Not even close. Nice whishlist though.

------
gosharplite
Thanks for sharing your knowledge.

