
Give Me Bare-Metal - jessaustin
http://www.subbu.org/blog/2014/11/give-me-bare-metal
======
programminggeek
The virtualization and containerization of computing is a long term net win,
but there isn't a lot of huge benefits financially or otherwise for the scale
most companies operate at.

For example, I witnessed a co-worker spend days messing around with docker to
host incredibly simple PHP/MySQL and Ruby apps. Instead of renting a few VM's
on say Digital Ocean or Amazon and spending probably $50 a month, my co-worker
spent at least 50-100 hours of time to put everything on Docker and Puppet.

Let's say the net cost of that orchestration was around $10,000 in terms of
billable time. You could rent a separate VM for every project and still come
out ahead without trying very hard and still spend less time doing the
project.

The total cost on that was way out of whack AND docker continued to be a
source of bugs and ongoing overhead costs in terms of time.

My point is that these tools are great when they save you time, energy, and
money, but a lot of people are overinvesting in tools like docker too early
and they will never get a good ROI on their investment.

I don't know at what scale these tools start to make a lot of sense, but I
imagine it is somewhere over 20 or 30 machines or projects before
containerization shows itself to be a good investment. I'd be curious to see
someone run the numbers on that.

~~~
viraptor
> You could rent a separate VM for every project and still come out ahead
> without trying very hard and still spend less time doing the project.

Doing the project? Yes. Running it - no!

These tools make sense at the scale of a single project with two hosts in my
opinion. Here's what happens in my experience when developer with little ops
experience takes care of a few VMs with small apps: (so a majority of services
out there)

\- where are the logs and what's the available space for them? depends on
which VM, some don't log at all, some log debug because they had some work
done weeks ago

\- which hosts are running which app version and what's the upgrade procedure?
likely every one has different manual steps

\- what's your system/libraries/application upgrade story? likely missing /
different for each host

\- what's your DR story? likely missing

\- what's the state of monitoring and are all hosts covered to the same
extent? unlikely

\- is the development and test environment at all similar to production?
unlikely

For each of those items you're likely to go back and introduce something that
docker/puppet would've either made easier or provided from the beginning.
Knowing that you can consistently build the same host/app every time and that
it's got the same options, utilities, tools, etc. available is a huge gain
even on 2 VMs.

Yes, the huge benefits start at 1X hosts, but I've seen the benefits and I'm
never deploying even one production machine without automation anymore. (salt
in my case though) Not doing so is actually more work in a long run. And as
soon as you deployed one app that way, you have your library of scripts -
another app will likely look very similar and anything around it (monitoring,
logging, backups, database deployments, ...) will likely be exactly the same.

So I think that you either pay that 10k for orchestration from the start, or
you end up paying it bit by bit, every time something needs to be added or
goes wrong.

~~~
KaiserPro
Ah, I think you're missing the point.

What the OP was saying is that spending all that time to make _docker_ work
was a bit pointless. Especially as a VM makes no real difference to the end
user at this stage.

Your points about DR and the like are valid, however if you are going to use
puppet, the difference between deploying n docker instances (and keeping the
state in the correct way) and VM instances are negligible. (yes, yes, docker
is a chroot, so inherits the host's state, but you'll still need to control
the docker environment.)

I use cgroups everyday, to avoid memory exhaustion, (25k CPU render farm) and
for that its grand. Running it on a public host? madness I say. For things
that require high IO(even then its questionable, AUFS...), or low security
perhaps.

~~~
vidarh
> What the OP was saying is that spending all that time to make docker work
> was a bit pointless. Especially as a VM makes no real difference to the end
> user at this stage.

I disagree. Given that you operate a 25k CPU render farm, perhaps you are no
longer aware of how poorly devops tends to be managed in small shops. I'd
argue Docker buys you _more_ in those small environments, because in large
environments you can build tools that works for your environment. In small
environment, the traditional "solution" has been to brush these concerns under
the carpet.

Just getting repeatable builds of an environment so that you can ensure the
dev, test, staging and live environments continue to bear a resemblance to
each other is a massive improvement for many of this type of setups. That is,
assuming that there _are_ separate dev, test, staging and live environments -
often you'll find all of it on a single server. Yes, really.

Basically, people that think like OP terrify me, because too often we've taken
over operations of applications from people who think like that, to find that
the operational cost is staggering. We have one customer, for example, that
have over the last few years easily paid several times what it would cost to
properly re-engineer their application from scratch in time spent for us on
operational issues because they've refused to accept that it's necessary.
(EDIT: Said customer pays for this time, so on paper it works out well for us;
in practice it's bad for us as it leads to lots of tension with the customer
over all the issues, and we'd much rather have spent that time making their
app better, and help them grow their revenue from it - long term that'd be far
better for our customer relationship). This is a "simple PHP and MySQL"
application.

VMs aren't really a solution by themselves. It's not app-level isolation that
"makes" Docker. It's the entire structure that encourages repeatable builds, a
repository, and the light weight nature that encourages breaking things into
smaller components. I agree that just Docker as isolation in a multi-tenant
system is still scary.

I'm currently working on deploying an application in VMs. With Docker in each
VM. The VMs gives us the isolation we need in a multi-tenant system. Docker
lets us ensure that dependencies between each component is well documented (or
the components won't have access...), and that we have a decent image
repository, and that we have ways of managing the entire lifecycle of each
component. The two are complementary technologies.

In a large enough environment you could solve the issues that Docker solves
with process. In a small environment that process is usually non-existent or
weak.

> AUFS

AUFS is not necessary. Btrfs or device mapper backends have been supported for
a long time. And even then, for any data volums that needs high throughput
you'd use a bind mount from the host.

------
DigitalJack
Bare metal doesn't mean what it used to.

~~~
gaius
You know, Unix has a pretty good abstraction already for managing resources: a
process.

Nowadays it is fashionable to deploy applications consisting of hundreds of
files in a complex structure, but this is fashion and not technology and
that's easy to demonstrate: in the old days, a C application consisted of
hundreds of files in a complex structure, that would compile into one self-
contained executable. There's no technical reason you couldn't take a RoR app
(for example) and "compile" it and manage it just like any other process.
Start it, stop it, move it around, one binary executable and one plain text
file for configuration, that's really all you need for _any_ application.
Really.

I look at things like Docker and think, you are solving a completely
artificial problem, and as soon as fashion swings the other way (10 years at
most) people will wonder why people ever did this.

~~~
vidarh
> You know, Unix has a pretty good abstraction already for managing resources:
> a process.

And that's what Docker is built on: processes + namespaces to restrict what
each process group can access.

> I look at things like Docker and think, you are solving a completely
> artificial problem, and as soon as fashion swings the other way (10 years at
> most) people will wonder why people ever did this.

Things like Docker will still have value with single-executable containers.
Problems that people have had for years. Even with single-executables.

You still need to manage privileges. You still need service discovery. You
still need to manage the network. You still need to manage data volumes. You
still need to manage the environment.

Docker provides mechanisms that either solves all of that, or provides
information for tools to hook into (e.g. ability to query what ports the
currently running containers expose).

I agree with you that there is distinct benefit in self-contained executables,
and I think you'll find a _lot_ of people working with Docker thinks the same.
Partly this is part of what Docker gives us: We get to treat a bigger hairball
of an environment pretty much like a single entity, even if we're not able to
slim it down.

But partly, note that a lot of the adoption of Go within devops tools appears
to at least partially be motivated by the simplicity of distribution and
installation from getting static binaries (without having to use C). I prefer
Ruby, and find Go to be a ugly, too low level, language, but I definitively
understand the appeal Go has when it comes to simplicity.

Yet despite having those devops tools as single executables, people want to
put them in containers to manage them properly.

If anything, I hope and think that things will move even further down this
route. For my part, I'm containerising my desktop environment part by part.
There's no reason we should not move to far more locked down process
environments.

~~~
custardcream
Couple of points/questions, at least from my perspective as a man of many hats
to anyone passing by the above comment:

1\. Docker appears to only allow a single entry point per container. What do I
do if I want to run a container consisting of say postfix and dovecot
together? If this is possible can someone please point me at it.

2\. Go doesn't allow you to bundle resources with the executables (as say
Java/C#) so you have to rely on 3rd party libraries to do this that aren't
very good. I can't easily ship my templates with my web server for example as
part of the "go build" process. Again if I'm wrong, please point me at a
solution.

~~~
vidarh
Personally I wouldn't run Postfix and Dovecot together under Docker. Instead,
I'd export a volume for the mail store, and share that volume between a
Postfix and Dovecot container.

But if you absolutely want to put them in the same container, the solution is
to run a process manager as the entrypoint. E.g. base it on something like
baseimage-docker [1]

Re: 2. I don't use Go, so I can't answer that part.

[1] [http://phusion.github.io/baseimage-
docker/](http://phusion.github.io/baseimage-docker/)

~~~
custardcream
Thanks for the info. I'd have a couple of problems with exporting the volume
because I need postfix to talk to a Unix domain socket that dovecot listens on
for SASL auth. I'll go with the process manager solution - thanks for your
suggestion!

Been trying to build a canned mail server for a few years so this is probably
a nice solution if I can get my head around it.

~~~
myhf
Unix domain sockets work in docker shared volumes. A common use-case is to
mount /dev/log into a container so that it doesn't have to run its own
syslogd.

------
voidz
Docker is written by developers of which there are too many. But it sucks to
many of us who are sysadmins. Docker's networking isn't all that, and the
storage is basically just nifty because of already-existent tech like AFS and
BTRFS.

I think docker is nothing more than a hype from a new generation who wants
everything hip, simple, fun, fantastic and awesuummm and the older and more
experienced sysadmins like me have seen this coming years ago. It was
inevitable, and now it's just a matter of waiting until the lazy generation
finds out what we've been seeing all along: Docker is an abstraction of other
technologies and you don't need Docker at all.

~~~
matthewmacleod
That's stupid. Developers want the ability to deploy applications easily, and
keep them stateless. The tools to do that are of course available, and Docker
builds on top of them to provide a much more effective interface than rolling
an entire infrastructure from scratch each time. What the hell is wrong with
that?

~~~
gaius
No, he has a point. Why are applications hard to deploy and manage? Because of
how they are structured. If deployability was factored in at the time
applications were being designed, they would look very different. You know
there were entire 1000-page books written on how to deploy RoR applications?
Why was it ever so complicated? Any why is creating a problem that never
existed before, then solving it, considered a good use of anyone's time?

~~~
matthewmacleod
It is complex to deploy applications because applications are complex. They
have dependencies, they require reproducible environments. We want them to be
scalable, so we can easily fire up more instances as required.

Hand-waving this away as a problem that "never existed before" is nonsense.
And increase in individual machine power and the demands of applications means
that we now have—quite validly—more complex applications, and more of them
running on each machine. Administration of this is objectively difficult, and
Docker is one of the tools that will help to make it easier.

~~~
voidz
But you are not addressing what I am saying at all.

I am saying that this is convenient for developers, but for us sysadmins, who
have to implement Docker apps in already existent systems, it is just not all
that convenient, so now we have a surplus of happy developers.... and the
sysadmin "should just stop being so grumpy".

Sysadmins are expected to cheer in the same happy way, but in my opinion, many
of us already knew how to build the kick ass systems /without/
containerization that the new developers ("devops"... yuch) are so cheery
about.

~~~
matthewmacleod
_Sysadmins are expected to cheer in the same happy way, but in my opinion,
many of us already knew how to build the kick ass systems /without/
containerization that the new developers ("devops"... yuch) are so cheery
about._

I don't think this is the case.

My goal as a developer is to be able to deploy applications quickly and
easily, to be able to scale them, and to avoid worrying about dependencies.
That's all good practice.

I still need a sysadmin to handle that infrastructure, so Docker et. al. are
not a tool to replace sysadmins.

That being the case, why do you think there is pressure from developers to
adopt these tools? If sysadmins are generally capable of building 'kick ass
systems' that offer the same features but without using Docker, then why
aren't developers happy with the situation?

Docker is a tool that allows sysadmins to provide these useful features
without having to arse about building them by hand. It's a tool for sysadmins,
and dismissing it out-of-hand in the way you do is totally stupid.

~~~
vacri
I'm a sysadmin in a place that's bringing docker on-board, and I'm finding
that I'm having to write more stuff for docker than docker is solving, from an
ops point of view. Developers don't care about where logs go or whether they
persist. Or about managing SSL certificates (don't want those in the repo). Or
starting/stopping things programmatically. Or a number of other things that
the guys who have to make this stuff work in production have to care about.
Docker is lovely for devs who can coddle their containers personally, but it's
an entire new ecosystem with its own can of worms for ops people.

An example: I can't easily rename containers, so if I want to install a new
container with a set name for programmatic use, I have to destroy the old
container, so there's no quick rollback. Instead I have to have a tag system
for the docker images, and track those instead, keeping aware of which current
unique name is 'old prod'.

Another example: docker doesn't really support having one repo support two
clients with their own config, not without a lot of faffing about. You get one
dockerfile, it is always at the root of its tree, and it only builds one
artifact.

Another example: until you set up your own base images and repos, you're
taking an app that used to be a few megabytes, and turning it into an image
that is hundreds of megabytes in size. We have an nginx image that uses a
stock docker base, and it's over 400MB! For nginx! That takes a non-trivial
amount of time to download on a base image update, a new dev machine, a newly
provisioned machine... if the docker hub happens to be up and not slow (we had
a couple of issues there recently). And if you do have your own base images,
you now have an extra operating system image that you need to keep up-to-date
with security patches.

Docker is nice and will 'get there' eventually, but in its current state it
benefits devs at the cost of sysadmins. I would counter that dismissing
sysadmin concerns out-of-hand in the way you are doing is totally stupid.

~~~
voidz
Thank you for making my point. English is not my first language, but this is
exactly what I meant, and your examples are spot on, too.

One other thing that is annoying is networking. IPv6 is once again being
ignored, and for the rest they offer either a simple NAT bridge (and who wants
to NAT anyway?), or you're on your own for the stuff you cannot proxy with for
example nginx.

I'm hoping that this will really improve because tools do exist already.
Openvswitch is one of them.

(It's also ironic to witness in this discussion how matthewmacleod even helps
prove my point - developers are usually not really aware of what sysadmins do,
although the other way around, we know what devs are doing, that's for sure.)

------
bluedino
I think a lot of people would be surprised at how many users and traffic you
can really handle on a single server, or single pair of servers (web/DB).

~~~
chubot
Well yes, because a lot of hosting companies claim 100+ customer VMs per
machine. Pretty sure Linode and Digital Ocean run somewhere in that
neighborhood.

So if you just run on a beefy bare metal server yourself, you can probably
serve 50-100x as much traffic as you could on a typical VM. I guess the issue
is that your software is probably not written to make use of the full machine.
You might have to start 50 copies of a process, which might be OK. Ruby,
node.js, and Python will probably neat that treatment.

------
akerl_
I'm not sure where folks decided there was a competition between containers
and VMs. They solve very different needs.

> If I’m a front-end app developer, it’s much more convenient for me to use a
> container-based cluster management solution on bare-metal without dealing
> with VMs.

If you're spinning up containers on bare metal, you need to manage bare metal
systems, or pay someone to do it for you. Or, you could put your containers in
VMs, where you manage the VMs or pay somebody to do it for you.

Bare metal gives you some advantages (like tighter control over the components
and the ability to customize the hardware to suit up-front needs) and some
disadvantages (harder to dynamically scale, spin up/down times are longer),
and VMs have their own advantages and disadvantages.

VMs are here to stay, just like dedicated servers are here to stay. Containers
change what we do inside of our systems.

