
Are Containers the Right Answer to the Wrong Question? - bdcravens
http://open.citrix.com/blog/328-are-containers-the-right-answer-to-the-wrong-question.html
======
phkamp
Before Linux adopted "Containers", jails were invented in FreeBSD (by yours
truly) and they ran Linux Apps just fine, thankyou.

And SVR4 and SCO binaries too for that matter.

The idea that your kernel can only give you one syscall-API is so 1990-ish.

~~~
justincormack
Linux was simpler to emulate back then, it has grown in complexity and the
emulation has not kept up, although it does work for some use cases. Having
worked on testing for the NetBSD Linux emulation (its getting better, didnt
help having no tests) there is quite a lot missing (epoll for example is not
yet emulated). FreeBSD still appears to only have 32 bit emulation which rules
out most modern Linux apps.

~~~
thejdude
What is it that makes "most modern Linux apps" not run on 32bit? Last I
checked, pretty much all packages were available for both x86 and x64.

If you're talking enterprise use, sure, some app servers might need more than
a few GB, but that's a memory problem, not strictly a bitness problem.

~~~
MichaelGG
Well the idea of using mmap for large areas to simplify development can mean
32bit isn't usable even if the program doesn't need that much memory. It's
about the address space in that case.

One program would spin up lots of threads, each with a 1MB stack, which was
hardly used. It'd die on 32bit even with no real memory pressure.

Also, 32bit is less likely to get as much testing for server apps.

------
ChuckMcM
Interesting article mostly because it tells me that Containers are clearly
getting traction (Citrix sees them as a threat to their business, so they
declare they are not a threat to their business.)

That said, there are much deeper questions here than 'container good or bad?'
Not surprisingly, they are nearly the same as 'compilers good or bad?' that
were all the rage before.

So hypervisors and containers are the same two systems, hypervisors abstract
to the hardware level and containers abstract to the OS level. By setting the
level of abstraction they control what can be shared and what can't, and what
differences can be accommodated in the same system.

At scale, efficiency is associative. Which is to say that you can be more
efficient in your use of resources (higher utilization) when you associate
similar abstractions together. You do that by maximizing the commonality. So
if the only common thing is 'runs on x86 hardware' that is your common point.
If its 'runs on Linux' (or Windows, or Solaris) as a common point then that
you get some efficiency by running one copy of the OS and multiple processes
in that OS. If the common point is 'Rails' or 'Go' then you can gain efficency
by keeping hardware, OS, and software stack in common.

The further up the abstraction level you go, it loosens constraints on the
bottom layers while increases them on the top layers (you have to write all
your code in Rails, but you can run on x86-64 or ARM-64 architecture machines,
versus you can write you code in any language/system that targets x86-64
hardware) Different organizations can dial that knob based on their ability to
be flexible. So organizations with a lot of legacy binaries targeting x86-64
they really need hypervisors, but organizations where all of their code base
targets Linux, they are much better off with containers. At a small scale the
numbers are a toss up, at Google/Walmart scale the numbers become company
controlling.

------
KaiserPro
And so the circle is complete:

TL;DR: this is timeshared mainframes all over again. I'll dust off my pipe and
white technician's coat.

In the beginning there were batch run mainframes, and it was good.

And Lo!, did the bean counters notice thy ever under utilized CPU cycles. So
became time sharing mainframes.

The masses rejected the cold noisy cave of the server room and so it was
decreed that the terminal shall forever more be remote.

Thy ever indiscriminate shrink ray doth break the bonds of the mainframe, and
set forth the world into mini computers. "Hark!" said the masses; we demand
the Personal computer.

The masses were mollified, until the becoming of cheap internets. "But wait!?"
they cried we require some sort of network computer.

Time passed, and a tower of babel was conquered by XML, JSON and eventually
the cloud.

~~~
mp8
Hate to be "that guy", but that should read "And Lo! ..."

[http://english.stackexchange.com/questions/38741/use-of-
and-...](http://english.stackexchange.com/questions/38741/use-of-and-lo-in-a-
sentence)

~~~
KaiserPro
much obliged.

------
ithkuil
Virtualization is very good at handling consolidation of legacy systems. Which
means that your 10 old servers, running various unix and windows versions and
hardware whose spare parts are difficult to find, can be easily replaced by a
single modern machine, usually not even particularly beefy by modern
standards.

It solves a lot of problems in small to medium companies; it's an answer to a
desperate call for help of sysadmins too tired to hunt 15 years old cpu fans
on ebay, and managers hearing always the same infernal stories about what it's
like to do anything down in the basement.

It solves the problem of the same small to medium companies having to buy new
hardware for each application they buy, usually because the vendor tells them
to do so. Many software started to get shipped as a whole OS stack, because
that was easier for the vendor to support.

It solves many other problems, but let's stop here. These problems are going
to stay, and so will virtualization.

However containers offer a way to isolate your environment so that they can
survive longer and coexist better with other apps. If you build your
environments using those building blocks, then after 10 years, you'll have a
slightly smaller mess (which you can still handle with virtualization).

If you really have a lot of machines then, you need to have a good system for
managing your hardware, for bulk installation etc. You have to manage new
kernel and container management software rollouts in more or less the same way
you'd need to manage hypervisor rollouts.

But since you have a lot machines, which means that you encounter hardware
failure more often, the relatively higher frequency of kernel and container
management software updates has still a smaller cost than virtualization
overhead (provided you have enough machines).

Or put in another way, virtualization overhead is negligible if you're a small
fish. It's cheaper for you to buy more machines or a stronger machines if all
you need is to cover for that overhead.

~~~
KaiserPro
As someone who battles legacy at scale everyday, Its rarely the platform that
is the trouble.

For example, I have a nasty little calendar script that I have to move. The
rest of the app is long dead, but this one little worm keeps on biting.

It uses regex to parse XML. (insert cuthulu joke here) its based on php5.old I
need to move it to 5.new+secure. The OS has an update function that is know,
tested, documented and works. I can install a new RPM/deb and be fairly sure
it'll work if it installs

Standalone apps, in the way that people are building them wont. Conatainers
rely on the underlying operating system for various things. Any exceptions are
bundled with the app.

a true VM is free to be moved to a new host, a container is not. You are
hamstrung to that machine and possibly that config.

So If you built the farm on docker, you're stuck with it, on that hardware.
unless you want down time, or was to re-engineer things

~~~
ithkuil
> Standalone apps, in the way that people are building them wont. Conatainers
> rely on the underlying operating system for various things. Any exceptions
> are bundled with the app.

I don't understand. I assume that for "underlying operating system" you mean
the OS that hosts containers. Unless you depend on some OS specific things
like iptables or stuff like that, it should be trivial to rebuild a container
(e.g. make a new docker image with a new php) and test it.

Containers are not so great if you migrate into container software that has
not been packaged for containers in the first place. It's not that it won't
work, but it's easy to find example where you could run into troubles.

The whole point of solutions like docker is to give you the consistent
environment that will help you to test your changes in isolation.

> [...] The OS has an update function that is know, tested, documented and
> works. I can install a new RPM/deb and be fairly sure it'll work if it
> installs

Until you have a distro that's too out of date for that to work properly. The
same thing happens if you use containers with too much of an "OS" in it. There
is currently a lot of debate about what's the right balance between
microcontainers or full fledged mini-OS inside a container. On one end of the
spectrum you move your problem to container orchestration, while on the other
you risk breaking the isolation abstraction provided by the container by
running an OS that doesn't have full control on it's environment (e.g. kernel,
disks etc) and thus breaking the guarantee that existing upgrade tooling will
work flawlessly.

> a true VM is free to be moved to a new host, a container is not. You are
> hamstrung to that machine and possibly that config. > So If you built the
> farm on docker, you're stuck with it, on that hardware. unless you want down
> time, or was to re-engineer things

That's true, only a VM will guarantee that an arbitrary legacy application can
be moved to a different physical machine in face of arbitrary incompatible
developments in the underlying hardware. Well, if you want to stretch that to
the extreme, only a full emulator can guarantee that you will be able to run
your legacy things on a completely different CPU architecture.

However, you must be aware that VM offer you a way to acquire a technical
debt, which is great in that it allows you to decouple your immediate hardware
upgrade needs from your software reengineering, but you must be aware that
you'd have to pay off this debt.

I have a 15 years old mandrake linux VM with some legacy software. It's well
known, tested and documented update system won't help me maintain the mess
that's inside that VM.

------
rwmj
Also, containers don't do live migration. If you can transparently migrate VMs
around your datacenter then you can consolidate VMs onto a few hosts to save
electricity at night, spread them out during the day to maximize performance,
or move them for scheduled hardware upgrades.

~~~
amirmc
I know that live migration is possible but I'm curious how many people
actually do this. I hear much more about spawning/culling VMs quickly, than I
do about live migration.

~~~
spydum
Pretty much all day on busy clusters. VMware esx with DRS does this to
rebalance resources.

------
sturadnidge
Wherein the author argues that requiring a single OS kernel for multiple
containers is somehow more restrictive than MirageOS, which requires every
application to be written in OCaml?

The overall concept is not without merit, the OSv site
([http://osv.io/resources/](http://osv.io/resources/)) has some more
interesting material. I'll still be sticking with Docker though.

~~~
mackwic
What's the big deal with OCaml ?

I found it to be a very nice, powerful, and enjoyable language.

~~~
vidarh
"The big deal with OCaml" is that most people have tons of code that isn't
written in OCaml.

Projects like MirageOS and ErlangOnXen are fantastic and I hope we see more of
them, but getting to the point where these kinds of runtimes are viable for
most people is a _long_ road.

~~~
amirmc
They are viable for people starting afresh. Not everything that comes along
_has_ to care about legacy systems so the advantage goes to those creating new
tools/products/services with the advanced approaches. In fact, I'd argue that
such step changes wouldn't be possible if we always had to be compatible. This
is pretty much 'circle of life' stuff.

------
jgrodziski
I think Unikernels are the next steps after the lightweight containers trend,
in short an execution environment free of all the unused clutter an OS brings
for just executing one application.
[http://t.co/JRxG2RGRgp](http://t.co/JRxG2RGRgp) What do you think of this?

~~~
amirmc
I completely agree and the workflow does away with the need for a vast number
of configuration management tools too. Also, John Carmack also thinks they're
"really interesting" [1] (the link in his tweet takes you to the academic
paper). If you've used Unikernels for anything, I'd _really_ like to know
about your experience. Please do get in touch with me (see profile).

Side note: Why use a Twitter short URL when the original [2] is pretty
succinct anyway?

[1]
[https://twitter.com/ID_AA_Carmack/status/476046680698003458](https://twitter.com/ID_AA_Carmack/status/476046680698003458)

[2]
[http://queue.acm.org/detail.cfm?id=2566628](http://queue.acm.org/detail.cfm?id=2566628)

~~~
jgrodziski
Hi Amir,

Thanks for your reply, don't have time to invest that much in unikernel, I
just keep an eye on it. But the use case is very appealing to me and to you
according to your writings on your blog :) I'll keep in touch if I investigate
further the Unikernel way. Thanks for the John Carmack's tweet, very
interesting, and a validation of our intuition by one of the greatest
programmers :)

Side note : I sent the comment from my smartphone and the quickest way to
retrieve the link to the ACM article on Unikernel is by browsing my twitter
feed :) hence the twitter short URL

------
pointernil
I think "Erlang on Xen" is as well a nice example pointing in this direction.

In this case it is not a application though. It is a runtime environment...
something in between a whole VM and a container?

[http://erlangonxen.org/](http://erlangonxen.org/)

------
thejdude
If you're one of the 0.01% whose app cannot run on Linux, or needs a very
specific patched version of the kernel to run, then sure, I agree.

Who knows, maybe Citrix has something to sell that makes the author's opinion
very "important"?

------
dschiptsov
Let's say that unless you are mega-provider, which trying to pack as many as
possible those cheap low-resources (10 pages, 3 forms, 5 tables, 10 requests
per day Rails or whatever is current buzzword) site into one piece of leased
hardware the visualization is a wrong question.

Just apache's virtual hosts isn't good enough anymore, customers want Rails,
Nodejs, Java for their $20/mo.

Virtualization for real-world production is nonsense, especially those idiotic
setups like Java under Linux under VMWare under Windows (you wouldn't believe
how many "professionals" are cock-sure that this kind of setup is "robust" and
most stable.)

Jokes aside, Docker is eating VMWare's lunch being a fresh new meme promising
miracles (yeah, problem of cache trashing and context switching for I/O in a
host OS will surely be solved by someone else.

------
nwalfield
This post is disingenuous. In particular:

    
    
      [Cloud Operating Systems] enable the vision of light and fast application servers, while preserving the superior security and flexibility of true hypervisors.
    

Hypervisors are only secure if you don't need any sharing. If you have two
applications running as separate guests and they need to exchange data, then
all of the security issues that you'd have on a normal OS return, but they are
harder to solve, because the Hypervisor doesn't provide much support!

As an extreme case, consider running all of your applications on your Desktop
in their own VM. Now, you are in your web browser and you download a Word
document that you want to edit. Somehow, you have to transfer this document to
the Word Processor VM to edit it (they have different file systems so you need
to use something like scp). If, after you're done editing it, you need to send
it to someone via email, then you'll have to copy it to the Email VM. It's not
how to see how this seriously impacts usability!

When sharing is required, then a Hypervisor is not helpful. That's not to say
that Hypervisors are useless, as the author claims his opponents say.
Hypervisors are good for running different operating systems on the same
hardware and they are good at isolating the different OSes from each other.
However, once any sharing is required, hypervisors are not the right tools for
the job.

~~~
amirmc
I'm confused. What you're describing seems equivalent to just connecting a
machine to the internet/network (ie you have security concerns). How many
scenarios are there where you have two guests _on the same host_ that must
communicate? Shouldn't those applications/VMs be architected in a way that you
could simply move one of them to a _different host_ without breaking? I guess
I'm confused about what you mean by 'sharing'.

Edit: As for your extreme example, I believe Bromium (ex-XenSource founders)
is doing what you describe.

[http://www.bromium.com/products.html](http://www.bromium.com/products.html)

