
How does Docker affect energy consumption? - rbanffy
https://arxiv.org/abs/1705.01176
======
raulk
Doesn't this article fail to consider the bigger, practical picture, i.e.
overall resource efficiency/footprint in multi-component architectures?

Containers allow us to condense workloads in a single OS runtime –while
preserving isolation– where otherwise the same workloads would have spanned
multiple machines or VMs, each with its overhead and slack (unused resources).

Example: consider you need to deploy not just a single instance of Wordpress,
Redis, Postgres, etc., but a complex application consisting of many of those
components.

You can either choose to (a) deploy each in a different machine [incurring in
the overhead of OS + unused resources]; (b) in different VMs [incurring in the
cost of OS, but being able to share a resource pool]; or (c) in containers,
sharing the OS and incurring in the overhead of the container daemon.

I would love to see an article that takes these points into account.

~~~
CrLf
> Containers allow us to condense workloads in a single OS runtime –while
> preserving isolation– where otherwise the same workloads would have spanned
> multiple machines or VMs, each with its overhead and slack (unused
> resources).

I'm starting to believe that the increased density provided by
containerization is a myth, in practice. Both because orchestration tools
bring their own overhead (compare all the proxies and filesystem/network
overlays with multiple virtualized kernels) but also because containerization
goes hand-in-hand with microservices thus increasing the number of components
(often times with no real reason other than to be hip).

If you're running 20-30 containers on a beefy VM, you're not really condensing
anything. You're just moving from running hundreds of small VMs on a single
server to running much less larger VMs.

~~~
raulk
> If you're running 20-30 containers on a beefy VM, you're not really
> condensing anything. You're just moving from running hundreds of small VMs
> on a single server to running much less larger VMs.

Of course you are condensing. Those small VMs would have been running a full
OS runtime __each __.

Assuming you relocate those workloads onto a single machine (1 process = 1
container), you no longer run 20-30 copies of an OS, just a single one.

And the container engine takes the place of the hypervisor.

~~~
acdha
> Of course you are condensing. Those small VMs would have been running a full
> OS runtime each.

That's something but probably not as much as you think since hypervisors can
share identical pages (e.g. the Linux kernel) across guests and the base
footprint for a server Linux install is not that high as a percentage of the
private data most applications use. Unless you're running a ton of unnecessary
services on those guests or have an application which uses almost no RAM
you're talking about a fairly modest percentage savings even before you factor
in all of the things you might be running for container management and other
overhead on that side.

The other thing to remember is that this works both ways: containers are great
for being able to upgrade one component independently but that means that e.g.
you might have a dozen different versions of a common shared library because
not all of your containers are using the same base image & version and with
Docker your storage driver might actually force shared libraries to be
duplicated across all processes anyway.

~~~
sdeziel
> since hypervisors can share identical pages (e.g. the Linux kernel)

With ASLR, I'm now sure the gains are that substantial.

~~~
derefr
The parent doesn't mean that the hypervisor will merge identical MMU->physical
page _mappings_ (like a Copy-on-Write process fork would); they mean that VM
pages' underlying host virtual pages literally get periodically hashed for
their current content by a background process on the dom0 and merged when they
are found to have identical hashes. The _underlying virtual page_ is then made
copy-on-write.

Or, to put that another way: the host memory for most modern hypervisors
consists of a heap of "new" pages, and then a generational garbage collector
that moves said pages, if still alive, into a _content-addressible_ "old"
store.

As such, if two VMs each have a process that

1\. calls malloc() 1000 times to get 1000 1-page buffers randomly spaced
through their memory, the mappings different for each VM; and then

2\. uses a fixed PRNG seed to generate random data [but _the same_ random
data] to fill those pages;

then those two processes' pages will _still_ get collapsed together for a 50%
savings.

------
rcarmo
Honestly, I'd very much like to see more deep studies into container overhead
- not just from a kernel standpoint, but actual tit-for-tat measures for
storage and networking overhead that spanned the different filesystem options
(I know that aufs is old hat, but I've used overlay and btrfs too) and,
probably most importantly, the networking overhead.

I've had slowdowns in the order of 30% in terms of requests per second on some
of my stuff inside Swarm as compared to running processes outside containers
(arguably with four or five moving parts and entirely anecdotal, but still
enough to give me pause), and I'd really like to understand how to shave this
yak.

~~~
gtirloni
It really depends on what is being used. There are studies showing the
overhead for containers is minimum, CPU-wise.

However, you might see docker-proxy eating a lot of CPU due to increased
traffic. Or dockerd adding overhead to collect statistics, manage log buffers,
etc.

My point is, pure containers using basic kernel features don't seem to add a
lot of overhead. It's the sugar on top that sometimes is the problem, but that
will vary depending on the container runtime, the app being contained, extra
features that were enabled, etc.

I've seen CouchDB perform just fine in a container when using Docker's
"bridge" network and drop to a halt if using Docker's "host" network. It
seemed to dislike the situation and was doing way more syscalls than usual.
Just to show the app might not like a certain environment.

Even with some overhead, it's a trade-off I'm willing to accept considering
the advantages in the development workflow and managing the infrastructure.
Most of what I've seen are rough edges that will get ironed out with time.

~~~
user5994461
FYI: VMWare is 0% CPU overhead.

CPU instructions are executed as-is. Running on VmWare, ESX, Xen, Docker, LXD
makes no difference.

It's a different story with network and storage access. Impact of
containerization/virtualization is variable(5 to 95% performance drop).

~~~
AstralStorm
Usually true. That said, some instructions and hardware are emulated or
virtualised. (E.g. APIC, clocks, privileged instructions)

The main overhead is in IO though. Closer to 10% with KVM virtio than 95%.

~~~
brianwawok
And this is why friends don't let friends run databases in containers.

~~~
chc
Don't Virtio and devicemapper have different performance characteristics,
though?

~~~
brianwawok
If any of them give near native IO, would love to see. Haven't tested anything
like that yet myself.

------
zxcmx
By eyeballing the graphs it looks like "tl:dr", yes by about 10%, and
sometimes just because a job takes longer.

Another thing to note is that redis and postgres are i/o heavy and many
production environments (esp in aws) will choose not to dockerise i/o heavy
stuff.

I wish they had included a statement like that so I didn't have to stick my
neck out and give an eyeball estimate, but I suppose this kind of statement is
harder to defend and the data they provided was more nuanced.

With that said, power is largely divorced from cost in actual operation except
at extreme scale.

This is because purchaseable and billable units of compute are usually not
utilised to 100% capacity, both in cloud and bare-metal situations. Another
way to say this is, most people essentially prepay more "power budget" than
they actually use.

Since a primary use of docker (esp via kube) is workload consolidation, it's
hard to know what real impact this has on the world.

------
mbertschler
It would be awesome if the comparisons would include tests of Linux in a VM.
Does anyone have such data? (bare Linux vs. Docker vs. VM)

~~~
searchfaster
I did a test of 'Bare metal Linux' vs 'Containers on bare metal linux' for our
product. In this case it is just 2 processes a 'search component' and an
'analytics and logging component'. Under heavy load the 'search' uses a lot of
disk reads, CPU and network, while the logging module uses a lot of disk
writes.

The comparison was done on

1) Ubuntu server 16.04 with both processes running as they usually do (Search
with higher priority)

2) Core OS - Both processes running each in a separate rkt container (search
with higher priority).

I saw no change in CPU / Network / Disk access metrics and my throughput
remained the same.

Please note though, in my case I do not have way too many microservices as the
general usage is. Also I use host networking. I also had no need for
orchestration services like Kubernetes / swarm etc.,

TLDR:; No change between running product in container vs no-container mode
with host networking, minimal containers and no orchestration.

------
nikanj
I wonder what the numbers look like for microservices, or the modern "burn
down and rebuild everything" deployment methods.

Fleet management tools easily create hundreds or thousands of new virtual
machines and run them through a complex npm bootstrap, when all you wanted to
do was edit a file in /etc and restart node.

~~~
icebraining
Fleet management tools usually guide you not to have any files in /etc,
injecting instead the configuration from outside the container, often stored
in Kafka or equivalent. So you shouldn't have to recreate the container image,
just stop them and re-launch.

------
jdoliner
The article mentions that most of the increased energy consumption was due to
the performance of I/O system calls. This was a bit surprising to me since
Docker shouldn't have much of an impact of I/O system calls unless you're
writing to the container's filesystem, which is copy-on-write. The only reason
you should be writing to this filesystem is if you want to incorporate the
data you're using into a later Docker image. For high I/O tests like the ones
they're doing you should use volumes. Unfortunately I didn't see the actual
`docker run` invocations they used to run the tests so I can't know for sure
if this was what they were doing. It's just a suspicion.

~~~
CGamesPlay
Wouldn't the copy-on-write nature of the filesystem mean that _reads_ are
going to be more heavily impacted?

------
StreamBright
I would be interested in the overhead of systemd. Amazon decided to drop
systemd in favor of sysv/upstart init. I guess this is based on energy
consumption but could not find any info online.

~~~
lclarkmichalek
Probably more likely due to the cost of breaking compatibility. It's probably
the most popular rolling release distribution out there, and I imagine a large
part of that is because they'll do anything they can to avoid breaking
compatibility.

~~~
StreamBright
I am not sure. In theory it is easy to convert init.d scripts to systemd
service programatically. I see energy/CPU consumption a much bigger problem
with it. We did some performance testing and systemd was using way much CPU
time than sysv/upstart under similar cirtumstances. I guess the difference is
huge when you apply it to 2M computers (this is how many Amazon used to have)

[https://www.bloomberg.com/news/2014-11-14/5-numbers-that-
ill...](https://www.bloomberg.com/news/2014-11-14/5-numbers-that-illustrate-
the-mind-bending-size-of-amazon-s-cloud.html)

~~~
icebraining
How did you compare the two? Systemd replaces more components than sysv alone,
like rsyslog; was that taken into account?

~~~
onli
Does it matter? If one were to find out that it uses more energy because of
those components, would that change anything regarding the decision of using
it or not because of that higher energy usage?

~~~
icebraining
It matters that you're measuring equivalent things. If systemd replaces three
components, you need to compare its energy consumption with those three
components, not just with sysv/upstart.

~~~
annnnd
Not OP, but if you are measuring the complete system with similar
functionality then the comparison would be fair.

------
supremesaboteur
> We compare the energy consumption of various scenarios run on bare-metal
> Linux—that is, the applications are running on one kernel, without any
> virtualization at all—in contrast to Docker-managed containers, using “off-
> the-shelf” Docker images

Are docker containers at least more power efficient than virtual machines ?

------
edpichler
This surprised me:

> "Results: In all cases, there was a statistically significant (t-test and
> Wilcoxon p<0.05) increase in energy consumption when running tests in
> Docker, mostly due to the performance of I/O system calls."

------
tyingq
I would guess that the process change that usually comes with Docker has more
of an impact.

Since most people adopt the "ephemeral" notion for containers, there's
probably a lot higher rate of wiping and recreating bits, especially during
dev and test.

I assume though, that a new process was designed to create efficiency in some
other area that is more significant than any increase in energy consumption.

------
tmdk
Running multiple containers on a single core or hyper-thread (which seems to
be the primary way that containers are used), will obviously have a lot of
non-cooperative context-switching, and will therefore have a very significant
overhead compared to a solution that takes advantage of cooperative
multitasking and cooperative context-switching.

------
frik
Would be interesting to see a comparison running WordPress, Redis, Postgres,
etc on Linux: BareMetal vs VirtualBox vs VMware vs KVM (Qemu) vs XEN vs
Docker, etc - with CPU, Memory, IO r/w and energy consumption (all measured
from host system) on Intel server hardware with VT enabled. (Optionally also
on AMD, ARM and POWER server hardware.)

------
DamonHD
Note The Register's take on this:

[http://www.theregister.co.uk/2017/05/05/docker_docks_wallets...](http://www.theregister.co.uk/2017/05/05/docker_docks_wallets_with_delays/)

which I posted separately a little earlier.

~~~
positivecomment
They just say that "the energy trade-off between containerization and enabling
TLS/SSL is comparable for PostgreSQL" and quote the lead on this research,
Abram Hindle: "If your expenses are people, then Docker is probably worth the
hit".

------
ram_rar
It would have been more useful if the paper compared Docker vs Virtual
Machines instead of bare metals. It pretty much intuitive that running docker
over bare metal might incur slight overhead in terms of power consumption.

------
marvy
I made no effort to read the paper and went straight for the pretty pictures.
The basic idea seems to be that you can save a few percentage points on your
energy usage by running directly on Linux without Docker.

------
rogaha
It would be more interesting to see "apple to apples" comparison such as
"Docker vs Mesos Containerizer vs CRI-O" or "Xen vs QEMU vs KVM". But I'm glad
to see this level of analysis!

------
bandrami
Not only that, but Docker's "containers are cheap" mentality _also_ means that
there are more calls being made in the first place, and those are being made
less efficiently.

------
Walkman
The whole idea of Docker is to make it possible to better saturate single
machines. How is that not more energy efficient than solutions before Docker?

------
hmottestad
Conclusion: 2 watt higher power consumption with docker.

------
throw9966
This is just a paper presented for the sake of a paper presentation, like
myriad of papers presented. Nothing to see here, should move on.

~~~
ajkjk
What do you mean?

~~~
throw9966
Its common sense to understand that when you run an extra application (docker)
in one machine and compare it with another machine which do have it,
everything else remaining the same, the machine which runs the extra
application consumes more energy. Don't need a paper or watts up to prove
that. That is the reason I said, its a paper presented for the sake of it and
nothing more.

~~~
ajkjk
Well, common sense won't tell you the amount of extra energy used, or which
specific aspects of the extra application are particularly responsible for it,
like this paper does.

------
kreetx
Why not measure just processor load?

~~~
bjackman
Power consumption is more complex than processor load - you have to think
about how P-state (cpufreq) and C-state (cpuidle) selection is affected by the
workload and now the scheduler distributes load across cores.

~~~
progval
There are other factors to power consumption: type of instructions used (eg.
cpuburn generates more heat than a regular 100% CPU usage), memory accesses,
cache hits/misses, ...

------
endymi0n
While an interesting observation and research area, I'd be curious how much of
these effects are offset by the better utilization that containers enable. I'd
bet that the reduction in idling server capacity much more than equalizes
these effects. However, this would probably be hard to measure.

------
eikenberry
Who'd have thought that an additional layer of abstraction has a small impact
on performance. Nothing else in computers work that way.

~~~
chrisseaton
Great - you've made a hypothesis using your intuition. It seems like it's
probably true. The science part is then checking that this it actually is,
rather than just assuming, which is what this paper is.

~~~
eikenberry
No hypothesis or intuition at play. I've read some of the many articles
covering the performance impact of using docker on various metrics. Useful
metrics, unlike an electric bill.

~~~
sirclueless
Huh? Electric bill is the #1 useful metric for any workload at scale. If
you're small, maybe human administration costs dominate that, but otherwise
it's virtually guaranteed to be the primary operating cost.

~~~
eikenberry
I guess so, but if it matters that means you have hardware and I hate having
to manage hardware. So I guess that colors my opinion on the matter.

------
tunnuz
Sort of related: mongodb seems to use a fair bit of energy as well on Ubuntu
linux.

~~~
dokem
That's not related at all.

