
Backdoored images downloaded 5M times removed from Docker Hub - cel1ne
https://arstechnica.com/information-technology/2018/06/backdoored-images-downloaded-5-million-times-finally-removed-from-docker-hub/
======
zimmerfrei
On the same topic, PyPI has recently moved to a new backend, and in the
process all end-to-end PGP signatures (created by the package owner upstream,
proving that no tampering happened on the online servers) have disappeared
from the UI, and that is seen as a "feature":

[https://github.com/pypa/warehouse/issues/3356](https://github.com/pypa/warehouse/issues/3356)

You can still get them through some obscure API and you still need to know the
right PGP key for verification, but this really signals the lack of consensus
and awareness on the path toward a secure software supply chain.

EDIT: typos

~~~
egjerlow
FWIW, here is a blog post by dstufft which might help contextualize this
behaviour: [https://caremad.io/posts/2013/07/packaging-signing-not-
holy-...](https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/)

~~~
msl09
That is discussed extensively in the issues related to the OP. The problem is
that package maintainers of distros actually check whether the GPG signature
has changed in order to repackaged python projects for their distros.

~~~
jonathonf
Why are distros packaging from PyPI and not from the upstream project?

~~~
jwilk
Because that's where upstream puts release tarballs.

------
TekMol
I wonder how much malicious code like this is doing its work deep down in the
endless pyramid of npm dependencies.

And how much as-of-now clean code will turn into malicious code when bad guys
take over npm repos in the future.

It might be possible to tackle this issue by some intelligent trust algo that
combines a trust rank similar to google-page-rank and signed messages.

Say somebody pushes an update to their repo. Now the first user of it might
read it and sign it with 'Looks OK /Joe'. And the next user sees the signed
message by Joe in some kind of package-review-message list. Based on all the
reviews and the trust of the reviewers, they then can calculate a trust score
for the update.

~~~
onion2k
...or CPAN modules, composer libs, cocoa pods, etc. Anything you use to
install unchecked external code is potentially dangerous.

At least npm has auditing now, so checking for problems is relatively trivial.
GitHub even does it automatically.

~~~
dvfjsdhgfv
Well, it's much, much easier for me to audit a CPAN module than a Docker image
- the latter is practically impossible.

~~~
raesene9
eh?

Assuming automated builds, you can just read the dockerfile (and it's
hierarchy if necessary), it's less complex syntax than perl by a long way.

Even assuming no automated build, all the information is in the manifest, and
using something like portainer it's pretty easy to read.

~~~
palotasb
Don't a lot of OS images start by importing a non-transparent prebuilt tarball
containing nontrivial binaries? I would hide the malware inside those.

~~~
ownagefool
Sure, but so could the OS or app distributors. You need to establish a
baseline of trust somewhere, and this will likely be on the official (or your
cherry picked official) images, and you build from there.

------
fpgaminer
Somewhat related, since this is about Docker security: I started looking at
Traefik today. It's a reverse proxy that runs as a Docker container and
automagically configures itself to expose your other services (that are also
running in Docker containers).

Neat idea. However, to accomplish this you have to mount the docker socket
into Traefik's container...

Which means that when a bug shows up in Traefik attackers can pivot out of the
container and onto the host; access to the docker socket is equivalent to root
on the host.

And of course Traefik is the thing you're exposing directly to the internet.

It's like giving the guards outside manning your castle's gate the skeleton
key to the rest of the castle.

Of course, Traefik is quickly becoming popular because of its simplicity. But
to achieve this simplicity it carves a giant hole in the security of your
application.

~~~
gnur
The integration of Traefik with the Docker daemon should mainly just be used
while developing (imho). Once you get to acceptance / production environments,
you are very unlikely to run plain docker containers, if you use kubernetes
you interface Traefik with the kubernetes api itself, and the service account
you create for Traefik can be (and should be) completely read only. Same for
Docker Swarm, Marathon, Consul and AWS ECS.

So no, Traefik is not the big security problem you make it out to be.

Sorry to be so harsh, but Traefik is one of the most amazin pieces of software
I have come across in the last years that has seriously made my life much
easier.

~~~
jakobegger
If software has an insecure mode "just for development" that absolutely
shouldn't be used in production, you can be certain that a large fraction of
developers will use that in production nevertheless.

Security today doesn't mean that you are safe if you do everything according
to best practices and follow the docs. Modern Security includes making sure
that default settings are safe, and that it should be impossible or hard to
set up the software in an insecure manner.

If you make it easy to shoot yourself in the foot, that's what people will do.

~~~
raverbashing
> If you make it easy to shoot yourself in the foot, that's what people will
> do.

Yes they will. Just repeating it here because it bears repeating.

Yes, they don't care. And yes, there will be an attacker trying to exploit it.

------
LeoPanthera
This bug: [https://github.com/docker/hub-
feedback/issues/1121](https://github.com/docker/hub-feedback/issues/1121)

raised over a year ago(!) is really interesting. It seems like many of the
downloads may have been malicious - the author of the malicious images was
scanning for open docker api ports and then installing their own images to
mine cryptocurrency.

So they're essentially using docker as a dropper. Clever, in a way.

------
bboreham
In what sense is this a “backdoor”? Seems to me the code is coming through the
front door, which the victims left open.

DockerHub is just the delivery mechanism.

~~~
imtringued
I'm scratching my head at where the /mnt mount is coming from. If you're doing
"docker run -v /:/mnt <sketchy_username>/mysql" then absolutely nobody can
help you.

~~~
b6z
Same for me.

From Kromtech's article I deduced that this only happens when a docker daemon
(or kubernetes interface) is exposed to the Internet and an attacker uses that
to download and start a docker image on the victim's host. Then they can bind
mount a host directory like described and attack the host computer.

------
cyphar
It should be noted that some of the reports talk about the Docker API being
publicly accessible over the internet which allowed people to run containers
on their machines. This is actually not the worst thing that could have
happened -- _having access to the Docker API gives you root access on that
machine without any authentication_!

(One of the ideas of rootless containers is to remove the possibility of any
privileged codepath, which helps eliminate this issue.)

~~~
avip
Docker api exposed over internet without TLS implies a head should be removed.
That’s not the default config. Not what the docs recommend. Why??

~~~
djsumdog
I don't think that's even possible. Docker doesn't let you expose the daemon
over HTTP without configuring certs. I had to write an ansible script to do
that, and even then I locked down my Docker port to my VPN subnet:

[https://github.com/sumdog/bee2/blob/master/ansible/roles/doc...](https://github.com/sumdog/bee2/blob/master/ansible/roles/docker/tasks/servertls.yml)

~~~
cyphar

        sudo dockerd -H tcp://0.0.0.0:8080
    

will happily start Docker with it listening on my IP address without TLS. It
will print an all-caps warning, but nothing else (you don't even need to pass
a --give-the-internet-root-access flag). However, I just submitted a PR which
adds the --give-the-internet-root-access flag[1] because it's pretty obvious
to me that _very_ few users do this intentionally (and with full knowledge of
the consequences).

[1]:
[https://github.com/moby/moby/pull/37299](https://github.com/moby/moby/pull/37299)

------
ccnafr
ORiginal report: [https://kromtech.com/blog/security-center/cryptojacking-
inva...](https://kromtech.com/blog/security-center/cryptojacking-invades-
cloud-how-modern-containerization-trend-is-exploited-by-attackers)

The ArsTechnica article, like most AT articles, glosses over most details and
focuses on a small-time cryptomining campaign

------
djsumdog
I don't understand why people use other people's Docker images. Unless it
comes from an official repository for the tool you're using, it's better to
look at the source code/Dockerfile in the github link and just roll your own.

A lot of times you're just installing the package you want with apt-get within
your Dockerfile anyway; a package you can't check for normal updates for
anymore since it's in a container. So now you need a tooling system around
making sure your packages in your containers don't have security issues.

Docker is kinda a mess.

~~~
y4mi
its not really a mess for its usecase.

its immutable infrastructure at its heart, so yeah, you don't do updates on
containers... what you do need is periodic rebuild of your images for upgrades
and each new image needs to run all integration and system tests again.

it just makes this process easier than it is without docker. But it doesnt
alleviate you of writing the system that actually keep everything updated in
an automated way.

It also doesnt help you deploy unless you're already experienced with docker.
and while we'Re on the topic... no, if you know how to execute 'docker run -it
--rm ubuntu bash' you still don't know shit about it. _sigh_ sorry, i'm just
remembering someone from work today...

------
ex_amazon_sde
For those who wonder why Linux distributions are "still" around, this is a
reason. Some have a good vetting process for packages.

~~~
HankB99
I wonder which ones have the best vetting. And if it is adequate.

I also wonder about other packaging systems. CPAN, pip (pypy?) AUR and so on.
It doesn't surprise me to see this happen. I wonder what other surprises might
be in any of these packages.

FWIW, I'm running mostly Debian and some Ubuntu. I _always_ prefer to install
packages via the package manager rather than directly from some tool specific
repository because I'll get automatic updates and some level of
testing/vetting.

------
crypt1d
>By the time Docker Hub removed the images, they had received 5 million
“pulls.” A wallet address included in many of the submissions showed it had
mined almost 545 Monero digital coins, worth almost $90,000.

This seems incorrect because its impossible to see wallet balances on the
Monero network. So I'm assuming they just came up with the numbers based on
some rough calculations.

~~~
ricANNArdo
If you go to the pool website with the address specified on the botnet you can
see how much it was mined. The main article [1] linked on the news said:

> The actor has been able to mine about 630 XMR to date, which at the current
> USD rate is more than $172,000 for just a little more than one year of
> activity.

[1]: [https://www.fortinet.com/blog/threat-research/yet-another-
cr...](https://www.fortinet.com/blog/threat-research/yet-another-crypto-
mining-botnet.html)

~~~
crypt1d
that makes more sense, thanks.

------
jchw
The worst part here is definitely the timeline. NPM is often criticized about
security, perhaps rightfully so, but at least the issues are handled promptly
after raised publicly.

~~~
paulie_a
There is no "perhaps" about npm, it is absolutely a shit show. It's like the
creators went out of their way to build an ecosystem of security nightmares.

~~~
outside1234
ok, hang on, how is npm different from maven, pypi, etc. ?

~~~
oneweekwonder
I will take a stap at comparing npm vs pypi.

While this will be anecdotal I found my `pip freeze` packages a lot more
manageable compared to `npm ls --parseable`.

My Full Stack Flask application sits at about 64 requirements. Where last time
I used expressjs my dependancies inside of node_modules inside of node_modules
border-lined to insanity.

I can see myself hand picking and reviewing my requirements.txt, which I did
to some extent.

But I just gave up with npm.

Which is a bit of a personal dilemma for me, because some of my tooling needs
npm.

~~~
ehnto
So many packages and authors creates an impossibly large surface area to
review and secure. It is my impression that most people don't even try so the
issue falls on deaf ears. But for a PCI compliant piece of software for
example, you would have to had reviewed every module in node_modules. As a
stack it makes it a non-starter.

------
raesene9
This is essentially a dupe of
[https://news.ycombinator.com/item?id=17303570](https://news.ycombinator.com/item?id=17303570)

FWIW that headline isn't great. Docker hub pulls in no way correlate to
innocent users pulling/using those images. It could be (and this is quite
likely) just other malware which made use of those images and just used Docker
hub as a repository.

There are official images for the software in question and I don't think it's
that likely that that many people ignored the official ones and got these
ones.

------
meuk
“For ordinary users, just pulling a Docker image from Docker Hub is like
pulling arbitrary binary data from somewhere, executing it, and hoping for the
best without really knowing what’s in it,”

This is basically what you do every time you install something (except when
it's via a walled garden like an 'app store'). Besides, I'm not sure I would
even classify mining for someone else as 'malicious'. It hogs your CPU a
little, but if that's malicious then visual studio should be considered
malicious as well.

~~~
laumars
Maybe it's not malicious in the strictest sense of the term but it's also not
the same as your Visual Studio example. In your case Visual Studio is a
productivity tool that brings you value and thus you chose to install it. In
the case of this docker image it's not adding you value and was installed
without your concent.

The JS example another commenter made is more apt however the argument there
is that you still requested the site and it's content (even if you didn't
really want it). Whereas many of the installs of this "dockerised" miner were
remotely via exposed Docker APIs. That I think is the real crux of the
potential "malice" (for want a better description) here.

------
INTPenis
This is exactly why I never liked Ansible Galaxy, and Docker Hub came into the
same category.

Screw the extra work, I'd rather write my own roles and Dockerfiles.

~~~
pmlnr
Most of these things are plain text instruction files - yaml for ansible,
docker's own thing for docker. It falls under the same category as random bash
install scripts: download the text file, read it, use it, if it's safe.

~~~
INTPenis
Yes read it, use it but the next step can't be update it because then you'd
have to read it again. I just don't have the time to audit someone elses yaml
constantly.

------
outside1234
where are the images enumerated?

~~~
rafaele
[https://kromtech.com/blog/security-center/cryptojacking-
inva...](https://kromtech.com/blog/security-center/cryptojacking-invades-
cloud-how-modern-containerization-trend-is-exploited-by-attackers)

search for "Figure 7"

------
etaioinshrdlu
A startup I'm aware of (not associated with) that aims to help tame this
problem a bit: [https://anchore.io/](https://anchore.io/)

------
yani
This is not a backdoor. I myself have a miner on Docker hub. The image can be
used by anyone with correct envars set. Should my image be removed if used by
other users no matter what their intensions are?

~~~
sleepychu
You are being facetious.

If your image is called monero-miner and a bunch of people download it, of
course it's not going to be considered malicious code.

If your image is called apache-webserver and a bunch of people download it and
you've stealth bundled a monero miner, of course it's going to be considered
malicious code.

EDIT: even worse than that, the images are _actually back doored_ they open up
a reverse shell to allow the remote to execute arbitrary commands.

