
Help:We Found a Bitcoin Mining Prog / Email Server Running on Our Server - gopi_ar
We saw that load spiked on one of Ubuntu servers yesterday and found this on the proc list using all our cores:<p>statd 7680 690 0.0 743976 58492 ? Sl Nov26 20914:53 .&#x2F;yam -c 1 -M stratum+tcp:&#x2F;&#x2F;binyu.crypto%40gmail.com:x@xmr.pool.minergate.com:45660&#x2F;xmr<p>This was running under the statd user.<p>What do we do? We checked firewall, SSH, all seem OK. How do we go about investigating this breach?
Do help!
======
shostack
On a related note... What stops companies from doing things like adding a
background mining script to a website that uses people's browsers to do the
mining?

I'm not very familiar with the tech behind BTC mining so there may be some
obvious reason that isn't feasible. I was always surprised some evil company
like EA hasn't added it to wait screens when matchmaking for various games.
They could install whatever they want, target machines likely have solid GPUs,
and they are sitting around waiting.

I also wonder about the legal aspects of this. Would someone need to opt in
this? Is the energy cost of something like this legally distinguishable from
sites that say... load a bazillion tracking tags and eat up your data cap?

~~~
r1ch
ESEA (a counter-strike matchmaking service) tried this. It didn't go well.

[https://www.google.nl/search?q=esea+bitcoin+scandal](https://www.google.nl/search?q=esea+bitcoin+scandal)

~~~
shostack
Interesting. So my takeaway from that is that they weren't really up front
with getting consent and opt-in for it, and that it is still legal and valid
if they were to have gotten that?

------
mcfrankline
Nope. Not bitcoin.

It's pointed at xmr.pool.minergate.com:45660/xmr

XMR is Monero. It has a lower hashrate so i see how the attacker can make
something out of this.

Are you sure it's not an inside job? Cause anyone with access to run this
under statd basically owns you right now

~~~
gopi_ar
Yes, it's a Redis vulnerability (caused by bad config on our part) in one
container where the firewall was down.

Strange thing if we run 'top' from the main host, all containers running redis
say 'statd' as their user; inside the container the user showed 'redis'. We
removed nfs and all related files, and now it shows a user ID number. Is this
something we should worry about?

~~~
alexginzburg
Could you elaborate what redis configuration could've caused this?

~~~
gopi_ar
Here you go: [https://kevinchen.co/blog/postmortem-server-
compromised/](https://kevinchen.co/blog/postmortem-server-compromised/)

------
gopi_ar
Update: I sent an email to the email on that script. And the person at the
other end replied and mentioned that he/she is doing it for extra pocket money
and was only mining on the server. We aren't going to pursue any legal
charges, might even pay the person a bounty for pointing out this
vulnerability. I'd like to thank all of you, with special mention to some
folks over at reddit for all your help!

------
throwbsidbdk
Hahahahaha that's a pretty creative way to monetize an attack.

Hopefully you're taking regular VM snapshots so you've got some logs they
can't delete. Otherwise good luck, someone Bitcoin mining is probably clever
enough to cover their tracks.

Realistically an breach bad enough that they have server control is probably
through the web. The most common way I've seen is through various CMS code
execution exploits. If your web apps allow file upload that's a really common
way to get code running on the server as well

~~~
gopi_ar
Funny thing is we don't do uploads anywhere and there's no CMS whatsoever..
Which leads us to believe it's an OS vulnerability.

Would you how we could hire professionals to investigate this for us? And
report it to appropriate groups..?

PS: These are dedicated servers :-/

~~~
nodesocket
Can you answer the following for us?

    
    
        Is ssh only allowed by public key? (in /etc/ssh/sshd_config => PasswordAuthentication no)
        Is Apache or NGINX running on the server?
        Is PHP/Ruby/Node/Python running apps?
        What ports are open in iptables (iptables -L)
        What does /var/log/auth.log say?

~~~
gopi_ar
Thank you for responding.

We searched the whole system for authorized_keys files and found one created
in a /var/lib/redis/ of a staging container (with no firewall) on this host.
We then came across the redis vulnerability
[https://kevinchen.co/blog/postmortem-server-
compromised/](https://kevinchen.co/blog/postmortem-server-compromised/) . A
junior dev had spawned this container without help from dev-ops and hence left
ports open.

What doesn't make sense to us is how this daemon (yam) was running under a
statd username when the container doesn't have such a user, but the host does?
Are LXC containers able to run daemons on the host?

~~~
jmgao
> What doesn't make sense to us is how this daemon (yam) was running under a
> statd username when the container doesn't have such a user, but the host
> does? Are LXC containers able to run daemons on the host?

This is because usernames don't exist, as far as the kernel's concerned. ps is
resolving the process's UID to the corresponding name for the outside context,
not the one inside the container.

~~~
gopi_ar
This makes sense, we can rest easy knowing they didn't break out of the
container. Thanks!

------
matiasb
Hi, I wrote a Redis security tool once and this attack is familiar to me.

I might be able to help you on investigating this issue. Contact details are
on my profile.

------
alexginzburg
we are in the same situation. Early morning today found one host with a high
cpu usage. Turned out it was running `./yam` process as a `redis` user. I shut
the host down for now. Before shutting it down I did a strace and saw json
stream clearly stating that it is a monero app. Looks like the cpu spiked
about 12 hours ago. We do have redis on a host but it should be behind the
iptables rules. Other hosts look ok.

~~~
gopi_ar
We were able to get in touch with the hacker and he told us he was just mining
and not stealing stuff. We're still cleaning the whole system; might even pay
him/her a bounty for this though.

------
max_
Its a mining program. But not for BTC. The hashing power points to a Monero
Pool

[https://getmonero.org](https://getmonero.org)

