
SaltStack Mining Attack - photon-torpedo
https://saltexploit.com/
======
alexandercrohde
>>> If you're here, chances are you're already compromised.

WTF does that mean? Our salt implementation is on an entirely private network,
so why would I be more likely than not to be compromised already?

\----

Edit: Re-downvotes -- This is a sincere question. Is there some evidence that
the majority of salt implementations are compromised, or some mechanism by
which this hits private networks? Or is that line just for dramatic effect?

~~~
twic
I assumed that the problem was that if a master is accessible on your
intranet, it could be hit with some sort of XSS attack from browsers inside
the firewall.

But apparently there are 6000 people just straight up exposing their masters
to the internet:

[https://gbhackers.com/saltstack-salt/](https://gbhackers.com/saltstack-salt/)

~~~
Spivak
But that's like half the point of SaltStack. It's supposed be secure enough to
run on the public internet to manage road-warrior endpoints.

"I found this site called Alexa that lists a bunch of companies that just
straight up expose their web servers to the internet. Crazy."

And I have some bad news about how many companies are exposing their VPN
servers to the internet too.

~~~
tilolebo
It's really not.

You also wouldn't expose your database just because it's password-protected?

I bet all these public servers have a firewall running to protect other
services.

Shouldn't be hard to include the saltstack port(s) and whitelist the relevant
IP addresses.

Also,the main difference with your example is that the clients that connect to
webservers and vpns are either not known in advance or don't have static IP
addresses.

~~~
Spivak
Anyone who's putting their Salt master on the internet is having to deal with
clients without fixed addresses. And it's not uncommon to have to deal with
clients that aren't known advance which is when autosign scripts are used.

Sans the current CVE this is otherwise a service that is safe to expose to
adversarial networks.

~~~
tilolebo
From Algolia's outage retrospective:

    
    
      What we did so far:
      We’ve secured the impacted SaltStack service by updating it and adding additional IP filtering, allowing only our servers to connect to it.
    
    

So clearly unrestricted access wasn't a necessity.

I understand it's a pain, I've been running a 1000+ server stack with puppet
on a public network and relied on iptables to secure it. But I'd rather cope
with the daily iptable rules update than having to fight a 0-day exploit...

------
formersaltuser
SaltStack has a long history of home brew protocols, internally written
encryption, security issues, and bugs. I'm not surprised they would end up
being used as a vector for attacks.

~~~
jlgaddis
Exhibit A: CVE-2013-2228.

As described inthe relevant entry [0] on Debian's Security Tracker:

> _SaltStack RSA Key Generation allows remote users to decrypt communications_

The fix [1]:

    
    
      - gen = RSA.gen_key(keysize, 1, callback=lambda x, y, z: None)
      + gen = RSA.gen_key(keysize, 65537, callback=lambda x, y, z: None)
    

\---

[0]: [https://security-
tracker.debian.org/tracker/CVE-2013-2228](https://security-
tracker.debian.org/tracker/CVE-2013-2228)

[1]:
[https://github.com/saltstack/salt/commit/e8ce66cf688b43aeb3e...](https://github.com/saltstack/salt/commit/e8ce66cf688b43aeb3e716e78b1af3a08e9940e3)

~~~
sushisource
Jesus that's some rookie stuff right there

~~~
lima
They also built their own low-level network protocol based on 0mq. Back when I
tried it, it had constant keepalive issues behind NAT and would randomly lose
messages (that was in 2014, probably better now).

~~~
xenophonf
It wasn't any better as of last year, and its constant connectivity issues
were part of the reason I abandoned it.

------
nerdbaggy
We all got lucky on this one. The outcome could of been much worse, like
secret leaking and rm -rf. We record everything that saltstack does and the
scripts didn’t even upload anything from the infected servers.

~~~
willjp
You may have been lucky, but that was not the case with everyone. Some of the
entries appear to be missing, but last night someone setup a honeypot and was
posting what they saw.

Adding keys to /root/.ssh/authorized_keys, scp'ing sshkeys, flushing iptables.

I am of course one of the idiots that had it exposed to the internet. I
wouldn't do it with a database, but I didn't think twice about salt. Where was
my head!

~~~
mrits
Those things you listed are easily fixed which is what I assume they meant by
lucky.

------
lndarj
Our server were affected. They added a cronjob which ran every minute wgetting
a .sh file and executing it. Most of the time the server which the file was in
was either offline or returned 404, but every once in a while it returned the
malicious script. When this script got executed, it killed our nginx server,
which is how we noticed something odd was happening. If it wasn't for nginx
dying, we might have not even noticed we were infected.

Here's the contents of the sh script for the curious.
[https://pastebin.com/CbupwQMG](https://pastebin.com/CbupwQMG)

~~~
nikanj
Looks like it goes through a massive amount of effort to eliminate competing
miners and malware. Should neuter that script and run it periodically on boxes
:)

------
lol768
This hit RamNode too.

> This message is to customers with VPSs on our legacy SolusVM system.

> At approximately 20:34 eastern (GMT -4) on May 2, recently published
> SaltStack vulnerabilities (CVE-2020-11651, CVE-2020-11652) were used to
> launch cryptocurrency miners on our SolusVM host nodes. The attack disrupted
> various services in order to allocate as much CPU as possible to the miners.
> SSH and QEMU processes were killed on some of our CentOS 6 KVM hosts,
> causing extended downtime in certain cases.

> Upon detecting the disruption, we quickly began to re-enable SSH, disable
> and remove Salt, kill related processes, and boot shutdown KVM guests. After
> careful analysis of the exploit used, we do not believe any data was
> compromised.

> RamNode was not specifically targeted, but rather anyone running SaltStack
> versions prior to the one released a few days ago (April 29).

~~~
londons_explore
> After careful analysis of the exploit used, we do not believe any data was
> compromised.

If someone had code running as root on their machine, they can't say that
statement with any confidence whatsoever.

~~~
lol768
> If someone had code running as root on their machine, they can't say that
> statement with any confidence whatsoever.

Indeed, I think you need to assume all guests running on the affected nodes
are now compromised.

------
scblzn
Algolia got impacted too, apparently their Salt masters were opened to the
whole internet: [https://blog.algolia.com/salt-incident-
may-3rd-2020-retrospe...](https://blog.algolia.com/salt-incident-
may-3rd-2020-retrospective-and-update/)

~~~
jlgaddis
One of DigiCert's Certificate Transparency logs was likewise open to the
entire Internet -- and likewise compromised [0]:

> _I 'm sad to report that we discovered today that CT Log 2's key used to
> sign SCTs was compromised last night at 7 pm via the Salt vulnerability._

Several other "high-profile" sites have been compromised as well, including
LineageOS and Ghost. I expect we'll hear of many more in the next few days.

\---

[0]:
[https://groups.google.com/a/chromium.org/forum/m/#!topic/ct-...](https://groups.google.com/a/chromium.org/forum/m/#!topic/ct-
policy/aKNbZuJzwfM)

------
docsapp_io
This impacted ghost.org hosting really bad
[https://status.ghost.org/](https://status.ghost.org/)

~~~
machbio
Impacted LineageOS too -
[https://twitter.com/LineageAndroid/status/125682105610016358...](https://twitter.com/LineageAndroid/status/1256821056100163584)

~~~
twic
Oh god, the responses. Half of them are completely unrelated, asking Lineage
to support particular phones.

And then there's someone who saw "salt" and tried to help:

[https://twitter.com/mvonwi/status/1256989787321438209](https://twitter.com/mvonwi/status/1256989787321438209)

------
rhakyr
Debian hasn't fixed this yet in their packaging so you might want to work
around that if you are using that as a salt-master server.

[https://security-tracker.debian.org/tracker/CVE-2020-11651](https://security-
tracker.debian.org/tracker/CVE-2020-11651)

------
willjp
I'm so grateful this happened over the weekend, when I had time to respond. My
girlfriend woke me up and told me the CPU-fan was going crazy in the living
room. I realize this isn't the case for everyone. I extend my deepest
consolations to those affected.

------
tormeh
Former SaltStack user here. If you're using it, just stop. Switch to Ansible,
a Kubernetes/Helm/Flux solution or experiment with Chef Habitat if you're
feeling futuristic. SaltStack is just bad. It's buggy, clunky and has really
bad error messages. Its only saving grace is that it came before Ansible, but
that's only relevant in a historical context. Just no.

~~~
lykr0n
None of those are compatible solutions.

Ansible is very inflexible when it comes to configuration and how you
structure your playbooks.

Kubernetes/Helm/Flux only works if you're inside k8s. I don't know why you
would be using Salt for that.

Chef Habitat looks nice, but again. I'm doing more then just management of
applications.

The only real alternative to Salt is Puppet. So, No. Just no.

------
circlingthesun
We got hit by this on Saturday night :|

------
thesz
How does things like these affect Monero's reputation?

I saw Monero mining in JS libraries, now it is in virus form.

Thus, why Monero? Does Monero somehow discourage these "practices"?

~~~
jackhalford
Maybe it's to do with the fact that monero tokens are non-fungible, unlike
some other crypto-currencies.

~~~
3456j34jht
I think you mean 'fungible'. Non-fungible means 1 XMR is not equivalent to
another 1 XMR. The use of ring signatures, key images and transaction amount
hiding gives Monero its fungible properties. This makes it computationally
difficult to trace a particular transaction graph. This provides fungibility
beyond that of Nakamoto style coins, in that you cannot easily block/identify
addresses or transactions.

