
Juniper screenOS authentication backdoor - master ssh password posted - ghshephard
https://community.rapid7.com/community/infosec/blog/2015/12/20/cve-2015-7755-juniper-screenos-authentication-backdoor
======
rdtsc
I'd like to hear how this was discovered and who's name ended up on git blame
line next to that password.

I have done a malicious source code injection as part of a network security
exercise at the university. That was before git or other sane source control
and I basically inserted a semi-obfuscated piece of code in the source
repository, which gave our team an advantage in the game (the game was to
crack and find a weakness in a protocol, but the whole machine was a
target/battlefield). The clever part was first rooting the server via suid
vulnerability.

I won the contest, but while doing it I thought, yeah, this shit will never
work in the real world. And then this story made me remember that. That's
pretty crazy.

~~~
jacquesm
If you think about it a bit longer you'll realize it can not be a single name.
Yes, someone wrote this code. But then someone else reviewed it and then a
third person signed off on shipping the end result. It takes (or really, it
should take) more than one person at a company like Juniper to get away with a
trick like this. Unless their processes are completely broken, let's for the
moment assume that they are not otherwise all bets are off. Juniper is going
to be scratched from a lot of POs because of this story, if it turns out they
don't have a process in place to protect against stuff like this and it does
not involve two people or more operating in concert to circumvent that process
then they're pretty much dead.

~~~
michaelt
If I was doing an attack like this, I'd wait for an opportunity to slip it
into a huge, mundane change.

For example, changing the repository from Mercurial to Git. Splitting or
combining two repositories. Moving lots of files between directories. Running
an autoformatter over the entire codebase. Something like that.

Every code review tool I've seen there's /some/ way you can get it to
highlight thousands of trivial changes. And I don't know about you, but I
can't promise I'd spot 1 evil change among 1000 trivial ones.

~~~
jacquesm
Yes, that would be the way to do it. And that is _exactly_ why a change like
that is _just_ as dangerous as any other.

I did quite a bit of code review during a contract earlier this year. One day
one of the programmers shows up with a 100K line changeset, a reformatting and
clean up that he'd embarked on of his own accord. Imagine his surprise when I
point blank refused the commit. He was extremely upset (understandably, it was
a lot of work and well intended) but at the back of my mind were two things:
one, we are already under pressure to get things to work and this does not add
anything functional. two, if it _does_ add anything functional it will be an
error or a bug and there is no way I'm going to catch this manually reviewing
all these changes (across 100's of files). So the safe thing to do is to
simply not accept the change unless I can free up developer time to break this
massive commit into smaller ones that can be reviewed and accepted one-by-one
over a longer period of time. And that was a luxury we could not afford.

So too bad, one _very_ inflexible reviewer and one _very_ pissed off
programmer. 1000 trivial changes in a single commit isn't going to fly (at
least, not with me), unless I review each and every one of those with just as
much attention as I would review a much smaller commit. It's that sort of
attention to process details that probably makes me 'less than easy' (to put
it mildly) to work with but I really feel that if a customer trusts me that
I'm going to have to earn that and rubberstamping a change set because it is
too large to review is definitely a breach of that trust.

~~~
junto
We have a rule that you can only prettify (to defined style rules) when you
are already making a change to that piece of code or file.

Otherwise it's hand off as a rule. It also stops programmers wasting time.

~~~
dalore
I believe that's the wrong way to do it. You can only prettify/reformat if you
aren't making changes to the file. That way you review only the prettify
changes and not the functional changes.

~~~
Myrmornis
Agreed, refactoring and functional changes should certainly be in separate
commits, and ideally in separate pull requests. jacquesm can still have
her/his rule to only change code involved in the course of functional changes;
it's just that the refactoring and functional-change PRs will come in close
succession and the refactoring PR exists only because the functional-change PR
does also.

------
_jomo
> The argument to the strcmp call is _< << %s(un='%s') = %u_, which is the
> backdoor password, and was presumably chosen so that it would be mistaken
> for one of the many other debug format strings in the code.

That's very clever from the attackers point of view, extra kudos to hdmoore
for finding it!

~~~
captaincrowbar
Not really clever. Format strings have a fairly restricted format; that one
can't have all that many bits of entropy. Once the obfuscation method occurred
to an attacker, it wouldn't take long to brute force it even if they didn't
have a file to search for suggestions. (And now that the idea is out there, I
bet any number of hackers, wearing assorted coloured hats, are trying this out
on other systems Even As We Speak.)

~~~
qb45
> Not really clever. Format strings have a fairly restricted format; that one
> can't have all that many bits of entropy.

I fully agree that _now_ is a great time to brute-force every ssh server found
on the Internet with randomly generated valid format strings.

However, did you know it before this password was published? I think it was a
novel idea.

------
tw04
Can I just point out I find it hilarious reading about all the people debating
how amateur it was to put the password into the code in plaintext was... all
while ignoring the fact that this backdoor survived for 3 YEARS of code
reviews? Obviously the way it was implemented was ingenious, and it likely far
surpassed whoever put it there's wildest expectations.

~~~
BinaryIdiot
> ignoring the fact that this backdoor survived for 3 YEARS of code reviews

While I don't disagree with the rest of your comment it's possible that this
portion of the code was only reviewed once (if at all) during the entire 3
years whereas your comment makes it sound like it was constantly reviewed over
the period of 3 years.

------
Deregibus
It'll be interesting to see a more in-depth look at the code surrounding this
and some theories on how it could have been implemented in a way that avoided
obvious detection. The naive assumption is that someone just added to the
code:

    
    
        if(!strcmp(password, "<<< %s(un='%s') = %u")) return true;
    

which is certainly possible, but seems too easily detectable and risky for
someone on the inside, and too lazy for someone on the outside that had
already gone through the trouble to get write access to the source.

The string itself looks like it's part of some logging system, so my guess is
that it already existed and was opportunistically chosen rather than created.
If this was passed through a macro, then it's possible that the attacker
didn't have to touch the auth code at all and may have been able to implement
this by changing only a handful of characters in an area of code that was more
amenable to obfuscation.

~~~
emillon
That is my opinion too, but I would go even further and say that this change
could have been made on the compiled binary.

This kind of statement must take 10-15 bytes max to patch and the build boxes
are typically less safe than source control systems.

~~~
revelation
If you look at the disassembly in the link, the backdoor was inserted smack in
the middle of the authentication function, which caused jump labels further
down to change.

This is all trivial for a compiler to adjust, but it's not what someone
manually tampering with the binary would do.

~~~
Deregibus
In addition, AFAIK this affects both the ARM and x86 firmware, so a patched
binary would imply two separate modifications. Though that would still leave
open the possibility that the toolchain was exploited before compilation
occurred.

------
ra1n85
Brutal. From conversations with other engineers, this is a very common network
operating system with banks in Europe.

Would not be surprised if urgent code reviews and security audits are taking
place at the campuses of other large network software/hardware vendors.

~~~
theklub
Yeah, some people aren't getting sleep this week.

------
zouhair
How can we know that it is the only backdoor? Wouldn't be safer to just stop
using Juniper's products altogether?

~~~
nl
What do you propose to use instead?

Cisco[1]? Huawei[2]? Open Source[3]?

[1] [http://arstechnica.com/tech-policy/2014/05/photos-of-an-
nsa-...](http://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa-upgrade-
factory-show-cisco-router-getting-implant/)

[2] [http://www.bmi-t.co.za/content/which-way-huawei-ban-or-
not-b...](http://www.bmi-t.co.za/content/which-way-huawei-ban-or-not-ban)

[3] [http://arstechnica.com/security/2014/01/how-the-nsa-may-
have...](http://arstechnica.com/security/2014/01/how-the-nsa-may-have-put-a-
backdoor-in-rsas-cryptography-a-technical-primer/)

~~~
rdtsc
So based on that [3], open source then.

What are the high speed and feature-wise comparable open source solutions to a
the compromised Juniper switches?

~~~
f2f
> So based on that [3], open source then.

nortel networks. no bugs in the last few years.

~~~
jhall1468
> nortel networks. no bugs in the last few years.

No _known_ bugs.

------
jdiez17
It was a matter of time before it leaked. If you're going to put a back door
in, at least make sure only you can use it (Nobody But Us). Why didn't they
use a keypair?

~~~
LukaAl
Well, simply storing and hashed version of the pass would be ok. Even an MD5
or a SHA1 would be enough to guarantee at least a "Very Few plus Us" security.

Hiding that in the could would be much more difficult. And using PKs would
make the attack even easier to spot.

And then you have to consider that a "Nobody but Us" backdoor exist only on
paper, not in reality. If a key exists and someone knows it, well sooner or
later it will be discovered (disgruntled employees, hackers etc). The only
safe backdoor is the none backdoor. Now, consider this problem from an
attacker that doesn't give a s __t of your security. He knows that as soon as
the backdoor is known (because someone find it in the code, because someone
leak the password, because whatever...) the backdoor will be useless most
systems. What you do, you design a safe backdoor or you design a backdoor with
as little footprint as possible?

Obviously, legally sanctioned backdoor have a different set or constraint and
making them safe is a requirement. The fact that it is impossible to prove
them safe unless some assumptions (that always prove to be wrong, but...) are
made, it's a totally different problem.

~~~
ghshephard
That's a really good question. Why did they store the plaintext instead of a
hash? Doesn't feel like the sort of thing an attacker with any sophistication
would do. Also doesn't feel like the sort of thing that underwent any code
review. I'm starting to wonder about the "intern" theory again.

~~~
brazzledazzle
You're assuming the attackers care if the backdoor is used by others. If a
hash would increase the chances of it being found and you don't care if other
attackers discover it, only if the vendor does then it's the right choice.

~~~
ghshephard
I absolutely assume that if the backdoor was placed by a vendor, they would
make sure it was a one-way function.

I (perhaps incorrectly) assumed that criminals would want to stick in a hash
function, given a choice.

I think the explanation, that adding a hash function will attract more
attention than a simple strcmp is a good one. As is the desire to stick in
something that gets by code review.

All signs point to this 100% not being Juniper engineering officially adding a
back door, and it being a party doing it without authorization.

------
kabdib
This smells like:

\- Underhanded C . . . maybe. Seems difficult to just insert a strcmp in the
middle of a sensitive piece of the login path

\- A compromised toolchain that is inserting the code

Would love to hear what Juniper has to say about it, but I doubt that they
will, or will be allowed to say.

~~~
ghshephard
Compromised toolchain/Underhanded C perhaps explains why whoever did this
didn't take even the slightest bit of effort to compare against a hash.

~~~
rdtsc
That was my first guess. Based on how password looked it could have been a
multi-line split or some sort of other such trickery. A set of macros
perhaps...

~~~
stordoff
That was my first thought on seeing the password: the string wasn't chosen per
se; it was a string they could assemble (or block of memory reused) in a way
that appears benign in a code review.

~~~
buserror
Actually, it's very likely the string was used for some other (valid) purpose
and the pointer is being hijacked afterward -- it's VERY unlikely to appear as
a singleton string constant anywhere.

The password looks like what someone would put at the start of a function to
trace stuff; and, there's very likely a macro that does that all over the
software.

Lets imagine what that might look like:

    
    
      #define TRACE(__f, ...) \
         const char *tok="<<< " __f;\
         printf(tok, __func__, __VA_ARGS__);
    
      u32 check_pass(const char *un, const char *tock) {
        u32 res = valid(un, tock);
        TRACE("(un='%s') = %u", __func__, un, tock);
        res += valid(un, tok); /* could look like a cut/paste blip... */
        return res; 
      }

~~~
rdtsc
Yap, agreed, that makes most sense.

------
revelation
So, did they have time for a _git blame_ yet? It's one thing to say you have
found a backdoor, another is to clear up how it got there in the first place.

~~~
StringyBob
If you are capable of inserting this kind of backdoor, how easy is it to
compromise a version control system? (or the login details of one of the
programmers?).

Is it possible to prevent history of the code being modified? Do DVCS use
blockchains?

~~~
hueving
If a git commit is modified, every commit hash after it will change and will
break every clone.

~~~
escape_goat
That's a system that would still be entirely vulnerable to collision attacks,
though, right?

~~~
vishbar
I'd imagine it would be nearly impossible to generate a collision that a.)
does what you want, b.) is small enough to be unobtrusive, and c.) can be
discovered in finite time with the computing power reasonably available to
NSA/GCHQ/insert SIGINT organization of choice.

~~~
escape_goat
I asked because git uses SHA1, which might be a bit low on (c) at this point.

~~~
vishbar
True. Generating collisions is doable, but a.) and b.) are still huge
constraints that make it orders of magnitude more difficult.

------
gbin
Anybody tried to see in their logs if someone used it ?

------
sandworm101
So ... if this is a backdoor for some three-lettered agency, what are the
chances that this backdoor, even this same password, is present in other
products?

~~~
BinaryIdiot
Unlikely in my opinion. You don't reuse the same backdoor across any products
if you don't want to be noticed. Reusing means someone could run some sort of
statistical analysis and find a commonality.

Ideally every product you get a backdoor into is one done in a different way
so that even if one is found the others won't be found easily.

------
wepple
This may be a good time to bring up the various big corps who are trying to
prevent reverse engineering... while even they can't keep tabs on knowing what
is actually being executed on the hardware we pay them for.

------
isido
Now that the cat's out of the bag, could the esteemed hacker(s) just post the
source code, so that we can see how it was implemented :) (j/k)

------
rmdoss
If you follow good security practices and restricted your SSH access, you were
safe against it:

[https://twitter.com/danielcid/status/678907293770059776](https://twitter.com/danielcid/status/678907293770059776)

If you are managing any login system, try to implement ip white listing
whenever possible.

------
rasz_pl
> We were unable to identify this backdoor in versions 6.2.0r15, 6.2.0r16,
> 6.2.0r18 and it is probably safe to say that the entire 6.2.0 series was not
> affected

this sounds fishy, like Juniper trying to push users to upgrade from _non
affected_ builds to a new firmware with a fresh set of NSA backdoors.

~~~
jwildeboer
Not really fishy. The 6.3.x series is the only version that is under active
development [1] so quite naturally it follows that at least since 2013 (EOL of
6.2) many users have upgraded to the supported version.

A bit more disturbing IMHO is that you need an active support contract to even
get the updates (based on information I got from Juniper customers, I couldn't
find a direct confirmation on their site) meaning that aftermarket users are
left in the cold.

[1]
[https://www.juniper.net/support/eol/screenos.html](https://www.juniper.net/support/eol/screenos.html)

~~~
rasz_pl
That makes more sense. Juniper lied about older firmwares being backdoored to
force old customers into fresh support contracts. Not as fishy, just more
Oracle.

------
mmaunder
Anyone running ScreenOS was already scrambling as fast as they could to patch
the issue before the backdoor password was posted. Just thought I'd point that
out.

~~~
_yy
You wish. Some orgs move slowly.

------
nodesocket
Forgive me if I am wrong, but these devices phone home right? Can't they just
hot fix this over the air?

~~~
davidu
No. Not a chance.

~~~
ajdlinux
Well, they _do_ have a working administrator password to log in to every
affected box out there...

~~~
qb45
But no customer pays them for surprise downtime.

