
NSA finds Snowden hijacked officials’ logins - moonlighter
http://arstechnica.com/information-technology/2013/08/sysadmin-security-fail-nsa-finds-snowden-hijacked-officials-logins/
======
res0nat0r
> The National Security Agency (NSA) is the font of information security
> wisdom for the US defense and intelligence communities. But apparently, the
> NSA's own network security is so weak that a single administrator was able
> to hijack the credentials of a number of NSA employees with high-level
> security clearances and use them to download data from the agency's internal
> networks.

Thats a bit hyperbolic and out of touch with reality. Sure I as a sysadmin
with root to most UNIX machines in my companies environment could have been
able to copy the raw Oracle db files to steal company secrets, SAP databases
for other juicy data that I could sell to a competitor, run a network sniffer
on important login servers to steal passwords, that is how the real world
works. If anyone believes that you can totally lock down access to every
system on your network from your trusted sysadmins and have 100% audibility
and accountability you are unfortunately living in a fantasy land. NSA or not,
this really isn't something that is 100% preventable.

~~~
MichaelGG
Is it? I'd expect a write-only logging system that runs at a low level
exporting logging to remote servers that other admins don't have access to. As
the article suggests, you layer admins so no single person can truly
compromise things.

It should be pretty basic to catch someone that's resetting user's passwords.

~~~
res0nat0r
Sure, you can remote syslog everything to a remote logging service, but:

1] Do you log _every_ single thing that happens on the machine?

2] If not, did you actually log the proper set of commands that were used to
possibly commit a crime?

3] Granted you are not a small startup, do you have a process in place already
to data mine, extract and analyze these potentially dangerous commands from
log sets that are potentially TB in size?

4] Do you have the personnel in place with the qualifications and time each
day/week/month to audit these potentially large results?

5] If you've caught something, aren't you already too late? Do you have some
type of magical software in place that will somehow recognize these types of
crimes in progress and cut off access in realtime?

Many of these articles totally ignore how these real world systems work
(mainly I'm guessing because the authors have never been involved with
companies that run 10k-200k servers around the world). It really is more
complex than you think. Most companies that aren't startup size are
continually playing catch up. Sure they need to put logging in place,
auditing, yada yada. That is being balanced with the other day to day pressing
tasks also. It always seems to be a game of catch up...

~~~
bigiain
Re #1 – at the rate the NSA is reported to be archiving data, do you _really_
think logging every single keystroke that every employee/contractor with any
elevated privileges at all ever types would be even vaguely a problem? TB-
sized logfiles and processes to datamine them are surely interview tasks or
intern familiarisation exercises at their scale?

~~~
res0nat0r
Yes. A previous employer of mine was apparently logging a PB of log data a day
between all departments, it isn't something that is just tossed up and somehow
works in a month.

Do you have infrastructure in place to store all of that? How are you storing
all of this content coherently so that you can go back and audit records of
systems at a point in time you are concerned about? Do you have an internal
Hadoop cluster configured (not trivial) and the hardware to ingest and process
these logs? How are you actually logging _everything_? Do you have dedicated
and qualified people to write this software and maintain it? When someone has
su -'d to root, how are you logging _everything_ they type coherently? Is your
regex smart enough to ingest and contextualize multiline commands as root and
add them to your report? What if no actual "bad" commands on your list were
typed as root? What if I as a luser copied the "bad" commands to
/tmp/kittens.txt, then su -'d to root and ran them there? Is this somehow
captured? Are you 100% sure syslog is running 100% of the time on all hosts
you are concerned about? Can't I as a sysadmin kill the syslog pid before I
commit a crime? Are you using redundant syslog hosts? Are you using TCP and
not the default UDP so that syslog doesn't drop packets under load?

Like I said, this isn't as simple as everyone thinks...

~~~
bigiain
Ballpark numbers - an employee typing at 100wpm for 40 hours a week generates
not quite 70MB worth of keystrokes in a year - and I'd guess that estimate is
probably something like 2 orders of magnitude too high - no-one actually types
100wpm nonstop all day every single day at work. Even ignoring that, I've got
enough disk space sitting under my TV to store (uncompressed) every keystroke
typed in a year by something like 100,000 such mythical "100wpm typists
working at 100% utilisation" employees. 1PB would, to a first approximation
store every single keypress it'd be possible for the rumored 4 million "top
secret or above clearance" people in the US typing flat out for something like
15 years.

So I don't think your "do you have the infrastructure" argument holds too much
water here.

The "do you have internal Hadoop clusters" question falls the same way. Sure,
I don't have that connected to my 8TB of external USB drives plugged into my
media server - but if I worked at Google or FaceBook or Twitter, I'd fully
expect to be able to provision and spin up adequately sized clusters of VMs
and storage to effectively consume and run reports on data that size and
bigger. And it's surely not just the NSA and Google/Facebook/Twitter routinely
dealing with collecting and processing data at that scale - any decent sized
telco, any non-trivial web analytics service, any large financial institution,
every HFT business, most bio-med businesses, probably every physics and
astronomy department at any university – there must be tens of thousands of
businesses routinely dealing with that sort of sized data sets.

It's not simple - certainly not simple enough for me to do it on a Mac Mini
and a bunch of external hard drives – but I also don't think it's anything
like "uncharted waters" territory. (I'm pretty sure I could find the expertise
required in my 1st level LinkedIn connections, and have absolutely no doubt
I'd be able to manage designing, developing and deploying exactly such a
system if someone came to me with a high six or low seven figure budget.)

~~~
mirkules
I think your approximations are focusing solely on keystrokes, whereas the
parent specified just "data", which makes me believe that network traffic,
application logs, etc are included in this. I can believe the 1PB/day number,
it's not that far fetched when you consider the above.

I also agree with the parent. There are so many possible scenarios and things
to log that eventually you're playing a "logging" version of whack-a-mole.
Even just managing these files (as the parent talks about) is really no
trivial task. Honestly, I wouldn't even know how to begin managing a petabyte
worth of _daily_ data.

------
ihsw
Snowden's escape from the NSA exposed two things: 1) _illegal_ kleptocratic
behaviour on part of the government 2) gross security incompetence on part of
the government. The NSA has spent all its time on #1 up until now, so they're
going to hype him up as much as possible so as to diminish the embarrassment
from #2.

> He _wasn 't_ just a community college stooge, he was brilliant! The obscure
> flaw that he exploited has since been fixed, hooray too!

Meanwhile 'sudo su' has been criminalized as a precaution.

~~~
jonhohle
that's why I like to use `sudo sudo su' ;) And of course, `sudo vim', :!bash

~~~
eru
On a more serious note, you should (almost) never do `sudo vim', but stick to
`sudoedit'.

~~~
onedev
I'm curious; what's the reasoning behind this?

~~~
aegiso
sudo vim means vim gets root privileges. Meaning vim can do anything it wants
if it's bugged or somehow compromised. Not likely on an otherwie secure system
but not impossible.

sudoedit (according to the manpage) makes a copy of the file first, lets your
editor edit it, then copies back when you're done -- so even a compromised
editor couldn't do much damage beyond corrupting the given file.

~~~
onedev
That's awesome I never knew that; thanks so much for explaining it!

------
GigabyteCoin
It's this kind of stuff that really scares me.

1) Some government agency builds massive computer system containing lots of
information about the general public.

2) There are numerous obvious holes in the "massive" computer system for
obvious reasons (government's haste, lack of oversight, etc).

3) The government's computers get hacked.

In my opinion, numbers two and three are inevitable when number one takes
place.

Something similar just happened in Canada a few years ago with all of our
driving information: [http://www.huffingtonpost.ca/2012/11/06/service-ontario-
kios...](http://www.huffingtonpost.ca/2012/11/06/service-ontario-kiosks-
ontario-government_n_2081077.html)

~~~
frank_boyd
And it will always be happening, as long as data gets stored.

The only way to prevent it is to not store data.

~~~
jleader
That's not entirely true. Take for example, passwords. If your requirement is
"store passwords securely, so they can only be used to check logins", then you
implement that as salted, one-way encryption. But if you add the shadow
requirement "and be able to retrieve anyone's password when the government
requests it", then your password storage will inevitably be much more
vulnerable.

~~~
dylangs1030
Short of a one-time pad, there's no such thing as truly secure, only
theoretically secure according to time and processing constraints.

I'm not trying to be pedantic, I'm just saying, it's not mathematically
impossible to crack a salted encryption without the cipher key.

~~~
jleader
Agreed, data not stored is more secure than data that's stored, but I stand by
my point that data stored in such a way as to provide the minimal necessary
access can be substantially more secure than data stored in such a way as to
provide that access, plus additional back-door access for spooks.

------
zero_intp
Personal observation: Internal security best practices in spy organizations
are rarely 'overlooked'. It is all about trade-offs. II think the more
important question is "why would the NSA have lax internal oversight on both
user-privledges AND audit-logs?"

The answer is that it is much easier for black bag operations to be scrubbed
from potential oversight when an individual holds the power to run the
hidden|illegal analysis and clean their own log trails.

------
femto
Spring clean time at the NSA? Apart from anything he might have done, Snowden
has provided a convenient dumpster, which may be used to neatly wrap up and
dispose of pesky unexplained incidents.

~~~
gojomo
"General, these logs say your account pulled the intercepts for a bunch of
corporate board meetings, everyone within two hops of Senator Wyden, and
General Petraeus. That was all just brilliant superspy Snowden, right?"

------
beedogs
AKA he typed "su". Or whatever the Windows equivalent is. I do this all the
time to diagnose problems and no one has ever written an article about me. :/

~~~
bigiain
FWIW, there are some things that lots of sysadmins do on a week-to-week basis
as part of their job – that other people have ended up with felony convictions
for…

See:
[http://en.wikipedia.org/wiki/Randal_L._Schwartz](http://en.wikipedia.org/wiki/Randal_L._Schwartz)

(Note: all 3 felony convictions were "expunged" eventually, but for 12 years
he had all the restrictions and problems a convicted felon lives with, all for
"doing his job", or possibly "overstepping the bounds of his authority while
doing his job".)

~~~
letney
Also see:
[http://www.washingtonpost.com/blogs/worldviews/wp/2013/07/30...](http://www.washingtonpost.com/blogs/worldviews/wp/2013/07/30/the-
free-web-program-that-got-bradley-manning-convicted-of-computer-fraud/)

Describing how Bradley Manning using 'wget' was considered computer fraud
because it was not on the list of approved programs.

~~~
krichman
CFAA is a ridiculous mistake. It's like if you used a programmable coffee
machine and the company's policy forbade making decaf you would be a felon.

------
CaveTech
Article is filled with speculation and extremely vague details on his
purported "attack". Seems largely like a filler article to me.

------
joshfraser
If they couldn't detect Snowden, it raises the question of what the Chinese
government has access to. I imagine every government and hacking group in the
world is doing everything they can to get access to that pile of data. Even if
you trust the US government not to abuse these capabilities, what will happen
when that data falls into even worse hands?

------
i386
Unless this audit was independent, can we really trust their analysis? The
NSAs credibility is shot at this point.

~~~
bigiain
Presumably "independently audited" by the Presidentially approved investigator
James "No means Yes" Clapper…

------
Nate75Sanders
"Snowden reportedly used high-ranking official's profiles to troll NSA's
intranet."

trawl?

~~~
advice4u
To troll means to fish. In fact, looking up the definition of "trawl," it
means "to troll"

[http://www.thefreedictionary.com/trawl](http://www.thefreedictionary.com/trawl)

~~~
Nursie
I think it's a US thing though. Obviously I know that in US English the word
has both meanings, but in the UK a troll is something that hides under a
bridge, and the fishing is trawling.

Took me a while to figure out the humour inherent in the double meaning when I
first came across it (in the 90s)

~~~
prawn
I always thought trawling involved dragging a net whereas you trolled with a
lure (enticing fish out to chase it).

e.g., you might trawl for prawns but troll for snook.

~~~
Nursie
Never heard that in UK english.

I just assumed that with an american accent the words sounded the same so
people stopped using 'trawl'... I've been wrong before though.

~~~
thaumasiotes
I can confirm that this is not the case; trawl and troll shouldn't rhyme any
more than Paul and pole, tall and toll, fall and foal, etc.

------
samstave
This appears to be a snowjob against Snowden, and BS -- Look at the language:

1\. ___Snowden impersonated NSA officials, sources say_ __

2. __ _Edward Snowden accessed some secret national security documents by assuming the electronic identities of top NSA officials_ __

3. __ _forensic investigation has included trying to figure out which higher level officials Snowden impersonated_ __

4. __ _if an employee was on vacation while the on-line version of the employee was downloading a classified document, it might indicate that someone assumed the employee’s identity_ __

5. __ _NSA has already identified several instances where Snowden borrowed someone else’s user profile to access documents_ __

6. __ _“The damage, on a scale of 1 to 10, is a 12,” said a former intelligence official._ __

7. __ _The NSA declined to comment_ __ <\--- WTF, then who are the above sources?

[Edit: I wanted to add a little bit of clarity here: the language used is very
vague and references things that could never possible be confirmed: sources
say, "might indicate", "has identified" \--- This story is like a bunch of
paragraphs typed out, randomly put into a hat then shaken onto the floor into
the pattern of the story. It is not a decisive, cohesive piece of information
-- then it is ended saying that the NSA has no comment.

THe TITLE is "NSA finds Snowden hijacked officials’ logins" NSA FINDS....

So, if the NSA doesn't comment - and the "analysis by NBC" and the NSA
declines to comment are all used -- then NOTHING in this piece can be
believed.

Even if the entire premise is true - this is hands down the worst framing of
the information, supposedly factual, one could imagine!

\---

In my informed IT professional opinion, they are using this to brand him a
hacker - and they make a bunch of "what if" type claims. Then they slide into
a confirmed report. Then they claim the damage is off the scale (12 on a scale
of 1-10)

This is a completely MISO built PR piece for the NSA.

As administrator on any system (administrator in Windows, and Root in *nix)
one will have access to whatever you want.

Whilst at lockheed, I had admin rights to every machine and document in my
realm - I would have had no need to "impersonate" any other lockheed
employee...

The mistake here is if NSA was using the same root passwords/keys across
entire tiers of machines. In that case - call it criminal negligence on the
part of whomever architected that disaster.

~~~
vpeters25
My thoughts exactly, they are setting him up for CFAA charges either to pile
up on him or so they can drop espionage taking away the reasons for Russia's
asylum.

~~~
samstave
I edited my post to add some stuff....

But, yes - it is pretty clear that they have the MSM on their freaking sqwak-
box right now against Snowden.

Funny how it was also revealed that they (the UK) were lying about being
"leaked" info from Snowden to the "Independent" \-- where Snowden came out and
said he never communicated to them anything, and the "leak" was a lie and
information that Snowden was specifically avoiding getting out because it was
too pointed at actual personnel...

This info needs to KEEP COMING -- as we need to get to a tipping point where
change is a reality.

~~~
jlgaddis
The "budget PDF" referenced "media exploitation" and my first thought was
manipulating the mainstream media into basically reporting what they wanted
and if some of the taxpayer funds given to the NSA goes into the pockets of
various reporters and journalists.

Maybe the "media exploitation" is what you mentioned in your original post; in
effect, "NSA officials", "sources inside the NSA", etc. all "anonymously"
making off-the-record comments to journalists.

~~~
samstave
Yes. One way to look at it is this:

If the biggest story in the history of the NSA is that some college drop-out
"genius" was able to exfiltrate 20,000 docs from the NSA - and the biggest key
argument they have against the guy, to prove that he were some "hacker" who
was impersonating people to gain access to said files -- how is it that "NBC"
is revealing this with "sources say" "an official" etc...

Don't you think that the most critical point to the credibility of the NSA is
to come out and state via an official means - that they have electronic logs
of this activity?

Digital forensics are binary: Either you have a digital trace of actions
taken, or you do not. Period.

Either they can determine 100% that Snowden logged into station X as person Y
to grab file Z -- or they do not.

THe language used in this article is textbook MISO/Psyop PR.

"Oh, an official source with knowledge of the incident has said.." \-- cool, I
guess they have it figured out then, huh! I shouldn't dare question that. But
if I do: "We can't comment" "we cannot reveal the details of an ongoing
investigation" etc...

Utter crap.

------
kyzyl
To me this simply looks like the folks in charge are locking in their cyber
crimes case against Snowden. If he's ever brought in--for whatever reason--and
even if he magically avoids every charge of espionage, treason, leaking,
spying, misuse of company keyboards, or whatever, they'll have the hacking
angle sunk so deep that it won't matter.

If Bradley Manning got what he did for scary wget wizardry (making no
statement about the validity of that charge or verdict) then I think Snowden
can safely expect more consecutive life sentences than he has fingers and
toes.

------
joelrunyon
Has anyone figured out exactly how high Snowden's clearance went?

They're really non-specific about what he did (and play it off like he
couldn't do anything), but it's coming across more & more like he really had
his crap together.

------
ajw0100
Am I the only one who finds it funny that the article quoted Oracle's
Enterprise Manager docs?

~~~
ozten
Clearly, Oracle is unbreakable.

------
bingeboy
Quis custodiet ipsos custodes?

