
Anonymous speaks: the inside story of the HBGary hack - abraham
http://arstechnica.com/tech-policy/news/2011/02/anonymous-speaks-the-inside-story-of-the-hbgary-hack.ars
======
trotsky
Computer security is obscenely asymmetric - an attacker only has to find one
flaw, once, somewhere. A defender needs to constantly monitor, test, review
isolate and basically never make any mistakes.

It is easy to look at almost any intrusion and attribute it to poor defenses.
If HBGary didn't have a SQL injection, they'd have had a XSS vuln. Or a
employee would get spearphished. Or an attacker at a local coffee shop would
compromise a mobile client. Or a backup service would get compromised and
unencrypted. Or a interviewee could plant a network listening device. Or the
CEO's daughter could win a pre-owned iPhone. Or a secretary gives out a VPN
login. And so on and so on.

Did HBGary suck worse than usual? Possibly - but consider Google china got hit
by ie6+acrobat vulns, DOD lost hundreds of thousands of classified documents
from an air gapped and physically controlled system to a private, Open BSD may
have included side channel backdoors, Kaspersky lost their source code,
PS3/iPhone/Xbox/HTC etc. are unable to secure their platforms.

The truth is, a motivated attacker will rarely fail. Anyone reading this would
be unlikely to survive 24 hours of a coordinated attack whether it's done by
16 year olds, chinese university students, russian mafia, FBI or simply nerds
that know how to google vulnerabilities.

Fighting back against a group like Anonymous provides the same asymmetric
warfare problems as the US military experiences in fighting terrorists,
including the inability to respond with similar tactics for legal reasons.

Bottom line is, almost any organization can be subject to this kind of
embarrassment without warning.

~~~
pyre
The 'real' story is that HBGary charges that big bucks to tell other companies
and/or government agencies about how they aren't following security best
practices, yet they themselves weren't doing so. I don't think that anyone
would be ragging on HBGary for lax security if Anonymous had pulled out some
0day kernel exploit to break into HBGary's systems.

They failed in:

\- Keeping their systems patched and up-to-date.

\- Convincing/forcing their users to use strong passwords.

\- Convincing/forcing their users to use separate passwords per system.

\- Convincing/forcing their power users/admins to use a unique, strong
password on key systems (i.e. Google Apps Admin).

\- Not-invented-here syndrome (or maybe security through obscurity -- hey! if
we use an obscure CMS then it won't be exploitable!) with respect to their
CMS. I can be a little lax on them here. Had they chosen 3rd-party software
people would doubtless be railing on them for _which_ off-the-shelf 3rd party
software they were using (e.g. had they been exploited through a Wordpress
vuln, then people would be lambasting them for using Wordpress vs <insert-cms-
here>).

~~~
trotsky
The 'real' story is that a motivated attacker will rarely fail.

You can take almost any intrusion and write it up in wildly different ways.

If HBGary had not failed in everything that you listed, odds are you would be
listing some other comparable set of failures:

\- something somewhere is always unpatched and out of date

\- humans always deviate from best practices

\- 99.99% of intrusions involve traditional threats, well known
vulnerabilities, unpatched systems and human error

I could write up 100 different intrusions done in 100 different ways and
almost always make the victim sound incompetent, or like a real life spy
novel, or make the defenses sound like fort knox, or make the intruders sound
like gods, or make it sound like my product would have prevented them, or draw
the conclusion that the security environment is hopeless and out of control.

In the end it doesn't really matter how it happened or what I make it sound
like.

Bottom line: Did you get owned [Y/N]

~~~
JoachimSchipper
Then again, a motivated defender is a _very_ though adversary. Think "web
server serves plain files only and is disconnected from the internal
network"[1], "secure OSes everywhere"[2], "password quality checker
installed"[3], "full-disk encryption for all serious data"[4], "SSH logins
only via public key"[5], etc. Yes, this takes (some!) real effort. No, getting
0wned by "please drop the firewall and send me the root password" is not
acceptable if you're a security outfit.

Really, you can go years without patching if you choose your software
properly. (Except browsers - those just suck.)

[1] e.g. <https://github.com/mojombo/jekyll> [2] e.g. <http://www.openbsd.org>
[3] e.g. <http://www.openwall.com/passwdqc/> [4]
<http://www.openbsd.org/faq/faq14.html#RAID> or the equivalent for other OSes
[5] All over the internet. Or set up a Kerberos environment and get single-
sign on too.

------
dantheman
Amazing write up - this is one of the best pieces of technical journalism I
that I think I've seen. There is no hype, it's informed, it's detailed - but
not super technical, i.e. math showing password complexity to rainbow table
size tradeoffs etc.

Any journalists out there, this is how it's done.

~~~
jokermatt999
Ars Technica regularly comes out with good pieces like this. Out of all the
tech news sites I've seen, they're generally the most professional (other than
their photoshopped story images; those usually go for humor) and thorough in
their coverage.

------
jgershen
Very well written article - it does a terrific job of explaining things like
rainbow tables for a non-technical (or at least, technically-but-not-security-
minded) audience. The only part that seems off is the theme that /all/ of the
exploited vulnerabilities were necessary to render HBGary vulnerable:

"Even with the flawed usage of MD5, HBGary could have been safe..."

They homebrewed their own password system. Can someone switch on the tptacek
bat-signal?

~~~
there
_They homebrewed their own password system._

the story says hbgary hired an outside company to make this cms for them,
which may explain the crappy security on that particular system.

 _Can someone switch on the tptacek bat-signal?_

thomas' security company also got hacked a couple years ago and had sensitive
information plastered all over a mailing list. rumor was that it happened via
their use of wordpress for their weblog.

i guess the moral of the story is... you will get hacked by crappy third-party
software?

~~~
rdtsc
> the story says hbgary hired an outside company to make this cms for them,
> which may explain the crappy security on that particular system.

Doesn't that make them look even more amateurish and incompetent? They chose
an insecure content management system and, most importantly, they didn't
isolate it enough. So penetrating that resulted in a complete penetration of
their site.

If they were selling hand-made baskets, nobody would blame them, but they sell
"security" and charge big bucks for it, so they deserve the ridicule.

It is an interesting perspective I guess on selling "security", both as a
service and a product. One can charge lots of money, but unless there is a
serious attack and penetration, it is hard to know what the quality of they
security product is. Of course once the penetration happened, there is at best
pity and at worst ridicule and blame.

~~~
iuguy
Not really, they should've tested the site themselves and there's evidence
that this actually happened on the main site but not federal. Normally a
company contracts an external company to do the work for them and either asks
the external company to independently check the security of the output or
organises it themselves. In the case of federal it may have been the case that
neither happened.

> If they were selling hand-made baskets, nobody would blame them, but they
> sell "security" and charge big bucks for it, so they deserve the ridicule.

I disagree. Anonymous were a highly motivated persistent attacker. It doesn't
matter whether or not there was SQL injection involved, they'd just keep on
going until they get in regardless. If there wasn't a SQL injection bug
there'd be something else. Tptacek's company has been hacked into, our website
got hacked into years ago (through having shared hosting - someone else had a
SQL injection bug on the same box and the hackers defaced every site on the
box. The difference is that we did a risk analysis beforehand and decided to
never to store sensitive data there nor use the same credentials for that
account anywhere else). Given a long enough timeline, everyone gets hacked.
While the SQL injection bug was the way in, the real schoolboy error was Aaron
Barr using a weak shared password for Google Apps admin.

~~~
towelrod
> the real schoolboy error was Aaron Barr using a weak shared password for
> Google Apps admin.

I've been reading through some of this HBGary stuff, and I have come to the
conclusion that Aaron Barr is kinda a dipshit.

Read the email analysis at <http://www.wired.com/threatlevel/2011/02/spy/> and
its filled with Aaron Barr "hacking" into people's facebook accounts and then
posting pictures of their kids as if he made some awesome discovery.

------
jmtame
Good TLDR: "So what do we have in total? A Web application with SQL injection
flaws and insecure passwords. Passwords that were badly chosen. Passwords that
were reused. Servers that allowed password-based authentication. Systems that
weren't patched. And an astonishing willingness to hand out credentials over
e-mail, even when the person asking for them should have realized something
was up."

------
bretthopper
HBGary isn't anywhere near the only company to have security holes like this
open. It's just worse because they're a security company and they happened to
piss off Anonymous.

Getting employees or users not to reuse passwords is probably the hardest
thing to do.

Also, Ars' coverage of this story has been great.

~~~
rst
One point in favor of requiring ssh keys for external access is that the users
don't get to blow it on passwords. Though it does require sysadmin staff who
are willing to walk users through the process of creating the keys --- and
stubborn enough to explain that this _is_ the procedure until following it
becomes the path of least resistance.

~~~
tedunangst
Changing passwords is a lot easier than changing keys.

~~~
cookiecaper
Also, passwords are compromised much more often than keys. To get someone's
SSH key you have to have access to their local workstation, which is probably
going to be more troublesome than access to a colo'd server, if for no other
reason than most workstations go to sleep after they've been inactive for a
while (there are other reasons, though).

Also, changing keys isn't that hard. You just re-run ssh-keygen and delete the
old key from authorized_keys and replace it with the new one.

~~~
tedunangst
replacing the private key is the hard part. it's the kind of thing where you
don't discover that the new private key for your server isn't on your backup
laptop until you need to login and don't have access to a system with they
key.

------
ahrens
<rant>As others have said, the most serious thing is that it is a security
company, and they seem to have ignored EVERY security best practice ever! Did
they do anything right?

As a security specialist myself, I always do my very best to follow best
practices, not only to protect myself, but to show others that I am willing to
follow my own advice. I have met security people doing presentations that need
to elevate to admin, and they are logged in as admin already. Sometimes even
with UAC turned off. This completelly shatters my confidence in them. I am
always logged in as normal user and have long (20+) passwords, and make sure
people see that I have long passwords. I don't hold others to the same
standard, but I know people. If they see that I use 20+ characters, they will
not think that the 10 that I want them to use is that bad.

This is the way the security landscape should work. We should all set
ourselves to much higher standards than the advice we give others. We should
always follow up on it. We KNOW lazyness will cause breaches, so therefore we
should never be lazy when it comes to security. For a security company - and
especially the president - to have such low security lowers the confidence for
the whole industry.

Yes, security is asymmetric. That is why companies must always follow at least
the recommended best practices. If they are followed, the target might be too
hard to break into and a hacker might go someplace else where it's easier to
break in. Targeted attackers might still get it, but we should all make sure
they have to work DAMN hard to succeed! If we start thinking that the
attackars will succeed anyway, we might as well drop all defences. Display the
admin passwords at the bottom of the "About us"-pages.

Bottom line, HBgary fucked up good. They showed the world that they give
advice that they don't follow. They deserve the burn and anybody thinking
about hiring them should think again. Even if they change the problems that
allowed this breach, the basic problem is that they obviously don't understand
security. If they did, none of this would have happened.

</end rant>

------
waterlesscloud
Wow. Did they do _anything_ right?

I can understand a typical organization making most of these mistakes, but a
security firm?

~~~
m0nastic
I don't mean to shatter your dream of how security firms are run, but on the
whole, I'd bet we're no better than the industry at large.

This might be a "cobbler's kids shoes" issue, or just a general failure of
people and process.

One of the only truisms I've found so far when dealing with breaches is that
almost no one gets this right proactively. You almost have to be the victim of
a breach (the more public the better) to actually rethink how your
people/processes are implemented.

This seems true for the largest banks in the world, and the smallest security
firms.

And it's easy to look back in hindsight and say "how could they have possibly
had things set up that way?", but the truth is that this was not an
opportunistic attack; if these vulnerabilities weren't present, then they'd
have looked for others.

That's the problem with securing your environment, you have to get everything
right, and the attacker only has to get one thing right. That being said, as
with most "disasters", this was a series of cascading failures (like most
airplane crashes, or oil rig explosions).

Hopefully they'll learn from this (assuming the negative fallout doesn't
completely bankrupt the company).

~~~
stcredzero
_I don't mean to shatter your dream of how security firms are run, but on the
whole, I'd bet we're no better than the industry at large._

If it's ever possible for me to hire a security firm that has higher standards
than this, I'm going to do that!

~~~
t3chg1rl
I agree with stcredzero. You need some standards if you're going to put
yourself out there as a security company. I'm a random chick with some web
programming and I know that you should iterative hash or salt your hashes. I
also know you shouldn't use the same passwords, and what sql injections
attacks are. Hey, maybe I should start a security company!

~~~
stcredzero
It's more of an issue with, "do they practice what they preach?" "Do they eat
their own cooking?"

When people at a company don't do this, it's often a symptom. A friend of my
girlfriend worked at an AT&T store. She could've gotten a huge discount on
AT&T mobile? Her answer: no thanks.

------
iuguy
This is a very well written summary of how it all went down. From this, we can
conclude:

* An initial entry point through SQL injection (from which we can infer that the federal site was probably never security tested)

* The use and re-use of weak administrative passwords

* The adoption of poor practices (such as sending passwords in cleartext via email) at rootkit.com

Of all of these, the attack would've probably been limited to the federal site
had Aaron used decent strength administrative passwords (or just not reused
them). The issue regarding the website comes down to risk ownership. If Aaron
Barr was the risk owner for the website, it falls down to him. If in his
contracting the company, Aaron had a plan to test the website then whomever
tested it missed the SQL injection (which could've been implemented after a
test by the CMS firm without him knowing, or any number of things). If Aaron
didn't have a plan to get the site checked out, then that's both strikes
falling on his shoulders. The third strike is quite clearly stirring the
hornets nest.

Moral of the story: Don't antagonise groups that will strike back without a
plan to deal with it.

Moral #2: Don't reuse passwords.

Moral #3: Retest your app on every significant change (including initial
deployment).

------
wh-uws
SQL injection and MD5... on a "security" company? In 2011?

I'm sorry but thats just egregious. Thats like being a bodyguard and not even
putting a lock on your own house.

The rest of the attacks could have happened to anyone. We all know its best
practice to use many different passwords but most don't because its more
convenient to only have one or a few. And if the email is coming from the
email address it should you could brain fart and give up the info without
thinking.

But the first two parts of the attack should NOT have been possible for them
to even pretend to call themselves a computer " _security_ " firm in 2011

~~~
hysterix
You are completely wrong to justify people using the same passwords in
multiple places because it is convenient.

~~~
RyanMcGreal
Understanding a behaviour isn't the same as justifying it.

------
OoTheNigerian
I just took an introductory course in web security. This explained many
terminologies that fly over my head on HN.

------
cookiecaper
This really makes the case for much more public-key cryptography everywhere --
if all of the emails between HBGary, even internally only, were encrypted,
HBGary would have gotten out with just a small DDOS and been meandering along
just fine today. I think that people that run a computer security company
should at least be able to figure out Enigmail.

~~~
askar_yu
not sure how e-mail encryption would have helped... ? They SQL injected and
got the DB, obtained the passwords and then proceeded further (social
engineering: FW policy change, ssh password through e-mail, etc.)

~~~
firebird84
If you protect your keys well enough, you can protect the content of your
emails. If they're stored on an IMAP server, downloading them will do you no
good without the keys. Additionally, compromising a single machine may only
yield the key to some subset of a company's emails.

------
Splines
_One: the root password to the machine running Greg's rootkit.com site was
either "88j4bb3rw0cky88" or "88Scr3am3r88"._

There must be more to it than this. If you know it's one of two passwords, why
bother asking - couldn't you just try both? (In retrospect, maybe it was to
give Jussi confidence that he was communicating with the real Greg? [Who else,
after all, would know the root passwords?])

~~~
weego
That was the root password, and remote root was disabled, so they had to
social engineer an account that could remote in over ssh so they could su to
do the real damage.

------
CaptainZapp

      HBGary used Google Apps for its e-mail services
    

So, this high flying super duper high tech hardcore company actually uses an
external email provider for their (I suppose) super secret emails?

The more I learn about this incident the more Mr. Barr and HBGary look like a
bunch of amateurish dolts that may give good power point presentations, but I
for one sure wouldn't take my security business there.

~~~
iuguy
We use Google Apps for our e-mail and we're a security company not entirely
dissimilar to HBGary (the main company, not federal, and in terms of services,
not practices).

We moved to Google Apps as they bought Postini. We had an adult discussion of
the benefits and drawbacks, and as everyone had PGP made it a straightforward
job of encrypting things according to policy. Google Apps (for business, at
least) offers SSL encryption on everything and offers relatively little by way
of additional risk compared to using someone like messagelabs for your AV.
Obviously they're storing the data for you, but you'd get that with any hosted
provider.

------
djm
FYI, it looks like Anonymous is aiming for total humiliation as far as HBG is
concerned - they have even set up a pretty web interface/search for the entire
collection of stolen emails - see <http://hbgary.anonleaks.ru/>

------
bane
Relevant: <http://news.ycombinator.com/item?id=2227181>

