
China Mafia-Style Hack Attack Drives California Firm to Brink - ssclafani
http://www.bloomberg.com/news/2012-11-27/china-mafia-style-hack-attack-drives-california-firm-to-brink.html
======
groby_b
"n April 2010, during a 6:30 a.m. check of his servers -- by then part of his
daily routine -- Milburn stumbled on a folder buried in an obscure Microsoft
directory, one that’s normally unused. What he found inside startled him. The
file contained the encrypted versions of all eight passwords in his system --
the keys to the entire network. The hackers could use the passwords to control
just about anything he could, from web servers to e-mail."

Ah. He just decrypted the encrypted versions of his passwords by looking at
them. I see.

And he "stumbled upon" a directory during his routine 6:30am check. Really?
He's manually inspecting servers every morning at 6:30am?

This entire story is several levels of /headdesk.

~~~
willvarfar
If your servers were under constant attack, yes checking your servers every
morning first thing could well be your routine.

You also assume he _immediately_ recognised them. It could well have been
someone one who later told him what he'd found.

Sloppy of the attackers to leave it in plain sight.

------
benmccann
A lot of the comments about this presume that the victim could have avoided
such trouble with better security knowledge. It's worth noting that the
Chinese have hacked Google, Yahoo, Adobe, Lockheed Martin, and over a hundred
other companies, which have many more security resources available to them
than Solid Oak does. No small business in America would ever be able to defend
itself from a sustained attack backed by the resources of one of the most
economically powerful nations on earth. For example, does anyone here think
they could defend against years of attacks from the NSA?

~~~
groby_b
As somebody who made dismissive comments about the victim in this case: No,
I'm not saying he could have avoided a nation state attack with better
security knowledge.

I'm saying that - at least the way TFA describes it - the victim doesn't have
enough security knowledge to tell if he's been attacked by script kiddies, a
nation state, or a roll of toilet paper.

I'm sure the real story is a bit more nuanced, but _nothing_ about the
description gives me any confidence that the allegations are even broadly
aimed in the right direction.

Is China a major vector of security attacks? Yes. Does that mean it has
anything to do with this particular incident? I wouldn't bet serious money
either way, and pocket money against.

~~~
ThisIBereave
"He had no idea who was behind it until last August, when he provided malware
samples to a security firm at the request of a Bloomberg reporter. A forensic
analysis of the malware by Joe Stewart, a threat expert at Atlanta-based Dell
SecureWorks, identified the intruders who rifled Solid Oak’s networks as a
team of Shanghai- based hackers involved in a string of sensitive national
security-related breaches going back years."

~~~
danso
What does that prove? The Bloomberg reporter doesn't show much evidence of
being able to discern what is a valid analysis and what is not. What is a
malware sample? Is it so easy to tell if a malicious script was actually used
by a group, or merely copied by a script kiddie?

To give you an idea of how stunningly bad conclusions can be drawn from
technobabble inferences, I present this:

[http://www.foxnews.com/tech/2012/05/30/powerful-flame-
cyberw...](http://www.foxnews.com/tech/2012/05/30/powerful-flame-cyberweapon-
tied-to-powerfully-angry-birds/)

Powerful ‘Flame’ cyberweapon tied to popular Angry Birds game

 _The most sophisticated and powerful cyberweapon uncovered to date was
written in the LUA computer language, cyber security experts tell Fox News --
the same one used to make the incredibly popular Angry Birds game.

LUA is favored by game programmers because it’s easy to use and easy to embed.
Flame is described as enormously powerful and large, containing some 250,000
lines of code, making it far larger than other such cyberweapons. Yet it was
built with gamer code, said Cedric Leighton, a retired Air Force Intelligence
officer who now consults in the national security arena._

FOX News claims that Angry Birds is a delivery vehicle for a virus because
both of them are programmed in Lua. And who can blame the reporter? Her source
is a former intelligence officer in the Air Force who is likely paid way more
than every developer on HN to hype up such threats.

------
anigbrowl
_Milburn, after all, had built Solid Oak’s network himself. “I thought they
might be able to get around some IT guy, but there’s no way they were going to
get around me,” he says._

/Facepalm.

I was going to write that at least he got a bunch of free publicity from the
article, but on the other hand the takeaway is that product is probably no
more secure than his servers.

~~~
dr_doom
I liked "He taught himself how to write code, and eventually mastered complex
Internet software protocols."

Seems like a lot of the issues would of been solved by moving to AWS. At least
they would of still been making money.

~~~
anigbrowl
Likely. I don't want to be too hard on the guy because I can recognize a bit
of myself in him, but this is a great example of why you should concentrate on
your core competency and pay someone else to do the stuff you're not expert
at.

------
robomartin
I'll preface this by saying that it is all too easy to criticize from the
outside and without all of the relevant information and history.

Once I got to the point in the article where it was obvious that he was under
some form of a coordinated attack I almost wanted to send myself back in time
to be able to go over to his office and yell "AWS! AWS!" in his ear.

In addition to that, it seems it should have been obvious that bringing a team
of experts to help secure the network (or transition it away) and fight the
fight would have been the smartest idea. I would think that the quoted losses
of some $58K per month would have covered this just fine. And, perhaps what is
more important, it is likely that the technology fight wouldn't have lasted
three years.

The other thing that struck me --again, don't have perfect data, don't know
all the facts-- is the apparent lack of help from the likes of the FBI. You
would think that they'd be there in some sustained fashion to help out.

This fellow was out of his league and paid a dear price for it. Hopefully the
settlement compensated for some of it. It sounds like he might have ended-up
with stress related health issues which are no laughing matter.

It'll be really sad if the Internet becomes just another weapon of war. That
should not be permitted. How? Not sure. Is it too late?

~~~
javajosh
_> It'll be really sad if the Internet becomes just another weapon of war._

Anything of value can be turned into a weapon of war. Think about what a
weapon is - it is a tool to disrupt something delicate. A missile destroys a
plane. A knife stops a heart. A virus stops a computer.

It really concerns me that a private US citizen was personally attacked by a
sovereign nation. It seems to me that the US has an important duty to protect
it's people, whether the attack is virtual or physical.

------
ThisIBereave
The HN reaction to this story is pretty sad, I have to say.

Yes, journalists are poor at providing technical coverage. That's not really
their job.

Yes, this guy could have created a more secure network system, but if a
government hacking group comes after you, you will likely not do any better.

------
danso
Ugh...this article reads like a movie about hacking. The story breathlessly
conveys how elite Shanghai hackers toyed with a company's lifeblood, wreaking
such prolific havoc that the owner literally crawled under the server building
to see if a bug had been planted...but the ultimate culprit may have been...

> _Examining the script that controlled the payment processing function in
> November that year, he noticed that a single character was missing from the
> string -- an apostrophe. That was enough to cause the page to time out,
> rather than to complete the credit card transaction. Customers were leaving
> in frustration_

Am I right in thinking that this was all hack via SQL injection?

~~~
frozenport
I think you are missing the point. China will steal your software and higher
people who aim to destroy your lively-hood and life.

~~~
danso
No, I think you're missing the point. If it's true that rival nations will
attempt to hack us for whatever reason, then it benefits us all to have a
better understanding of basic cybersecurity than seeing foreign hackers as the
Hand of God. Instead of examining the geopolitical problems here, this article
takes us through a terrifying cyberstorm whipped up by mystical superhackers
when the real perpetrators might as well have been script kiddies.

The problem is that if cybersecurity continues to be framed in this fashion,
then all that shit that HN continually complains about -- security theater
(via homeland security) and draconian Internet laws (remember SOPA) -- will
continue to be status quo.

~~~
saurik
(I think frozenport was telling a joke.)

------
jakejake
It sounds a bit like the site owner was playing wack-a-mole for three years
when he should have just wiped out everything. Once you have an intrusion,
unfortunately you can't trust anything that a machine has access to and you
should start again from a totally clean install. If this was a huge corporate
network I could understand that may be impossible. But for a small shop with a
few people, there's no reason not to go around and just wipe out every machine
in the company.

If this was a government sanctioned job, though, I imagine they could have
stepped things up if the script-kiddie stuff didn't work. Why bring out the
nuclear weapons if bottle rockets are doing the job just fine.

------
pyre

      > high-tech spies and digital combatants seek to gain
      > a brass-knuckle advantage in the global economy
    

Digital combatants? Does enlistment in the PRC army now include light cycle[1]
training?

[1]: <http://en.wikipedia.org/wiki/Tron_(franchise)#Light_cycles>

------
toyg
Apart from the unverifiable security blunders (this is a Bloomberg article,
not a Full Disclosure post), I have to say I can't feel very sympathetic
towards somebody in the censorship business being harassed by fellow censors.

------
ajays
I stopped reading at this point:

 _Commercial hacker hunters -- who refer to the team as the Comment group, for
the hidden program code they use known as “comments” ....._

/facepalm

~~~
sanswork
Why? It's seriously what they do. The user installs the malware, it goes to a
normal looking web page checks the html comments and receives it's commands.
On the face of it everything looks fine to the user but the malware knows then
what to do.

------
clayrichardson
I'm going to try to suggest a couple of solutions regarding problems
encountered in the article that people might find to be useful to know.

First, some pointers (non-exhaustive).

For workstations: \- Antivirus \- Firewall \- Disk encryption \- Password
complexity requirements \- Updates

For servers: \- Key-based authentication \- Updates \- Firewall rules on host:
explicit ingress/egress \- Run services as non-privileged users in a chroot
jail

Firewall rules for network: explicit ingress/egress

Yes, that's two firewalls for your hosting infrastructure. Attackers would
have to escalate privileges from a service user to a root user in order to
modify the firewall rules, at which point they'll have access to the other
components of your internal network. To get outbound connectivity, they'd have
to gain root access to the external firewall as well. Keep in mind they can
still upload data if they can execute arbitrary commands as the web service
user.

Now, onto more specific items mentioned in the article.

    
    
      >realizing only later that the e-mail address was a couple of letters off.
    

Tell your employees to only accept cryptographically signed email from you.
S/MIME is simple to implement, and works across multiple platforms and
clients. There is even support for it in iOS, Android, and Mail.app!

<http://en.wikipedia.org/wiki/S/MIME>

You can get certificates here: <http://www.symantec.com/verisign/digital-id>

You can even get free ones if you're on a budget! (although they do say for
personal use, implement at your own risk if you're a business!)
[http://www.comodo.com/home/email-security/free-email-
certifi...](http://www.comodo.com/home/email-security/free-email-
certificate.php)

You can do a Google search on how to get these certificates installed in your
mail and mobile clients of choice.

    
    
      >clicked on the attachment
    

Use Google Apps. I'm not sure what the success rate of Gmail's 0-day detection
is with attachments, but I'd be willing to bet that they're better if not as
equally effective as the average desktop antivirus email scanner.

    
    
      >Microsoft operating system
    

The article doesn't say if they used any sort of antivirus or firewall
protection on their workstations. If you absolutely have to have Windows
somewhere in your business, you need to have a decent antivirus and firewall
solution running. Also, make sure all of the latest updates are applied in a
timely manner. IIRC, you can set Domain Policies from Active Directory to do
this sort of thing for you.

    
    
      >automatically uploading more tools the hackers could use to control the network remotely.
    

This could have been addressed in a couple of ways. (non-exhaustive list)

1\. If the malware was listening for incoming connections, explicit ingress
(aka no port forwarding) with a NAT (most residential routers) would have made
it more difficult, as another layer must be evaded to connect to the internal
machines.

2\. If the malware was using a connect-back shell, explicit egress (aka
outbound) traffic would make it difficult to get a shell on a compromised
machine. As soon as you realize you're under attack, you should go into
hardcore lockdown paranoid mode, and only allow network ranges that are
absolutely necessary for business operation.

    
    
      >Then the company’s e-mail servers began shutting down, sometimes two or three times a week, slowing e-mail traffic, the main way the company provides customer service
    

Hosting an email service is a ridiculously difficult task security-wise.
Postfix and Sendmail both have a history of security vulnerabilities, and the
only way to properly host an email server is to harden a *nix system to the
extreme, and always be paranoid. But that's hard and most people aren't in the
business of hosting email services. That said, I'm pretty sure Google Apps has
a great cost/benefit factor when you look at ease of implementation and
security, because the Gmail team is more likely better at server hardening
than you are. Oh, and make sure you backup your email and support PIN, in case
anything goes awry.

    
    
      >Similar problems began plaguing the web servers -- a bigger problem since web sales of CYBERsitter supply more than half of Solid Oak’s revenue.
    

I would have to recommend putting CloudFlare in front of your servers, as
(through their blog posts about several incidents) they have demonstrated an
ability to successfully defend against a large number of attacks. This also
has the benefit of providing a proxy, masking your backend server IPs, which
I'll talk about more later.

    
    
      >Similar problems began plaguing the web servers
      >figure out how the hackers might be behind it.
    

Bro, [http://en.wikipedia.org/wiki/Host-
based_intrusion_detection_...](http://en.wikipedia.org/wiki/Host-
based_intrusion_detection_system)

    
    
      >But the agency shed almost no light on the situation, he says, and he was never told if the material was useful.
    

I'm surprised they didn't offer any advice on how to mitigate attacks such as
these. At the very least, some pointers similar to the ones I'm writing now
would probably gain some sort of ground.

    
    
      >alternating between four different cell phones from three different carriers.
    

This seems a little extreme, if not excessive. I've used burners (temporary
pre-paid cellphones, disposed of after a duration of use) before when I
attended DEFCON (hackers like to do sketchy things with RF) but never more
than one at a time. I'm sure you could probably use something like Google
Voice with multiple Google Accounts or Twilio for temporary numbers while not
exposing your actual number.

    
    
      >constantly had to reboot servers
      >couldn’t trace the source of the network problems
    

If you're going to have to do it more than twice, it's probably best to
automate it. If reboots are required due to a crash, you should probably
identify the cause of the crash (go through logs, etc.) and attempt to rectify
it. If you seem to be way in over your head, bringing in an expert should
definitely be on your mind.

    
    
      >to find that his commercial-grade SonicWALL firewall had failed
      >He spent a good part of the next day on the phone with the manufacturer, who was stumped.
    

If you have any sort of hosting infrastructure, and your expertise isn't
network administration or security, I'd recommend getting (full time, or
contracting) a certified network administrator to help you get your networking
shit together. If I were to pick equipment, I'd probably go with the big dogs,
Cisco or Juniper.

    
    
      >He began writing his own software to monitor the connections his computers were making to outside networks, looking for tell-tale signs of the hackers at work.
    

Bro,
[http://en.wikipedia.org/wiki/Network_intrusion_detection_sys...](http://en.wikipedia.org/wiki/Network_intrusion_detection_system)
Snort: <http://en.wikipedia.org/wiki/Snort_(software)>

    
    
      >servers
      >obscure Microsoft directory
    

Contrary to popular (or at least some peoples') beliefs, it is possible to
harden a Windows server, you just have to really know your shit and be able to
tolerate all the graphical clicky stuff.

    
    
      >all eight passwords
    

If you're running a business, chances are you have more than 8 accounts.

1\. Use unique passwords for each service/account. 2\. Store them encrypted,
in a password manager. 3\. Use keys/certificates whenever possible. 4\. Use
two-factor authentication whenever possible.

    
    
      >The folder was gone two days later, he says, and in its place were several pieces of software he didn’t recognize.
    

You generally don't want to leave a compromised server in production rotation.
Take it off the network and perform forensics to determine how it got
compromised, and how to prevent that in the future. Make sure to wipe the
server and reconfigure it before placing it back on the network.

    
    
      >Net losses averaged $58,000 a month
    

(aside) I'd fix the CRAP out of your security problems for that much a month.

    
    
      >A hacker could certainly edit the script and break it so it wouldn’t work
    

This means they have access to your web servers, and/or to your deployment
process/service. One of the ways you can mitigate direct access to your web
boxes is to use some sort of load balancer (HAProxy) or web application
firewall (CloudFlare). Then only allow connections to your web boxes from the
proxy, as well as a specified IP address to allow for maintenance.

    
    
      >That would be a great way to do it without calling attention to the fact that they were in the system.
    

If you choose to use git as a tool to deploy your code, you can continuously
monitor the codebase for modifications.

I'm quite tired and I think I'm rambling at this point. I hope this is clear
enough for some people to get something out of. I'll probably come back later
and make some edits for clarity/correctness.

------
cnlwsu
from the Analysis that the article claims identified "thousands of lines
copied of code" (sounds like they copied a blacklist):

We found evidence that a number of these blacklists have been taken from the
American-made filtering program CyberSitter. In particular, we found an
encrypted configuration file, wfileu.dat, that references these blacklists
with download URLs at CyberSitter's site. We also found a setup file,
xstring.s2g, that appears to date these blacklists to 2006. Finally,
csnews.dat is an encrypted 2004 news bulletin by CyberSitter. We conjecture
that this file was accidentally included because it has the same file
extension as the filters.

------
rbanffy
So... for how long did the guy keep credit card processing running on
compromised servers? And it took them months to realize that?

------
fkdjs
Hopefully that teaches him to rely on windows.

~~~
69_years_and
And that's what I was thinking as I read this - a guy/team who can write net
filtering software would surely know what side is up. I was waiting for the
part where he set up a GNU/Linix box and just rebuilt his site and perhaps
directing his support email to/via GMail, heck how long would that take to
setup, a week and cost far less than the lost sales and it would keep the
engine running till a more permanent setup was devised. Or just clean
installed to all his Windows machines. So many options and he choose the hard
one. There has to be more to this story.

------
louischatriot
The scary thing is indeed that most software companies do not have security
experts nor the means to hire ones and are basically defenseless against such
attacks. Kind of reminiscent of patent trolls.

tldr version of the article: <http://tldr.io/tldrs/50b6e4acbb22039977000f5b>

------
sown
I hope tptacek chimes in. It sounds like a terrifying situation.

------
wildranter
Why do we do business with those sleazy bags? Just to save a few bucks on our
next gadget? No. I guess we have to thank the Wallstreet bonus whores for
dropping this f-bomb on us. Not even Bin Laden would've done better. Thanks
boys, you rock!

~~~
bruceboughton
That's a mighty accurate username.

~~~
wildranter
Thanks I guess. It's not that hard living up to that these days.

