
How We Got Owned by a Few Teenagers (and Why It Will Never Happen Again) - cardmagic
http://blog.phpfog.com/2011/03/22/how-we-got-owned-by-a-few-teenagers-and-why-it-will-never-happen-again/
======
sriramk
I feel really bad for the phpfog guys. But given the situation, I think they
handled it admirably well - kudos to them. No software is secure and this
could have happened to anyone. Especially startups who have to take shortcuts
at the very beginning.

I know the attackers were just kids but I have to admit pursuing legal action
sounds very tempting - even to just act as a deterrent to others. If they had
just put up phpfogsucks.com, it _might_ have been ok. But tweeting trash from
their twitter account, redirecting their root domain to phpfogsucks, etc - are
all not cool at all and should have _some_ consequences.

~~~
dolinsky
What I find most disturbing about this whole situation is the way in which
these teenagers are handling themselves, especially after the fact. The
continued denial of responsibility and half-hearted mea culpa, coupled with
the monetary damage to those businesses who had been running on PHPFog, leads
me to sincerely desire that these teenagers face a penalty of some magnitude,
not just a slap on the wrist.

Maybe then they'll stop with the half-assed apologies and recognize that
there's a right way and a wrong way to do things.

~~~
parfe
_Maybe then they'll stop with the half-assed apologies and recognize that
there's a right way and a wrong way to do things._

PHPFog built a castle out of sand and you're upset that a wave came and
demolished it. I'm always surprised at how thin-skinned a lot of HN commentary
is. "Oh, Zed shouldn't be so rude" "These kids' lives should be destroyed for
playing games with an wholly insecure website." "I stopped reading that
article because it used the word blowjob."

I don't get angry at my dog when he shits in the house. Being angry at
something that can't understand only satisfies the urge to shift blame.

My dog shits in the house and it's my fault for not walking him sooner. If
some children compromise every level of your company then getting mad at them
is only trying to deflect the blame. PHPFog is the only responsible party in
this mess. I feel for the customers who still trust them.

~~~
lambda
This isn't a wave knocking over a sandcastle or a dog shitting in the house.
These are 16 year old kids, old enough to know right from wrong, and with the
knowledge and skills to exploit the system. And once the exploit worked, they
didn't then responsibly disclose the problem to PHPFog; they started
vandalizing, changing passwords, and the works.

This is like someone finding an unlocked door to the apartment building's
maintenance office, taking the master keys from there, rifling through a bunch
of people's personal belongings, sticking signs in the windows saying "this
building's landlords suck," and changing the locks on some of the doors to
make it hard to clean up the whole mess.

They absolutely are the responsible party; you should never blame the victim
of a crime just because the victim didn't take adequate steps to defend
themselves. If I accidentally leave my door unlocked one day, that does not
make it suddenly OK to come in and take my stuff and it's my fault for not
having locked my door, instead of yours for taking my stuff.

Now, in this case PHPFog does bear some responsibility, because they have a
duty to protect their customers as well as possible, and from reading about
how this happened, it sounds like they were amazingly sloppy and irresponsible
about it (passwords stored in the clear on the server, passwords shared
between various accounts, leaving unsecured shared systems running after beta
launch, etc). But that doesn't reduce the culpability of the attackers; they
acted maliciously, with full knowledge of what they were doing, vandalized
systems, changed passwords, and bragged about it.

~~~
PostOnce
I'm now nearly 25, and the amount I have changed since I was 16 borders on the
immeasurable. Teenagers are glorified children. We seem to forget how little
reflective capacity we all had when we were teenagers.

If I were 16 and hacking some stupid website, I would think, sure this is
"wrong", and I might get in trouble, but I probably wouldn't think that it
would matter 5 years from then, or 20. Truth is, a criminal infraction can
fuck your life up worse than is deserved. You could be forbidden to immigrate
to your wife's home country, denied any worthwhile job you can find, or spend
a sizable portion of your life in a smaller cage than they house animals in.

There's a reason juvenile law exists, but we forget it. We charge 12 year olds
as adults. This isn't _that_ different.

Hell, where I live, a lot of employers will check the sheriff's website for
public arrest records. These aren't convictions, they're arrests. People are
being denied jobs based on accusations.

I've never gotten so much as a speeding ticket, but even I realize what a load
of shit this all is.

~~~
billybob
So when they turn 21, a "be responsible" switch will magically flip in their
minds?

The job of parents and society is to teach children responsibility. That means
having consequences for your actions. And the closer you get to adulthood, the
more adult those consequences should get.

When I was a teen, some real estate developers tore down a bunch of woods
where I had always played and started building a house. I was ticked and I
vandalized the construction site. But my crime was discovered, and I had to
work carrying lumber in the hot sun and scraping glue off windows to pay back
the damage. Why did I have to do this? Because the developer talked to my
parents. And my parents made me do it.

This was light punishment - I wasn't taken to court, and I didn't get a
criminal record. But parents are to children as legal system is to adults:
they set the rules and enact the punishments. If they don't, someday the legal
system will have to address their failure to do so by locking up their grown
children who never learned right from wrong.

I'm not sure what the consequences here should be, but the argument "they're
kids, they can't be held responsible" is silly. If they're not expected to be
responsible, they won't learn to be responsible.

~~~
intended
What you're saying is fine. Its probably the better adult way to handle it -
talk to the parents. Other people are suggesting FBI/Criminal law. How would
you have fared/grown up if they sent you to court?

Do note that since he is tech oriented, he probably now knows what a S __*
storm he has kicked up - the blog response pretty much ensures that he is
aware. I assume you would be sweating bricks if you knew that the FBI was
coming after you AND that you had hand delivered a bullet point confession. I
assume thats a pretty strong deterrent (in his specific case)

(Incidentally fighting to stick up for your woods was a kinda nice thing to
do, modulo the amount of vandalism you pulled off.)

------
nodoubt
The blog post is riddled with the words "luck" and "timing" which brings doubt
into my mind that the team can actually take full responsibility for their
actions.

"aware of the potential security threat " but they left it for the next week,
who honestly here would do that?

I have also seen comments around the web of migrating to Php Fog because of
how they handled the situation. If you are one of these people please
enlighten my mind as to how you came to such a logical decision or how much
you get paid per year.

Also if Php Fog could enlighten us on how their terms of agreement will work
in the case where our intellectual property is stolen on no fault of our own.

Save your sympathy for the sites that are still down, four days and counting

~~~
wooter
I couldn't agree more. The phpFog team cut corners to deliver quickly. We
(devs) all do it. The important part is to clean up after yourself.

The whole blog post seems a bit melodramatic. I mean seriously, who here
hasn't spent 3 all nighters in a row fixing a mistake? sack up and do what you
should've done before deploying other people's data.

...and who would seriously sue these kids? they handled it poorly but they're
smart (definitely smarter than i was at 16) you're lucky it was curious kids,
rather than malicious (and experienced) hackers that would've been harder to
catch. Do you really want to burden them with a criminal record for life?

~~~
Kudos
Am I the only one who becomes functionally useless after the first 24 hours?
I'm nearly 30 now, but the maximum I'd have done 10 years ago was 36 hours.

------
eel
I am bothered by some of the language in this post:

\- _we were aware of the potential security threat behind post-deploy hooks
and were about to disable them [...] but..._

\- _we were days away from replacing this server_

\- _They were a short-term stopgap measure we had been planning to replace_

To me, it sounds like the real problem could have been stated as "We were lax
on security," but almost worse than that is the lack of accountability that I
sense from company. Yeah, maybe it won't happen again, but it's hard to be
full of confidence to buy into a service like that.

~~~
CGamesPlay
The article starts out in that tone, but it changes to be pretty remorseful
after that. For example the "why it won't happen again" part.

~~~
krobertson
I agree with parfe's comment below.

They sound incredibly laxed on security and the "we were days away from fixing
it" could be complete bull. To Lucas, it probably sounds better to say they
were close to fixing it instead of admitting they were unaware of these
exploits.

I find the disclosure in the blog post great, but the conditions they had
leading up to the hack very disappointing.

If they were aware of the exploits, they should have taken quicker action.
They'll probably be focusing on security big time now... they have no other
choice.

------
geekfactor
"We have hired professional white hat hackers with government level security
experience to attempt regular pen tests on our system..."

I guess whenever I read this kind of statement from now on I'll be thinking of
HBGary and chuckling a bit inside.

~~~
bmelton
At the risk of that comment being taken as a joke, I've done a lot of work
with the federal government, and I can assure you that while the level of
hilarity that HBGary has generated, the typical level of talent in government
cleared individuals is not necessarily great.

I don't mean to impune the capabilities of the people involved (I don't know
who they are,) and it isn't to say that you can't find some AMAZING talent in
the government realm, but as in all fields, it's the exception, not the rule.

~~~
jarin
Reminds me of when I was in the Navy and I went to Navy Security and
Vulnerabilities Technician school. I was all excited, so I went out and bought
a copy of Hackers Exposed and read through the whole thing, learning
everything from how to determine what family and version of operating system a
computer is running by what ports are open, to how a buffer overflow attack
actually works.

Fast-forward to the class, and we're sitting there running tools like
BackOrifice that exploit vulnerabilities that had been patched for _years_ ,
and learning that a SYN flood is "a malicious attack". That's it, just "a
malicious attack". When I asked about the difference between a SYN flood and a
Christmas tree attack, I got a blank stare and "they're both malicious
attacks".

I spent the rest of the class in the back of the room, reading the Armadillo
Book.

Also, I did not once in my brief Navy career get to hack an enemy computer.
Hugely disappointing.

~~~
mattdeboard
Yeah, military schools on scientific/technological subjects can be very
disappointing. Not all of them, obviously, but computer science topics get
distilled down to be accessible to the bottom 20% of attendees. Having spent
over a decade in the Marines, I learned not to get excited about anything like
that... unless the schools were taught by civilians.

------
citricsquid
I mentioned this last time, but I don't think anyone was interested, but the
"John" guy is compwhizii (same handle on Twitter) who runs the forums
(facepunch.com) for garrysmod, a very popular game. I will be curious to see
how garry (owner person) responds to this, or if he already has.

Elliot is apparently VERY scared and blames John (compwhizii) (edit: not john,
he blames someone else called supersnail1):
[http://www.facepunch.com/threads/1071855-A-member-of-
Facepun...](http://www.facepunch.com/threads/1071855-A-member-of-Facepunch-
may-cause-me-to-be-sued)

Here is (compwhizii) Johns reply:
[http://www.facepunch.com/threads/1071855-A-member-of-
Facepun...](http://www.facepunch.com/threads/1071855-A-member-of-Facepunch-
may-cause-me-to-be-sued?p=28754506&highlight=#post28754506)

~~~
nbpoole
And here's Elliot's "official statement": <http://elliotspeck.com/phpfog.html>

And for anyone who missed it, here's what Elliot posted in the previous HN
discussion about the phpFog breach:
<http://news.ycombinator.com/item?id=2346161>

~~~
jedsmith
_Hi, I'm <full name>! My site says what city I'm from. I've written publicly
to admit that I committed multiple crimes, definitely without consulting legal
counsel first! I even put them in a nice bulleted list that can be copied and
pasted right into a complaint. They're pressing charges, but that's bullshit.
I'm 16!_

Not too bright, are we? Instant message the company you just hacked and bust
out from behind your handle, then provide evidence for the prosecution in the
form of a Web page? What is with kids these days?

It only takes one episode of Law & Order to figure out how to proceed here.
Clue: Attorney.

~~~
trotsky
Somebody really needs to get in touch with that kid's parents and let them
know that they need to find the mute button on their kid and retain counsel.

After reading that I think it's a pretty strong argument against those
claiming adult status for him. He clearly doesn't understand the situation
he's in or the second order impacts of what he's done.

------
tjarratt
The phpfog guys really deserve praise for being so open on this issue. As a
fellow engineer, being able to learn from their mistakes and see exactly what
they could have done _ahead of time_ to avoid the disaster is priceless.

Just goes to show that those with the time to spend are the most likely to
break your stuff, even if you pay "professional white hat hackers" to test
your system.

~~~
mtogo
On the contrary, they knew of security vulnerabilities and intentionally left
them unpatched, then blamed it on chance and timing when they got owned
because of it.

Avoid phpfog if at all possible, in my opinion.

~~~
tjarratt
If I lost $1 for each time I left a known vulnerability unpatched because I
was convinced I had more important work to do, I would be a very poor man
indeed.

Honestly, there are very very few developers that fix security problems in
beta environments before anything else. In my experience, it's more likely
that you're fighting fires, handling outages, and dealing with problems of
scale than fixing security vulnerabilities.

Besides, isn't a beta the correct time to find these security issues? (Design
/ Alpha would be the ideal time, granted, but sometimes that's not possible.)

------
noonespecial
It seems like incredible coincidence that allowed this to happen but when I
think back to all of the security incidents I've been involved in, it always
seems this way.

I guess the best way to think of it is that badness on the internet is like
water. It will flow into every tiny crack in your wall you haven't sealed up
tight. A crack in a dam doesn't leak less because its in an "obscure"
location.

------
Aaronontheweb
Goes to show you why the DRY principle (I might be stretching that analogy
here, but bear with me) is important here - if you have old stuff lying around
in production that was cloned a long time ago, you might forget about it and
open yourself up to unfortunate incidents like this.

PHP Fog is doing great work to make the PHP ecosystem easier to work with, and
I hope they didn't suffer too much from this mistake.

~~~
tjarratt
Still stretching the analogy, but the same could be said of their password
reuse.

~~~
uxp
That's exactly what I thought of.

DRYP - Don't repeat your passwords.

------
brisance
While it is admirable and good that they have learned from their mistakes and
are taking steps to reduce the likelihood of getting hacked in future, to say
"never again" is to paint a big red bullseye on yourself.

------
tzs
Wait...their model is an EC2 instance per customer? The normal limits Amazon
imposes are 20 reserved or on-demand instances and 100 spot instances per
region. You can request more, but will Amazon really accommodate a one
instance per customer model?

~~~
jasonwatkinspdx
Amazon is happy to. The limits you cite are merely the point at which you need
to have a conversation with Amazon staff. They are quite happy to accomodate
_much_ heavier usage from customers.

~~~
zmmmmm
When I asked for a raise to my limit they denied me, on the basis that my
usage was insufficient. Of course, my usage was low because I hadn't launched
my product fully because I didn't have enough instances to serve a lot of
customers ... catch 22.

So I had to build out a rather convoluted architecture that used the loophole
of deploying to multiple regions and failing over to whichever region would
give me an instance ... which gives me up to about 80 instances ... just
barely enough for me to get going with a trial beta program.

Which is all just to say, it is _slightly_ more than just a "conversation"
that you need to have to get a higher limit.

~~~
jasonwatkinspdx
Sorry you had a bad experience. I have gotten nothing but superb support from
Amazon staff. I'm surprised they weren't willing to accommodate you, doubly so
if you were ready to pay.

One thing that surprises me is when people talk about utilizing multiple
availability zones in EC2 as some sort of burden. It's very clear from their
documentation and architecture that you need to be capable running in at least
2 availability zones regardless if you want any sort of availability.

~~~
zmmmmm
> people talk about utilizing multiple availability zones in EC2 as some sort
> of burden

My use case isn't for an ongoing server where you require availability. It's
purely about compute power - I don't care where the compute power comes from
but preferably I want low latency to my customer. So ideally I would just get
all instances for any given customer from a single region.

I did find in the end that, as you say, I would sometimes not be able to get
an instance in a region even when I was below my 20 limit for reasons internal
to Amazon, so the failover work was going to be something I had to deal with
anyhow ... but it just added complexity to my life earlier than it would have
otherwise.

Edit: I would also mention that I certainly don't think of it as a "bad"
experience. I think it is something of a small miracle that Amazon offers the
service they do in the first place and I certainly understand why they have
caution about handing out large limits to just anyone. I only made my comment
above as a kind of caution to not just assume you're going to get a raised
limit from Amazon immediately and especially don't leave talking to Amazon
about it until the last minute if you're planning to launch something.

~~~
jasonwatkinspdx
Thanks, that's an angle I haven't seen since I've mostly used EC2 as a hosting
service. What little batch work I've done on it hasn't involved instance
counts where I ran into limits. (Now, EBS i/o limits and other things... :/ )

------
drivingmenuts
Leaving the doors to your house wide open does not grant every passerby the
right to enter.

So, yeah, PHPFog screwed up and did that. Then these kids went in, threw paint
on the walls, smashed some windows, etc.

PHPFog was stupid - they admitted that.

The kids were criminal.

The first is not illegal - the second is.

------
pdenya
What a crazy story. If the timelines are accurate there was an extremely small
chance of this happening. Bad luck all around.

My site is still down, guess i'm in the unlucky 1%.

~~~
greendestiny
Yeah but the problem with 'things planned for the near future' is that they
have a tendency to stay in the future until something like this happens.

~~~
16s
Yes, and after the fact, they sound like excuses (even when they are true and
they are not meant to be excuses as in this case).

------
djcapelis
Ugh, you shouldn't try writing an apology after not sleeping for days. Sleep
on it first, always sleep on it. Talking about prosecution and explaining this
with a framing that it was all a fluke caused by the only person who was silly
enough to IM you with a confession... add one more person who will never be a
customer of yours with an apology like that. Now I know you're irresponsible.

Seriously don't write official blog posts for your company while you're
experiencing "I was just in the field for days trying to fix this stuff"
emotions.

Calm down, then try and be graceful about the fact that you were hacked by a
few clueless kids. (Clueful kids don't let you know who they are.) Then try
and figure out how to protect yourself against people with a clue.

------
Stormbringer
Wow, that is quite the list of security measures that they had almost but not
completely/correctly implemented, or hadn't got around to yet.

I guess the real moral of the story is to finish what you begin, or don't keep
putting security off until it is convenient for you.

------
jschuur
Never? I would be cautious about issuing a challenge like that.

~~~
biot
They might still have a security hole big enough to drive a freight train
through, but that specific attack will never happen again given that they've
shut down the shared failover server.

------
samjohanssen
Congratulations to PHPFog. They've managed to direct the attention to the 16
year old kids rather than their own incompetence.

Is it me or no one mentions the lack of expertise of the PHPFog team in PHP
and Systems Administrations.

Sure kids broke in and the way they published their findings was despicable.
The fact remains that PHPFog was utterly broken to pieces and the exact
essence of the problem is simply the lack of knowledge in their field.

I am very disappointed by the tone of the blog post and think PHPFog don't
really have a notion of what they are doing. I would much rather seem them
where they belong, in the Ruby world where their experience is.

------
intended
Their response and abilty to turn the situation around is a case study in
dealing with a difficult situation. Kudos! I'm saving their response and will
use it when dealing with things. Being able to have a counter party to
identify has definitely helped in handling the situation. I didn't realize how
powerful that can be until I saw this, I learnt something new.

Its a brilliant piece and a great start/way to restore faith and recover from
what must be a pretty grueling ordeal. Good job.

------
nethsix
Great to see disclosure. This can happen to anyone, and more so for startups,
where labor is short, focus is on developing features. Using the phrase "Never
Happen Again" is a bit strong though. Security is risk management; spend until
you can accept the remaining risk while still maintaining profit and avoid
being a hacker's low-hanging fruit.

------
rexreed
This post convinced me _not_ to use PHPFog. They reveal more in their lack of
foresight and security prevention measures than their response to what was
otherwise a fairly trivial exploit. I am not sure this blog post was helpful
in convincing customers like me that want to feel that their infrastructure
providers are on top of things.

------
Popcorned23
Here's an interesting tweet from one of their developers.

<http://twitter.com/ReinH/status/50348989366796288>

> Your password in the database is SHA512 encrypted, but we're not taking
> chances.

I hope he knows what he's talking about and is just tired from the past few
days.

~~~
16s
Let's hope they are salted and iterated.

~~~
jarin
Or swapped over to Bcrypt

~~~
lyonheart
we've cleared the old SHA512-salted passwords out of our production database
and have upgraded the password hashing to bcrypt, with a cost of 10.

~~~
jarin
Good call :)

Just in case anyone doesn't know why Bcrypt is so awesome, it's because it
actually takes _longer_ to hash (based on the difficulty level you set, and
you can bump up the difficulty level as hardware gets more powerful).

For other applications, you want hashing to be fast. But for passwords, you
want hashing to be as slow as possible without compromising user experience.

~~~
lyonheart
I wanted Lucas to link to Coda Hale's post on bcrypt (found by googling "Coda
Hale bcrypt") in the blog post, but he edited that out. So it goes.

------
zaidf
I remain in two minds about idea of charging the kids.

There is no doubt they did some things they should not have. And I don't doubt
there can be a decent case built against them. But as someone who actually had
something from his teen years come to bite years later, it's not pleasant. At
least in my case it was a MAJOR maturing moment(also the worst day of my
life). May be it will take a lawsuit to get these kids to mature up...to that
extent anything that gets em to mature up before they _really_ get screwed
would be fair.

I'm not merely advocating another chance but actually something that gets
these kids to be a tad more thoughtful about their actions. It's not always
easy to do that when you are 16 and full of adrenaline.

------
skbohra123
I am sure, many of the HN users here would have found at least a loophole in
similar systems in the course of time. What I do in such situation is letting
the service know about the flaw. Isn't that the ideal behaviour ?

------
benatkin
> Eliminate shared hosting failover server – We may never do shared hosting
> failover again if we can not guarantee its security. We might do a non-
> realtime failover to automatically launch a new instance for you, but this
> experience taught us what a bad idea this can be.

What does _realtime_ mean in this case? Anyway, this isn't the only option.
They could keep a few bare instances of their php stack online and simply run
the deploy script instead of the image creation script. That ought to be able
to run in under ten seconds I think.

~~~
jasonwatkinspdx
Realtime means a request in flight is retried if there's a failure. Said a
different way, the load balancer already knows about a spare at all times.
This is compared to a ~5 minute downtime if spawning a replacement EC2
instance is required.

Nice idea on the hot spare instances however.

------
hinathan
This feels like a business model where the lean/MVP approach isn't quite
appropriate. A lot of things fall out of that decision, not the least of which
is that the exposure surface area you get from an environment that allows
user-sourced code on purpose is enormous. I feel for the guys going through
this but there were a lot of errors in the wild all at once to allow this to
happen.

~~~
gsharma
IMO, in this sort of business it is important to define the right MVP.

------
RobMcCullough
There is no such thing as bad publicity! Kudo's for turning lemons into a
viral blog post! Although, if I understand correctly, you were reusing
passwords and storing them in plain text! This is an ABC123 computer security
nono. Thank goodness it was just some young script kiddies and not someone
with malicious intent!

------
dashr
great to hear all the details so quickly so that others building similar
systems aren't in the same situation. as fellow PHP'ers its also great to hear
that you are not blaming it on PHP somehow (no fuel for the php haters).

------
getsat

      2:56:45 AM Elliot : then I used the method detailed by turby
      2:56:46 AM Elliot : to gain root
    

Has anything been said about what this method was?

~~~
nbpoole
If it's what was on the PHP Fog Sucks website, it worked something like this
(I may be misremembering a step):

1\. Use the post-deploy hook to chmod _/home/ubuntu/.ssh_ so that it could be
written to.

2\. Upload a PHP shell, use it to write your public key into
_/home/ubuntu/.ssh/authorized_keys_ , and get the public IP of the EC2
instance.

3\. SSH into the box, _sudo su_ will get you root.

~~~
getsat
Ok, thanks. I was wondering if there was some other root exploit at play, not
just bad system configuration.

------
pdaviesa
So, shouldn't the first thing you learn as a hacker include how to mask your
physical location so as not to have the FBI knocking on your door?

------
teyc
I feel for the people at phpfog.com, but this is a bigger blow to cloud
computing.

Customers who are already pretty risk averse to their data being stored in the
cloud would see this as another reason not to take the risk.

The cloud computing consortium needs to work on a stable stack as well as
figure out how to audit that it works properly.

In addition, it calls for security ahead of features. Given that phpfog is
funded, they'll need to implement the equivalent of a bleeding edge stack and
a locked down stack.

------
svlla
php... a language by amateurs, for amateurs. phpfog... a service by amateurs,
for amateurs.

~~~
mikey_p
Your argument could also apply to the Linux kernel.

~~~
skeletonjelly
I know the point you're trying to make, but it's a weak one.

75% of the kernel is written by corporate employees.

<http://apcmag.com/linux-now-75-corporate.htm>

~~~
zaidf
_75% of the kernel is written by corporate employees._

And that implies what about the kernel?

~~~
skeletonjelly
I'm not sure where you're heading with this. I was trying to tell the parent
comment that arguing against 'amateur' coders writing php is the same as the
Linux kernel was factually incorrect.

