I know the attackers were just kids but I have to admit pursuing legal action sounds very tempting - even to just act as a deterrent to others. If they had just put up phpfogsucks.com, it might have been ok. But tweeting trash from their twitter account, redirecting their root domain to phpfogsucks, etc - are all not cool at all and should have some consequences.
Maybe then they'll stop with the half-assed apologies and recognize that there's a right way and a wrong way to do things.
PHPFog built a castle out of sand and you're upset that a wave came and demolished it. I'm always surprised at how thin-skinned a lot of HN commentary is. "Oh, Zed shouldn't be so rude" "These kids' lives should be destroyed for playing games with an wholly insecure website." "I stopped reading that article because it used the word blowjob."
I don't get angry at my dog when he shits in the house. Being angry at something that can't understand only satisfies the urge to shift blame.
My dog shits in the house and it's my fault for not walking him sooner. If some children compromise every level of your company then getting mad at them is only trying to deflect the blame. PHPFog is the only responsible party in this mess. I feel for the customers who still trust them.
This is like someone finding an unlocked door to the apartment building's maintenance office, taking the master keys from there, rifling through a bunch of people's personal belongings, sticking signs in the windows saying "this building's landlords suck," and changing the locks on some of the doors to make it hard to clean up the whole mess.
They absolutely are the responsible party; you should never blame the victim of a crime just because the victim didn't take adequate steps to defend themselves. If I accidentally leave my door unlocked one day, that does not make it suddenly OK to come in and take my stuff and it's my fault for not having locked my door, instead of yours for taking my stuff.
Now, in this case PHPFog does bear some responsibility, because they have a duty to protect their customers as well as possible, and from reading about how this happened, it sounds like they were amazingly sloppy and irresponsible about it (passwords stored in the clear on the server, passwords shared between various accounts, leaving unsecured shared systems running after beta launch, etc). But that doesn't reduce the culpability of the attackers; they acted maliciously, with full knowledge of what they were doing, vandalized systems, changed passwords, and bragged about it.
If I were 16 and hacking some stupid website, I would think, sure this is "wrong", and I might get in trouble, but I probably wouldn't think that it would matter 5 years from then, or 20. Truth is, a criminal infraction can fuck your life up worse than is deserved. You could be forbidden to immigrate to your wife's home country, denied any worthwhile job you can find, or spend a sizable portion of your life in a smaller cage than they house animals in.
There's a reason juvenile law exists, but we forget it. We charge 12 year olds as adults. This isn't that different.
Hell, where I live, a lot of employers will check the sheriff's website for public arrest records. These aren't convictions, they're arrests. People are being denied jobs based on accusations.
I've never gotten so much as a speeding ticket, but even I realize what a load of shit this all is.
The job of parents and society is to teach children responsibility. That means having consequences for your actions. And the closer you get to adulthood, the more adult those consequences should get.
When I was a teen, some real estate developers tore down a bunch of woods where I had always played and started building a house. I was ticked and I vandalized the construction site. But my crime was discovered, and I had to work carrying lumber in the hot sun and scraping glue off windows to pay back the damage. Why did I have to do this? Because the developer talked to my parents. And my parents made me do it.
This was light punishment - I wasn't taken to court, and I didn't get a criminal record. But parents are to children as legal system is to adults: they set the rules and enact the punishments. If they don't, someday the legal system will have to address their failure to do so by locking up their grown children who never learned right from wrong.
I'm not sure what the consequences here should be, but the argument "they're kids, they can't be held responsible" is silly. If they're not expected to be responsible, they won't learn to be responsible.
Do note that since he is tech oriented, he probably now knows what a S* storm he has kicked up - the blog response pretty much ensures that he is aware. I assume you would be sweating bricks if you knew that the FBI was coming after you AND that you had hand delivered a bullet point confession. I assume thats a pretty strong deterrent (in his specific case)
(Incidentally fighting to stick up for your woods was a kinda nice thing to do, modulo the amount of vandalism you pulled off.)
While I agree that the full force of adult law is probably inappropriate, some punishment should be exacted. Perhaps some sort of injunction regarding limited access to the Internet for a time (to be enforced by their parents, and to include replaying any fancy 'net capable phones with a good old not-even-got-GRPS mini-brick phone for the duration).
If I left my front door open and some kids came in, raided the fridge, broke the TV and ran up a huge phone bill, I would not expect them to get away completely unpunished. I'd not want them put away, and I wouldn't want it on their permanent record unless something particularly immoral was done (harming the cat, for instance), but I would want something to be seen to be done to educate them on right/wrong and act as a deterrent to others. I would also expect to be laughed at for being daft enough to leave my front door open!
These are 16 year old kids, old enough to know right from
Also, whether they know right from wrong is a question whose answer definitely isn't as clear-cut as you make it out to be. There's a reason we don't consider them adults yet.
This is like someone finding an unlocked door to
the apartment building's maintenance office,
taking the master keys from there, [...]
I've been a 16 year old idiot myself. After doing some stupid things to a website I was threatened with some trouble. I could avoid it by having extensive talks with the site's owner, and paying for their losses, but still, I learned my lesson: everything you do on the internet, in the virtual world, has an effect in the real word. Also it was an urgently needed wake-up call for me; I learned to think twice and since then haven't done any nasty stuff without considering its impact.
So in my opinion some sort of punishment is needed for the kids to learn their lesson. The FBI or other federal institutions shouldn't be involved. Talk to them, I'll guess they're nervous as shit right now! Just bill them the time you needed to tidy up the mess they left (or a fraction thereof), and I think both sides are good.
Door locks are extremely "exploitable", but if a 16 year old were to use a bump key to gain access to PHPFog's corporate office and vandalize the place, all of the sudden it's a much bigger deal.
I have a theory that it has to do with familiarity and empathy. Door locks are a pretty standard solution. We all have them on our homes, and we think to ourselves, "I've done a reasonable job of securing my home." When someone's home/office is broken in to, we can easily identify with them. We look at the scenario and realize that we could easily suffer the same. We empathize with them.
Move the playing field to the Internet and all of the sudden everyone is expected to have Fort Knox level security. When someone's infrastructure is compromised, everyone stands atop the high hill, looking down on the drowning masses as the tide comes in, but the reality is that we're all vulnerable at some point.
A startup could easily spend as much on security as they do developing their core product. Why? As a startup, I'm not going to invest in double-reinforced steel doors, bullet proof glass windows, armed guards, and a centralized vault. That's wasted money in my view, because I have a reasonable expectation that people will act with civility. If someone does break in, I'm insured, and I will report the crime to authorities who will investigate. If the criminal is caught, there are real penalties, and they'll carry the stigma of having to check "YES" next to the "Have you ever been convicted of a felony" on their job applications.
I'm not saying we should try these kids as adults, but when I was 14, some kids who shared a bus stop with me broke in to a house near our bus stop and trashed the place. They got caught and suffered some severe penalties. It was a valuable lesson for everyone involved. A couple of the kids were from really bad homes and suffered from greater influences than the threat of the law, but the other two turned their act around really quickly. Had they gotten away with it, or had the attitude been "they're just kids", I'm not sure they would have realized the impact of the crimes they committed. I think we need more of this balance in our views of internet crimes.
The fact remains that the site was insecure enough for a 16-year-old to find his way in. And the contributing factors to this insecurity might not have been identified had he not performed the attack in the first place.
This reasoning is not applicable when the victim is a corporation who gives implicit or explicit guarantees to their customers about security. Your example should be: if you stored your stuff at a paid storage facility and someone there accidentally left the door yo your unit open.
I wouldn't care if it's a bunch of teenagers or Chinese cyber-warfare team who breaks into my Gmail account, I'd be mad as hell with Google for letting this happen.
Pressing charges against the proprietors of the crime will not change the fact that this host acted in a very irresponsible and sloppy way, and is in short untrustworthy.
Your analogy is slightly off. A wave is an act of nature: this is more along the lines of a jealous kid who knocks down someone else's sandcastle because he can't build his own.
"I don't get angry at my dog when he shits in the house. Being angry at something that can't understand only satisfies the urge to shift blame."
While some HN posters might feel the the 16 year olds involved in this incident have the same mental capacity as your dog, I'd like to give them slightly more credit. ;)
I can only speak for myself here, but I do think the people involved in compromising PHP Fog should be punished. No, I don't think they should get life in prison (</hyperbole>): I hope they can learn from their mistakes. However, they did commit a crime, as they've admitted both here and elsewhere online. They should be capable of understanding that their actions have consequences, so I think some consequences are in order. What those consequences should be is up to PHP Fog.
In terms of moral culpability, sure. But when I put systems on the internet, I basically treat "intrusion attempts" as in practice part of the environment, like "mosquito bites" are in Texas. Perhaps they're best thought of as kids knocking down sandcastles rather than ocean waves, but their ubiquity makes them feel more like ocean waves, because you can basically assume that there are tons of those kids, and they're going to kick at your sandcastle every day.
The fact that there's a whole ecosystem of bots running automated intrusion attempts makes them feel a little bit force-of-nature-ish as well. If you lived in some neighborhood where thousands of roving robots were constantly checking doors to see if they could find an unlocked one, you'd have to treat "roving robots" as a quasi force of nature. Well, either that, or come up with a policing method that finds the controller of the robots and shuts them down, but I have relatively low hopes for how much of a dent "cybercrime" policing will make in the overall online-intrusion ecosystem.
"Act of God is a legal term for events outside of human control, such as sudden floods or other natural disasters, for which no one can be held responsible"
Do you think nobody can be held responsible for this breach?
Just because we can hold individual humans accountable (and should) doesn't mean we shouldn't have the perspective of "CONSTANT VIGILANCE."
Certainly should have to treat intrusions as inevitable in designing the system, but there still is responsibility on the part of the intruder.
I lock my door because I consider it inevitable that someone will eventually try and break in. However, if someone does break into and vandalize my apartment, I sure as hell would consider them responsible and not consider in an act of God.
Does it matter to you if some kid in Australia is brought up on charges? No?
Does it matter to you if a hosting company is competent in securing their servers? Yes?
Any discussion of who did the hack servers no purpose other than to distract from the only issue that matters to anyone which is PHPFog's security.
But your post seems to imply that the kids who did this have no responsibility for their actions; "it was just waiting to happen". This I strongly disagree with (not least because I work with 16 year old kids and they are completely able to take some level of responsibility for their actions).
Sure, at 16 your world view is incomplete and you can make rash decisions ("for the lulz") that backfire bigger and faster than you imagine. On the other hand there is no doubt they knew the illegality and the ethical issues with undertaking this - even if only vaguely.
And if they do not completely understand those issues, do you not think they should be taught them? As responsible adults we should be getting across to them in a sensible fashion that this was not a nice thing to do, and that the impact could have been a lot wider than it was.
Because if we don't and next time they do some real damage, well, that was an opportunity lost.
FWIW I think he crossed the line by causing damage. If I caught a kid breaking into my house I would probably drag him home to face his parents. But if he started smashing plates I'd be a little more pissed, that is a wanton act and probably needs a more severe punishment.
Sure, criminal charges are a silly approach in this case (no need to ruin his life for one silly mistake). Call his parents, explain what has happened and then get him to do some sort of "community service". That's an important lesson in consequence.
In this modern world 16 year olds are not nicking alcohol from the corner shop any more; they are breaking into websites. And that has potential for much more dramatic and widespread impact. We need to stop saying "oh, they're only kids". Instead we should recognise that 16 year old "hackers" exist on the internet and think of ways to communicate with them (ideally in a way that gives them an avenue for their curiosity without risking too much damage :)).
Bottom line; a 16 year old kid is a far cry from your dog when it comes to the ability to "understand" what you have done.
Just my 2p :)
Both parties bear responsibility, and it's absurd to think otherwise.
Very well. These kids caused harm to a business. So what's that change? The business screwed up, badly. The agent of destruction is quite irrelevant. Had it been a power failure, backup failure, permissions failure, data leak, or data corruption would PHPFog deserve any less blame? This need to shift some responsibility to a bunch of kids is nauseating.
>I'm just much more impressed with the way that PHPFog is handling their business after the fact than these kids are.
This is another example of the weird HN mentality when it comes to companies "apologizing" (Like WakeMate blaming their Chinese manufacturer for flunky power supplies). Are you actually impressed that a corporation has better PR than a bunch of children? Does that even make sense to you? I'd be impressed if they had managed to actually apologize while accepting all the blame without trying to pawn off the responsibility for their mistakes on some kids.
They aren't shifting responsibility. The kids are responsible for their own actions. They did something illegal. They are responsible for it.
Now, PHPFog is also responsible for protecting their customers; they are supposed to provide a secure hosting environment. PHPFog is a victim here, but has also acted irresponsibly with regards to security (not criminally irresponsibly, but if harm did come to their customers due to this, there could be possible civil liability). The fact that PHPFog bears some blame for their security practices doesn't take responsibility off the kids who broke in and vandalized their systems.
I was appalled at the frequent mentions of 'luck' in that blog post. Your job as a sysadmin is to eliminate luck. To eliminate chance. To make _sure_ everything stays running, everything stays secure, everything stays confidential.
Yes PHPFog is. The only reason these kids are even mentioned in the blog post is to shift blame. Their part in the post serves no other purpose.
The entire event could have been recounted without a single personification of the hackers in the blog post.
I can't honestly imagine what kind of moral system you have in which you don't believe that criminals are responsible for their own actions. If someone breaks into your home, is it your locksmith's fault, or the police's fault, or your alarm company's fault? No, it's the fault of the person who broke into your home. Perhaps one of the other parties mentioned was negligent, or perhaps not negligent but they could improve their security practices (install stronger locks, upgrade your alarm system, do more patrols in your neighborhood), but it's still the fault of the person who broke in and vandalized your home.
"They" are not relevant in any way! "They" are tabloid meat for an internet drama. "They" are a distraction from the fact that a hosting company had piss poor security surrounding the core product. The entire story could be told without mentioning the hackers by name, or providing any biographical information. The only reason to include them is to distract from the real issue.
Replace "16 year old" with Russian, Chinese etc. Yes, vandals are bad. That's not exactly in question. In question is the sheer gall of PHPFog to shift blame to some kids to try and cover their embarrassment.
I think the blog post should be rewritten. Instead any mentions of the hackers should be completely neutral. Then it will be PHPFog getting out there and taking responsibility for their mistakes.
No distraction. No hand waving. And no tabloid drama.
Lets put it this way - a 16 year old kid who got lucky broke their site.
If it was anyone with intent, we would not know.
This is a case study in how to handle a situation like this. Its brilliantly done, inclusive of the comment where he says "the community is standing by us".
Its actually a brilliant brilliant PR piece.
Perhaps a little mention would be good, but the way they make it the key points of the post, and so many people commenting on it accept it, disgusts me.
Technology analogies invariably suck, but I'm pretty sure this is provably better:
When a teenager smashes in a storefront window, do we say they should've had bars over it?
If a storefront was under constant attack, then yes, they should have bars over it. In fact, in my hometown, there was a streak of vandalisms where kids were throwing bricks through windows. After getting hit 3 times, one store replaced their huge glass windows with smaller plexiglass ones.
Let me be perfectly clear, lots of the blame lies with that attacker. But it is also the responsibility of the a web host to fortify their systems sufficiently, which clearly wasn't done in this case.
If I give the bank my money and the next day I get an email saying "Sorry, we didn't feel like locking up last night and some kids looted the vault." I'd have a hard time calling the bank the victim.
And something concrete:
PHPFog knew the holes existed and were negligent. I would say they even have some contributory negligence (IANAL). Especially after admitting they knew they were vulnerable.
And we certainly don't call the bank the victim.
Bank robber, 13, could get 21 years in US jail (http://www.breitbart.com/article.php?id=CNG.cb17379375828ffc...)
Teen bank robber to be held for two years (http://www.morningjournal.com/articles/2011/02/18/news/doc4d...)
Boy, 15, Charged in Armed Bank Robbery in Lancaster (http://articles.latimes.com/2002/aug/09/local/me-bankrob9)
2. Why do you think banks aren't considered victims in cases of robbery? Because they could have done more to prevent the robbery from happening?
Depends what's behind the storefront windows.
On the other hand I can certainly agree that we stop with the technology analogies before a car analogy is let loose and someone gets hurt.
If I leave my front door unlocked and someone walks in, there are several possibilities. If they just look around and leave, or say lock the knob behind them and walk out, that is one thing. If they smash my furniture and knock over all my plants, that was their choice, right? That action is illegal and immoral regardless of the fact that it was my negligence which made it possible. We all know these dumb kids should have reported the vulnerability responsibly. That would have benefited everyone, especially themselves. They might have been getting job offers instead of bad reputations.
He is naive and immature and realizes what he did is wrong. To make amends, in true 16 year old fashion, he gives a bullet list of errors to try and help undo the damage.
For his inability to communicate, show restraint, maturity, planning and foresight, for the very crime of being young and immature, he gets people wanting to throw the book at him? In that case, can we please, pretty please, torch wall street?
Its quite likely that he realizes that giving an error list is BAD and STUPID, and now is trying to back pedal by putting on a brave face to ignore the bone headed-ness of his (compounding) mistakes.
Try and imagine exactly how YOU would feel if you had a huge amorphous mob saying "The FBI should come for you"? At 16 you have NO scale in your head to cope with that.
Restraint is (one of) the hallmarks of maturity. As is intelligence and not taking good faith for granted - like not sending a list of errors you can be prosecuted for.
Here is what I would do - call this kid parents and Leave it at that. Let the family know how close he is to being in BIG trouble. If you want to do one better, give him a constructive outlet. He is already probably one very, very, very, miserable and frightened kid right now. And he should be.
Its called grace forgiveness and wisdom. As adults, we are supposed to have it. You are NEVER going to deter kids from being kids. So you need to ensure that they are scared and know where the line is drawn, so that they can become effective productive Adults.
I think they should definitely get something a little stronger than a slap on the wrist, but also remember that they're just teenagers who don't know any better. All they really need is enough of a punishment to learn their lesson, and they'll probably end up productive members of society.
"We have hired professional white hat hackers with government level security experience to attempt regular pen tests on our system, both as regular users as well as giving them special access and seeing if they can get through."
"If you find a security flaw and report it using the Full Disclosure Policy to firstname.lastname@example.org with notice, we will help strengthen your security reputation in a very public way and reward you generously."
Step 1. Open up Metasploit
Step 2. Button Mash.
Step 3. Profit
So called 'white hat hackers' tend to be fraudulent script kiddies who couldn't hack their way out of a gibs0n.
They often attend classes like this http://www.infosecinstitute.com/blog/ethical_hacking_compute... and read a book or two like this http://www.google.com/products/catalog?q=hacking+exposed&.... Some times they'll even have a sweet certification like this https://www.eccouncil.org/certification/certified_ethical_ha....
And at the end of the day all they're doing is getting the down low on your system with nmap and then going all turbo with metasploit. And if they are feeling up to a challenge they might even rip someone's exploit from milw0rm.
99% of them are frauds and the other 1% are sellouts.
The way you become a 'top tier hacker' is by exploiting and reversing in the community (black/grey hats). Much of this is done through working with people who you respect and may have more experience than you.
Through this comes a certain level of respect, to where going public with exploits and such is a hindrance to the community and only serves your selfish agenda for fame or whatever.
Your exploiting all the people you've worked with in the past, ripping their ideas, and handing them off to assholes who then try to teach classes on 'ethical hacking' or whatever.
So yes, I'm calling them sellouts.
*Edit - Another problem I have is that these people generally tend to misrepresent 'black hacks' as a whole and try to play themselves off as bigger than life personas.
I mean, I'm all for praising gymnastics, but if we're trying for truth here shouldn't endgame be valued over execution?
Example: imagine the country of Dictatoria where if you jaywalk you are publicly tortured for a couple of weeks and then put to death. "As a detterrent"
A little extreme? Well, consider the 10-20 (or more) year sentences for cyber-terrorism these teenagers are going to get if the FBI throws the book at them.
That kind of law enforcement agency doesn't understand the concept of restraint - they are set up to go for the kill, for the maximum charge, for the maximum sentence every time.
You can lay charges thinking to give them a slap on the wrist, but the steam roller that gets set in motion is designed to crush them flat with no mercy.
Some parents had an argument with their teenage son, and he went off in a huff taking the family car (technically without asking permission). His parents thought to themselves "I know, we'll teach him a lesson", so they reported the car stolen. They intended to later on drop the charges (which would have royally peeved the police, making a false charge is also a crime).
Anyway, the cops caught the kid, and because Grand Theft Auto (or whatever the NZ equivalent is) is rated as amongst the most serious crimes, he got sent off (before formal charges were laid) to a maximum security facility in the back of a paddy wagon.
He didn't even make it to the maximum security facility. One of the other prisoners being transported there in the same van killed him.
* it wasn't recent (2006)
* he was sent to prison because his parents denied bail, not because GTA is some heinous crime in NZ
Nothing better to sharpen your skills and higher your standards than a _good crisis_
A good example of this would be how Chris Putnam got hired at Facebook.
I remember him getting trolled out of there after showing any kind of ambition beyond posting on the forums. Same thing happened to the guy who started Imageshack (originally an image host for the SA forums), Eli Hodapp (who later became one of the main TouchArcade writers), and probably some others I don't remember.
Well dang, good for him!
I'm struck by the similarity in some respects of the two cases. I would guess that the same people that got really upset about Valve doing that would be a bit saddened by this, too.
Unfortunately, this situation was much more severe than your average multi-million dollar AAA gaming title being leaked before release (http://www.ea.com/crysis-2/blog/crysis-leak
), so this requires the FBI and prosecution.
"aware of the potential security threat " but they left it for the next week, who honestly here would do that?
I have also seen comments around the web of migrating to Php Fog because of how they handled the situation. If you are one of these people please enlighten my mind as to how you came to such a logical decision or how much you get paid per year.
Also if Php Fog could enlighten us on how their terms of agreement will work in the case where our intellectual property is stolen on no fault of our own.
Save your sympathy for the sites that are still down, four days and counting
The whole blog post seems a bit melodramatic. I mean seriously, who here hasn't spent 3 all nighters in a row fixing a mistake? sack up and do what you should've done before deploying other people's data.
...and who would seriously sue these kids? they handled it poorly but they're smart (definitely smarter than i was at 16) you're lucky it was curious kids, rather than malicious (and experienced) hackers that would've been harder to catch. Do you really want to burden them with a criminal record for life?
"aware of the potential security threat " but they left it
for the next week, who honestly here would do that?
- we were aware of the potential security threat behind post-deploy hooks and were about to disable them [...] but...
- we were days away from replacing this server
- They were a short-term stopgap measure we had been planning to replace
To me, it sounds like the real problem could have been stated as "We were lax on security," but almost worse than that is the lack of accountability that I sense from company. Yeah, maybe it won't happen again, but it's hard to be full of confidence to buy into a service like that.
. Security is never binary. No matter what security measures you take, there are always zero-day exploits, social engineering, physical access, heavily-researched-and-highly-targeted attack vectors, etc.
Security is the opposite of convenience and accessibility. The right thing to do is to analyze what you are trying to secure and ensure an appropriate level of security proportional to the sensitivity and business impact of the potentially-exposed system.
There's no such thing as "secure." It's a continuum and it's always a tradeoff. Would you spend $5000 to protect something that's worth $50? It sounds like this site was in beta mode, and they made an understandable decision to focus on building the product and growing a customer base in lieu of ensuring top-notch security. In retrospect it was the wrong decision, but you don't hear about the companies who follow this approach and don't get publicly hacked. If they spent all their time on security from the outset, they wouldn't have anything to protect.
Lucas's post does not say "We screwed up." He says "We got screwed by Elliot."
I'm saddened most because Lucas is not embarrassed to point out he was outwitted by children.
When I foul up at my job I don't send an email detailing how some nasty client did something. I summarize what went wrong, how it should have been prevented and what steps I will be taking to prevent it in the future.
I would never write an email:
James Smith, a really evil customer (who happened to be working while there was thunder and lightning like Dr Frankenstein!), decided to try system("rm -fr /"). I knew it was possible, but I didn't feel like fixing it. Also I didn't feel like securing any of our other systems which explains those tweets, blog posts, DNS changes, and email compromises. I was lazy, but It's not my fault.
P.S. Credit cards probably didn't get compromised. Tim the intern was the one who implemented the payment system and he had his own passwords set.
(Note: I move this comment as I replied by mistake to CGamesPlay.)
Lucas's post says: "This was really naive and irresponsible of me." That doesn't sound like he's shifting blame to me.
You say: "I summarize what went wrong, how it should have been prevented and what steps I will be taking to prevent it in the future."
The article is essentially just that, with one exception; they didn't list steps they "will be taking" to prevent it, they listed steps they have already taken in the last 3 days.
As for Credit Card:
"Credit cards – We have never stored credit cards on any PHP Fog server. There was never any possibility that credit cards could have been compromised by this attack."
I mean, yes, by all means implement measures to avoid this sort of thing from happening in the future but "It Will Never Happen Again" is a very, very bold statement on security. The kind I associate with people who still don't really "get it".
They sound incredibly laxed on security and the "we were days away from fixing it" could be complete bull. To Lucas, it probably sounds better to say they were close to fixing it instead of admitting they were unaware of these exploits.
I find the disclosure in the blog post great, but the conditions they had leading up to the hack very disappointing.
If they were aware of the exploits, they should have taken quicker action. They'll probably be focusing on security big time now... they have no other choice.
I guess whenever I read this kind of statement from now on I'll be thinking of HBGary and chuckling a bit inside.
I don't mean to impune the capabilities of the people involved (I don't know who they are,) and it isn't to say that you can't find some AMAZING talent in the government realm, but as in all fields, it's the exception, not the rule.
Fast-forward to the class, and we're sitting there running tools like BackOrifice that exploit vulnerabilities that had been patched for years, and learning that a SYN flood is "a malicious attack". That's it, just "a malicious attack". When I asked about the difference between a SYN flood and a Christmas tree attack, I got a blank stare and "they're both malicious attacks".
I spent the rest of the class in the back of the room, reading the Armadillo Book.
Also, I did not once in my brief Navy career get to hack an enemy computer. Hugely disappointing.
Like I said, it isn't all bad. Two of the best security guys I know work in the government, and one of them was actually ex-Navy. But the hurdle for finding people that can get top secret clearances AND tie their shoes often proves too high to hire anybody, much less somebody qualified.
Armadillo book: http://www.amazon.com/dp/0596003439
Elliot is apparently VERY scared and blames John (compwhizii) (edit: not john, he blames someone else called supersnail1): http://www.facepunch.com/threads/1071855-A-member-of-Facepun...
Here is (compwhizii) Johns reply: http://www.facepunch.com/threads/1071855-A-member-of-Facepun...
And for anyone who missed it, here's what Elliot posted in the previous HN discussion about the phpFog breach: http://news.ycombinator.com/item?id=2346161
Not too bright, are we? Instant message the company you just hacked and bust out from behind your handle, then provide evidence for the prosecution in the form of a Web page? What is with kids these days?
It only takes one episode of Law & Order to figure out how to proceed here. Clue: Attorney.
After reading that I think it's a pretty strong argument against those claiming adult status for him. He clearly doesn't understand the situation he's in or the second order impacts of what he's done.
"Following this, I took a hold of their Twitter account and posted a couple of bits to draw attention to the fact. This did two things. One, it showed people the system was insecure, but on the other hand people always subconsciously root for the underdog; I drew attention to the company and the product. I know a number of people have actually registered (or intend to register, registration is closed) for phpFog since the incident thanks to the attention drawn to it by myself."
However, I won't post them here. That would be irresponsible disclosure. ;)
Kids (and like it or not, that IS what they are) are notoriously dramatic and these kids are definitely experiencing this as a crisis, no matter how they present in their comments.
They are also now being border-line, and in some cases overtly cyber bullied by some of their peers online which may be devastating if they don't have great face-to-face support. As adults, we have a responsibility not to add fuel to that fire and remember that if teenagers were predictable and transparent, there wouldn't be so many tragic cases in the news where they hurt themselves and each other.
Part of the problem is that teenagers and young adults aren't great at predicting consequences. This isn't opinion - it's science.
Especially in males, the part of the brain that is responsible for predicting consequences is not fully developed until the early to middle twenties. This is a double whammy because it leads to their bad decision making, but also to their belief that when something bad happens as a result, that it's literally the end of the world... that there is no way out.
There is no disputing that what these boys did was a big deal and that there have to be consequences. But it's also just a symptom of a bigger problem: as a society, we have failed to keep pace with the challenges that are the result of the first generation of digital natives (kids) being raised and educated by a generation of adults who are (at best) digital immigrants.
This post is getting way too long so I'm going to finish it on my blog @ thenewtag.com But thanks, criticsquid and nbpoole for posting some of the aftermath. I hope that one of the 1st things phpfog (or the authorities) did was involve these boys' parents?
Just goes to show that those with the time to spend are the most likely to break your stuff, even if you pay "professional white hat hackers" to test your system.
Avoid phpfog if at all possible, in my opinion.
Honestly, there are very very few developers that fix security problems in beta environments before anything else. In my experience, it's more likely that you're fighting fires, handling outages, and dealing with problems of scale than fixing security vulnerabilities.
Besides, isn't a beta the correct time to find these security issues? (Design / Alpha would be the ideal time, granted, but sometimes that's not possible.)
I guess the best way to think of it is that badness on the internet is like water. It will flow into every tiny crack in your wall you haven't sealed up tight. A crack in a dam doesn't leak less because its in an "obscure" location.
PHP Fog is doing great work to make the PHP ecosystem easier to work with, and I hope they didn't suffer too much from this mistake.
DRYP - Don't repeat your passwords.
So I had to build out a rather convoluted architecture that used the loophole of deploying to multiple regions and failing over to whichever region would give me an instance ... which gives me up to about 80 instances ... just barely enough for me to get going with a trial beta program.
Which is all just to say, it is slightly more than just a "conversation" that you need to have to get a higher limit.
One thing that surprises me is when people talk about utilizing multiple availability zones in EC2 as some sort of burden. It's very clear from their documentation and architecture that you need to be capable running in at least 2 availability zones regardless if you want any sort of availability.
My use case isn't for an ongoing server where you require availability. It's purely about compute power - I don't care where the compute power comes from but preferably I want low latency to my customer. So ideally I would just get all instances for any given customer from a single region.
I did find in the end that, as you say, I would sometimes not be able to get an instance in a region even when I was below my 20 limit for reasons internal to Amazon, so the failover work was going to be something I had to deal with anyhow ... but it just added complexity to my life earlier than it would have otherwise.
Edit: I would also mention that I certainly don't think of it as a "bad" experience. I think it is something of a small miracle that Amazon offers the service they do in the first place and I certainly understand why they have caution about handing out large limits to just anyone. I only made my comment above as a kind of caution to not just assume you're going to get a raised limit from Amazon immediately and especially don't leave talking to Amazon about it until the last minute if you're planning to launch something.
So, yeah, PHPFog screwed up and did that. Then these kids went in, threw paint on the walls, smashed some windows, etc.
PHPFog was stupid - they admitted that.
The kids were criminal.
The first is not illegal - the second is.
My site is still down, guess i'm in the unlucky 1%.
* It will go wrong at the worst possible time.
* If there are 4 possible ways for it to go wrong and you prevent all of them, it will go wrong in a 5th way.
Seriously don't write official blog posts for your company while you're experiencing "I was just in the field for days trying to fix this stuff" emotions.
Calm down, then try and be graceful about the fact that you were hacked by a few clueless kids. (Clueful kids don't let you know who they are.) Then try and figure out how to protect yourself against people with a clue.
I guess the real moral of the story is to finish what you begin, or don't keep putting security off until it is convenient for you.
Is it me or no one mentions the lack of expertise of the PHPFog team in PHP and Systems Administrations.
Sure kids broke in and the way they published their findings was despicable. The fact remains that PHPFog was utterly broken to pieces and the exact essence of the problem is simply the lack of knowledge in their field.
I am very disappointed by the tone of the blog post and think PHPFog don't really have a notion of what they are doing. I would much rather seem them where they belong, in the Ruby world where their experience is.
Its a brilliant piece and a great start/way to restore faith and recover from what must be a pretty grueling ordeal. Good job.
> Your password in the database is SHA512 encrypted, but we're not taking chances.
I hope he knows what he's talking about and is just tired from the past few days.
Just in case anyone doesn't know why Bcrypt is so awesome, it's because it actually takes longer to hash (based on the difficulty level you set, and you can bump up the difficulty level as hardware gets more powerful).
For other applications, you want hashing to be fast. But for passwords, you want hashing to be as slow as possible without compromising user experience.
There is no doubt they did some things they should not have. And I don't doubt there can be a decent case built against them. But as someone who actually had something from his teen years come to bite years later, it's not pleasant. At least in my case it was a MAJOR maturing moment(also the worst day of my life). May be it will take a lawsuit to get these kids to mature up...to that extent anything that gets em to mature up before they really get screwed would be fair.
I'm not merely advocating another chance but actually something that gets these kids to be a tad more thoughtful about their actions. It's not always easy to do that when you are 16 and full of adrenaline.
What does realtime mean in this case? Anyway, this isn't the only option. They could keep a few bare instances of their php stack online and simply run the deploy script instead of the image creation script. That ought to be able to run in under ten seconds I think.
Nice idea on the hot spare instances however.
2:56:45 AM Elliot : then I used the method detailed by turby
2:56:46 AM Elliot : to gain root
1. Use the post-deploy hook to chmod /home/ubuntu/.ssh so that it could be written to.
2. Upload a PHP shell, use it to write your public key into /home/ubuntu/.ssh/authorized_keys, and get the public IP of the EC2 instance.
3. SSH into the box, sudo su will get you root.
Customers who are already pretty risk averse to their data being stored in the cloud would see this as another reason not to take the risk.
The cloud computing consortium needs to work on a stable stack as well as figure out how to audit that it works properly.
In addition, it calls for security ahead of features. Given that phpfog is funded, they'll need to implement the equivalent of a bleeding edge stack and a locked down stack.
75% of the kernel is written by corporate employees.
The original comment just struck me oddly because I've come to assume that a large portion of the software that I use regularly was originally developed as an amateur project, or by amateurs.
And that implies what about the kernel?