
Code Spaces data and backups deleted by hackers - leejacobson
http://www.codespaces.com/?hackernews
======
invertedohm
Leaving aside the obviously deficient sysadmin work here: the timeline of the
story doesn't add up. I can only hope this explanation is not accurate.

You find notes in your AWS control panel saying you should contact some
Hotmail address. OK. So the first thing you do is reach out to that address
and take the time to communicate intricate extortion details? Only after that
you think maybe it's a good idea to start changing passwords, and right then
the other party takes action and deletes all the things?

If that's what actually happened then I'm afraid something like this was bound
to happen sooner or later.

~~~
Kequc
I feel that a lot of people here are being unnecessarily harsh. It was all a
bit of a silly mistake in hindsight but Code Spaces was a very new service I'm
not even certain it had secured funding yet.

The timeline looks to me like email address shows up. Check email address.
Email address contains extortion details. Try to change passwords. Hacker gets
in again and again while deleting stuff. Cannot get rid of hacker. Do not have
money. Within 12 hours everything is gone.

~~~
tylerlh
They have existed for a number of years[0]. Even still, being a new service is
no excuse for such poor handling of OpSec.

[0]:
[https://twitter.com/CodeSpaces/status/265757401368637440](https://twitter.com/CodeSpaces/status/265757401368637440)

------
jacquesm
Is it possible to 'lock' your amazon control panel to a specific set of IP
addresses?

In the payment world it is a fairly common feature to use a block-by-default
strategy for such crucial controls.

Hosting your project management and your sources with other companies always
did feel strange to me. I can see how it works well for open source project
and git (after all, every repo is a complete copy) but to host the master of a
subversion repo 'in the cloud' and to have your project management in the
cloud feels uneasy to me.

If this or something like it would happen to github and all the github issues
would be lost that would be a fairly major disaster.

You never know how solid the infrastructure and solutions chosen behind a nice
looking web front are until it goes down, and this one went down hard.

Condolences to the _users_ of codespaces.com, they are the ones who lost most
in all this.

From codespaces backup page:

" Real Time Backups Backup

All your Source Code is backed up in real time, so that in the unlikely event
of a system break down your data is safe.

Not only do we Back Up your data we also give you access to our backup up data
via the Code Spaces Admin console so you can keep your own copies.

Whenever you make a change we make a backup."

So much for that I guess, if it is spinning and online _it is not a backup_.

~~~
toomuchtodo
> So much for that I guess, if it is spinning and online it is not a backup.

True, although with S3, you can make backups very difficult to remove:

[http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMFADelet...](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMFADelete.html)

"If a bucket's versioning configuration is MFA Delete enabled, the bucket
owner must include the x-amz-mfa request header in requests to permanently
delete an object version or change the versioning state of the bucket. The
header's value is the concatenation of your authentication device's serial
number, a space, and the authentication code displayed on it. If you do not
include this request header, the request fails."

~~~
jacquesm
Yes, so all it takes then if for amazon to either be buggy or to fail for some
reason. You can't really outsource responsibility for stuff like backups. They
should be under your control and yours alone, and they should not live in the
same DC or with the same provider where you store the rest of your data.

And when you've made your backup you take it offline so that no matter what
you can get back in business.

~~~
toomuchtodo
Okay then. Where would these backups go? S3 is easy to backup to. Tarsnap is
nice. Rsync.net works as well. But these are all online backup options.

If you're advocating offloading to physical media, you need someone who is
going to religiously do it (execute code, pull to removable flash drive/SATA
dock), and the more up to date you want your backups to be, the more tedious
it becomes.

How much are you willing to spend to have AWS Export send you a physical SATA
drive nightly?

~~~
__david__
> Where would these backups go?

Why not to a big disk sitting on a computer in your house?

The server doesn't push to it, the home computer pulls from the server (via a
cron job or something). That way an attacked can come in from somewhere else.
It doesn't have to be perfect, and it shouldn't be your _only_ backup. But
having something like that is great for catastrophes like this.

~~~
notwedtm
and then your house burns down, floods, is robbed.

~~~
jacquesm
One of the basic premises behind a solid backup strategy is that if disaster
hits that it does not hit simultaneously in all places. If that does happen
then I think you have different problems to contend with than trying to
restore your backups.

------
pjc50
That's extraordinary .. but it does drive home the distinction between
"backup" and "online second copy of your data". Proper backups should be
offline when not in use.

~~~
__david__
> Proper backups should be offline when not in use.

I'm not sure I'd go that far, but I'd say that backups on the same service as
the data aren't a good idea. It would smart of use a different cloud provider
for the backups and (depending on the size of the operation), even a disk
sitting in someone's house somewhere, just in case.

~~~
twistedpair
That's why I have a Mini on the bookshelf that SSH's into an EC2 instance (IP
restricted) during a 1 hour window each night when the router is set to open
and copies that day's PGP'd backups over. Then it goes back to sleep to wake
on schedule the following night. Not the same as a tape in a vault, but it's
my _last resort_.

------
bagosm
All I can say is this:

If you can delete it with a single control panel, it doesn't count as an
offsite back. Fire the devops

~~~
toomuchtodo
DevOps here.

There are some things you aren't going to expect (compromise of your AWS
console). This could have been solved by having MFA enabled, as well as having
the app push backups in realtime, versioned with delete protection, to S3
buckets under the control of another account (write access, but no delete
access).

Show of hands how many people here are doing it this way.

~~~
mentat
Seriously, if your root account and all full admin accounts aren't using MFA
you're just asking for it. Also if you're not using purpose specific access
keys, you're just asking for it. If the first thing you do isn't calling AWS
support, wow...

~~~
guiambros
Couldn't agree more. Everything under a single platform, no MFA, no (real)
_offsite_ backup, and on top of that they spent 12 hours _corresponding_ with
the attacker, instead of immediately calling Amazon to ask their help to shut
down everything, while they still had time?

I'm sorry, but this is a succession of things _not_ to do in terms of system
operations. Probably the team never managed mission critical platforms before,
and hopefully they now learned the lesson.

~~~
jacquesm
How many companies have not yet learned that lesson? There are probably a lot
of codespaces on AWS. My reasoning is that if you make it so that a developer
can set up a virtual datacenter but does not have the background of actually
running such an installation then you're going to have to assume that it is
probably quite fragile.

Software people tend to make all kinds of assumptions about hardware that do
not work out in practice.

~~~
guiambros
That the problem with the recent "DevOps" trend. Lots of people coming from a
"Dev" background, but no real "Ops".

And now that spinning up a couple of servers on AWS and creating snapshots on-
the-fly are so easy, it gives the false impression that you don't need much to
act as a sysadmin.

------
mkal_tsr
They ran a code hosting service and they had _no_ offsite backups?

Wow, just wow.

~~~
nevster
From their homepage "We offer Rock Solid, Secure .. hosting". Perhaps not so
rock solid...

------
aaroncampbell
Saw this earlier today. That's rough. Obviously they had some problems with
their architecture (backups shouldn’t be able to be deleted like that), but
it's still pretty messed up. I hope they catch the guy. I won't help Code
Spaces, but whoever it was deserves to be caught.

~~~
informatimago
s/I/It/

~~~
qworty
It hope they catch the guy?

~~~
taspeotis

        I won't help Code Spaces
    

s/I/It/

    
    
        It won't help Code Spaces

------
Arnor
From the Amazon RDS documentation:

When the backup retention changes to a non-zero value, the first backup occurs
immediately. Changing the backup retention period to 0 turns off automatic
backups for the DB instance, and deletes all existing automated backups for
the instance.

[http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overvi...](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.BackingUpAndRestoringAmazonRDSInstances.html)

~~~
jacquesm
> and deletes all existing automated backups for the instance.

That could do with some sort of gracetime, both against shoot-in-your-foot
scenarios and bad guys.

------
codeddesign
while I feel bad for thoer customers, I have no remorse for the creators.
Stupid and mediocre managment of their code...not to mention...how does a code
hosting service not understand the difference between a backup and an off-site
backup? how could they delete an account and not notice (or bother to check)
the other created accounts? No notification on account access creation? So
mamy mediocre mistakes...

------
opendais
Seems to be down so cache:
[http://webcache.googleusercontent.com/search?q=cache:qpjW4k2...](http://webcache.googleusercontent.com/search?q=cache:qpjW4k253l0J:www.codespaces.com/+&cd=1&hl=en&ct=clnk&gl=us)

"We are experiencing massive demand on our support capacity, we are going to
get to everyone it will just take time. Code Spaces : Is Down!

Dear Customers,

On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against
our servers, this happens quite often and we normally overcome them in a way
that is transparent to the Code Spaces community. On this occasion however the
DDOS was just the start.

An unauthorised person who at this point who is still unknown (All we can say
is that we have no reason to think its anyone who is or was employed with Code
Spaces) had gained access to our Amazon EC2 control panel and had left a
number of messages for us to contact them using a hotmail address

Reaching out to the address started a chain of events that revolved arount the
person trying to extort a large fee in order to resolve the DDOS.

Upon realisation that somebody had access to our control panel we started to
investigate how access had been gained and what access that person had to the
data in our systems, it became clear that so far no machine access had been
achieved due to the intruder not having our Private Keys.

At this point we took action to take control back of our panel by changing
passwords, however the intruder had prepared for this and had already created
a number of backup logins to the panel and upon seeing us make the attempted
recovery of the account he proceeded to randomly delete artifacts from the
panel. We finally managed to get our panel access back but not before he had
removed all EBS snapshots, S3 buckets, all AMI's, some EBS instances and
several machine instances.

In summary, most of our data, backups, machine configurations and offsite
backups were either partially or completely deleted.

This took place over a 12 hour period which I have condensed into this very
brief explanation, which I will elaborate on more once we have managed our
customers needs.

Data Status

All svn repositories that had the following url structure have been deleted
from our live EBS's and all backups and snapshots have been deleted:
[https://[ACCOUNT].codesapces.com/svn/[REPONAME]](https://\[ACCOUNT\].codesapces.com/svn/\[REPONAME\])

All Svn repositoies using the following url format are still available for
export but all backups and snapshots have been deleted:
[https://svn.codespaces.com/[ACCOUNT]/[REPONAME]](https://svn.codespaces.com/\[ACCOUNT\]/\[REPONAME\])

All Git repositories are available for export but all backups and snapshots
have been deleted

All Code Spaces machines have been deleted except some old svn nodes and one
git node.

All EBS volumes containing database files have been deleted as have all
snapshots and backups.

Code Spaces Status

Code Spaces will not be able to operate beyond this point, the cost of
resolving this issue to date and the expected cost of refunding customers who
have been left without the service they paid for will put Code Spaces in a
irreversible position both financially and in terms of on going credibility.

As such at this point in time we have no alternative but to cease trading and
concentrate on supporting our affected customers in exporting any remaining
data they have left with us.

All that we can say at this point is how sorry we are to both our customers
and to the people who make a living at Code Spaces for the chain of events
that lead us here.

In order to get any remaining data exported please email us at
support[at]codespaces.com with your account url and we will endeavour to
process the request as soon as possible.

On behalf of everyone at Code Spaces, please accept our sincere apologies for
the inconvenience this has caused to you, and ask for your understanding
during this time! We hope that one day we will be able to and reinstate the
service and credibility that Code Spaces once had!"

~~~
lugg
Thanks for the cache just getting to this.

Edited because someone didn't like my original tone. Was a bit rushed to be
honest.

Few things seem off about this:

\- Offsite backups were also deleted, I don't think they had offsite backups,
or at least backups you could legitimately say were "off site."

\- EC2 has two factor auth, why you wouldn't use this for your business I
don't know. [1]

\- Corresponding with extortionist is a really dumb move. It would be better
time spent locking things down - contacting amazon directly to get an account
lock / getting your ducks in a row.

[1]
[http://aws.amazon.com/iam/details/mfa/](http://aws.amazon.com/iam/details/mfa/)

~~~
jacquesm
There is something about this whole story that feels weird, I can't name it
but it is as if this isn't the whole story.

~~~
rcthompson
Probably because of the part where they say this isn't the whole story: "This
took place over a 12 hour period which I have condensed into this very brief
explanation, which I will elaborate on more once we have managed our customers
needs."

~~~
jacquesm
That could be it. But there is a certain dissonance about this whole thing, I
try to imagine myself in the same situation and the whole thing weirds me out.
How could this mysterious hacker have known they had no other backups? Have
they talked to LE at all at this point? Why not string the guy along, buy
time, _immediately_ alert amazon to lock the account completely?

So many questions. Anyway, they'll be updating this sooner or later, I just
can't help but feel a bit weirded out by some of the things in there (and
things that should be in there that are not).

This is most likely just my professional paranoia acting up. And of course it
is easy enough to be back-seat driver here, I'd hate to be in their shoes, no
matter how they got there.

~~~
tzs
> How could this mysterious hacker have known they had no other backups?

I don't think we can infer that he knew that. It seems more likely to me that
he expected the outcome of deleting all their Amazon stuff he could reach
would be that they would be down for a day or two as they reconfigured
everything and then restored from offsite backups, costing them overtime or
comp time for their IT guys, a few disgruntled customers who leave, a few more
disgruntled customers they have to placate with freebies, and making them more
likely to pay next time an extortionist comes around.

I would not at all be surprised if the extortionist is very surprised that
they did not have other backups and his actions have probably killed the
company.

He's probably also somewhat worried, as this probably knocks the monetary
damages up enough to (1) make it much more likely that this will get some
serious law enforcement attention, and (2) if he is ever caught and convicted
greatly increase his sentence and/or fine by moving the severity level of the
offense way up.

For instance, here are some examples for 18 USC 1030(a)(5), which covers
causing damage or loss on a computer via unauthorized access, assuming no
other factors that increase the sentence:

    
    
           LOSS    MONTHS           FINE
    
           $10k       0-6    $ 1k  - 10k
           $30k      6-12    $ 2k  - 20k
           $70k     10-16    $ 3k  - 30k
          $120k     15-21    $ 4k  - 40k
          $200k     21-27    $ 5k  - 50k
          $400k     27-33    $ 6k  - 60k
         $1,000k    33-41    $ 7.5k- 75k
         $2,500k    41-51    $ 7.5k- 75k
         $7,000k    51-63    $10k - 100k
        $20,000k    63-78    $12.5k-125k
        $50,000k    78-97    $12.5k-125k
       $100,000k    97-121   $15k - 150k
       $200,000k   121-151   $17.5- 175k
       $400,000k   151-188   $17.5- 175k
       above that  188-235   $20k - 200k
    

Trying to cost someone a few thousand dollars worth of damage and instead
killing their $10 million dollar company, for instance, changes it from 6
months tops to 5 years minimum. Ouch.

~~~
jacquesm
> I would not at all be surprised if the extortionist is very surprised that
> they did not have other backups and his actions have probably killed the
> company. > He's probably also somewhat worried, as this probably knocks the
> monetary damages up enough to (1) make it much more likely that this will
> get some serious law enforcement attention, and (2) if he is ever caught and
> convicted greatly increase his sentence and/or fine by moving the severity
> level of the offense way up.

That's plausible. It makes some sense that if you destroy something that you
should be responsible for that. At the same time, even for a hacker the
assumption that there would be back-ups would be a fairly logical one, though
I'd hate to be in a position of fielding that defense.

~~~
danielweber
If I throw a rock into your garage, and knock over your precariously balanced
anvil onto a Lamborghini, that's 100% on me.

[http://en.wikipedia.org/wiki/Eggshell_skull](http://en.wikipedia.org/wiki/Eggshell_skull)

------
twistedpair
Reminds me of LiveJournal's dead harddrive induced closure. When will startups
learn how it's done?

~~~
gaadd33
When did that happen? I was under the impression that LiveJournal was still
around however it was mostly popular in Russia. Any links to that?

~~~
csixty4
Definitely still around. My wife uses it daily and is part of several
communities.

------
gnu8
Why is it possible to destroy an entire enterprise by compromising an Amazon
account? Where the fuck is their 2FA? What about a cooling off period before
committing changes like deleting all of your storage? Amazon's infrastructure
seems to be built without essential safeguards.

~~~
TkTech
Amazon has extremely tight security, including 2fa, fine-grained IAM
permissions, instance security groups, VPC, and more.

The fact that Code Space's didn't bother to use them is their own problem, not
a failing on Amazon's side.

Additionally, storing all of your backups with the same service as your
production environment was outright moronic.

------
taspeotis
Another point for distributed version control systems. If the server hosting
my repository exploded I'd have the entire repo on my computer.

Why hosted subversion is a thing, I don't know. It's a horrendous experience
once you introduce any latency between the client and server.

------
Globz
How the hell would you let the faith of your entire enterprise data in the
hand of Amazon? Offsite/offline backup are mandatory especially when you deal
with this kind of data.

~~~
twistedpair
Real companies don't. I know a major video provider in Cambridge that
replicates everything to S3 and Google Cloud storage for this very reason.
Now, neither has ever gone down for them, but if you've got to stay up, you
have a resilient architecture.

------
wasamasa
Does anyone know of noteworthy projects that were hosted there?

~~~
eterm
Without naming names I know of at least one company who were using it for
their main svn repo. I am hoping they have come out of this relatively
unscathed. I am not with them anymore and last I heard they were switching to
git so perhaps they have managed to dodge this bullet.

------
_cipher_
Cloud kicks ass. Totally secure along with the high-level fellowship of
paladins^Wadmins.

------
msantos
When will these cloud people learn that storing a so called _backup_ in the
same place or under the same domain or root account as the original data is a
_copy_ not an actual backup.

------
rahimnathwani
offsite backups != offline backups

------
franklyjeremy
Either that or the "hacker" pressed the "Close Account" button in the EC2
panel.

------
randunel
Am I the only one feeling cheated when people post referral / tracking links?

~~~
taspeotis
The ?hackernews in the URL is to work around HN's duplicate URL detection. It
could have been ?lulz and had the same effect.

