

$5,000 Security Breach - mantrax4
http://blog.joemoreno.com/2014/04/5000-security-breach.html

======
sergiotapia
>'They' spin up spot instances which isn't subject to Billing Alerts. You'll
need to cancel those spot instances, revoke your AWS credentials, and change
your account password," he said.

This doesn't make sense at all. Amazon should let us if monthly bill > X send
me a priority email and phone call. Why do they hide behind these dark
patterns? I thought they were better than that.

~~~
theboss
The fact of the matter is amazon is a giant company and cannot thoroughly
think through each piece of logic in their system.

The reward for focusing on this before-hand is much lower than just writing a
check for $5k to this person and then fixing later (lot of $5k checks from
amazon today. Wheres mine?)

------
bequanna
>"'They' spin up spot instances which isn't subject to Billing Alerts.

Spot Instances aren't subject to AWS Billing Alerts? Is this common knowledge?

~~~
ceejayoz
I'm not sure it's _true_.

------
matthuggins
Why the cliffhanger? Why not just finish your thought? As is, this story
provides nothing other than showing the Amazon provided some helpful customer
service.

------
Jare
We had an almost identical event last Sunday in one of our dev accounts:
multiple high end spot instances in multiple regions with a newly created
security group pointing to a suspect IP.

We caught and corrected it quickly, but we still don't know how the keys
leaked out - we have chalked it up to lower security practices since it's not
a production account and is shared by more people (e.g. no 2-factor on it). We
started to investigate, but then Heartbleed happened.

I wish there were more mechanism in AWS to prevent bills from mounting up, but
the basic billing alarms worked in this case. I can't imagine how or why spot
instances would be excluded from alerts, their cost certainly is included in
the estimates that alerts are based on.

------
colinbartlett
Another reminder to be extra careful about checking in AWS credentials into
version control. You never know when you might open source that repository and
someone can easily yank it from Github.

~~~
friendstock
Actually this is what happened to us... our AWS credentials had accidentally
been put into Github for a hackathon project several months ago.

Coincidentally, the incident also occurred around the same time (April 1-2).
We were hit with $13,000 worth of EC2 usage before we shut them down and
changed our AWS key... We reported to Amazon, and they are working on a
refund.

------
nilved
The instance was spun up on April 2, but Heartbleed wasn't disclosed for
almost a week later. I highly doubt anybody used the Heartbleed 0-day to
access your account.

~~~
viseztrance
According to Cloudflare ([http://blog.cloudflare.com/answering-the-critical-
question-c...](http://blog.cloudflare.com/answering-the-critical-question-can-
you-get-private-ssl-keys-using-heartbleed)), exploiting heartbleed may
actually be very difficult. So yeah, it's very unlikely for that to have
happened.

~~~
dmix
Well, getting an SSL private key is difficult as they don't often get into
memory and are quite long (difficult to get from 64k at a time). Whereas AWS
credential keys are something that get into your servers RAM much more
frequently and are shorter strings. So it could easily be remote memory
exploitation. But more likely social engineering or some other easy path in.

~~~
mschuster91
Heartbleed only exposed SSL memory (like incoming or outcoming connections),
but not other memory (particularly not program memory), containing AWS keys.

------
abhimir
A similar thing happened to me when I made public a previously private repo on
Amazon and forgot to scrub it for AWS key. Luckily I got an email from Amazon
within 24 hours of the instances being started, and as such my bill was just
$360, which in any case they waived off. You might want to check your Github
repos.

------
jes5199
I have 2-factor auth enabled on my AWS login - but am I right in thinking that
if someone has my API keys that they don't need the 2nd factor?

~~~
prattbhatt
Yes, someone with your access and secret keys can spin up instances, create
buckets, and do everything else that the stolen keys are authorized for.

~~~
ceejayoz
Which is why most things should be done with IAM keys specifically locked down
to minimal privileges.

~~~
jes5199
Apparently I have access keys that predate the release of IAM ! Fortunately
there's a convenient "disable" link on security keys page.

------
Havoc
The fact that the consultant - unprompted - knows exactly what is going on
suggests to me that Amazon needs to fix this. If one customer gets burned then
the customer is an idiot. If lots of customers get burned then maybe its not
the customers that are the problem.

------
sandy23
Is there anything you have done that might have compromised your account.
Moreover bill could as well been much larger than $5000 i guess.

------
josephagoss
Is it too pedantic to want the tech support to just say they were probably
mining cryptocurrency instead of bitcoin? It's most likely the intruder was
mining either litecoin or whatever coin is most profitable for the month. I
know it's all very much the same but they were almost certainly not mining
bitcoin.

A $5,000 AWS instance would mine about $1 worth of bitcoin and would not be
worth the time logging into someones account.

