
How I became $12,000 Amazon's debtor just in 24 hours - manlog
https://bitcalm.com/blog/howi-became-12000-amazons-debtor-in-24-hours/?utm_source=news.ycombinator.com&utm_medium=referral&utm_campaign=hacker.debtor.04.04
======
sytelus
Amazon has avoided allowing users to set hard billing cap and that makes me
worried a lot too. If your website gets suddenly too popular or DDoSed then
you can get toast and might even go bankrupt. Amazon has reasoned that they
don't want to disrupt your service by enabling this feature (because
technically their safest choice to implement hard cap is to shutdown all
resources assigned to you until you intervine). But as a user that's exactly
what I want! I rather have my service go down then me go bankrupt from my ad-
free website. It should be customer's choice to enable this feature if they
want to but Amazon doesn't want to do it regardless of their supposedly
"sharp" focus on customer needs.

The feature they did enabled is CloudWatch where you get alerts if you go over
the limit. But that's rather weak. If it happened while I was sleeping or on
vacation, it is very easy to miss these notifications and face giant financial
losses in just few hours. Having a non-commercial service hosted on AWS is
like a sword hanging over your head that can fall anytime on you.

~~~
Terretta
You can't set a dollar figure but you absolutely can set resource count caps.

The dollar figure is less useful. If you set $1000 should it kick in when you
hit it 12 hours before the end of the month? What if you hit $990 by the 2nd
day of month? These are wildly different, and the dollar figure doesn't know.

Service Limits, on the other hand, let you be sure the max you use every day
won't overrun your EOM budget.

[http://docs.aws.amazon.com/general/latest/gr/aws_service_lim...](http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html)

~~~
manlog
But, why the hack they don't allow this option? Those people who wants to do
that - let them go ahead... What's the problem with that?

------
altcognito
Kudos and smart thinking on Amazon's part to vacate these fees. If you heard
the opposite story (about how people were filing for bankruptcy as a result of
this), there's no way I'd take the chance of using the service, and I would
feel hesitant to recommend their services to small businesses I work with.
5k-10k for some of those groups is their entire operating budget. (mostly
volunteers)

~~~
manlog
You never know... unfortunately

------
tbronchain
I found that project recently [https://code.google.com/p/amazon-
pricewatcher/](https://code.google.com/p/amazon-pricewatcher/)

It's old, and probably won't work. But the idea is interesting. It monitors
your AWS account, and shutdowns EC2 instances/removes S3 files if your bill
goes up. It could be interesting to re-write it using boto. Maybe even make it
simpler: If the bill goes over a "hard" limit, shutdown anything that can be
shutdown (but don't delete anything!). I guess I would prefer to have a short
downtime than a crazy bill. Added to a well setup alert monitoring, it could
be a great way to limit crazy billing.

@WestCoastJustin point to use dedicates roles account seems like a quite
import thing as well. I think I would also trend to avoid large-range ASG.

~~~
manlog
Nice idea!

------
WestCoastJustin
There has probably been a half dozen of these threads on HN, and every single
time, they have the fees waived. You see a much different strategy with public
telco horror stories regarding roaming/data fees and things like that.

There are a couple pro tips for anyone interested:

\- Configure billing alerts, so that if your bill goes over an expected
threshold you will get an email/sms/etc [1]. If your bill is usually
$20/month, then set the alert trigger for $30/month, and you will get an
alert. In this case, it looks like they just racked up the bill, likely via
crypto-currency mining, but what if they were deleting data, etc. You want to
be pretty proactive about this stuff. This only takes a couple minutes.

\- Use dedicated role accounts, rather than the AWS root account, so that you
can roll keys without a huge impact [2]. You can also give narrow rights to
your role accounts, so that they do not have access to everything, this will
greatly limit the scope of an attack. It seems that people mistakenly publish
their root AWS keys to github, then this type of stuff happens, but luckily
AWS is pretty good about it. Although, AWS cannot help if you had your
database exfiltrated, or production instances deleted, so you really _MUST_
create role accounts, and limit their abilities [3].

\- Monitor the activity on your account [4]. Create some type of alerts so
that you will get notified about suspicious activity on your account. For
example, if you do no generally create new instances, have an alert fire on
this event type, via cloudtrail [5]. Same goes for deleted instances, etc,
then you will be in the loop at things are happening.

[1]
[http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/...](http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-
what-is.html)

[2] [http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-
creati...](http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-
creatingrole.html)

[3] [http://arstechnica.com/security/2014/06/aws-console-
breach-l...](http://arstechnica.com/security/2014/06/aws-console-breach-leads-
to-demise-of-service-with-proven-backup-plan/)

[4] [http://docs.aws.amazon.com/general/latest/gr/aws-security-
au...](http://docs.aws.amazon.com/general/latest/gr/aws-security-audit-
guide.html)

[5] [http://aws.amazon.com/cloudtrail/](http://aws.amazon.com/cloudtrail/)

~~~
manlog
You can't imagine the feelings a person experiences, when he sees 12k+ debt on
an almost not used account!

