
Using a single AWS account is a serious risk - hellomichibye
https://cloudonaut.io/your-single-aws-account-is-a-serious-risk/
======
cperciva
For Tarsnap, I have several _completely independent_ AWS accounts. This is
partly because I needed privilege separation before IAM existed; but I keep
this setup mainly because it's dead simple and completely avoids the risk of
user error: When I'm doing development work, I don't even have access to the
Tarsnap production accounts.

~~~
objectivefs
By using IAM roles for EC2 you can also move more of the key management to
Amazon. When you attach an IAM role to an EC2 instance Amazon will provide
temporary credentials through the local metadata server and automatically
rotate them for you. This avoids storing and copying your credentials, and
makes it easy to separate your environments.

~~~
cperciva
IAM roles for EC2 terrify me. They completely break OS privilege separation,
since every user has access to the keys.

~~~
xenophonf
The servers I have set up this way aren't really multiuser machines, but your
point is a good one.

------
impostervt
About three weeks ago I started working on a new open source project meant to
interact with AWS. Coding fast and dumb, I cut and pasted my personal AWS
credentials into my source code, committed it, and pushed it to Github.

The next day I got an email from Amazon, alerting me to the problem.
Apparently, they scrape github looking for just that kind of stupidity. I
instantly deleted the project, but it was too late.

Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully.
I'm now a zealous advocate for making sure a person never even HAS AWS
credentials. Instead, make a new user without a password for each use case,
and manually select the privileges that account has.

If you have a password to AWS, you shouldn't have credentials.

~~~
e40
I posted to the AWS forum and accidentally copy/pasted my secret key. Within
24 hours $11k of charges. I called them, they wiped them. It's amazing how
quickly people find and use these things.

What kills me is there is no easy way to stop _all_ the instances for an
account, in a region. It took me hours to kill all the instances. They had
maxed out the number of instances in every single region. Very, very annoying.

~~~
mrestko
Any idea what they were doing with them? Mining bitcoins? Hosting CnC for a
botnet?

~~~
fpgaminer
It's always Bitcoin mining. The attacker spins up a bunch of GPU EC2 instances
and mines as long as they can. I don't think the profit ends up being very
large (those GPU instances put out a puny amount of hashing power compared to
modern mining ASICs)...

~~~
icelancer
ROI for BTC mining on standard AWS pricing is about -90%. So attackers get 10%
free BTC on spend.

~~~
tajen
So, out of $11k, they make $1k. That's the kind of money they need to make
between once and three times a month minimum depending on where they live.
Doesn't Amazon notice patterns in terms of source, scripts which are uploaded
and scaling profile: Who uses 2 medium instances for a year then spins off
2,000 in 20 minutes?

~~~
icelancer
Certainly there's a pattern there, but it's not THAT far away from people who
intermittantly scale stuff for short bursts of huge processing. False
positives in those cases for people that really intended to spend $10-50k in a
short time might mean a HUGE loss of revenue and/or incurred customer support
and service costs.

------
developer1
"If someone gets access to your AWS access credentials, you’re in trouble."

I know we're not supposed to post negative comments that don't "add value" to
a discussion, but the only thing that comes to mind is "really?". Your setup
is as secure as you make it. How you use your API access is up to you. Putting
all your eggs in one basket is not insecure. This article doesn't actually
bring to light anything important. There is no risk involved so long as you
pay attention to what you are doing.

Any set of credentials, if leaked, destroys security. So... don't set yourself
up to leak your credentials? I mean, come on, seriously?

*edit: I unfairly used the word "incompetent". Change to a phrase about paying attention to what you are doing.

~~~
zippergz
Pay attention to what you're doing, and never make a mistake. Or have anyone
in your team/company make a mistake. While you're at it, please make sure you
don't put any bugs in the code also.

We're humans. We're imperfect. Sometimes a safety net isn't a bad thing.

~~~
angrybits
Except for the fact that you can explicitly contain the potential damage by
controlling which accounts can do what. It's not like you have to have one god
account with no MFA that has the keys to the city.

------
Havoc
//Somewhat off topic

Using any kind of AWS account (in a personal capacity) is a serious risk in my
eyes.

Its much like I'm happy to take the risk of buying shares, but don't like
leveraged derivatives.

I don't want to deal with something that is not my area of expertise and has
very real potential to blow up in my face.

I wish Amazon allowed me to cap spending to say 200USD...that I can plan for
much easier than X thousands.

~~~
eropple
If you look around (or spend a few minutes writing one), you can use a scram
switch that goes off of AWS billing alerts.

~~~
Havoc
>AWS billing alerts

I don't want alerts, I want a fixed amount limit. If watching this sht was my
day job then sure I'd be OK with billing alerts. I'm busy fighting other fires
in my day job though & often won't see mails for days. By that time Amazon
might have bankrupted me.

Amazon has some decent engineers & this is not technically difficult. Its very
obviously a concious decision to not provide this very useful feature as any
kind of capping feature will negatively impact revenue...but its also
literally the only reason why I don't have a paid AWS account for personal
use.

~~~
eropple
_You 're in AWS._ Why do you think alerts mean you need to be watching for
them? Alerts come from CloudWatch, to SNS, and from that can go wherever you
want them to. So do that, with the tools they've already given you. And this
lets you kill only what you want to kill--I mean, do you want them to dump all
your S3 buckets? Or just, say, down all your EC2 nodes?

It's not about capping features being negative for revenue, it's that the
customers who want it _don 't matter_; if it's not something a company
dropping $20K/month is going to use, I doubt it's really worth the time of
day. You are a Linode or DigitalOcean customer, and _that 's not a bad thing_,
but I think that AWS is not the right place for you and I think that they know
that.

~~~
Havoc
>from that can go wherever you want them to

You miss my point - its not a technical problem. As I said this isn't my day
job.

Sure a hacker/sysadmin/whatever could re-route the AWS warning to whatever
system. For me this would be a hobby/toy though and I can't have a toy that
exposes me to a potential multi-thousand bill out of the blue. Thats not a
hobby - thats a gamble.

As I said in my initial post - its like leveraged derivatives. If I were a day
trader living that reality then that kind of uncertain would be acceptable.
I'm not a day trader though so I don't want that kind of blank cheque
responsibility on something I'm not actively watching.

~~~
eropple
Complaining that they don't cater to people who don't make them even couch-
cushions money and won't use the tools they have to do _exactly what they
want_ is really silly.

Not all services are intended for all types of customer. Right now in their
business lifecycle, you're the kind of customer AWS should fire.

~~~
Havoc
>won't use the tools they have to do exactly what they want

What I want is to reduce my _legal_ liability from unlimited to something less
than unlimited.

Show me how to do that and I'll concede that you're right & I'm wrong.

~~~
wpietri
I believe his point is that you aren't their target customer. It's like
walking into a freight distribution center and asking to buy $0.50 worth of
chewing gum. Sure, somebody might accommodate you, but complaining about them
not doing so doesn't make a lot of sense.

In Amazon's shoes I'm not sure I'd offer a feature like that at all, because
someone might set that to be "smart", forget about it as the business grows,
and then have a bunch of servers terminated and/or data destroyed right during
an excitingly busy period. It would be an extraordinarily bad customer
experience. It could be much better for them to do as now and then
occasionally credit people who accidentally go over.

~~~
vacri
> _and /or data destroyed right during an excitingly busy period_

The idea that service providers destroy data when a cap is reached is just
plain weird. Usually your access to the data is simply blocked - then you can
pony up some more $$ if you want access to it.

Similarly, it's also weird that people think Amazon doesn't care about 4- and
5-figure accounts. They do. Big accounts bring in money... but so do a lot of
small accounts... and there are a hell of a lot more small accounts than big
accounts. At my last job our monthly bill was $1-2k, and I'd get regular
contacts from the AWS account manager, along with free training days.

I think that the real reason is that AWS is generally not at capacity, and if
someone does have a hostile overage event, it's quite short term, and the only
money that AWS really loses is from the wattage used by the VMs. There's no
extra labour or hardware on their side. Refunding overages is good PR and not
all that expensive, as long as it doesn't prevent other paying customers from
accessing existing resources.

~~~
wpietri
> The idea that service providers destroy data when a cap is reached is just
> plain weird.

This fellow is proposing an account that's capped. In AWS-land with by-the-
hour servers, that would presumably mean terminating running servers. Servers
that have data in RAM. Servers with volatile local disks that get recycled as
soon at they're terminated. Maybe you have an idea of how to shut down all of
Amazon's 40-odd services with no risk of data loss, but I'm not seeing one.

If they're not going terminate them, then either they're going to keep
charging or not. If they keep charging, then it seems like it's more a billing
alert plus a service interruption. If they don't charge, then it's more like
what they do now, which is waive charges for mistakes, except with a service
interruption.

> Similarly, it's also weird that people think Amazon doesn't care about 4-
> and 5-figure accounts.

I think they do care about them. I'm just saying they won't go out of their
way for the $50/month accounts if it means putting serious customers at risk.

~~~
vacri
You're mixing and matching your incredulity. On the one hand, AWS has 40
services. On the other hand, everything is terminated like an EC2 instance...
Which really isn't the case.

So, to begin with, most of their services can either be powered down (with
chickenfeed storage costs that they could cover as part of the deal) or are
outright free. Most things that you fork over money for in AWS can be disabled
with some form of networking block. Have code in Lambda? Well, it doesn't get
destroyed, it just gets blocked. Have RDS databases or Elasticache? Block
access, and perhaps power them down after X time (which saves them to s3, and
block storage for AWS's internal use is very cheap - _retail_ s3 is
3c/GB/month). S3 itself also gets your access cut. SES and SNS just stop
processing their queues. Things like VPCs and IAM are free to begin with. The
costs of keeping these things running behind a block is trivial compared to
the overage charges they already routinely waive.

And then we come to EC2... and the story still isn't 'must be terminated'. EC2
instances can be powered down and ELB access blocked, leaving the config all
in place and the instance's drive saved to s3 (which is where AMIs and volume
snapshots/unpowered instances live). Yes, you will lose data in RAM, but you
just get the account holder to accept that the machines can be shut down by
AWS, just like already happens with spot instances. If the client _opts in_ to
capped pricing, then they can take that into account and design their system
around the sudden downing of the instances.

~~~
wpietri
I certainly didn't say everything is terminated like EC2, so I don't know
where you're getting that from.

And it looks like we agree: any attempt on Amazon's part to create something
like this would be complicated and would still not remove the risk of data
loss. An service interruption is, of course, a certainty.

So as far as I can see, the feature still doesn't make much sense. It's only
really useful to people who aren't doing anything in Amazon that matters.

------
astral303
Another risk consideration is that anyone getting unauthorized access to your
AWS account can delete all your resources and all your backups (snapshots,
etc), effectively putting you out of business. [1]

One solution is to backup to a separate backup-only AWS account, with super-
serious access controls (MFA and password physically locked away somewhere).
Set up a "write-only" link, such that backups can be added, but never removed.
This way, in the worst case, your runtime infrastructure can be decimated, but
your data backups would be safe.

1 - [http://arstechnica.com/security/2014/06/aws-console-
breach-l...](http://arstechnica.com/security/2014/06/aws-console-breach-leads-
to-demise-of-service-with-proven-backup-plan/)

~~~
badmadrad
I prefer this method. Having multipe aws dashboards sounds like a nightmare. I
would rather use the backup account approach with MFA tied to the
administrator accounts on each aws site.

~~~
eropple
Multiple AWS dashboards is actually really easy with multiple Chrome profiles
(not that I go to them often, there are APIs for a reason) and tends to
encourage much better, much more isolated application design. I've worked in
environments with a single account and in environments with multiple accounts
and I can't imagine going backwards.

------
kikibobo69
At Zalando we have an account per team, and have been releasing a bunch of
tooling to help do this securely.[1] It's not completely easy (e.g., account
creation at Amazon can not be automated), but the security aspects are really
nice, and it lets us give teams more or less full access to just their
account.

[1] [https://stups.io](https://stups.io)

------
whisk3rs
The MFA Condition is a huge win, and I'm surprised Amazon hasn't built this a
tool to make this easier yet.

However, I question the merit of using two separate AWS accounts. While this
separation of responsibility sounds nice in theory, doesn't it introduce
additional maintenance burden because you now have two accounts to administer?
You can't define or manage the roles in the 2nd account without credentials to
do so.

~~~
Rapzid
Usually those other accounts have much fewer people with access=much lower
risk of unauthorized access. For instance, I'm a big proponent of a backup
acct that the main acct can push to but not delete from. That backup acct can
have very limited and tightly controlled access. It's unfortunate RDS does you
no favours in helping out with this; you pretty much have to dump your DB and
push it off AWS or into another acct's S3 bucket.

IAM is just a PITA is what this boils down to. Create an IAM policy that
allows users to push updates to elastic beanstalk but not touch any other
resources in the account.. It's a major, major hassle. AWS has no concept of
resource groups and each service has different ways of restricting access(ec2
can do it on tags, other services you kinda have to use naming schemas and
wild cards in your policies). So you are often left needing to have users with
a little too much access, and/or spending a LOT of time testing and crafting
IAM policies..

IAM is a really good idea and powerful in many ways but unfortunately AWS's
lack of consistency and UX across individual services really shows through
sometimes, and with IAM in particular.

------
djhworld
I think the comment about using temporary credentials is an interesting one.

Where I work, we have a federated access system that generates temporary
credentials so you can access the Console without having to sign in, and it
uses our work authentication mechanism to do the initial handshake.

The thing is this process is useless if you want to use it with the AWS
command line client, or any other tools that rely on the use of credentials.
It would be nice if AWS adopted a plugin system or something where you could
plug in a "credentials provider" of some sort, and the command line client
queries that for credentials each time you make a request, instead of having
to stash keys in ~/.aws/credentials

------
chris_wot
Is there a good primer on AWS and what it provides?

------
anu_gupta
Is there an easier to use UI for setting up IAM users. I get hopelessly
confused every time I attempt it, and worry about losing access to services
I've already set up.

~~~
fletchowns
I don't think so. Mine is always a painful experience of trial and error using
a combination of things cobbled together from the policy generator, manual
editing, copy/pasting from AWS documentation, and using the policy simulator.

And then after hours of that I can determine that IAM policies fall way short
off what I needed (isolation between different product groups in the org) and
that we need to use separate root accounts.

------
nodesocket
I like the approach that Google cloud takes with projects. Projects are
independent and isolated under a single account.

------
setheron
What's cool is that in AWS as a developer they disable the ability to even
login with username and password. You can set it up such that the only access
is through a site that grants federated access.

------
azinman2
Like the solution via MFA & temp credentials! Simple & elegant.

------
autotune
You can also use ssh forwarding to log into your servers through the SSH
bastion without dropping your private keys everywhere on the bastion itself,
securing access even further.

------
alexchamberlain
I seriously question any business running just on AWS. There are plenty of
other services that "look like" AWS and give you complete isolation.

------
evook
aye aye captain obvious!

