
AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely - dustinrcollins
https://aws.amazon.com/blogs/aws/aws-secrets-manager-store-distribute-and-rotate-credentials-securely/
======
scrollaway
First reaction: Holy crap! They finally turned ParameterStore into a proper
product!

Second reaction: Holy shit that's expensive [for what it does].

ParameterStore is free (minus the KMS component). The only value-add is secret
rotation and that's not something that most of the time makes sense to use.
[Edit: I'm not advocating for no rotation; see replies]

Edit: Had more time to think about it. Someone enlighten me: What's the
difference between writing a rotation lambda for this new product, vs. writing
a rotation lambda for ParameterStore that you then cron? The pricing really
doesn't make sense.

~~~
toomuchtodo
> The only value-add is secret rotation and that's not something that most of
> the time makes sense to use.

From a security perspective, you should be rotating secrets somewhere between
annually and every 90 days, depending on your business/security/compliance
requirements and the nature of the data secured by the secret.

~~~
tptacek
It depends on the secret and the degree to which the secret is exposed. SSH
creds should get rotated _constantly_ ; a one-hour SSH login cred is a
significant exposure. But an API secret that is kept in Parameter Store and
not exposed to developers doesn't really benefit from rotation every 3 months
in proportion to the amount of mechanism required to do that.

~~~
JoachimSchipper
What are you assuming about exposure? If a SSH key lives on a well-secured
workstation or bastion host (and you ideally don't agent-forward it to
insecure hosts), rotating that key once per hour doesn't seem a top priority
to me? E.g. a sudo password is (lower-impact, but) more likely to get exposed
to compromised hosts?

("Well-secured workstation" is arguably an oxymoron, of course...)

~~~
cassianoleal
Wait, you keep private SSH keys on bastion hosts?

It's much better practice to use the bastion as a proxy to the other hosts.
This is easily achieved using the ProxyCommand option of OpenSSH.

No agent forwarding, no secrets kept in random hosts.

~~~
scrollaway
Do you know a good guide with standard practices for setting up and securing
bastion hosts (preferably on aws)?

~~~
toomuchtodo
If you don’t require interactive sessions, consider using AWS SSM run command
[1] instead. You install the agent on the instances, with commands sent from
the client through the AWS control plane (with IAM and SSM documents for
access control and CloudTrail logs of all commands issued).

I’m currently deploying it in an enterprise for ~5k users, and it works
surprisingly well for providing the ability to run arbitrary commands on
instances without ssh access.

[1] [https://docs.aws.amazon.com/systems-
manager/latest/userguide...](https://docs.aws.amazon.com/systems-
manager/latest/userguide/walkthrough-cli.html)

------
whateveracct
This looks like an AWS equivalent of the Amazon-internal secret management
tool called Odin. Which is very nice because Odin was pretty much universally
loved from what I saw.

~~~
ronnier
After leaving Amazon you really go on to understand how good the internal
tools are at Amazon and how they probably have the best tools in the industry.

~~~
mabbo
Brazil + Apollo + Pipelines. I left (and later boomeranged) and was surprised
that the outside world hadn't solved these problems nearly as well. Don't get
me wrong, Brazil and Apollo probably are ready for a rethink and rewrite, but
they do a great job.

~~~
mrep
What would you change about them?

~~~
mabbo
Hard to say (especially in as public forum like this).

Brazil's dependency model is, I think, very much worth rethinking. Just try to
use it for NodeJS. Apollo is fine but slow and dated and needs love. Pipelines
works for specific software models and not others (ever seen a pipeline with
500 envs?). Great design for building custom approval steps though (long ago I
wrote a now-popular one).

------
sowbug
For those of us living in the service-development stone ages, is the idea that
a secret-manager service replaces any number of ad-hoc local secret-storage
and configuration mechanisms with a single robust mechanism that takes only a
single root credential to retrieve all the individual secrets that your
service needs?

You do still have to figure out a way to securely provide the root credential
to your service so that it can fetch the secrets from the secret manager,
correct? Otherwise this would be magic of a kind I think is impossible.

If my questions aren't too far off in the weeds, then this service sounds like
a personal password manager but for a service rather than a person, though I'm
sure AWS's service has finer-grained controls than just the all-or-nothing
master passphrase. Similar risks apply: an attacker obtaining the master
passphrase is a major issue, losing the master passphrase is devastating
(though recoverable here because you probably didn't lose your personal AWS
login credentials), and unavailability of the password database is
catastrophic. But the usability benefits of having everything in one secure
place, behind a service managed by experts, should outweigh those risks.

I have more questions about the credential-rotation feature, but this is
enough for now.

~~~
lmartel
The big missing piece is roles. No service uses a root access key directly.
Instead, there's a webserver role with access to a relevant secrets group but
no access to data warehouse secrets, for example.

Access keys can be provisioned and downloaded straight onto the box from the
service. Sure, a compromise is bad, but only exposes the secrets that would be
available on the pwned box regardless.

~~~
sowbug
OK, so "root" wasn't really the right term. I get an X credential so I can be
an X, and nobody needs to worry that I also got enough to be any part of a Y.
Thanks.

~~~
notoverthere
This sort of model also fits nicely with the AWS ecosystem. EC2 instances
(virtual machines) can be given an IAM Role when they boot-up. An IAM Role is
essentially an automatically generated access key which is unique to that EC2
instance, and has pre-determined permissions.

So in other words – a unique key is generated every time a virtual machine is
created. It's fully automated, never shared between instances, and never needs
to be handled manually. That key will give the virtual machine permission to
access other AWS services, in this case the AWS Secrets Manager.

So as long as you're using EC2 instances, you won't need to worry about
securely passing a 'master password' to your VMs in order for them to access
secrets.

~~~
sowbug
I see. So there actually is a bit of magic involved.

I exist, therefore I can.

~~~
mwarkentin
Yep - they have similar functionality for task roles (aka docker containers on
ECS) and Lambdas as well.

------
kondro
At $0.40 per secret per month and $0.05 per 10,000 requests this is much more
expensive than the practically free SSM Parameter Store product, even if you
factor in the auto-rotating bits.

~~~
0xCMP
It'd be cheaper than running Vault with a backing Consul cluster which also
provides rotation and other features.

There is a point where Vault is more cost effective, but I believe it'd
require a ton of requests and secrets to justify min 6 machines of at least
t2.micro that also need to managed and secured.

~~~
013a
Depends on how many secrets you're storing.

You can also back Vault with something other than Consul. You can back it with
DynamoDB, which would be much cheaper than managing your own Consul cluster.
You can even back it with S3, which would be dirt cheap (cost of the vault
instance + a few cents for storage).

~~~
0xCMP
I wasn't aware you could use Dynamo or S3, that's pretty interesting

------
epberry
Besides the secrets rotation how is this different from EC2 Parameter Store?
I’m genuinely curious and will move away from parameter store if this provides
some benefits.

~~~
neuronexmachina
Skimming the blog post, the main difference seems that it allows you to
basically store a dict of key/value pairs for each secret. So for example, you
could store all the user/pass/host/port for a DB connection as a secret. If I
recall correctly, Parameter Store could only store a single SecureString for
each secret.

~~~
scrollaway
ParameterStore lets you store String, StringList or SecureString. But there's
no limit to SecureString.

A SecureString can be `postgres://admin:hunter2@localhost:5432/db`. It can
also be `{"username": "admin", "password": "hunter2", "host": "localhost",
"port": "5432"}`.

~~~
neuronexmachina
Sure you could do that if you don't mind adding in your own string->dict
conversion. I wouldn't be surprised if the internal implementation is near-
identical. Secrets Manager seems like a slightly more easy-to-use version of
Parameter Store that's also visible, instead of hidden away inside Systems
Manager.

~~~
epberry
It did take me a minute to actually find Parameter Store. There's so much
stuff hidden away in the EC2 menus.

------
013a
> $0.40/secret/month

WOWZER. I get having a managed solution is great, but you don't have to store
many secrets before running your own Vault server makes sense.

~~~
crazypyro
At my work, we have a vault service, but it takes 3 servers for vault plus
another 3 for consul and they are not very cheap servers (m4.larges). Our
rough calculations, using a 3x3 vault/consul architecture, estimated you'd
need over 1100-1300 secrets to make it worth implementing vault (not including
development/maintenance cost which could be significant. and if you have
multiple, independent environments, it gets worse.

The real issue for some of the people at my work is the vendor lock-in versus
vault.

~~~
outworlder
Consul only makes sense if you are already using it for something else. If
just Vault, you can use DynamoDB (with full HA) or S3 (without HA support),
same with Azure and Google Cloud (although without HA). There are other
backends, like PostgreSQL.

------
thepumpkin1979
Kinda similar to what Hashicorp's Vault does for secret management but hosted.

~~~
ryanSrich
There's really no comparison. You could use Vault to build a PKI. You couldn't
say the same for a proprietary, vendor locked solution like this.

------
marvinpinto
I'm still trying to figure out how different this is from their "parameter
store" offering in AWS Systems Manager. The main thing I guess is you get more
control over key rotation.

------
Thaxll
Looks like the equivalent of Vault? Anyone can compare the two?

------
netvisao
I am curious as to how it handles those race conditions where a connection is
made with the older credentials just after the time the rds master key
rotates, or a connection is made with the newer credentials just before the
time the rds client key rotates. Short of using two credentials accounts ...

~~~
ranman
there’s a pretty nifty 4 stage rotation mechanism that solves most, but not
all, of those race conditions. More info on it in the post and the docs.

~~~
netvisao
Are we talking about these? createSecret setSecret testSecret finishSecret

These were mentioned around the custom rotation strategy, any pointers to docs
describing how it is done if the secret is an RDS Secret?

------
jtwaleson
As others have noted, the pricing is really the odd thing here. AWS seems to
be moving to value based pricing rather than cost based pricing for some of
its niche products.

I used to think that Cloudwatch metrics were very expensive at $.50 per custom
metric per month, but this seems waaayyyy cheaper to store.

------
chatmasta
How is this different than KMS? “Key Management Service” is practically
synonymous with the name of this new product, so how exactly do the two
differ/interact?

~~~
mratzloff
Well, you can use KMS to do something similar, but without the auto-rotation.

1\. Use a KMS key to encrypt a secrets file (obviously, never check this into
source control)

2\. Store the encrypted secrets file in an S3 bucket

3\. Tie a new IAM role with kms:Decrypt and s3:GetObject policies for the
relevant resources to your EC2 instance

4\. On app start, get the KMS key and secrets file, decrypt, and set
environment variables

In practice, rotating using this scheme just means creating a new KMS key, re-
encrypting the file and pushing the updated copy to S3, and updating the IAM
role's kms:Decrypt policy. It's not too bad unless you have a million
services.

~~~
mattwad
This seems like the easiest solution, and we've been pretty successful. We
have sub-folders based on your environment and only machines have roles that
can read production. We also realized you don't even have to encrypt and
decrypt.. just turn on encryption for the S3 bucket, I believe it's the same
thing.

~~~
jeremya
It's not exactly the same thing. Encryption at rest on S3 is good, but
security is about layers. If you only use that, then you are one mis-
configured S3 bucket policy away from exposing your secrets. Bad bucket
policies are one of AWS' biggest footguns- plenty of intelligent people have
done it before, don't assume it's a mistake you'll never make. It's so easy
because S3 is a service that has to make publicly-accessible storage easy to
achieve.

KMS keys on the other hand, do not have that use case. If the secrets in your
S3 bucket have been encrypted using a KMS key (or better yet, a data key
derived from a KMS key) and your bucket is compromised, your secrets are safe
as long as the attacker cannot ALSO get access to the KMS key. By all means,
enable encryption at rest on S3, but don't make that your only defense.

------
talawahdotnet
At $0.40 per secret it would have been nice if they had a 5 secret free tier.

That would get smaller users to start using it instead of parameter store and
eventually realize the value of automated and audited secret rotation

~~~
kungito
Yeah I'm thinking about starting to use it for personal projects but don't
want to bother with these 2 dollar bills. I'm pretty sure they can benefit
from me playing with it first and then advocating for it at work

------
pageandrew
Does the Secret Rotation for the RDS-integrated credential store actually
update the password for the user in the SQL database?

If so, thats pretty damn cool.

~~~
rconti
Oh, I misread it as a secret for Twitter (which, I guess, makes no sense,
because tweets are generally public).. and I was wondering how the password
got updated on the Twitter side. So I guess, naturally, secrets can only be
rotated for AWS services that support it.

~~~
mwarkentin
You can write the custom lambdas to rotate 3rd party creds.

------
ppierald
There is a lot of commentary about the use of vault as an alternative, number
of secrets needed, etc. I think the inclusion of Secrets Manager is a great
addition for AWS and will definitely help people get better control over their
secrets, however, vault contains richer functionality than just secrets
key/value storage.

It can provision users to backends like SSH, databases, cloud providers, and
such. Use is audited, can be revoked, and has a TTL associated.

Additionally, vault contains a full "crypto-in-a-box" implementation that
allows for sign/verify, hmac/verify, encrypt/decrypt, random number
generation, and other functions.

So I applaud AWS for doing this and hope the will continue developing
KMS/HSM/Parameter Store/Secret Store/??? in the future and innovating, but
evaluating Secret Store vs. Vault simply on price may be a short sighted
comparison.

Disclaimers: Employer is an AWS customer using vault

------
bestbear
Compare this to the cost of Hashicorp Vault. You might be surprised at the
difference.

------
spullara
$0.40 per secret per month is outrageous.

------
drodgers
I was hoping to use this for RDS parameters (to get automatic rotation) and
leave everything else in ParameterStore.

Unfortunately — although I'm sure it's built on-top of ParameterStore
internally — I just checked and you can't see SecretsManager secrets in
ParameterStore, so an application would need to read from both and merge them,
or switch entirely to SecretsManager to take advantage of the automatic
rotation.

------
0xCMP
This will be nice for systems and budgets which can't afford a consul+vault
cluster to handle this for you.

------
outworlder
"Not just for secrets" Yay, we can go and store all types of secrets!

But not with this price per secret...

------
kerng
Anyone know how this compares to Azure Key Vault?

~~~
spydum
Vault already did secrets. Nothing stops you from writing an azure function to
handle rotation, but you gotta build it yourself. AWS has BUILT IN support for
RDS.

However, the biggest delta is: I'm like 90% sure Azure key vault doesn't have
fine-grained access policy per VAULT. That kind of stinks.. you need a vault
per role ideally.

------
tyingq
Au revoir Cyberark. I won't miss you.

