
Why We Need Dynamic Secrets - mafro
https://www.hashicorp.com/blog/why-we-need-dynamic-secrets
======
maxxxxx
Secret management can be a really infuriating problem. No matter what you do
there never seems to be a good solution between dealing with employees who
know passwords leaving, automated processes needing credentials and other
business processes. It's one of the toughest problems I have ever seen without
a good solution. At least not one that the business people will accept without
complaining that it's too expensive or burdensome.

~~~
XorNot
Secret management tends to suffer from the logical holders of secrets
(managers) not actually having the time or expertise in a lot of cases to deal
with the implementation (or even understanding) of the process.

~~~
mahmoudimus
This. The lack of focus faced today from key custodians and the lack of
undergoing continuous training to understand the different threats and keeping
up-to-date with implementations is a gaping hole in this particular type of
model. It's never a core competency, which is why the industry needs a
different paradigm.

~~~
Piskvorrr
As in, "Job title: The Holder Of Keys"?

------
viraptor
This is all cool, but... once you get to a setup like this (which has many
moving parts and config rewriting), why isn't that solution compared to
kerberos? You get the temporary credentials giving you similar possibilities.

~~~
armon
Kerberos was a major inspiration for us!

The goal of Vault was to be a modern Kerberos, but invert the integration
model. I think that is the Achilles heel of Kerberos, since it has a complex
API and only works if the endpoint systems are tightly integrated.

Vault operates in much the same way, and could be viewed as a "KDC". However,
instead of requiring the Authentication Service (AS) to be Vault aware, Vault
uses authentication plugins to do the integration in the other direction.
Similarly, instead of network services being Vault aware, Vault uses secret
plugins to do the integration with endpoint systems.

This lets Vault easily be extended to support new authentication systems and
endpoint systems, without needing those systems to be modified. Otherwise, its
conceptually very similar to Kerberos!

Edit: I'm a co-founder of HashiCorp, and one of the early authors on Vault.

------
mahmoudimus
This is a great first step on building the right kind of infrastructure to
stop applications from exposing secrets. Leases on secrets with dynamic policy
enforcement is, IMHO, the future of application security. Happy to see
companies like Hashicorp discussing more of the vision on how Vault can be
used to secure applications that we depend on for infrastructure.

I would caveat, however, that encryption is just one part of the tool in the
toolbox. A correct security posture is about having layers of defense. The
next layer is absolutely the application and isolating secret management
responsibility from the app layer to the infrastructure layer is the right
strategy, IMO.

------
sapphire_tomb
I'm still stuck on "Yeah, but I need a credential to authenticate with Vault
to get the other credential that I want".

Very chicken and egg.

~~~
infamousjoeg
That was my first thought, too.

"But... there's a root account... creating another account... which is
typically a privileged action. What's protecting that? Is that root account
being rotated, too?"

~~~
Flenser
the master key is split into shards. See under "Why" here:
[https://www.vaultproject.io/docs/concepts/seal.html](https://www.vaultproject.io/docs/concepts/seal.html)

~~~
infamousjoeg
Sure, the master key is split into shards...

I'm talking about the account creating the database user. Let's take MSSQL,
for example. The equivalent to a root account there is `sa`. So, Vault will
have control of the `sa` account in order to create leased database users.

If I'm a malicious actor inside the environment, what's stopping me from
compromising the `sa` account and mimicking dynamic secrets? I'd need to be
comparing Vault logs with Database logs constantly to ensure it was legit.

It's just madness, if you ask me.

~~~
zie
In some cases, vault can change the password of the root account on the
service, so that only vault is known to have the password. But you are
correct, there is a problem here. There is nothing stopping you from attacking
the DB or service directly.

You can do a lot to minimize that, many DB auth systems let you specify where
connections for users are allowed to come from, only allow a single concurrent
connection, etc. Plus you can encrypt and verify that connection with TLS,
build up FW rules, etc.

Security is never "perfect" but to say dynamic secrets are wrong and crazy
just because of this one problem isn't very well thought out. Your wordpress
application will be hacked a LOT easier than your MSSQL sa account.

So dynamic secret people are saying, we all know our apps leak/get compromised
all the time, minimize their leakage with a short lived, one-time use secret,
that can only be used from X machine, etc, instead of a long-term, static
secret that can be used anywhere, etc.

For sure, for high-value targets you best be auditing everything you care
about, not just vault, but with a vault audit log, you can automate a lot of
that comparison between MSSQL and vault, where before it was much harder to
automate since the logs from every application using MSSQL will be different,
etc. But if you are in a high-value org like this, you are almost certainly
already auditing your MSSQL account creations, this doesn't change that.

------
andreygrehov
This is awesome. Somewhat reminds me of Short-Lived SSL Certificates (e.g.,
[https://medium.facilelogin.com/short-lived-certificates-
netf...](https://medium.facilelogin.com/short-lived-certificates-netflix-
fd5f3ae5bc9))

------
blr246
Vault has always appeared to me to be a great technology for a larger tech org
looking to implement more granular access control and increased auditability.

At a smaller scale, we've been satisfied with encrypting secrets using KMS and
then placing the resultant ciphertext into environment variables that we
commit with our terraform scripts. Our system does not allow granular access
control, but it is relatively simple to implement using a couple of Bash
scripts and allows committing all of the config, including secrets, all at
once so that deploys are fully reproducible.

To allow a bit more granularity in isolating different environments, we use
different encryption keys depending on whether we have a dev/uat/prod
deployment. We grant the target application role access to particular secrets
based on which KMS keys it can use.

The other trick is that we decrypt all of the secrets in our environment at
the entry point of our application code, so that the environment is fully
decrypted by the time the service runs. This means there is no trace of secret
management in our application code.

There are intermediate steps of this scheme where we could revoke access to
KMS keys for some developers, so that we could allow users to deploy services
but not necessarily be able to use and access the encrypted secrets.

Regardless, I always appreciate knowing these kinds of technologies exist
because they seem to me to solve very challenging problems at scale.

------
rakoo
So this is like authentication tokens that pretty much any developer who's had
to use a web API probably knows already ? It definitely is welcome in a more
"traditional" setup.

------
whip113
IME, dynamic secrets don't scale well. First, you're going to have a tough
challenge getting a large org to stop sharing secrets amongst apps/hosts.
Tougher still is providing a justification for why host1 for Salesforce needs
a unique credential from host2 for Salesforce that is doing the same thing.
Second, ensuring privilege creep doesn't occur becomes nightmarish when you
consider an environment today that has 500k secrets and having to audit the
authorizations an ID was granted in multiple backend authentication systems
after that ID has been deleted. Finally, generating dynamic ids for an
environment like this is a staggering amount of compute workload. Taking that
same 500k environment and breaking these out to unique credentials dynamically
generated puts you into 7 figure numbers pretty quick. Even if we ignore the
complexity of such an arrangement, just consider what the performance demands
are going to be to generate an id, assign it privileges and a password,
provide that to a client, then clean the same id up later. We'll ignore how
this looks in an environment where you have to replicate the identity that was
created across multiple authenticating backends. (think Active Directory
domain replication)

In short, I think dynamic secrets will increase the threat surface
dramatically, while simultaneously increasing complexity. Complex systems are
expensive to maintain, hard to secure and notoriously fragile. There is a
limited scope use case where dynamic secrets may be a good fit, but I for one
wouldn't base a purchasing decision on whether or not my secret store can do
dynamic secrets. Instead, I'd want a secret store that has some provision for
secrets rotation that I can tailor to the needs of my specific use cases.

~~~
armon
We work with many Fortune 2000 customers, and having 500K secrets is on the
extreme side and most certainly puts you in an infrastructure where you have
50K-100K+ machines under management.

In terms of the "compute cost", for an infrastructure of that size this is a
negligible amount of overhead. For dynamic secrets that live 30 days, rotating
500K secrets works out to 1 secret every 5 seconds.

The advantage would be avoiding an incredible number of static credentials
sprawled across a very large estate, plus having a unique audit trail that
lets you identify points of compromise. Treating those credentials as dynamic
will also reduce the human overhead of managing so many credentials, instead
focusing on roles and high level intents.

I question if there is an non-disclosed bias given the anonymous user, created
just in advance of the comment.

~~~
whip113
How is a dynamic secret that has a TTL of 30 days avoiding static credentials
sprawled across the estate? It's static (for 30 days) and the only difference
I see between rotating the password of an ID every 30 days and a dynamic
secret with a TTL of 30 days is that when I cycle the dynamic secret I have to
reassign entitlements to the new user ID and correlate this new user ID with
my monitoring systems and Vault audit data. I disagree that the compute cost
to doing all of that is a negligible amount of overhead.

Like I said, I think there is a use case here for dynamic secrets, but I have
questions about what it looks like when it comes to trying to do them at
scale. If you have solutions to the worries I outlined, I'd love to hear them.

------
pjc50
This sounds an awful lot like .. Kerberos?

------
seniorThrowaway
Here's what I see happen with Vault in actual real world usage. Applications
use the token auth method with extremely long lived tokens, and then the auth
tokens are stored in source or in config files or whatever. It just moves the
problem and makes it look better because they aren't called "passwords"

~~~
zie
This is absolutely not how _WE_ use it, but I'm sure lots of people do stupid
stuff like that.

We actually deploy vault tokens to every user(happens at login time, since
login authentication happens against vault). So we can deploy secrets directly
to end users using vault policy.

Applications that are not ran/started directly by the user find other means of
getting their secret 0(VAULT_TOKEN), usually through nomad, but in the rare
case it's a standalone application not run under nomad, we seed the startup
with a token by hand, and then have a process handle renewals, etc.

------
t-moe
Funny how hashicorp promotes security products on one hand and fails to
implement proper security in their remaining products on the other hand.

For instance: Vagrant Images/boxes hosted by hashicorp (vagrantcloud) are
neither signed nor is there any author information available (if not
explicitly provided by the uploader).

~~~
kgilpin
Also, Terraform stores "all settings, including usernames, passwords, port
numbers and literally everything else" in the tfstate file (1).

I believe that using Dynamic Secrets is HashiCorp's proposal for how to
mitigate this; leave the secrets in the log files, but make sure they expire
in a timely manner.

[1] [https://tosbourn.com/hiding-secrets-
terraform/](https://tosbourn.com/hiding-secrets-terraform/)

~~~
jugg1es
Yea, I wish they had thought of sensitive stuff in the state files from the
beginning.

------
throw4w4yv4ult
How does an application (say, a RESTful API server) deal with dynamic secrets?
I suppose it could be simply restarted when a secret changes (assuming it is
requesting the secret from Vault at startup). Is that how it is typically
handled?

~~~
armon
That is exactly right. At the bottom of the blog post we touch on this, but if
you are using consul-template to provide secrets via a configuration file, it
can either restart or reload (signal) the application to pickup the changes.
Alternatively, an application could be Vault aware and use the SDK
programmatically.

------
infamousjoeg
My concern is how you can tell me whether a database user was created by Vault
and not by a malicious actor with knowledge of Vault's Dynamic Secrets method
mimicking it.

How the heck can I know if it was a legit created database user or not?

~~~
FLUX-YOU
I assume you can check Vault's audit logs and see if the user was created
there or not.

~~~
whip113
I have 500k ids today. Now you want me to make those ids dynamic and correlate
how much Splunk data with Vault audit data? And what is my pattern matching
regex going to look like when the ids I'm trying to match on are randomly
generated? And how do I pick out anomalous behavior from the noise I just
intentionally created because Terraform can't stop leaking my secrets? And
what about the performance? How does Vault scale to generate that many
identities? And how do I audit my authorizations since on Vault I'd just see
what groups the IDs were added to but not the groups those groups belonged to?
What about replicating my authentication backends? Active Directory
replication takes minutes to replicate a password in some environments, it's
going to take longer to replicate a new identity and its group memberships.
And while I can revoke an identity after some time, that doesn't mean existing
authenticated sessions are terminated, it just means subsequent authentication
with the same secret will fail.

~~~
FLUX-YOU
Maybe this feature isn't a good fit for your organization?

But based on what you've just asked, I'd definitely never create anything like
what you're working with because it sounds like ANY addition to your
infrastructure is problematic.

------
skybrian
How does this different from an SSO (single-sign-on server)?

~~~
maccam94
SSO is for users, and credential lifetime is usually on the order of months.
Vault is for service to service authentication keys/secrets, which are
traditionally static because changing them is a complex process (you need to
make sure you don't break services by forgetting them or updating credentials
in the wrong order). Vault automates credential creation and rotation.

------
admax88q
"Why you need to buy our product."

~~~
armon
Vault is free and open source. We do have an enterprise product, but all the
dynamic secrets capabilities exist in the open source!

~~~
drablyechoes
There are also excellent free and open source frontend interfaces for vault
that are out there, such as vault-ui and goldfish. We have used both at my
org, since the HashiCorp enterprise pricing is unfortunately not within our
budget.

~~~
mahmoudimus
I think they just released it: [https://www.hashicorp.com/resources/vault-oss-
ui-introductio...](https://www.hashicorp.com/resources/vault-oss-ui-
introduction)

