

Security of Infrastructure Secrets - smurfpandey
https://journal.paul.querna.org/articles/2013/11/09/security-of-infrastructure-secrets/

======
kgilpin
Disclaimer: infrastructure secrets management is my profession.

This is a lot harder problem than people realize.

If you have a fixed set of machines that need secrets, then encrypting a bag
of secrets with each machine's private key works ok.

But in auto scaling / automated / ephemeral scenarios, it doesn't work. You
need an RBAC scheme for machines that builds layers of trust; each machine is
placed into a role by a trusted service, script or person. Communication
between the machines and the secrets service is verified TLS. Each event of
access to, or modification of, a secret is recorded for audit purposes. And
people and machines should both be treated as first-class actors.

Furthermore, secrets should be kept off permanent media; per the 12factor
guidelines, secrets should come from environment variables.

Don't entangle secrets management with other tools like configuration
management; otherwise you impede yourself from switching architectures down
the road.

Don't create workflows that only ops can control, leaving developers out in
the cold, or you are increasing organizational friction.

And if your secrets management processes are opaque to security and compliance
people, then they won't have the same level of trust that they would have in a
transparent system.

Here's an example of how we approach the problem:
[http://blog.conjur.net/chef-cookbook-uploads-with-
conjur](http://blog.conjur.net/chef-cookbook-uploads-with-conjur)

~~~
csirac2
Something that bothers me is (seemingly) widespread use of passphraseless ssh
keys or, using ssh-agent without a timeout setting (so your keys are always
loaded). I have to wonder if part of this is because ssh-agent -t starts the
timeout clock for automatically unloading keys from whenever the agent was
launched, rather than resetting the clock at each signing operation (which
would mimic the familiar sudo behaviour).

This makes using ssh-agent with a reasonable timeout incredibly painful.

So you're left with either reentering your passphrase every 5/10/15mins, or
basically never. Using smartcards for humans and TPMs for servers is a step in
the right direction, but it seems ssh-agent is still missing this basic
functionality - or am I missing something?

~~~
kgilpin
Most people we have worked with don't take ssh key passwords seriously,
because they can be stripped out. We advocated for the idea that password-
protected ssh keys are a form of 2-factor auth, but nobody bought into that.

Organizations that want 2-factor auth are typically setting up bastion / jump
hosts that require a second factor like a phone-delivered one-time password.
This can be configured through the PAM stack.

Once on the bastion, the user can get to other machines within the accessible
network using their passwordless ssh key. In effect, each bastion serves as a
mini-perimeter.

And yes, people spend a lot of time entering their second factor. Dozens of
times per day is not unusual.

Re-reading your question, I'm not really answering it. But maybe this anecdote
is useful in some way :-)

~~~
jlgaddis
_> Once on the bastion, the user can get to other machines within the
accessible network using their passwordless ssh key._

I really really hope that bastion host never gets compromised.

~~~
jessaustin
Well, sure, but that's inherent in the "perimeter" concept referenced above.
The design assumes there is an advantage to relaxing the hardening
requirements of hosts connected only to the subnet. These could include: only
having to keep ssh whitelists current on the bastion, instead of on all hosts.
Being able to completely reorganize reverse proxies without having to update
all the app servers. In general, it can be valuable for any particular host to
trust that any particular incoming connection is not related to a DDOS. If a
host connects to the public internet, that's not possible.

------
sjbase
Storing secrets is fundamentally imperfect ("it's not a secret if someone [or
something] else knows it"). This article calls for aid in the form of
standards other than PCI-DSS, and those standards do actually exist. NIST
800-53 and 800-130 to name a couple; the EU has others in different industry
flavors.

Now, I'm not going to defend these govt. standards as up-to-date or
comprehensive. But they're a good philosophical reference for how to manage
keys/secrets. Some COTS technologies (which I won't advertise here) try to
automate/enforce strong key management for infra, but are typically only
affordable for enterprise deployments.

~~~
kgilpin
If secrets are rotated or time-limited, they become a lot better. For an
example, see the AWS notion of Token Vending Machine.

It's much easier to feel comfortable handing out secrets of each of them had a
fixed lifespan. It reduces anxiety greatly.

------
proksoup
The attack vectors that surprised me but should not have:

\- MongoHQ support person has access to data in customer database.

\- CircleCI stores _everything_ in the MongoHQ database, that is used to
deploy/control customer servers.

\- CircleCI's Customers' CircleCI controlled environments mixed with
production environments.

I am guessing everyone just expects most companies, especially those with
maybe just Series A financing or close to it, expects those companies to
employ this level of security paranoia?

~~~
jlgaddis
I think we (those of us using "cloud services") put entirely too much trust in
the providers that we use.

We all just pretty much assume that they're doing the right thing(TM) with
regard to security even after we've seen, time and again, that this is
certainly not the case.

------
jffry
Needs a [2013] tag. I thought they had been compromised again.

------
cddotdotslash
The link to the MongoHQ page about their compromise is giving a cert expired
error (1 day ago), which then redirects to a page not found on compose.io.
Odd.

