
Ask HN: How does your company manage its encryption keys? - 2mol
We just had an interesting data loss at work, that was due to data being encrypted at rest. We somehow managed to delete the encryption keys (still figuring out how), which became an obvious problem once our main database instance was rebooted.<p>Luckily we were able to restore the data, but now I (we) really want to learn what a proper setup would look like.<p>If you have any clear overview reading on the topic I&#x27;d be very interested to to know about it.<p>In particular I&#x27;m wondering: how do you back up your encryption keys, or even put them in escrow somewhere? Assuming we don&#x27;t rotate the keys constantly I would love to just save them in somewhing like a passsword manager that&#x27;s secured with 2FA&#x2F;FIDO.<p>Would love to hear your thoughts!
======
thieving_magpie
Someone at my company generated the keys. They then put them on a network
share without any security restrictions. They've been there for 5 years with
no rotation. At least 2 are checked into source control.

~~~
panpanna
If this is a public repo then you are already hacked.

There are multiple automated systems that scan public repos for credentials. 5
minutes later you are mining bitcoins for them.

~~~
ganstyles
In a boneheaded movei I accidentally committed my SendGrid creds to GitHub.
Pretty quickly after, GitHub alerted me. However by then my SG account was
sending thousands of automated spam messages. Those automated scammer systems
are FAST.

Not particularly germane to the discussion, but really disappointed in how
SendGrid handled things. I notified them immediately, rotated all API tokens,
and tey could not turn it off, so the spammer sent messages for days and
eventually my SG account got suspended.

~~~
bkandel
So this has happened at more than one company that I've been at, only
difference is that these were AWS keys and used to mine bitcoin. AWS was
actually pretty good about it, we rotated the keys as quickly as we could and
they dropped all the charges.

~~~
meddlepal
I once worked for a co-founder who despite all the warnings I gave about not
committing infra credentials to source control still went ahead and explicitly
committed credentials to public source control because "the developer
experience was better".

The CEO was not pleased with the 30K (or maybe it was 60K) bill... and I just
pointed at the CTO and was like "I fought this battle and was overruled"

~~~
mb7733
The most frustrating part of this is that the developer experience _isn 't_
better when you check credentials into source control.

It seems convenient until the credentials change (which they ought to now and
then). Then when you check out an old revision of the project, it is broken.
You end up having to copy and paste the new creds back in time and it's
finicky as hell.

------
toomuchtodo
[https://www.vaultproject.io/](https://www.vaultproject.io/)

We use Hashicorp's Vault product to manage SSH credentials, TLS certificates,
as well as application secrets across thousands of users, tens of thousands of
virtual machines, and hundreds of applications.

We pay for the enterprise version, but the free version is more than capable
for most needs. Avoid a password manager if you can, it leads to poor security
practices and availability issues depending on how and where the data is
stored (IMHO, YMMV). Use single sign on where ever possible.

Disclaimer: No affiliation other than a satisfied customer and frequent end
user of the product.

~~~
varikin
This just pushed the problem further down the stack. You should have keys to
unlock vault when it is restarted. How do you secure those keys?

~~~
mmm_grayons
I can't seem to understand how a "secrets manager" helps things. Could someone
who does ELI5 why it's better than a config file with permissions locked down?

~~~
bvandewalle
It makes sense at scale. If you are a company of two there are probably better
solutions.

At scale, you can very granularly define policies for each secret. When a
secret is accessed, it is done so through a user or application identity. Each
access is also logged.

~~~
scott00
So then how do you manage the secret that authenticates an application's
identity? And what good is the logging if after an application has the secret
it can do whatever it wants with it?

~~~
bvandewalle
if it is an instance on the cloud, GCP and AWS let you define ServiceAccounts
that get populated on the Instance at boot time.

you should only let the instance access the secret it requires.

~~~
rdslw
and how do you manage secrets that let you define that ServiceAccounts?

As OP wrote, you did not solve it, just moved it to a different level.

------
jwr
Here's what works for small and medium organizations for data which needs to
be encrypted at rest, but is not often accessed (so, backups):

1\. Buy a bunch of Yubikeys, minimum of 2.

2\. Create GPG keys and store them on YubiKeys. Follow this guide:
[https://github.com/drduh/YubiKey-Guide](https://github.com/drduh/YubiKey-
Guide) (if you want to, keep the secret keys, but in case of multiple YubiKeys
I would not keep them anywhere). Remember to set the keys to always require
touch.

3\. Use GPG to encrypt your backups to multiple recipients (all of the
YubiKeys).

4\. Take care of the physical keys with proper storage and procedures. Do not
store the keys together, have at least one in a _really_ secure location,
check if you have all the keys regularly, etc.

5\. Test restores at least once per quarter, with a randomly selected key.

The advantages of this solution is that it is simple, works pretty well, and
gets you a lot of mileage with relatively little inconvenience. You don't have
the risk of keys being copied, and guarding physical keys is easier than
digital ones.

You still have the problem of guarding the passphrases to the Yubikeys (if you
use them), but that is much less of a problem than guarding the encryption
keys. A passphrase without the physical key is useless.

This setup works for organization from size 1 up to fairly large ones.

Note that some recently fashionable security consultants crap on GPG from
great height, but do not provide an alternative. It's a tool that while having
multiple flaws, does many jobs better than anything else out there.

~~~
rrdharan
This is all generally good advice, but I think there's huge potential
complexity lurking here:

> 4\. Take care of the physical keys with proper storage and procedures. Do
> not store the keys together, have at least one in a really secure location,
> check if you have all the keys regularly, etc.

Would be great to see what folks think this concretely looks like for joe
random startup in Capital City, Somewhere.

e.g. Does "really secure" mean "find a bank that still offers safety deposit
boxes"? Does it mean paying for something like Iron Mountain
([http://ironmountain.com/](http://ironmountain.com/)) or one of its
competitors?

~~~
techsupporter
> Does "really secure" mean "find a bank that still offers safety deposit
> boxes"?

Realistically? Yes. This is what several of the companies I've done contract
work for have done. You can still find at least one bank or self-storage place
(look for the ones that don't have a nationally-advertised brand and don't
look like they're made entirely out of corrugated metal) that do regular
safety deposit boxes in pretty much any city. They may only be offered at a
couple of locations and I've noticed credit unions bailing the hell out of
this market as fast as they can decommission the vaults but boxes still exist.

Let's assume all variables work in the other way, though. If you can't find a
safety deposit box and don't have somewhere that's not your office you can
drill into a floor or wall and you're storing a small device like Yubikeys or
USB sticks, buy the heftiest portable gun safe you can find, one with a steel
cable that loops back into the device, and stick it under your bathroom sink
with the cable wrapped firmly around the water supply or drain pipe.

~~~
kelnos
One problem with a safe deposit box is what happens if things go to hell at
2am? The bank isn't going to open for another 6 or 7 hours, and meanwhile
you're sitting there with possibly business-destroying downtime.

~~~
tomschlick
Well the idea is that the safety deposit box key is if everything falls apart,
all the employees go missing and you dont have any other options. Ideally
multiple employees would have keys as well that could respond in those crucial
hours.

But if for instance, your whole security team got in a car accident and all
the keys burned up, you'd have a way to recover the creds and save the
business.

------
parasec
The more complex your system grows, the more often it will fail and shoot you
in the foot. I'd advise against systems like Hashicorp Vault - they just
increase the complexity and while they have their merits in complex setups,
you seem to be too small to be able to operate such a system.

Have an offline backup printed along with the disaster recovery checklist and
documentation and put them in a safe in your company - the checklist should be
dumb enough that your drunk self can use it at four in the morning, because
you were the nearest employee when everything went down.

Ensure that you have stupid manual processes in place on rotation of the
safe's PIN and encryption keys in general, including a sanity check if the
newly generated keys actually work (e.g. if they are used for your backup
storage, actually back something up _and_ restore it). Ensure that the safe's
PIN is available to at least another person and used regularly (e.g. because
you store your backup tapes there).

If you feel that you need to change from this very simple system to a more
complex one, ask yourself why. What does your change actually add in terms of
security and what risks does it add.

In the end, you want your system available to customers and the security you
add is to not only secure the data, but actually to know _who_ can access it
(the auditing part).

~~~
2mol
Great answer, thank you. I agree with your point about testing the restore
process, right now I'm trying to think of a way to automate it.

As a side note: for example we had some backups that are probably useless,
because they are way too small. Catching this would mean more manual regular
checks, or some automated rules, at which point it becomes quickly more
complex again.

~~~
parasec
The backup part is quite easy - generate a well-known file with 1kb of data
and include it in your backups. After the backup completed, validate that you
can restore the well-known file and compare the content of the original and
the restored. Easy to automate if you run a well-customizable backup system.

Storytime: I did _not_ check the restoration of my backups some time ago and
had a faulty harddrive, so I needed to restore the backup. Backup was also as
dumb as possible - essentially a tar and encrypting it with openssl. So I
reinstalled the server, tried to decrypt it - got the error, that the key was
wrong. Took me a good weekend to find out, that openssl changed the default
hash algorithm between openssl 1.0 and 1.1. This would not have been catched
with the proposed system, but now I really pin all default options in my
scripts as well.

------
openfinch
Can't speak to my current employer as it's above my pay-grade to know, but at
Job-1 we did the following:

\- All "hot" keys were stored in an offline credential manager in specific
vaults depending on who needed access to them. Only staff with actual
clearance could request temporary access to a vault (fully background checked,
1 year employment, etc).

\- Copies of each vaulth and our master CA cert were written to 4 encrypted
USB sticks. Two stored on-site in the fire-safe and two off-site at our safety
deposit box that only c-level staff could access. (We had the same process
with our tokens and master logins for AWS).

\- Any work using those keys was on a pair-up basis, so at least two people,
one doing the work and the other observing.

\- We had a detailed policy around this that covered each step in the process
and who needs to approve them; everyone who could feasibly need to access the
keys was briefed annually as part of our security awareness training.

We handled a LOT of sensitive financial data, so this was the most appropriate
way that we could find that maintained both sensible availability and key
control.

So in order to get to the keys you needed:

\- Access to the fire safe (Senior Ops, Senior Security and C-Level only).

\- The LUKS passphrase for the USB sticks (Senior Security and some C-Level
only).

\- The passphrase for the specific vault (Senior Security and some C-Level
only).

I don't know how the passphrases were managed by our sec team, but I know that
the C-Level staff had physical envelopes in their home safes.

~~~
brutus1213
An expanded version of this would make a valuable book (or blog post at
least).

------
AgalmicVentures
YubiHSM 2 has worked fantastically well for us as a root of trust in a variety
of applications, at a very reasonable price ($650) (I am unaffiliated with
Yubico other than as a very satisfied customer).

Accessible over USB or HTTP, it supports every major crypto algorithm [1], and
keys can be backed up onto another HSM via a wrap key (if they are marked as
exportable -- you can also control what can and cannot be exported -- in fact,
every operation may be allowed or disallowed per key).

Every operation is logged for audit, of course, and the device may be setup to
require logs to be read before they are overwritten. In combination with
configuring a special authentication key to access the logs, you can ensure
that every operation on the HSM is logged to a remote store before additional
operations may be completed.

It does depend on your existing physical security, so that has to be taken
into account when designing architectures including it. The micro form factor
at least makes it trivial to put into an internal USB port.

And of course, if you require a more enterprise grade tool, you may want to
use an HSM in combination with a tool like Hashicorp Vault to manage your keys
throughout your orgnaization.

[1]
[https://developers.yubico.com/YubiHSM2/Product_Overview/](https://developers.yubico.com/YubiHSM2/Product_Overview/)

------
up_and_up
At my prev company we generated keys and split them using `ssss-split` and
handed out shards to specific individuals via a keybase exploding message. Our
system required at least 3 shards (combined via `ssss-combine`) to reboot.

FYI: Hashicorp vault just uses Shamir's Secret Sharing scheme under the hood:
[https://github.com/hashicorp/vault/blob/45b9f7c1a53506dc9722...](https://github.com/hashicorp/vault/blob/45b9f7c1a53506dc97221f0915daeaeb0a6fe894/website/pages/guides/operations/rekeying-
and-rotating.mdx#L20)

------
hn_throwaway_99
I highly second the people saying KMS (AWS KMS, Google KMS, or KeyVault).

* The pricing for just storing keys is incredibly cheap.

* At least with Google KMS you can't delete the keys without a 24 hour waiting period (and you can alert on the deletion attempt), so that's a huge safeguard.

* You get key access auditing out of the box.

~~~
NovemberWhiskey
This is fine if you're committed to using (say) AWS KMS for your encryption
needs as a service with its per-API-call pricing.

The costs of that obviously scale in a completely different way from the per-
key storage costs (which are actually zero, I think).

~~~
kevindong
Google's KMS pricing structure is $0.06/key/month + $0.03/10,000
encrypts/decrypts [0].

At $0.06/key/month, that's practically free for most reasonable use cases. For
example, if there's 10k secrets that's $7,200/year.

If you encrypt/decrypt your secrets 1 million times per day (~11.6 times/s),
the access charges would be $1,095/year (1 million operations/day * 365
days/year * $0.03 / 10,000 operations).

[0]: [https://cloud.google.com/kms](https://cloud.google.com/kms)

------
gingerlime
We use Bitwarden[0] for our secrets. It's open-source with a hosted option.
Makes sharing passwords and keys across the team pretty straightforward.

In addition to it, we use envwarden[1], which is a simple open-source wrapper
around the Bitwarden CLI to manage our server secrets. It's super simple, but
does the job for us well. We can then manage both passwords and keys in one
place.

Disclaimer: I created envwarden. I'm not affiliated with Bitwarden in any way
however. Just a happy customer.

[0] [https://bitwarden.com/](https://bitwarden.com/)

[1]
[https://github.com/envwarden/envwarden](https://github.com/envwarden/envwarden)

~~~
shekhardesigner
I didn't know about envwarden till you mentioned it here.

We migrated from bitwarden to offline client machine for the reason that we
didn't wanted a copy of our encryption key be available anywhere else online.

------
ComputerGuru
We commit encryption keys, themselves encrypted, to git alongside the code and
everything else. They’re fully versioned and therefore protected against data
loss, and we don’t treat dev keys as any different from production keys (just
stored in a separate file).

I first wrote about it back in 2017 (1) and we released an open framework for
multiple languages/frameworks (2).

1: [https://neosmart.net/blog/2017/securestore-a-net-secrets-
man...](https://neosmart.net/blog/2017/securestore-a-net-secrets-manager/)

2: [https://neosmart.net/blog/2020/securestore-open-secrets-
form...](https://neosmart.net/blog/2020/securestore-open-secrets-format/)

------
hellcow
We use shh for secrets ([https://egt.run/shh](https://egt.run/shh)).

It's designed to integrate really well with your existing CLI tools like vim,
xargs, and diff. It offers user-based permissions, and secrets are encrypted
into a single file that's safe to commit into your git repo. We can stream
secrets out of it directly to our remote servers during deploys.

Unlike Vault you don't need to manage infra to run it -- it's just a file.
Unlike cloud secret managers, there's no lock-in.

~~~
ahnick
A couple of questions..

1\. Does every client have a copy of shh to interact with the secrets? Or are
the secrets in the file served from a single centralized node?

2\. What is your process of exchanging user keys with shh?

3\. If someone leaves the company, what is the process you go through to
change the secrets and rotate keys?

~~~
hellcow
1\. Every client (i.e. developer) has their own copy of shh but interacts with
a shared .shh file in the project root. Secrets can be decrypted locally in
memory then streamed to the remote servers over ssh. You could in theory run
this on a server to help address your third point.

2\. Exchanging public user keys is done via email when an employee starts,
then added to the .shh file and committed in the repo.

3\. You'd need to write a script for this, which probably involves `shh rm-
user {email}`. There is no silver bullet here for changing secrets; since
developers have access to secrets, any secret they had would need to be
regenerated. `shh` makes no assumptions about how you deploy, what secrets you
keep, or their formats.

------
z3ugma
We use Bitwarden
[https://bitwarden.com/#organizations](https://bitwarden.com/#organizations)
to store passwords as well as encryption secrets as a backup.

Application repos have the encrypted secrets (meta) stored in their repos
using Ansible-vault and the .vault_keys are stored in Bitwarden.

~~~
pqdbr
We do exactly the same.

------
okso
We store all keys that do not required automated access on Yubikey with the
option that requires a physical touch per use.

Usage includes SSH authentication, file encryption (backups and exchanges),
git commit signatures and password/secret storage using `pass`.

Copies of the offline master keys keys are stored on flash in safes onsite and
offsite in bank vaults, and sub-keys are valid for one year.

We use Hashicorp's Vault for secrets that require automated access.

------
d0ne
Disclosure: Founder

[https://dev.ionic.com](https://dev.ionic.com)

Utilized globally by individual developers, large enterprises such as JP
Morgan & Chase[1], and integrated into the KMS services such as Google
Cloud[2].

1\. [https://venturebeat.com/2019/02/27/ionic-security-
raises-40-...](https://venturebeat.com/2019/02/27/ionic-security-
raises-40-million-to-encrypt-files-accessed-on-the-cloud/)

2\. [https://cloud.google.com/blog/products/identity-
security/clo...](https://cloud.google.com/blog/products/identity-
security/cloud-external-key-manager-now-in-beta)

~~~
Predrag3141
dev.ionic.com is in particular an answer to, "how do you back up your
encryption keys, or even put them in escrow somewhere?"

~~~
mfloyd5218
In particular, this solution implements a key vault that you can safely store
keys using the security model provided by your OS (Windows, MacOS or Linux).
The tool provides a CLI that lets you easily create and store keys from the
command line.

------
mbreese
At one job I was afraid of losing a set of keys, so as an extra backup, we had
a physical copy of the the key printed out as a QR code. This was then
physically locked away (safe deposit or similar). But I like this as an option
for a “failure resistant” offline backup. The main benefit is that doesn’t
assume a stored away USB stick will still be readable while still having a
somewhat machine readable format. Yes, the QR code was very large, but could
be split into multiple files if needed.

------
qubex
I worked in a place that had a very nice in-house blade server setup with an
attached NAS, which was encrypted (at rest, on storage).

This was early in the “cloud” epoch and at any rate they preferred in-house
iron for entirely understandable reasons. Also, they needed to do some pretty
interesting things, so they had a NAS populated with enterprise-grade SSDs and
a native AES-based encryption scheme. This kept the key on what was basically
a glorified USB key.

(For those of you who wish to know these details, the network between the
blades and between the blades and the NAS was a nicely spec’d fibre channel
network, and there was some iSCSI involved. The blades featured Itanium
processors, which kind of gives the manufacturer away, and the firm had
invested quite heavily in producing very high-performance code for those ill-
fated microprocessors, but I digress.)

So... it happened that somebody lost the USB key. Well, not quite. Somebody
took it home during a weekend (whilst the system was shut down) and their kid
used it for school work.

This proved to be a “significant problem”. There was a backup, and it was
encrypted and stored on an adjacent SAN. It wasn’t exactly stale, but it
wasn’t entirely pristine either.

There was much woe and gnashing of teeth.

Nobody was fired because the dolt who maintained custody of the only USB key
was the founder/CEO, so he couldn’t exactly blame himself.

But, yeah. That happened, sadly.

------
z3t4
You need to have a threat model. Then work out a solution from that. Ask
yourself, why are you encrypting the hard drivers ?

One threat model might be: Burglars sneaking in during the night and stealing
the hard drivers. Then you would store the keys on a different location then
the disks.

Then you make it a routine to reboot parts of the fleet, like scheduled
simulation/training so that everyone knows what to do when you actually need
them.

------
munchbunny
How paranoid do you want your security to be?

In general I would suggest using a key vault. AWS, GCP, and Azure all have
cloud versions that are backed by virtual HSM's built on top of actual
physical HSM's. For the vast majority of usages they're good enough. Use admin
account management to enforce 2FA/FIDO for all AWS/Azure/GCP logins. (You
should be enforcing 2FA with phone/FIDO auth anyway.)

If you need truly paranoid backups, you can back the key up onto a portable
hard drive that you lock in a safe in the closet, with a few key people who
know the code.

I recommend against using a (cloud-synced) password manager. Cloud key vaults
do the same thing but offer specific features relevant to server stuff. And if
you want more paranoia, a physical safe is probably safer than extending your
attack surface to a cloud-synced password manager.

Also: make sure that you set up a ~quarterly ritual of opening and verifying
the backup. For crucial backup fallback systems, you want to make sure you
actually use the system so that you know if it fails.

------
ReptileMan
> How does your company manage its encryption keys?

Poorly seems to be the answer industry wide. Both encryption and disaster
recovery are hard tasks. Combine them and you have recipe for a total mess.

------
ecesena
We use Knox:
[http://github.com/pinterest/knox](http://github.com/pinterest/knox)

Think of a lighter vault, with ACLs for people and/or machines to access keys
and versioning to rotate keys.

In our case, Knox depends upon AWS KMS to "lock/unlock" its storage.

~~~
lux
This looks way easier than getting Vault setup! Unfortunately, googling for it
returns a ton of info about Apache Knox which is an unfortunate name clash. I
would love to see this catch on. Thanks for the share!

------
dijit
Not sure the "right" way to do it. But this is what we did:

For context: We run a centralised salt-master, salt master unencrypts content
using gpg filters as part of variable generation (salt "pillars"). So it's
encrypted at rest and encrypted in our git repositories.

What we do/did, is:

* grab a pair of differently branded USB sticks.

* LUKS encrypt the USB sticks; we used a keyfile which is encrypted on our machines with our GPG key.

* encrypt salt's GPG private key with all of our keys.

* encrypt some of the "irreplacable" private keys (IE: CA roots) with all of our GPG keys.

* store it all on the pair of USB drives

* put the USB drives in a real-life vault, give the keys to the office team.

We haven't needed to recover, but it's clearly documented how to recover if
anything went wrong.

~~~
openfuture
Bit rot is a thing.

It'd be better to take a page out of the cryptocurrency handbook and print the
keys out as QR codes (maybe with a simple password so it can only be restored
if you know the pw).

------
bashinator
AWS KMS, and equivalent for other cloud providers. They are in every sense
better qualified to handle encryption secrets than my lone ass is.

------
Edmond
Lots of decent advise here already...be wary of a complicated (fancy product)
approach and security theater....frankly when it comes to secret management
the KISS principle should apply, with the caveat that it should be secure of
course.

------
dudus
[https://aws.amazon.com/kms/](https://aws.amazon.com/kms/)

------
ransom1538
My humble security advice:

1) If you don't need it - don't store it. If you need it - but it needs to be
encrypted - probably! don't store it.

2) Think one way hashes with salts, think deletion policies, rotations, small
disk drives.

3) You will get owned!!

~~~
dfc
I agree with only store what you need. But I can't think of many things in a
professional setting that a company should store but not encrypt.

------
XorNot
The dream for me has always been to get LDAP/Kerberos tied into container/VM
orchestration, and do everything that way with per-instance accounts.

LDAP is ubiquitous enough as an auth method (how do you auth to Vault? You
auth to LDAP with it...) that any service you run or use is likely to speak
it.

Why this isn't done more often is a mystery to me and probably the number 1
source of credentials being baked into things accidentally: oh we need a
service account into the <enterprise system> which uses Active Directory.

Might have answered my own question there though.

------
uxp100
I think it really matters how sensitive these keys are.

It's been quite a few years since I interacted with them, but for some keys
there is a server somewhere with an HSM installed, and two people have
credentials for it. If you need something signed you send it to them, with a
justification for why it needs to be signed with the real keys, and they will
send you a path to get the signed file, and remind you to delete the signed
file when you no longer need it.

This is overkill for some things, and probably would be considered sloppy for
others.

------
caipre
Perhaps a dumb question, but how did you restore the data without the keys?
Was a prior backup unencrypted?

------
kissgyorgy
Put it in a password manager. Nothing can be any simpler than that.

~~~
KMnO4
I like this approach because 1Password provides a nice CLI that I can use in
scripts:

    
    
      op get document UNIQUE_DOCUMENT_ID > pem
    

Anyone in my department (or who can access the shared vault) can run this
script by simply logging into 1Password cli (`op login`) before running.

------
weitzj
Online HSMs where you can exchange keys. Databases connect to the HSMs for
crypto operations + offline HSMs with PCI cards in a safe for root keys.

For other parts HashiCorp Vault/AWS kms

------
awayfromme
Hi, this issue brought me back in the days when i was just a IT handyman in a
small company. The priority of that company was to don't share keys or
everything related to them with no one. For no one i mean, third-parties
software of wherever a password or an encryption would be watched to someone.
At the time, i thought this "obsession" was clearly a sign of mental illness,
cause the company was very small and we were in the nowhere of nothing.(maybe
nowdays i still think it). Our method was based on a selfmade MD5 encryption
script using ruby on rails. Put your password into it, it print it on a
datacoin blockchain that generates a univocal MD5 hash. This hash goes around
5 (later 6) local server those collect the encrypted key. Obviously these
servers were without an internet connection, running only for internal company
purposes.(such as this). For sure a weakness of this procedure was the
slowness for obtain a new password or to change it. I think that the most
secure place is where there'snt an internet connection. Thanks for bring back
memories :D

------
bloopernova
When managing/deploying code we use SOPS (Secrets OPerationS)
[https://github.com/mozilla/sops](https://github.com/mozilla/sops)

For standard password style secrets used by Ops, we use Team Password Manager.
Which we chose about 5 years ago because it was self-hosted, the database was
encrypted, and it had fantastic audit capabilities.

------
lormayna
In my previous experience, I was working for an HSM and Data Protection
vendor. If you want to be resonable secure to don't lose keys and keep them
safe, just use an HSM. If you need to encrypt filesystem, you can use a DP
product (most of them are not so expensive) If you want to database content,
you can use tokenization services.

------
Spooky23
I try to push as much to certificate auth as possible. Internal CA keys are
stored on an offline USB HSM, which is locked in a cabinet. Access to the key
requires 2/10 individuals to be physically present.

There are different CAs for different purposes. There's an intermediate for
device management, and another for user or service auth purposes.

~~~
ahnick
So are your secrets served from a central node that authenticates the
certificates? What does your process look like for changing secrets when
someone leaves the company?

------
bvallelunga
A couple of our customers shared this thread with me so I thought I’d chime
in. For transparency I am the CEO of Doppler (YC W19), a hosted secrets
manager service. I know secrets management isn’t directly related to key
management, but it’s a cool security topic we think about often. A one-liner
about Doppler - lovable secrets manager built for the everyday developer, that
works across all stages, from local development to production, on all stacks,
and infras.

For anyone who needs a super simple place to store their encryption keys that
works with Heroku and has versioning, I think Doppler could help. It doesn’t
have all the fancy (and really cool) features of KMS as it’s designed to be a
kv store for secrets, but it could be helpful. We have a free tier for anyone
who wants to try it out. [https://doppler.com](https://doppler.com)

------
matheusmoreira
> I (we) really want to learn what a proper setup would look like

Signing and authentication keys are expendable but encryption keys are worth
keeping even after they've been rotated since decryption of existing data may
be necessary.

The key can be printed on paper and stored in a physical safe. Paper isn't a
high density storage medium but it is remarkably durable and perfect for small
amounts of data such as encryption keys. It also counts as an offline backup.

Keys can also be printed as QR codes. They support error correction and enable
automatic data restoration. Even 4096 bit RSA keys fit in a binary mode QR
code and the smaller ECC keys allow use of high error correction modes, making
the data even more durable.

I wrote a binary decoding feature for ZBar in order to support this exact use
case:

    
    
      zbarcam --raw --oneshot -Sbinary > key
    

It's available on version 0.23.1 and above.

------
lsllc
Anyone using any KMIP based solutions? (looks like Hashicorp's Vault product
implements a KMIP server).

[http://docs.oasis-open.org/kmip/spec/v1.4/kmip-
spec-v1.4.htm...](http://docs.oasis-open.org/kmip/spec/v1.4/kmip-
spec-v1.4.html)

------
shekhardesigner
We keep the encryption key stored in plain text files on client offline
machine.

Catch: Client machine is encypted with VeraCrypt. Veracript hidden drives, one
password is kept by product owner and another password is kept by head of
security.

Offline client machine is key for us.

We rotate encryption keys after quarterly security audit.

------
Faaak
We use vault, but sometimes I just `openssl` gpg encrypt the secrets with the
keys of all the members of my team and commit the .gpg to git. We all use
yubikeys and use them to SSH.

Not ideal, but it works... At least until one of us resign (but turnover is
quite low here, so crossed fingers).

~~~
ahnick
What is your process for handling key exchange with team members?

~~~
Faaak
when a new guy arrives to the company, we generate the keys on an air-gapped
computer (with cahoskey et al) and upload the to the yubikey (they keep a
separate encrypted usb key with the private keys). Then, some employees verify
the new employee and sign their keys. There are then uploaded to teh
keyservers and an internal mail is sent.

Quite old school but it works quite well, alas we're small though (120).

------
zmmmmm
We check many of our passwords into a git repository ... but they are all
encrypted with unix pass with the public key of each individual in the team
authorised to access. Our services require the gpg agent running to launch,
allowing them to read the secrets.

------
thesimon
Vault for big stuff, git-crypt for file-level encryption of secrets (in
Terraform files etc.)

~~~
dijit
What happens if vault gets destroyed or corrupted?

~~~
NovemberWhiskey
This is why you have geographically separated disaster recovery replicas and
backups.

------
clintfred
[https://github.com/IronCoreLabs/ironhide](https://github.com/IronCoreLabs/ironhide)
is a tool we built for managing our own developer secrets. It allows encrypted
files to be checked in to git or stored elsewhere. You can think of it similar
to gpg, with the upside that ironhide has the ability to change who can
decrypt the secret _without re-encrypting the data._

Check it out and if you have any questions, feel free to ask here or open an
issue in github. We also have a Rust version in the works for those interested
in something native.

------
Shicholas
GitHub Actions, Terraform Cloud, and strict RBAC for everyone on our team so
they can't see/change these values on GitHub/Terraform/our cloud environments.
This doesn't need to be hard.

------
tlarkworthy
The new Google secret manager is a pretty nice way of having an easy to use
web interface protected by IAM, with a REST API for applications to pull. You
could easily prevent users from deleting keys. It's not as hardcore as Vault
but it's a much simpler way of getting keys out of source control IMHO. You
can have 2FA and audit logs easily too. Simpler than Google KMS too.

[https://cloud.google.com/secret-manager](https://cloud.google.com/secret-
manager)

------
nitwit005
I've had these variations at work: Checked into the code encrypted, a custom
secret vault, stored encrypted in a S3 bucket and downloaded/decrypted at
startup.

The main observation I'd make that if you put your keys somewhere only
accessible in production, you've made it impossible to test anywhere except
production. If you do that, you need to create a process where people can ship
some small bit of code to test if the production key setup genuinely works
(hint: it won't).

------
cj
AWS KMS for most thing.

AWS ACM for SSL certs.

AWS SSM for ssh, eliminates the needs for ssh keys.

Not everyone loves AWS (me included), but this stack works nicely in removing
the need to ever touch raw encryption key files locally.

------
gramakri
I am curious how you managed to restore it if you lost the keys.

~~~
2mol
"accidental" backups

~~~
gramakri
ha ha, cheers mate. I asked because I had encrypted backups in the past for
which I lost the key and wasn't so lucky :(

I don't know your company size but we use stackoverflow's blackbox. It's very
nice because you can check in the repo into version control. You can add and
remove users on the fly as well.

------
pianoraptor
Piecing together a key management solution from open source is possible, but
if you're trying to encrypt data and are open to commercial products, we do a
lot of support of Vormetric.

All of the key management is "built-in" and managed, so there really isn't
much overhead. All software-based, with FIPS certified key management. It's
very easy to encrypt data this way. It is expensive though.

Disclaimer: while I used to work for Vormetric / Thales, I no longer do.

------
greyhair
Hardware key control (HSM) with smart card controlled access.

Locked in a vault.

Keys are an actual secret key (within the HSM), unknown to any human.

Two people with two different access cards have to be present to enable key
operations.

So, you don't have to worry about changing keys as employees come and go,
because no one knows the actual keys.

There is a whole structure of spare cards stored in offsite secure storage in
case a card is damaged, lost, or stolen.

Card set one is stored separately from card set two.

------
gerritjvv
the simplest option is to store your keys in aws secrets manager (if you use
aws), and then write some tooling around it.

 __ __self promotion __* You did ask how people do it :), this is my way, Ive
written my own service which has been in production for more than 3
years,[http://pkhub.io](http://pkhub.io) (if you would like to try it send me
an email to admin@pkhub.io). This was before aws secrets manager, the tooling
is usefull cause I wrote: running your app with its needed secrests
dev/stage/prod, accessing dbs, downloading and installing ssh keys to ssh
agent, utilities.. ________end __ __ __ __

of course you could write all these yourself with aws secrets manager.

there is hashicorp's vault but tbh it always seemed like way to complicated to
setup.

my advice in general would be: to get something secure but simple enough that
your engineers can do their work and access the resources they need, without
the oh only bob has the keys on his laptop situation.

------
eqvinox
We have a main config/setup git repository which contains keys (and passwords)
in git-crypt. 4 people have GPG keys able to decrypt it, and this is used in
normal operation for deployment (i.e. for ansible to feed the keys into
systems.) The repo is cloned in quite a few places.

None of our secret material is particularly hard to revoke or replace; we
don't run an internal CA or anything like that.

------
nickbauman
I have used EJSON and putting everyone's public key in a single keyring. Works
but the secrets and keys end up taking up a lot of space.

------
sly010
If you run on kubernetes it has first class support for secrets [0]. You can
reference secrets in environment variables, or mount them as files in your
containers.

[0]
[https://kubernetes.io/docs/concepts/configuration/secret/](https://kubernetes.io/docs/concepts/configuration/secret/)

~~~
pricechild
Check out
[https://kubernetes.io/docs/concepts/configuration/secret/#ri...](https://kubernetes.io/docs/concepts/configuration/secret/#risks)
\- this isn't free.

------
tzs
Shamir's Secret Sharing algorithm [1]. 4 people have shares, with 2 shares
required to reconstruct the key.

How the 4 shareholders store their shares is up to them. Mine is in a secure
note in 1Password.

[1]
[https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing](https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing)

------
N_A_T_E
We have so many secret values it made sense to build our own internal product,
an audited system which holds secrets in a backed up, locked down database.
Apps pull from that system at runtime (or deploy time) but they can only
access their own secrets using access control. We also use AWS KMS for AWS
related resources.

~~~
ahnick
Before building your own, did your company consider using Vault? If so, what
factors led you to go down the build your own path?

------
sudhirj
Some on a USB hardware dongle under lock and key. We also run HSMs and the key
cards to those require quorum. So the CTO, Founder and head of devops each
have multiple cards at home and work such that any two of them can make a
quorum with all their cards, or all three can make a quorum with some of their
cards.

------
r1ch
I put them into a password manager which is backed up both on Dropbox as part
of sync and Backblaze for long term.

------
speedgoose
We save the keys in a cloud vault. For the most important keys I also print
them on paper, in text and in a QR-Code (that I generate with an offline
tool). It is then placed into a physical safe. This is in case we lose access
to our cloud vault, or if the keys are deleted from the cloud vault.

------
Ididntdothis
Very badly and inconsistent in my company ! We have a lot of people raising
concerns about whatever you want to do. But not many actually have any
contributions for how to do it right so every team rolls their own little
solution.

I will be very interested in hearing how others do it.

------
rogerkirkness
We use Doppler. They're a YC Co. Like an easier to use Hashicorp. Has been
great so far.

~~~
bvallelunga
[https://doppler.com](https://doppler.com)

------
rraval
We use StackExchange's blackbox to check in GPG encrypted secrets into our
monorepo:
[https://github.com/StackExchange/blackbox](https://github.com/StackExchange/blackbox)

------
moxylush
I've asked myself this question many times. Or "Who holds the keys to the
keys?".

Here is an air-gapped solution:

[https://www.arcanus55.com/?trusted55=A55PV2](https://www.arcanus55.com/?trusted55=A55PV2)

------
drcongo
Depending on what the keys are for, we use either 1Password or EnvKey for
storing them.

------
certera
Plugging my project Certera as a means to manage keys used for Let's Encrypt
certificates: [https://docs.certera.io](https://docs.certera.io)

You can rotate keys and facilitate key pinning scenarios.

Cheers!

------
BillinghamJ
They are kept on Keybase >_<

Sadly now forced to figure out what to move to. We're considering 1Password
with it's CLI as a short term option, but will be wanting to move to Hashicorp
Vault or similar on the mid-term

------
mavdi
AWS secrets manager or SSM or KMS for any kind of secrets, keys etc. Works
well because our entire stack is on AWS. Otherwise hashicorps Vault would do I
guess but that’s yet another service on life support.

------
person_of_color
Related question: How do big companies (FAANG) store their root private key?

------
uditj
Used Doppler to store encryption keys and other secrets, and it worked well.
Was pretty easy to setup and use their secrets store across our dev machines,
CI, and production environments.

------
haidrali
We use keybase

~~~
ahnick
We also use Keybase. We wrote an extension to our existing lightweight
management script,
encpass.sh([https://github.com/plyint/encpass.sh](https://github.com/plyint/encpass.sh)),
to be able to access/manage our secrets from the CLI and in our infrastructure
scripts. The extension is encpass-keybase.sh
([https://github.com/plyint/encpass.sh/blob/master/extensions/...](https://github.com/plyint/encpass.sh/blob/master/extensions/keybase/KEYBASE.md))
kept in the same repo.

The extension makes use of the per-user keys of Keybase. This allows you to
manage access permissions using the Keybase GUI client. Once a user is added
to the appropriate Keybase team, they will immediately have access to that
teams secrets that are stored in the encrypted Git repos.

To use it you just need to download the 2 shell scripts to a directory in your
path (e.g. /usr/local/bin/) or you can clone the git repo.

------
virgilp
Nice try, NSA.

------
Dansvidania
My company is too big for me to know about how everyone manages them (my guess
would be sharepoint shared folders, with variable degrees of accesses).

My team uses lastpass

------
MangoCoffee
We use Azure KeyVault

~~~
maxbaines
+1

Worth noting accessible using api via REST [https://docs.microsoft.com/en-
us/rest/api/keyvault/](https://docs.microsoft.com/en-us/rest/api/keyvault/)

and powershell [https://docs.microsoft.com/en-us/azure/key-
vault/secrets/qui...](https://docs.microsoft.com/en-us/azure/key-
vault/secrets/quick-create-powershell)

~~~
idatum
+1 for Az KeyVault. I use it in my Docker deployment scripts using Azure CLI.
Example here is a secret, but similar concept for certs using CLI:
STORAGE_ACCOUNT_KEY=`az keyvault secret show --vault-name=<your keyvault>
\--name=<secret name> \--query=value | tr -d "\""` This populates a .env file,
referenced in Docker-compose.yml.

------
ppafin
On clean hardware under four eyes policy:

[https://imgur.com/a/FE8sslv](https://imgur.com/a/FE8sslv)

------
rgmvisser
We use Doppler for our encryption keys, great way to easily store dev /
staging / production keys in a secure and obfuscated way

------
vbhakta
We use lastpass for sharing keys internally and we use AWS SSM ParameterStore,
this section of AWS is only accessible by a few engineers.

------
asldfhasdf2
wrote our own kerberos-aaS clone with less features and vulnerable to more
internal attacks than plain kerberos and more reliant in a central cert (not
cert authority, cert), that is only used sporadically for cross services, not
users (there's something else from major vendor there)

and that team now keeps growing and the feature never improves :)

------
pengaru
There's a reason gpg supports --armor; print a backup copy of any important
keys and stick it in a safe deposit box.

------
delduca
What about [https://www.envkey.com/](https://www.envkey.com/)?

------
bsder
Lots of answers, and I still don't see anything valid for a 5 person startup--
2 of whom are executive types.

------
recursive
On a USB drive in a locked cabinet. It's not a hardened cabinet. A hard yank
would probably open it.

------
MattGaiser
At a prior company they were printed off and stored in a typical file folder.

------
shifty1
Generated by an automated process via service now and stored in cyberark

------
megavolcano
AWS KMS. Access is added invidually using IAM roles. That's it.

------
based2
anyone use
[https://www.rcdevs.com/products/spankey/](https://www.rcdevs.com/products/spankey/)
?

------
ykevinator
We have a shared Dropbox account with 2fa hardware and no otp

------
mancini0
We email them around lol. Wish we used istio.

------
fortran77
We check them into github! (sadly...)

------
elwell
Honeypot post, don't get doxxed

~~~
Zezima
Seriously. Who would be dumb enough to answer this question when they know
their key security is non-existent.

Most people have their employer or work email on their HN profile.

Really people?

------
bulletsvshumans
Wouldn't YOU like to know?

------
snow_mac
We use Azure Key Vault

------
rataata_jr
Lastpass?

------
winrid
Vault.

------
paloswag
Notepad

------
itpragmatik
AWS KMS

------
ahmetcetin
we use lastpass

------
nihil75
We use cloud-provider managed encryption because we're not paranoid and don't
have legal requirements to manage our own keys.

We don't have SSH keys because it's not the 90's and we don't have servers.

~~~
nocman
> We don't have SSH keys because it's not the 90's and we don't have servers.

This seems unnecessarily snarky.

There are _lots_ of businesses in 2020 that still maintain their own servers,
use ssh keys, and are staffed by admins and developers who very much know what
they are doing (and are not at all "behind the times", as this comment seems
to imply such businesses are).

If that's not what you meant, well, OK, but I find it difficult to interpret
it any other way.

~~~
pnutjam
He'll learn...

probably the hard way.

~~~
NicoJuicy
Or the expensive way

~~~
nihil75
My serverless infra costs me 0$ per month.

