

Keywhiz: Square's system for distributing and managing secrets - bascule
https://square.github.io/keywhiz/

======
mtsmith85
Coincidentally Coda Hale's sneaker
([https://github.com/codahale/sneaker](https://github.com/codahale/sneaker))
just popped up in my Twitter feed earlier. Sneaker stores the secrets on S3.
Keywhiz stores the secrets in a central database and then ephemerally on the
client servers. I guess if you started with something like sneaker on AWS
using Amazon's KMS you could then move to Keywhiz if you eventually moved out
of AWS.

~~~
toomuchtodo
You'd probably still want to use Sneaker until you were massive, even if you
moved out of AWS. S3 provided tremendous value compared to its costs (3
cents/month/GB for storage, requests are cheap as well) compared to EC2.

~~~
kingryan
Though the sneaker page makes it very clear that its not ready for production
use.

~~~
bri3d
I don't think "Keywhiz should be considered alpha at this point" really
screams production ready, either. For me, the Sneaker README's detailed
enumeration of which threat models had been thought over really helped inspire
confidence, as did the acknowledgement that no professional cryptographers had
evaluated its soundness (most people just ignore this idea and rampage onwards
unencumbered by reality). I believe that both are probably better than storing
plaintext keys and passwords at rest in Git or on developer machines.

------
tdicola
Wow this looks great! If the whole online payment system thing doesn't work
out for square the engineers there should fire the entire management chain and
pivot into a developer services company, because the projects and libraries
they publish are just stellar (particularly the Android stuff).

------
bradleyjg
Maybe I'm missing something, but how do you get the client certificate private
key on the e.g. new webserver which allows it to connect and retrieve its
server certificate private key? Isn't that the same key distribution problem,
just one step removed?

~~~
kbar13
[https://github.com/square/certstrap](https://github.com/square/certstrap)

~~~
bradleyjg
I saw a reference to that PKI package in the linked article, but it doesn't
really answer the question.

A newly brought up server can generate its own private key and issue a key
signing request to a certificate authority, but how does the CA server
authenticate the request? You still need a shared secret (perhaps implicitly
through the username / password of an administrator doing things manually).

Any way you cut it, I don't see how you avoid this problem outlined in the
article: "Common practices include putting secrets in config files next to
code or copying files to servers out-of-band. The former is likely to be
leaked and the latter difficult to track."

I do take the point made by some of the sibling comments that this allows for
better mitigation of the damage in the case of the loss of the bootstrap
shared secret, and I can certainly see the value in that.

~~~
cssx
You can solve this by having a bootstrapping process that issues the
appropriate credentials when bringing up a new server.

~~~
donavanm
And how do you trust the identity of the new server/instance during boot
strapping?

~~~
0x44
You could leverage the TPM and some version of remote attestation and only
permit key-requests from attested machines. Alternatively (or concurrently),
you could PXE boot all devices with a parameterized shared-secret
individualized for each node.

------
staunch
> _KeywhizFs is a FUSE-based file system, providing secrets as if they are
> files in a directory. Transparently, secrets are retrieved from a Keywhiz
> Server using mTLS with a client certificate._

I hope this doesn't bring their entire infrastructure tumbling down when a
network problem causes processes to block while reading from the mount point.

~~~
bigmac
There's an in-memory cache that keeps the secrets locally even when the server
disappears either due to downtime or network issues.

------
AndrewWright
Also coincidentally, this was just released today -- Credstash: a utility for
managing secrets using AWS KMS and DynamoDB. Written in Python.
[https://github.com/LuminalOSS/credstash](https://github.com/LuminalOSS/credstash)

------
toomuchtodo
Oh thank goodness! I was about to have to implement something like this using
S3 signed requests, ramdisks, etc. I owe somebody a beer! (or coffee)

------
doublerebel
Previous short discussion:
[https://news.ycombinator.com/item?id=9375620](https://news.ycombinator.com/item?id=9375620)

------
amelius
I want this in a smartwatch.

------
zobzu
mm it says "java" :(

~~~
peterwwillis
People downvote this, but there's a couple good reasons it shouldn't be in
Java.

This doesn't need to be a complicated service. You could write an alternative
to Keywhiz in bash in a weekend. This is a tremendous win over Java because 1)
it's interpreted, so modification is trivial and fast, 2) it's a commonly
known language by nearly everyone on *NIX platforms, 3) it's simple to
troubleshoot, 4) highly extensible and 5) fast to develop with.

In terms of how to do this securely, you could of course still write it in
bash and use FUSE to distribute your secrets, but why? Personally i'm more of
a fan of push services when it comes to opening up what is literally all the
keys to your kingdom. Design a host-and-user-and-service-specific trigger that
can request the key server push the proper credentials to the machine and you
avoid opening up your server to attacks on open services.

If you want to keep your secrets from getting on disk (and honestly, this
isn't a concern for most servers as 99% of them are on all the time and will
more readily leak secrets from memory than from disk), just push the secret to
a tmpfs mount. The only annoyance is when your machine reboots each service's
init script needs to request the secret be pushed, but I can't imagine that
takes any more time than doing the same thing through Keywhiz.

I know the draw of writing everything imaginable in your favorite language. I
used to write all my tools in Perl, but it turns out that's annoying for
people who have to admin my services later that don't write Perl. For really
simple admin tools like this, Bash is really the best thing for you. (Though
i'll admit, some web app language more secure than Bash to handle the
client->server requests would be handy)

~~~
chamakits
The downvotes are likely the result of not adding anything to the
conversation.

More specifically to your point, I strongly disagree. People should build
systems in their favorite language if the language fits the use case. And
usually, the language chosen isn't as important as being familiar enough with
the language that your solution is clear and well written.

However, contradicting myself for a second, bash is probably not a great
choice to build complex systems. The problem with bash is that it's easy to
write something that works, but it's much harder to change it and keep it
working. Bash is very powerful for one liners, but the second your program
grows, you start to miss all the nice things that most general purpose
languages have. And another problem I've ran into with bash is that not all
*nix systems actually have bash, and even those that do have many different
versions which make it hard to write a large code base that works without
issue across different services. Even more so when people tend to use non-bash
binaries, and then you have to write up a script to install the dependent
binaries, and have to make it work for different package managers and the
complications go on and on.

