Keywhiz uses KMS to encrypt config data and then put it in DynamoDB. This means that access to Keywhiz-managed keys is ultimately mediated by access rules around who has access to the KMS key used to encrypt the secret.
With an S3-based solution, either Amazon can manage the keys for you and you use IAM Roles to mediate access (which I find to be a cleaner solution, personally, though open to other perspectives), or you can use KMS keys in S3 and then we're back to access being mediated by KMS keys, plus IAM roles, but without a DynamoDB table to manage or pay for (i.e. S3 will be super cheap b/c secrets occupy so little space).
I still appreciated the OP's intro to KMS, which I found insightful.
I had been planning to do this with keys stored in S3 and IAM roles to retrieve the keys at instance boot time, stored in a ramdisk, but this saves all of that trouble.
Thanks for rubber duckying this with me!
Hypothetically the problem of boot strapping trust is solved by tpm. However thats predicated on functional hardware attestation. And thus the hypothetical bit.
A newly brought up server can generate its own private key and issue a key signing request to a certificate authority, but how does the CA server authenticate the request? You still need a shared secret (perhaps implicitly through the username / password of an administrator doing things manually).
Any way you cut it, I don't see how you avoid this problem outlined in the article:
"Common practices include putting secrets in config files next to code or copying files to servers out-of-band. The former is likely to be leaked and the latter difficult to track."
I do take the point made by some of the sibling comments that this allows for better mitigation of the damage in the case of the loss of the bootstrap shared secret, and I can certainly see the value in that.
I hope this doesn't bring their entire infrastructure tumbling down when a network problem causes processes to block while reading from the mount point.
This doesn't need to be a complicated service. You could write an alternative to Keywhiz in bash in a weekend. This is a tremendous win over Java because 1) it's interpreted, so modification is trivial and fast, 2) it's a commonly known language by nearly everyone on *NIX platforms, 3) it's simple to troubleshoot, 4) highly extensible and 5) fast to develop with.
In terms of how to do this securely, you could of course still write it in bash and use FUSE to distribute your secrets, but why? Personally i'm more of a fan of push services when it comes to opening up what is literally all the keys to your kingdom. Design a host-and-user-and-service-specific trigger that can request the key server push the proper credentials to the machine and you avoid opening up your server to attacks on open services.
If you want to keep your secrets from getting on disk (and honestly, this isn't a concern for most servers as 99% of them are on all the time and will more readily leak secrets from memory than from disk), just push the secret to a tmpfs mount. The only annoyance is when your machine reboots each service's init script needs to request the secret be pushed, but I can't imagine that takes any more time than doing the same thing through Keywhiz.
I know the draw of writing everything imaginable in your favorite language. I used to write all my tools in Perl, but it turns out that's annoying for people who have to admin my services later that don't write Perl. For really simple admin tools like this, Bash is really the best thing for you. (Though i'll admit, some web app language more secure than Bash to handle the client->server requests would be handy)
More specifically to your point, I strongly disagree. People should build systems in their favorite language if the language fits the use case. And usually, the language chosen isn't as important as being familiar enough with the language that your solution is clear and well written.
However, contradicting myself for a second, bash is probably not a great choice to build complex systems. The problem with bash is that it's easy to write something that works, but it's much harder to change it and keep it working. Bash is very powerful for one liners, but the second your program grows, you start to miss all the nice things that most general purpose languages have. And another problem I've ran into with bash is that not all *nix systems actually have bash, and even those that do have many different versions which make it hard to write a large code base that works without issue across different services. Even more so when people tend to use non-bash binaries, and then you have to write up a script to install the dependent binaries, and have to make it work for different package managers and the complications go on and on.