
Crypto Tools for DevOps: Git-Crypt – Tozny - eamann
https://tozny.com/blog/crypto-tools-for-devops-git-crypt/
======
tptacek
Don't keep encrypted secrets in your git repositories, if for no other reason
than that it makes access revocation deceptively difficult --- but also
because it encourages you to have a development team in which ordinary devs
have a full complement of secrets on their laptops at all times.

Instead, keep secrets "out of band" and supply them to applications as part of
your deployment process.

~~~
hdhzy
Just curious - what's the preferred way of passing secrets from the deployment
process to the application?

Environment variables? Simple to use and understand but can leak to child
processes and crash reports.

Files on disk? Usually can also be read by children.

Via stdin on app startup? Seems simple...

Some other way?

~~~
peterwwillis
What app? They all take credentials different ways. The ideal would be an
unprivileged child talking to a privileged parent, where the child asks for
the secret and is authorized by the parent, and then both the child and parent
erase the [unencrypted] secret when not in use. If your app doesn't do this,
executing some app that then feeds the secret via a file descriptor
(pipe/socket/etc) (and hopefully erases all traces of it after) is similar.

The simplest hack is to use a configuration management tool to start your app,
and have its pre-exec step provide the credential, and post-exec remove any
trace of it (which, again, will not work everywhere). The config management
tool can be authorized to pull the credential from a remote host by some
public-key protocol and a hole in your firewall, or by a local cache of
secrets pushed by the config management tool only to the hosts and accounts
that specifically need them. These have security and reliability tradeoffs.

You can also use a credential management suite for a more robust solution, of
which there are both commercial and free options.

~~~
yeukhon
> "The simplest hack is to use a configuration management tool"

> "You can also use a credential management suite for a more robust solution"

You have to start somewhere, this is why automated system with credentials
isn't a simple task to solve.

Imagine this, you use consul to store your secrets, as you wrote up a script
for Ansible to look things up, this include secrets. Great, but you need to
authorize. So you need to keep the token somewhere. It's okay if you have a
person typing in, but if you deploy through Jenkins on a regular basis
automatically, you need to add that token as a password to Jenkins's data
store.

But you provision and configure your Jenkins using your Ansible playbook, and
your playbook needs to get to Consul to get to the secret so you can compare
the key on Jenkins. Now this is getting tricker.

Okay, say you are on EC2, you can use instance profile to grant instance
access to S3 / DynamoDB where you put your ultimate secrets. But if you are
paranoid or for some reason you decided to do client-side encryption and
keeping the master key to yourself (you generate them and you keep them, never
given out to AWS), then you are back to square one. Where the heck do you keep
your master key safe? KMS? No? In your own Consul? Okay where should we keep
the token? Ugh.

What if you don't use AWS? Google Cloud?

You have to trust at least one place. If your repo can be trusted, let it be.
There are three major factors in combating a system that has to put trust in
password/cert/key:

* keep rotate the credentials

* everyone should pull from the same credentials management infrastructure

* least privilege

But #2 takes a while to complete, mainly because now you need to develop /
rewrite your script on Jenkins to not use Jenkins's password datastore, and
the password masking plugin. You have to develop your own masking plugin.

The most trusted place is where you will need to add extreme security measure.
Assuming the encryption can't be broken in reasonable amount of time, the only
way to keep a secret unrecoverable is if the secret holder is dead ("only one
can live if two people know a secret).

~~~
dozzie
>> "The simplest hack is to use a configuration management tool"

>> "You can also use a credential management suite for a more robust solution"

> Imagine this, you use consul to store your secrets, as you wrote up a script
> for Ansible to look things up, this include secrets. Great, but you need to
> authorize. So you need to keep the token somewhere.

You see, the problem springs from you mistaking Ansible for configuration
management tool. It is not, it is somebody's deployment script. What you want
is CFEngine, Puppet, or Chef, and you need it download configuration templates
and fill them with secrets on the server that you want configured. (CFEngine
can easily do that; Chef -- probably, since it's not a standalone tool, but a
Ruby framework, so you have ERB at your disposal; Puppet? no idea if it can
use ERB in masterless mode.)

Now you need to solve the problem how to get the secrets to the server being
configured. This would be easy if you had a specialized database service that
ships credentials, and only the ones that are relevant to the server are sent.
It's a pity we don't have such a database (or at least I am not aware of any),
but it's not exactly a famine problem to solve, it's just juggling with access
lists and some simple log replication protocol.

~~~
yeukhon
When you said

> "You see, the problem springs from you mistaking Ansible for configuration
> management tool. It is not, it is somebody's deployment script."

You lost credibility from my PoV. Ansible is capable of doing what Chef can
do. Ansible can use dynamic inventory which means Ansible can use dynamic
secret infrastructure. That's not the default mode of Ansible out of box, but
much like Chef and Puppet, Ansible is made up of somebody's deployment script
as well. You and I didn't just write up Puppet out of no where, and there are
community plugins available for used, and those plugins are just somebody
else's deployment script/component.

~~~
dozzie
> Ansible is capable of doing what Chef can do.

So is gawk, except nobody tries to shoehorn it to do so or claim that it's
designed for this. Ansible's very architecture and operation model are causing
troubles where should be none, like applying configuration changes to hosts
that happened to be temporarily down.

> [...] much like Chef and Puppet, Ansible is made up of somebody's deployment
> script as well.

I don't know the history of Chef, but I assure you that Puppet was never
somebody's deployment script that happened to be published and gain traction.
It was designed ground up as a configuration management tool, after its author
deemed cfengine 2.x too troublesome, which again was a configuration
management since its early 2.0 days (and probably since 1.0 version).

Configuration management requires completely different operation mode than
deployment; one is asynchronous and unattended, the other is a synchronous
list of steps to take and often needs to stop on the first trouble to appear.

------
mackenbach
The series of articles the author intends to write are a great idea. Managing
secrets is a fast growing problem that needs more exploration.

However, I must say I disagree with the sentiment that separation of secrets
from source code is a bad thing. Git-crypt and similar tools use git for
versioning. While this sounds great, it is not desirable for key management.
Software and secrets have different management cycles. You always want to keep
a copy of past software versions, but this is not the case for secrets. For
instance, what if you want to make sure a secret is deleted? It will be a hard
challenge to remove this from the git history on all copies of the repo.

As commented before, separating your development workflow from your secret
management flow is actually a must. Not only are the management cycles
different, the access policies also differ. Giving only a few trusted
individuals access to the encrypted bag inside your git repo may work for a
very small team with a few similar servers. However, when you have multiple
sets of people and services that need access to different sets of secrets,
controlling who has access to what secrets with these encrypted data bags
quickly becomes impractical.

I agree with tptacek, having a decoupled and secure place to manage and
distribute secrets is more secure and scales better. At SecretHub we allow
access control per secret or secret-group to solve this complex mapping
problem. We believe it should be easy for developers to create secrets, but
only easy for machines to use them. Developers rarely need access to secrets
in production.

Take a look at our website to find out more:
[https://secrethub.io](https://secrethub.io)

Disclaimer: I'm the co-founder of
[https://secrethub.io](https://secrethub.io), an encrypted SaaS to help teams
manage and distribute secrets.

------
u801e
Several years ago, I worked on a proof of concept for storing secrets in a git
repository (making use of smudge and clean config options) where each secret
was encrypted using the public ssh keys (of each server's host key pair) of
the servers that would be deployed to.

Once the code was pushed to those servers, the clean filter would use the
private ssh key of the host key pair to decrypt the secrets.

------
mankash666
Vault from hashicorp was built to solve the use case described in the article.

However, encrypting source code, in general, especially from hosting services
like github is still a valid use case for many scenarios. GitZero attempts to
solve it, but it provides no specific guarantees that the source code (for
free tier) is inaccessible to the hosting service

~~~
eamann
Here's the full article about Vault, if you're curious:
[https://tozny.com/blog/crypto-tools-for-devops-hashicorp-
vau...](https://tozny.com/blog/crypto-tools-for-devops-hashicorp-vault/)

------
tachang
No mention of blackbox by stackoverflow? On a phone right now but it is pretty
useful for this type of stuff. The only thing bad about it is key management
and new developers have to generate their own keys.

