
Don't use ENV variables for secret data (2017) - cimnine
https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/
======
y7
So the author offers two alternatives:

1\. Using docker-secret inside of a Docker swarm

2\. Using Keywhiz [1], a Java server together with a FUSE client.

This seems overkill for a lot of cases. If environment variables are such a
security problem, why not just use a config file (not checked into the source
code repository) with proper permissions set?

[1] [https://developer.squareup.com/blog/protecting-
infrastructur...](https://developer.squareup.com/blog/protecting-
infrastructure-secrets-with-keywhiz/)

~~~
Someone1234
It reminds me a lot of .Net Core's newish "Secret Manager."

They created a user profile storage vault that's outside the deployment/source
code path (like ENV variables), and then for some inexplicable reason tell you
not to use it in production without _good_ justification anywhere and to use
their paid service instead ("Azure Key Vault" $3/100K requests) which is
multiple extra points of failure (even ignoring Azure's reliability problems).

Naturally people will repeat Microsoft's advice verbatim without justifying it
themselves like this SO answer[0]:

> Don't use app secrets in production. Ever. As the article says DURING
> DEVELOPMENT.

But WHY?! And the article they linked doesn't tell you WHY either, just points
to a paid service. And when these people get poked for an explanation, they
just wrap the same secrets in another layer of abstraction, but really haven't
changed the security of the operations they're performing.

For example if a server node gets compromised and that node is authorized to
make requests to "Azure Key Vault," it too can request the keys. The
abstraction may make sense for public-private key scenarios where the actual
private certificate is never returned, but a lot of what the "Secret Manager"
returns are raw database credentials and encryption keys, making this paid
abstraction more beneficial for centralized management than _actual_ security.

If people want to argue for more abstraction: Fine. But they have to explain
the logic behind the security.

[0] [https://stackoverflow.com/questions/39668456/how-to-
deploy-a...](https://stackoverflow.com/questions/39668456/how-to-deploy-asp-
net-core-usersecrets-to-production)

~~~
lostmsu
Another reason is that you have a single place to store your secrets shared
between multiple services. Now you only have one place to update when the
secret expires/is revoked.

------
ablanco
I don't agree at all. The reasons in the article all seem like "envs are bad
because if you make a mistake you can expose them". This is not exclusive to
envs, it applies to all secrets, independent of the medium used to make it
available to the process using it.

In my experience, if you prevent using envs for secrets (as docker swarm does)
all you get is a disgruntled programmer reading the contents of a secret file
to an env in the entrypoint.

~~~
michaelt
I think what the author means is environment variables are _particularly
vulnerable_ to being logged by accident, because:

1\. They're stored right next to variables like PATH, JAVA_HOME, LC_ALL and
PYTHONPATH which people might plausibly decide to log out every time

2\. They'll get printed any time someone writes a shell script with _set -x_
then uses the environment variable.

3\. They'll probably end up in your developers' ~/.profile or ~/.bashrc,
meaning _any_ program logging the environment will log it, not just _your_
program

4\. Because they'll be in ~/.profile or similar, the secret will be in a file
on disk _anyway_ and a secret that's in one place is always better than a
secret that's in two places.

With that said, a lot of CI servers that support "secure variables" offer
those as environment variables and nothing else. So I can understand why
people might end up stuck with environment variables despite their downsides.

~~~
imtringued
>1\. They're stored right next to variables like PATH, JAVA_HOME, LC_ALL and
PYTHONPATH which people might plausibly decide to log out every time

No, they aren't. You just create a .env file in your project root and run
"source .env" before you run your application.

Unlike Windows there are no global environment variables on Linux. All of them
are hierarchical and only exist within the process they were created and its
children.

------
kwhitefoot
Great :-(

Had to read to the end of the description of why environment variables are bad
to discover that it is effectively an advertisement for Docker. I don't use
Docker so the article told me pretty much nothing that wasn't fairly obvious
already, although it is a valuable reminder.

~~~
cimnine
This is not Docker related.

If an application spawns a sub-process, that sub-process will inherit all
environment variables. Which might be fine or might not be, e.g. if the
spawned application is user controlled.

Also tools such as Airbrake or Sentry often send all your ENV variables to the
error collection server, effectively exposing your secret values. Most such
tools offer to filter variables, but that's in my experience almost always not
done pro-actively.

The only thing that's Docker related in that post is that it does offer a
turn-key solution for Docker-based projects. The solution's principle can be
applied to other projects though, i.e. secrets should be read from a config
file (or something like Vault).

~~~
chme
> If an application spawns a sub-process, that sub-process will inherit all
> environment variables. Which might be fine or might not be, e.g. if the
> spawned application is user controlled.

Right. But isn't that well known?

Of course you have to explicitly define which environment variables are passed
through to the process you are going to spawn, just as you would have to drop
permissions to (configuration) files and possible limit capabilities if you
start spawning untrusted processes.

IDK... For me it make sense that you should not give secrets over the command
line, because they will appear in the program listings, but environment
variable is pretty much ok in many cases.

------
gchamonlive
I am progressively adopting Hashicorp Vault as the secrets manager of choice.
It can be used by a variety of different scenarios -- directly into the
application using AppRole, with Terraform using it's secrets provider, by
developers, during vault authentication when they get their secrets and access
regenerated.

This way I am not bound to docker swarm, or keywhiz, or god forbid AWS Secrets
Manager.

As of now, I am still exposing secrets with Env Vars, but the next step is to
use Vault directly. Vault has been pretty reliable so far. It is using AWS KMS
for managing the master key and a scalable DynamoDB table for high
availability backend.

~~~
oceanfloor
What's wrong with AWS Secrets Manager?

If I'm already working 100% in Amazon, I'm tempted to use Secrets Manager
rather than justify the cost in hours to deploy and maintain a Vault cluster.

Interested in your opinion.

~~~
falcolas
IMO, it gets expensive quickly, and it was designed around AWS' use case,
which involves credentials that roll on a regular basis. Outside of RDS, its
value for the price goes downhill quickly.

To a sister comment who mentions parameter store - that UI is the biggest
leaking bag of horse manure I've ever had the displeasure of using. We made
the mistake of using it, and moved over to Vault at the first available
opportunity. Vault isn't a silver bullet either, but the UI is at least
usable.

~~~
Lazare
The web UI is bad, but tools like chamber
([https://github.com/segmentio/chamber](https://github.com/segmentio/chamber))
are excellent, so there's no need to deal with the web UI.

------
photon12
Also this pattern makes path traversal vulnerabilities (a thing not uncommon
in web frameworks) have the potential to allow for privilege escalation on
Linux via the /proc/self/environ file.

I've been on a pentest where a recently disclosed path traversal bug in Rails
was not patched in the environment I was testing and I thought I would get at
least some credentials from at least one service, but every host used a
dedicated API for secret retrieval and there was nothing sensitive exposed via
any system.

Maybe your threat model doesn't care, just adding a data point.

~~~
joshspankit
Any chance they documented that setup publicly? Would be interested to dive in
to how all that works and gain any new insights

~~~
photon12
It's a large company that built their own bespoke internal credentials service
running over a TCP port to the application with another proprietary protocol
to push key material to hosts.

Can't say much more due to NDAs.

Edit: this service handles credentials and rotation for hundreds of thousands
to millions of hosts.

~~~
joshspankit
!

Wild.

I guess with custom protocols there’s not much that can be learned from that
setup. Presumably wrap the request in security, handshake to verify authority,
use the custom protocol to deliver the secret which is also wrapped in
security.

Too bad they keep it close to the vest, but I certainly don’t begrudge them
for it.

~~~
photon12
I mean the same company built a service with better architecture that they
sell as part of their managed computing environment options.

Some people complain about not wanting to use it due to "lock-in."

------
joosters
If you think that putting secrets in ENV is bad, you probably shouldn't take
the advice of running ' _docker service create --secret= "secure-secret"
redis:alpine_' either!

Putting secrets in a command line makes them just as visible, in fact more so,
to other processes. It also makes the secrets available to anyone regardless
of permissions, since everyone can monitor the currently running processes and
their args (e.g. a ' _ps waux_ ' or ' _cat /proc/12345/cmdline_')

~~~
IshKebab
`secure-secret` is not the secret itself. It is the name of the secret.

~~~
joosters
Oops, my bad. I saw the '\--secret="secure-secret"' part of the command and
assumed that was the secret itself! Thanks for the correction.

------
yakshaving_jgt
The proposed solution appears to be "use Docker", which is a bit lame.

------
hellofunk
Every production product I’ve ever worked on, the entire team put database
credentials in the environment variables.

~~~
Minor49er
The author's arguments aren't compelling. Most of his points can be solved by
properly configuring any logging processes to grab only what's necessary and
not letting newbies commit unreviewed code.

~~~
IshKebab
But the author's argument is that people don't do that. You're pretty much
saying "The argument for seatbelts is not compelling. The problem can be
solved by driving carefully."

~~~
Minor49er
No, I would be saying that wearing a seatbelt improperly is obviously
dangerous and ill-advised.

------
hankchinaski
We have recently started removing credentials in env vars and started using
google secret manager (previously berglas) and its been amazing so far. AWS
and azure should have the same. More challenging when you don’t have all the
infrastructure and services in place (ie. when working on plain VPS or other
systems without those tools)

~~~
seer
Hm not sure I understand how google secret manager relates to berglas to be
honest, thought those two were separate apis...

As for berglas itself, we also use it and have been very happy with it. Since
you put just the names of your secrets into the ENV files, not the secrets
themselves, they can be easily stored in version control, passed around in
chat and you can just do whatever you want with them. Instead of:

    
    
        ENV_PASS=my-secret-pass
    

You do:

    
    
        ENV_PASS=berglas://bucket/secret-id
    

And it will be decrypted at the last possible moment - e.g. when the system
starts. Or even later if you need to, if you use the apis provided.

Funny enough we had implemented the almost the same approach with AWS SSM apis
ourselves ([https://github.com/ovotech/ssm-env-
secrets](https://github.com/ovotech/ssm-env-secrets)). But I think it should
be possible to use berglas in AWS directly without issue.

------
mrkeen
For those in the comments suggesting to use another system to store
credentials instead of ENV:

Do you need a password to access that system? Why not?

------
djsumdog
Are secrets specific to docker-swarm? If you're using plain docker, I imagine
this wouldn't work?

k8s has its own secrets system built into deployments. I haven't used it
though. I've been at shops that use k8s+vault, and other places that uses
marathon/DCOS+consoul.

If you're on AWS you can use pod2iam in a k8s cluster and then use the SSM
parameter store to encrypt/decrypt secrets based on pod roles. I'm sure Google
Cloud must have similar services.

The most agnostic way would be to mount in a file or volume at runtime. It's
still accessible to the process, but just via the filesystem and not via
environment variables. You still need to program with security in mind, but
it's less likely for inadvertent leaks; basic layers of security. From there
you could use something that encrypts that mount at rest and decrypt it when
you start the container.

> Environment variables are passed down to child processes, which allows for
> unintended access.

Doesn't this depend on how you create the new process? fork() would keep a
copy of the env in both parent/child processes and exec would keep the env
because it replaces the current process. But if you start a process using
something like the subprocessing module in Python, it would give you a fresh
shell for that process, right?

------
hobbescotch
The way I got around this was to store secrets in Google KMS encrypted files
in Google Cloud Storage. The KMS key and encrypted files share the same name
and can be accessed by that name programmatically. This secret storage method
works really well for me and lets you easily access & manage secrets across
all environments. It's so convenient, I sometimes even use this system as a
simple key/value store.

~~~
ravenstine
That seems nice, but won't you be pwned the day that GCS accidentally loses
your files, or your entire Google account gets locked because its algorithm
mistakes it as a bot? The danger of that sounds greater than the possibility
that your environment variables get leaked.

~~~
alchemism
Neither of those scenarios quite applies to the cloud side of Google.

------
eihli
I came across a blogpost describing this workflow recently and I'm curious to
hear HN opinions about it. Any pitfalls?

[https://matthewdowney.github.io/encrypting-keys-in-
clojure-a...](https://matthewdowney.github.io/encrypting-keys-in-clojure-
applications.html)

1\. Generate a new set of API keys.

2\. Read my encrypted map of keys from disk, decrypt it with a passphrase,
assoc in the new key & secret, encrypt it again, and write it to disk.

3\. At the entry point for my application, use (.readPassword
(System/console)) to securely read in the passphrase, and then use it to
decrypt the key file and read it into a Clojure map.

4\. Instead of passing the key map around (allowing it to potentially escape
into a debug log, or be printed at the REPL if I do something dumb), the top
level code of my application passes the credentials into a signer-factory for
each api that closes over the credentials.

    
    
        ;; The factory is shaped something like this
        (defn request-signer-factory 
          [{:keys [key secret]]
          (fn [request-to-sign]
            (sign-request request-to-sign key secret)))
           
        ;; Then an API endpoint looks like this
        (defn place-order! 
          [signer {:keys [price qty side market post-only?]}]
          (let [request (comment "Format the order data for the exchange")
                signed (singer request)]
            (do-http-request! signed)))
    

I like this workflow more than others which are centered around only
encrypting credentials inside of your Git repository, and decrypting them when
you clone / pull, because it means that not even on my development machine are
keys just sitting around in plaintext.

------
xiwenc
From experience, it's good to support multiple ways of configuring an app.
Depending on your case, this could become hard or requires naming conventions.

This way you can move secret data to files if needed depending on the
deployment choice.

My own rule of thumb is:

1) sensible default value

2) read from file

3) read from env

Examples:

\- [https://github.com/spf13/viper](https://github.com/spf13/viper)

\- [https://docs.spring.io/spring-
boot/docs/current/reference/ht...](https://docs.spring.io/spring-
boot/docs/current/reference/html/spring-boot-features.html#boot-features-
external-config)

~~~
djsumdog
When I was doing Scala development, we went the same route using TypeSafe
Config. Default/file in the src/resources which could be overridden by ENV.

It seems like secrets aren't orchestration agnostic. You can't seem to use
Docker secrets without being in swarm mode, and k8s has its own secrets
management system (or if you're running on AWS, you can use pod2iam and ssm to
store/get encrypted parameters). I've been at places that use k8s+vault as
well.

------
RedShift1
And fix it by making it dependent on something docker specific?

------
worldhistory
I use and recommend Mozilla SOPS
[https://github.com/mozilla/sops](https://github.com/mozilla/sops)

------
irjustin
I'm surprised at the amount of responses here that hate this article.

It's not without cause/justification. Any stack tracking software (PagerDuty,
Rollbar, NewRelic...) gets huge amounts of secrets pumped out to them
regularly because of things like this.

And the author is not wrong, the environment doesn't need the secret. The
application does.

Sure, you may not like the proposed solutions, but there are plenty out there,
he just named 2.

------
tonetheman
Maybe if all of your stuff is in docker does this article make sense.
Otherwise it is just directly wrong. Use ENV variables they work.

------
fortran77
So to mitigate the security risk of ENV variables I should run a JavaVM, a
Java Server, and a FUSE client? No thanks.

------
makethetick
I just did a write up about how we use a secrets manager to load our
environments allowing an easy centralised management across multiple
projects/envs.

[https://news.ycombinator.com/item?id=23822681](https://news.ycombinator.com/item?id=23822681)

------
agentultra
If migrating your infrastructure to swarm is not feasible:

\- make sure to sanitize the environment before spawning any child processes.

\- Be sure to `set +x` (or your shell's equivalent) in your CI process

\- that your secrets never get interpolated into a string through your
scripting language.

------
joshspankit
Oddly not talked about much:

In a lot of cases, if an attacker gets even limited access to the application
environment, dumping the ENV variables is trivial as they were never intended
to be secure.

`process.env`, `printenv` (including php attacks), `ENV`, etc

------
tyingq
Is transferring them to memory in your startup routine, then doing unsetenv()
a reasonable mitigation?

It seems like it addresses several of the listed concerns. It's not perfect,
of course, but perhaps better, and straightforward.

~~~
ptx
I don't understand why we have to put them into the environment in the first
place (and then make sure we scrub it). Isn't it just as easy to read the
secret from a file?

~~~
gregoriol
The difference could be that files are managed by the devs and env is managed
by the ops. Depends on the team structure you have.

------
arnaudsm
What's a best practice that doesn't use this Docker-specific feature ?

~~~
wolco
Use a config file not checked into git

~~~
jimktrains2
This comes back to where does the config file live and how is it managed?

~~~
linkdd
The plain config file must live in memory (tmpfs). The encrypted config file
on a hard drive that is accessed only to decrypt the file to memory.

~~~
jimktrains2
No, I mean management of the secrets. Though it is a rather moot point as the
same problems apply to most systems anyway.

~~~
wolco
Wiki protected pages by role. When roles change so do passwords.

------
fomine3
I'm curious why such opinion is unpopular. ENV is too global and easily
exposed in accident. I think people just don't want to think to leave 12factor
(including me).

------
kazinator
> _At a previous job I helped solve this problem with a really elegant
> solution_

That distinction might go to something packaged in a single .h and .c file, if
it passes additional smell tests.

------
gfiorav
I mean, if the argument against it is: what happens if I run arbitrary code in
your server? The lease of your problems is having secrets in your env...

------
sahoo
There is secret manager is AWS and gcp.

------
shuringai
env vars are not collected by default within containers. if OP's alternative
is to use docker for every service then the whole problem is nonexistent since
containers cannot access the host env.

------
biznickman
Rails already solved this with credentials files...

------
quotemstr
The advice is good. But why is ENV capitalized? The term is just "environment
variable". One heuristic I use for evaluating technical material is
orthography: if you spell or capitalize or spell something improperly, it's
likely you'll get a lot else wrong too.

As for secret storage: didn't we solve this problem with keyrings? If I must
put a secret in long-term plaintext storage, I might as well put it in a file,
where I can see, access-control it, and audit it. Where's the audit log for
someone reading an environment variable value?

~~~
cholmon
Maybe he's got a history of working with PHP? $_ENV is an autoglobal array
that gets filled with environment variables.

[https://www.php.net/manual/en/reserved.variables.environment...](https://www.php.net/manual/en/reserved.variables.environment.php)

~~~
Izkata
The example they use at the top of the article is in bash/shell, and seeing it
written as in the article (as "ENV variables") is kinda like a code smell,
indicating the author isn't really familiar with what they're criticizing. I
think that's what GP is getting at, and it threw me as well - the article
really only makes sense when looking at it from the perspective of the author
not knowing how env vars are scoped.

~~~
coldtea
It's very common to write ENV in all caps. You can be a bona fide master in
UNIX and write it that way (just like some people write UNIX and others Unix).

Sorry, but technical knowledge/experience is not about particular spelling
conventions and word proofing, anymore than they are about wearing "business"
clothes...

~~~
quotemstr
> Sorry, but technical knowledge/experience is not about particular spelling
> conventions and word proofing

Yes, actually, it is. Sloppy language indicates a sloppy mind. I _absolutely_
crank down my estimate of someone's technical competence when I see him
writing badly and will continue to do so. Once in a while, you do see bad
writing coupled with good technical insight --- especially in infosec for some
reason --- but it's the exception, not the rule.

~~~
coldtea
> _Sloppy language indicates a sloppy mind._

No, that's a 19th century schoolmaster idea.

Many great hackers are messy with grammar and spelling, and even dyslexic.

~~~
quotemstr
The 19th century had a lot of good ideas that we've managed to forget.

