
Ask HN: How do you manage SSH keys and SSL certificates in your company? - johnmathew123
Do you use - Universal SSH Key Manager, ManageEngine Key Manager Plus, Cyberark Solution, or any other tool? Tips please.
======
mheiber
It baffles me that nobody seems to have mentioned Hashicorp's Vault yet:

[https://www.vaultproject.org](https://www.vaultproject.org)
[https://github.com/hashicorp/vault](https://github.com/hashicorp/vault)

It comes with both, a full blown PKI (want a new cert? Use an authenticated
REST endpoint!) and SSH backend. On top of that you can use it to manage
accounts for many other third party applications as well (e.g. PostgreSQL,
MySQL) while leveraging a multitude of authentication backends to delegate
granular access for all of these features.

It's been an absolutely eye-opener for somebody like me who's used to managing
SSH keys/PKIs (what a pain) and I wouldn't want to use anything else right
now.

~~~
serkanh
So let say you have bunch of servers that your team has to access via ssh, how
would vault help adding users public keys add it to ~/.ssh/authorized_keys of
each machine? i am familiar with vault locking/unlocking secrets but not sure
if Vault can help centralizing and deploying those keys to individual
machines.

~~~
jen20
A good solution to this problem is to use an SSH Certificate Authority - then
you need only configure the CA certificate on each box, and you can either
issue semi-long-lived certificates to each user who needs access, or use
something like Vault to issue short-lived certificates intended for one-time
use.

This model is described in an excellent post by Facebook from a while back
[1].

(Disclaimer: I used to work at HashiCorp, and put this model into production
there, though the Vault support for issuing short-lived certificates was added
after I left)

[1]: [https://code.facebook.com/posts/365787980419535/scalable-
and...](https://code.facebook.com/posts/365787980419535/scalable-and-secure-
access-with-ssh/)

~~~
fragmede
Using an SSH Certificate Authority is also my recommendation, but be aware
that it's relatively new, so associated tooling with it is not super mature
yet. In particular, the user still needs some bits in order to login, and
whether they generate it themselves and send it off to get signed, or the bits
are generated for them on the backend and the user simply needs to receive
them, there's a management aspect to it that isn't a totally solved problem
with open source tools.

It's not a difficult problem, mind you, but there was custom code written that
runs on developer laptops (OS X and Ubuntu) to support this workflow.

(Despite being a very similar looking string of bytes as more traditional
pub/private keys, it's different in the SSH-Agent protocol, so don't assume
all ssh-agent-looking daemons support it.)

~~~
bsabrin
ScaleFT uses a certificate based approach and issues short lived certain for
SSH and RDP. Also has a policy engine to set additional access controls.

~~~
true_tuna
I tried to deploy them to grant developers limited time access to the hosts
running their services. Didn't really work out and I killed the project.

------
jamiesonbecker
I'm the co-founder of a startup that has developed a tool for this called
Userify[1].

It creates and removes _local_ accounts and manages the ssh keys and sudo
permissions centrally, so you don't have to worry about not being able to get
in if your LDAP/AD is down. (Our Enterprise edition, self-hosted in your VPC
or in your DC, can optionally integrate with LDAP or AD for dashboard logins,
MFA, etc.) We also have an AWS server edition in the AWS marketplace. When you
remove someone in the dashboard, their account is removed across all of the
servers they have access to, _and_ their sessions are terminated, but their
home directory is retained for archival or forensics.

Our primary foci are paranoid levels of security (hardened, all data encrypted
at rest with X25519, in motion with TLS), flexibility, and speed: fast to
deploy or integrate (w/ ansible, chef, puppet, terraform, cloudformation), and
fast to install (milliseconds), with both SaaS/Cloud and self-hosted versions
and open source python agent (we call it the 'shim').

We were founded in 2011 and serve thousands of servers and users in real time;
I'm first.last at userify for any questions, and/or info at userify.

1\. [https://userify.com](https://userify.com)

~~~
ytta
I thought about doing something like this in the past, did you find this to be
relatively profitable? Was it something that a lot of people wanted?

~~~
jamiesonbecker
No, it's a terrible space. Pick something less noisy and with fewer
competitors. Like log management. ;)

------
sp0ck
LDAP as a public key service and servers configured via PAM to use that as
source for pub keys. Nothing to distribute. Delete key from LDAP and second
later user can't log on any machine. We are analysing teleport ssh suite for
possible migration direction. SSL is different story :)

~~~
Pyxl101
Can anyone log into hosts if LDAP is down? Is that a concern? Is there an easy
way to mitigate the concern if you wanted to? Interested in exploring this
solution, but worried about the availability risk.

~~~
stephenr
Apart from the caching stuff mentioned in other replies, this is why you have
multiple ldap servers in a HA replication setup.

~~~
scurvy
Yeah running multiple servers is easy and the daemons are very mature at this
point. LDAP daemons aren't crash prone. I haven't had an LDAP outage in over
10 years.

------
janvdberg
We use Ansible to deploy/manage peoples' SSH keys on our servers. From their
laptop or a jumpbox (within the management VLAN) with their personal key (and
a passphrase!) they are able to login in to all those servers. So logins are
personal (as opposed to shared accounts which have to be updated when people
leave). Now when new people arrive or when people leave we just run an Ansible
playbook and all our 400+ servers are updated.

~~~
drinchev
How do you manage to "remove" SSH keys. Since sensible is stateless, you
probably run once for removing and once for adding a new key.

~~~
Spivak
There are two states that we're talking about.

* Ensure that the following key is present.

* Ensure that only the following keys are present.

You can accomplish the second one with the 'exclusive' flag.

[http://docs.ansible.com/ansible/authorized_key_module.html](http://docs.ansible.com/ansible/authorized_key_module.html)

~~~
falcolas
There's a third state as well: absent. Move keys from one file to another, and
they go from allowed to denyed.

That said, I do like the "exclusive" tag too, I just tend to have a few one-
off keys laying around as technical debt.

------
grogenaut
SSH: Daily key generation and rotation with a 2FA registration system. Keys
come from this local daemon, once per ui session / 12 hours. Port knocking to
get through bastions. All bastions and config done via enterprise managed
setup. You're not on a managed machine, you're not on prod/vpn/ssh/etc.

SSL: AWS KMS style solution which predates it on internal, and new system
built on KMS. These systems are merging as KMS takes a lot of the load off.
Then it's down to building a key distribution system. All secrets actually
stored in this type of system. Devs rarely access secrets directly. Lots of
nice "client" wrappers which look like DynamoDBClient, or Mysql driver but
actually fetch passwords on a 5 minute rotation from the secret store so a
rotation means push new key, wait 5 minutes, pull old keys. Secrets preferably
never hit disk.

Not a big fan of vault because it wants to connect in to your hosts to manage
and rotate passwords.

~~~
Tharkun
Interesting. What kinds of tools are you using for your SSH setup?

~~~
grogenaut
in house. you may be able to figure out where I work from my history which
would explain why.

------
rocmcd
Regarding SSH keys, for anyone using Github organizations, we created a
service called GitWarden[1] for automatic syncing of local user accounts/SSH
keys with organization teams. This makes it very easy to manage users across
your entire infrastructure directly through the Github UI, and have any team
changes (add/remove members) reflected locally in near real-time. It also
makes it incredibly easy for users to login, as they just use their Github
username and any SSH key from their Github profile. There's a full
demonstration in one of our blog posts here[2] for anyone interested.

This is a new service we recently launched, so any feedback would be greatly
appreciated! My email is in my profile, if you would like to reach me
directly.

[1] [https://gitwarden.com](https://gitwarden.com) [2]
[https://gitwarden.com/blog/2017/05/04/first-steps-with-
gitwa...](https://gitwarden.com/blog/2017/05/04/first-steps-with-gitwarden)

~~~
uiri
Is there any logging of which ssh key is used? I don't think that I would be
comfortable having access to work material from a personal machine, assuming
the company is large enough to issue work desktops/laptops.

~~~
rocmcd
Not currently, but that's a great point that we haven't considered yet.

------
aren
My company Foxpass (YC S15) has a product to manage SSH keys. It serves as
access control too -- the keys are only available on the hosts where a user
should have access.

[https://www.foxpass.com/](https://www.foxpass.com/)

~~~
gingerjoos
We've been using Foxpass for more than a year now and can definitely
recommend. We have Amazon Machine Images with the required packages installed
and configured; we use the web interface to grant/revoke access to users and
add SSH keys. So each user logs in as themselves.

Aren has been awesome with responding to emails and helping us set it up too.

~~~
voltagex_
How does it handle sudoers?

~~~
aren
I didn't mention it in the original post, but Foxpass also has an LDAP
endpoint so it can manage users and groups on your linux machines.

This means you can set up a linux group with sudo capabilities (sudo or wheel,
usually) in /etc/sudoers. Then using Foxpass you can manage the membership of
that group by adding users on a permanent or temporary basis.

------
subway
For SSH:

I'm a fan of public key distribution via directory service (LDAP) with
failover/caching via sssd. Private key storage on a smartcard (Yubikey in my
case)

SSL: It depends on where it's being deployed -- AWS apps use ACM, situations
where I need to access the secret from within a running system image (gce/ec2)
generally uses the platform's kms long with their object store.

Can't help but notice this account only has two posts, both asking about
commercial solutions folks use for secret management. I'm gonna take a stab in
the dark and say this is market research?

------
deepsun
I would prefer to not manage them at all. Let my cloud provider care about SSL
(AWS already provides it).

I don't even want anyone (including me) to have access to our SSL
certificates. There's no lock-in, I can always change my mind and buy new SSL
certs somewhere else.

------
ineiti
We were working on a more decentralized solution here:
[https://github.com/dedis/cothority/tree/master/cisc](https://github.com/dedis/cothority/tree/master/cisc)
\- unfortunately the update is still 1 month away...

This system uses a set of untrusted nodes that form a permissioned blockchain.
Updating the chain requires a threshold of keys stored in the first block. The
private keys are distributed over laptops/phones.

A person can have multiple devices that accept/deny new keys, while the
servers check periodically for updates and can verify the new ssh-keys are
legit by verifying the signatures.

I did a small demo at HotPETs 2016: [https://www.securityweek2016.tu-
darmstadt.de/fileadmin/user_...](https://www.securityweek2016.tu-
darmstadt.de/fileadmin/user_upload/Group_securityweek2016/pets2016/1_Managing_identities_bryan_ford_etc.pdf)

I also hope to have it running again, soon. If anybody is interested, don't
hesitate to contact us at linus.gasser@epfl.ch

------
inopinatus
Last time I did ssh key distribution on a global scale it was with cvs, cron,
and makefiles. Self-generation of key with gated registration to central
repository, auth'd via k5, distributed via cvsup (remember that?). Automatic
expiry according to configurable policy. Two-person-rule for root trust and
policy changes. We also modified sshd for better key usage auditing.

------
bluecmd
We used ScaleFT for Dreamhack ([https://www.scaleft.com/blog/how-dreamhack-
used-scaleft-to-s...](https://www.scaleft.com/blog/how-dreamhack-used-scaleft-
to-secure-the-worlds-largest-digital-festival/)) and it was very easy to work
with. A very good product if your company simply wants something that works.

~~~
bretpiatt
Another happy ScaleFT customer at Jungle Disk
[https://www.jungledisk.com/blog/2016/07/13/behind-the-
scene-...](https://www.jungledisk.com/blog/2016/07/13/behind-the-scene-at-
jungle-disk-first-peek/)

------
nikcub
For self-hosted PKI checkout Lemur from Netflix:

[https://medium.com/netflix-techblog/introducing-lemur-
ceae88...](https://medium.com/netflix-techblog/introducing-lemur-ceae8830f621)

------
agrinman
We just release Kryptonite ([https://krypt.co](https://krypt.co)), an HSM for
your SSH private key on your phone.

The private key never leaves your phone. Pair your phone with your computer to
create a secure channel (channel is encrypted + signed with session keys only
known to your computer and phone). Every time you SSH, the computer calls out
to your phone over this channel and asks you to approve a signature.

SSH logins with simple push notification approvals. Code is public:
[https://github.com/kryptco](https://github.com/kryptco).

------
lucd
We use CFengine 3 to manage personal accounts and ssh keys on servers. As
CFengine continuously checks and converges configuration, if someone set bad
permissions on their ssh file or remove the authorization it is automatically
corrected before five minutes have elapsed.

------
rcarmo
At my previous company, we used Puppet to distribute both. SSH public keys
were part of a developer's intranet profile, and SSL certificates were managed
on a central repo.

Updates to either would be passed on to Puppet and distributed automatically
across 1400 machines.

(Edit for typo)

------
wgoulet
The company I work for (Venafi), provides enterprise grade SSL certificate and
SSH key management solutions. The on-premise enterprise solution provides
certificate lifecycle management (monitoring for expiration, automatic
renewal, and automatic installation). It provides a REST API so you can
automate the process using your toolkit of choice. We've also recently
released a new cloud service that offers free certificates for use in dev/test
environments (and will have more functionality added soon!)

------
sneak
We use Yubikeys as GPG smartcards, and use them for gpg-agent as ssh keys.

Everyone puts their hsm keys on their github account (and removes all others).

We fetch the keys for each user from github on system init. e.g.

[https://github.com/sneak.keys](https://github.com/sneak.keys)

When we need to add/remove people, we just update the list of usernames in the
script that fetches keys, and then kill off instances one at a time to force a
redeploy.

~~~
desdiv
A comment above,
[https://news.ycombinator.com/item?id=14321901](https://news.ycombinator.com/item?id=14321901),
does a similar thing as you, except they use the Github API to fetch the list
of usernames. I don't know your use case, but maybe this is useful to you too.

>[https://github.com/sneak.keys](https://github.com/sneak.keys)

How do you do that? I mistook you for a Github employee at first.

~~~
jon-wood
Appending .keys to your Github user profile URL returns SSH keys. Hugely
useful in organisations that are using Github for code management already.

~~~
voltagex_
Is there any danger to adding keys to the Github account that are used
elsewhere?

~~~
Tharkun
"It depends".

I dislike that Github doesn't explicitly mention that it publishes your public
keys, because they can be used to figure out your identity across multiple
services. I believe someone a while back posted a demo on HN, where you could
SSH in and it would greet you with "hello $yourname", which it derived from
your github keys.

My advice: if you use different (user)names for different services, you should
probably consider using different SSH keys for them as well. That is, if you
don't want those two identities to be tied together.

~~~
Foxboron
Wrong solution.

The main problem that Fhilippo pointed out with the service is that ssh by
default gives all keys present inn your keyring, or all keys named
id_{rsa,dsa,ecdsa}. What should be done is to never present ALL your keys, but
turn on "IdentitiesOnly yes" in your SSH config.

Project:
[https://github.com/FiloSottile/whosthere](https://github.com/FiloSottile/whosthere)

~~~
Tharkun
Yes, that's part of the solution indeed.

------
cloudposse
We use the Github API to retrieve public keys and teams to manage membership.
Using this app to provision users on servers automatically and serve public
keys. Read more here: [https://github.com/cloudposse/github-authorized-
keys](https://github.com/cloudposse/github-authorized-keys)

It supports caching to etcd (incase GitHub API is unavailable).

------
jacques_chester
At Pivotal we're building CredHub for managing SSH keys, TLS certs, as well as
passwords and arbitrary values.

It's specifically being built for particular systems to consume, and slowly
generalised as it goes. The first consumer is BOSH. Typical BOSH deployments
can have dozens or even hundreds of sensitive credentials to manage. Across a
large company it quickly escalates into the thousands.

------
seroland86
I am the author of a PAM module called Keeto that lets you reuse all key
management processes of a PKI for X.509 certificates for OpenSSH. Furthermore
a layer has been added to centrally manage access permissions for OpenSSH
servers. For further information see: [https://keeto.io](https://keeto.io)

------
mjh__
In our new stuff at aws we are storing public SSH keys in IAM and using a
slightly modified version of [https://github.com/widdix/aws-
ec2-ssh](https://github.com/widdix/aws-ec2-ssh) to authenticate / synchronise
users. So far it looks like it ticks all boxes for us.

------
Kiro
The sysadmin just adds each employee's key to the servers. From reading this
thread it sounds like bad practice?

~~~
JoiDegn
Do you know how the sysadmin organizes himself? Is there a process where he
gets notified of leaving engineers to remove their keys?

edit: typo

------
sapphire_tomb
We've been looking into deploying Conjur
([https://www.conjur.com](https://www.conjur.com)) for the SSH part of this
question at least, as they provide a rotator which doesn't require changes to
our server build (i.e. no special PAM modules or the like).

------
buschtoens
We don't.

As much as I hate do admit it, but I belong to the small Linux minority in my
otherwise Windows only business.

~~~
Tharkun
We don't either, and we're mostly a Linux shop.

We only have ~10 employees. About half of them are technical staff with some
access to certain systems (git, mostly). Only two users have global root
access.

Vault (which was mentioned somewhere in these comments) looks interesting, but
at our current scale it looks like it might be overkill.

------
dddw
keybox, quite handy.

[http://sshkeybox.com/](http://sshkeybox.com/)

------
skulblock
We use Phabricator (installed locally) to manage most things in our company.
We have a 20 member team, all repositories, ssl certificates, issue tracking
etc

[https://www.phacility.com/](https://www.phacility.com/)

------
ReligiousFlames
monkeysphere for personal ssh private keys stored in gpg

chef and hashicorp vault

Another neat thing to deploy into dns is sshfp records so there's almost never
ssh fingerprint verification prompts for deployed hosts. Alternatively, ssh
host fingerprints can be deployed to LDAP.

~~~
shabble
> _Another neat thing to deploy into dns is sshfp records_

For those wondering, [1] provides a bit of a background on SSHFP records. You
can only skip host-key checking entirely if it's served with DNSSEC, although
that might be easier if you're running internal DNS.

How do you have your system working? Its something I've fiddled with briefly,
but ultimately gave up on for now.

[1] [https://matoski.com/article/sshfp-dns-
records/](https://matoski.com/article/sshfp-dns-records/)

------
darkr
SSH keys - OpenPGP smartcards (E.G yubikeys)

SSL Certs - vault PKI (internal) AWS ACM (external)

------
ageofwant
This works for me
[https://github.com/grahame/sedge](https://github.com/grahame/sedge) not vault
no, but better than straight .ssh/config.

------
rvanlaar
I have a similar problem. Keeping check of which certificates are about to
expire. I built an MVP to help with that:

[https://captaincert.net/](https://captaincert.net/)

~~~
billyhoffman
Your "Account" link is failing, and also returning a verbose debugging output
for Django...

[https://captaincert.net/accounts/login/?next=/account/](https://captaincert.net/accounts/login/?next=/account/)

~~~
rvanlaar
Thank you for taking the time to look around and report back.

These two issues are known. The next step is notifying users via e-mail of
expiring certs. Then the account button will be functional.

We're validating the value proposition. I've a real need for this application,
but we're trying to see if others have the same need.

Would you like to be notified when the account and reminder functionality goes
live?

------
scurvy
Maybe we're old school, but we use autofs and nfs to mount homedirs and ssh
keys on demand. Puppet handles host ssh keys across machine rebuilds. It works
well, no need to replace it IMHO.

------
revicon
SSL certs are managed by AWS certificate manager and all VMs are deployed via
elastic beanstalk or exist as Lambda functions. No SSH access is enabled
across any of our infrastructure.

~~~
cheeze
What has your experience been with AWS Certificate Manager?

~~~
grogenaut
Cert manager is only for a subset of the aws offerings and I don't think it
works for EC2 instances. I know it covers cloudfront, which means it covers
static websites under https (CF fronted S3), Api Gateway (which is CF under
the covers for the most part), so also Lambda apis. A seperate but same thing
covers ELB. I'm not sure that it covers ALB yet.

That said for these services it works very well, except there are some issues
with it lagging on sending confirmation emails that just make you retry.

------
based2
Another one:
[https://www.rcdevs.com/products/spankey/](https://www.rcdevs.com/products/spankey/)

------
fjallstrom
we use the public keys from existing github accounts in our organisation. a
little cronjob runs daily on all servers to fetch all keys.
[http://labs.earthpeople.se/2016/04/controlling-ssh-access-
wi...](http://labs.earthpeople.se/2016/04/controlling-ssh-access-with-github-
organizations/)

no idea how secure this is or what implications this could have but it's easy
and works well for our use case.

------
postdevops
shadowd - Secure Login Distribution service

shadowc - client of shadowd

[https://github.com/reconquest/shadowd](https://github.com/reconquest/shadowd)

[https://github.com/reconquest/shadowc](https://github.com/reconquest/shadowc)

------
mlosapio
FreeIPA

------
throwaway18917
Dropbox and 1Password. It's, uh, not great.

