
Client Certificates at Netflix - rchiniquy
https://www.scaleft.com/blog/client-certificates-at-netflix/
======
bazzargh
It's unclear from this and the slides what the UX for the engineers is? The
repo contains a bless-client that'll fetch a newly minted cert, but the slides
talk about integration with SSO - is there another piece that invokes bless
and drops the cert on disk?

[https://github.com/Netflix/bless/blob/master/bless_client/bl...](https://github.com/Netflix/bless/blob/master/bless_client/bless_client.py)

~~~
spydum
You linked right to it.. what do you mean? $ ./bless_client.py region
lambda_function_name bastion_user bastion_user_ip remote_username
bastion_source_ip bastion_command <id_rsa.pub to sign> <output id_rsa-
cert.pub>

If successful, I imagine it signs the pub key "id_rsa-cert.pub" which you then
can use to SSH as you please (until the cert expires).

~~~
bazzargh
That isn't the workflow described in the slides. They talk about signing in
via SSO to generate the cert, so it looks like this client is just an example
of what would be run at the back end of the web interface you sign into (with
2FA etc), to drop the cert. Look at that as a UI as well, it expects the
bastion user to know their own IP to put into the command? Nope. That's a
command to be run by something else.

But that just brings up more questions. Generally you want as little as
possible running on the bastion; it really is just that choke point. So I
presume the SSO front end is also not running on the bastion, but in another
subnet, then pokes the key back out...somehow?

~~~
russell_h
Author of the blog post here.

I wasn't at the talk, but had the same question.

BLESS, as released, appears to be intended to drop onto an AWS bastion host
and recommends an AWS IAM role to authenticate the bastion to the Lambda
function that acts as the CA. Like you mention, there are a lot of limitations
to that, but its still a neat demonstration of using ephemeral client
certificates without having to spin up a bunch of infrastructure.

At ScaleFT we're building a solution on the same principles that offers some
of what you're talking about out of the box:

    
    
      1. Authenticating users against a variety of SSO systems
      2. Treating bastions as nothing more than an untrusted TCP proxy
    

For me the critical takeaway though, is that whatever automation you're
layering on top, client certificates make great ephemeral access tokens and
they're increasingly catching on. Its just a matter of building the right
automation, integrations, and tooling on the client to make the experience
good.

------
blakesterz
[https://speakerdeck.com/rlewis/how-netflix-gives-all-its-
eng...](https://speakerdeck.com/rlewis/how-netflix-gives-all-its-engineers-
ssh-access-to-instances-running-in-production)

That's a great slidedeck, looks great and I can actually learn something from
it just reading the slides.

------
zokier
I'm not sure how much BLESS really improves the overall situation. It just
shifts the problem from securing SSH secrets to securing AWS/IAM secrets,
which seems pretty much the same to me.

~~~
spydum
The AWS/IAM secrets wouldn't be passed around, except on initial creation..
Not sure what you mean?

The problem they are fighting is host-level SSH authentication. Developers
tend to have SSH keys stored all over the place, and it's near impossible to
know if they protected their private keys with a password, or if they ever
accidentally published them (how many private keys can you find on github
these days?

Using client certificates instead, the developers have to ask for a
certificate (presumably with some BLESS client), and after authenticating
(with their credentials + 2FA) get issued a SHORT-LIVED certificate (5 mins)
signed by your own internal trusted CA, to use for SSHing into bastion host or
whatever.

So now, there is no key material handed to developer which will be trusted for
more than a few minutes. Even if their password/secret were leaked, you still
need 2FA and network access to get a client certificate. If you leak the
client certificate, unless someone snatches it up within it's validity period,
still no worries.

~~~
russell_h
Author of the blog post here. You totally nailed it about the problems with
SSH keys.

I think what zokier is referring to is that the BLESS client uses an AWS IAM
role to authenticate users, so if you wanted to invoke it directly from an
end-user machine they would need their AWS credentials on disk there. Instead
BLESS seems to be intended to be invoked from a jump host which has a machine
role giving it access to the Lambda function - which gives you a nice choke
point for auditing, but obviously has some other limitations.

I get the impression from the slides that Netflix might have some additional
internal tooling for authenticating users against SSO. I'm just excited to see
client certificates taking off as a form of ephemeral access token. I see too
many organizations that require MFA to access the corporate wiki but let users
authenticate to production with a private key from 3 jobs and countless
laptops ago.

~~~
shortj
For a layer of security, when using the client you could do IAM role
assumption in your local machine. You have your static creds and an MFA
device, then use those to export temporary credentials to actually run the
bless client.
[http://docs.aws.amazon.com/STS/latest/APIReference/API_Assum...](http://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html)

Some tools to make the process easier: [https://github.com/remind101/assume-
role](https://github.com/remind101/assume-role)
[https://github.com/trek10inc/awsume](https://github.com/trek10inc/awsume)

