
SSH Emergency Access - based2
https://smallstep.com/blog/ssh-emergency-access/
======
timeattack
It's cool and interesting application of the technology, but doesn't really
seem to be practical.

When you're unable to access machine using your standard SSH keys usually it
means that it's highly unlikely that it will be possible to login remotely via
other means.

As an emergency login there are two common options:

* in case of cloud: use remote VM console provided by the hosting provider.

* in case of bare-metal: use IPMI to access machine console directly.

~~~
tashian
Hey there — I'm the author of this post.

There's a few scenarios where I imagined this approach being useful:

* If you have any kind of remote dependency in your SSH auth flow (LDAP, or an online CA, or automated Ansible playbooks to push keys), any of those might fail and render the host otherwise inaccessible.

* It's becoming more common to not ever SSH into machines. So, what if emergency SSH access is the only way to access a host? Some companies even go a few steps further: When a host is SSH'd into, it is considered "tainted by humans", is quarantined and eventually shut down.

* Some hosts should never allow root access to anyone. For example, there's no reason for anyone to have root on a bastion host. So, what if the only way to get root on some hosts is with the emergency key?

While you could use the cloud VM console for emergency access in these cases,
having a hardware key provides even more security and would let you turn off
cloud VM access.

Of course if you broke your SSHD config, or have a network issue that prevents
you from reaching the host, this won't magically fix any of that. IPMI is good
for that though.

~~~
CodeWriter23
There’s also the access control feature of this approach. You can give someone
temp access to a host.

~~~
nokya
That's exactly the reason why we use certificate based ssh access at my
employer's: for our suppliers. It looks like the author went far away to find
alternate reasons to deploy this :/

------
ed25519FUUU
Here's what I like to do on my server(s) in cron, which pulls my keys from
github:

    
    
        @hourly <username> ssh-import-id gh:<github username>
    

If I lose my keys to this host, I can simply update github.com with my new
ones and go to lunch. I'll be able to login again shortly.

And on all of my hosts:

    
    
        @reboot <username> ssh-import-id gh:<github username>
    

This is REALLY helpful on devices like raspberry pi, where they may stay
shutdown / offline for _years_. The minute they're powered up again they'll
get my fresh keys and I can login to them without needing a console.

[http://manpages.ubuntu.com/manpages/bionic/man1/ssh-
import-i...](http://manpages.ubuntu.com/manpages/bionic/man1/ssh-import-
id.1.html)

------
cbb330
Why not use certificates as your primary authentication for SSH? Facebook has
a great blog post on implementing this at scale:
[https://engineering.fb.com/security/scalable-and-secure-
acce...](https://engineering.fb.com/security/scalable-and-secure-access-with-
ssh/)

~~~
tashian
Yes! Shameless plug — we (smallstep.com) offer a service that makes this
frictionless at scale and super easy to set up. You'll never want to go back
to public keys.

------
masonhensley
Neat - something I feel that often gets overlooked in most SAAS systems (think
internal side) be it customer service, ops, etc tooling is break the glass
escalation functionality. Most systems I’ve seen in the wild completely lack
this and will result in over provisioning of admin “god mode” accounts.

NoodlesUK points out alerting which is a pretty important concept to
incorporate.

Largely a solved concept in Electronic Medical Records & as outlined in the
post.

------
noodlesUK
I think another thing we might want to learn about is how to sound the alarm
when the break glass is used. Is there an easy way of doing that with SSH?
Running a command to page the ops/security team when a server receives a login
attempt with an emergency credential?

~~~
rsync
"I think another thing we might want to learn about is how to sound the alarm
when the break glass is used. Is there an easy way of doing that with SSH?"

Yes - quite simple and old-fashioned, actually ...

I have this line in the SSH users' .login file:

    
    
      /usr/local/sbin/sms 4153331111 4158882222 "USER LOGIN TO XXX - $DATE" >& /dev/null
    

... where the 'sms' command, above, is a shell script I wrote to call twilio
messaging with the curl command. A very simple example of that would be:

    
    
      curl -X POST -d "Body=$msg" -d "From=$from" -d "To=$to" "https://api.twilio.com/2010-04-01/Accounts/$accountsid/Messages" -u "$accountsid:$authtoken"
    

... and this works like a charm.

Alternatively, you could rick-roll your on-call sysadmin:

    
    
      /usr/local/bin/curl -XPOST https://api.twilio.com/2010-04-01/Accounts/$accountsid/Calls.json --data-urlencode "To=$number" --data-urlencode "From=$callerid" --data-urlencode "Url=http://demo.twilio.com/docs/voice.xml" -u $accountsid:$authtoken
    

(the voice.xml demo is, in fact, Rick Astley)

~~~
rnotaro
Direct link of the audio that the XML will play:
[https://demo.twilio.com/docs/classic.mp3](https://demo.twilio.com/docs/classic.mp3)

------
modinfo
What a coincidence, 3 days ago I ordered two pieces of yubikey 5, today
arrived a package and today I read a post on how to use them in an interesting
way for emergency access to my server via SSH.

I'd like to add that the way it's described really works.

But... Now I don't know to leave one yubikey in case I need to use it for
emergency access to ssh? I have a server since 2011 and I have never problems
with access through ssh, I use the same keys to this day and everything works.

I think this way with yubikey to emergency access is overkill.

It's just an interesting way to use yubikey.

~~~
danmur
If you need to the option to give someone temporary access it seems like a
good option. I don't think it would add anything to my personal stuff since
there's no reason I can think of to give someone else access. At work
definitely.

~~~
dcow
Right, this is more about a cryptographic grant of temporary emergency access
to someone who doesn't have a user account or admin keys already on a machine
(and ideally nobody should have persistent admin access in a well-oiled
production setting) in the event that existing access control mechanisms have
failed. And backing the signing operations by a YubiKey lets you physically
secure the key in ways that you wouldn't an entire laptop and provides all the
benefits of tamper resistant, proximity aware, hardware. Probably not
something most people will want or need to bother with for personal stuff, but
very reasonable expectation as soon as you're working on a team or managing
many hosts, etc.

------
alexandrerond
I'm very confused, given Yubikeys have smart card fuctionality and they can be
used by gpg-agent to SSH with the regular gpg key (you can add to
authorized_keys just like any other keys) and you don't have to go through
this whole mess of setup to create a CA and install it.

What am I missing?

~~~
munchbunny
It's a chicken and egg problem: if you can't SSH into the machine, how do you
add your key to the SSH config on the target machine?

You could use a very long lived key, but then as soon as you have multiple
people who might need production SSH access, you've got access control and
revocation issues. The SSH CA is a good minimal solution, because the CA can
issue only short-lived SSH keys (few hours at a time) that you use once and
throw away. Also, CA trust scales better because it moves user management
burden to the certificate issuing process and removes the need to modify the
SSH config every time you onboard a new user.

It's a pretty standard practice. Here's a post from Facebook about it from
several years ago. This post is just about how to do it using YubiKeys.
[https://engineering.fb.com/security/scalable-and-secure-
acce...](https://engineering.fb.com/security/scalable-and-secure-access-with-
ssh/)

~~~
harikb
This can also work as a solution where the “setup” (of trusting CA) is baked
in to the image. Then there is no ssh related setup until the day you actually
need to ssh to the host. And you get the guarantee that no ssh login can
happen until you issue a temp-pair.

This is actually quite useful for deploying clusters of machines that one
doesn’t want normal ssh access until there is a real need. I think this was
also mentioned in another comment

------
aaronmdjones
To add to the blog post; you don't need brew or step or any of that nonsense
to inspect certificates.

    
    
        $ ssh-keygen -Lf the-cert.pub

~~~
dcow
You can, but `ssh-keygen` is about as nice to use as `openssl` which
practically means you spend a lot of time with your head in the manual. The
`step` tools have a nicer UI:

    
    
        $ step ssh inspect the-cert.pub
    

Also the post already mentions that using `step` instead of `ssh-keygen` is
_optional_ , so I'm not sure why you feel the need to repeat it...

~~~
aaronmdjones
Right, you'd have to look up the switches in the manpage if you don't remember
them, but that's already the case with the generation portion, which is why
the post includes the switches for that. I'm just saying it could have
included the inspect switches too.

~~~
dcow
We actually plan to update the post to demonstrate doing it entirely with the
`step` tool. We just want to do a pass on the UX to make sure it is as easy an
foolproof as possible before bringing more attention to it.

------
munchbunny
This is a pedantic detail, but if you're trying to implement this system, it
does matter: "resident key" is not a required feature here. You're not using
the hardware token for its WebAuthn capability, you're using it for its smart
card capability.

You just need PKCS11 token support for SSH, which the YubiKey's smart card
capability can do. YubiKey 4 and YubiKey FIPS can both do it, and so can
regular old smart cards even though that form factor is a lot less popular
now.

The workflow is the same: generate a key pair on the hardware token, have the
CA sign it, install the signed cert onto the hardware token, and then SSH with
it.

~~~
closeparen
The procedure here is actually using WebAuthn, which is now explicitly
implemented by OpenSSH.

------
markpeek
Using certificates with SSH is the way to go for shared access servers. Here's
an open source way (yes, I'm involved in the project) to manage authorization
and access with asynchronous approvals:

[https://github.com/cloudtools/ssh-cert-
authority](https://github.com/cloudtools/ssh-cert-authority)

~~~
dcow
Smallstep also offers an open source ssh-aware kms-backed certificate
authority.

[https://github.com/smallstep/certificates](https://github.com/smallstep/certificates)

One nice advantage is its support for different provisioning flows. The oauth
flavor allows you to hook into an existing identity provider to authenticate
certificate requests.

Simply:

    
    
        $ step ssh login
    

and boom you've got a short-lived ssh certificate in your ssh-agent using a
private key that never touched the disk.

------
jlgaddis

      Valid: from 2020-06-24T16:53:03 to 2020-06-24T16:03:03
    

Um...

------
ascotan
I don't get it. When would u need a backup ssh key other than if a user lost
access? Most VMs have console access for this purpose.

~~~
asdfasgasdgasdg
I'm not sure of the exact scenario but I would just note that there are other
types of computing environments than virtual machines. For example, there a
physical machines, sometimes hosted in a colo where you have no employees on
the ground.

~~~
gnufx
Surely you'd have some sort of remote KVM in such cases (like IPMI, as
mentioned in another comment). That's critical in the clusters I've run and,
of course, the manufacturers' implementation of that critical functionality in
IPMI is likely to be rubbish and you can't get it fixed...

------
dmitrygr
Website unreadable on mobile. Commands cut off and not scrollable.

------
ausjke
what about port knocking?

