
SSH gets protection against side-channel attacks - throw0101a
https://www.undeadly.org/cgi?action=article;sid=20190621081455
======
Koffiepoeder
RAM encryption for sensitive data is overlooked in so many applications, even
"highly secure" applications like veracrypt [0] only recently started adding
it.

In my opinion server-applications of all sorts should encrypt their private
keys by default; this makes cold-boot attacks and other memory-escape attacks
so much harder, since now two totally unrelated memory chunks have to be
combined in order to retrieve the private key (in fact one could argue that,
the higher the quantity of memory chunks is, the harder it is to correctly
decrypt the original private key; although to me this has the security through
obscurity smell).

In practice however, this is rarely done. OpenSSL for example has a whole lot
of memory-management functions [1], but none of them seem to include RAM
encryption (AFAIK it is not present in the source code elsewhere either, but
it's a large one, so I am not familiar with all code. Related: in fact some
people even argue it's way too large for it's own good [2]).

Even better still, would be to use the CPU Cache for private keys, since this
memory is even more difficult to access through any sort of cold boot. An
interesting paper with much more information about this subject can be found
here [3], to whom it may concern.

[0]
[https://github.com/veracrypt/VeraCrypt/commit/321715202aed04...](https://github.com/veracrypt/VeraCrypt/commit/321715202aed04dd9892d1c0686d080763ab212d)

[1]
[https://www.openssl.org/docs/man1.1.0/man3/OPENSSL_zalloc.ht...](https://www.openssl.org/docs/man1.1.0/man3/OPENSSL_zalloc.html)

[2]
[https://queue.acm.org/detail.cfm?id=2602816](https://queue.acm.org/detail.cfm?id=2602816)

[3] [https://www.ieee-security.org/TC/SP2015/papers-
archived/6949...](https://www.ieee-security.org/TC/SP2015/papers-
archived/6949a003.pdf)

~~~
reza_n
Yup, we added this feature to Varnish Cache a few years ago, random key
encryption. It generates a random key at startup and encrypts all memory with
it. Since this kind of memory is only resident for the lifetime of the
process, it works. We stored the random key in the Linux kernel using the
crypto API [0] just because its not safe storing any kind of keys in a memory
space used for caching (Cloudbleed [1]). We then use the key to generate a per
object HMAC, so each piece of data ends up with its own key, which further
prevents something like Cloudbleed. Since we used kernel crypto, overhead was
about 50%. If you stay completely in user space, its probably much lower.

[0] [https://www.kernel.org/doc/html/v4.17/crypto/userspace-
if.ht...](https://www.kernel.org/doc/html/v4.17/crypto/userspace-if.html)

[1]
[https://en.wikipedia.org/wiki/Cloudbleed](https://en.wikipedia.org/wiki/Cloudbleed)

~~~
thijsvandien
Could this be implemented at the OS level, i.e. whenever a proces launches,
the OS generates a key that it will keep to itself and use to transparently
encrypt all memory allocated by that process?

~~~
reza_n
Possibly, but memory is accessed using plain CPU instructions, so it would be
hard to transparently encrypt all memory for an application at the kernel
level. You do have virtual memory, but I dont think that could be leveraged
for this. But who knows whats possible there, maybe if you align and address
each memory value at the page boundaries and always force a page fault you
could have a really poor implementation :)

Transparent disk encryption, not a problem since devices have filesystems
which can implement encryption at that layer.

~~~
repolfx
Modern Intel chips can encrypt memory on the fly without performance loss (SGX
does this). However I think it's not exposed for non-enclave use. Perhaps it
should be.

Note: inside the enclave there _is_ a performance loss but that's due to MAC
checks. If you just want encryption without integrity against tampering you
don't need that.

------
jasonh3
When I first saw this post, I got excited that someone had finally addressed
the side channel from 17 years ago, which gathered inter-packet timing from
ssh packets (and the fact that passwords are not echo'd) to a) detect when
passwords were being typed, and b) use Hidden Markov Models to extract
keystroke timing to try to extract the likely passwords.

There have been defenses proposed over the years, but none were accepted by
OpenSSH AFAICT.

[https://users.ece.cmu.edu/~vsekar/Teaching/Spring19/18731/re...](https://users.ece.cmu.edu/~vsekar/Teaching/Spring19/18731/reading/SideChannel_Song.pdf)

[https://www.researchgate.net/publication/2907598_Timing_Anal...](https://www.researchgate.net/publication/2907598_Timing_Analysis_of_Keystrokes_and_Timing_Attacks_on_SSH)

~~~
wahern
The proper defense is to not use passwords. It's 2019, nobody should be using
passwords on SSH-enabled shell accounts.

Heck, I use secure tokens for authentication so this memory encryption hack is
useless to me--even _I_ don't know what my private key(s) are, nor does
OpenSSH or any other software on my computers. But I appreciate that we're a
long way from ubiquitous hardware-based authentication.

~~~
gnur
After you are connected you might still want to enter a password (for sudo
perhaps).

~~~
rphlx
The solution there is to have the app use line buffering rather than a "raw"
term mode that exposes inter-char timing on the network. How widely that's
followed in practice, I do not know, but one would certainly hope that sudo
does it.

------
yjftsjthsd-h
Unfortunate that there's no commentary on performance impact. It's symmetric
encryption on a few kB, so probably fast, but I'd like to have numbers.

~~~
dangerface
It's not mentioned because it's irrelevant. There is no security / performance
trade off for a secure program. It's secure or its not.

~~~
simias
There are a lot of knee-jerking in this thread. I don't think anybody is
arguing that this is a bad patch, but if it _does_ have a notable performance
impact (which it probably does not) it's still worth knowing. If only to be
able to answer questions like "why do our backups suddenly take 10% more time
to complete? Do we have a problem with our network infrastructure?"

It's not about tradeoffs, it's about understanding what's going on and
anticipating potential problems.

By your logic we shouldn't have to discuss the performance regressions on CPUs
who implement spectre/meltdown mitigations because "it's irrelevant".
Obviously these patches are necessary but the performance impact is very
relevant for many users.

~~~
archgoon
By the parent's logic, there's no concept of improving security, because if
there remains attacks, then it is still not secure. You can't take a position
of "it's either secure or it isn't."

------
ajaurio
Could password managers like 1Password (AgileBits Inc.) employ similar
techniques? They seem to make very little effort in this regards. E.g. on
1Password,
[https://discussions.agilebits.com/discussion/101551/article-...](https://discussions.agilebits.com/discussion/101551/article-
just-published-in-washington-post-is-saying-1password-and-others-have-
security-flaws).

~~~
eridius
This kind of mitigation really only makes sense on shared machines (such as
servers). On a desktop OS, if an attacker is in a position to read memory from
other processes, it's pretty much game over already.

Browsers implement Spectre/Meltdown mitigation on desktop OSes because without
that, JS could read secrets from other JS contexts executing in the same
process. One of the mitigations is in fact to just segregate JS contexts into
different processes depending on the domain they belong to. But most apps
don't execute untrusted code so most apps don't have this sort of in-process
attacker to worry about.

~~~
sedatk
> On a desktop OS, if an attacker is in a position to read memory from other
> processes, it's pretty much game over already.

Not really. Password managers have access information to your other accounts
in the cloud. Your desktop may not have your most valuable data.

------
rurban
I've looked at it. There's still a spectre window of opportunity to get the
shielded private host keys.

sshkey_shield_private => explicit_bzero() openbsd-compat/freezero.c

[https://github.com/openssh/openssh-
portable/commit/4f7a56d5e...](https://github.com/openssh/openssh-
portable/commit/4f7a56d5e02e3d04ab69eac1213817a7536d0562)

It's only using the insecure freezero, which is using the insecure
explicit_bzero. A simple compiler barrier only, no memory barrier. so it's
unsafe against the advertised spectre/meltdown sidechannel attacks, the
secrets are still in the caches.

------
mehrdadn
> Attackers must recover the entire prekey with high accuracy before they can
> attempt to decrypt the shielded private key, but the current generation of
> attacks have bit error rates that, when applied cumulatively to the entire
> prekey, make this unlikely.

It seems the real mitigation isn't the prekey size, but the temporal
sparseness of the symmetric key -- since I would've imagined attackers would
just try to obtain the symmetric key rather than the prekey. Weird to see they
they didn't even mention this... I imagine attackers would try to find a way
to get the symmetric key to stay in memory for a while.

~~~
the8472
The prekey is hashed into the symmetric key. Both the hash function and the
symmetric cipher have avalanche effects that mean that N bit errors require
the attacker to bruteforce 2^N combinations.

unprotected RSA keys on the other hand have structure and are dense in memory.
That means fewer bit-errors and and the ability to guess the missing bits
faster than O(2^N).

~~~
mehrdadn
Yeah but my concern was about direct leakage not brute force. That's what
Spectre is about.

~~~
the8472
Oh yeah, but the temporal sparseness doesn't just apply to the symmetric
memory encryption key. The most important part is that it also applies to the
asymmetric host keys, which are the actual thing one wants to have protected.

prekey -(hash)-> memory key -(decrypt)-> host private key

------
nabakin
Is this only for OpenBSD's implementation of SSH?

~~~
chasil
If we wait long enough, it will go all the places that the portable release at
openssh.com pushes it.

It will take a few service packs, but even Windows will get it.

    
    
        C:\>ssh -V
        OpenSSH_for_Windows_7.6p1, LibreSSL 2.6.4

------
knorker
Does this only protect the SSH host key?

What about the session keys for the sysadmins who have logged in, and maybe
keep their session up 24/7?

~~~
throw0101a
Session keys are rotated. See "RekeyLimit" in sshd_config(5).

------
foobar_
Take my money!

~~~
gnomewascool
Great!

You can donate to OpenSSH[0], whose "funding is generally done via the same
donation framework" as the rest of OpenBSD, to which you can donate either
directly[1] or via the OpenBSD Foundation[2]. If you're serious about donating
obviously please check that these links are legitimate and I'm not a scammer.
(I'm not affiliated with OpenBSD in any way.)

[0]
[https://www.openssh.com/donations.html](https://www.openssh.com/donations.html)

[1]
[https://www.openbsd.org/donations.html](https://www.openbsd.org/donations.html)

[2]
[https://www.openbsdfoundation.org/donations.html](https://www.openbsdfoundation.org/donations.html)

~~~
foobar_
50$ donated! I hope that's good for some beers :)

On a sidenote paypal donation took way too long. If anyone is interested in
making open source donations simpler feel free to ping me.

~~~
chasil
For your donation, you also received preliminary code for quantum-resistant
key exchange.

    
    
        $ sed -n '/NTRU/,/enabled/p' ChangeLog
        sntrup4591761x25519-sha512@tinyssh.org using the Streamlined NTRU Prime
        4591^761 implementation from SUPERCOP coupled with X25519 as a stop-loss. Not
        enabled by default.

------
vaxman
Krypton Authenticator app anyone? [https://krypt.co/docs/security/privacy-
policy.html](https://krypt.co/docs/security/privacy-policy.html)

------
Klasiaster
It could use the x87 floating point register stack to store the encryption
secret. These registers are not used unless there would be some assembly in
the SSH libs that accesses them.

~~~
Liquid_Fire
Wouldn't they be saved to memory anyway on a context switch?

~~~
Klasiaster
Yes, but I thought this is done in kernel space and a userspace attack has no
access to that while the syscall is served.

~~~
Liquid_Fire
That is true - it's probably more secure than storing them in the process, but
several of these side-channel attacks can apply to kernel space (depending on
hardware and security patches applied).

It's also not portable - OpenSSH runs on non-x86 architectures, and they might
not have spare basically unused registers lying around.

Finally, I'm not sure the x87 registers have enough space to fit these keys.
You have 8 80-bit registers, for a total of 640 bits. Your typical SSH private
key might be 2048 bits or more.

So it's a fun and creative line of thinking, but probably not practical in
this case.

------
sedatk
> Hopefully we can remove this in a few years time when computer architecture
> has become less unsafe.

Narrator: It hasn’t.

------
jnordwick
Couldn't they also move the keys around in memory to every second or keep the
bytes of the key separated (this seems like it would be similar to
encrypting)?

These side channel attacks even under ideal conditions take a very long time
and part of the problem is they basically need to guess at memory addresses.
Even when data is in a known location, it is sketchy. Anything that slows down
locating data would help immensely.

------
captn3m0
Does this support all Private Keys across all operations?

Trying to understand the coverage.

~~~
throw0101a
Host keys only, as those are the only ones that are long-lived.

Session and cipher keys are ephemeral for forward security, and are rekeyed on
a regular basis: see "RekeyLimit" in sshd_config(5).

------
tr33house
Anyone with the actual link to the specific commit?

~~~
mulander
Sure:

[https://github.com/openbsd/src/commit/707316f931b35ef67f1390...](https://github.com/openbsd/src/commit/707316f931b35ef67f1390b2a00386bdd0863568)

~~~
ktpsns

       -  /* $OpenBSD: authfd.c,v 1.113 2018/12/27 23:02:11 djm Exp $ */ 
       +  /* $OpenBSD: authfd.c,v 1.114 2019/06/21 04:21:04 djm Exp $ */ 
    

what weird tool/infrastructure do these guys use, that they maintain CVS/SVN
string identifiers in the files but nevertheless use git?

I actually liked these identifiers. There were tools like ident
([http://manpages.org/ident/1](http://manpages.org/ident/1)) to deal with
them.

~~~
guan
They use CVS, not git. The repository that’s on GitHub is a mirror of the CVS
repository; I don’t know what tool is used to do the mirroring.

Here is the original change in authfd.c in cvsweb:

[https://cvsweb.openbsd.org/cgi-
bin/cvsweb/src/usr.bin/ssh/au...](https://cvsweb.openbsd.org/cgi-
bin/cvsweb/src/usr.bin/ssh/authfd.c?rev=1.114&content-type=text/x-cvsweb-
markup)

------
baq
> Hopefully we can remove this in a few years time when computer architecture
> has become less unsafe.

I think the author misspelled 'decades' here :)

~~~
mtgx
I believe all consumer AMD Ryzen CPUs support Secure Encrypted Memory (SEM),
but for some reason it's not enabled by default on most Ryzen
devices/motherboards. It's a shame.

I'd also love to see AMD bring an updated and patched version of Secure
Encrypted Virtualization (there have been some attacks against it, although
still fewer than against Intel's SGX) to consumer Ryzen in the near future.
With so many cores available in consumer AMD CPUs (up to 16 now), people will
start to use VMs more. Even Windows 10 has the easy-to-use Windows Sandbox
now, as well as the App Guard sandbox for Edge.

Not to mention they could use this as yet another "killer app" of their many-
core CPUs, because otherwise people will eventually start to wonder why even
get CPUs with so many cores over CPUs with fewer cores but higher singlethread
performance. No different than say Verizon promoting high-quality 4k streaming
on its new 5G network.

I would've already preferred to see this in Zen 2, but at least Zen 3, which
will otherwise bring few performance improvements and remain on the 7nm node,
should come with these as some sort of "security-focused generation of Zen".

~~~
chasil
Isn't AMD's encrypted memory meant more for the case of someone with physical
access aggressively cooling a running system, then cutting power and removing
the chilled memory for analysis (which will preserve contents much longer when
cold than at normal temperatures)?

The case of an attack on the SSH Agent would take place within the CPU,
perhaps even in the same core on a separate HT/SMT thread, where the memory
would be cleartext.

