Bitbucket dev here. Firstly, thanks for the post, this topic is too often skimmed and as someone who worked directly on the host key rotation here, it was shocking to find how many devs had zero idea what "that prompt from git on a new computer" was.
Secondly, I just wanted to say that there is an interesting side-effect to everything covered in that blog that effectively leads to a gap in most people's mental models (or maybe just mine). Because of the way these systems were defined/the way people interact with them, there are several conundrums when a large centralized host needs to deprecate an old key.
If a key were suspected of being compromised, it seems obvious that you could just stop serving the old key and tell all of your users to be prepared to re-TOFU. However, since _most_ devs blindly type "yes" when interacting with a large git host, this effectively primes the pumps for any bad actor that have MITM control but have not actually stolen the keys. This gives them a relatively long (if not infinitely long) window of time where a user will not be surprised to see "the prompt" and blindly accept trust of a host that could be controlled by the bad actor. If a key was completely successfully compromised by a bad actor, and said bad actor had MITM control of the victim network, then requests with the _old_ key would never actually reach the correct host and just quietly continue working (assuming the bad actor was savy enough to setup a remote system to properly behave as a git host without a prior copy of the target repository).
Damned if you do, damned if you don't, so always carefully TOFU.
Also,not sure why the author linked off to a Bitbucket Cloud blog post for the SSH keys, they are documented here[1] along with our recommended best practices WRT TOFU.
We actually ship a bunch of vetted pubkeys in our developer tools package (in ssh_known_hosts2) . We tried to get a little bit out of the loop and tell our devs that they could use the instructions at https://bitbucket.org/blog/ssh-host-key-changes (confirm it doesn't cert fail, then read the curl https://bitbucket.org/site/ssh bit and see that it also checks certs, so we can leverage "trusted https cert" to "trusted ssh key"... much better than TOFU...
Didn't work, because there's no trailing newline on the output of site/ssh. So even if it works, it corrupts the next addition.
Optimistically, I like to think that people were just not _quite_ bothered enough to file a bug report. The pessimist in me thinks that people went back to blindly typing "yes" when prompted. Either way, thanks for the feedback, the fix should be live!
Much worse performance (which might have been idiosyncratic, or might have been improved in recent years, but repeated authentication vs. single stream design hasn't changed.)
I don't work for them, but: BitBucket changed their SSH keys earlier this year due to possible compromise of their secret keys. [0] I don't think they change them periodically.
sshd also supports online key rotation where it tells clients about new keys, but I think it requires the new key be installed on the server, so it ends up getting compromised at the same time as the existing key.
Nobody ever mentions that TOFU requires the attacker to commit to the attack before they know whether you know the fingerprint. Somebody could set up on that public wifi and intercept ssh, but they'd be detected as soon as anybody with a previously known key tries to connect. They can't wait for you to connect to an unknown server, because they don't know what you don't know. If you're not sure about the network, you can connect to a known host first to check for funny business.
That is correct. And still, many, many devs and admins would do exactly that. Which is why in many, many cases, Tofu really isn't Tofu but "trust whatever."
When designing or evaluating security, one should not ignore that this is a part of reality.
That's a bit of a different issue from the "Let’s say you’re on a fresh machine" that this article is about.
I agree that "automatic trust on first use" is "good enough" for most cases and people (especially with sshfp records), and to be honest I think the warning you get once that fails is strong enough:
% git clone git@github.com:madmurphy/libconfini.git
Cloning into 'libconfini'...
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
Please contact your system administrator.
Add correct host key in /home/martin/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/martin/.ssh/known_hosts:118
Host key for github.com has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
It's a strong warning, with a manual fix that's more than "just press ok" (probably intentionally), and if you choose to ignore that then that's your problem.
I'm not really sure what could be done better? A centralized https-like system comes with its own downsides.
(The only complaint is that last "Please make sure you have the correct access rights and the repository exists" line, which is from git and not ssh, and a tad confusing; maybe it's possible for git to do better here?)
That warning IMO is over the top and too long. All that noise with key fingerprints and filenames makes the eyes glaze over. Any user who knows what a "key fingerprint" is will be able to find and compare them if they are interested. The message should just say something like:
WARNING: Remote host identification has changed.
If you did not expect this, verify this change with the remote system administrator before proceeding.
Ideally the message should be able to be customized so that organizations who provision computers for their employees could include e.g. the phone number or email address for the internal help desk.
Good point. If we're being imaginative, there's various side channels an attacker could use to carefully target their attack, such as seeing a new MAC or DHCP lease on the local network.
True. Also depends on what the target is. I.e., if the attacker wants to impersonate machines that are only avaliable on the local network, then it's probably reasonable to conclude that a machine which was never seen on that LAN before also hasn't yet connected to the impersonated host.
However, that isn't true for "public" SSH endpoints, such as github, gitlab etc - e.g. if the attacker is impersonating the wifi if the nearest Starbucks to snoop on all the hipster solo-devs there, he'll probably be unable to impersonate github, because even people who use that wifi for the first time have probably connected to github before, on a different network.
I had this phase where I tried to do everything the right way. I used to check whenever I thought I could, including to my own servers. And you know what? 100% of the time the check passed.
After a while I sat back and thought to myself "is this really worth the cost in time?" And I concluded that no it wasn't. Plenty of people run scanners looking for bad stuff online, ranging from poisoned SSH to supply chain attacks on OSS. These are the people that find the baddies and the baddies eventually get targeted. Me as a lowly developer is just not spending time fruitfully by checking github.coms SSH fingerprint. It would be better if I scanned the web app that I'm working on for, say, SQL injection bugs or misapplied CORS headers or whatever else.
Ultimately security checks are an economic activity. The likelihood you are targeted via a given threat, the payoff to the attacker if you are targeted, the downside to the attacker if they are caught, etc. I think being realistic is prudent.
I think that's the crux of the problem: we should do this check, and perhaps the first few times we do - but it's a chore every time and infrequent enough that the process has to be rediscovered, with the associated uncertainty that entails (sure I checked - but did I do it properly?)
If it's so important to validate then it shouldn't be left up to a human to choose, it should be mandatory - but the process needs to be straightforward. Why isn't there a built in tool that we can trust to check for us when these alerts happen? Or at least streamline the process and provide certainty
Nothing stopping Apple from working in SSH fingerprint gossipping with an opt-out option for weird corporate people. I already trust Apple to secure my computer and my SSH client I see no downside to protecting this threat vector automatically too.
>Me as a lowly developer is just not spending time fruitfully by checking github.coms SSH fingerprint
... and when you don't you drive up the probability of enabling the 4-stage attack that leaks private keys to Chines govt. Of course the probability is only 0.01% and it's worth to spend time on something better. And when you do check and other 1000 devs in the organization don't, what's the point, right?
This is how this stuff happens and we all read a nice post morten here somewhat 2 years later.
It's implied in the examples on this post, but when using public key authentication, an attacker with a stolen SSH host key can pretend to be the server to the client, but cannot pretend to be the client to the server. Which means that the attacker has to pretend to be the server, without being able to see how the server would answer. For instance, if the ssh connection was a "git pull", the attacker has to know enough about the repository state (and about how much of that state the git client has already seen) to avoid complaints from git; if the ssh connection is a shell connection, the attacker has to know enough about the target machine to make the command outputs seem realistic enough for that particular machine; and so on.
And as for "SSH has no forward secrecy", I thought SSH used Diffie-Hellman or similar to derive the session key? The SSH host key (and the client's public key) is for authentication only, stealing them does not reveal anything about past sessions.
> [the attacker] cannot pretend to be the client to the server
Isn't the most common way of identifying ssh clients still just public key authorization? Meaning the attacker can MITM the real client connecting to the server, satisfy the real client's request, and then likely carry on doing whatever it likes.
edit: actually no - I can see where my intuition is wrong (out of date from the time of widespread password auth). If the ssh server only accepts input that's been effectively signed by the authentication key, then there's no way for the MITM to send its own requests. Yet another reason to not use password auth.
SSH could really use a better default key fingerprint. It's 42 characters of base64. So a hodgepodge of upper and lower case Latin characters with some numbers and special characters mixed in. Not even any spaces to give it some structure. Basically just the output of of a 256 bit hash dumped directly to the terminal.
Perhaps more people would check it if the usability was improved...
$ ssh -o VisualHostKey=yes tty.sdf.org
Host key fingerprint is SHA256:ZjwbO7AU8rHJExYrmZS2LqGZ7WfdoELfMrF54W92PYA
+--[ED25519 256]--+
| ... |
| .oo o |
| .=.* |
| . .* B |
| = o O S. |
|+ + o.o*E=. |
| o o O.++ o |
| o X = +.. o |
| + + +.. . |
+----[SHA256]-----+
user@tty.sdf.org's password:
This graphic looks like a pigeon to me. Similarly, the one for github.com looks like a cat to me. For each server I connect to via SSH I know what the ASCII art for the host key looks like. This familiarity aids in performing TOFU in a hassle-free manner.
I would be interested in how good "looks like a cat to me" would be. It's trivial to generate 100,000 SSH keys and automatically select the most visually similar one. But would you notice the mismatch.
Usability would be improved if there was a one-click way to verify it. Off the top of my head a QR code you could scan so it'd be enough "out of band" that the odds of a compromise affecting both paths would be unlikely. Ideally someone smarter than me would come up with a way that it wouldn't require TOFU at all.
I was using a system at work, and so eventually I needed to ssh into it, and I was confronted with the usual TOFU prompt. So I contacted Support, to ask how I could ascertain the Host Keys in use and check their fingerprints. And they replied, "Mmm, we don't do that."
In the real world, "trust on first use" is literally true - it's never "check on first use." SSHFP DNS records would help a lot, or some other scheme. For all of the positive image ssh has, mostly because telnet and ftp were so ridiculously terrible, ssh-as-used is pretty mediocre.
Personally I always manually check on first use, but I agree this puts me in a small minority.
I said something similar in a comment last year. [0] It's unfortunate that the term TOFU is ambiguous. In practice it's used to refer to check on first use and blindly assume it's ok on first use.
Yeah I've done it. Not for Git but for SSH. I'll call the admin out-of-band and read him part of the fingerprint, and ask him to read another part back. Only done it a few times when I'm setting up a new machine and don't have an already-authenticated one to check against.
If I've got suspicions about the network connection I'm on, I'll first ask him some questions about our adventures years ago, that nobody else would likely know about.
And one time when the host key changed, and I wasn't expecting it, he said my call was how he knew he replaced the right box. Tongue-in-cheek, but only barely.
The only time I immediately dismiss the key-changed message is when I'm cycling yet another raspberry pi image into the same IP as an old one or whatever, and I'm on the local network so I know exactly what's going on.
If I have to connect to a new machine, I identify it by its hostname, and send it my credentials (whatever credentials the owner just sent me in plaintext).
If I get the warning and I'm pretty sure I connected to this machine before, I ask whoever owns the machine "hey did you do something weird with this machine? Like reimage it or rotate SSL certs or change the DNS entry so it's actually a different machine?"
99% of the time I'm not connecting to a machine for the first time, so (if I may be allowed some bad maths) this reduces my attack surface by a factor of 100.
(If the owner didn't send me plaintext credentials but instead added my SSH key, I don't expose anything by using that, but I immediately follow up by doing other sensitive things like scp'ing code or credentials there).
We've switched to them based on Hashicorp Vault and I think they are an improvement in security and convenience.
Before SSH host certificates, using tooling like ansible was seriously annoying, especially to the monthly ansible users, compared to the daily ansible users. If you tried using ansible once a month without our host key certificates, you'd have ansible barf about a dozen unknown host keys due to them being rebuilt and other things. Then you fix that, then they do that one ansible run, then it lies around for a month and back you go to square one.
I'm not sure HOW you'd stage an actual attack there, but people got used to just accept SSH host keys whatsoever, so the vulnerability was there.
Now we have host key signing based on Vault. This is a huge improvement in my book. Base infra guys know when they've reset a VM to a base image (which resets SSH host keys), so they know when to expect a TOFU request from their SSH upon first connect to the just rebuilt system, so they accept those. Afterwards, one of the first things the config management does is to sign those host keys and then the accepted host keys are usually deleted again.
There probably is a window of vulnerability in there, but that's getting pretty hard to attack and I'm sure there are easier ones in the overall infrastructure.
The main problem at this however is that you need some safe and secure place for the CA, and ideally automation to sign those host keys. And it needs to be enough of a problem to do all of that.
MITM for GitHub or another code forge is one of the things I'm least concerned about, because GitHub to me is a place where things go to be made public. One of the security principles I live by is not to upload anything to the Internet that I wouldn't mind being shared completely in public.
The only exceptions I make are video conferencing and email. Jitsi provides great end-to-end encryption for the former (although the 8x8 server's recent prohibition of anonymous use is disappointing).
> MITM for GitHub or another code forge is one of the things I'm least concerned about, because GitHub to me is a place where things go to be made public.
This is not just about secrecy, but also about integrity. Someone who MITMs your connection could not only look at the code you published, but also modify it. Then later someone who was trusting you (or possibly even yourself on a new machine) will end up downloading and running the modified code.
In this particular case, I'm not sure it would be possible to modify the code in transit, because although you could trick the user into submitting the code into Fake GitHub, you wouldn't be able to authenticate as the user to submit your modified copy to Real GitHub.
More importantly, in my opinion, the end-user has no way of verifying the integrity of the code if authentication was the only security measure. By signing releases or the commits themselves (which can be done with both PGP and SSH keys currently) the end-user can verify the integrity themselves.
Same. I presume every company I have an account on will be compromised at some point, and all my account data will be out in the wild. This presumption hasn't disappointed so far ;)
I tried to check but 99% of our clients rotate host keys whenever they restart the container or the LB sends you to a different node or whatever. And if you email them to verify they take 3 weeks to respond, which is 20 daily reports they now don't get and so don't want to pay for, so... nope.
The last time I was jump-starting an internal wiki, for a place that had no docs... when I was documenting how to set up a workstation for checking out and running the code from Git SSH... it turned out that no one seemed to document or know what the fingerprint should be. Not even any documentation I could find from the major Git provider.
I can understand not expecting every user to do a one-time 10-second diligence process. But the "enterprise" provider seemingly not expecting any of its customers' IT people to even do a one-time 10-second check when integrating mission-critical infrastructure... is a concern.
I think the attack in all cases would require pointing DNS of gitprovider.net to their own server. This means they control DNS on that network, which means they're running it.
What am I doing on a network I don't trust with a machine I've never used? The example given is that I've started a new job. Well okay, their machine, their network, and I guess their corporate gitprovider.net account they're getting me to use, so to heck with it.
If we're talking about using _my_ gitprovider.net account however, that's a different story. I simply wouldn't use my own account on a laptop that's not mine.
> "I wonder if SSH should really be used at all for git. While TLS certificates are relatively short lived these days thanks to Let’s Encrypt, SSH host keys live a very long time. If GitHub’s SSH host keys were stolen, how long would an attacker get away with intercepting communications?"
The solution is to use SSH Certificate Authentication as opposed to SSH Key Authentication. The Certificate has a forced expiry and is verified by a CA. Then interception is rendered a moot problem since keys are verified with the CA before use.
No, often I don't check, because my managers rush me to produce results on tight timelines, so I don't have time to "do things right".
Slack message at 3pm asking for updates to be in a doc by 6pm. What do you expect?
I guess the problem is less serious on a corporate VPN, but still, the principle applies everywhere. If you want security, give your employees time to do things right. Usually that means at least double what you think it will take.
Only once. ssh needs to make the fingerprint shown in the terminal match what is stored in known_hosts. If I already have public keys why can't I match them against the new connection? I could preload the file but why not copy it from my desktop's file into the terminal.
Secondly, I just wanted to say that there is an interesting side-effect to everything covered in that blog that effectively leads to a gap in most people's mental models (or maybe just mine). Because of the way these systems were defined/the way people interact with them, there are several conundrums when a large centralized host needs to deprecate an old key.
If a key were suspected of being compromised, it seems obvious that you could just stop serving the old key and tell all of your users to be prepared to re-TOFU. However, since _most_ devs blindly type "yes" when interacting with a large git host, this effectively primes the pumps for any bad actor that have MITM control but have not actually stolen the keys. This gives them a relatively long (if not infinitely long) window of time where a user will not be surprised to see "the prompt" and blindly accept trust of a host that could be controlled by the bad actor. If a key was completely successfully compromised by a bad actor, and said bad actor had MITM control of the victim network, then requests with the _old_ key would never actually reach the correct host and just quietly continue working (assuming the bad actor was savy enough to setup a remote system to properly behave as a git host without a prior copy of the target repository).
Damned if you do, damned if you don't, so always carefully TOFU.
Also,not sure why the author linked off to a Bitbucket Cloud blog post for the SSH keys, they are documented here[1] along with our recommended best practices WRT TOFU.
1 - https://support.atlassian.com/bitbucket-cloud/docs/configure...