
SSH Key Audit on Github (required) - ericelias
https://github.com/settings/ssh/audit
======
pilif
What makes me the most happy about this is that they ask for the password in
order to add a key now.

I was always very afraid of XSS attacks (I know - there shouldn't be any - but
there could and were, though not for this) that would add another key, so I
always hoped they would add that additional bit of protection.

As such: Another huge thanks to @homakov for forcing the issue.

~~~
natrius
This change is orthogonal to @homakov's attack. Silently changing the user id
on a key wouldn't generate an email.

~~~
ajross
The change is good and was quite clearly precipitated by Homakov. Thus it
seems entirely reasonable to thank him. So I don't see what your point is
beyond irrelevant pedantry.

~~~
orblivion
I don't think it's pedantry. I think it's good to point out that while this
change is good, and is in the same realm, if you give the impression that it
would prevent an attack like the one this weekend, it's security theater.

~~~
pilif
It doesn't and I really hoped that I was clear enough in my initial comment. I
know that this has _nothing_ to do with the attack on Sunday.

But ever since that XSS vulnerability was shown maybe a year ago, I was scared
that there could be another one of them as they are very, very easy to miss,
especially when you have to rely on filtering instead of escaping.

Github has to because you can freely use HTML in Markdown, which is the main
markup language on github.

So there we are: Relying on filtering out bad stuff from user content, instead
of blindly escaping it, at least one XSS vulnerability already happened and
the public key interface still allows adding keys without confirming the
password.

It would really suck to be hit by a XSS attack that silently adds a key to my
account, not only giving the attacker the possibility to impersonate me in
commits, but, even worse, giving access to my private repositories.

Regardless of the fix for the problem on Sunday, I have always hoped Github
would add the check for the password. Not to mitigate the mass-assignment
problem, but to prevent a possible XSS attack from being used to deal much
worse damage.

I see this package of changes that we see now happening at Github as a direct
result of the Sunday hack, so while we got the mass-assignment fix ( _of
course_ that had to be fixed), we _also_ finally got the password recheck
which we likely would not have gotten without the hack.

Hence my "thank you" to @homakov. Not just for uncovering the mass-assignment
thing (which is a simple code fix), but also for forcing a change of policy
(with negative usability repercussions and thus probably not universally
accepted between github internals) elsewhere.

------
memset
Here is the command you use to obtain your fingerprint for this audit:

`ssh-keygen -lf ~/.ssh/id_rsa.pub`

~~~
cschmidt
Thanks for this. I found it kind of weird that the email they sent didn't
include any instructions on what to do. The linked webpage didn't have any
guidance either. I guess they assume we all know what to do.

~~~
charliepark
This. In fact, it creates an even more dangerous situation, as users could go
to the site, see their keys, and say "I dunno. Looks fine?" and approve all of
the keys, without actually confirming that the keys are legitimate.

Not giving instructions _on the page_ on how to verify the info was weak.
Github people, if you're reading this, please update that page.

~~~
julian37
I very much agree they should have added instructions to the page. However
when I went through the process there was a prominent note saying that when in
doubt, you should reject keys and upload new ones. So the "I dunno. Looks
fine?" case seems like it would be a problem only for the careless.

~~~
mikeash
Sort of like how the Rails default settings only caused problems for the
careless....

I bet a lot of people would have verified their keys with instructions but
didn't bother without.

~~~
bricestacey
Disagree for anyone with more than one key. The problem with verifying all
your keys at once is that I'm not going to find all my devices (I don't
practice falconry). It would have been better if you could delay answering for
some keys. I'm not sure you could have, but I didn't feel that way when
performing my audit so I accepted them all, they all had recognizable
hostnames.

~~~
mikeash
It looked like you could put off dealing with keys by just not doing anything
to them. Anything you didn't approve or deny would stick around. However, I
didn't actually test this, and I only had one key which is now approved so
it's too late.

~~~
lotu
That is correct I did exactly that. I got the message at home, and I had a key
for a work computer on there, so I confirmed the home keys and left the work
key disabled.

------
spicyj
The accompanying email:

    
    
      A security vulnerability was recently discovered that made it possible
      for an attacker to add new SSH keys to arbitrary GitHub user accounts.
      This would have provided an attacker with clone/pull access to
      repositories with read permissions, and clone/pull/push access to
      repositories with write permissions. As of 5:53 PM UTC on Sunday,
      March 4th the vulnerability no longer exists.
    
      While no known malicious activity has been reported, we are taking
      additional precautions by forcing an audit of all existing SSH keys.
    
      # Required Action
    
      Since you have one or more SSH keys associated with your GitHub
      account you must visit https://github.com/settings/ssh/audit to
      approve each valid SSH key.
    
      Until you have approved your SSH keys, you will be unable to
      clone/pull/push your repositories over SSH.
    
      # Status
    
      We take security seriously and recognize this never should have
      happened. In addition to a full code audit, we have taken the
      following measures to enhance the security of your account:
    
      - We are forcing an audit of all existing SSH keys
      - Adding a new SSH key will now prompt for your password
      - We will now email you any time a new SSH key is added to your
        account
      - You now have access to a log of account changes in your Account
        Settings page
      Sincerely, The GitHub Team
    
      --- https://github.com support@github.com

------
rdl
Why are ONLY keys at risk, which this implies?

Presumably someone could have added a key, done evil, then removed the key.
Evil includes all sorts of interesting things, like checking in code under the
name of an existing contributor. This could potentially be really subtle and
would be difficult to find in an audit later.

(Remember the stink over OpenBSD potentially having backdoors in the IPsec
stack, revealed in late 2010?
<http://blogs.csoonline.com/1296/an_fbi_backdoor_in_openbsd>)

~~~
archivator
I thought git itself provides authentication and integrity? In fact, if
someone modifies the history of a branch that will raise all sorts of red
flags the next time a legitimate user pushes.

~~~
rdl
Not modifying stuff on disk outside git, but using substituted keys to
masquerade as a legitimate user and contribute evil code. It leaves an audit
trail, but means you need to audit all contributes even those ostensibly from
project owners.

------
andrewjshults
They also did a notification when you tried to push:

ERROR: Hi andrewjshults, it's GitHub. We're doing an SSH key audit. Please
visit
[https://github.com/settings/ssh/audit/<removed>](https://github.com/settings/ssh/audit/<removed>);
to approve this key so we know it's safe. Fingerprint: <removed> fatal: The
remote end hung up unexpectedly

A little weird to see when you're doing a push but good that they put it in
there. Their email got flagged as bulk in gmail so until I saw this I didn't
know they were doing the audit.

~~~
kapitalx
> "good that they put it there"

I think this was handled very badly. This broke alot of automated jobs for
many people. The email actually arrived after some of our automated jobs had
stopped running. There should have been an advanced warning that such an audit
would take place. I'm glad we had monitoring on our automations, but many
people won't have those and might still be unaware that they are not running
until they authorize their keys.

------
finnh
Sadly the link in the email isn't direct (it's a tracking link through
"news.github.com"), so Thunderbird flags it as a possible phishing attempt =(

Edit: github send out an email with a link to the ssh audit page; that's the
email to which I refer

------
pak
As an interesting side effect, they will have pretty exact stats on how many
active users they have; might help them sunset old accounts or move them to
the slowest servers.

(Because of the offline nature of most git actions and different habits on
pushing/pulling, it's probably hard to otherwise estimate how much a user
cares about their github.)

~~~
arohner
Why is this more effective than say "number of push / pulls in the last
month"?

~~~
pak
Some people try to pool local commits into larger, less frequent pushes and
pulls so the number of push/pulls is perhaps less relevant than their
cumulative size. But pushes/pulls will never correspond well with user
involvement because people use github for all kinds of scenarios. For
instance, I might be developing branches that I don't want to push or pull to
github yet--maybe I don't even intend to ever make them public. However, I may
still want people to clone from my github repo and report issues to me.

The amount of time between someone getting an email from github and re-
activating their account is probably the best metric github will ever have on
users' attachment to their accounts.

------
avar
Correct me if I'm wrong but the nature of the vulnerability was that someone
who's _not_ you had to submit a page with certain POST variables they could
have determined after the fact to be malicious while logged in.

So the fact that they're sending out this E-Mail tells us that they either
don't keep logs on requests + POST contents, or that they haven't had the time
or inclination to analyze this data if they have it.

~~~
chrisguitarguy
Or they realize that psychology is just as important to security (and customer
confidence) as logs, analysis, and good code.

Every person that got this email now feels more secure about Github. They
audited their own private keys. They were reminded that they can remove keys
at will. And they know Github has improved its code and given users more power
(email alerts, etc) to be in control of content.

~~~
avar
I for one am feeling way less confident in them with every announcement they
make. They clearly have no idea if and how they were exploited, and their
communication with their users only asks their users to check for one attack
vector, while in actuality the attack was _not_ limited to just adding ssh
keys.

------
spullara
I guess this answers my questions about how long this vulnerability existed (a
long time) and whether or not they could verify no other accounts were
compromised (no).

------
niels_olson
Um, is anybody else having the experience that their keys really do seem to be
different?

~~~
rmc
You should probably tell GitHub

~~~
niels_olson
I thought I deleted this comment. Sorry. Turns out I had an old key still in
my .ssh folder. Two current and valid keys and no others. My bad.

------
jgrahamc
It would be interesting to know the details of the vulnerability. Given that
they've patched it, it would be good to see what the error was in case others
are affected.

Was this Rails-related and what was it?

~~~
chimeracoder
There are a number of articles that surfaced on Sunday/Monday, but in short,
yes - it was a Rails mass-assignment vulnerability.

~~~
oscardelben
I'd like to point out that this is not a rails vulnerability, but a mistake
Github engineers made, which happens to the best of us. Mass assignment is a
feature and I guarantee the problem has been know for years and Github
engineers were probably well aware of it.

~~~
lucian1900
It most certainly is a vulnerability in rails, by encouraging bad practice by
design. Mass assignment should work by defaulting attributes to protected, if
it should exist at all.

------
rwmj
[http://unix.stackexchange.com/questions/2116/given-keys-
in-s...](http://unix.stackexchange.com/questions/2116/given-keys-in-ssh-
authorized-keys-format-can-you-determine-key-strength-easi/2146#2146)

This script is very useful when doing this audit, because you can turn your
.ssh/authorized_keys file into a list of key names and fingerprints to check
against what github is showing you.

------
Ecio78
I've just registered yesterday on Github (it's suggested for Coursera's Saas
Class i'm attending) but they've sent it to me too, even though the
vulnerability has already been resolved _before_ my account was created. Maybe
they've not checked account age..

~~~
sophacles
I suspect that since they only closed the hole late Sunday or early Monday,
that they decided the number of new accounts was so small, it wasn't worth
introducing the complexity (and potential bug/insecurity sources) to handle
such a small percentage of the user-base. A reasonable tradeoff since getting
this audit done in a timely manner is more important than waiting to handle
edge cases such as yours.

------
tomjen3
Why are you guys praising GitHub? They basically screwed up thrice: first by
not catching such an obvious flaw (granted it should have been changed in
Rails, but still), second by breaking half the scripts that rely on their
service and finally by sending such an obnoxious email (really required
action? Who the hell to do you think you are?).

Anyway it is pretty moot at this point since I have long ago forgotten my
password and changing the orgion to somebody else is pretty easy.

That said, can anybody recommend alternatives? I know Bitbucket and they seem
pretty great, especially as they allow private repositories, but it seems the
consensus here doesn't like them for some reason?

~~~
unoti
I love Bitbucket and have been using them exclusively for a long time. I have
many private repos histed there for free. I have several private repos there
shared between a few users, also free. I'm unnerved by how they havent asked
me for money, but grateful. Free jira, wiki, and hipchat, too.

I'm a Mercurial guy, but they do git too.

------
homakov
Is it a good idea to check created_at != updated_at ?

People update public keys very rarely. I would even say NEVER.

Just make an sql against your table to see what are the most possibly are
malicious keys.

(i see no reason to update timestamps doing 'the trick'. I believe attackers
didn't)

~~~
aparadja
The vulnerability probably also allowed the attacker to edit the created_at
and updated_at columns in the ssh key table.

------
joshklein
Several comments below praise the Github team response to this vulnerability.
I agree. But it should also be mentioned that the first email I sent to my
company this morning read, "should [our product] source code be in the cloud?"

~~~
rdl
At the very least, I think it is irresponsible for your product's source code
to ONLY be in the cloud. Luckily git provides an easy way to keep a mirror
(it's kind of automatic), but some kind of regular off-github backup, signing,
etc. would make a lot of sense.

~~~
joshklein
This is a vulnerability where an attacker would be able to add his SSH key to
a private repository and pull proprietary source code he was not authorized to
see. This is why we pay for Github, not use a free account. We don't want
people to be able to walk off with the intellectual property of our company.

That being said, we don't have the resources to deploy a more secure
alternative without hamstringing our development capabilities (e.g. no
internet connectivity).

~~~
rdl
Oh, I agree. I'm just saying the baseline for everyone should be backups and
integrity checking. A lot of companies don't have great value in their code
remaining confidential, a lot do, so that has to be factored in. Other
infrastructure (including runtime environments/hosting) need to be factored
in, too, and there's confidentiality plus a lot of other concerns like
availability.

Why isn't Github Enterprise an option for you? Too expensive? (plus of course
you have to run it; if you don't have a good VPN or premises network, sysadmin
resources to run it, etc., it's entirely possible a self-hosted thing could be
less secure than a SaaS solution)

(The irony of my running a cloud tech startup and not trusting "the cloud" for
our source control, email, file storage, compute, ... is not lost on me. It
definitely adds costs, but I think this is an appropriate level of paranoia.
The providers of business services need to provide convincing arguments why
their services are secure enough to use, at least for b2b.)

~~~
joshklein
You hit the nail on the head. Github Enterprise is too "expensive" in
attention required. While we are security-minded about our proprietary code,
we also recognize that we have a limited budget for "distraction overhead." We
chose infrastructure largely based on how little we have to think about it. In
this case, the distraction cost would be significantly higher than the risk-
weighted cost of IP theft. I still would prefer to minimize that risk, but
without the additional staff and systems you mention, the only reasonable
alternative we have is a local server with no internet connection. Alas,
connectivity is a fundamental requirement.

------
benatkin
It was easier for me to just delete all of the keys. I had some I didn't need
anymore. I also didn't pick great names for the keys I had. It's easy to add a
key so instead of checking the fingerprints I can just create a new key.

------
zby
I've just seen it and I headed to Hacker News to verify if it was legit :)

~~~
oelmekki
Same here, wasn't sure seeing no reference to github.com in mails' header,
even if the link pointed directly on their domain.

i guess that's the precise moment when I should feel _too_ _much_ paranoid.

------
skrebbel
Damn I envy the GitHub guys. They can send a mail to their users about SSH
Keys and nearly all users simply understand it and get it over with.

In any other business, the result of a similar mail would be an overloaded
helpdesk, a significant reputation hit and a massive bucketload of competitor
FUD.

------
levigross
They also added a audit log so you will be able to track and address any
future issues.. <https://github.com/settings/security>

------
homakov
you got balls guys. It is hard to force everyone to do something but you did
it. Kudos

also, if we go back few years ago this way would be a bit secure to handle
keys @key.body = params.. @key.title = params.. I am sure update_attributes is
good choice when you got 5+ fields and update database scheme pretty
frequently. Just my 2 cents

~~~
technoweenie
We sent someone back in time to make this change. Somewhere there's an
alternate universe with a secure GitHub by default. Thanks!

------
my8bird
while this was a good response to their security issue a little heads up would
have been good. they broke all of our auto builds and by the time we figured
it out the guy who's key was used for the builds was gone on his vacation.
luckily, we got ahold of him prior to him turning his phone off.

------
ricardobeat
Did this change just disable re-use of deploy keys across multiple repos?

