
GitHub mirror compromise incident report - amaccuish
https://wiki.gentoo.org/wiki/Github/2018-06-28
======
rich-and-poor
He should have sat on the password. He should have watched for PRs, and
started pushing updates immediately after they had received approval, and then
merging. Instead, he panicked and kicked out all the maintainers, who realized
the intrusion only 10 minutes after he gained access. And all he did was add
`rm -rf /*` to build scripts, and the N word to the readme.

The malicious commits:

[https://github.com/gentoo/gentoo/commit/e6db0eb4](https://github.com/gentoo/gentoo/commit/e6db0eb4)

[https://github.com/gentoo/gentoo/commit/afcdc03b](https://github.com/gentoo/gentoo/commit/afcdc03b)

[https://github.com/gentoo/gentoo/commit/49464b73](https://github.com/gentoo/gentoo/commit/49464b73)

[https://github.com/gentoo/gentoo/commit/fdd8da2e](https://github.com/gentoo/gentoo/commit/fdd8da2e)

[https://github.com/gentoo/gentoo/commit/e6db0eb4](https://github.com/gentoo/gentoo/commit/e6db0eb4)

[https://github.com/gentoo/gentoo/commit/c46d8bbf](https://github.com/gentoo/gentoo/commit/c46d8bbf)

[https://github.com/gentoo/gentoo/commit/50e3544d](https://github.com/gentoo/gentoo/commit/50e3544d)

~~~
Buetol
Forcing any commit to be a PR merged by another dev would have solved it

~~~
rich-and-poor
Do you know of any companies that do that? That seems like a big hassle.

~~~
kace91
For mine every feat/fix/refactor is a new branch which then requires 2
approvals to get merged, among other things (style guide enforced as linting,
minimum test coverage threshold, etc ).

Tbh it's cool, coming from a previous job at a startup where version control
meant just zipping the project from time to time.

~~~
ff_
Same here. All code goes in a Pull Request, and requires at least 1 approval.

Since Pull Request review is a high priority activity there's no "bottleneck",
and you double the bus factor for free + prevent bad things from happening.

------
akerl_
Something to note here: "Evidence collected suggests a password scheme where
disclosure on one site made it easy to guess passwords for unrelated
webpages."

For any folks out there who've been using or promoting formula-based
passwords, this is the potential impact: a leak on one site can be leveraged
by an attacker towards other sites you use.

~~~
simias
I suppose it's only as good as the formula itself and at least it protects you
from automated attacks. Still, I wouldn't advise anybody from going with
formula-based passwords, just use whichever password manager you prefer and
use unique strong passwords per-website.

Maybe one day the web will get its shit together and let us login everywhere
with a certificate that we'll be able to securely store on a HSM and easily
revoke and update if necessary but in the meantime it's the best we can do.

~~~
berkes
I'd like to add "truly random" to

> use unique strong passwords per-website.

... use unique, truly random passwords per-website.

"Strong" is ambigous and hard to explain how to do right. Random means, in
practice: "don't come up with one yourself, but let a tool generate one for
you".

~~~
ChrisSD
To nitpick slightly, I think "truly random" is the wrong phrase here.
Computers can't generate such numbers (only pseudo-random). What they can do
is generate unpredictable passwords. Unpredictability is the feature we want
in a password, whether that happens through "randomness" or something else
doesn't matter.

Of course computers are far better at coming up with unpredictable passwords
than humans are.

~~~
zokier
RDRAND is supposed to generate very true random numbers

~~~
ChrisSD
Well either a RNG generates true random numbers or it doesn't. There's no room
for "very" here. From Intel[0]

> With respect to the RNG taxonomy discussed above, the DRNG follows the
> cascade construction RNG model, using a processor resident entropy source to
> repeatedly seed a hardware-implemented CSPRNG. Unlike software approaches,
> it includes a high-quality entropy source implementation that can be sampled
> quickly to repeatedly seed the CSPRNG with high-quality entropy.

So it's a cryptographically secure pseudo random number generator that takes
entropy from the processor. It's not a True Random Number Generator. And
again, if it does work well for cryptography, it's the unpredictability that
matters, not the randomness itself.

[0] [https://software.intel.com/en-us/articles/intel-digital-
rand...](https://software.intel.com/en-us/articles/intel-digital-random-
number-generator-drng-software-implementation-guide)

~~~
zokier
From the link you provided:

> This method of digital random number generation is unique in its approach to
> _true random number generation_ in that it is implemented in the processor’s
> hardware

> The all-digital Entropy Source (ES), also known as a _non-deterministic
> random bit generator (NRBG)_ , provides a serial stream of entropic data in
> the form of zeroes and ones.

> The ES runs asynchronously on a self-timed circuit and _uses thermal noise
> within the silicon_ to output a random stream of bits at the rate of 3 GHz

What on earth would classify as TRNG if not above?

Just fyi, checked, and NIST SP 800-90B defines NRBG as following:

> Non-deterministic Random Bit Generator (NRBG):

> An RBG that always has access to an entropy source and (when working
> properly) produces outputs that have full entropy (see SP 800-90C). Also
> called a _true random bit (or number) generator_ (Contrast with a DRBG).

------
gkoberger
To be clear, GitHub itself was NOT compromised. The password of a Gentoo
contributor, who presumably wasn't using 2FA, was.

The title should probably clarify it's a "Gentoo mirror on GitHub compromise
incident report"

~~~
SquareWheel
Glad I clicked through. When I scanned the Hacker News titles earlier, I'd
assumed Github had been compromised. Very misleading.

------
snowwolf
As they have now done, they could have reduced the threat exposure by
requiring 2FA to join the organisation, and using a password pattern scheme
does expose you to a targeted attack so ideally encouraging password manager
generated passwords is probably also recommended.

But Github could probably have helped by detecting logins from unusual IPs for
that user - i.e. a login attempt from an IP they haven't logged in from
before, and required something like email verification too. Although if they
were using an easy to guess pattern, then likely the admins email could have
been compromised too.

Edit: GitHub could also warn people whose accounts have admin access to
organisations if they don't have 2FA enabled.

------
noobermin
Small piece of annoyance, the clickbait press had a field day with this. From
the positive which I appreciate[0] to claiming putting `rm /*` in the wrong
place so that it didn't even work is "file wiping malware."[1]

For the most part, this appeared to the work of a teenage skiddy (given the
addition of a readme with a racial slur as the text), and not any actual
sophisticated attack.

[0] [https://www.techrepublic.com/article/gentoo-stops-github-
rep...](https://www.techrepublic.com/article/gentoo-stops-github-repo-hack-in-
an-hour-setting-standard-for-security-response/)

[1] [https://www.bleepingcomputer.com/news/linux/file-wiping-
malw...](https://www.bleepingcomputer.com/news/linux/file-wiping-malware-
placed-inside-gentoo-linux-code-after-github-account-hack/)

------
driverdan
Always require 2FA for something as important as your code base.

~~~
berkes
While true, many apps, including github, make this problematic for something
like an admin account.

2FA, by it's nature, is bound to one single software, tool, or piece of
hardware[1].

This limits the access of e.g. an administrator-login to one person and her
personal phone, only. A bus-factor that is unacceptable to many.

Software like AWS allows more granular set-up, but still complex. Linode,
Digital-Ocean and even docker.io, last time I looked, make it impossible to
share the admin-account by allowing multiple 2FA devices active on one account
simultaneous. And if they did, that would greatly lower the security of that
account (still better than no 2fa though)

[1] 2FA, like google authenticator (or one of the much better open source
alternatives thereof) make it possible to share a 2fa secret across devices,
but that is both insecure and hard.

~~~
snowwolf
Github allows multiple administrator accounts.

------
petters
Github supports U2F (two-factor auth). Please use it.

~~~
fulafel
Github's SMS or TOTP requirement for u2f use is a shame - why would you use
u2f if you have already set up these? And if you want to use u2f because you
don't trust your phone as a security device (hello Android users) - not good.
Or maybe you don't want to mix your personal phone with work it infra, it's a
bad equation there too.

I guess you could still use a software TOTP implementation on your workstation
to fool Github, but then you are not getting the additional security from u2f
because the totp codes are a substitute for the u2f token.

~~~
leni536
What if you just throw away the TOTP secret right after you get it and verify
it on Github? You don't have to use your phone either. Just because you used
TOTP to set up u2f on Github doesn't mean you have to store the TOTP secret
indefinitely.

~~~
fulafel
Yes, I guess you could navigate it this way. But clearly this is not something
Github wants users to do, so I wonder if this way is bad somehow. Maybe there
is no backup mechanism to recover from u2f token loss other than the old 2fa
mechanisms, for example.

------
badrabbit
Are there any generic incident response plans or playbooks for foss projects?
Is gentoo's documented anywhere?

Gotta say,a bit impressed by the response.

~~~
throwawayqpth43
I don't normally post on hackernews (so I've made a throwaway). I'm Antarus
from the report.

We don't have a plan for Gentoo. I work for Google and I mostly used a vaguely
similar plan to the Google incident plan.

1) Communicate early. For publically visible stuff (defacement was very
obvious) you want to get a message out quickly before a natural narrative
forms.

2) Communicate often. 3) Mitigate the problem first (e.g. prevent the
malicious stuff from being downloaded) then investigate second.

4) Assign roles to people and be clear who is responsible for what. 5) Collect
lots of data.

------
zenexer
I'd sleep better at night if every organization were this transparent.

------
Belphemur
Mirror is the important word here.

I'm pretty sure it didn't impact anybody.

~~~
akerl_
From the incident report it sounds like the gentoo/systemd GitHub repo is
__not __a mirror: the official repo is on GitHub and they 're now looking to
mirror it onto Gentoo infrastructure.

Does anybody know why this is the case?

~~~
Semaphor
> The main Gentoo repositories are kept on Gentoo hosted infrastructure and
> Gentoo mirrors to GitHub in order to "be where the contributors are."

Doesn't sound to me as if it's not a mirror.

Though I guess you could say as their CI depended on this mirror, it had a
higher status than _normal_ mirrors.

~~~
akerl_
I'm specifically talking about
[https://github.com/gentoo/systemd](https://github.com/gentoo/systemd)

From their incident report (under
[https://wiki.gentoo.org/wiki/Github/2018-06-28#What_went_bad...](https://wiki.gentoo.org/wiki/Github/2018-06-28#What_went_badly)
): "The systemd repo is not mirrored from Gentoo, but is stored directly on
GitHub."

And from their Action Items: "mirror systemd repo on git.gentoo.org"

I'm curious why they seem to treat this repo differently from the others,
using GitHub as authoritative and adding a mirror to git.gentoo.org rather
than making git.gentoo.org authoritative and mirroring to GitHub.

------
taf2
Github supports two factor authentication, maybe make it mandatory for all
admins?

------
jrochkind1
> Clearer guidelines that even if users got bad copies of data with malicious
> commits, that the malicious commits would not execute.

Eh, that sees unwise. To suggest based on what you think you are "sure" about
security protections, that users should not worry about having a copy with
malicious code.

------
Thaxll
Not enabling MFA is just unacceptable at that level...

------
imdsm
No 2FA?

------
LeoPanthera
Actual link to full report:

[https://wiki.gentoo.org/wiki/Github/2018-06-28](https://wiki.gentoo.org/wiki/Github/2018-06-28)

~~~
sctb
Thanks, we've updated the link from
[https://lwn.net/Articles/759046/](https://lwn.net/Articles/759046/) which
points here.

------
PretzelFisch
what mitigations do you use to protect systems that automatically updated
after every git push?

~~~
badrabbit
Don't auto-update production like that? If a change will affect all
users,manually review it and as another commenter said,gpg signatures. Have
every prod commiter use mfa too.

~~~
PretzelFisch
I have seen a lot of people describe their continuous delivery where the code
is checked in and moves into production after passing tests. I wasn't sure if
they had a way to deal with this type of issue or I am misunderstanding their
process somehow. I

