We can confirm that on 2019-07-06 there was a Canonical owned account on GitHub whose credentials were compromised and used to create repositories and issues among other activities. Canonical has removed the compromised account from the Canonical organisation in GitHub and is still investigating the extent of the breach, but there is no indication at this point that any source code or PII was affected.
Furthermore, the Launchpad infrastructure where the Ubuntu distribution is built and maintained is disconnected from GitHub and there is also no indication that it has been affected.
We plan to post a public update after our investigation, audit and remediations are finished.
Thank you, your trust in Canonical is important to us, which is why we make privacy and security a priority.
At least it wasn't as juvenile as the (possible actual) juvenile "hack"[0] of Gentoo's repos that had insertions of "rm -rf /" at the wrong places (where they wouldn't even execute) and insertions of racial slurs into readmes.
I just recall the takeover of the Lubuntu project by this kid, kicking out all others and his attempt to threaten others in the project at Christmas time as a ubuntu tm licensee for his website impostering as a Ubuntu trade mark owner. His forged evidence provided worked with Github staff to get others blocked. I don't trust Ubuntu accounts anymore and I don't trust Canonical because they let it happen.
Launchpad is public, think GitHub but for packages. The Canonical Launchpad repositories weren't tampered. Presumably a lot fewer people have commit access to LP since it's used for package distribution.
Probably not. The vast majority of Ubuntu development occurs on Launchpad. A few important projects are developed on Github, though. Notably, LXC and LXD. The maintainers of these projects are quite active, and would likely have seen any harmful changes being made.
The LXC and LXD repositories are under https://github.com/lxc which is a different Github organization with a much much smaller set of members (17 currently) than CanonicalLtd.
We require 2FA for all accounts under the lxc organization and only grant the access that's actually needed by those contributors. So while it's not impossible that one of our members' credentials may get compromised, especially when considering the use of access token, current access is as restricted as you would expect for those repositories.
It's also worth noting that because of Git's own design, even should one of those accounts get compromised, it would be fairly simple to spot and revert any changes that may have occurred.
I do not see how this will make you safer than running the distro from a disk drive. I believe you are at a higher risk than others because you are running apt update on every boot and thats how you would get infected if launchpad was compromised - but it is not.
While it’s troubling that this happened isn’t it sort of cool that it’s out in the open for everyone to see? At least due to open source the community can know what happened and even look for changes.
I was told this several hours ago on IRC - this is not an official statement:
someuser@somewhere.canonical.com:
I can confirm that we're aware of the issue, have done some initial remediation (e.g. shutting down CI systems that might pull potentially compromised code) while we do a more in depth investigation.
someotheruser@ubuntu/member/username:
And we've of course revoked the access that was abused.
We are lucky that they decided to vandalize instead of hiding something. Remember, on Github you can actually submit your commit with another’s email, and github automatically puts the identity associated with the email allowing people to spoof.
>> We are lucky that they decided to vandalize instead of hiding something.
> We do NOT know this.
Well, at least we know that the attackers wanted their activities to be noticed at this point in time.
I have reasons to suspect that this is a compromised account, in which case there should be enough forensics info to figure out what the scope of the damage is.
Not knowing the details around this, but if they haven't done so, Github could add something to indicate whether particular commits were signed by the PGP key Github knows to belong to the user the commit claims it was authored by. A user could have a setting to enable Github warnings where there are commits supposedly authored by user but not signed. They could also always warn where there are commits authored by a user but signed by a key the user didn't inform Github of.
Likewise, I wonder if git differentiates signatures where the keys are set as trusted in gnupg, and those that are not.
> Not knowing the details around this, but if they haven't done so, Github could add something to indicate whether particular commits were signed by the PGP key Github knows to belong to the user the commit claims it was authored by.
Github could simply reject commits with email addresses not added to your account. As you need to verify every email you add to your account this problem would be solved instantly.
One wouldn't be able to upload an existing open source repo with loads of existing contributors. That seems to really go against Github's mission of being a universal, easy-access hub for open source git repositories.
> Github could simply reject commits with email addresses not added to your account.
Sure, if GitHub wanted to completely abandon the decentralized nature of Git and be a completely centralized system. Which it would love to, I'm sure, but I don't think the user base is quite ready for them to completely shut off any repository with any commits not made by Github users, and posted to GitHub through the account of the user who made the commit, which is what you are suggesting.
and be replaced by a bunch of other problems instead. E.g. no publishing of existing repos on Github if they happen to contain a commit with a non-linked address...
To verify old hashes, with a backup. But anyone with access can just push a new commit with an arbitrary author field. Without code signing it'd be very difficult to tell who actually pushed it. (Probably GitHub has logs you could check, but if the attacker could get on the corporate network it'd just be a NAT IP, and more sophisticated attackers can spoof or MITM)
If they had an SSH key only then signing would be a defense. But if the attackers got OAuth access, stolen credentials, or a personal access token to someone with access to the org, they could do everything through the web interface which is signed by a common Github key.
Isn't commit signing a bit like NuGet package signing? Not worth it unless _everyone_ does it all the time? Any valid commit doesn't necessarily invalidate unsigned commits. It only means this commit is really done by me, but it doesn't me the other commit wasn't done by me.
When you sign git commit, you do not sign your diff but the whole source tree as you see it. Effectively, you're vetting all prior commits on the branch.
Thus maintainers usually sign merge commits and it's their job to confirm that commits don't do anything shady and come from people they claim to come from.
Unfortunately PGP is fundamentally broken. Any identity (email address) can be trivially DoS'd by anyone, because the keyservers are (by design) write-only databases which anyone can add to.
GitHub’s PGP signing feature doesn’t make use of keyservers. You have to explicitly add a PGP key to your account for GitHub to acknowledge that your commits are signed.
They still pretty visibly log the push event with the identity of the user whose credentials were used, so what you're describing is pretty easy to spot.
Are these not just mirrors though or do they actually use git? Woulda thought they used SVN primarily since they pull from Debian? Or do they just pull tarbals... And this all leads to me wondering how they do their process... Anybody got good docs on how Ubuntu or similar distros that base off a parent distro do their work?
First, I would like to challenge the assumption that Debian primarily uses SVN for packaging. We can get a rough idea from the use of Vcs-$name package metadata in the testing distribution:
Which shows that git is the overwhelming favorite and about 40 times more popular than SVN for packages which provide VCS metadata.
The sources can be pulled directly from Debian as tarballs using [syncpackage]. Ubuntu maintainers are free to use their preferred VCS for maintaining Ubuntu-specific packaging. Using the same VCS as the Debian maintainer (or upstream developer) is often convenient, but not required.
That’s true, but somewhat misleading. Though most Debian maintainers use a VCS for packaging, it’s incidental to what actually ends up in the Debian package tree. That’s all still based off of tarballs submitted off of a developer’s machine.
