Zerodium[0], the magic string searched for in the fake user agent, is a zero-day acquisition platform, in case anyone else is wondering. The third parameter to `zend_eval_string` in the commit [1] is the "filename" [2] of the executed code, which is probably used for stack traces. Which is quite a strange choice, because seeing that in the logs would probably ring alarm bells immediately. Overall, this commit seems strange in its structure IMO - not in a "good" covert way, but in a "why does it not try to hide better" way.
EDIT: According to zerodiums charts, this would have been a payout of up to 250k$. However, I'm not quite sure what the attackers plan on cashing out on that would have been, given that the exploit would have quite obviously referenced zerodium and therefore would make the submitter the primary suspect of being the author. Maybe some APT used this "simple" format to disguise the actual power? But then again, it would be in their interest for the exploit to actually go through.
The commits say it was 'sold to zerodium, mid 2017', which would lead me to believe that someone has already been paid for this.
I'd say the reason it doesn't hide better is because it's specifically meant not to hide. It's burning the vulnerability that has apparently existed for almost 4 years.
The vulnerability most people here are talking about was introduced to the PHP codebase last weekend, as part of the block of code containing the "sold... mid 2017" message -- that vuln wasn't there previously, at least in that form. It's not clear what exactly is going on here, but if something was sold to Zerodium in 2017, it wasn't that vuln.
So, the message "sold to zerodium, mid 2017" is there, but may not be true -- it might very well be deliberate misdirection.
I don't see much point in discussing these latest commits. It's very unlikely they were actually malicious in intent (at least to PHP).
The vulnerability that is of interest is the one that allowed these commits to be injected. That is what I figure was sold years ago, and was burned recently.
Problem is, burning the exploit this way only works if someone does the work to find it.
Of course, it could appear in the server logs (or be detailed on a file dumped on the filesystem) - but it seems PHP have just decided to move to Github, and may not actually care about this, so it may not see the light of day.
Sorry, I'm not sure what you mean. The commit was almost certainly never meant to make it in to some server log, it was meant to be seen, and it was. In that way it 'burned' (ie destroyed) the backdoor in to the PHP code repo.
And PHP does seem to care about the real exploit, the one in their infrastructure, which is why they're moving to Github.
Mind expanding on your above comment to help me understand?
I mean exactly what you did - "The vulnerability that is of interest is the one that allowed these commits to be injected".
Yes, it will be evident that the commits appeared in the git history, but not necessarily how they got there. If the git repo on disk suddenly has that commit, and no other usual indicators of compromise, that doesn't really give you anywhere to start looking for, what we agree is, the interesting vulnerability. All you have is "we know this box/service was definitely rooted, so look at all our infrastructure end-to-end".
They care about the exploit enough to move - of course. But do they care enough to investigate and find the actual exploit? Or hand over the server image to someone who does? Unless they've wrapped it in some PHP, it's probably not their core competency.
I don't follow the PHP project, so I don't know the answer to that.
Okay, I see what you mean. I guess I assumed they wouldn't just accept that their server was hacked and not do a through investigation (or have someone do so), but I suppose that's mostly because I couldn't imagine not if it was me.
Are you sure about that? If anyone wanted to demonstrate their access/power, they would probably leave some way to identify themself as the author - with a half-hearted concealment like this, anyone could claim to have been the hacker.
The only way this discovery would help the hacker, is if they intended to force a security audit of the karma system and/or the move to GitHub.
> The only way this discovery would help the hacker, is if they intended to force a security audit of the karma system and/or the move to GitHub.
I agree. Of course I can't be sure of anything here, but to me this commit seems suggestive that there's other injected code somewhere since mid 2017.
You also make a good point on attribution. I had considered that too and if not wanting to take credit could tell us anything about the author.
For example say you were an intelligence agency that knew about this access and knew it was being used by an enemy. Perhaps you couldn't let php know about the vulnerability without exposing that you had access to other data allowing you to know about it. Creating this commit could be a way to share knowledge without it being known where/how it was found.
Meant to be seen with a commit named "fix typo" wtf are you talking about? It's only due to luck that it has been reverted quick enough before being available through binaries.
Maybe the person that reported the exploit to Zerodium already cashed out in mid 2017 and was angry that the exploit hasn't been fixed yet. Therefore, he decided to make it very obvious and force php to react.
That was my assumption given that they explicitly mentioned zerodium. But so far only the commit itself is public, so until we know more about the attack or the attacker, this is pure speculation. The tweet from zerodium is not clear on whether they received an offer for this vulnerability, unfortunately.
> If you are not part of the organization yet, or
don't have access to a repository you should have access to, contact me at
nikic@php.net with your php.net and GitHub account names, as well as the
permissions you're currently missing.
Migrating accounts of PHP core developers by asking them for their username over unencrypted email? What could go wrong? A security incident already occurred, so we should be good on that front for a few years, right?
The whole part of signing commits is providing proof that you are who you say you are with the same level of trust provided by other systems built using Public Key Crypto.
You need people to properly secure their keys of course, but it makes life a lot harder for an attacker.
I sign my work-based git commits. I've been using GPG since the late 90s. It's not a silver bullet (especially when a git server has been owned).
When you git pull, then do your work and git commit/push, if you are not careful and you do not examine git status/diff you could sign and commit the hackers change (that you pulled from the repo initially) along with your new changes. So care is required even with PGP signed commits.
This is another good reason to not interrupt developers while they are working. Doing so, makes mistakes such as this more likely.
Can you walk me a scenario where you and I are hosting our code on a malicious git server and Chuck can get their code in our our codebase? (assuming only signed commits)
You don't really pull down code though, you pull down commits. I'm not seeing a case where you sign the attackers code. You might sign a merge commit that has an attack commit somewhere in the tree, but that's very different and much more easily audited.
Even if we assume the git server is completely malicious, I'm still not seeing a case where an attacker can hide a change because I didn't 'status/diff'. I'm open to considering attacks I haven't thought of though.
I think the problem on GitHub is that if you compromise someone's GitHub account (from a leaked password or whatever), you can just add a new public keypair to the account, and now you can sign code as that user.
Requiring two-factor auth for the GitHub account should be good for this though.
I assume you mean that you are only handling people for whom you can confirm their recent php username AND can verify that the specific username connects to/is owned by the same person as the github account provided?
Just to make sure that someone doesn't go in and find a PHP account with recent commits, then email you with a third party github account.
Isn't there an existing, working PKI for this? "I'm xy and have been signing commits using the key 0xdeadbeef for some time now, signing this message with the same key."
(Yes, this still has holes, but it's a far smaller attack surface than "hello, it's me, trust me because I say so".)
Looking at the commit history [0], it seems that the majority of the commits aren't signed. So no, this would probably not work for identifying everyone.
If the person had a GitHub account before, it could be newly created by the attacker. Or they could have had different e-mails for php.net and GitHub, i.e. something like php@xy.com and github@xy.com.
In a perfect world, this would be fine, yes. But when adding work on top of a probably already very stressful workday, the likelihood of this creating additional problems is there.
> You can say your email is Linus Torvalds’, and GitHub will not question it (and even link to his GitHub profile in the commit history!)
That's the other way around though -- claiming the victims GitHub profile made a commit controlled by the attacker, instead of claiming the attackers GitHub profile made a commit controlled by the victim.
No, you could claim the commits made by the victim this way. The steps would be:
1. Look for a contributor without GitHub account (i.e. xy@gmail.com )
2. Add xy@gmail.com to your GitHub account/create one with that address [0]
2.1 The commits made by the victim will now link to your account on GitHub
3. Send a mail to Nikita with your faked GitHub-profile
None of this will survive a thorough check, but under pressure this will easily pass a surface-level check.
[0] I think you need to confirm your primary E-Mail, but not secondary ones. Alternatively, you could look for commits made from an now deleted mail account.
Yup, AFAICS GitHub associates secondary addresses with your account without verification.
Although the victim would be notified that their address was added to a rogue account, the notification mail says: "If this wasn’t you, please ignore this email." Ugh.
Trying to sneak that diff in was a hugely wasted opportunity, it sticks out like a sore thumb. If they were going to commit anything, it should have looked like a legitimate change. Another option might have been to wait until just before a release and fiddle with the tags or the newly opened QA branch (if any).
I don't see any way they could have snuck anything in unnoticed as a large patch. The number of people who regularly do big patches to PHP is not that large. If you send a big commit as one of them, that person would likely notice and raise an alarm they didn't do it. If you send as somebody who doesn't regularly make big changes, then people would wonder how come this person, who never submitted anything big, suddenly submits a large change without any discussion. So it would have stuck out anyway.
It could work if you do it in some poorly maintained extension, maybe, but that extension probably wouldn't be much used either.
Committing to release branch would be the worst way of doing it, since only RMs commit there, and RMs are those people who are going to notice if somebody commits to their branch and it's not them. Also, since releases are tagged manually, it'd not get into the release anyway, unless you somehow trick the RM into merging the change into their local repo without looking. Which they have no reason to since they created the branch and only they are supposed to commit there. So it'd require some serious trickery.
I think that's a good point. Apparently they had the ability to make commits as any user, which was a huge opportunity. Then they waste it on such an obvious backdoor? It does seem possible that this was a diversion tactic.
Edit: Elsewhere someone mentioned, this could have been "marketing" - to demonstrate their ability to take over accounts.
> this could have been "marketing" - to demonstrate their ability to take over accounts.
Hot take - They burned it, in a very visible way, to prove they've had access since 2017 (or whatever the investigation will reveal). A full security audit isnt very feasible.
Would force everyone to either downgrade and face known vulnerabilities, or do nothing and face high-risk unknown vulnerabilities.
but why would you test that so obviously ('fix typo' but add code)? and they made it clear that they likely had access to the git server (by pushing 'from' different accounts), which (obviously, if that's what is actually happening) can kill their access
Note: I did not keep the original title (Changes to Git commit workflow), because it sounds mundane, rather than being related to an apparent compromise of the git server of a high profile open source project.
Moving to github doesn't change anything fundamentally. It just moves infrastructure hosting to someone else (even though it may or may not be a more secure hosting).
Something like signing commits, and requiring all commits to be signed properly with a known key (and checking this on push and before release) would be an independent verification layer, and a more fundamental response to the compromise.
It moves infrastructure to someone else _whose core job it is to manage this infrastructure_. I definitely trust GH more than myself in managing a git infrastructure securely
That's assuming the self-hosted git server itself was compromised, and it wasn't just a simple matter of the attacker stealing some committer's private keys.
If that's what happened, then moving to Github solves nothing.
> I definitely trust GH more than myself in managing a git infrastructure securely
That's FUD, and appropriate too considering who owns Github now.
Did you notice that the commit was reverted, but then almost immediately re-applied? This is a strong indication that the server admins did not know the how the server was compromised.
With Github, you can: (1) use "events" API to see which account did the push and (2) remove the account's ssh keys and change password from a secure machine to stop compromise right away. This stops the compromise completely.
Compared to that, if there was a compromise on our own server, it is quite hard to tell that everything was cleaned. Remember kernel.org compromise in 2011 [0]? They took a few weeks to recover, and this was with some of the best programmers.
While it's not clear yet whether it was pwned infra or leaked privkey moving to GitHub would definitely lower chances for the former. When SDL moved to github recently I brought this up as one of the advantages when the typical "but M$ suxx" comments started pouring in.
After learning Django for a few months I decided to switch back to php/wordpress yesterday, php7 and php8 look solid and php remains to be the best language for the web in my opinion, Php8 added JIT which might extend itself outside of web applications. Hope moving to github will free the team to do even more on Php development.
I wonder if GitHub will ever support requiring signed pushes. It supports requiring signed commits, but that isn't always feasible, since you could want to merge commits signed by an external contributor without rebasing those commits in order to sign them yourself.
That does seem like a really good idea. If I make a protected branch that requires signed commits I really only care the signatures on the commits you'd see with `git log --first-parent`, as those are the ones by people I trust.
My PHP is poor, but it looks like it was being changed to check for a string coming in via a "typo'd" HTTP_USER_AGENTT header, that started with zerodium, and then eval the string without the zerodium prefix on the server.
Reminds me of that Linux compromise years ago where they picked it up because the change was directly on the build box, except at least that one was hidden behind an apparent `=` vs `==` "bug" - this one has a bit less style.
Maybe this was someone close to Zerodium burning the compromise against PHPs Git server to bring attention to other zerodium commits dating back to 2017?
<!-- git web interface version 2.11.0, (C) 2005-2006, Kay Sievers <kay.sievers@vrfy.org>, Christian Gierke -->
"'Gitweb' is a Git web interface. It is written in Perl and can be used as a CGI script, or as a mod_perl legacy script (run by ModPerl::Registry handler). It allows browsing a git repository (or a set of git repositories) using a web browser." - https://git.wiki.kernel.org/index.php/Gitweb
There's a strong possibility the vulnerability lay in their hosting setup rather than in the software application itself.
There's an argument to be made that self-hosted software that are non-trivial to configure for security have "dev ux" type security bugs, but I doubt that'd constitute a CVE. It could also be unrelated to the git-web hosting config.
Was PHP's Git server using a self hosted Gitlab instance? I can't find the past threads, but I thought there was an announcement a while back that they were moving to Gitlab self-hosted. Maybe I'm misrembering which project that was?
Well it's still vulnerable. At least in Gitlab you can use fancy features like requiring 2FA for members of a group. But you still need to secure its infrastructure.
I think the most common attacks on Github repos and groups have been on individual accounts. Which 2FA would mitigate.
There is a reason why some services are worth it... for one you don't have to worry about all those issues that could pop up. It's someone else's problem.
Feel for the maintainers, tough to keep own stuff together -- but also first reaction is that another massively influential part of the web/opensource being stored on GitHub. I'm not anti-MS by any means but just reminded of the dominance of so much of our global codestores being on there
It's git. That part is easy to migrate. Exporting issues and discussions might take a bit of work, but if/when GitHub starts going downhill there'll be a script to migrate to GitLab (or whatever will be the preferred option by then)
I wonder if the signed features of GitHub would have helped here. Probably can't ask all the 3rd party contributors to setup signing commits, but it would help to spot commits not signed.
If the attackers would have used their own name/email linked to a GitHub account and signed the backdoor commits, GitHub would display a green "Signed" label, and everything would look OK to an outsider.
That's why it's just a gimmick. Signing only works as protection against hosting compromise if the signing/verification is separate from the hosting itself.
Someone responsible for release would have to manually keep a list of keys of authorized comiters and check the repository against this list at the very least prior to a release.
The number of people with direct commit access is almost certainly less than the number of commits in a release though. Without signatures you have to verify every commit.
PHP already uses GitHub and pull requests there, it's just not been the canonical git repository until now. It's not really a big change for most; if anything it's easier.
Of course, there's a vast ecosystem still running shopping carts, forums and whatnot. It's much more prevalent that you seem to think. Just because it's not cool anymore (or well ever) doesn't mean it's dead.
That's a very dated view of PHP. Modern PHP is far "cooler" than it was a few years ago. Just look at the entire https://laravel.com ecosystem — the most starred web framework on all of GitHub, across all languages, is Laravel.
Discovering Laravel a couple of years ago was perhaps one of the most significant things that made PHP interesting again after 20+ years of working with PHP.
It's refreshing (as far as PHP goes) and worth checking out with an open mind. Performance on the other hand, that take a little more work.
Performance keeps getting better, particularly with PHP 7 and 8. And just last week, “Laravel Octane” was announced... it lets you serve a Laravel app using Swoole, and early benchmarks on a MacBook Pro suggest 25k+ HTTP requests served per second!
Laravel Octane is using Swoole which implements a complete webserver, so there won‘t be any php-fpm more. As you would need to restart the server for every code change, there won‘t be a performance difference between docker on mac and non-docker.
Oh, I get that, it was more directed at the OP. It's not your Rust, Nim or Crystal or... which is what I meant about 'cool'. Coolness is always in the eye of the beholder :)
Facebook wrote a new runtime and has, over time, changed the language rather significantly (eg strong static typing, async/await, generics, overhauled much of the standard library.) It's called Hack. At this point, I don't think you could reasonably say facebook's written in php any more than you could say various c++ projects are written in C.
EDIT: According to zerodiums charts, this would have been a payout of up to 250k$. However, I'm not quite sure what the attackers plan on cashing out on that would have been, given that the exploit would have quite obviously referenced zerodium and therefore would make the submitter the primary suspect of being the author. Maybe some APT used this "simple" format to disguise the actual power? But then again, it would be in their interest for the exploit to actually go through.
[0] https://www.zerodium.com/
[1] https://github.com/php/php-src/commit/c730aa26bd52829a49f2ad...
[2] https://bugs.php.net/bug.php?id=78224 (only mention I found with a documented use of the parameter)
[3] https://zerodium.com/program.html