Hacker News new | past | comments | ask | show | jobs | submit login
PHP's Git server compromised, moving to GitHub (php.net)
405 points by jsmith45 on March 29, 2021 | hide | past | favorite | 135 comments

Zerodium[0], the magic string searched for in the fake user agent, is a zero-day acquisition platform, in case anyone else is wondering. The third parameter to `zend_eval_string` in the commit [1] is the "filename" [2] of the executed code, which is probably used for stack traces. Which is quite a strange choice, because seeing that in the logs would probably ring alarm bells immediately. Overall, this commit seems strange in its structure IMO - not in a "good" covert way, but in a "why does it not try to hide better" way.

EDIT: According to zerodiums charts, this would have been a payout of up to 250k$. However, I'm not quite sure what the attackers plan on cashing out on that would have been, given that the exploit would have quite obviously referenced zerodium and therefore would make the submitter the primary suspect of being the author. Maybe some APT used this "simple" format to disguise the actual power? But then again, it would be in their interest for the exploit to actually go through.

[0] https://www.zerodium.com/

[1] https://github.com/php/php-src/commit/c730aa26bd52829a49f2ad...

[2] https://bugs.php.net/bug.php?id=78224 (only mention I found with a documented use of the parameter)

[3] https://zerodium.com/program.html

The commits say it was 'sold to zerodium, mid 2017', which would lead me to believe that someone has already been paid for this.

I'd say the reason it doesn't hide better is because it's specifically meant not to hide. It's burning the vulnerability that has apparently existed for almost 4 years.

The vulnerability most people here are talking about was introduced to the PHP codebase last weekend, as part of the block of code containing the "sold... mid 2017" message -- that vuln wasn't there previously, at least in that form. It's not clear what exactly is going on here, but if something was sold to Zerodium in 2017, it wasn't that vuln.

So, the message "sold to zerodium, mid 2017" is there, but may not be true -- it might very well be deliberate misdirection.

I don't see much point in discussing these latest commits. It's very unlikely they were actually malicious in intent (at least to PHP).

The vulnerability that is of interest is the one that allowed these commits to be injected. That is what I figure was sold years ago, and was burned recently.

Problem is, burning the exploit this way only works if someone does the work to find it.

Of course, it could appear in the server logs (or be detailed on a file dumped on the filesystem) - but it seems PHP have just decided to move to Github, and may not actually care about this, so it may not see the light of day.

Sorry, I'm not sure what you mean. The commit was almost certainly never meant to make it in to some server log, it was meant to be seen, and it was. In that way it 'burned' (ie destroyed) the backdoor in to the PHP code repo.

And PHP does seem to care about the real exploit, the one in their infrastructure, which is why they're moving to Github.

Mind expanding on your above comment to help me understand?

I mean exactly what you did - "The vulnerability that is of interest is the one that allowed these commits to be injected".

Yes, it will be evident that the commits appeared in the git history, but not necessarily how they got there. If the git repo on disk suddenly has that commit, and no other usual indicators of compromise, that doesn't really give you anywhere to start looking for, what we agree is, the interesting vulnerability. All you have is "we know this box/service was definitely rooted, so look at all our infrastructure end-to-end".

They care about the exploit enough to move - of course. But do they care enough to investigate and find the actual exploit? Or hand over the server image to someone who does? Unless they've wrapped it in some PHP, it's probably not their core competency. I don't follow the PHP project, so I don't know the answer to that.

Okay, I see what you mean. I guess I assumed they wouldn't just accept that their server was hacked and not do a through investigation (or have someone do so), but I suppose that's mostly because I couldn't imagine not if it was me.

> it was meant to be seen, and it was

Are you sure about that? If anyone wanted to demonstrate their access/power, they would probably leave some way to identify themself as the author - with a half-hearted concealment like this, anyone could claim to have been the hacker.

The only way this discovery would help the hacker, is if they intended to force a security audit of the karma system and/or the move to GitHub.

> The only way this discovery would help the hacker, is if they intended to force a security audit of the karma system and/or the move to GitHub.

I agree. Of course I can't be sure of anything here, but to me this commit seems suggestive that there's other injected code somewhere since mid 2017.

You also make a good point on attribution. I had considered that too and if not wanting to take credit could tell us anything about the author.

For example say you were an intelligence agency that knew about this access and knew it was being used by an enemy. Perhaps you couldn't let php know about the vulnerability without exposing that you had access to other data allowing you to know about it. Creating this commit could be a way to share knowledge without it being known where/how it was found.

Of course that's all wild speculation.

Meant to be seen with a commit named "fix typo" wtf are you talking about? It's only due to luck that it has been reverted quick enough before being available through binaries.

Perhaps the vulnerability in the git server that allowed this is what was sold, rather than the commit.

Zerodium say they have nothing to do with this: https://twitter.com/cBekrar/status/1376469666084757506

> Zerodium say they have nothing to do with this

Would they admit if they did?

"zerodium investigated zerodium and zerodium found out that zerodium had nothing to do with this"

Well the opposite may be true too - the comment may be lying about it having anything to do with zerodium or that date.

Could be just a meme or a bad attempt of misdirection in case it went undetected for a while.

That's unknown, but it would be strange for them to be involved - it would not make much sense for them to pay out a bounty to themselves.

I think it's far more likely that someone tried to plant this there in order to cash out via zerodium.

Maybe the person that reported the exploit to Zerodium already cashed out in mid 2017 and was angry that the exploit hasn't been fixed yet. Therefore, he decided to make it very obvious and force php to react.

I believe the exploit itself was introduced with the commit in question, it wasn't there before.

As has been pointed out elsewhere, the exploit is the ability to inject code into the PHP Git repository. Not this RCE.

The other exploit, the one that allowed the commit to be injected.

am I understanding it correctly that somebody had breached the PHP git server and tried selling it to 0-day and similar exploiting businesses?

That was my assumption given that they explicitly mentioned zerodium. But so far only the commit itself is public, so until we know more about the attack or the attacker, this is pure speculation. The tweet from zerodium is not clear on whether they received an offer for this vulnerability, unfortunately.

>Likely, the researcher(s) who found this bug/exploit tried to sell it to many entities but none wanted to buy this crap

but why so snarky?

twitter. the takes are hotter than the sun, the style is snarkier than zkSNARKs.

> If you are not part of the organization yet, or don't have access to a repository you should have access to, contact me at nikic@php.net with your php.net and GitHub account names, as well as the permissions you're currently missing.

Migrating accounts of PHP core developers by asking them for their username over unencrypted email? What could go wrong? A security incident already occurred, so we should be good on that front for a few years, right?

For now I'm only handling people for which I recognize the account or can confirm recent commits.

Are you requiring signed commits? If you're not, I strongly suggest that you do in the future.

I've been pointed out once that only tags should be signed - and referred to this: [1] https://news.ycombinator.com/item?id=12290873

The whole part of signing commits is providing proof that you are who you say you are with the same level of trust provided by other systems built using Public Key Crypto.

You need people to properly secure their keys of course, but it makes life a lot harder for an attacker.

I sign my work-based git commits. I've been using GPG since the late 90s. It's not a silver bullet (especially when a git server has been owned).

When you git pull, then do your work and git commit/push, if you are not careful and you do not examine git status/diff you could sign and commit the hackers change (that you pulled from the repo initially) along with your new changes. So care is required even with PGP signed commits.

This is another good reason to not interrupt developers while they are working. Doing so, makes mistakes such as this more likely.

Can you walk me a scenario where you and I are hosting our code on a malicious git server and Chuck can get their code in our our codebase? (assuming only signed commits)


> if you are not careful and you do not examine git status/diff you could sign and commit the hackers change

I assume he meant that you pull, get an auto merge and then inattentively sign it. With this, you confirm the hackers change with your signature.

You could add a push hook to only allow fully signed chains, but given that we are talking about a compromised server, this does not help.

You don't really pull down code though, you pull down commits. I'm not seeing a case where you sign the attackers code. You might sign a merge commit that has an attack commit somewhere in the tree, but that's very different and much more easily audited.

Even if we assume the git server is completely malicious, I'm still not seeing a case where an attacker can hide a change because I didn't 'status/diff'. I'm open to considering attacks I haven't thought of though.

But how does the hacker's commit goes through in the first place, if all PRs with unsigned commits are rejected outright?

If he hacked the server (which was assumed in the grand parent), he can probably shut off that check rather easily

And then you have a singled commit without a signature lying there for everyone to see and call attention to you.

I think the problem on GitHub is that if you compromise someone's GitHub account (from a leaked password or whatever), you can just add a new public keypair to the account, and now you can sign code as that user.

Requiring two-factor auth for the GitHub account should be good for this though.

I assume you mean that you are only handling people for whom you can confirm their recent php username AND can verify that the specific username connects to/is owned by the same person as the github account provided?

Just to make sure that someone doesn't go in and find a PHP account with recent commits, then email you with a third party github account.

For most contributors we know GitHub identities, as PHP was using pull requests etc already from GitHub.

Aah, k, makes sense :)

The notice starts out by saying:

> Yesterday (2021-03-28) two malicious commits were pushed to the php-src repo [1] from the names of Rasmus Lerdorf and myself.

…and ends by saying:

> We're reviewing the repositories for any corruption beyond the two referenced commits. Please contact security@php.net if you notice anything.

Are you sure that using recent commits as confirmation is a good idea?

There will be increased scrutiny on this repo anyway. It's not necessarily the safest, but probably a low-risk and easy way to migrate committers.

Can you provide more details about the breach? Did they get in via munin?

Are github usernames especially sensitive?

"I'm xy and had push access to the php core git server, but I do not yet have access to the PHP organization. Could you add @xyfake to it? Thanks!"

Combine that with a spoofed/typo'd sender and you're good to go.

Isn't there an existing, working PKI for this? "I'm xy and have been signing commits using the key 0xdeadbeef for some time now, signing this message with the same key."

(Yes, this still has holes, but it's a far smaller attack surface than "hello, it's me, trust me because I say so".)

Looking at the commit history [0], it seems that the majority of the commits aren't signed. So no, this would probably not work for identifying everyone.

[0] https://github.com/php/php-src/commits/master

You can mitigate that pretty easily by checking whether recent commits from this person are linked to that GitHub account.

If the person had a GitHub account before, it could be newly created by the attacker. Or they could have had different e-mails for php.net and GitHub, i.e. something like php@xy.com and github@xy.com.

In a perfect world, this would be fine, yes. But when adding work on top of a probably already very stressful workday, the likelihood of this creating additional problems is there.

> If the person had a GitHub account before, it could be newly created by the attacker

Doesn't GitHub require e-mail validation before it associates commits with that e-mail address?

> Or they could have had different e-mails for php.net and GitHub, i.e. something like php@xy.com and github@xy.com.

Then require them to add their php.net mail address to their GitHub account.

> the likelihood of this creating additional problems is there.

That I agree with.

> Doesn't GitHub require e-mail validation before it associates commits with that e-mail address?

IIRC, nope! You can say your email is Linus Torvalds’, and GitHub will not question it (and even link to his GitHub profile in the commit history!)

> You can say your email is Linus Torvalds’, and GitHub will not question it (and even link to his GitHub profile in the commit history!)

That's the other way around though -- claiming the victims GitHub profile made a commit controlled by the attacker, instead of claiming the attackers GitHub profile made a commit controlled by the victim.

No, you could claim the commits made by the victim this way. The steps would be:

1. Look for a contributor without GitHub account (i.e. xy@gmail.com )

2. Add xy@gmail.com to your GitHub account/create one with that address [0]

2.1 The commits made by the victim will now link to your account on GitHub

3. Send a mail to Nikita with your faked GitHub-profile

None of this will survive a thorough check, but under pressure this will easily pass a surface-level check.

[0] I think you need to confirm your primary E-Mail, but not secondary ones. Alternatively, you could look for commits made from an now deleted mail account.

Yup, AFAICS GitHub associates secondary addresses with your account without verification.

Although the victim would be notified that their address was added to a rogue account, the notification mail says: "If this wasn’t you, please ignore this email." Ugh.

> recent commits from this person

Given that the attacker managed to push unauthorised commits, I don’t see how recent commits can be considered a reliable method of authentication.

You need to audit the commits anyway, and presumably after the audit they can be trusted again.

Trying to sneak that diff in was a hugely wasted opportunity, it sticks out like a sore thumb. If they were going to commit anything, it should have looked like a legitimate change. Another option might have been to wait until just before a release and fiddle with the tags or the newly opened QA branch (if any).

Check out https://freedom-to-tinker.com/2013/10/09/the-linux-backdoor-... for another attempt, that one at least requires careful review to notice the problem

In the diff, could they be hinting that they've had access since 2017? [1] Maybe they just wanted to burn it now for some reason.

[1] https://github.com/php/php-src/commit/c730aa26bd52829a49f2ad...

I don't see any way they could have snuck anything in unnoticed as a large patch. The number of people who regularly do big patches to PHP is not that large. If you send a big commit as one of them, that person would likely notice and raise an alarm they didn't do it. If you send as somebody who doesn't regularly make big changes, then people would wonder how come this person, who never submitted anything big, suddenly submits a large change without any discussion. So it would have stuck out anyway.

It could work if you do it in some poorly maintained extension, maybe, but that extension probably wouldn't be much used either.

Committing to release branch would be the worst way of doing it, since only RMs commit there, and RMs are those people who are going to notice if somebody commits to their branch and it's not them. Also, since releases are tagged manually, it'd not get into the release anyway, unless you somehow trick the RM into merging the change into their local repo without looking. Which they have no reason to since they created the branch and only they are supposed to commit there. So it'd require some serious trickery.

It does make me wonder if something else was changed and this was what they wanted you to notice (as a diversion).

I think that's a good point. Apparently they had the ability to make commits as any user, which was a huge opportunity. Then they waste it on such an obvious backdoor? It does seem possible that this was a diversion tactic.

Edit: Elsewhere someone mentioned, this could have been "marketing" - to demonstrate their ability to take over accounts.

> this could have been "marketing" - to demonstrate their ability to take over accounts.

Hot take - They burned it, in a very visible way, to prove they've had access since 2017 (or whatever the investigation will reveal). A full security audit isnt very feasible.

Would force everyone to either downgrade and face known vulnerabilities, or do nothing and face high-risk unknown vulnerabilities.

They were presumably testing the waters to see if they were noticed first.

but why would you test that so obviously ('fix typo' but add code)? and they made it clear that they likely had access to the git server (by pushing 'from' different accounts), which (obviously, if that's what is actually happening) can kill their access

It does beg the question if this was only the first time they were noticed.

Note: I did not keep the original title (Changes to Git commit workflow), because it sounds mundane, rather than being related to an apparent compromise of the git server of a high profile open source project.

Run your own infrastructure. It will be fun, they said.

Moving to github doesn't change anything fundamentally. It just moves infrastructure hosting to someone else (even though it may or may not be a more secure hosting).

Something like signing commits, and requiring all commits to be signed properly with a known key (and checking this on push and before release) would be an independent verification layer, and a more fundamental response to the compromise.

It moves infrastructure to someone else _whose core job it is to manage this infrastructure_. I definitely trust GH more than myself in managing a git infrastructure securely

That's assuming the self-hosted git server itself was compromised, and it wasn't just a simple matter of the attacker stealing some committer's private keys.

If that's what happened, then moving to Github solves nothing.

> I definitely trust GH more than myself in managing a git infrastructure securely

That's FUD, and appropriate too considering who owns Github now.

Did you notice that the commit was reverted, but then almost immediately re-applied? This is a strong indication that the server admins did not know the how the server was compromised.

With Github, you can: (1) use "events" API to see which account did the push and (2) remove the account's ssh keys and change password from a secure machine to stop compromise right away. This stops the compromise completely.

Compared to that, if there was a compromise on our own server, it is quite hard to tell that everything was cleaned. Remember kernel.org compromise in 2011 [0]? They took a few weeks to recover, and this was with some of the best programmers.

[0] https://lwn.net/Articles/464233/

It's the style of the times: techno-feudalism. Serve a powerful enough feudal lord by doing your business on their servers and you'll be protected.

in comments on the first commit it was said that the access was revoked, but another one appeared, which points at the server

While it's not clear yet whether it was pwned infra or leaked privkey moving to GitHub would definitely lower chances for the former. When SDL moved to github recently I brought this up as one of the advantages when the typical "but M$ suxx" comments started pouring in.

HaHa I think some point like the challenge until...

After learning Django for a few months I decided to switch back to php/wordpress yesterday, php7 and php8 look solid and php remains to be the best language for the web in my opinion, Php8 added JIT which might extend itself outside of web applications. Hope moving to github will free the team to do even more on Php development.

I wonder if GitHub will ever support requiring signed pushes. It supports requiring signed commits, but that isn't always feasible, since you could want to merge commits signed by an external contributor without rebasing those commits in order to sign them yourself.

I wish git would support signing with tools other than GPG. I hate using it and tools like minisign/signify are a huge improvement.

Hell, the entire thing is simple enough to just be integrated in git.

That does seem like a really good idea. If I make a protected branch that requires signed commits I really only care the signatures on the commits you'd see with `git log --first-parent`, as those are the ones by people I trust.

It would have been interesting if GitHub had purchased Keybase instead of Zoom. I'm still a bit bitter over that sell out.

My PHP is poor, but it looks like it was being changed to check for a string coming in via a "typo'd" HTTP_USER_AGENTT header, that started with zerodium, and then eval the string without the zerodium prefix on the server.

Reminds me of that Linux compromise years ago where they picked it up because the change was directly on the build box, except at least that one was hidden behind an apparent `=` vs `==` "bug" - this one has a bit less style.

It is C not PHP in the linked diff :)

Its not actual PHP code, it's C, which PHP is written in

Ah - that explains a lot, I was expecting to understand a lot less of it ;)

Maybe this was someone close to Zerodium burning the compromise against PHPs Git server to bring attention to other zerodium commits dating back to 2017?

Zerodium buys vulnerabilities and exploits, not direct access though. You can’t sell them this kind of backdoor.

I mean with keys to the kingdom, you can just plant backdoors and such every now and then and sell it off on Zerodium et al.

Poking around on https://git.php.net, it appears to be using "gitweb":

  <!-- git web interface version 2.11.0, (C) 2005-2006, Kay Sievers <kay.sievers@vrfy.org>, Christian Gierke -->
"'Gitweb' is a Git web interface. It is written in Perl and can be used as a CGI script, or as a mod_perl legacy script (run by ModPerl::Registry handler). It allows browsing a git repository (or a set of git repositories) using a web browser." - https://git.wiki.kernel.org/index.php/Gitweb

The last commit looks to be Dec 2020: https://github.com/git/git/tree/master/gitweb

Though I don't know that this was how they got into the git server.

Disclaimer: wild speculation below

There's a strong possibility the vulnerability lay in their hosting setup rather than in the software application itself.

There's an argument to be made that self-hosted software that are non-trivial to configure for security have "dev ux" type security bugs, but I doubt that'd constitute a CVE. It could also be unrelated to the git-web hosting config.

Looks like Softlayer / IBM Cloud. There are some high numbered ports listening and accepting connections, including git/9418, which is interesting.

Was PHP's Git server using a self hosted Gitlab instance? I can't find the past threads, but I thought there was an announcement a while back that they were moving to Gitlab self-hosted. Maybe I'm misrembering which project that was?

It seems they use some combination of gitolite and their own stuff.

The post mentions karma, there's some code here: https://github.com/php/karma/

Thanks. That's a bit of a relief, given that we use self-hosted GitLab instances.

Well it's still vulnerable. At least in Gitlab you can use fancy features like requiring 2FA for members of a group. But you still need to secure its infrastructure.

I think the most common attacks on Github repos and groups have been on individual accounts. Which 2FA would mitigate.

> Which 2FA would mitigate.

Or Rust, right? </sarcasm> https://github.com/php/karma/search?q=exec

This looks like the work of either a talented but neophyte black hat, or an intentional burning of an exploit.

commits history have following:

  - 8d743d5  Revert "Revert "Revert "[skip-ci] Fix typo"""
  - 2b0f239          Revert "Revert "[skip-ci] Fix typo""
  - 046827a                  Revert "[skip-ci] Fix typo"
  - c730aa2                          [skip-ci] Fix typo

There is a reason why some services are worth it... for one you don't have to worry about all those issues that could pop up. It's someone else's problem.

Feel for the maintainers, tough to keep own stuff together -- but also first reaction is that another massively influential part of the web/opensource being stored on GitHub. I'm not anti-MS by any means but just reminded of the dominance of so much of our global codestores being on there

It's git. That part is easy to migrate. Exporting issues and discussions might take a bit of work, but if/when GitHub starts going downhill there'll be a script to migrate to GitLab (or whatever will be the preferred option by then)

(Yes, there's already such a script for GitLab.)

Unless the first step in MS Github's downhill journey will be to cut off APIs needed for those exports.

I wonder if the signed features of GitHub would have helped here. Probably can't ask all the 3rd party contributors to setup signing commits, but it would help to spot commits not signed.

If the attackers would have used their own name/email linked to a GitHub account and signed the backdoor commits, GitHub would display a green "Signed" label, and everything would look OK to an outsider.

That's why it's just a gimmick. Signing only works as protection against hosting compromise if the signing/verification is separate from the hosting itself.

Someone responsible for release would have to manually keep a list of keys of authorized comiters and check the repository against this list at the very least prior to a release.

The number of people with direct commit access is almost certainly less than the number of commits in a release though. Without signatures you have to verify every commit.

I'm not against signatures. I just think relying on third party system to verify them is a gimmick.

It's not about the green label. A set if commits signed by some unknown person is a lot easier to spot and clean up.

Looking at these commits they say mid-2017. Without signatures any previous commit could be by the same author.

> I wonder if the signed features of GitHub

Uh, you mean the features of ... Git?

Quick migration of toolchain. Curious how that will work out. If it does, my deepest respect to their engineering.

PHP already uses GitHub and pull requests there, it's just not been the canonical git repository until now. It's not really a big change for most; if anything it's easier.

This was so dumb. Even mentioning zerodium and skip-ci and sloppy commit message.

This was not dumb, most probably this was marketing.

They chose to burn -very publicly- the two accounts they had access to in order to showcase that they have an in that they can use somewhere else.

Dunno, they just provoked a hardening and possible review of the remaining infrastructure.

It seems likely it was meant to be seen and fixed.

Bold move


Of course, there's a vast ecosystem still running shopping carts, forums and whatnot. It's much more prevalent that you seem to think. Just because it's not cool anymore (or well ever) doesn't mean it's dead.

That's a very dated view of PHP. Modern PHP is far "cooler" than it was a few years ago. Just look at the entire https://laravel.com ecosystem — the most starred web framework on all of GitHub, across all languages, is Laravel.

Discovering Laravel a couple of years ago was perhaps one of the most significant things that made PHP interesting again after 20+ years of working with PHP.

It's refreshing (as far as PHP goes) and worth checking out with an open mind. Performance on the other hand, that take a little more work.

Performance keeps getting better, particularly with PHP 7 and 8. And just last week, “Laravel Octane” was announced... it lets you serve a Laravel app using Swoole, and early benchmarks on a MacBook Pro suggest 25k+ HTTP requests served per second!

Cool, I'll have to get updated and checkout Laravel Octane soon after clearing a work backlog, did notice Taylor tweet something about it.

I'm guessing your benchmarks are using native php-fpm and not through docker on your MBP?

Laravel Octane is using Swoole which implements a complete webserver, so there won‘t be any php-fpm more. As you would need to restart the server for every code change, there won‘t be a performance difference between docker on mac and non-docker.

I’m not too sure, I was just quoting the figures Taylor gave after his Laracon demo. I think the beta is being released next week.

Also, WordPress. WordPress is pretty much everywhere.

and there are reasons for that. As bad as it is for a developer experience, the ui for non-tech people just can't be beat.

Oh, definitely. As a blogging engine it's quite usable and very easy to get used to even for people who are IT-illiterate.

I'm not convinced it's the best for building e-commerce sites and whatever people are piggy-backing on top of WordPress nowadays.

Oh, I get that, it was more directed at the OP. It's not your Rust, Nim or Crystal or... which is what I meant about 'cool'. Coolness is always in the eye of the beholder :)

You must be living in s pretty air tight bubble.

facebook, vk, youtube, wordpress, shopify - a lot of stuff runs on php

Facebook wrote a new runtime and has, over time, changed the language rather significantly (eg strong static typing, async/await, generics, overhauled much of the standard library.) It's called Hack. At this point, I don't think you could reasonably say facebook's written in php any more than you could say various c++ projects are written in C.


Google would never let YouTube use PHP. I guess it did originally though.

YouTube was originally Python, not PHP.

Afaik at least facebook uses a heavily modified version of PHP ? No way they would run it pure.

VK.com also uses their own php compiler (VKCOM/kphp on github)

Perfect time to get your php commit bit now. Just ask:)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact