I can empathize with them, this is actually quite difficult when you're an open source project.
You have your source code public, so you can't commit/push because then you've leaked info.
Your build pipeline is also likely public, so you can't just build from another source code... source.
You can create a private repo, but now you also need to build a private build system.
You don't get the same generous build resources for private repos, so you either:
a) simplify the build to just the essentials
b) pay
Unless you have significant funding, you'll probably go for a).
Now you have to maintain two systems, also with some differences between them.
Plus you need a way to get the build artifacts from the private system to the public one, typically at a fixed point in time.
Any failure/delay (e.g. less powerful build on private repo, errors because you exercise it rarely) causes people to get upset that you didn't respect the announced time.
Maybe you're also using things like image attestation, another can of worms.
Those are all challenges that surely need to be overcome!
But they had _two months_ to figure out how to overcome them. For them to not have figured out a solution and have pre-built images ready to push as soon as the embargo lifted is ridiculous.
I'd say that for someone like PHP, in this era of AI-driven vulnerability/patch race, having a private CI/CD pipeline is a necessity.
PHP is written in C, I'd be surprised if similar problems didn't lurk in other places.
Maybe they will construct the private pipeline after this crisis. Millions of websites are probably now waiting for all the downstream Docker images to be built and pushed.
The images everyone is waiting for are built by Docker though? So now if you're Docker you have to have a private CI/CD pipeline with all library projects?
It would be great, but it's also something to manage. Just keeping it working with all the changes that happen across all projects would be something.
I don't know about PHP, but at work our artifacts are signed, and you certainly cannot have the signing certificate on your laptop.
It's stored in some secure server, and only accessible from some allowlisted build nodes, for specific builds (release branch, ...).
Now of course, there are ways around that if required, but they are, by design, manual, cumbersome and slow. I could not create a release build locally in a few minutes. A few hours maybe if the whole organisation dedicated itself to it.
Setting up all that for single man project might be a hassle but for any sofware with actual dev team I wouldn't call it hard to do.
I'd worry more about the fact that essentially publishing binary-first means that it ever gets compromised that makes it easier for attacker to smuggle exploit using "this is security update" pretext.
Making a cloud CI/build/release system whose behavior can't be reproduced locally requires some kind of brain damage. I can't believe people tolerate this.
The problems you describe in your first post are only problems when you can't reproduce your build process locally. With a local build you can always do it in private for free.
The problems you bring up here are eminently solvable too. Yes there are many solutions for building for multiple architectures and versions. Cross builds, containers, VMs, or just having more than one computer locally. You can address building the wrong thing with reproducible builds, but for a one off patch like this it's not the most likely problem and relying on a trusted maintainer temporarily during an emergency is better than waiting for a glacially slow cloud build while exploits spread. Pushing builds to the cloud is inherently a cloud step, so it's fine to leave keys for that in the cloud, but there should be a process for publishing local builds and it should be essentially instant when the build is already done.
You should always have a local build process that replicates CI. There is no excuse.
There's over 3 million downloads per month for the official PHP image. That may certainly not be "most PHP users". But still is a a lot of users.
I'm quite certain that your exact sentiment is why the images weren't updated immediately: I'm afraid PHP foundation doesn't see docker images as a particularly important distribution system.
> I'm quite certain that your exact sentiment is why the images weren't updated immediately:
I wouldn't be that certain.
I think it's likely that the vast majority of PHP Docker image downloads are for local development, for one thing.
But nevertheless, it would not come close to the number of websites running PHP from the Ubuntu Server packages. It is absolutely right that the effort was focussed on the distribution route. Because PHP is not "expert" hosting territory; it is still more often whole cloud VMs running the distro packages.
I find it incredibly naive that they’d hope it would not leak after weeks/months of embargo.
Being given advance notice of an upcoming security fix is an immense privilege, organisations should turn around quickly when given such information, they shouldn’t be given more than a couple of days to update IMHO, it’s not like the rest of the world can wait.
Keep in mind that some of the organizations are presumably patching ancient versions of PHP like 5.6. I am sure this is a process which takes considerable time, especially as upstream PHP has washed their hands of such old versions and the downstreams are effectively on their own.
(It is often much cheaper for an enterprise to just get a long term support contract than to properly upgrade to a modern PHP stack.)
> (It is often much cheaper for an enterprise to just get a long term support contract than to properly upgrade to a modern PHP stack.)
I've experienced similar to this. I know I shouldn't be, but I still get amazed every time I hear that some small company is still using an old DOS machine running core software based on dBase that nobody has any clue on how works and the developer has since moved on to a higher plane. One person told me that they sweat every time the power goes out longer than the attached UPS can run holding their breath that the machine powers back up.
The PHP Foundation should manage it, the Docker ""Official" Images" approach clearly just doesn't work properly for responding to vulnerability patches.
PHP foundation also doesn't "Run" the apt or rpm packages, yet they certainly coordinated with debian, and other big distros to push the updates out timely.
Now I don't know what happened here. Maybe PHP foundation reached out to docker but got no help, maybe they simply forgot about docker as distributor of their software.
