
Mystery meat OpenJDK builds strike again - hocuspocus
https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-May/009330.html
======
CaliforniaKarl
A few emails later in the thread[0], it is noted...

> ...who actually maintains the "official" Docker images for 8u and 11u?
> Github history for Dockerfiles points to no one I recognize from around
> OpenJDK.

That, I think, is the main problem.

[0]:
[https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-May/0...](https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-May/009335.html)

Edit: Actually linking to source (which I should've done from the beginning…)

~~~
Arnt
Main problem or not, having the "release" version number on anything other
than an actual release version is begging for problems.

------
chungy
As an observer, not any kind of authority:

The release methodology of OpenJDK from Oracle is obscure at best, and it is
hard to imagine how package managers are supposed to cope. I notice both of
these examples come with Debian version strings attached, and they likely
build from tags in the version control repository. As example, in these two
links:

[https://hg.openjdk.java.net/jdk8u/jdk8u/tags](https://hg.openjdk.java.net/jdk8u/jdk8u/tags)

[https://hg.openjdk.java.net/jdk-
updates/jdk11u/tags](https://hg.openjdk.java.net/jdk-updates/jdk11u/tags)

The "-ga" tags do pretty clearly delineate official releases, but they also
get pushed to the public repository well (often weeks) after Oracle posts
official binaries for these releases. It is confusing and makes it not all
that mysterious that distribution packaging will take the best that they can
see for doing a clean build, as they like to do.

It would help quite a lot if the +1/+2/+3/etc releases were instead delineated
with "-alpha1", "-beta1", "-rc1" tags. Much clearer that they aren't meant to
be production.

~~~
djsumdog
I think this has lees to do with the OpenJDK team and more to do with the part
of the Docker team and/or community that maintain this repo and not pulling
correct release versions.

I knew someone who worked at Gradle and noticed how RedHat did all their own
gradle packaging that broke everything they had in their offical rpms and spec
files.

Some projects like Elastic Search maintain their own v2 repos on their own
domain (all there's are mirrored on Docker hub but their docs point to their
own repo) and other projects have official Docker hub accounts (like Jenkins).

I'm guessing the right solution might be Docker hub working with the OpenJDK
guys so they maintain their own organization on Docker hub and integrate
building and pushing to it from their own CI.

~~~
repolfx
It's an age old problem. Upstreams were complaining (legitimately) about how
Linux distributions would (re)package their software and break it along the
way 20 years ago. Broken packages in Docker being pulled from Debian is no
surprise, it was ever thus.

------
jillesvangurp
This is indeed not the first time something like this happens and a good
reason to get your openjdk builds from someone that has licensed the test
suites and supports their builds.

I remember having weird issues a few years ago with builds for both Centos and
Debian/Ubuntu where they first took ages to get release builds out and then
made it worse by basing their builds of pre-release versions rather than
actual stable releases. This was several months into the JDK 8 cycle. In their
defense, JDK 8 had some nasty bugs early on. But still, a lot of this seemed
accidental to me and not intentional.

You can get decent openjdk builds from Amazon, Azul and Red Hat. They each put
in quite a bit of effort to ensure these builds are solid and tested and they
each offer these builds for free they also each offer Docker images with their
builds and make it very easy to use them. I've been using Azul docker images
for quite some time.

Amazon particularly seems to have made an effort with their openjdk builds. If
you read their commit log:
[https://github.com/corretto/corretto-8/commits/develop](https://github.com/corretto/corretto-8/commits/develop),
you'll see a few backports of fixes from openjdk 9 and lots of efforts to get
the packaging right. The last bug fix they added was from just a few months
ago. I'm currently considering switching to this.

BTW. Oracle seems to have stepped away from this and only offers limited time
support for their own JDKs which also cost money. So, not including them in
this list on purpose. Unless you must use them (for legal/business reasons)
and fully understand the legal situation that implies, you should avoid using
them.

~~~
jacques_chester
Pretty much every enterprise vendor has an OpenJDK build. I work for Pivotal.
We've been building and shipping the OpenJDK from source for years (mostly
Java buildpack, but we use it for anything that we ship that requires a JVM).
Like our peers at other enterprise software vendors we spent a lot of
engineering making the process smooth and traceable.

I think more of the industry will move towards AdoptOpenJDK builds. There's a
lot of value in combining forces to produce the same byte-identical binaries
for everyone.

~~~
noir_lord
Jetbrains do as well, they store theirs on bintray.

It's all a bit of a mess tbh and particulary frustrating from the outside
where you aren't a full time java dev.

------
saagarjha
Not completely related, but this is one of the reasons why I prefer to add
commits bumping the version number when a release is made instead of before–it
means that any builds made from source don't have version number they
shouldn't.

~~~
phyzome
It took me a long time to understand why Maven uses version strings like
4.2.1-SNAPSHOT but this is exactly why it's useful.

(The standard practice is to strip the "-SNAPSHOT" off for the release commit,
then stick it back on for the next commit, but with a bumped minor version.
Both commits are then pushed at once, so stuff built locally always is marked
as a SNAPSHOT.)

~~~
Boulth
The problem here is that Debian's OpenJDK maintainer removed the "work in
progress" indicator string manually. No approach protects against this.

------
karianna
The AdoptOpenJDK community (a consortium of vendors and users that produce
business ready OpenJDK binaries) will work with the Debian maintainers to
course correct this going forwards. If you’d like to get involved see
[https://AdoptOpenJDK.net](https://AdoptOpenJDK.net)

~~~
makomk
I'm still astounded how much of an (intentional?) mess Oracle have made of
Java and OpenJDK. If you search for Java you get to a Oracle page with a big
download button and a cryptic yellow warning about license changes, with a
little link at the bottom directing people to OpenJDK. The official OpenJDK
builds they send people to have no Windows installer - they're just a ZIP of
the JDK. There is no way for an ordinary Windows user searching the internet
for a way to install Java on their work PC to end up with a working Java
install that doesn't leave their employer on the hook for Oracle licensing
fees. AdoptOpenJDK have working builds with a proper Windows installer, but
unless you already know about them somehow (for instance, by reading HN
regularly) there's no way to find them.

~~~
ptx
The current thinking is that end users are not supposed to install Java - it
will be bundled with every application. That kind of solves the problem, but I
guess we'll have to forget about writing small utility programs in Java.

Also, complicating things a bit if you want a traditional shared Java
installation, AdoptOpenJDK has apparently been refused a license for the test
suite, making it not quite properly offical Java.

~~~
pron
> but I guess we'll have to forget about writing small utility programs in
> Java.

Developers will continue using the JDK (for which no installation is
necessary; just setting JAVA_HOME). A tool suite intended for users will share
the same runtime. And when it comes to small, user-oriented, standalone
utilities, Graal's native image makes that more compelling than ever.

------
dotdi
Ubuntu Java packages are a joke as well and I stopped using them when I was
having issues with the OpenJDK-11 package from apt. It turned out the the
package wasn't packaging Java 11, but Java 10 instead, with a fake name [0]. I
didn't check but they might rely on bad Debian packages just like Docker.

\---

[0]:
[https://twitter.com/d53r/status/1055403252286717952](https://twitter.com/d53r/status/1055403252286717952)

~~~
brazzy
There was a deliberate decision, not a random mistake:
[https://askubuntu.com/questions/1037646/why-is-
openjdk-10-pa...](https://askubuntu.com/questions/1037646/why-is-
openjdk-10-packaged-as-openjdk-11)

TLDR: Ubuntu Bionic came out before Java 11 was officially released, they did
not want to have Java 8 as their default, but Java 10 would not fit their LTS
strategy since it is not itself an LTS release.

Having Java 10 in a package that claims to be Java 11 was a stopgap measure,
and it has been fixed now, i.e. if you update that package you now get Java
11.

~~~
mrighele
Unfortunately by doing this they made another mess.

When they updated OpenJDK to 11 [1] they didn't realize that some things have
been dropped from the core project, and others have moved; as a result (among
other things) JavaFX is now broken in Ubuntu Bionic. [2]

I think this is something inexcusable for an LTS.

[1] [https://blog.ubuntu.com/2019/04/19/announcing-
openjdk-11-pac...](https://blog.ubuntu.com/2019/04/19/announcing-
openjdk-11-packages-in-ubuntu-18-04-lts)

[2]
[https://bugs.launchpad.net/ubuntu/+source/openjfx/+bug/18250...](https://bugs.launchpad.net/ubuntu/+source/openjfx/+bug/1825054)

~~~
dcminter
TLDR; I agree, but I understand how this happened.

The logic seems to have been:

* Ubuntu 18.04 is a LTS release.

* LTS releases should only incorporate software that is supported on a similar lifecycle.

* OpenJDK 8 was moving out of its support window early in the lifetime of Ubuntu 18.04.

* OpenJDK 11 (the next long term supported JDK) was not going to be released until after the release of Ubuntu 18.04

* OpenJDK 9 and 10 were short term releases so they weren't options

* The move from Open JDK 8 to Open JDK 9 included a lot of breaking changes (from project Jigsaw/modularisation)

So a compromise was necessary. From a purist packaging point of view it makes
sense; release JDK 10 as "JDK 11" and silently upgrade under the covers when
possible. In theory this would only involve minor changes and the 'wrong'
version would only be there for the first few weeks of the release. In
practice there were some breaking changes - enough that the backporting of
OpenJDK 11 into Ubuntu 18.04 actually took rather a long time. It's also, of
course, really confusing when you "sudo apt install openjdk-11" and something
totally different happens.

The problem with staying on OpenJDK 8 (perhaps the obvious approach) is that
in an unsupported state it would lack security fixes. In practice of course
several alternative upstreams are now available supporting it.

Hopefully it will all go with fewer surprises when we get to the next LTS
edition (Ubuntu 20.4).

~~~
jrwr
I saw earlier in this thread of blaming Debian, but all I'm seeing is Ubuntu
snafus

------
liveoneggs
debian is not a trustworthy source of software if you expect it to be close to
the upstream version.

They will change things around completely (MySQL/MariaDB), cause terrible bugs
(OpenSSL), create crazy/renamed/weird versions (apache httpd -> apache2), etc
etc.

Frankly you are lucky to get something this close to real java and not a shell
script wrapping GCJ or whatever.

~~~
Lt_Riza_Hawkeye
>And any day Debian decides to stop using Foo, is always a happy day for Foo's
upstream. Finally, the stream of bugs matching the below template will stop
spamming Foo's bug tracker:

>> Debian X.x (stable) >> foo <5-10 years old release> >> >> <XXth duplicate
of a 7 year old issue that has been fixed 3+ years ago | 5 year old Debian-
only bug caused by a Debian patch>.

>The only appropriate reply to such bugs IMHO is:

>> Fuck off. And don't come back before switching to a distribution >> with a
non-broken definition of stability.

>Unfortunately, most upstreams always try to stay _civil_.

------
javagram
[https://lists.debian.org/debian-
java/2019/05/msg00008.html](https://lists.debian.org/debian-
java/2019/05/msg00008.html)

> I can only comment on the OpenJDK 11 backport in Stretch since I'm the one
> who uploaded it last month. Debian 9 "Stretch" is far from being usable with
> Java 11 (we poured a lot of work into Debian 10 to make this possible) and
> this backport can only be reasonably seen as a technology preview aimed at
> gathering feedback.

More at link, but that’s concerning. We’ve been using the “OpenJDK” images but
maybe we should look at using a different set of them.

------
ars
There don't seem to be any debian emails about it:
[https://lists.debian.org/debian-
java/2019/05/threads.html](https://lists.debian.org/debian-
java/2019/05/threads.html)

~~~
CaliforniaKarl
The place where you'd want to look is the Debian Package Tracker entry for the
source package, openjdk-8.

Here's the URL:
[https://tracker.debian.org/pkg/openjdk-8](https://tracker.debian.org/pkg/openjdk-8)

That has the current status of the package, and links to the bug tracker,
which is where I'd expect things to appear!

~~~
kevin_b_er
That seems an odd place to look, considering this is at the top of the page:

> package is gone

> This package is not in any development repository. This probably means that
> the package has been removed (or has been renamed). Thus the information
> here is of little interest ... the package is going to disappear unless
> someone takes it over and reintroduces it.

So it is an obsolete package?

------
qxcbr
This is the norm for Debian, including their own mystery meat in every package
without asking upstream, and then making it look like an official release. I
remember there was a post about that from the Redis guy a few days ago.

The guy who made those Docker images is not to blame in my opinion because he
just trusted Debian, which is a renowned distro; he expected them to do the
right thing. I believe Debian has a good reputation that they don't deserve
anymore.

------
rocky1138
To be fair, I wouldn't have known what "WE" meant either. Why not "WIP"
instead?

------
CaliforniaKarl
Unrelated, but one thing that's confused me, as someone who doesn't really use
Docker.

I don't think of a programming language as being the thing upon which I would
base my layers of containers. My expectation, as someone coming from not-
containers, would be that I should first start with a core-libraries
container, _then_ a language container, and then packages on top.

Having this clarity of composition, I think, is important if I need to start
bringing other stuff into the environment, particularly stuff that requires
compilation, so that I know what to target it against. It also makes it easier
to know when a layer needs to be upgraded, for example because of a security
issue.

The downside I see is that maintainers of layers higher up in the stack would
possibly have to build multiple variants of their layer. But maybe that
wouldn't be too big of a problem?

~~~
chrisseaton
Core library of what programming language, if you haven't picked a language
yet?

~~~
jlarocco
I think the OP's talking about stuff like Qt, OpenGL, BLAS/LAPACK.

