
Project Jigsaw: Complete - artbristol
https://mreinhold.org/blog/jigsaw-complete
======
clra
I don't follow Java too closely and the world "module" is so hopelessly
generic that I had a hard time understanding what the article was talking
about. It links to this document [1] though, which is a long read, but does an
excellent job of describing of describing all the ins and outs of the module
system.

(tl;dr Packages can now declare themselves as modules instead of simple
namespaces, and they get to choose exactly what they export and what other
modules they require. Projects can now resolve their types from within modules
with a _module path_ instead of on a per-type basis from the class path. It's
a huge step forward for Java modularity and dependency management.)

I have a hard time being _too_ impressed because more modern languages are
doing this out of the gates and with far fewer hacks and more out-of-the-box
tooling (e.g. Rust), but acknowledge just how monumental of an effort it was
to design the new system and then build it into Java's core in a way that's
mostly backwards-compatible. Java's going to be around for a long time, and
this is a big long-term gain for the ecosystem.

I'd _really_ love to see languages like Ruby tackle this next. It's sorely
needed.

\---

[1]
[http://openjdk.java.net/projects/jigsaw/spec/sotms/](http://openjdk.java.net/projects/jigsaw/spec/sotms/)

~~~
chrisseaton
> I have a hard time being too impressed because more modern languages are
> doing this out of the gates and with far fewer hacks

Well of course - it's easier to build something like this into a clean-slate
language isn't it? It's harder to build it into an existing language and VM
spec with an incomprehensibly large volume of existing code to be compatible
with. It's backwards to say it's not impressive because someone else with zero
constraints to work with also managed it.

~~~
clra
> _Well of course - it 's easier to build something like this into a clean-
> slate language isn't it? It's harder to build it into an existing language
> and VM spec with an incomprehensibly large volume of existing code to be
> compatible with. It's bizarre to say it's not impressive because someone
> else with zero constraints to work with also managed it._

I hope my original comment was relatively clear on this, but sure, it's a big
accomplishment and will be a quantum leap for the ecosystem. However, given
that languages elsewhere have had better systems for quite some time now, it's
not like they're advancing the state of the art in dependency management.

But again, it wasn't an easy thing to do (a very hard thing even) and everyone
involved deserves major felicitation.

~~~
smallnamespace
I guess the real question is, why do you find advancing the state of the art
to be more impressive or deserving of acclaim than integrating improvements to
existing systems?

Both activities seem valuable and complementary to me -- there's no point in
advancing the frontier if nobody will bother to make those advances
practically useful.

------
djhworld
I think it's going to take me a while to learn about the module system in Java
9, I'm probably going to bank on watching other projects do it first to see if
any best practises emerge as developers get used to the new way of working.

One thing I will say though is Java 9 has had a definite improvement on Java
applications that run on my Raspberry Pi 3, I think because this got included
[http://openjdk.java.net/jeps/297](http://openjdk.java.net/jeps/297)

One application used to take 55s to start up, now it takes around 15s!

~~~
Koshkin
> _15s_

Still sounds ridiculous. Assuming this is how much time it takes to initialize
the VM, I wonder if it would be possible to have a pool of VM processes that
are already running and ready to accept an application code.

~~~
djhworld
It's a spring boot application so comes with the baggage of initialising
Spring + Spring MVC, that took a long time on JRE 8

~~~
curun1r
I don't know if it's still a problem with Spring, but at a previous job about
8 years ago I was able to cut our application's start time from about 80
seconds to ~12 by writing a custom implementation of the logic that Spring
used to find bridge methods to determine the proper annotations. We profiled
the startup and found that over 80% of the time was being spent in reflection
to resolve those bridge methods and that could be done significantly faster by
using asm to load the classfile bytes and resolve the bridge method by looking
directly at the invokevirtual call. Spring avoided this approach because it
could fail in situations where the SecurityManager prevented filesystem
access.

I don't know if that's still the case, but it always bugged me that Spring
would slow down the >99% case by so much just to accommodate the <1% case.

~~~
jacques_chester
Dave Syer looked at this topic recently[0]. Most of the time spent in loading
isn't due to reflection or annotation processing: it's proportional to the
number of classes loaded.

That said, Spring is fairly enthusiastic about pulling stuff. As he says:

> _Since more beans mean more features, you are paying at startup for actual
> functionality, so in some ways it should be an acceptable cost. On the other
> hand, there might be features that you end up never using, or don’t use
> until later and you would be willing to defer the cost. Spring doesn’t allow
> you to do that easily. It has some features that might defer the cost of
> creating beans, but that might not even help if the bean definitions still
> have to be created and most of the cost is actually to do with loading and
> parsing classes._

I imagine that they'll keep poking at it, especially since they're working on
Spring Cloud Function[1], where cold starts for single invocations are common.

[0] [https://github.com/dsyer/spring-boot-startup-
bench/tree/mast...](https://github.com/dsyer/spring-boot-startup-
bench/tree/master/static)

[1] [http://cloud.spring.io/spring-cloud-
function/](http://cloud.spring.io/spring-cloud-function/)

~~~
semi-extrinsic
> most of the cost is actually to do with loading and parsing classes

It boggles my mind that loading and parsing even a few thousand classes is
something that a human can perceive as "dog slow" when carried out by a quad-
core 1.2 GHz CPU.

~~~
jacques_chester
My understanding is that it's I/O bound, that the JVM more or less fetches
each class from the JAR file or disk individually.

I am not sure if that's an implementation decision or whether it's imposed by
the JVM specification. At some point I want to tinker with OpenJ9 to find out.

------
fndrplayer13
I think its very exciting they were able to get the number of discrete modules
up to 26. That's extremely impressive to me given the age and complexity of
Java! I went to a talk a few years back from one of the contributors to this
project (forget the name, sorry!) and I think if my memory serves me they had
significantly fewer discrete modules at that time -- perhaps 6?

Anyhow, I think this is a great step forward for Java.

------
morecoffee
Jigsaw was what held up the Java 9 released, due to some members rejecting the
proposal. It was changed and revoted upon, but it's hard to follow what was
modified to make it acceptable.

Is there anywhere the design goals and constraints for modules are? I'd really
like to know how they ended up in their current form.

~~~
needusername
> Jigsaw was what held up the Java 9 released, due to some members rejecting
> the proposal.

That was only a few weeks. If you check the official release schedule
[http://openjdk.java.net/projects/jdk9/](http://openjdk.java.net/projects/jdk9/)
Jigsaw was supposed to be feature complete a year before that vote happened.
Jigsaw was simply not close to ready when it was merged to master. And yes,
Oracle still claims it hit every milestone on that schedule.

> It was changed and revoted upon, but it's hard to follow what was modified
> to make it acceptable.

[http://openjdk.java.net/projects/jigsaw/spec/minutes/2017-05...](http://openjdk.java.net/projects/jigsaw/spec/minutes/2017-05-18)

[http://openjdk.java.net/projects/jigsaw/spec/minutes/2017-05...](http://openjdk.java.net/projects/jigsaw/spec/minutes/2017-05-22)

[http://openjdk.java.net/projects/jigsaw/spec/minutes/2017-05...](http://openjdk.java.net/projects/jigsaw/spec/minutes/2017-05-23)

Not much really. Basically Oracle called their bluff. At that point is was
really too late as Jigsaw was already merged to master and more or less
touched every part of the JDK.

> Is there anywhere the design goals and constraints for modules are? I'd
> really like to know how they ended up in their current form.

Not really, it was really just Oracle making things up as they went. For the
past ten years Mark Reinhold would give talks at Java conferences what the
module system comping in the next Java version was supposed to do and every
year the content was different. For example they have no good explanation
while they mashed the module declaration into a .class file, basically
everybody except Oracle thinks it's a bad idea but the just went with it
anyway. They claim "reliable configuration" as a goal but without versioning
that's meaningless.

~~~
pron
> They claim "reliable configuration" as a goal but without versioning that's
> meaningless.

Versioning is not currently _built into_ Jigsaw, but Jigsaw provides all the
building blocks necessary to support it in third-party tools.

> For the past ten years Mark Reinhold would give talks at Java conferences
> what the module system comping in the next Java version was supposed to do
> and every year the content was different.

Plus the entire development was done in the open, in an open source project
with an active mailing list and frequent prototypes.

~~~
needusername
> Versioning is not currently built into Jigsaw, but Jigsaw provides all the
> building blocks necessary to support it in third-party tools.

If I need third-party tools to validate my dependency graph at compile time
and at runtime to ensure "reliable configuration" then why do I need Jigsaw at
all? That's the current state of affairs with Java already, Jigsaw provides no
value.

> Plus the entire development was done in the open, in an open source project
> with an active mailing list

Ostensibly. There is a difference between having an open mailing list (where
people could give feedback and get a superficial thank you before getting
ignored) and a readonly repository and actual open development. Quite frankly
I find it telling that in 2017 we have to stress that there was and open
mailing list and readonly repository. The most important thing, the decision
process, was completely intransparent and all Oracle/Sun internal.

> and frequent prototypes.

Which weren't useful because none of the frameworks, libraries and tools would
support it.

~~~
pron
> Jigsaw provides no value.

I think you should read more about Jigsaw.

> I find it telling that in 2017 we have to stress that there was and open
> mailing list and readonly repository.

We don't need to _stress_ it, just to correct an incorrect report.

> The most important thing, the decision process, was completely intransparent
> and all Oracle/Sun internal.

This is simply not true, as a quick look at the mailing-list archive would
show, as well as from the fact that the project took 9 years.

> Which weren't useful because none of the frameworks, libraries and tools
> would support it.

They were welcome to support them. I'm not sure what it is that you want,
exactly.

------
didibus
I'm still not sure of the practical benefits? Seems to be purely for IOT and
nothing else. Am I correct?

Edit: To add to my question, jar hell was solved by tooling which generates
your classpath for you. So it hasn't been an issue for me in years. And they
refused to add versioning, so you still need to use those tools. Strong
encapsulation just means they disabled reflection access to non public
members, which I consider a regression on functionality. Either way, not a
particularly useful feature to me, in fact it breaks some of my code. So I'm
left with being able to have small JVMs which don't bundle the full standard
lib, which seems to be mostly of use for memory constrained environments like
IOT. But I'm actually hopeful there's some bigger practical benefits I'm not
thinking of, so I'd love to hear from people who know more about jigsaw.

~~~
pjmlp
Encapsulation is another one.

Before Java 9, the only way to hide private APIs would be to have everything
in the same JAR.

If a library is splinted across multiple JARs, then private APIs only to be
used internally by library become exposed to everyone, and there will always
exist someone making use of them even if explicitly marked as internal.

This is nothing new to Java, other languages e.g. Ada, Delphi, .NET and even
Go have this kind of visibility concept.

~~~
alkonaut
.NET does have the module level directive "InternalsVisibleTo" that declares
another assembly as a friend assembly that can use internal API's from the
first assembly. So if you have e.g MyLib.Core and then two impls on top say
MyLib.Windows and MyLib.OSX, where everyone would only use one of the impl
libs but always the core lib (which is why it was split - then you could make
internal API in the core lib visible ONLY to the Windows/OSX libs by using
InternalsVisibleTo, and users of the library could not see that internal API.

~~~
pjmlp
Yep, that is what I meant by mentioning .NET in "This is nothing new to Java,
other languages e.g. Ada, Delphi, .NET and even Go have this kind of
visibility concept."

~~~
alkonaut
Right. Thought you meant "the file/assembly is the module, no way around it"
was the concept.

------
sverhagen
In the organization I work for, we're contemplating a move from Oracle JDK to
OpenJDK, when we move from 8 to 9. I'm still nervous about OpenJDK from some
historical scary stories of things not working well in OpenJDK or at least
different. I've not found much on the subject, or even on OpenJDK as a
functioning open source project. (Or is it just a code drop of the Oracle
JDK?) If anyone has good insights, or links to some realistic comparisons, I'd
be thrilled.

~~~
tjfontaine
Note: Oracle is my employer, the following statements are my own — (insert
Safe Harbor notification)

Good news regarding any trepidation on the differencesa between the runtimes —
recently we announced that there will no longer be a difference between
OpenJDK and OracleJDK [1]

To quote: “The Oracle JDK will primarily be for commercial and support
customers once OpenJDK binaries are interchangeable with the Oracle JDK
(target late 2018)”

[1] [https://blogs.oracle.com/java-platform-group/faster-and-
easi...](https://blogs.oracle.com/java-platform-group/faster-and-easier-use-
and-redistribution-of-java-se)

------
puppetmaster30
I think this is for people that had issues w/ 'jar hell'. I did not.

~~~
mschwaig
The biggest pain for me was that it felt like there was no good way to create
a library with some internal structure in the form of packages without
exposing parts of your internal implementation as public.

When Jigsaw was delayed I was playing with the idea to create a project called
shitty-jigsaw, which would just merge all of the packages in a module and
change the access modifiers to be more restrictive according to some module
definition.

~~~
sverhagen
Would shading have done the job? Not to say that's a _good way_, but it _is_ a
way.

~~~
mschwaig
Sure you can use shading to move everything you don't want in your public API
into some other package, or you can use something like ProGuard to obfuscate
everything but your intentional public API. However those things will still be
public.

The only way to not have those things visible for the user of your library is
to only use one package in your implementation.

------
smegel
Does it have any positive impact on startup time for simple programs if you
are not loading large chunks of the JVM?

~~~
merb
yes. especially since in the future you can actually compile modules as
libraries and aot load them. at the moment this is experimental and should
probably only work for java.base

------
sscarduzio
Will this resolve the slow start problem in serverless apps?

~~~
brianwawok
It's a step. Still 10x slower start I suspect compared to node, but much
better "warm" performance.

~~~
needusername
Nope, it makes things (a little bit) worse because now you also have to start
up a module system.

~~~
abiox
> start up a module system

i thought jigsaw was about compilation artifacts. are you saying there are is
a (nontrivial) runtime component? what does it do?

~~~
needusername
> i thought jigsaw was about compilation artifacts.

Nope, jigsaw doesn't affect compilation artifacts other than an additional
module-info.class present in a .jar file.

There is an intentionally completely unspecified, Oracle/OpenJDK proprietary
extension (yes) called jlink that generates different artifacts. The aritfacts
are unspecified, only work with Oracle/OpenJDK, and the licensing restrictions
for redistribution are unclear.

> are you saying there are is a (nontrivial) runtime component? what does it
> do?

1\. module definitions have to be parsed into memory structures to build a
runtime model of the module system

2\. the runtime model built in 1. has to be validated

3\. every access has to be checked if it is permitted by the module system
based on the information from 1.. If it is not permitted a decision has to be
taken what should be done, this can include figuring out whether it is the
first such access which needs additional memory structures (this is the
default). See [1] for more information.

Probably some additional things that I forgot about.

[1] [http://mail.openjdk.java.net/pipermail/jigsaw-
dev/2017-June/...](http://mail.openjdk.java.net/pipermail/jigsaw-
dev/2017-June/012841.html)

------
badsectoracula
I remember this from... 11 or so years ago, i think. I was making some Java
applet based game and the slow startup time and big JVM download was something
that an issue among people making such games (which by the time i was into
were dying - although they'd soon get a short lived shot in the arm thanks to
Minecraft, but that was later). There were some posts in javagaming.org forums
from some people who were talking with Sun (that was before Oracle - Sun at
the time was actually interested into uses of Java in gaming and talked with
devs, including from smaller/indie developers) about a new improvement for the
VM to modularize the packages so that only things that are required would be
loaded (faster startup) and the default JVM download would be much smaller
(sort of comparable to Flash) which was a big issue at the time for those
making web-based games (that niche for Java games would soon die with the
introduction of Flash Player 9 and ActionScript 3, although it took a bit for
people to actually upgrade).

I remember it was released at some point, but at the end it didn't amount to
much since most of the JVM's actual implementation relied on everything else
and the "optional" bits were only a small part, so it didn't help almost at
all with applets. And by that time Java applet gaming was dead (it never
reached Flash gaming's heights, but for a few years in the early/mid 2000s it
was still something you could make a living off - if you didn't expect much).

I never heard about it much after that and personally moved away from Java
with Flash becoming dominant on the web game scene and soon with Oracle being
an dick to everyone. The only reason i used Java was for NetBeans' GUI editor
for my tools but with Lazarus [0] becoming very stable by that time and
NetBeans looking and behaving weird in my -then- brand new iMac, i abandoned
Java for good (the only active Java project i still have is a tilemap editor
[1] but this only if you stretch "active" to "hack on it a little every few
years").

I also never liked Java much after Java 1.4 - it moved further away from its
"simple language" roots and i liked how everything was solved in the library
instead of adding extra stuff to the language. Well, that, and things like
generics felt very "bolted on" while annotations like `@override` felt
unnecessary verbosity (and ugly syntax-wise).

My favorite Java was probably around 1.1 or so (i have a book on that too),
some times i think i should make a new language and VM after it but i lose
interest quickly - i think i have a bunch of parsers for Java-1.1-like
languages lying around in my backups, from every time i get that urge :-P.

[0] [http://www.lazarus-ide.org/](http://www.lazarus-ide.org/) [1]
[http://runtimeterror.com/rep/mapas](http://runtimeterror.com/rep/mapas)

~~~
pjmlp
That was the Kernel VM project.

[http://www.oracle.com/technetwork/java/javase/kernel-135055....](http://www.oracle.com/technetwork/java/javase/kernel-135055.html)

[http://openjdk.java.net/jeps/148](http://openjdk.java.net/jeps/148)

