Martijn Verburg here - London Java Community (LJC) representative on the Java Community Process (JCP) Executive Committee (EC) (dear Cthulu, the acronyms) otherwise known as the Java standards body.
Here's our reasoning for voting "Yes" the 2nd time around:
This is a great step forwards for Java as a platform as it lays the foundation for a smaller, more secure and lightweight footprint. There is still more to tackle on the modularity story (expect to see further updates in Java 9.1 and beyond).
It's very important to note that you can still run your normal Java classpath based application with Java 9. The Classpath and Classloader mechanism is still there, although you'll see a few more warnings pop up if your application or its dependencies are relying on a deprecated API or dangerous internal lib.
If you choose to port an existing project to the new module system and module path way of loading then you can gain some of the benefits but of course there will be a learning curve and I'd argue that unless you're really needing that modular help then wait for Java 9.1 or even 10 until some of the edges have been ironed out.
I'm relieved to see how this turned out. Very reasonable path forward. Was concerned that Java 9 would break everything which thankfully that has been averted, yet progress can still continue on making Java more useful with modularity.
They are highly motivated by internal (Oracle cloud) and external factors (Oracle as A technology leader) factors as well as professional pride (core Java folks are ex Sun) so I'm personally highly confident
What gives you this confidence? What kind of Java applications do you expect to run on Oracle could because they seem to target none of the existing ones? I don't see how Jigsaw can help existing Java applications with Oracle could:
They seem to be assuming nobody will use JAXB or activation either directly or indirectly because that will force you to deploy the full java.desktop module with all Swing PLAFs and everything including sound. They seem to be completely fine with this and do not seem to want to spend any time on splitting up java.desktop, having at least a headless version java.desktop or improving JAXB so that it doesn't depend on java.desktop anymore.
They seem to not care about existing Java EE APIs at all and have posted no replacements.
They seem oblivious to any needs of any dependency injection container (Spring, Guice, …).
Jigsaw only helps with disk footprint. It does not help with runtime footprint (which is usually several times larger than disk footprint) or startup time, if anything it makes both of them worse.
What gives you this confidence? The Java EE spec lead seems to stubbornly ignore Jigsaw. To give you some examples.
First and most important is module naming. Because Jigsaw dependencies work on modules and not packages (OSGi found out this was a mistake long ago) and because you can't make a module stand in for an other one the module name once decided can not be changed without breaking everyone depending on it. If you want to use any Java EE API in a Jigsaw application, e.g. Servlet API, you need to have a module name, even if you're using in Java SE. Right now if you're a Java SE application that uses any Java EE API at all you can't move to Jigsaw because none of the module names are set. Some Java EE specs use Java SE module guidelines because they say they are the only ones around (they ignore the ones from jodastephen) and then point the finger at the EE spec lead saying that since there are no guidelines doing what they do is fine. They also wrongly argue that they could change the module name later should naming guidelines be formalised.
Secondly a lot of the objections Red Hat made were because of the difficulties of making existing Java EE APIs run in Jigsaw, they were shut down with "you only want to promote JBoss modules".
And finally the current GlassFish 5 promoted build do not even start on Java 9 in classpath mode with all the internal API access checks disabled.
Lots of noise here about what it actually consists of. This blog post is a good overview[1]
> Creating a module is relatively simple however. A module is typically just a jar file that has a module-info.class file at the root - known as a modular jar file. And that file is created from a module-info.java file in your sourcebase (see below for more details).
> Using a modular jar file involves adding the jar file to the modulepath instead of the classpath. If a modular jar file is on the classpath, it will not act as a module at all, and the module-info.class will be ignored. As such, while a modular jar file on the modulepath will have hidden packages enforced by the JVM, a modular jar file on the classpath will not have hidden packages at all.
Does anyone who has been following this in more detail have a summary of the sort of benefits developers 'at the coal face' will see from this change?
I've read a few articles and they've left me very confused.
Is this about parts of the Java standard library (like the mentions of SQL and logging imply) and meaning it's only relevant for people targeting small devices?
Does it replace JARs, like the mentions of 'JAR hell' imply?
Does it add a maven-style package manager that can resolve dependencies by downloading code from the internet, like the mentions of npm imply?
Or are the changes / benefits something else entirely?
The module system is taking the place of whatever you were previously using to order your dependencies on the classpath.
Now, instead, the module files list the dependencies of a module and, Java does a resolution of all of them to determine the dependency graph. So the order of modules on the module path doesn't matter, unlike the classpath.
The only advantage I know of is that the standard lib has been split in over 92 modules. So if you don't use everything, your resulting Jar will be smaller. Basically, making Java more modular in the sense that packages are now grouped into smaller Jars.
Now Java 9 has some other features, one of them is to prevent reflection access of non public methods. This was made optional for Java 9. But might be made default in 10 and mandatory in 11. This is the biggest issue people have with Java 9 modules, because it will not be backwards compatible, and certain code can't even be ported, since no public APIs exist for certain things.
Finally, most programmer at the cast, like me and you, we're hoping that modules would be version aware, and allow multiple versions of the same module to coexist, but they didn't do that unfortunately.
> So if you don't use everything, your resulting Jar will be smaller
I'll admit that I'm not super well-versed in Java packaging, but I kinda assume that typical jars do not include any bits from standard library, so modularizing standard library wouldn't have any impact on jar size. I'd imagine that the effect would be instead visible for self-contained distributions, like e.g. docker containers.
One of the worst things about the Docker fad is that people are wasting effort minifying things like the JRE and libc so that having many copies on the same machine isn't quite so blatantly foolish. They could be accomplishing something but instead they're wallowing in a problem that was solved decades ago.
> but I kinda assume that typical jars do not include any bits from standard library, so modularizing standard library wouldn't have any impact on jar size.
That is correct.
> I'd imagine that the effect would be instead visible for self-contained distributions, like e.g. docker containers.
In theory. In practice you need to build a JDK contains only the bits you need. Sounds great in theory, you can get rid of the CORBA and GUI modules, that should save you a lot of disk space. In practice the CORBA module is only 2.7 MB and the desktop module is 14 MB and almost every framework indirectly requires the desktop module because Java Beans are in there.
Now Java 9 has some other features, one of them is to prevent reflection access of non public methods. This was made optional for Java 9. But might be made default in 10 and mandatory in 11.
Really? Well, that sucks.
In a large and complex project I'm currently working on, we've embedded a clojure REPL and we use reflection to poke at private data/methods of internal objects. Its been extremely useful for debugging and monitoring the state of the system. If we lose the ability of poking at private stuff, then we won't be able to do this anymore (unless we make everything public or add accessors for everything).
Yes, it was the most controversial change to Java 9. Currently, the compromise was that there's a jvm arg you can set to re-allow reflection access. But the idea is that it will eventually be taken away, so it's a stop gap.
I'm not sure if it will impact the Clojure REPL. I know Clojure is aware of the Java 9 changes, and I think they have some ideas how to keep things working somehow.
There's also going to be a Java Repl now, called jshell. ButI don't it will be given special permissions such as that.
Its not that it will impact the Clojure or the Clojure REPL, but rather that it will affect the utility functions that we wrote to inspect our running system.
For example, we have a function 'inspect' that takes an object and series of Clojure keywords - it converts the object into a clojure map where the keys are keywordized member variable names and the values are its value. It then acts recursively consuming the series of keywords to look of the fields. This makes it trivial to look into deeply nested object graphs. (It also does some other things like pretty printing the results, accessing containers by index etc)
BUT this is only useful because 'inspect' sees all fields: public and private. It does this by using the reflection API to access private fields.
would be the same as the following java (were java to ignore visibility modifiers):
rootobject.someField[0].privateField.foo
or (depending on the type of someField):
rootobject.someField.get(0).privateField.foo
or if, for example, privateField is a map:
rootobject.someField.get(0).privateField.get("foo")
...etc
We don't want to have to make things public that shouldn't be semantically public and we similarly don't want to provide accessors when they are not required, just for the sake of being able to poke at them with our debugging/monitoring tools.
Yeah this modularization will do nothing for your own jar's size.
Basically, if you the developer weren't using OSGI before, and planning to switch to java modules, then likely the only effects you will see are negative ones.
This is really for the benefit of the jdk itself, and tiny-platform installations.
The threat of java 10, and onward -- breaking stuff is the real thing to watch out for.
It does quite a lot. I suspect it basically breaks Maven, since Maven jars aren't valid for Jigsaw. It also breaks code that does things like poke private member variables in other people's classes.
I think the breakage will be significant. I also think it'll be worth it.
> It also breaks code that does things like poke private member variables in other people's classes.
> I also think it'll be worth it.
Yeah, it's fun to stick to abstract design principles, until an important dependency in your project refuses to play ball, in a way that cannot be trivially fixed by simply subclassing something and calling it a day. In a language like Java you essentially have three options - submit an issue and wait months (or years) 'till it gets resolved, fork the project now (if open source) and deal with costs of supporting your own version, or just poke at an appropriate private static variable somewhere and have a working version deployed in an hour. I know which one I prefer.
It just occured to me that the second and third options are more similar than they initially appear. In both cases you've taken responsibility for the library in a new way.
It's no longer a breaking change for the library authors to break you if you're messing with private state, so you'll need to do some vetting of all changes to the library.
But why take away my paddle? Let me have a working solution right away, and when you push an update that breaks my code, let me also deal with fixing it. I'm not dumb, I understand the risk of depending on private APIs.
I understand that it's impossible to both implement a rule and not implement a rule, but what is actually the benefit of disabling this? All nerd sleeps safe at night confident that no one changes their private variables?
Does this mean that things like jmockit's deencapsulation stops working?
There is none, appart from making the Oracle employees life easier. They don't have to deal with the backlash of breaking customers by accident when touching private APIs, so it gives them more liberty. They decided to take the blow in one go, instead of spread over time. Now, the blow was bigger then they thought, so they've retracted and spread the change over Java 9 and 10, by making 9 an optional setting, and saying 10 will be the cutoff.
In an ideal world, it makes sense. This is defensive programming, they're protecting themselves against unplanned usages of their libraries. That's always good when you're a maintainer. You know how annoying it is when you're trying to change and improve things and you realize you can't because someone was using your stuff in a completely dysfunctional way and now you'd break them if you proceeded.
But in a practical world, the world of business especially. You should also know that part of the popularity of your software comes from all those use cases users have managed to somehow satisfy without you explicitly planning and building them. Take those away, and your software also loses a lot of value, until you reimplement all these use cases in a more sane way. Tjis will take time, and your customers might move on to something else. We'll see how this impacts Java.
I'm not going to try to convince you, but getting classpath isolation would be a huge deal for my use case. (We could theoretically do OSGi or similar, but that's quite a heavy-handed approach to what should be a simple -- even "default" -- thing).
I know the risks, but in this case, the private methods' implementation haven't changed, their signature hasn't, it's a completely artificial restriction.
That's where the frustration comes from. The code is there, you were allowed to use it, and now you are not, even though the code is still there.
If they had refactored, or oprimized, I would understand my code being broken. But this seems more ideological then practical.
> Well, you knew the risks, so I'm not sure why you're so upset that Java 9 precludes you using what were ultimately private APIs.
I think this is going too far. You have to jump through enough hoops already to mess with private state of library code; if some developers are dumb enough to do that irresponsibly and get hurt, let them. Don't take away a last-ditch tool those of us who know what we're doing sometimes really need.
Actually, there's an easier way. If you load the class as a resource, it's possible to still access the private methods. Not sure of the details, but I read a blog somewhere who was showing how to do it.
I wasn't serious, of course. Well, I have edited binary executables in Emacs on very rare occasions, but I"ve never tried to edit a classfile. I think there are classfile-modifying libraries one can use if one has to get that down and dirty.
> I suspect it basically breaks Maven, since Maven jars aren't valid for Jigsaw.
What do you mean by that? Maven JARs are just normal JARs downloaded with Maven. They are therefore valid in both the classpath and the module path (as automatic modules). Jigsaw modules are just normal JARs with a module-info.java, there is nothing preventing you form managing them with Maven. In fact in already have several of them in Maven Central, they went through the SonaType quality check without issues. In fact the compiler plugin will detect if a dependency in a modular JAR and if so put in the module path rather than the classpath (the javadoc plugin does not yet).
Modules add another layer above the current access control constructs in Java world (public, private, etc). There are several benefits to them, like discovering a missing dependency much earlier on (statically), not importing the code we are not using, avoiding cyclic dependencies between modules, etc. More importantly it enforces better design and provides some tooling to have a better 10000-feet view of your project. I think its going to come in very handy in managing larger codebases.
You could visit these links to know more from a web-dev backend point of view.
Ps... I created a library in Nodejs that implements concepts of modularity in JS codebases. Its known as archiejs (on github). I think modularity is a very useful feature that lot of people are going to want to use in Java (and also other programming languages).
ETA: That being said, I think those who campaigned the hardest against Jigsaw with technical arguments were wrongheaded (e.g. "Just do it like OSGi, it works fine.")
How about the whole module versioning thing? I get the impression that the real discussions and negotiations happen in private and even the somewhat technical messages in the mailing list are only the tip of the iceberg.
It looks like none of the technical problems that resulted in it being voted down last time have been addressed?
I liked this comment about #MultipleModuleVersions:
"...Addressing this issue may entail reconsidering the multiple versions non-requirement. [Mike Hearn]
"Resolution Defer a specific solution of this problem to a future release since effective, though crude, solutions already exist and the lack of an immediate solution should not block developers who want to modularize existing applications. [Proposal]"
I'm the Mike Hearn named in the document. I think they probably made the right call here.
The stated justification for the decision goes like this. With the current design of Java/the JVM, the only way to load two incompatible versions of the same module is to use different classloaders. You can in fact do this with Jigsaw. There is an API that can be used by an app loader or container of some kind to put every module into its own classloader.
This leads to an obvious question: if doing that lets you eliminate version conflicts, why not do it by default? The reason is compatibility. It turns out that lots of software makes assumptions about the classloader structure and which classloaders load it. They tried doing it that way early in Jigsaw's life and backed off because too much stuff broke.
It may seem odd to worry about breaking apps when Java 9 spent most of its development period unable to run basically any real Java software due to the widespread reliance on internal APIs, but:
• They backed off on that, one of the changes that let Jigsaw get voted through this time is that Java 9 no longer breaks apps that use internal APIs by default.
• Encapsulating internal APIs is at least a change that was always signposted as being possible - developers were warned not to use them in Java docs from decades ago. Whilst the classloader structure isn't something guaranteed by the spec (as far as I know), changes to it were at least not widely telegraphed in advance.
Ultimately, backwards compatibility concerns have won the day. So Java will not allow conflicting modules to be loaded by default. If you find yourself in a situation where you need that, there's an easy fix: change your app's startup code to use the Layer API to put every module in its own classpath and fix any modules you use that make now-invalid classloader assumptions.
> It turns out that lots of software makes assumptions about the classloader structure and which classloaders load it. They tried doing it that way early in Jigsaw's life and backed off because too much stuff broke.
Do you have a source for this? I have trouble accepting this argument for two reasons. First WilFly 7 and later (modular service container) completely changed the classloader layout to a non-traditional, non-hierarchical one. The amount of breakage introduced was negligible. In my experience the biggest source of breakage was read-only JNDI which is completely unrelated.
Second they (Oracle people working on Jigsaw) were completely unaware of the amount of breakage encapsulating internal APIs would introduce and had to back out of it at the last minute (i.e. this year). This suggest to me they did at best minimal testing, they certainly never even started a hello world Spring or Guice application.
My recollection from one of many Jigsaw talks that Mark Reinhold gave at was they would have to implement a SAT solver (like OSGi) and didn't want to do that because reasons.
My source is a comment to that effect on the JPMS/Jigsaw mailing lists, from Mark Reinhold. I don't have the link off hand unfortunately and pipermail is from the dark ages so searching for it again would take time.
I think the Oracle guys were aware of how much would break when they blocked access to internal APIs, but felt the costs were worth it and/or that people would adjust their code during the Java 9 development cycle so the breakage would be less by the end.
Another cited reason was simply getting to the point of shipping something. Java 9 is already quite late. I can see why they'd punt features to future releases to try and ship what they have.
Multiple module versions was one of the single greatest advantages of OSGi. I don't understand why this was considered a non-requirement except because it was a "difficult" problem... which OSGi had resolved.
Module isolation should actually mean that you can have multiple versions of a set of classes (a module) available in the system (JVM process) and each gets their own classloader. Dependency expressions permit the deterministic wiring of these classloaders to each other... Simple.
Why wasn't the OSGi spec imported (at least in part) again?
Great to see this happen. It was supposed to be one of the big features of Java 8 and to have it pushed to 10 because of last minute squabbling and objections that had never been raised before seemed ridiculous.
Your comment is disingenuous, this was not last minute minor squabbling as you pretend, instead objections raised during the previous ballot (https://jcp.org/en/jsr/results?id=5959) were serious and long-standing (especially RedHat's position was not new) and delaying the project to address them is a good thing and the very reason the JCP exists, in the interests of the Java community.
Martijn Verburg here - London Java Community (LJC) representative on the Java Community Process (JCP) Executive Committee (EC) (dear Cthulu, the acronyms) otherwise known as the Java standards body.
Here's our reasoning for voting "Yes" the 2nd time around:
https://londonjavacommunity.wordpress.com/2017/06/28/our-yes...
This is a great step forwards for Java as a platform as it lays the foundation for a smaller, more secure and lightweight footprint. There is still more to tackle on the modularity story (expect to see further updates in Java 9.1 and beyond).
It's very important to note that you can still run your normal Java classpath based application with Java 9. The Classpath and Classloader mechanism is still there, although you'll see a few more warnings pop up if your application or its dependencies are relying on a deprecated API or dangerous internal lib.
If you choose to port an existing project to the new module system and module path way of loading then you can gain some of the benefits but of course there will be a learning curve and I'd argue that unless you're really needing that modular help then wait for Java 9.1 or even 10 until some of the edges have been ironed out.
Any other Q's throw my way!