
Removal of Unsafe in Java 9 – A disaster in the making - pdeva1
http://blog.dripstat.com/removal-of-sun-misc-unsafe-a-disaster-in-the-making/
======
geofft
The quote stops short, and for someone like me who doesn't follow Java very
much, it feels like it's misquoting via omission:

"Let me be blunt -- sun.misc.Unsafe must die in a fire. It is -- wait for it
-- Unsafe. It must go. Ignore any kind of theoretical rope and start the path
to righteousness _ _/ now/__. It is still years until the end of public
updates to JDK 8, so we have / _years_ /to work this out properly. But
sticking our heads in the collective sands and hoping for trivial work arounds
to Unsafe is not going to work. If you're using Unsafe, this is the year to
explain where the API is broken and get it straight....

"Please help us kill Unsafe, kill Unsafe dead, kill Unsafe right, and do so as
quickly as possible to the ultimate benefit of everyone."

The quoted engineer understands that people use it, claims that there's no
immediate risk (since Java 8 will stay around), and very very strongly implies
that there is _a plan_ to move from Unsafe to something better than Unsafe
providing equivalent functionality in a stable way, and asks for help with
that. Is this correct?

~~~
pdeva1
The post directly links to the full quote. If the intention was to misquote
that would not be the case.

> there is a plan to move from Unsafe to something better than Unsafe

It is this plan and the implications of it that the post is discussing

~~~
scott_s
The _intention_ may not have been to misquote, but that is the effect that I
saw. I had a similar reaction that geofft did when I read the whole email: it
sounds like a reasonable effort to start the process of moving to something
better than what is currently available. The submission here does not seem to
take that into account.

------
sreque
There's a lot of backlash here against the author with comments like:

"You used a sun.* package? You deserve to die in a fire as well!"

These comments look like they are coming from people who have never used
Unsafe. First, Unsafe provides some immensely useful features to the Java
ecosystem. Second, there is no alternative to Unsafe on the JVM. That includes
JNI (native code integrated within the JVM).

So far, Oracle has stated their intent to remove Unsafe from Java 9. To all
the users of Unsafe, Oracle is basically saying, "tell us your use cases for
Unsafe, and if we decide they are valid we may try to create an alternative
for you. If you're lucky, we may even ship that alternative in Java 9 or 10!
Have a nice day!"

Many people, myself included, find this unacceptable. For instance, it looks
like several groups are currently collaborating on a document discussing uses
cases for Unsafe and whether alternative features satisfying those use cases
are expected to appear in Java 9:

[https://docs.google.com/document/d/1GDm_cAxYInmoHMor-
AkStzWv...](https://docs.google.com/document/d/1GDm_cAxYInmoHMor-
AkStzWvwE9pw6tnz_CebJQxuUE/edit?pli=1)

As you can see, the majority of the cells in the column "Expected in Java 9"
flat out say "no". There is a good reason people are up in arms about the
removal of this API.

Personally, if I had to choose between Oracle keeping Unsafe and Oracle
implementing Java 8 Lambdas, I would have picked Unsafe in a heartbeat. At the
end of the day, Lambdas are mostly syntactic sugar, especially in their
current implementation. Unsafe, on the other hand, actually empowers you to do
something more.

~~~
pjmlp
Except that those of us on the Java community that follow Java development are
fully aware of the 2014 survey, done by Oracle about how Unsafe is being used
and what is our opinion on it.

Of course, many HNers that hate Java aren't aware of those discussions and
post comments without knowing the full background.

~~~
MichaelGG
What is the purpose of enforcing module boundaries? That appears to be why
none of these internal classes will be accessible. Outside of a sandboxed
environment, who cares what you do? Is it for some magic optimization? Or just
some folks on a rampage? (The quote from the guy saying it needs to die, and I
read the mailing list, well it seems to be baseless.)

~~~
pjmlp
One of the purposes of Jigsaw is offering a way to define a public API based
on multiple jar files, similar to what OSGi does.

Similar in concept how Assembly Modules work in .NET, with friendly internals,
which incidentally not many devs know about.

In Java's case a linker will be part of the SDK, so that only the Java code
required for a given application will be deployed.

Since Oracle needed to clean up dependencies to implement Jigsaw, they are
taking the opportunity to also remove most internal APIs, Unsafe being one of
them.

They are also in the process of replacing JNI with something more programming
friendly, P/Invoke style, but it will only come in Java 10.

There is also the reason that in a memory safe language every use of JNI and
Unsafe is another security bug waiting to happen. If the compiler/runtime are
able to offer safe APIs with similar performance, then those exploits can be
avoided.

------
ante_annum
So the explanation is:

    
    
      Let me be blunt -- sun.misc.Unsafe must die in a fire.
      It is -- wait for it -- Unsafe. It must go. Ignore any kind of theoretical rope and 
      start the path to righteousness
    

And what the author takes away is:

    
    
      This engineer hates the Unsafe class for no reason at all.
    

The reason the engineer hates it is that it's unsafe. Disagree with it, sure.
But it's a reason, and it's a good reason (imho). I understand that it's
commonly used by apps I use, but it's hard to keep reading after this kind of
knee-jerk.

~~~
snuxoll
In addition, it's in the sun.misc package, if it's not in java.* or javax.* it
is ripe for pruning, you'd think people would have learned by now to stop
using internal classes since pretty much every JRE/JDK release known to
mankind has changed them or removed them.

~~~
kasey_junk
The problem with that, is unlike most of the other internal classes, there is
_zero_ other way to do the things people are doing with unsafe without it.

It's not a matter of expedience, it's a matter of necessity.

~~~
dlubarov
Yes, removing Unsafe would make things like PowerMock and Robolectric
impossible to (re)implement. So what? Those libraries never should have been
possible, and we can live without them.

Languages need to be selective about what features they support, and Java was
never intended to support things like mocking static methods, or creating
instances without invoking any constructors.

~~~
kasey_junk
Without the current unsafe package most performance sensitive applications
(whether throughput, latency, or pause sensitive) will have no recourse to
meet their current performance levels.

That will leave people to choose between staying on Java 8, worse (and often
unacceptable) performance, and migrating to a different language.

At this point, there is a very good chance if your library/application is
known for performance and exists on the JVM it uses unsafe and without a
migration plan all of us who care about performance will likely have to move
off the language.

~~~
dlubarov
With some effort, I'm sure the folks behind Cassandra, Hadoop etc. could
achieve acceptable performance with allocateDirect(). It doesn't support
explicit freeing, but you can always reuse buffers and write your code to
allocate slices within them.

------
ZitchDog
Entire companies exist around functionality provided in Unsafe. I don't see
this going well if they remove it without providing access to the
functionality it gives.

Java has plenty of problems, but backward compatibility over the years has
been simply amazing. This would chip away at one of the few legitimate reasons
to use Java in 2015.

~~~
geofft
Entire companies existed around DOS extenders and TSRs. Microsoft
rearchitected and everyone is better off for it.

The argument needs to be that the use of Unsafe by these companies (or the
projects listed on the blog post) both is good and has no alternatives, not
that the use exists.

~~~
taeric
There should be some argument made that the use by these companies is causing
trouble, as well. DOS had real identifiable reasons not to use it. The
rhetoric used so far just has Unsafe upsetting some engineers for existing.

------
carsongross
Once again, The Platonic Engineer gazes hatefully at the gronky, ugly and
awkward things that make things work in real life, rub his hands together and
says...

"Burn it to the ground."

~~~
xg15
_Or_ you could actually stop for a second and find out _why_ your current
model needs so many ugly and awkward workarounds and try to adjust it
accordingly - which seems to be what Oracle is trying to do in the long term.

~~~
taeric
This is an odd one, though. I can not remember seeing/hearing any real issues
with Unsafe. For the most part, people don't use it. There are a few that do,
to ridiculous benefit. As things stand right now, without it, you _can not_
get anything close to the performance people are seeing with it.

So, what many of us are seeing here is someone is upset with basically the
shape of an API. They have no evidence that I have seen of trouble caused by
it. Only a desire that such things not exist. This feels almost puritanical in
how it is being exercised. Which is just plain silly for a technical debate.

~~~
carsongross
_puritanical_

I think the correct term is platonic: each of us has an Inner Platonic
Engineer who cannot abide the ugliness of real world systems and insist on The
Great Rewrite. A lot of progress relies on that engineer.

However, in large, functioning and mature software systems he becomes the
enemy.

~~~
taeric
I question just how much progress relies on that engineer. Be honest here,
there are no identified problems that are being solved here. Merely an
undesirable in the system that is very effective at its job.

Progress, on other hand, advances something in the way of the problem space.
Not merely rearranges something in the solution space.

------
jart
I just read an article that says in order to even use Unsafe, you have to use
reflection to unwind the stack and go searching for a reference to the Unsafe
instance, because the Java authors worked so hard to make it private. So it
seems to me that they'd be well within their rights to remove the class.

Normally when you go to that level of epic hackery, you can't expect for
official support or backwards compatibility. So the author of this article
using scare tactics to keep that feature seems rather disingenuous.

~~~
orf
Could you provide a link to the article? I'm not saying you're wrong (I'm not
a Java guy) but that sounds convoluted.

~~~
twic
This is how you get access to Unsafe:

    
    
      Field theUnsafe = Unsafe.class.getDeclaredField("theUnsafe");
      theUnsafe.setAccessible(true);
      Unsafe unsafe = (Unsafe) theUnsafe.get(null);
    

Reflection, but no stack unwinding.

------
pron
Chill.

There is no disaster in the making. Oracle is very much aware of the issue,
and is actually already taking the very same steps suggested by the author. It
has assembled a public working group that is coming up with a spec for a
public "Unsafe" API[1] (that is safer than Unsafe).

 _In the meantime_ , use of Unsafe in Java 9 will be possible with a command-
line flag in order to allow the working group more time to create the new API
and for Unsafe consumers to migrate, and of course to allow existing libraries
to run on Java 9.

[1]: [https://docs.google.com/document/d/1GDm_cAxYInmoHMor-
AkStzWv...](https://docs.google.com/document/d/1GDm_cAxYInmoHMor-
AkStzWvwE9pw6tnz_CebJQxuUE/edit)

~~~
jryan49
The point is if you have to pass a flag to allow the use of Unsafe, you're
breaking "works-out-of-the-box" backwards compatibility.

~~~
snuxoll
Backwards compatibility is only guaranteed for official Java SE and Java EE
API's existing in the java.* and javax.* packages, anything in sun. _,
com.sun._ , etc is internal to the JDK and has no such guarantees.

Every time you hear of an application that only works on a specific release of
the JRE you can know for sure it's because they use undocumented internal
API's — keep in mind this not only impacts compatibility with other releases
of OpenJDK/Oracle JDK, but the sun.* API's are not typically part of third-
party JVM's like JRockit or the IBM JVM.

Stop using internal API's and then bitching when Oracle makes changes to them,
you only have yourself to blame.

~~~
smacktoward
This is exactly the same line Microsoft took when they broke poorly
architected applications with the new (and much better!) security model in
Vista. "You're the one who wrote an app that needlessly requires root
privileges, you have only yourself to blame."

It... didn't go well.

~~~
MichaelGG
And MS did a lot of work to detect such apps and make them work OK. What's the
real argument against Unsafe? People are upset that it exists? Really?

------
kkmickos
Many people seem to be upset with Oracle, but I think they should be upset
with the vendors who ship applications/libraries that depend on the "sun.*"
packages.

Oracle (then Sun) have since _1998_ strongly discouraged the use of them:
[https://web.archive.org/web/19980215011039/http://java.sun.c...](https://web.archive.org/web/19980215011039/http://java.sun.com/products/jdk/faq/faq-
sun-packages.html)

~~~
Someone1234
Fine, but why isn't the argument: "This then needs to be standardised in a
Java.* package and correctly documented."

Unsafe is useful, and is evident by how popular it is. Heck even Sun/Oracle
must agree because they created it for their own internal usage. So why not
just go ahead and standardise it.

------
ihuman
What is "Unsafe?" Is it unsafe, as its name implies? If it is unsafe, why to
so many programs use it?

~~~
dikaiosune
It's part of the OpenJDK/Oracle VM implementation, not an official part of the
Java standard library. It allows for all sorts of fun stuff like native
compare and swap operations (used for concurrent collections iirc), object
allocation without calling the constructor, accessing blocks of memory
directly without object references, and other things. The reason it's called
unsafe is because it allows developers to bypass a lot of the safety
guarantees that java is supposed to provide as a memory managed language. But
it's useful for getting more performance out of the VM.

There's currently an effort underway to push through a variety of Java
enhancement proposals that would add a lot of the unsafe functionality to the
official standard library. This would enshrine the unsafe functionality as
official, remove access to some of the more dangerous and less useful parts of
unsafe, and would also mean that the functionality would be properly
documented, which it is currently not.

~~~
Daneel_
Thanks for the thorough explanation - you summarised the issue well, as well
as explaining the impact and flow on effects.

I have to say, I agree with the engineer.. I'm cringing just knowing this sort
of practice is seen as acceptable, and it reinforces my distaste for Java. If
you have to resort to insecure practices to achieve performance, there's
something wrong with the framework.

~~~
alextgordon
Reading this thread as C programmer has me scratching my head.

 _Of course_ you have to resort to low-level functionality to achieve
performance. That's how computers work. We can't all live in fuzzy wuzzy land.

Surely Unsafe is safer than writing the whole thing in C and using JNI?

~~~
trb8
Yes. Also there are many "safe" operations on "unsafe". Such as memory fences.

~~~
vardump
Multithreading is somewhat unsafe in general. And so is ability to do most
I/O, you might overwhelm the system.

~~~
trb8
Sure. Unsafe in terms of program correctness. By that measure, new Thread() is
unsafe.

But that's not a reasonable definition of "unsafe". This could crash the JVM
is a reasonable definition. And fences and ordered instructions cannot.

------
sbov
This seems a bit hyperbolic. Especially the comparison of Python 2 to Python
3. Its not like you could run Python 2 and Python 3 under the same interpreter
by passing a flag.

------
hyperpape
Why on Earth is it a big deal that you need to pass a flag to the JVM at
startup?

I understand that backwards compatibility is valued, but if I went to my CTO
and said "we're gonna drop Java 9 in production, and we're not gonna test
beforehand"...the reaction would not be pretty.

------
exabrial
Document it, make it public. It's very much needed!

------
jkot
Compression in my library is 3x faster with unsafe. I think it is lesser evil
then native code integration.

------
clamprecht
At least he didn't name the article "Unsafe considered harmful."

------
xg15
> _Oracle plans to remove sun.misc.Unsafe in Java 9. This is an absolute
> disaster in the making and has the potential to completely destroy the
> entire ecosystem around Java._

You know your language has a problem if the entire ecosystem threatens to fall
apart as soon as you actually try to enforce the language's conceptual model
and try to make use of the guarantees it provides.

~~~
bitmapbrother
With the caveat that you need to believe in the hyperbole of a chicken running
around with its head cut off.

------
bowyakka
Ok moving past the rhetoric of don't use internal classes, there is a value to
sun.misc.Unsafe. It has been well known by Oracle for some time that it has to
go away right.

Realise that those of us that do use it, are fully aware that it is a hack,
and that we should be doing cleaner things. However know that those cleaner
things _dont exist_; and that you, Dear application developer might not quite
realise some of the shadow puppetry in making your favourite library work,
please lay off the bile.

As others have said, this blog post stops short, the full quote is part of a
very long and ongoing process to rid ourselves of Unsafe, replacing its majors
usages with new API's.

This is a process Oracle started last year after the previous
sun.reflect.Reflection.getCalleeClass mess. Yes, it no one should have been
using an internal class. This method did not go away; it was made only
available to the JVM internals, locked out from end users. The upshot of this
was that, with no sane alternative, suddenly logging became between 2x and
100x more expensive where class names become involved. The JDK team somewhat
back peddled on this change, and we are stuck here in a strange limbo land.

Was it right of Oracle to do this? Absolutely, everyone who goes down the
internal package route is fully aware that these are open-state secrets, sure
we all know them, but we should not be talking about them. Support for these
damn things requires version to version checking, hideous reflection hacks and
being prepared to read a lot of JVM source code for when bugs occur.

In light of all of this, and being the general level of awesome the JDK folks
are they started a very direct community outreach to remove safely Unsafe.
Last year they publicly asked people on the mailing list, and privately via
email to do a survey ([http://mail.openjdk.java.net/pipermail/core-libs-
dev/2014-Ja...](http://mail.openjdk.java.net/pipermail/core-libs-
dev/2014-January/024650.html)) on usages of Unsafe. I know this because I got
an email from Paul Sandoz asking me to comment on Unsafe abusages. Out of
previous thinking from the JDK folks, as well as from the aforementioned
survey, we have a bunch of JEP's for improving things regarding Unsafe.

Where things have gone slightly sideways is that we have, currently no unified
working group to deal with these changes. What this document is, is a proposal
to make a working group, focused on getting the right changes into the JVM. In
the same ways as Project Jigsaw is for modularising the JVM (and is indirectly
responsible for removing Unsafe), or the MLVM project handled making the VM
better support other languages.

Unsafe is dead, long live Unsafe! I fully expect that we will achieve the
right changes in during Java 9 to allow those of us that need to break the
rules, to do it in a supported fashion. I am _looking_ forward to the day when
I can do setMemory or getAndSetInt via a supported API, and not one where I
have to know the words Unsafe.theUnsafe. The above is a storm in a teacup, its
open-source democracy at work.

------
karavelov
What about shipping sun.misc.Unsafe as 3rd party package? Is there anything in
it that can't be done as through JNI?

~~~
vardump
It'd be unusably slow. "sun.misc.Unsafe" compiles in the instruction stream.
JNI implementation would need to go through a complicated calling mechanism. I
think it'd be up to 100x slower.

~~~
karavelov
At least in OpenJDK it is in fact implemented as a JNI package, so it is
already going through the JNI calling convention:
[http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/s...](http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/share/classes/sun/misc/Unsafe.java)

~~~
vardump
I think these are a strong hint they're not truly JNI:

Line 61:

    
    
      * <p> Most methods in this class are very low-level, and correspond to a
      * small number of hardware instructions (on typical machines).  Compilers
      * are encouraged to optimize these methods accordingly.
    

Line 90:

    
    
      /// peek and poke operations
      /// (compilers should optimize these to memory ops)

------
vbezhenar
Why libraries need Unsafe, after all? What problems does it solve for them?

~~~
kasey_junk
It's almost entirely about performance. Unsafe allows you access to concurrent
intrinsics as well as memory layout functions.

It's true that it is widely used, but the people that use it also understood
the ramifications of using it. If you were trying to make jvm agnostic code
you didn't use it. If you were trying to make code that could run on a variety
of jvm versions, you didn't use it.

A lot of what people use it for is being rolled into documented/standard
libraries and this is just the migration pains coming out.

~~~
vbezhenar
Why those concurrent intrinsics are not available with "safe" code? They are
not compatible with Java Memory Model? They are not implemented at some
platforms? What's so unsafe about concurrent primitives? There is API in
java.util.concurrent.atomic, for example.

I understand that Unsafe is used for direct access to memory, but why this
direct access is needed? May be there are other ways to solve those problems.

~~~
benjaminjackman
AFAIK there aren't other ways to get the performance required to be
competitive in the financial space. People that are using these libraries
don't want to use undocumented, Unsafe code. However, their hands are forced
given the races they are competing in. They could switch to C++ but that's not
really a great option for firms that have java code bases going back 10+
years.

Plus, personally, and for a lot of others, the jvm plus a dash of
sun.misc.Unsafe, is a lot better than going over to C++.

More concretely, Here are two sets of trading libraries that use unsafe for
direct memory access: [https://github.com/real-logic/simple-binary-
encoding](https://github.com/real-logic/simple-binary-encoding)
[https://github.com/OpenHFT](https://github.com/OpenHFT) (various libraries in
here use it, once you drink the Unsafe Kool-aid it's hard to turn back from
the performance benefits you obtain).

If Oracle just rips off the Unsafe band-aid without a replacement, the
business decision for these firms is still sound. They will just stay on 8 and
start making migration plans (likely to the C++ bandwagon).

However, It sounds like Oracle plans to give a sane alternative which is good
and what everyone using Unsafe would prefer (along with deprecating it the
meantime to give them some time to adapt).

One real world example: The CME (Chicago Mercantile Exchange) reworked the
entire way they distributed market data (v3 of their Market Data Protocol) to
mirror the cutting edge in performance oriented serialization libraries (e.g.
Cap'n Proto). They also commissioned an Open Source library for parsing that
data (real-logic SBE), it uses unsafe to approach the performance of C++ code.

~~~
kasey_junk
Exactly. Though I'd add that its not just the financial space that is doing
this for performance. Lots of people that are trying to scale high throughput
systems use unsafe for similar reasons:

Akka for instance uses it:
[https://github.com/akka/akka/blob/0de9f0ff40fc5e43540df58718...](https://github.com/akka/akka/blob/0de9f0ff40fc5e43540df58718ecb1818df35673/akka-
actor/src/main/java/akka/dispatch/AbstractBoundedNodeQueue.java)

------
rebootthesystem
This is related and unrelated and maybe even meta to some degree:

There are a couple of companies that are guilty of using their free software
update mechanism to try and trick people into installing crap they don't want
(and might even mess-up their systems): Sun and Adobe.

I almost happened to me last night as I updated Flash on a test system and
some bullshit anti-virus crap-ware was going to be installed because I was
moving fast and neglected to un-check the check-box on the site. I was able to
stop the install process and start over. No harm done.

These companies need public shaming to stop using this bullshit method to make
a few more bucks. Are they so hard-up that they need to trick users to install
crap to make money? I sure hope not.

Sun, Adobe, please STOP trying to trick users into installing crap-ware they
were not looking for in the first place.

~~~
kasey_junk
You understand of course that Sun has been out of business for a very long
time right?

~~~
rebootthesystem
Sorry, brain fart, I meant Oracle

------
scubaguy
I read this article yesterday and I really dislike how it is worded. The "this
engineer" has a name, Donald Smith, this is what he said:

"If you're using Unsafe, this is the year to explain where the API is broken
and get it straight.... Please help us kill Unsafe, kill Unsafe dead, kill
Unsafe right, and do so as quickly as possible to the ultimate benefit of
everyone."

It reads like a request to have a discussion on Unsafe, to talk about it and
find solutions. Unfortunately, I don't see anyone pointing out why Unsafe
should be kept around on that thread
([http://mail.openjdk.java.net/pipermail/openjfx-
dev/2015-Apri...](http://mail.openjdk.java.net/pipermail/openjfx-
dev/2015-April/017028.html)).

And this quote from Oracle shows an effort to do things right:

[http://mail.openjdk.java.net/pipermail/discuss/2013-October/...](http://mail.openjdk.java.net/pipermail/discuss/2013-October/003162.html)
"This is definitely something we plan to propose for SE 9. I expect to see a
JEP from the current maintainers of sun.misc.Unsafe..."

This ([http://blog.codefx.org/java/dev/how-java-9-and-project-
jigsa...](http://blog.codefx.org/java/dev/how-java-9-and-project-jigsaw-may-
break-your-code/)) seems like a more balanced article on the situation.

"So far we focused on the problematic aspects of Project Jigsaw. But that
should not divert from the exciting and – I think – very positive nature of
the planned changes. After reading the documents, I am impressed with the
scope and potential of this upcoming Java release. While it is likely not as
groundbreaking for individual developers as Java 8, it is even more so for
everyone involved in building and deploying – especially of large monolithic
projects." near future. (This is far too small to require a whole JSR.)"

------
razorsese
Literraly meme language

------
skrowl
Still using Java in 2015 – A disaster in the making

~~~
bitmapbrother
What would a MS fanboy suggest as an alternative?

