
How We Found a Missing Scala Class - drob
https://heapanalytics.com/blog/engineering/missing-scala-class-noclassdeffounderror
======
koube
FYI this domain is blocked by default for uBlock users.

~~~
kazinator
Wild-assed kazinator guess: probably be a substring/regex match on
"analytics". Must be a tracking domain!

~~~
manigandham
It's on the Pete Lowe Adserver list directly, since Heap is in fact a tracking
system and domain:
[https://pgl.yoyo.org/as/serverlist.php?showintro=0;hostforma...](https://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=hosts)

------
drob
Heap CTO here – would love to answer any questions you have.

This was my first exposure to btrace, which a super useful swiss army knife
for JVM debugging. That made this a worthwhile adventure for sure.

~~~
andonisus
Great article. I had been hunting down a similar issue in Java 8 with maven
dependencies. I was getting the same error, but confirmed that the dependent
jars were correctly included on the classpath. I eventually gave up and
decided to take a different programming approach that did not include these
missing classes, but I think I will revisit it to see if a class loader is
getting closed somehow.

~~~
drob
Ooh, check it out and let me know what you find! It would make me really happy
if this post helped someone debug something when they had previously hit a
dead end.

------
userbinator
_NoClassDefFoundError? But it’s right there!_

Although in this case the cause was very different, it reminds me of an old
"trap for young players" with loading shared libraries dynamically --- the
library itself can exist and be readable and executable, and yet attempting to
load it fails with a "file not found" error. This happens when one of its
_dependencies_ , directly or indirectly, is missing.

~~~
Lazare
I ran across another similar-yet-very-different example of this once in a
completely different language.

In my case, we added a new class, it worked fine on dev, then failed on
staging (and would have failed in production if we'd let it go that far). This
was confusing, because we were using Vagrant to ensure our dev and staging
environments were identical. What could be going on?

Well, our linux VMs were being _hosted_ in OS X, with shared folders for the
code, and by default OS X volumes are _not_ case sensitive. Meanwhile the
linux staging and production environments were using actual linux filesystems,
which _were_ case sensitive. So someone added a new class MyFancyClass, then
tried to import it as MyFancyclass, and it worked great in dev since the
underlying FS of the host OS could find the file, then failed on staging.

A fun debugging ride, and a good reminder that 1) having dev and staging the
same is really important and 2) that might be harder than you think. :)

~~~
mcherm
> a good reminder that 1) having dev and staging the same is really important

Actually, I think it proves that having staging and PROD the same is
particularly important.

~~~
Lazare
Oh yes, that too!

------
SuspiciousSwan
It sounds like you have a lot of operation issues due to the technologies that
you used. I mean, at least you aren't doing your backend in node, but running
an actor system on top of an actor system is going to be brutal to properly
analyze once you actually have scale.

What sort of process do you have for picking trendy technologies vs tested
ones, and how much do you talk to people who have built large scale systems
before implementing things like scala?

~~~
jjjensen90
Yeah... Reading this, it smacked of a possible combination of poor tool choice
and over-engineering (which I've been guilty of plenty). I built a video
processing/workflow application in Scala with Akka a few years ago and
debugging that was hard enough, eventually it was refactored to a simpler
Kotlin/Spring application... Actor systems are great for certain use cases but
you can really hurt the transparency of your app if you aren't careful. I
can't imagine maintaining the OP's application at scale for this use case, but
maybe they have someone smarter than me!

~~~
rozap
Counterpoint: debugging erlang systems in production is a cakewalk. The
tracing and introspection tools that come bundled in OTP make tracking
problems down really easy. It's _really_ hard to go back to systems that don't
have erlang level visibility, so much so that it's kind of a crutch sometimes.

This is an ecosystem problem and not something inherent in a program using an
actor abstraction.

~~~
jjjensen90
Ah yes, that is a great point. Erlang/OTP were designed to be used as actor
systems, whereas the actor implementations in Scala and other JVM
languages/frameworks are at least one level of abstraction above that.
Definitely agreed on Erlang/OTP having a wonderful set of tools for
debugging/visibility, but I still stand by my assessment that OPs problems are
from over-engineering (and secondarily from the ecosystem).

~~~
rozap
Definitely. In the handful of instances I've seen akka used, regular jvm
abstractions would have been simpler and more coherent.

------
nambit
Why doesn't java just spit out a classLoaderClosed error?

~~~
th3iedkid
Yes, believe that was the bad part from JMV implementation!

------
GrumpyNl
I keep running in this type of problems all the time with our developers.
Please keep it simple. Take a step back and ask yourself, do i need all this
stuff, is this the best approach. Often they just blindly accept all the
external libs. For me as an old school guy, i don't trust all those
dependencies at all.

------
fgheorghe
How do you lose a class in a programming language?!

~~~
mb720
The title is misleading. The class was there all along, the
NoClassDefFoundError was thrown because the class loader was closed when
trying to load a class.

~~~
saati
But why can you even do that?

~~~
rad_gruchalski
It says right there in the article.

------
djsumdog
The moment the article mentioned "Fat jar" I knew that'd be the problem.

I don't recommend using any type of fat jar plugin (like OneJar) or even
Google Guice for that matter. Custom class loaders are a nightmare.

Thanks to Docker containers, you should never really need a far jar again.
Just find a decent Docker packager for your build system (sbt, gradle, etc.)
and it can plop all your dependencies in there in a nice, isolated container
that uses the standard class loader.

~~~
thecatspaw
What is problematic about Fat jars?

The problem seemed to be Flink's implementation to unload the FatJar's
classloader when erroring. This would have happened with slim jars as well,
wouldnt it?

I also dont see how docker relates exactly, you can have hundreds of library
jars in a classpath with standard classloaders, no docker required

~~~
tjoener
Also Scala's lambdas create new anonymous classes, but Java's lambdas are kind
of bootstrapped static methods in the existing class file.

It does get converted to a class at runtime, but then it's long past the
classloader anyway.

~~~
yawaramin
Scala 2.12 lambdas compile to Java 8 lambdas.

