Yup, and work super aggressively to restrict it -- back when the JVM was supported, a large amount of the browser sandbox had to be extended to support the arbitrary behavior of the JVM, subsequently moving it out of process still required providing unrestricted access to OS components that you would rather they didn't.
At a very basic level it can be summed up with: the JVM approach to GLSL was to just throw it at the GPU, whereas the browsers worked on restricting WebGL to a super constrained subset (this was my fault, but is the correct thing to do :D )
By the time browsers were considering out-of-process plugins, the JVM was already thoroughly eclipsed by Flash. I think Flash's general brokenness (over half my browser crashes/hangs were caused by Flash) was the main motivation for out-of-process, not JVM's hugely broken sandbox.
Applets were eclipsed because of their monumentally expensive launch requirements - both time and memory.
The applets also made it hard to do a bunch of the simple games that were super popular - flash included a large amount of multimedia functions built in, and an editing environment that was geared towards interactive design.
The reason java took so long to move out of process was because the java<->browser bindings were unique to java, it did not use any of the normal plugin apis, and expected a lot of direct linkage to the host system.
At a very basic level it can be summed up with: the JVM approach to GLSL was to just throw it at the GPU, whereas the browsers worked on restricting WebGL to a super constrained subset (this was my fault, but is the correct thing to do :D )