The OS class I took at UC Berkeley (back in 2010) was in Java, which sucked all the fun out of it. Seeing other universities' OS courses always makes me feel a little jealous.
How can an OS be written in Java? The only experiment into this direction that I know of is Jnode (http://www.jnode.org/), which is not really mature and half-dead.
It ran user programs on a virtual CPU, also written in Java. Syscalls and page faults would call out to your code, which ran in the same JVM as the virtual CPU. You can read more about it here: http://www.cs.berkeley.edu/~kubitron/courses/cs162-F05/Nacho...