Compiling an emulator is quite easy: have a look at simh. It's very portable and should just work out of the box.
Once you've got that working, try installing a 2.11BSD distribution. It's well-documented and came after a lot of the churn in early Unix. After that, I've had great fun playing with RT-11, to the point that I've actually written some small apps on it.
The PDP-11/03 emulator itself is good enough that it can run the RT-11 installer to create the disk image you see in the browser version. The VT240 emulator is good enough that the standalone Linux version can be used as terminal emulator for daily work. Once I have time, I plan to make a proper blog post describing how it all works / what the challenges were and post it as Show HN eventually.
The daves garage youtube has an episode where he documents the pitfalls of compiling 2bsd for a PDP-11/83. https://www.youtube.com/watch?v=IBFeM-sa2YY basically it is an art on a memory constrained system.
What I found entertaining was when he was explaining how to compile the kernel, I went Oh! that's where openbsd gets it from. it is still a very similar process.
> It's somewhat picky about the environment. So far, aap's PDP-11/20 emulator (https://github.com/aap/pdp11) is the only one capable of booting the kernel. SIMH and Ersatz-11 both hang before reaching the login prompt. This makes installation from the s1/s2 tapes difficult, as aap's emulator does not support the TC11. The intended installation process involves booting from s1 and restoring files from s2.
good luck though. my emulator is not particularly user friendly, as in, it has no user interface. i recommend simh (although perhaps not for this thing in particular).
I've been messing around with RSX-11M myself! I find these early OSes quite fascinating. So far I've set up DECNet with another emulator running VMS, installed a TCP stack, and a bunch of compilers.
Once you've got that working, try installing a 2.11BSD distribution. It's well-documented and came after a lot of the churn in early Unix. After that, I've had great fun playing with RT-11, to the point that I've actually written some small apps on it.