As I remember it (and I had no inside knowledge, I was just an application developer who read the docs), NT was supposed to have 3 distinct APIs for interfacing with the kernel: the POSIX subsystem, Win32 for compatibility with Windows and the next-generation OS/2 API.
Then IBM and MS fell out and Microsoft put all their resources into Win32.
Windows NT was originally supposed to be "OS/2 3.0". Then IBM and Microsoft broke up.
However, Windows NT did actually have those three APIs – the Win32 subsystem (which also incorporated Win16/DOS support), the POSIX subsystem, and the OS/2 subsystem. But the POSIX and OS/2 subsystems were always rather half-baked, and never saw that much use. The OS/2 subsystem was removed in Windows 2000; it only ever supported OS/2 1.x apps, and only character mode unless you purchased a separate graphics mode add-on from Microsoft. The POSIX subsystem was not removed until Windows 8.1, although by then it had changed its name to "Subsystem for Unix-based Applications", and had grown a lot compared to the original NT 3.x POSIX subsystem. WSL is effectively the replacement for the POSIX subsystem, however its implementation is very different (which is true of both WSL1 and WSL2, despite the fact that they are also very different in implementation from each other.)
Interix wasn’t really a “replacement” - the fundamental architecture hadn’t changed, PSXSS was still there. From what I understand, Interix was just an enhanced version of the original POSIX subsystem code, not some from-scratch rewrite.
I thought that Softway replaced the Microsoft PSXSS with their own but I don't remember for sure. This would be an interesting software archaeology rabbit hole to go down.
Interix 2.2[0] shipped with its own PSXSS.EXE. I don't have a contemporaneous version of the PSXSS.EXE that shipped with Windows at that time to compare handy.
Finding a copy of OpenNT prior to the Microsoft acquisition is proving to be difficult.
Wasn’t their updated version a modified version of Microsoft’s original code, which they’d licensed from Microsoft? If that’s true, their PSXSS was more an enhancement than a “replacement”.
That would be interesting to know. I’d love to see a copy of OpenNT. Sadly, my experience started with it when it was already Interix. I know Microsoft negotiated source access/licensing with various third-party software “manufacturers” (like w/ Citrix for WinFrame). I’d love to know more about the arrangement with Softway.
However (putting aside the legalities of downloading "abandonware"), they won't let you download anything unless you first upload something they don't have. Heck, I probably do have something they don't have in my box of old floppy disks (which I keep on telling myself I will image one of these days), but my curiosity about this topic isn't strong enough to motivate me to do that.
notably the actual subsystem wasn't really necessary, if i recall it was some poorly implemented stubs for unix style system services. (think a backend for functions like getpwnam)
the important part was the posix api functions that made porting unix daemons that had been written in c/c++ to nt pretty easy. there were some quirks around shared memory/mmap, getpwnam, the sockets api and threading support, but beyond that doing a port was pretty easy.
the work there was plugging into the service control and event logging apis. and doing an installshield, that was no fun.
The subsystem absolutely was necessary to properly support fork(). The NT kernel has always supported fork, but Win32 can’t cope with it, forking confuses CSRSS.EXE. POSIX/Interix/SFU/SUA used PSXSS.EXE instead which wasn’t upset by forked processes.
that's right, i remember now fork being an issue. wasn't that much of a deal if i recall as most programs only really used it to spawn new processes by calling exec immediately after. (so, just swap it for CreateProcess)
if a program did use fork as part of its concurrency model (like say, preforking apache) then making use of actually functional nt threading support and gratuitous ifdefs was usually the answer.
exec() was another problem - NT doesn’t support reusing a process for a new executable; exec() on Windows actually spawns a child process and then exits the parent - which means exec() changes your PID.
Both Cygwin and POSIX/Interix/SFU/SUA use the same workaround - maintain a separate Unix PID for each process. exec() changes the NT PID but leaves the Unix PID unchanged. So exec() creates a new process, but to Unix apps it looks like the same process.
You are right that for many apps, rather than rely on emulated fork/exec, it was better to switch to native Windows facilities. Still, there was some value in being able to recompile Unix software with minimal changes (as Cygwin demonstrates). Even WSL now confirms that, albeit in a rather different way. Also, it was necessary to pass the POSIX test suite, which Microsoft briefly was concerned about (until they successfully lobbied the US government to drop POSIX certification as a procurement requirement).
Posix compliance was/is often a checkbox requirement to bid for government contracts. So M$ tossed in the Posix subsystem so they didn't get excluded based on that requirement. Lots of otherwise odd features in software fall into this category (See also: Apple A/UX).
the sockets api was modeled after berkeley and the libc itself is posix compliant to this day, i believe.