In the 1980s, XENIX 286 was able to use the 286 protected mode to deliver real protected memory and pre-emptive multitasking on a PC-AT. That more or less guaranteed a certain market share, because a PC-AT was a lot cheaper than anything with a 68k or a 386.
Xenix was successful because it delivered a lot of bang for the buck. It ran on relatively inexpensive hardware and delivered a lot of UNIX flavor despite the limitations.
Really early versions of Xenix would run on an actual 8086, with no protected memory whatsoever. I don't think any other UNIX ever managed that.
Back then I built systems with Xenix on a PC-AT that had three simultaneous users, one on the console and two WYSE-60 terminals attached to the serial ports. The system ran a database server and the applications that I built to manage factory inventory and orders.
Mind you, on a 4.77 Mhz PC at home, I ran Windows 1.0 and used it to multitask two sessions. In the foreground GUI I used Notepad to write C code, while the DOS session in the background ran the compiler. Back then compiling took a long time. Few people even then realized that Windows 1.0 did real pre-emptive multitasking on the original IBM-PC between the front-end GUI and a backend DOS session.
I worked at a MicroAge during the heyday of Xenix, selling any system we could get our hands on. We even had some weird System/36?/38? variant because IBM was still trying to keep the PC from killing midrange sales.
Hard to believe but we had three different UNIX offerings: Microsoft Xenix, AT & T 3B1/PC 7300 and what sold by far the best: Altos systems. We sold the hell out of 4, 6 and 8 user Altos systems because everything was connected to cheap dumb terminals. The Xenix systems could not come close at the time for speed.
But then Novell commoditized cheap networking and file sharing on the PC and that was that. The PC had far more software selection and despite the price difference offered Lotus 123 which took the industry by storm.
It depends. The Windows 3.x 386 "enhanced" mode kernel was in fact what we would call a "hypervisor" today. A 32-bit hypervisor to be specific. It ran 16-bit Windows applications inside one VM, along with any number of DOS VMs. The VMs themselves were preemptively multitasked, but the design of the 16-bit Windows API prevented preemption in that VM (a problem also faced by Apple).
If you weren't using an "real-mode" drivers then it never bothered to talk to DOS, it just did everything itself... for example the "32-bit disk mode" stuff was a 32-bit disk and filesystem driver that completely replaced DOS disk routines.
Microsoft being Microsoft, if you had DOS TSRs, DOS drivers, or other random DOS crap they would hook those things and drop down to real mode to call them the way they expected to be called... going so far as synchronizing the DOS in-memory data structures before and after the calls to make sure the lie was as complete as possible.
This should be no surprise- the 286 was faster than the PDP-11 where UNIX originated.
I used SCO Xenix on 386. It had a very good Lotus 123 clone called SCO Professional- you could have 5 or 6 users all sharing a PC. SCO was sued over it, had to stop selling it.
As for 8086 and MMU-less unix systems, xenix is certainly not unique in this. In 90's there even was heavily modified Linux kernel that ran on such systems.
And as for 286, there were (and probably even still are in production systems) multi-user multi-tasking CP/M derivates. Two large niches for such systems were industrial automation (IIRC some versions of Siemens Simatic ran that) and point of sale (IBM).
But it's still pretty unstable. If you have a spare 8086 and want a Unixoid, go look up Minix 2 (not 3). It's not very comfortable on a 8086 with 640kB, but it works. Proper filesystem, processes, Bourne shell, the lot. It even has a self-hosted C compiler. You can get real work done on it (slowly).
Even beyond the 90s, in uClinux (microcontroller Linux). While it was technically bad to access hardware directly, it was definitely possible to, for example, write a bit-banging I2C driver in userspace.
Xenix was successful because it delivered a lot of bang for the buck. It ran on relatively inexpensive hardware and delivered a lot of UNIX flavor despite the limitations.
Really early versions of Xenix would run on an actual 8086, with no protected memory whatsoever. I don't think any other UNIX ever managed that.