Historical look at quite a few
Note: A number were concurrency safe, had a nucleus that preserved consistency, or were organized in layers that could be tested independently. UNIX's was actually a watered down MULTIC's & he's harsh on it there. I suggest you google it too.
Burrough's B5000 Architecture (1961-)
Note: Written in ALGOL variant, protected stack, bounds checks, type-checked procedure calls dynamically, isolation of processes, froze rogue ones w/ restart allowed if feasible, and sharing components. Forward thinking.
IBM System/38 (became AS/400)
Note: Capability architecture at HW level. Used intermediate code for future-proofing. OS mostly in high-level language. Integrated database functionality for OS & apps. Many companies I worked for had them and nobody can remember them getting repaired. :)
Note: Brilliance started in Lilith where two people in two years built HW, OS, and tooling with performance, safety, and consistency. Designed ideal assembly, safe system language (Modula-2), compiler, OS, and tied it all together. Kept it up as it evolved into Oberon, Active Oberon, etc. Now have a RISC processor ideal for it. Hansen did similar on very PDP-11 UNIX was invented on with Edison system, which had safety & Wirth-like simplicity.
Note: Individual systems with good security architecture & reliability. Clustering released in 80's with up to 90 nodes at hundreds of miles w/ uptime up to 17 years. Rolling upgrades, fault-tolerance, versioned filesystem using "records," integrated DB, clear commands, consistent design, and great cross-language support since all had to support calling convention and stuff. Used in mainframe-style apps, UNIX-style, real-time, and so on. Declined, pulled off market, and recently re-released.
Genera LISP environment
Note: LISP was easy to parse, had REPL, supported all paradigms, macro's let you customize it, memory-safe, incremental compilation of functions, and even update apps while running. Genera was a machine/OS written in LISP specifically for hackers with lots of advanced functionality. Today's systems still can't replicate the flow and holistic experience of that. Wish they could, with or without LISP itself.
BeOS Multimedia Desktop
Note: Article lists plenty of benefits that I didn't have with alternatives for long time and still barely do. Mainly due to great concurrency model and primitives (eg "benaphors"). Skip ahead to 16:10 to be amazed at what load it handled on older hardware. Haiku is an OSS project to try to re-create it.
Note: Capability-secure OS that redid things like networking stacks and GUI for more trustworthyness. It was fast. Also had persistence where a failure could only loose so much of your running state. MINIX 3 and Genode-OS continue the microkernel tradition in a state where you can actually use them today. MINIX 3 has self-healing capabilities. QNX was first to pull it off with POSIX/UNIX compatibility, hard real-time, and great performance. INTEGRITY RTOS bulletproofs the architecture further with good design.
Note: Coded OS in safe Modula-3 language with additions for better concurrency and type-safe linking. Could isolate apps in user-mode then link performance-critical stuff directly into the kernel with language & type system adding safety. Like Wirth & Hansen, eliminates all the abstraction gaps & inconsistency in various layers on top of that.
Note: Builds on language-oriented approach. Puts drivers and trusted components in Java VM for safety. Microkernel outside it. Internal architecture builds security kernel/model on top of integrity model. Already doing well in tests. Open-source. High tech answer is probably Duffy's articles on Microsoft Midori.
So, there's a summary of OS architectures that did vastly better than UNIX in all kinds of ways. They range from 1961 mainframes to 1970-80's minicomputers to 1990's-2000's desktops. In many cases, aspects of their design could've been ported with effort but just weren't. UNIX retained unsafe language, root, setuid, discretionary controls, heavyweight components (apps + pipes), no robustness throughout, GUI issues and so on. Endless problems many others lacked by design.
Hope the list gives you stuff to think about or contribute to. :)