The x86 is actually fairly straightforward, as long as you don't try messing around with A20-gates, GDTs and legacy modes (i.e. not staying in protected/long mode the OS has given you). It's only when you try to build things such as bootloaders that x86 starts getting nasty. Backwards-compatibility can be quite evil.
It's not that it's complicated. Few assembly languages manages to be all that complicated, at least if you avoid some of the more bizarre addressing modes ...
It's that it's horribly ugly and full of warts.
The biggest issue I have with x86 is not even trying to make things orthogonal (until x86_64, which does a tolerable job cleaning up a lot of the mess).
Some of it could probably be improved simply by ditching the syntax, and renaming stuff and pretending a lot of the rest simply doesn't exist, but you'd be left with something that looks very different from a typical x86 assembly listing..
But some stuff you don't avoid unless you go 64 bit: The registers alone gives me gives me hives, starting with the naming convention, where the M68k for example is straight forward: D0 through D7 = data registers. A0- A7 = address registers. For a lot of things they can even be used entirely interchangeably.
No addressing parts of the registers with special names, or mumbo jumbo restrictions on what can be done with different registers (that was excusable when the 6502 was a viable competitor, but even that followed clear and simple rules (Arithmetic on the accumulator (A), indexing with the X and Y registers) and didn't layer complexity on top of complexity. Bank switching / segments? Nope, we don't clutter the CPU with junk like that. Either you use a memory address or IO lines (or, as it happens, on the 6510 in the C64, IO lines mapped to a memory address).
Maybe I'm just still sore the M68k lost the performance race and I "had to" get a PC.
As you noted, the x86_64 sorted out some of the mess.
Personally, I think a big issue is the whole backwards-compatibility thing. It causes all sorts of nastiness, and a complete overhaul of the architecture is impossible because everything depends on it! (Which is one of the reasons the IA-64 never really became a hit.)
I vote we design our own processors, and write a new kernel to boot. ;)
Um... ARM got assembly "right" three or four separate times at last count (depends if you count thumb and thumb2 as distinct architectures). And I believe they now have three equally "right" (and mutually incompatible) floating point architectures.
Everything has warts (and x86 more than most). But don't pretend that ARM is clean. Or VAX (integer polynomial instructions? Non-IEEE floating point what now?). Or MIPS (A branch delay what? Multiply outputs to a separate register?). Or even 68k (let me puke garbage on your stack, I'll just tell you how much room you need).
As a matter of fact, I agree. The fact that x86 won the consumer market is, IMHO, very much one of those clever-marketing-and-quick-standardization-cases.
