You don't have to worry about segmentation unless you're working with more than 64K of code or data, which is plenty when you're writing in Asm. And when you do, it's not hard to learn the few extra rules that come with that.
Protected mode and the 32-bit extensions are two separate (but related) topics, you don't need to learn the former to use the latter, and you can already do a lot without knowing either. Unfortunately the 64-bit extensions have gone in a very different direction, and you will need protected mode to use them. But for learning the basics of Asm, I don't think you need 64 bits.
The 8080/8085/Z80 are a little simpler, but learning them coudl be useful if studying x86 since they're its ancestors.
It's no harder (and in many ways easier) than learning the irregularities of English or any other language. One side-effect of learning x86 is that all the RISCs then become really boring and straightforward. x86 has character. :)
Now that's interesting. What is the deal there?