Most annoying for me personally: it doesn't support proper HiDPI scaling. X11 apps are blurry on my HiDPI Wayland desktop, which makes me not want to use them.
In general, X11 is absolutely inadequate for the modern world. It's a legacy protocol that has the "any client can be a keylogger" security model, drawing primitives from the 80s that no one uses anymore plus all the history of bolting on more modern ways of rendering. (And don't get me started on the input systems, of which there are several and no one knows how that code even works.) And the protocol is synchronous, so you'll always have some slowdowns. Oh and the fucking screen tearing! In modern display systems, every frame is perfect, by design, by protocol design. X11 only has various "tearfree" hacks in the server that work with varying levels of success.