
Understanding the working of X11 from the ground up (2017) - signa11
https://magcius.github.io/xplain/article/index.html
======
Jasper_
Author here. I stopped working on this a few years ago, so now's as good a
time as any for an overly-long, rambly, unedited retrospective.

When I started this, I was working on GNOME's window manager full-time, and
wanted to learn intricately how X11's drawing model worked, so over the course
of a few weeks in a hotel room, I recreated large parts of X11's drawing model
in a web browser, fixing artifacts as I went along, until I feel I had a
really good grasp of it. My initial test scene was a traditional desktop-like
approach with a taskbar and xeyes, both of which are still in the codebase
today, but untested [0].

I didn't know what I wanted to do with it, until I settled upon using snippets
of it to build a long-form article. I learned a lot about the difficulty of
writing, of pedagogy, of that blurred line between being technically correct
100% of the time vs. telling a few small lies here and there to keep the flow
consistent and help people see the broader picture.

At my day job, I had mostly moved onto Wayland, where some of the bits I
picked up here really helped me design better protocols and systems. My goal
with the series was to try to be as neutral as possible, and my original
design was to have a giant caution sign around "Author Opinion Zones" where I
would talk about how certain design features haven't held up well in practice.
But quickly, people on Hacker News or Phoronix or Reddit seemed to skim the
article, pick up a piece here or there, and go on straight to bashing Wayland,
gleefully unaware that I was one of the people making it.

So, the end result was that I basically stopped working on Xplain basically
after the second article. The COMPOSITE article was one I made after a
colleague was having trouble understanding COMPOSITE, and I figured it was
easier to write with my framework than explain in a chatroom, and maybe some
others would appreciate it.

I have a deep passion for sharing my knowledge, and Xplain was the format I
first really used to do it widely, so I tried to keep it exciting for me by
changing it from "Xplain" to "Explanations", and opening up the topics from
X11 to just about anything, but at some point I was just unhappy working on
it.

The last thing I was working on was a continuation of my Basic 2D
rasterization article, where I had a fun code editor you could use to make
your own graphics [1] [2], but as fun as the technology was, I couldn't find a
satisfying flow to the article, so I stopped it. Parts of it were later
recycled for an article on the histories of 2D and 3D graphics. [3]

Around mid-2016, I had stopped working on Linux and open-source graphics
entirely, and by 2017 I had exited the open-source industry completely and
jumped ship to professional game development. I still have a deep love for
graphics and a passion to explain things. I just released a new side project a
few days ago for it, even.

Here's some stuff I'm working on these days. It's much cooler than X11/Wayland
flamewars, in my opinion.

[https://noclip.website/](https://noclip.website/)

[https://blog.mecheye.net/2018/03/deconstructing-the-water-
ef...](https://blog.mecheye.net/2018/03/deconstructing-the-water-effect-in-
super-mario-sunshine/)

[https://www.youtube.com/watch?v=8rCRsOLiO7k](https://www.youtube.com/watch?v=8rCRsOLiO7k)

\--

[0] [https://github.com/magcius/xplain/blob/gh-
pages/src/clients/...](https://github.com/magcius/xplain/blob/gh-
pages/src/clients/xeyes.js)

[1] [https://u.teknik.io/XdSbC.webm](https://u.teknik.io/XdSbC.webm)

[2] Sort of up here at
[https://magcius.github.io/xplain/article/rast2.html](https://magcius.github.io/xplain/article/rast2.html)

[3] [https://blog.mecheye.net/2019/05/why-is-2d-graphics-is-
harde...](https://blog.mecheye.net/2019/05/why-is-2d-graphics-is-harder-
than-3d-graphics/)

~~~
bscphil
The examples in this link don't work for me in the latest Firefox or Chromium
(or Brave) on Linux. [https://blog.mecheye.net/2018/03/deconstructing-the-
water-ef...](https://blog.mecheye.net/2018/03/deconstructing-the-water-effect-
in-super-mario-sunshine/)

Edit: neither does anything here:
[https://noclip.website/](https://noclip.website/)

~~~
vetinari
It doesn't work in Firefox for Mac either.

It does in Chrome.

~~~
mrguyorama
Using Firefox nightly on 2015 Macbook Pro and it works just fine

~~~
vetinari
Firefox 69.0.1 on 13" 2015 MBP.

------
raphlinus
Looking over this stuff, I get an uneasy feeling. X was a brilliant solution
to problems 35 years ago, where you barely had enough memory for a full frame
buffer and the graphics model was primitive (no antialiasing or transparency).
From what I've seen, Wayland is a good solution for the model maybe 15 years
ago, where you're willing to accept the frame-and-a-half latency hit from
having a compositor in the path.

Basically, my question is whether anybody is working on a solution for the
hardware we have today, where there are hardware overlays, the GPU throughput
is orders of magnitude higher than what's needed to composite a desktop, but
we still care about memory bandwidth, largely because all of resolution, bit
depth, and frame rate is still going up. And you really want to be able to
schedule every pixel that gets presented with a high degree of timing
accuracy.

I've become familiar with Windows, but feel like it's a bunch of hacks. They
will promote a window to an overlay, but it's all a heuristic, and I'm pretty
sure latency will stutter while it does the transition in and out. And the new
flip-model stuff actually regresses the ability to do smooth window resizing.

I'm not even sure macOS is working on lowering latency. I haven't seen any
mention from them of hardware overlays.

Most of the attention these days seems to be in the VR/AR space, where latency
is obviously important, and there's more of a green field with respect to
legacy compatibility, but I'd love to see this problem space actually solved
on the desktop. Doing this right is not easy - it probably requires some form
of soft real time discipline across the graphics stack (including GPU
scheduling) so you can race the beam.

~~~
Jasper_
I have a lot of replies to this, but the back-and-forth format that HN has
isn't the best for collaborative discussion. Imagine us sitting in a room with
a whiteboard, rather than bickering at each other at loud volume. If you're
ever in south bay area and want to meet up at a coffee shop, let's :)

> Basically, my question is whether anybody is working on a solution for the
> hardware we have today, where there are hardware overlays, the GPU
> throughput is orders of magnitude higher than what's needed to composite a
> desktop

Hardware overlays are... not that popular on desktop devices? You just stomp
on the gas pedal with bandwidth and power consumption on consumer desktop
gaming GPUs. Besides a couple YUV planes and a cursor overlay, you don't see
that much in the way of CRTC capabilities on desktop. All of that's happening
in the mobile space, because of power consumption reasons, where you basically
have an SNES PPU in your CRTC with multi-plane compositing.

I remain unconvinced that compositors need to add a frame and a half of
latency. 16ms is a lot of time to render a frame, and composition isn't too
difficult of a task. It just comes down to buffer handoff, which can be
smoothed out over time. Similarly, the new flip-model isn't the thing that
makes smooth resizing difficult, buffer allocation and resizing has always
been a pain and racy on Windows/X11. This is one of the things that Wayland
fixes -- it was actually _the_ thing that caused Kristian to start Wayland
forever ago.

Presentation and timing is something I think _should_ be possible, but I don't
think it makes sense for every app to have to handle it. Multi-monitor
scenarios are what make global timing difficult, and having per-monitor
surfaces is just a pain to deal with from an API standpoint; if we did the
work to have per-monitor timing information and split surface rendering,
nobody wants to go through the work to set up two devices and present
different ones at the right time.

You might protest that all monitor timing is roughly the same these days, but
someone hooking up their laptop to a projector, you get radically different
timing.

In many respects, I think graphics is in a local minima right now: it works
well enough and we understand it well enough that nobody's really interested
in pushing us to the next thing. VR/AR research might, but that's been
relatively unimportant as a target platform, commercially.

~~~
pcwalton
> If you're ever in south bay area and want to meet up at a coffee shop, let's
> :)

For what it's worth, I would be very interested in chatting about this too, as
this is my area right now. I'm working on what will hopefully become the
Firefox OS compositor integration library, with an eye toward optimum power
consumption, and am knee-deep in details of cross-platform composition. It's a
very interesting topic.

~~~
zaxcellent
This topic is also of interest to me. I work on a compositor integration
protocol called virtio-wayland that is intended to allow Wayland apps running
inside of VMs to communicate with a Wayland compositor on the host, shared
memory and all. It's used for the Linux apps support for Chrome OS.

~~~
raphlinus
I don't get to the south bay often, but sounds like it might be good to have a
little compositor meetup. I can certainly vouch for pcwalton's interest and
knowledge in the topic :) Follow up by email?

~~~
pcwalton
Sounds good to me!

------
sprash
Money Quote:

> So, a lot of the design of X11 is giving clients and apps a large set of
> tools to design and implement their app, without any real opinionated design
> about how an app should be made. An X client can use this tool or that tool
> to battle this tradeoff or that tradeoffs, and neither is fundamentally a
> wrong choice.

That is something that can not be said about Wayland. Besides the "every frame
is perfect" mantra Wayland has no design philosophy at all. It is also a weak
argument to introduce a new protocol. With modern drivers it is rare to
experience tearing on X11 and on some occasions it even makes sense to disable
vsync (e.g. competitive gaming).

The introduction of Wayland was a mistake because X11 has a huge ecosystem
with lot of little programs and a large diverse users base. Almost every Unix
OS supported X11 at some point. There is a reason why Intel still stuffs the
i368 instruction set in their processors, Microsoft Windows still can execute
programs from 1995 and IBM is still able to make money with vastly overpriced
mainframes. Successful systems have the tendency to stay and accumulate cruft
as a result. The abandonment of X11 will be a heavy blow to the FOSS community
(and maybe that is exactly the intend).

~~~
pcwalton
As someone who is working with X11, I strongly disagree. It really needs
replacing badly. It doesn't even _work_ ; only a few code paths even get
tested on for example GLX, which is why complex X11 apps like browsers need
blacklists.

> There is a reason why Intel still stuffs the i368 instruction set in their
> processors

If you want a CPU analogy, Wayland is more like ARMv8, which was mostly a
clean break in terms of the AArch64 ISA, but which still supports 32-bit at
the process level. 32-bit and 64-bit code cannot be intermixed in the same
address space, but you can run 32-bit and 64-bit apps simultaneously on the
same OS, analogously to the way you can run X11 and Wayland apps on the same
Wayland desktop using Xwayland. This actually worked out beautifully, and I
would have preferred that x86 had followed the same trajectory.

~~~
vrodic
> As someone who is working with X11, I strongly disagree. It really needs
> replacing badly. It doesn't even work; only a few code paths even get tested
> on for example GLX, which is why complex X11 apps like browsers need
> blacklists.

Blacklists are needed because OpenGL drivers for specific hardware have bugs
and can have missing functionality. Those can affect Wayland too, since the
same drivers are used.

Modern Xorg supports direct rendering well, Keith Packard authored a lot of
modernized infrastructure such as DRI2/DRI3, and even support for VR (work
he's been doing for Valve).

There's also the important VK_display_timing extension implementation he's
done with Croateams Alen Ladavac:

[https://twvideo01.ubm-
us.net/o1/vault/gdc2018/presentations/...](https://twvideo01.ubm-
us.net/o1/vault/gdc2018/presentations/Ladavac_Alen_ElusiveFrameTiming.pdf) on
Xorg

Sadly, I don't see much more recent activity from him or Alen about these
topics, but it shows that Xorg is still the main platform of inovation for
Linux real-time graphics needs.

------
hota_mazi
What's fascinating to me is that X11 and Windows started at complete opposite
ends of the spectrum.

Then X11 never managed to be a really effective window manager on a local
workstation while Windows became really, really good at remote displays.

~~~
asveikau
I feel like this is short changing X. I have seen it as a good local solution
over 20 years or so.

Did you ever see those 90s commercial Unix workstations? Like from SGI?

How about a Nokia N900? X on a smartphone. It ran pretty well. Today's
smartphone hardware would run circles around that thing so I can only assume
if somebody ported X to those it would also do fine.

I feel like people see things they don't like or imperfections and bash the
whole thing as useless. X had a lot of mileage. I use it frequently today and
don't see a lot of problems.

------
kebman
This was a really nice demo! It can even be used to understand concepts used
in the HTML5 canvas, such as aliasing. Hell, it could possibly used as en
entry to learning how to program ray tracers. At the bare bones, construction
of complex bodies in POV-Ray was used by Constructive Solid Geometry
hearkening back to the basic union, xor, intersection, etc, operations done in
the demo, at least on a conceptual level.

------
krychu
I haven't used linux as my primary OS for many years now. If not X11 or
Wayland, what other alternative? Is there some other relevant development
going on?

~~~
sverige
OpenBSD is still actively developing xenocara, which is their fork of X11.

~~~
0xdeadb00f
That still counts as "X11" IMO.

------
shmerl
_> I've made a personal decision to leave the Linux, X11, and Wayland worlds
mostly behind._

I understand X11 has a lot of unfixable problems, but what's wrong with
Wayland world?

~~~
jchw
I'm running Sway WM right now, a Wayland compositor that implements an
i3-compatible window manager, and in fact working on the code (hopefully
merging my first contribution soon.)

I'd say the biggest issue is really maturity. I actually find Wayland to be a
surprisingly nice piece of software engineering. I don't know if I like the
architecture really, but it comes together fairly nicely in the end.

For example, there's unanswered security questions. It would be good if
applications couldn't just go and manipulate and spy on other apps like they
can with X11. Wayland indeed solves this issue, but it leaves other problems.
How do I capture the desktop?

wl_roots offers some extensions that help alleviate the problem, but they
don't solve the full problem; for example, you can capture the screen, for
which programs and an OBS plugin exists, but there's no upstream Wayland
extension that can make WebRTC desktop capture work. I think what we really
need is the ability to integrate policy and consent requests, but I'm not sure
whether there's progress on that front, or how long it will take. Once the
problem is solved, upstreamed to WebRTC, and that upstream version trickles
down to Electron apps, Firefox and Chrome stable versions... well, things will
be in a better place.

Speaking of web browsers, today Firefox still has some issues to go before it
is fully working in Wayland natively. You can do it today, but it is not
perfect; there are sometimes places in the code where it assumes it is running
on X11, and things simply break. The screenshot tool is just flat out not
present when running under Wayland due to the above issue.

Applications are slow on the uptake too. XWayland covers most of them, but it
is not perfect. Like anything that captures the screen, of course, is not
going to work under XWayland, but also, XWayland apps may behave a little off.
In Sway WM, one bug is that when the mouse travels from an X window to a
Wayland window, sometimes it behaves as though the mouse is still hovering
over the X window at the last position it touched, which leads to tooltips
showing up where they shouldn't. There's still a decent amount of GTK2 apps,
and even Qt apps aren't all Wayland yet; Krita plans to start working on
support I believe next year.

In my case, here's the itch I'm currently scratching: Sway WM also does not
have support for drawing tablets. I found this out, by, well, loading up a
drawing program and noticing the lack of pressure sensitivity. So, that's what
I'm currently working on. Needless to say, this is a non-starter for anyone
that relies on a graphics tablet, although if you are just using GNOME 3 it is
unlikely to be an issue since I'm sure GNOME 3 supports graphics tablets just
fine.

NVIDIA drivers are still a pain in the neck. I gave up and bought an AMD
graphics card, and frankly I can't recommend it enough. Basically everything
works better on Linux, but certainly Wayland works better. It's worth giving
up some raw power. Still, for many people the proprietary NVIDIA drivers are
going to need to work, and today support varies from lackluster to poor
depending on what compositor you use.

~~~
crispinb
> The screenshot tool is just flat out not present when running under Wayland
> due to the above issue.

That's odd - it's present (and functioning) in my Firefox (69.0.1) running on
Gnome-on-wayland

Firefox is generally quite crashy, however, which I'm guessing is Wayland-
related

~~~
gmueckl
So Firefox supports screenshots when running on the Gnome compositor, but not
running on sway? Looks like we're heading into interesting times where
applications are getting tied to certain DEs with little to no cross-
compatibility.

~~~
MayeulC
IDK. Sway is my daily driver. I just checked Firefox, and the screenshot tool
works as expected (including clipboard). So unless GP is talking about
capturing non-Firefox windows with that tool, I do not have that issue.

This Firefox instance is running on native Wayland. The only issue I have with
it is sometimes the coordinates for the context menu are wrong.

But yeah, there might be compat issues, if some protocols are required for
stuff. But Wayland itself is the baseline: you'll always be able to draw
windows and interact with them.

What's left to extra protocols isn't guaranteed to work: control of the
compositor with a client, for instance (color correction, virtual keyboard,
screenshots, window placement, screen geometry, window decorations, etc.) is
completely up to the compositor (and the protocols it can speak with the
client). So is other stuff, like graphics tablet input (with pressure
sensitivity, tile, etc), minimizing windows, window miniatures as used for
KDE's taskbar as well, I think.

But then, it's not like everything worked flawlessly everywhere on X either :)
And as long as the baseline is solid, we can build on it.

~~~
jchw
I may have done something wrong. I am not highly knowledgable here. For
example, I just learned of Pipewire.

That said, the problem occurred for me when using the NixOS package firefox-
wayland. It is possibly better to use upstream firefox with Wayland, but I
have not tried; I am just running firefox on Xwayland for now.

~~~
MayeulC
I don't know how the Firefox package is built on NixOS, but I just run the one
from Arch Linux with the following environment variables:

    
    
        MOZ_WEBRENDER=1 MOZ_USE_XINPUT2=1 GDK_BACKEND=wayland

------
pcr910303
This is a bit of topic but:

Every time I see the X11 protocol, I realize that the UNIX philosophy
(everything is a file) is dead.

It’s a pity that instead of improving the speed of the file system, everyone
started to ignore it, hence making it’s speed even less important, and on and
on...

And now nobody uses the file system for these purposes.

~~~
mmrezaie
How do you think it should have evolved? I personaly think posix way of
filesystem needs to be changed into specialized oject filesystems, but I am
not sure what object filesystem for a UNIX OS should look like specially since
it needs to be generalized into something all the other sub-systems can
benefit from.

~~~
pcr910303
Personally I also preferred the ‘Everything is an object’ from the lisp
machines. However, I also believe that files are an acceptable object format,
since if several applications know how to manipulate a file format, it’s
basically duck typing.

I believe file systems could have a more formal method to query the file’s
format, and has less latency so that applications could use files as an inter
communication format.

