Though I don't know much about this topic, this seems an underrated post. I think it would get more readership/discussion if it followed the principle of proceeding from the concrete to the abstract. Specifically, in its initial paragraphs it would be good to mention what it was a unifying theory of: a large laundry list of languages, libraries, and JavaScript frameworks that are relevant to the topic. That way anyone who uses any of them would know it's relevant.
According to the second half of the post (the one starting at “Case studies: Ok, since the above was so abstract…”), it is relevant to Imgui, Flutter, Jetpack Compose, React, “DOM”, etc, but even someone who uses or knows a lot about any one of those, unless they got to that part, may not become aware that this post is relevant to them or contribute to the conversation.
Thanks for these great articles. I love your blog and your work on druid. Keep them coming!
This is a space I'm very interested in. Having a high-quality, reactive UI toolkit for developing native desktop apps might make it easier for web developers accustomed to these reactive frameworks (e.g.: React) to develop native apps (without having to resort to hacks like Electron).
Linux in particular could really use more developers developing native desktop apps and I feel the current UI toolkits (QT/QML, GTK) and in particular their language choice (C++, but I'm aware there are bindings for other languages) are not appealing enough to the casual web developer. It could be argued that Rust might not be the answer for these developers anyway, but one can still hope.
How is Rust a better choice for the „casual web developer“?
Both Qt and Gtk have Python bindings, which would be far more in the realms of a web developer (dynamic typing, no compilation, garbage collection...).
Maybe you’re asking the wrong question. A different way to phrase this question will lead you to answers about why this is so exciting for a lot of people.
Why is Rust a great option for building native applications? Rust is a great option for building applications that would traditionally be in C/C++. These languages are traditionally chosen to have lower overhead than other options. You could build a new Qt or GTK in Rust, and put similar Python bindings over it, just as you described to go after those developers, if that’s desirable.
In addition to that, many casual web developers are recognizing the benefits of a compiler with a type checked language, it’s why Typescript is growing so quickly.
So you want to choose Rust when you want a lighter weight application, that has lower memory overhead, generally faster runtimes, and a long term plan around maintenance (the type system and various safety guarantees). Or more succinctly, because you’re someone who enjoys writing programs in Rust.
Weird, because as web developers we've been pushing towards static typing (TypeScript, ReasonML, etc). I'd much prefer to be learning Rust than having to brush up on my Python, to be honest.
> It could be argued that Rust might not be the answer for these developers anyway, but one can still hope.
I fail to see how a systems programming language notorious for it's novel/different take on memory management will cater to a community known by their extensive use of high-level languages that are very permissive and abstract away memory management.
Well, unless we're talking about buzzword-oriented programming.
What native apps do you feel are missing on Linux? All the small apps and utilities are there. In some areas there are even more options available...
What is missing is big-ticket suites like Adobe's software or video editors etc. In other words, huge software packages that I would very much like to prefer to be developed by proper skilled developers who can manage to write modern C++ (is it so hard? We're not talk assembly here...), not by "casual web developers" who are used to writing subpar code because the web and the browsers let them get away with it.
> proper skilled developers who can manage to write modern C++ [...] not by "casual web developers" who are used to writing subpar code because the web and the browsers let them get away with it.
Since it mentions various other frameworks and approaches (imgui, flutter, etc.) - this video here gives overview of the three tree hierarchies flutter itself uses: in their lingo: widgets, elements and renderObjects - https://www.youtube.com/watch?v=996ZgFRENMs
Basically, widgets (in this context) are the immutable parts (e.g. "config"), "elements" would keep the updates, and relationships (parent/child), and renderObjects - layout, drawing, etc.
Thanks, great resource. I've had Incremental explained to me by a Jane Streeter and I'm sure it influenced my thinking, but it's always good to have public explanations.
Good summary of techniques. I'm very interested in this subject matter and really want to get to a formal definition for this architecture of UI. As mentioned in the article, FRP is the closest here and where the techniques initialized from. The patterns deal with how to break down the problem of asynchronous rendering and user interactions (which inherently deals with state changes). The best papers I've read use some Category Theory to show the structure of the flow of certain patterns, and there are also adjacent fields like parallel and concurrent programming that I think are overlapping with this area of how to manage UI state. The recent articles on CRDTs and Causal Trees made me think that could be used not just across a network but to manage the asynchronous state of components.
If you can make it nicer and cleaner than that, go for it. The specific constructs for specifying view modifiers and properties could of course be slightly different in each language; for example we could do away with the {}s in Python, or the ()s might be []s. And maybe we could reduce the impression of "magic" that SwiftUI's @State, @Binding etcetera seem to give.
It would just be a model for describing the UI, that a hypothetical engine backend would take to spit out OS-native widgets. For another OS, you'd plug in a different engine.
Having a universally agreed-upon syntax would provide the benefit of "learn once, write anywhere", greatly reducing the friction in building cross platform apps with native APIs.
There have been many surprises. Microsoft making C# dockerizable, making the best lightweight opensource IDE, and having a literal version of Linux running inside windows, have been some of the big shockers for me.
And yeah, I totally hate XML, would still never use it for data or config, but for some reason JSX just kinda feels good.
I did not read full post but the introduction kind of surprised me. The example of "object oriented style" the author gives is artificially inflated to look ridiculously verbose. Modern languages with OOP and anonymous functions capabilities and some library help can make code the same or even less verbose representation than the declarative example mentioned.
I believe the main motivation for these code snippets was not to point out the verbosity of the OOP style, but to draw attention to the fact that the onclick handler mixes the concerns of updating the state and updating the UI. Or in the author's words:
"[...] duplication between the initial construction of the UI and updates. [...] to add a slider showing the value, the onclick handler also needs to be amended to add an update for that".
Totally agree. He lost me there as well, which is a pity because I like these kind of theory posts. Any reasonable modern language (Swift, C#, JavaScript) can represent the positive sample very closely.
I also do not see the relevance of this argument for for the reactive UI topic.
You should take a look on Qt and QML
The render tree is only one aspect whis is dealt by JS frameworks, because all of them are sitting on the fat neck of browser. For a stand-alone UI you will need not only bubbling through layers events like mouse clicks but also non-layered functionality like keyboard focus, tab orders and screen reader support for blind users.
Note that their images don't scale properly so if you browse on mobile and it looks like you're only seeing half a chart try opening the image in a new tab. I think this is because of the way they're including SVGs.
Works for me (tested in Pixel / Chrome and iPad / Safari). A bug report, or better yet, PR against the repo would be welcome, not just for this but because I plan to use SVG for most of my 2D/GUI related content in the coming months. (And I have a lot planned!)
Well, that's quite the subtle question. Yes, in my plan for world domination, the representation of display lists is also immutable trees (or, more precisely, DAG's) hosted on the GPU, and the incremental nature of the pipeline is preserved, so small diffs to the UI result in small uploads of graph to the GPU. But it also includes damage region tracking, which is not really a tree, and the very end of the pipeline is pixels, which is definitely not a tree.
Side question: How do you make DAGs immutable? Unlike trees where changing a node only requires a path copy of all ancestors, changing a node in a DAG has the potential to affect all other nodes. There are techniques such as using fat nodes[1] but I don't know of anyone using them in the real world.
You do need to trace all predecessors in this case, but in practice I don't think it's a serious problem. The kinds of graphs you'll get in UI are probably best thought of as trees with the possibility of shared resources (for example, all buttons of the same size might share the same background). On something like a theme change, I'd just rebuild the tree, although in theory you could be smarter and apply changes incrementally at a fine grain.
There seems to be a misunderstanding. On all mobile phones and increasingly desktops, if your UI involves the GPU, you have failed; the goal is to have large layers of a scene handled entirely by the display compositor that produces the final pixels. Using the GPU as an enhanced blit engine is a colossal waste of memory bandwidth and power that no one can afford.
This will the the tl;dr of my upcoming blog post "the compositor is evil", referenced in the post.
I hear what you are saying, but I think the situation is a lot more complicated than that. Certainly, these days, if you're not using the GPU then you have failed; with high dpi and high frame rate displays, combined with the end of Moore's Law, CPU rendering is no longer viable. So the question is how you use the GPU. I agree with you that power is the primary metric of how we should measure this (it certainly correlates strongly to memory bandwidth, and is the metric users will actually care about; if there were a magical way to achieve high memory bandwidth at low power, I don't see any problem with that).
One way certainly is to use the compositor, and this is very seductive. The major platforms all expose an interface to the compositor, and it's generally pretty well optimized for both performance and power usage. Since animation is part of the API, it's possible to do quite a bit (including, for example, cursor blink in an editor) without even waking up the app process. Scrolling is another classic example where a compositor can do very well.
However, the compositor comes with downsides. For one, it forces the UI into patterns that fit into the compositor's data model. This is one reason the aesthetic of mobile UI design is so focused on sliding alpha-blended panes of mostly static content.
But even from a pure performance standpoint, heavy reliance on the compositor has risks. It's tempting to think of the composition stage as "free" because it has to blit the intermediate surfaces to the final composited desktop, but this is not strictly true. On mobile devices, and in the process of coming to desktops (there's some implementation in Windows 10 for Kaby Lake and higher integrated graphics) is the use of hardware overlays to replace actually blitting the active window to an in-memory surface. When the overlay is available, it's both a lower latency path and also saves the GPU memory bandwidth of that blit. And generally the heuristic is that the application window is a single surface, in other words that it does not rely on the compositor.
In order to justify doing the final scene assembly under GPU control in the app, that has to be as efficient or more so than the compositor. I have some evidence that, for 2D scenes typical of UI, this is possible. Regions that are flat color, for example (which occupy a nontrivial fraction of total area) can be rendered with no memory traffic at all, save the final output stage. And most of the other elements can be computed cheaply in a compute kernel on the GPU.
The tradeoffs are complicated, and depend a lot on the details of the application and the platform it's running on. In the 2010s, a strong case can be made that a compositor-centric approach is ideal. But in the 2020s, I think, it's increasingly not. The compositor is evil because it saps latency, and because its seductive promise of power and performance gains holds us back from the future where we can use the GPU more directly to achieve the application's rendering goals.
> Regions that are flat color, for example (which occupy a nontrivial fraction of total area) can be rendered with no memory traffic at all, save the final output stage.
GPU rendering and compositing amount to the exact same thing most of the time. A "region of flat color", at least in principle, is just a 1x1 pixel texture that's "mapped" onto some sort of GPU-implemented surface that in turn is rendered onto the screen.
Hardware overlays merely accelerate the final rendering step; one can implement the exact same process either in hardware, or as a software-based "blitting" step.
This is good feedback that I need to be clear and avoid terminological confusion when I write that blog post.
Of course the compositor is using the GPU. The difference is entirely in how the capabilities of the GPU are exposed (or not) to the application. My thesis is that doing 2D rendering in a compute kernel is ideal for 2D workloads, because it lets the application express its scene graph in the most natural way, then computes it efficiently (in particular, avoiding global GPU memory traffic for intermediate textures) using the GPU's compute resources.
Of course you could in theory have a compositor API that lets you do region-tracking for flat color areas, and this would save some power, but no system I know works that way.
According to the second half of the post (the one starting at “Case studies: Ok, since the above was so abstract…”), it is relevant to Imgui, Flutter, Jetpack Compose, React, “DOM”, etc, but even someone who uses or knows a lot about any one of those, unless they got to that part, may not become aware that this post is relevant to them or contribute to the conversation.
(This comment has the same problem.)