
Towards a unified theory of reactive UI - raphlinus
https://raphlinus.github.io/ui/druid/2019/11/22/reactive-ui.html
======
svat
Though I don't know much about this topic, this seems an underrated post. I
think it would get more readership/discussion if it followed the principle of
proceeding from the concrete to the abstract. Specifically, in its initial
paragraphs it would be good to mention _what_ it was a unifying theory of: a
large laundry list of languages, libraries, and JavaScript frameworks that are
relevant to the topic. That way anyone who uses any of them would know it's
relevant.

According to the second half of the post (the one starting at _“Case studies:
Ok, since the above was so abstract…”_ ), it is relevant to Imgui, Flutter,
Jetpack Compose, React, “DOM”, etc, but even someone who uses or knows a lot
about any one of those, unless they got to that part, may not become aware
that this post is relevant to them or contribute to the conversation.

(This comment has the same problem.)

------
vially
Thanks for these great articles. I love your blog and your work on druid. Keep
them coming!

This is a space I'm very interested in. Having a high-quality, reactive UI
toolkit for developing native desktop apps might make it easier for web
developers accustomed to these reactive frameworks (e.g.: React) to develop
native apps (without having to resort to hacks like Electron).

Linux in particular could really use more developers developing native desktop
apps and I feel the current UI toolkits (QT/QML, GTK) and in particular their
language choice (C++, but I'm aware there are bindings for other languages)
are not appealing enough to the casual web developer. It could be argued that
Rust might not be the answer for these developers anyway, but one can still
hope.

~~~
Longhanks
How is Rust a better choice for the „casual web developer“?

Both Qt and Gtk have Python bindings, which would be far more in the realms of
a web developer (dynamic typing, no compilation, garbage collection...).

~~~
bluejekyll
Maybe you’re asking the wrong question. A different way to phrase this
question will lead you to answers about why this is so exciting for a lot of
people.

Why is Rust a great option for building native applications? Rust is a great
option for building applications that would traditionally be in C/C++. These
languages are traditionally chosen to have lower overhead than other options.
You could build a new Qt or GTK in Rust, and put similar Python bindings over
it, just as you described to go after those developers, if that’s desirable.

In addition to that, many casual web developers are recognizing the benefits
of a compiler with a type checked language, it’s why Typescript is growing so
quickly.

So you want to choose Rust when you want a lighter weight application, that
has lower memory overhead, generally faster runtimes, and a long term plan
around maintenance (the type system and various safety guarantees). Or more
succinctly, because you’re someone who enjoys writing programs in Rust.

------
malkia
Since it mentions various other frameworks and approaches (imgui, flutter,
etc.) - this video here gives overview of the three tree hierarchies flutter
itself uses: in their lingo: widgets, elements and renderObjects -
[https://www.youtube.com/watch?v=996ZgFRENMs](https://www.youtube.com/watch?v=996ZgFRENMs)

Basically, widgets (in this context) are the immutable parts (e.g. "config"),
"elements" would keep the updates, and relationships (parent/child), and
renderObjects - layout, drawing, etc.

------
75dvtwin
Thank you for sharing the write up.

I think the model you are describing, has parallels with Yaron Minksy's Data
Driven UIs, incrementally

[https://www.youtube.com/watch?v=R3xX37RGJKE](https://www.youtube.com/watch?v=R3xX37RGJKE)

Perhaps if you could review the above (if you have not done already),
additional ideas/data points might help with your formalization effort.

~~~
raphlinus
Thanks, great resource. I've had Incremental explained to me by a Jane
Streeter and I'm sure it influenced my thinking, but it's always good to have
public explanations.

~~~
fyp
If the academic stuff is interesting to you, then take look at functional
reactive programming stuff from the scala ecosystem.

For example there is a paper by Martin Odersky himself on incremental
computations on lists (which should be very familiar to you due to your work
on Xi's ropes):
[https://link.springer.com/chapter/10.1007/978-3-642-39038-8_...](https://link.springer.com/chapter/10.1007/978-3-642-39038-8_29)

------
proc0
Good summary of techniques. I'm very interested in this subject matter and
really want to get to a formal definition for this architecture of UI. As
mentioned in the article, FRP is the closest here and where the techniques
initialized from. The patterns deal with how to break down the problem of
asynchronous rendering and user interactions (which inherently deals with
state changes). The best papers I've read use some Category Theory to show the
structure of the flow of certain patterns, and there are also adjacent fields
like parallel and concurrent programming that I think are overlapping with
this area of how to manage UI state. The recent articles on CRDTs and Causal
Trees made me think that could be used not just across a network but to manage
the asynchronous state of components.

[http://archagon.net/blog/2018/03/24/data-laced-with-
history/](http://archagon.net/blog/2018/03/24/data-laced-with-history/)

------
Razengan
Importing the discussion from previous posts on this topic [0]:

I hope all new UI systems aim for SwiftUI's syntax. Not to copy Apple, but
because you can't really get more succinct that this:

    
    
        ZStack() {
    
            Rectangle()
                .foregroundColor(.unicornPuke)
    
            VStack() {
                Text("Hello")
                Text("World")
            }
        }
    

If you can make it nicer and cleaner than that, go for it. The specific
constructs for specifying view modifiers and properties could of course be
slightly different in each language; for example we could do away with the {}s
in Python, or the ()s might be []s. And maybe we could reduce the impression
of "magic" that SwiftUI's @State, @Binding etcetera seem to give.

It would just be a model for describing the UI, that a hypothetical engine
backend would take to spit out OS-native widgets. For another OS, you'd plug
in a different engine.

Having a universally agreed-upon syntax would provide the benefit of "learn
once, write anywhere", greatly reducing the friction in building cross
platform apps with native APIs.

[0]
[https://news.ycombinator.com/item?id=21495338](https://news.ycombinator.com/item?id=21495338)
\- Rust 2020: GUI and Community

~~~
antihero
What if we used JSX? It's got quite a high level of adoption and is pretty
readable.

~~~
paulddraper
> pretty readable

Not in a million years would I have guessed JS devs would praise XML for
readability.

~~~
antihero
There have been many surprises. Microsoft making C# dockerizable, making the
best lightweight opensource IDE, and having a literal version of Linux running
inside windows, have been some of the big shockers for me.

And yeah, I totally hate XML, would still never use it for data or config, but
for some reason JSX just kinda feels good.

------
FpUser
I did not read full post but the introduction kind of surprised me. The
example of "object oriented style" the author gives is artificially inflated
to look ridiculously verbose. Modern languages with OOP and anonymous
functions capabilities and some library help can make code the same or even
less verbose representation than the declarative example mentioned.

~~~
dthul
I believe the main motivation for these code snippets was not to point out the
verbosity of the OOP style, but to draw attention to the fact that the onclick
handler mixes the concerns of updating the state and updating the UI. Or in
the author's words:

"[...] duplication between the initial construction of the UI and updates.
[...] to add a slider showing the value, the onclick handler also needs to be
amended to add an update for that".

------
tz-lom
You should take a look on Qt and QML The render tree is only one aspect whis
is dealt by JS frameworks, because all of them are sitting on the fat neck of
browser. For a stand-alone UI you will need not only bubbling through layers
events like mouse clicks but also non-layered functionality like keyboard
focus, tab orders and screen reader support for blind users.

------
iudqnolq
Note that their images don't scale properly so if you browse on mobile and it
looks like you're only seeing half a chart try opening the image in a new tab.
I think this is because of the way they're including SVGs.

~~~
raphlinus
Works for me (tested in Pixel / Chrome and iPad / Safari). A bug report, or
better yet, PR against the repo would be welcome, not just for this but
because I plan to use SVG for most of my 2D/GUI related content in the coming
months. (And I have a lot planned!)

~~~
iudqnolq
My bad. Issue occurs in FF mobile preview and not latest non preview FF
mobile, so it's probably not your problem. I filed a bug report with Mozilla.

------
mwilcox
Is it really complete if it ends with "-> GPU"?

~~~
raphlinus
Well, that's quite the subtle question. Yes, in my plan for world domination,
the representation of display lists is also immutable trees (or, more
precisely, DAG's) hosted on the GPU, and the incremental nature of the
pipeline is preserved, so small diffs to the UI result in small uploads of
graph to the GPU. But it also includes damage region tracking, which is not
really a tree, and the very end of the pipeline is pixels, which is definitely
not a tree.

~~~
stefan_
There seems to be a misunderstanding. On all mobile phones and increasingly
desktops, if your UI involves the GPU, you have failed; the goal is to have
large layers of a scene handled entirely by the display compositor that
produces the final pixels. Using the GPU as an enhanced blit engine is a
colossal waste of memory bandwidth and power that no one can afford.

~~~
raphlinus
This will the the tl;dr of my upcoming blog post "the compositor is evil",
referenced in the post.

I hear what you are saying, but I think the situation is a lot more
complicated than that. Certainly, these days, if you're _not_ using the GPU
then you have failed; with high dpi and high frame rate displays, combined
with the end of Moore's Law, CPU rendering is no longer viable. So the
question is _how_ you use the GPU. I agree with you that power is the primary
metric of how we should measure this (it certainly correlates strongly to
memory bandwidth, and is the metric users will actually care about; if there
were a magical way to achieve high memory bandwidth at low power, I don't see
any problem with that).

One way certainly is to use the compositor, and this is very seductive. The
major platforms all expose an interface to the compositor, and it's generally
pretty well optimized for both performance and power usage. Since animation is
part of the API, it's possible to do quite a bit (including, for example,
cursor blink in an editor) without even waking up the app process. Scrolling
is another classic example where a compositor can do very well.

However, the compositor comes with downsides. For one, it forces the UI into
patterns that fit into the compositor's data model. This is one reason the
aesthetic of mobile UI design is so focused on sliding alpha-blended panes of
mostly static content.

But even from a pure performance standpoint, heavy reliance on the compositor
has risks. It's tempting to think of the composition stage as "free" because
it has to blit the intermediate surfaces to the final composited desktop, but
this is not strictly true. On mobile devices, and in the process of coming to
desktops (there's some implementation in Windows 10 for Kaby Lake and higher
integrated graphics) is the use of hardware overlays to replace actually
blitting the active window to an in-memory surface. When the overlay is
available, it's both a lower latency path and also saves the GPU memory
bandwidth of that blit. And generally the heuristic is that the application
window is a single surface, in other words that it does _not_ rely on the
compositor.

In order to justify doing the final scene assembly under GPU control in the
app, that has to be as efficient or more so than the compositor. I have some
evidence that, for 2D scenes typical of UI, this is possible. Regions that are
flat color, for example (which occupy a nontrivial fraction of total area) can
be rendered with no memory traffic at all, save the final output stage. And
most of the other elements can be computed cheaply in a compute kernel on the
GPU.

The tradeoffs are complicated, and depend a lot on the details of the
application and the platform it's running on. In the 2010s, a strong case can
be made that a compositor-centric approach is ideal. But in the 2020s, I
think, it's increasingly not. The compositor is evil because it saps latency,
and because its seductive promise of power and performance gains holds us back
from the future where we can use the GPU more directly to achieve the
application's rendering goals.

~~~
zozbot234
> Regions that are flat color, for example (which occupy a nontrivial fraction
> of total area) can be rendered with no memory traffic at all, save the final
> output stage.

GPU rendering and compositing amount to the exact same thing most of the time.
A "region of flat color", at least in principle, is just a 1x1 pixel texture
that's "mapped" onto some sort of GPU-implemented surface that in turn is
rendered onto the screen.

Hardware overlays merely accelerate the final rendering step; one can
implement the exact same process either in hardware, or as a software-based
"blitting" step.

~~~
raphlinus
This is good feedback that I need to be clear and avoid terminological
confusion when I write that blog post.

Of course the compositor is using the GPU. The difference is entirely in how
the capabilities of the GPU are exposed (or not) to the application. My thesis
is that doing 2D rendering in a compute kernel is ideal for 2D workloads,
because it lets the application express its scene graph in the most natural
way, then computes it efficiently (in particular, avoiding global GPU memory
traffic for intermediate textures) using the GPU's compute resources.

Of course you could in theory have a compositor API that lets you do region-
tracking for flat color areas, and this would save some power, but no system I
know works that way.

