
How to Achieve Polished UI - robin_reala
http://chrislord.net/index.php/2017/05/27/free-ideas-for-ui-frameworks-or-how-to-achieve-polished-ui/
======
panic
_As careful as you may be, it’s practically impossible to write software that
will remain perfectly fluid when the UI can be blocked by arbitrary
processing._

Well, that's what they did! All UI events and layout on the original iPhone
were handled on the main thread. I doubt asynchronous layout or event handling
would have improved the experience on its single-core CPU.

The key technical advantage the original iPhone had was Core Animation, which
_composited_ the laid-out views and applied animations to them in a separate
process. It ensured that all views would appear at the correct position in
their animations each frame with no jitter, and kept most of the per-frame
work in one place. But the animations were all initiated on the same main
application thread that handled events, performed layout, and so on.

~~~
yorwba
> All UI events and layout on the original iPhone were handled on the main
> thread.

But what else happens on the main thread? The way I understand the article, UI
and input are delegated to a dedicated compositor thread to prevent the heavy
processing on the main thread from interfering with responsiveness.

I would assume that iOS also separates the timing-sensitive UI handling from
anything that might take longer than a single frame to process. Either way,
you end up with one thread doing the heavy lifting and another keeping the UI
responsive.

~~~
zigzigzag
I found the article a little confusing to be honest. I wonder if the author
has written a traditional widget toolkit that isn't Firefox oriented.

In old widget toolkits, going back to the 90s here, there was a single UI
thread per app that did all drawing and sending of commands to the graphics
hardware. Keeping the UI responsive on such toolkits simply meant doing things
as much as possible in the background. Touching the UI data structures from
other threads was forbidden.

This architecture was adopted due to painful experiences with attempts to
build thread-safe toolkits in the 80s such as Motif and the original Win32
widget library. None of it worked very well. Motif apps tended to be deadlock
prone and Win32 was just a total API nightmare because it tried to hide the
thread affinity of the underlying widgets, but didn't do a good job of it.

Some systems in the 90s like NeXT and BeOS started experimenting with moving
the rendering into a separate process, the window server. Note that X Windows,
despite having a window server, did not use "retained mode" rendering and
still required the app to respond to do every repaint such as if an occluded
window was moved to the top. Systems with this sort of retained mode rendering
pushed "draw lists" into the window server so the OS could draw the window
from memory without having to wait for the app to respond. This used more
memory but meant that overall window UI stayed responsive and fluid even if
apps were under heavy load. However, anything that could _change_ the UI like
needing to respond to user input, of course stayed in the app and on the UI
thread.

MacOS X introduced a variant of the design, which I know less about, but I
believe it basically just stored fully rendered copies of the image. _Very_
RAM intensive and one reason MacOS X was considered very slow and heavy in the
early days, but it made it possible to do things like the genie effect and
exposé later on where the window server could animate the contents of windows
without the app needing to be responding.

All that is OS level compositing. The app itself did not do any asynchronous
compositing. So dragging windows around was fast, but animations inside the
app didn't benefit.

So the next level of asynchronicity is toolkits that push app level rendering
into a separate thread too. iOS, JavaFX, modern versions of Qt and modern
versions of Android work this way. In these toolkits, the app's GUI is still
constructed and manipulated on the primary/UI thread, but when the main thread
"renders" the UI, it doesn't directly draw it, it constructs a set of draw
lists for the apps own use. Again, these draw lists look a bit like this:

1\. Clear this area of the window to this colour.

2\. Draw a gradient fill from here to there.

3\. Draw texture id 1234 with that shader at these coordinates, at 50%
opacity.

4\. Invoke remembered draw list 111.

5\. Remember this set of instructions as draw list 222.

Once these lists are created they're handed off to a dedicated render thread
which starts processing them and turning them into commands to the GPU via an
API like OpenGL or Direct3D. Note that these APIs are, in turn, simply
creating buffers of commands, which eventually get dispatched to the GPU
hardware for actual rendering. Because the render thread doesn't run any
callbacks into app code, and because it's cooperating with the GPU hardware to
remember and cache things, it doesn't have that much actual work to do and can
process simple animations very fast and reliably.

However, responding to user input is still done on the main thread. If you
block the main thread, your UI will continue to repaint and may exhibit simple
behaviours like hover animations, but actually clicking buttons won't work.
That's because the most common thing to do in response to user input is change
the UI itself in some way, and that must still be done on the main thread.

I hope that helps.

~~~
lenkite
Thanks for this very informative post! However, I think you disparage good old
win32 just a tiny wee bit. It was quite well designed for its time - perhaps a
little too ahead of its time. Everything was async, to communicate between
windows you needed to post events to queues, you could customize the window
classes any way you wished - it was perhaps an extraordinarily sophisticated
and flexible framework and lots of talented folks made it dance. It had a
really good and long 20+ year run. (and still running strong in some desktop
software)

~~~
derefr
To me, Win32 was already falling away from the more intriguing model, which
was Win16.

Effectively, a Win16 system is/was basically exactly equivalent to an Erlang
node, but one where your "processes" just happened to be paired, component-
wise†, with handles to structs of GUI properties held in window-manager
memory.

Like an Erlang node, Win16 is/was:

• green-threaded — i.e. they both have very low-overhead in-memory structures
containing a tiny heap/arena and a reference to an in-memory delegate code
module, through which execution would pass in turn. In Erlang, these are
"processes"; in Win16, these are _windows_ —i.e. actual windows, but also
"controls" in the controls library.

• cooperatively-scheduled (yes, Erlang is cooperatively scheduled—when you're
writing C NIFs. When you're executing HLL bytecode in the Erlang VM, this fact
is papered over by the call/ret instructions implicitly checking reduction-
count and yielding; but if you're writing native code—like in Win16—you do
that yourself.)

• Message-passing, with every process having a message inbox holding
dynamically-typed message-structs that must be matched on and decoded, or
discarded.

• Offering facilities to register and hold system-wide handles to large static
data-blobs (Erlang large-binaries, Win16 rsrc handles);

• Capable of doing IPC only by having processes post messages to another
process's queue, and then putting themselves into a mode that waits for a
response;

• Based on a supervision hierarchy: in Erlang, processes spawn children
(themselves processes) and then manage them using IPC; in Win16, root-level
windows spawn controls (themselves windows) and then manage them using IPC.

And, crucially, in both of these systems, real machine-threads are irrelevant.
Both Win16 and Erlang were written in an era when "concurrency" was a desired
goal but multicore didn't yet exists—and so they don't really have any concept
of thread affinity for processes/windows. Both systems are designed as if
there is only one thread, belonging to a single, global scheduler—and then
their multicore variants (SMP Erlang and Win32) attempt to transparently
replicate the semantics of this older system (though in different ways: Erlang
allows processes to be re-scheduled between scheduler threads, while Win32
pins windows to whatever scheduler-thread they're spawned on.)

Win32 later introduced an alternative model to take better advantage of
multithreading: COM "multi-threaded apartments", allowing windows (or, as a
generalization, COM servers, which could now execute and participate in IPC on
a thread without spawning any window-instances) to interoperate across thread
boundaries without requiring a message be serialized and passed through the
scheduler/window-manager process.

† [http://cowboyprogramming.com/2007/01/05/evolve-your-
heirachy...](http://cowboyprogramming.com/2007/01/05/evolve-your-heirachy/)

~~~
zigzigzag
Well, I hope we're not getting sentimental about Win16. I've written more than
my fair share of Windows API code and even though some of the concepts are now
coming back into fashion again, largely due to the limits of browser engines,
it really isn't an era I'd return to at all. Those models were all abandoned
for solid reasons and Apple's failure to do so in time nearly killed the
company.

I'm not sure your description of COM is quite right. The way I remember it,
windows (HWNDS) were and still are objects with thread affinity. COM had the
notion of a "single threaded apartment" which basically meant the COM server
received RPCs using regular Window messages, and the MTA that you mention
simply meant no inter-thread marshalling was done at all i.e. the object was
inherently thread safe using locks or whatever. But Windows never changed to a
model where the controls library was thread safe: changing the contents of an
edit box from another thread, for instance, always required a context switch.

COM's usage and abusage of the window message system for fast inter-thread
switching was only ever an ugly hack, which caused all kinds of weird problems
and glitches. Most obviously it caused Windows' reliance on actually having a
GUI layer to deepen considerably because now inter-thread/inter-process RPC -
that on Linux and MacOS were well modularised into things like Mach IPC,
SunRPC, DBUS etc - were totally tied to the windowing system.

IIRC the entire apartment concept was also stupidly designed, so there were
constant problems with Microsoft using COM internally to implement some APIs,
which would by default enter an STA and require the _caller_ of the API to
pump the message queue otherwise the API they'd just used would silently fail
to work. In the era I was working with it that fact wasn't always properly
documented, I think.

~~~
derefr
> COM's usage and abusage of the window message system for fast inter-thread
> switching was only ever an ugly hack

This is effectively the entire point I wanted to dispute in my original post
above; I guess I didn't get it across clearly enough.

As can be seen from how pre-COM (e.g. DDE, OLE) IPC was achieved on Windows,
Microsoft truly believed that sending messages through the window-manager was
a _good_ way to do IPC. Their designs just kept doing it, over and over. STA
COM messaging wasn't a hack; it was more of the same, a doubling-down on a
long-standing design paradigm. _MTA_ COM messaging was the hack—a way to make
everything continue to _look like_ HWND messaging (with an abstraction layer
added), but have it transparently optimize to SHM IPC in cases where that was
beneficial [and where the developer had ensured their ADTs were compatible
with it.]

> Most obviously it caused Windows' reliance on actually having a GUI layer to
> deepen considerably because now inter-thread/inter-process RPC - that on
> Linux and MacOS were well modularised into things like Mach IPC, SunRPC,
> DBUS etc - were totally tied to the windowing system.

And, driven by the "evidence" of this repeated doubling-down above, I would
conclude that _this was the point_ : Microsoft considered Windows to be about,
well, _windows_.

As I was saying above, a "window" in the Win16 sense was effectively the same
thing as an Erlang process, but with some extra (optional-to-use!) GUI data
stuck to it. The "correct" way to achieve async parallelism in Win16 was
literally to create a "background window" that would register a kernel timer
to send it tick events, and then do work when it received one. Which is the
_same_ thing you do if you want to write an Erlang process to wake up and poll
some data source every so often.

My point isn't just that there are parallels here; my point is that Microsoft
expected you to _use_ the "window" primitive in _exactly_ the ways that Erlang
expects you to use the "process" primitive. Windows _are_ the "process"
primitive of Win16—they're tiny, green-threaded processes, and the window-
manager _is their scheduler_.

That statement should make Microsoft's views on IPC clearer. Of course Windows
IPC is achieved by putting messages through the window-manager. The window-
manager _is_ the scheduler†; knowing about other window-processes and routing
messages to them is its _job_. It _is_ DBUS—and it is also, given DDE, the
equivalent of macOS's LaunchServices daemon.

\---

† ...or rather, the window-manager is the scheduler _for anything that 's not
a DOS VM_. Windows, from 2.x through to 9x, was effectively a two-layer
system: a bare-metal hypervisor "kernel" (KERNEL.EXE/KRNL386.EXE) with one
Windows dom0 and N DOS domUs; and then an OS "kernel" running in that Windows
dom0. That dom0 OS kernel _is_ GDI.EXE, and cooperative message-passing _is_
its scheduling algorithm. It also happens to do graphics. (It's a
_paravirtualized_ kernel that relies heavily on the hypervisor kernel above
it, yes, but it's still the kernel of the Windows domain.)

------
dep_b
It's quite useless without examples. I know some frameworks do all rendering
in a separate thread but you almost completely forfeit all UIView based
rendering. With the native UI toolkit you always go back to the main thread so
I only put long running processes like network requests, image rendering or
database access in a separate thread.

------
korijn
Reminds me of common practises in game engines.

~~~
subb
Yea. It's not very hard - if you want something responsive, you have a limited
amount of time between when you read input and when you present something on
screen.

This delta can be greater than the rate at which you present images. For
example, most game engines take 30ms to read input and simulate on a thread,
then take 30ms to renders on another thread while the next frame is simulated.
This means that they present something every 30ms, but the delay between input
and rendering is 60ms. This delay can be up to 100ms and not be noticed by
players.

~~~
pikzen
A 100ms delay is extremely noticeable, especially for faster paced games.
That's 6 frames at 60FPS, or 3 at 30. While you may get away with that on 4X
games, no FPS will ever get to more than 50ms input lag, because it seriously
becomes unpleasant to play.

------
retox
In landscape mode on mobile there is a bar that takes up 20% (at a guess) of
the screen just to contain a single icon.

Does everyone browse mobile in portrait? My eyes are horizontally arranged.

~~~
ivanhoe
How do you hold your book, in landscape or portrait mode? Pages in almost all
scripts are traditionally always in portrait orientation.

~~~
gurkendoktor
Pages are also traditionally larger than a cellphone. On a 4"/5" smartphone,
text in landscape orientation is much closer in font size and line length to a
normal book than in portrait orientation.

------
wruza
It is ironic how the author is speaking about polished iPhone UI and his blog
has big stupid "to the top" button^ fixed at the bottom, covering two lines of
the text. Every iphone user knows that he can go to the top by tapping above
the address field, no buttons required.

^ not actually a button, but "polished" invisible plane that randomly cuts
text.

Can we please, please return to the era when ui was humane?

~~~
PretzelFisch
not every iphone user. magic UX like that is not very discoverable.

~~~
wruza
I'm pretty sure it was described on small introduction booklet in iphone 4
package. "Shake to undo" was also there.

Do you guys ever read instructions? :)

~~~
PretzelFisch
I don't recall paper instructions in the iphone 6 package. Maybe they put up
an ill timed tutorial on screen with the first boot. I needed to skip that to
finish setting up the phone in store. Sometimes I think getting a user to read
instructions is the worlds hardest problem.

------
an27
Would anyone here have resources on writing similar highly-responsive GUI
frameworks?

I'm considering writing a Rust-based one with Linux, BSD, macOS, Windows and
Android backends.

