
Immediate Mode GUI - sytelus
http://behindthepixels.io/IMGUI/
======
60654
Oh no, not again! (Just kidding :) )

Immediate mode UIs come up every now and then, especially in games, and for
relatively simple things they're a neat idea (imagine not retaining any extra
view state!) But - and there's a big _" but"_ coming up - it does not scale to
complex and stateful UIs.

The problem is that, for games that have complex needs - think simulation and
strategy games, not racing or arcade - complex UIs are needed to express
complex relationships between underlying data and the relevant UX flows. But
how do you do that in IMGUI? This information about UI state, layout, etc. is
not in the UI anymore, but it has to live _somewhere_ \- so you end up having
to build _your own_ layout and UI management stuff, and basically duplicate
what a RMGUI library would offer, but now with less generality and potentially
more bugs.

For the probably best known example, Unity used to have an IMGUI [1] but
they've actively moved away from it. It's still there in vestigial form, the
API is not removed, but game developers don't really use it.

[1]
[https://docs.unity3d.com/Manual/GUIScriptingGuide.html](https://docs.unity3d.com/Manual/GUIScriptingGuide.html)

PS. And there's also the lack of visual layout tools, which is directly
related to IMGUI's reliance on code to produce layout.

~~~
mellinoe
> For the probably best known example, Unity used to have an IMGUI [1] but
> they've actively moved away from it. It's still there in vestigial form, the
> API is not removed, but game developers don't really use it.

It's true that the newer retained mode GUI has supplanted the older IMGUI for
"game UI" (e.g. what users interact with when playing an actual game built
with Unity). On the other hand, the Unity editor itself is still built
entirely with their immediate-mode GUI library, and I haven't heard of any
plan to move away from that. Building "editor-like" tools (including custom
user tools) is still significantly easier to accomplish using immediate-mode
libraries.

~~~
60654
Oh, they're switching to RMGUI in the editor as well, though more slowly. More
info here:

[https://blogs.unity3d.com/2019/04/23/whats-new-with-
uielemen...](https://blogs.unity3d.com/2019/04/23/whats-new-with-uielements-
in-2019-1/)

------
sytelus
TLDR;

Typically we build GUIs using event driven callbacks. This causes problems in
managing life times, dynamic layouts, threading issues etc. The idea behind
immediate mode GUI is that you get a frame and in each frame you create/update
layout, check for user actions - just like you do in render frame in game
loop. So there are no call backs and you don't have to worry about issues
arising from event-driven code.

~~~
jboles
Does this pretty-much rule out (without a huge amount of effort) nonmodal
GUIs? It seems without callbacks, you would need to roll your own window
management, Z-ordering of overlapping UI elements, state transition validation
(e.g. clicking on the parent dialogue’s Cancel button while a child window is
still open), ...

~~~
mmozeiko
Here's an browser demo for dear imgui
([https://github.com/ocornut/imgui](https://github.com/ocornut/imgui))
library:

[https://pbrfrat.com/post/imgui_in_browser.html](https://pbrfrat.com/post/imgui_in_browser.html)

For modal demo check "Popups & modal windows" -> "Modals" example.

This is how it looks in C++ code:
[https://github.com/ocornut/imgui/blob/master/imgui_demo.cpp#...](https://github.com/ocornut/imgui/blob/master/imgui_demo.cpp#L2273)

~~~
tokyodude
Here's another

[https://greggman.github.io/doodles/glfw-imgui/out/glfw-
imgui...](https://greggman.github.io/doodles/glfw-imgui/out/glfw-imgui.html)

In the "ImGUI Demo" window start expanding the collapsed items or from the
menus pick "Examples"

Note that no effort has been spent to actually make the demo web browser
friendly so there are rough edges.

------
Stratoscope
Note to authors: please define your terms when you first use them, or provide
links to the definitions.

Of course I know what a GUI is. I've been building them for decades, and
building tools that other developers have used to build GUIs. (I was one of
the authors of the "visual" part of Visual Basic.)

But I have never before seen the terms "immediate mode GUI" or "retained mode
GUI". And I'm pretty sure there are many developers like me who will have no
idea what you're talking about. It takes only a moment to add a quick
definition or a link the first time you mention a specialized term, and your
readers will definitely appreciate it. Thanks!

(Edited to remove fluff and make the essential point more clear.)

~~~
tokyodude
What a rude comment. It's not a specialized term. It's a common term.
Immediate mode GUI as a concept is well known. Tens of thousands of
programmers use Immediate Mode GUIs. There are several well known Immediate
Mode GUI libraries used by tens of thousands of projects.

When I don't know a term I google it. I don't assume the entire world revolves
around me and that it's the author job to make their article more verbose
because everyone reading it might not know every term.

Here's are galleries of some from one of the more popular libraries

[https://github.com/ocornut/imgui/issues/2265](https://github.com/ocornut/imgui/issues/2265)
[https://github.com/ocornut/imgui/issues/1902](https://github.com/ocornut/imgui/issues/1902)
[https://github.com/ocornut/imgui/issues/1607](https://github.com/ocornut/imgui/issues/1607)
[https://github.com/ocornut/imgui/issues/1269](https://github.com/ocornut/imgui/issues/1269)

~~~
donaltroddyn
I don't think it's rude at all. I did Google "immediate mode GUI" after
reading the first paragraph, because I also didn't know what it meant. The
first three results were a Wikipedia snippet which didn't help my
understanding at all, a Gist that begins with "Before you continue, if you
don't know what IMGUI is don't bother reading this", and then third result was
this HN comments section.

I would argue that "immediate mode GUI" is an extremely specialised term.

~~~
Arnt
Specialised enough that I didn't come across it during my years of work on Qt.
Work that included reading all the manuals, style guides and UI books we could
get hold of.

~~~
loup-vaillant
> _Specialised enough that I didn 't come across it during my years of work on
> Qt._

Of course you didn't, Qt is retained.

It's a bit like imperative programming vs functional programming. Before
functional programming gained any significant traction, imperative programming
was just "programming".

~~~
Arnt
_Of course_ I didn't? I filled a wall with the books I bought and didn't come
across the term. What did I do wrong, what books could I have bought but
didn't?

~~~
loup-vaillant
How many of those books weren't about the mainstream GUI toolkits (Qt, GTK,
MFC, Cocoa, Swing…)? Has even _one_ of those books described something that
resemble "let's redraw everything every frame"? If so, how that technique was
called?

> _what books could I have bought but didn 't?_

Possibly none at all. Immediate mode _anything_ is mostly a game dev thing,
and what happens in game dev tends to stay in game dev. They don't seem to
document their practices as much as other fields do.

Hence my suspicion that the authors of your books didn't even know about
IMGUI. They just knew about "the" way to do GUI, which happens to involved
retained state.

~~~
Arnt
You're letting the tail wag the dog. You're assuming that people know terms
that are used in "possibly none [no books] at all", or put differently, that
the set of terms used in books for software developers is uncorrelated with
the set of terms software developers may be assumed to know.

BTW, the code in Qt that draws rotated/sheared text/pixmaps was influenced in
part by a book or chapter by Michael Abrash, as well as by articles in either
Graphics Gems or conference papers (memory's fallible I'm afraid).

------
userbinator
_Some also argue that executing UI logic of every widget is wasteful. This
might indeed be a concern when implemented poorly_

For a game, which is already quite busy even when "idle" (and there are
probably other things going on in each frame which take far more CPU and GPU
to render), it won't be much of a concern; but with an application where most
of the time is spent waiting for user input, spinning the CPU to re-render the
GUI every frame is very energy-inefficient.

~~~
frnkng
I don’t think that an imgui app will eat more resources than a comparable
electron app.

Thus the Performance Argument against any IMGUI is pretty pointless.

~~~
theamk
I disagree. With all downsides of electron, it does not generally re-layout
and re-render entire screen on any mouse move. IMGUI does.

So while Electron will lose in memory/disk used, IMGUI will lose in CPU usage
(battery life) once the software is complex enough.

~~~
tokyodude
I disagree with your disagreement :)

Electron (and nearly all retained mode GUIs) will execute 10x more code trying
not to execute the rendering code. Just skipping all that checking and jumping
to the rendering code is arguably a win in many many cases.

To put some empirical data on it, game devs use ImGUIs because they keep up
with perf where as that 10x overhead of retained mode guis creating gui
objects, deleting gui objects, marshaling data into and out of gui objects,
adding and removing event handlers and callbacks, and then trying to write a
bunch more code to optimize all of that never leads to a GUI that is
performant enough for the GUIs needed for game dev.

I think if you check actual apps, especially phone apps, you'll find it's not
so clear which one wins. Phone apps almost always move in relation to the
users fingers meaning the screen is being re-rendered. Usually new GUI
elements are moving on and off the top/bottom of the screen. The overhead of
creating/deleting/managing those objects is arguably greater than just re-
rendering. The screen is already going to be re-rendered. At best you're
getting a few cached textures from your retained mode gui all at the expensive
of managing GUI objects.

For desktop apps it really depends on the app. It's got nothing to do with the
complexity of the UI and everything to do with what the app does. A text
editor might be a win for a retained mode GUI. Photoshop/Maya it's less clear.
For example Unity is ImGUI and has very complex UI. Much of the UI has to
respond in realtime to the state of the user's document that is changing
constantly.

~~~
theamk
> Electron (and nearly all retained mode GUIs) will execute 10x more code
> trying not to execute the rendering code. Just skipping all that checking
> and jumping to the rendering code is arguably a win in many many cases.

How do you figure? Rendering a single Truetype letter will have dozens of
conditional branches, use complex floating point math, and touch tens of
kilobytes of cache. And there are many thousands of them in the GUI. Even if
you have rendered cache, there is still lots of memory pressure copying data
back and forth.

Compared to that, retained mode GUIs should be much more efficient. Even 50
checks are nothing compared to ability to void rendering one line.

Note this is all predicated on screen being mostly unchanging. For example, as
I am typing this comment, everything is stationary except for a single line
where I am typing.

> The screen is already going to be re-rendered.

This is the key! As you said, if you have to re-render whole screen anyway,
then retained mode GUIs will have to maintain draw state _and_ re-render every
element every time anyway -- a strictly worse performance.

But most of the regular apps, like editors and chat clients and web browsers,
do not re-render whole screen every time. They only do one thing at a time.

For an empirical evidence, look at WinAPI design: this is a retained mode GUI,
with lots of effort dedicated to figuring out invalidated rectangles and which
controls need to be rendered. This was the only way to make a GUI which is
performant enough.

~~~
tokyodude
And yet the WinAPI is not actually preformant for types of apps that an ImGUI
excels at

There's best case for Retained mode GUIs and best case for ImGUIs. ImGUIs
excel when lots is changing. In a scrolling mobile app the entire page is
being re-rendered constantly as the user moves the page up and down. GUI
widgets get created and deleted, data gets marshalled in and out, various
algorithms are applied to try to minimize re-creating stuff, all code that
doesn't need to be executed in ImGUI mode code.

There are tons of cases where ImGUIs win in perf. In fact the reason they are
so popular with for game dev is exactly because they vastly out perform
retained mode guis. There's a reason there are 100s of books and 1000s of blog
posts and articles trying to get retained mode guis to run smoothly. They
almost always fail without lots and lots of specialized client side code to
try to minimize GUI widget object creation/deletion, reuse objects, minimize
state changes, etc, all that code disappears in an ImGUI and they run at 60fps
where as all the retained mode guis struggle to get even 20fps consistently as
they hiccup and sputter.

------
chriswarbo
IMGUI always brings a couple of things to mind for me, which don't seem to be
discussed often:

Smalltalk/Self GUIs:

AFAIK, the idea (or at least terminology) of MVC comes from the Smalltalk80
GUI. MVC seems to be intrinsically retained-mode: the "view objects" (buttons,
forms, etc.) are separate to the "model objects".

Self was inspired by Smalltalk, and its Morphic UI seems to be immediate-mode:
the UI objects often _are_ the model objects. This idea is more explicit in
the "Naked Objects" approach. Squeak Smalltalk uses Morphic, although its
descendents (e.g. Pharo and Newspeak) seem to have replaced it (I'm not sure
of their paradigm though)

Denotative, Continuous-Time Programming (DCTP):

This was originally called "Functional Reactive Programming" (
[http://conal.net/papers/frp.html](http://conal.net/papers/frp.html) ), but
that term is now mainly applied to discrete, event-driven systems like Elm and
React, which can make some articles/papers a bit confusing to read ;)

In this approach, anything which varies over time (e.g. user input, display
widgets, application state, etc.) becomes a function taking the current time
as a parameter, called a "behaviour". These behaviours can be composed
together in various ways to create an application's overall behaviour, still
parameterised by time. The "main loop" just samples this behaviour at an
appropriate rate, e.g. a fixed frame rate, or varying based on system load,
how much activity is happening, etc.

This is very much IMGUI, but has some nice advantages over the fixed ticking
of a traditional main loop. For example, different components of a behaviour
can be sampled at different rates, e.g. rapid sampling for smooth animations,
coarse sampling for expensive calculations. Interpolation behaviours can
remove jerkiness (e.g. a smooth animation based on an expensive simulation).

~~~
scroot
> Squeak Smalltalk uses Morphic, although its descendents (e.g. Pharo and
> Newspeak) seem to have replaced it (I'm not sure of their paradigm though)

Pharo is still using Morphic, though there is a new graphics layer called Bloc
(vector and constraints based) that has been under development to replace it.

------
bdowling
Immediate mode GUIs are an extension of the simple display update loop: (1)
update display, (2) check input, (3) goto (1). Immediate mode GUIs add a
_context_ parameter to the update function. In the _draw_ context, the update
function draws to the screen. In the _input_ context, the update function
doesn't draw; instead it checks if mouse clicks or keystrokes should do
something.

For example, part of the update function might look like this:

    
    
        if (Button("Do That Thing")) 
            doThatThing();
    

In the draw context, the call to Button will draw a button on the screen and
return False. In the input context, the call to Button will check if a mouse
click fell within the area it would have drawn. If so, it will return True and
the callback will run.

An immediate mode GUI is basically the opposite of a typical Model-View-
Controller GUI because it deliberately mixes the Model, View, and Controller
together into one function. Just one function draws the GUI based on whatever
state it's in and whatever the model is. The same function gets reused in a
different context to update the GUI state or handle callbacks.

~~~
tokyodude
I'd argue an ImGUI does't mix anything. It's just a library. If you want MVC
it's up to you to implement your MVC and use your ImGUI as a library. Retained
mode GUIs don't magically make your code MVC and ImGUIs don't magically make
your code not MVC. That's entirely up to you.

Also ImGUIs are not multiple pass. That's in implementation detail. Unity's
ImGUI is multi-pass. One of the most popular ImGUIs (Dear ImGUI) is single
pass

[https://github.com/ocornut/imgui](https://github.com/ocornut/imgui)

------
nickflood
I've taken part in some WPF application development and I think there are some
benefits in immediate-mode rendering of the UI. Particularly when the UI
changes faster than a monitor refresh rate or there's too much data complexity
and you want the data update and redraw processes to be independent.

For my first example, updating controls and overall layout in a WPF
application feels much more lightweight than in WinForms because the renderer
just shows the result at the next monitor refresh instead of trying to update
the layout in realtime or using hacks that disable redraw until you do
everything.

For the second example, I've used SciChart to display trading data and it
again was much more responsive because it replaced a control-based rendering
pipeline with just an immediate-mode canvas tgat would render the selected
data window once per monitor refresh rate.

And I think overall once you hit the performance barrier where you can't just
fill a panel with controls and let them render and instead have to make a
custom draw routine, the immediate-mode rendering adds this extra smoothness
to the UI and gives you a bit more performance headroom.

------
BoorishBears
I find it ironic that Unity is used as an example... the immediate mode UI
was/is considered a joke.

Terrible performance, doesn’t scale to UIs more complex than a form (... like
the ones many games tend to have)

It got to the point where there was a website dedicated to track when we’d get
the current non-immediate UI, and Unity actually hired the developer of the
leading non-immediate mode UI library in the asset store to work on their own
offering.

~~~
tokyodude
I think you're mis-understanding why Unity switched. They only switched their
in game GUI system. The entire app of Unity itself uses an ImGUI system and it
is massively more complex than any Game UI and runs just fine.

The reasons they switched the in game system are many but they have nothing to
do with performance.

~~~
BoorishBears
The editor has none of the constraints the games made with it have... like
running on mobile and maintaining 60 FPS while doing it.

... but IMGUI is on the way out there too:
[https://docs.unity3d.com/Manual/UIElements.html](https://docs.unity3d.com/Manual/UIElements.html)

The old system would stack up draw calls like nobody's bussiness, doubling
drawcalls if you used the built in layout tools. OnGUI being called period
would be expensive, even when you had properly implemented your rendering
logic, so you’d have to disable the entire game object when you didn’t want to
draw.

The docs for mobile literally used to say not to use the OnGUI stuff during
gameplay.

------
dmix
Please add [2015] to title

~~~
soylentgraham
Also, 2011, 2007, 2001, 1996 :)

Immediate vs deferred, is just another paradigm that goes back and forth in
trends like keyboard input/buffering, deffered/one pass lighting, monoliths vs
modules, State vs events, flat vs 3D

------
jpnelson
Are there any popular implementations of immediate mode GUI for the web? It
might be interesting to have one that could then "eject" into a stateful UI
for when things start to get more complicated.

~~~
rtpg
I think that React is like 90% of the way there and you would just need to
mess around with the VDOM a bit for this.

To be honest, given that web render output is a tree instead of flat "render
to screen" stuff, writing a nice immediate mode GUI is not obvious.

If you're... very clever you could take React and the State stuff, at least
conceptually, and get what you want here.

~~~
kkarakk
wouldn't you need some sort of gpu access to have a smooth UI rendering with
immediate gui(in production)?

~~~
rtpg
This is if you expect to build an immediate mode with the same flat rendering
model.

imgui's model is "you draw onto pixels". You could create another mental model
which is "you draw onto a tree." so instead of saying "drawText(x, y, text)"
you would do something like "drawText(domLocation, text)"

Granted, this model gets a bit messier, and _some_ might say defeats the
purpose of immediate mode. But I think you could still get conciseness
benefits on this.

------
i_don_t_know
While I find the idea appealing, I haven't used them because I haven't figured
out how to make immediate GUIs accessible to people who are visually impaired.
How do you make them work with screen readers? You're drawing text onto a
canvas but you don't retain a copy of the text for the screen reader.

~~~
loup-vaillant
I don't know about screen reader APIs, but… can't you just send the text
you're drawing to the screen reader when you drawing it?

How do screen readers work anyway?

~~~
i_don_t_know
Good question. I don't know. When you use the native GUI you get all of that
for free. You don't have to do anything special for accessibility. So I always
thought that screen readers somehow traversed the window/control hierarchy to
find the control that has the focus and ask it for any text that should be
read.

Maybe you're right and you can send the text to be read directly to the screen
reader. Not sure how well that works when your GUI updates every 20ms. You
don't want to keep resetting the reader.

~~~
loup-vaillant
I was thinking maybe resend the stuff only when the text changes, so you'd
have to have a cache of some sort, hidden in the library.

Anyway, I believe Qt is screen reader compatible, so it must access the
relevant API _somehow_. I'd be surprised if it was the exclusivity of the OS
vendor.

------
mschaef
I'm going to date myself here, but this reminds me in some ways of Microsoft's
BASIC 7.0 (circa 1989).

One of the new features of 7.0 was a library for producing text mode GUIs with
dialog boxes and the like. Because the language didn't support anything
resembling a callback, everything had to be written in what was essentially
"immediate mode". Lots of control flow that would normally be down in the
library stayed up in user code. The net result aligned well with the
description in this article: "...much shorter but more intensive, every thing
is packed in a single function."

This approach also presented lots of opportunities for errors caused by
failing to correctly code something that would be default behavior in a normal
library.

Never seen anything else quite like it, aside from this. (Which, quite
frankly, I'm happy about.)

------
childintime
How does Flutter sit in the spectrum from Immediate mode to Retained mode? I
presume it might provide an interesting erspective. On the one hand widgets
are constructed on the fly, on the other it uses an OOP language to do so. So
I suspect this maps to statefulness and statelessness?

~~~
raphlinus
Flutter is implemented in layers. The RenderObject layer looks very much like
a traditional retained mode UI. It's _possible_ to write Flutter entirely at
this level, nobody forces you to use the Widget layer above it. The Widget
layer is basically React-style, where the API looks fairly similar to
immediate mode, but it uses caching and diff computation to reflect everything
into the retained RenderObjects below it.

------
avodonosov
Again IMGUI. Several years ago there was a chain of videos and blog posts by
different authors. People were drawing two buttons and saying "see how easy it
is".

In this post the author seems to go further than two buttons, but I'm
skeptical. I wish there was a video showing how it works in dynamics - inputs
clicked, texts entered, scrollbars scrolled, dialogs dragged around. Or source
code available to build and try locally.

From the code in the post I suppose the majority of UI is dead - can not be
interacted with.

I think IMGUI can only be convenient for very simple cases.

In real frameworks the lowest level is immediate mode (see WM_PAINT in
WinAPI), but real UI needs quickly evolve them into the traditional RMGUI.

BTW, this thinking is my source of doubts about React.

~~~
tokyodude
I think you have it exactly backwards. Most ImGUI use cases are extremely
complex UIs and that's where they excel.

One of the most complex apps out there, Unity, is entirely ImGUI based for the
app itself and often has 1000s of controls visible. Going though the
screenshots of typical ImGUI usage most of them are on the complex side.

[https://cloud.githubusercontent.com/assets/8225057/20628927/...](https://cloud.githubusercontent.com/assets/8225057/20628927/33e14cac-b329-11e6-80f6-9524e93b048a.png)

[https://raw.githubusercontent.com/wiki/ocornut/imgui/web/v16...](https://raw.githubusercontent.com/wiki/ocornut/imgui/web/v160/editor_white.png)

[https://user-
images.githubusercontent.com/16171743/50757906-...](https://user-
images.githubusercontent.com/16171743/50757906-f2bace80-1260-11e9-9ce5-4173b68cd9a2.jpg)

[https://user-
images.githubusercontent.com/8225057/51042368-6...](https://user-
images.githubusercontent.com/8225057/51042368-65f37600-15bc-11e9-99cf-36e9f9822610.jpg)

[https://user-
images.githubusercontent.com/870320/51073482-ba...](https://user-
images.githubusercontent.com/870320/51073482-ba0c6200-1671-11e9-9b44-0c573ab0fae5.png)

See
[https://github.com/ocornut/imgui#Gallery](https://github.com/ocornut/imgui#Gallery)
for more

------
alex-e
Do you think it's possible to build an immediate UI wrapper on top of Win32
and Cocoa?

------
whateveryou381
2015

