I wrote the terminal canvas renderers in VS Code that has been called out a few times here. Initially I implemented a canvas renderer using just a 2d context to draw many textures which sped things up "5 to 45 times"[1] over the older DOM renderer.
Since then I moved onto a WebGL renderer[2] which was mostly a personal project, it's basically the first canvas renderer but better in every way since it works by organizing a typed array (very fast) and sending it to the GPU in one go, as opposed to piece meal and having the browser do its best to optimize/reduce calls. This was measured to improve performance by up to 900% in some cases over the canvas renderer, but actually much more than that if for example the browser has GPU rendering disabled and tried to user the canvas renderer on the CPU.
My opinion here is that canvas is a great technology, capable of speeding things up significantly and getting close to native app performance. It comes with very real trade offs though:
- Cost of implementation and maintenance is much higher with canvas. This is particularly the case with WebGL, there have been very few contributions to xterm.js (the terminal frontend component) in the WebGL renderer because of the knowledge required.
- Accessibility needs to be implemented from scratch using a parallel DOM structure that only gets exposed to the screen reader. Supporting screen readers will probably also negate the benefits of using canvas to begin with since you need to maintain the DOM structure anyway (the Accessibility Object Model DOM API should help here).
- Plugins/extensibility for webapps are still very possible but requires extra thinking and explicit APIs. For xterm.js we're hoping to allow decorating cells in the terminal by giving embedders DOM elements that are managed/positioned by the library[3].
More recently I built an extension for VS Code called Luna Paint[4] which is an image editor built on WebGL, taking the lessons I learned from working on the terminal canvas renderer to make a surprisingly capable image editor embedded in a VS Code webview.
Okay, if anyone else after reading Accessibility needs to be implemented from scratch felt ashamed, raise your hands with me. SW engineers suffer from assuming everyone is like them and there are no corner cases. My Mom was recently sued for violating ADA with her real estate website not working with screen readers well enough.
I’ve never worked somewhere that treated a11y as a feature, with the commensurate resources put towards implementing it. It’s always ignored and then sometimes maybe worked on as an afterthought, during a hackathon or whatnot.
In other words, even when engineers are aware of it and inclined to do something about it, mgmt still has to care, and I’ve just never seen that once.
Do you think the large performance benefits can be achieved for any general web app (e.g. if I rewrite my Vue app's render functions to using a canvas instead of the DOM) or is the canvas' benefits mainly for niche workloads?
Definitely niche workloads or when the performance benefit from a UX perspective is worth the cost of implementation. Start out with virtualizing the DOM so only the visible parts are showing, if the framerate isn't acceptable after that then consider switching to canvas.
Using the terminal as a case study, its DOM renderer needs to swap out many elements per row in the terminal every frame (the number depends on text styles and character widths) and we want to maintain 60fps. It's also not just a matter of maintaining low fps since more time rendering means less time processing incoming data because they share the main thread, which means commands will run slower.
In my experience no - at least on current iterations - for very specific things such as a text editor where the DOM isn't really prepared to deal with the way it (the editor) has to be structured probably yes, if you knew what you're doing - but for most things not really - and to have the same functionality you would need to implement a lot of things by yourself (even if functionally it would work it wouldn't have the same accessibility unless you did that yourself and not sure how much you can fully emulate it).
From what I understand DOM is pretty shit in terms of performance because it needs to support so much legacy crap (eg. float layouts) - so even using sandboxed WebGL (which adds overhead over native APIs which your browser would use to render) you can still be much faster.
> Supporting screen readers will probably also negate the benefits of using canvas to begin with since you need to maintain the DOM structure anyway (the Accessibility Object Model DOM API should help here)
The fact that the DOM elements are invisible (don't affect layout) should eliminate the majority of the performance cost, right?
They can't be display: none as that would mean the screen reader can't access them. To do this properly and help low vision people, you need to make sure the textarea is synced with the cursor position and that all the text is positioned roughly where the text on screen is. By doing this the screen reader will correctly outline the element being read.
There may also be additional costs like the string manipulation required to build the row text in the terminal, this is nothing that can't be optimized but then that's more memory and cache invalidation to worry about.
Since then I moved onto a WebGL renderer[2] which was mostly a personal project, it's basically the first canvas renderer but better in every way since it works by organizing a typed array (very fast) and sending it to the GPU in one go, as opposed to piece meal and having the browser do its best to optimize/reduce calls. This was measured to improve performance by up to 900% in some cases over the canvas renderer, but actually much more than that if for example the browser has GPU rendering disabled and tried to user the canvas renderer on the CPU.
My opinion here is that canvas is a great technology, capable of speeding things up significantly and getting close to native app performance. It comes with very real trade offs though:
- Cost of implementation and maintenance is much higher with canvas. This is particularly the case with WebGL, there have been very few contributions to xterm.js (the terminal frontend component) in the WebGL renderer because of the knowledge required. - Accessibility needs to be implemented from scratch using a parallel DOM structure that only gets exposed to the screen reader. Supporting screen readers will probably also negate the benefits of using canvas to begin with since you need to maintain the DOM structure anyway (the Accessibility Object Model DOM API should help here). - Plugins/extensibility for webapps are still very possible but requires extra thinking and explicit APIs. For xterm.js we're hoping to allow decorating cells in the terminal by giving embedders DOM elements that are managed/positioned by the library[3].
More recently I built an extension for VS Code called Luna Paint[4] which is an image editor built on WebGL, taking the lessons I learned from working on the terminal canvas renderer to make a surprisingly capable image editor embedded in a VS Code webview.
[1]: https://code.visualstudio.com/blogs/2017/10/03/terminal-rend...
[2]: https://code.visualstudio.com/updates/v1_55#_webgl-renderer-...
[3]: https://github.com/xtermjs/xterm.js/issues/1852
[4]: https://marketplace.visualstudio.com/items?itemName=Tyriar.l...