As of right now the GUI you are looking at is built with HTML and CSS (later on this will change). But what renders the page in the application's canvas and the published webpages is all within the canvas element.
Aside: The use of "partially sighted" in this context seems odd to me. After all, totally blind people also use screen readers; indeed, totally blind people rely most completely on screen readers.
So I'm curious about the rationale behind your word choice here. Are you afraid that partially sighted (or in fewer syllables, low vision) people like me would object to the word "blind". I do often clarify that I'm not totally blind. But in some cases (e.g. if I'm explaining to an Uber driver why I can't see them), I just say "I'm blind". And I would in no way feel excluded by "blind users with screen-readers", though as a low-vision user, I sometimes use a screen reader too. Most of all, like anyone, I dislike a multi-syllabic phrase where a short word will do.
Or is it something you never gave much thought, because you picked up the phrase from somewhere else?
To be clear, I'm not offended or anything like that. Just curious, and sharing my perspective as one of the people you were referring to.
Finally, Thanks for raising the accessibility concern in the first place.
- "partially sighted users with screen-readers": For example, the final interface will expose a set of controls to set attributes that help screen readers do their jobs. The HTML structure is pretty much determined by the order of elements inside the content panel.
- "keyboard controls. E.g. can I find / highlight / copy text?": The canvas piggybacks from the generated HTML. So for example if the user presses the tab key to alter the input focus in the page, with a listener the changes can be reflected on the canvas depending on which element is focused. DOM elements are linked to the entities being rendered in the canvas. Text picking with the mouse would be handled by queying the scene graph and mirroring how text gets highlighted in the browser. Copy and pasting would be handled by the Clipboard API.
- "scrolling, especially on resource constrained browsers. Do you have a massive framebuffer? what about long pages? animation and region invalidation?": While the amount of elements in the page will correlate directly with the amount of memory that page is consuming. We have to keep in mind that most (if not all) of the things that are rendered are being treated as quads. This isnt a 3D mesh. Infinite scrolling is a feature I do have in mind implementing, but I don't have an specific answer for that yet unfortunately. Region invalidation (I'll assume you mean with space partioning) would be handle with the scene graph. Could you tell me more specifically what you want to know about animation?
- "script blockers": If the user is blocking scripts in the browser then all he will get to see is plain HTML unfortunately. AFAIK this is something that cannnot be stopped.
Thanks for the feedback! If you have more questions keep asking them.
If there's html behind the canvas that elements on the canvas are mapped to, why render everything in the canvas at all instead of just using html/css?
Lets use two applications that are out on the market as examples.
Webflow: Produces good HTML and CSS but you still as the designer are expected to deal with ID's and classes directly. You also need to at least have a rudimentary understanding of css styling and the box model, designers are allergic to that.
Webydo: All design, but try to use the html inspector and check the code it produces. It is bloated. Check out one of the websites in their showcase, http://muccatypo.com/
One of the advantages using the canvas this way is that it allows the page to be rendered only through the use of numerical data saving bandwidth when delivering a webpage.
WF founder chiming in here ... the beauty of giving designers the ability to create CSS classes, build with the box model, and think about HTML/CSS concepts is so they can leverage the power of the browser to help them design a production ready, responsive website. There may be designers that are allergic, but there are also many that embrace it.
Thanks for the answers. This raises even more questions!
It sounds like you're allowing a hidden DOM to be rendered by the browser and then you're going to render it again in a canvas element? Is the DOM "live" in the browser, so to speak? I am very interested in seeing more details!
The reason I was asking about scrolling / invalidation was just to try and guess how you're doing it. Like, if you are rendering all the text in a canvas and you have a hover or click state that requires a change to what's on screen, are you re-rendering the whole canvas or working out the minimum clipping rectangle or what.
Broadly, my concern is this: When people re-implement stuff that already exists in the browser or operating system, they miss things. Or they make it work the same in every browser, when browsers and operating systems are different, and people choose them for a reason. Off the top of my head:
- If you render text yourself then you don't get the subpixel / hinting behaviour or kerning that's distinctive to the OS (or browser engine). This looks out of place, and you discard a lot of hard work and performance tuning that has gone into making the text look good.
- For example, maybe you hijack the command for "find". Do you hijack ctrl-F and F3 on Windows? cmd-F on Mac? What about Linux? What about "find in page" menu item on my Android phone? On a Mac, cmd-G is "find again". That may be different on Windows.
- I use shift-space to scroll up. Cmd-up to scroll to the top of the page. On Windows that's the 'home' key. Did you think of that?
- If I double-click I can use shift-left and right to select characters. Shift-alt-right to select words. Shift-page-up to select to the top of the page. Triple click to select a paragraph. Right-click to read a paragraph out-loud with speech synthesizer. Command-click to bring up a context menu. Double-click and drag to select multiple paragraphs.
These are standard Mac OS things, nothing special.
I am not asking "have you thought of these?", I am pointing out that they are different from platform to platform, and exist for a reason, and you will find yourself reimplementing a platform-in-a-platform. And it's a constant race to copy and maintain compatibility, when the purpose of an OS and browser platform is to allow the creator to focus on building on top of those abstractions, not to re-implement them.
Or you will be removing functionality from end-users and making the sites worse to use.
I hope you don't find this too negative. These are big issues, and your project touches on them.
1) The DOM is hidden and never rendered, plain HTML will only be rendered if the user does not allow Javascript to execute.
2) Yeah the scene graph would tell the renderer to only render the bare minimum with a clipping rectangle. Re-rendering the whole canvas is only done when some sort of "global" change like scrolling occurs.
3) I agree with you completely on this one. There are ways to do it for sure but they require some tinkering. Like I said in another reply, one of the main goals is to keep everything consistent. And actually, with the newer browsers the text rendering has actually been really good canvas.
4) For the set of questions with hotkeys, I do not intend to remove any functionality from end user. I never really thought about these things so thanks a lot, now I can keep them in mind. I cannot give you a specific answer as of right now, but I do know there are ways of capturing text selection and keeping that selection independent from how the user tries to highlight text.
I don't find any of this negative, on the contrary I am thankful for the feedback.
I agree with you in principle. But, needless to say, most developers don't design for accessibility above all else. We who care about accessibility need to meet developers and designers where they are, helping them incorporate accessibility into the way they're already building the UI or content, rather than tell them they need to do it a completely different way.
Haha yup. The interface you are seeing now is not the final one, you can be sure that in the future the application is going to be using wasm. Just like the guys at Figma are doing.
Does this mean the entire UI is drawn using the canvas element, which then outputs a standard HTML/CSS webpage?