
Render HTML via WebGL - PixelsCommander
https://github.com/PixelsCommander/HTML-GL
======
SoapSeller
This project(at least for me) is a little bit misleading. Because it isn't
rendering HTML using WebGL.

It using html2canvas[0] to rasterize the browser rendering into a canvas,
upload it to WebGL, apply effects and patch-back events(i.e. mouse clicks)
back to the hidden browser dom elements. (On the other side, font hinting and
such are not really a problem)

Implementing full UI framework over WebGL has it merits, and hopefully someone
will make a useful(and not html+css based) one some day.

[0]
[https://github.com/niklasvh/html2canvas](https://github.com/niklasvh/html2canvas)

~~~
Polarity
u know [http://famo.us](http://famo.us) ?

~~~
kowdermeister
Famous is CSS based, not WebGL

~~~
liviu
Actually it has WebGL renderer.

 _Famo.us is the only JavaScript framework that includes an open source 3D
layout engine fully integrated with a 3D physics animation engine that can
render to DOM, Canvas, or WebGL._

~~~
argonaut
Actually it doesn't. Have you actually used famo.us? You do realize you're
just quoting the website copy. Famo.us doesn't have WebGL rendering yet.

~~~
jiyinyiyong
They've shown demos of WebGL renderer. It's called "Mixed mode" as I can
remember. But it's probably not released yet.

------
phn
Make it feature complete and you'll get the same speed the browser has.

If we're going to build a new rendering engine for the web, maybe we could try
to create something nicer than HTML and CSS, whose only advantage at the
present is the number of developers familiar with it. I know I wouldn't miss
them.

~~~
emodendroket
What is there that's so much better?

~~~
aout
Because you could do something easier to understand in terms of 3D for a very
simple use case and find a generic use for it.

~~~
aout
Your browser is slow when applying transform to DOM elements because it has to
translate everything 3D in terms of JavaScript and DOM. If you want to
manipulate images and elements effectively in 3D then you have to learn matrix
computations, quaternions and stuff which is horrible for the common web
developer. I'm saying that if you offer a new approach that could mix HTML/CSS
and encapsulate those complex 3D concepts it would be interesting.

------
aout
Wait wait wait... Handling textures via CG Buffers is quite fast you are right
and you can basically do whatever you want in terms of image manipulation
using the shaders but you'll lose efficiency as soon as you introduce complex
picking (text selection for example) and / or computations (aliasing etc...)
between JS and your pseudo-DOM.

In the end the only difference you'll have compared to the real browser
renderer is that you do less computations from CG(GL) to RAM (JS), hence the
performance.

You can check this very old project of mine where we tried to link DOM and
WebGL Nodes: [https://github.com/aout/SAGE](https://github.com/aout/SAGE)

edit: just saw the repo wasn't entirely up to date. Done now.

~~~
PixelsCommander
Nice, we may use it as a renderer for HTML GL. Shaders and fast rendering
wirth it I believe. Try the demo in the bottom
[http://htmlgl.com/](http://htmlgl.com/)

~~~
aout
I tried the demo of course and I thought you did a great job. I'm not saying
you should abandon the project because of performance issues you'll encounter
but rather focus on simplicity with clear boundaries. Again, go for it! (but
don't go too far)

~~~
PixelsCommander
Thank you. Agree, simplicity is what it is created for, will keep it that way.

~~~
aout
Just updated the repo with multipass rendering including picking and lights.
Might interest you.

------
pierrec
Very cool. This could be useful for some applications, but I don't see it
becoming a thing for general use cases unless it's implemented and exposed
directly by the browsers.

Even then, the main problem is that the internet is still mostly text, and if
you can't implement or access font hinting, your text will be blurry and ugly
on any display that isn't ultra-high DPI. You can't really render text to a
texture and _then_ manipulate that texture with sub-pixel precision. It must
be done the other way around: text rendering must be done last, and with the
help of a hinting library so it will look sharp and readable.

~~~
PixelsCommander
There is an intent to make rasterization native
[http://pixelscommander.com/en/javascript/state-of-html-
conte...](http://pixelscommander.com/en/javascript/state-of-html-content-
rasterization-draw-html-to-canvas-image/)

------
methyl
So we have browser in the browser, we have to go deeper...

~~~
PixelsCommander
It is basicaly not. Just converting HTML to texture with some tricky proxying.
The reason is to be able to do that
[http://pixelscommander.com/polygon/htmlgl/demo/filters.html](http://pixelscommander.com/polygon/htmlgl/demo/filters.html)

~~~
MrDosu
What he means is that is exactly what the browser does already. It's just
slower because it handles more of the standard.

It rasterizes the DOM.

~~~
PixelsCommander
Browser rasterizes based on it`s assumptions and you cant control it
completely which is a bottleneck for performance optimization. Also how would
you apply displacement filter on that content?

------
moron4hire
This creates an interesting change in how applications are deployed to users.
Instead of the user choosing the HTML rendering engine, the developer now gets
to choose it. Instead of having to design sites with multiple rendering
engines in mind, this makes it possible to manage only one. And it's not like
the user cares that content is being rendered by Firefox or Chrome or
whatever, the user just cares to see the page, leaving the selection of the
actual rendering engine only of concern to the developer.

I think that's important enough to warrant going through the paces and
implementing this. Don't listen to the naysayers. Reinvent wheels. The wheels
we currently have largely suck.

~~~
perdunov
C++ with asm.js + WebGL effectively bring back desktop programming to the web.

~~~
moron4hire
That is so not even close to a good way to build applications. I mean, I
really can't think of anything more painful right now. Other than, maybe being
asked to make a modern web application on classic ASP.

------
slaction
"I really wish I could use bump mapping on this div" \- No one ever

------
bsenftner
Really great work. I've had in the back of my mind that this was possible, but
now you've gone and done it, and quite well!

I'm curious if there would be any conflict with usage of Three.js on the same
site? Could they end up clobbering each other's textures? (I've not gotten
into webgl's texture memory management, so that question is a stab in the
dark.)

~~~
PixelsCommander
I would not expect any conflicts there since they are different canvases and
textures uploaded just as files. You may ask mat@goodboydigital.com for more
details.

------
danbruc
This was at least kind of possible (not sure if it worked only for images or
also other elements) with Internet Explorer 4 [1] in 1997, almost two decades
ago. It was obviously non-standard.

[1] [https://msdn.microsoft.com/en-
us/library/ms532849.aspx](https://msdn.microsoft.com/en-
us/library/ms532849.aspx)

~~~
cordite
Although non-standard, Microsoft sure did several novel things back in the day
here. Then they decided not to do the actual standards correctly and held a
lot of things back since nobody updated these things.

On the topic of Microsoft and the web, ActiveX gets a lot of bad rep (and for
good reason), though it seems like they were trying something akin to web
components back then. It worked better in actual applications than the web
though.

------
mandeepj
Then all the hard work that flipboard did to achieve 60 fps rendering speed
was not required?

Reference -
[https://news.ycombinator.com/item?id=9029159](https://news.ycombinator.com/item?id=9029159)

------
bsenftner
Would forms and input fields still work? Could one push their wysiwyg rich
text editor into this? (I'm thinking game / VR content where an in-3D-world
web browser on some in-3D-world device screen is fully operational.)

~~~
PixelsCommander
It depends on how far it will go. At the moment there is a plan how to do
that, but not enough time to implement... You may use it for editor, but keep
in mind that text itself would be sent to GPU every time it updates. Anyhow
you may ping me in skype lp_funky if interested in doing this, would be glad
to help.

------
cturhan
What if we merge it with React and use its virtual Dom management?

~~~
PixelsCommander
I think could be nice. There is an intention to create React-GL, but nothing
serious yet. All you basically need is to call something like new
GLElement(this.getDOMNode()). Or how is it in 0.13?...

~~~
cturhan
Actually there is an implementation called react-canvas does the same thing on
canvas. Flipboard uses it on mobile website and it has a great performance
even for mobile.

------
josephmx
Very minor bug on the demo: The alert on click reads "clicked X" (eg, "Clicked
Avatar") on all of the titles except for Quantum of Solace

------
heldrida
Holy cow! This is a great idea : D the responsiveness is a bit sloppy but it's
really good you guys have it working! haha really cool : D

~~~
PixelsCommander
Thanks! We are working on improving responsiveness a bit. Will be better.

------
kolodny
fiddle: [http://jsfiddle.net/409gxzm6/](http://jsfiddle.net/409gxzm6/)

~~~
bshimmin
The text is blurred on my retina MacBook Pro. (And not selectable, as others
have mentioned.)

~~~
PixelsCommander
Yes, there is an issue on that. [https://github.com/PixelsCommander/HTML-
GL/issues/11](https://github.com/PixelsCommander/HTML-GL/issues/11) I d like
to fix it before going public but HTML5WEEKLY made announcement without
notice, so everybody sees know that bug =( As for selection - it is selectable
in it`s bounding box, but selection is not highlighted yet.

------
jeremiep
It looks great, but I don't like the fact that I can't select the text
anymore!

~~~
PixelsCommander
It can be selected at same bounding box DOM node have and the selection will
not be highlighted at the moment, but we will fix that.

~~~
riskable
If you can get text selection (and copy/paste) working please post how you did
it. I wrote Gate One which is a web-based terminal emulator. Right now I have
to go through great pains to ensure that text being rendered to the screen
happens in a performant way while still allowing for copy & paste
functionality. It also has some features that will highlight text that you
select with your mouse and other text formatting stuff.

I've experimented with canvas in the past to see if it could improve the
performance of, say, a terminal that updates very quickly but it ultimately
ended up being just as slow as the regular DOM because of all the extra code
that had to be running in order to keep the hidden buffer in sync with what
was displayed to the user.

I'd be curious to know how the performance of WebGL compares to the
(accelerated) canvas.

~~~
PixelsCommander
Performance is much better if translation / scaling / opacity / rotation are
used or shaders ofcourse. If image content redraws by CPU it sucks. Basically
it is what GPU can do quickly. Selection is possible since elements are there
but just invisible and when webgl textures were animated we can lazily update
DOM positions without affecting performance a lot. Is there a repo for your
terminal?

------
kluck
Recreating existing desktop GUI frameworks one useless HTML/JS library at a
time.

~~~
PixelsCommander
Come on
[http://pixelscommander.com/polygon/htmlgl/demo/filters.html](http://pixelscommander.com/polygon/htmlgl/demo/filters.html)

------
outworlder
Now we need a WebGL demoscene.

~~~
theandrewbailey
There is. It's been around since WebGL was a thing.

[http://www.pouet.net/prodlist.php?platform[]=JavaScript&page...](http://www.pouet.net/prodlist.php?platform\[\]=JavaScript&page=1)

~~~
PixelsCommander
Cool, thx

------
perdunov
As far as I understand, this is roughly how iOS UI works (UIView - CALayer).

Obviously, this can be done at the browser engine level.

I have also tried to do this in my pet UI engine, but found out that it was
tricky to figure out caching rules to avoid memory blowup and performance
degradation for all possible cases.

~~~
PixelsCommander
It is done but there is no control over rasterization and no possibility to
add filters as in the demo
[http://pixelscommander.com/polygon/htmlgl/demo/filters.html](http://pixelscommander.com/polygon/htmlgl/demo/filters.html)

~~~
perdunov
Kudos for the smart solution.

This approach has proven itself very well in iOS, in fact, it was the absolute
best mobile animation technology for a long time. Actually, the older versions
of iOS used _software rasterization_ with hardware accelerated bitmap caching
and it worked smoothly on the original iPhone.

------
designml
Ahh Nice! I was going to start working on something similar to this.

~~~
PixelsCommander
Come on! We can do much more together =) Still so many WOW things to
implement...

~~~
designml
Haha yea, I am going to delve into your code and see if I can contribute. But
nice work! Your site is probably getting hammered now with the traffic but I
am going to come back and check it out.

------
thecatspaw
it looks blurry. does it support retina displays?

------
curiously
this idea is so crazy it just might work but need to see some more examples.

one might be a crazy heavy SPA or javascript webpage that lags and show it
through HTMLGL.

this really could go somewhere, the idea is solid but it might take some work
to actually deliver, a constant fps web.

~~~
PixelsCommander
Yes, the only bottleneck here is rasterization which we are trying to solve by
making native Rasterization API live. You can read about here
[http://pixelscommander.com/en/javascript/state-of-html-
conte...](http://pixelscommander.com/en/javascript/state-of-html-content-
rasterization-draw-html-to-canvas-image/)

