
Show HN: Mixing WebGL and HTML, Both Rendered Out in React - mlsarecmg
http://subdued-health.surge.sh/
======
westoncb
This looks really nice. I did a three.js CAD tool [1] for a client during this
last year and really wanted to use something like this, but ended up building
something custom.

The main thing I knew I wanted was a declarative specification of the objects
in our scene. I knew it would be a nightmare to fiddle around with manually
adding and removing bits of the scenegraph in response to input.

Ended up just doing diffs on the part of the Redux state that mapped to the
scene, and automatically discarding and re-running the mesh construction code
for the parts of the state that changed.

It actually worked well enough and didn't cause any problems I noticed (in
terms of convenience of use and performance), but I wouldn't want to re-write
it for each project that could use it.

One thing I do wonder about with this though is whether there's really a
benefit to using the tag language to specify three.js objects. For folks
already familiar with the three.js library it's going to be a nuisance to use
the tag language instead, and it doesn't appear to me to be more concise or
convenient in any way. But maybe I just feel that way because I've been
writing js to construct three.js objects for years :)

[1]
[https://www.youtube.com/watch?v=e21tqZebl60](https://www.youtube.com/watch?v=e21tqZebl60)

~~~
mlsarecmg
Hey that's amazing, and your tool is really advanced! It's exactly the purpose
of this, we're making a cad system as well and it's been hell with the current
way of how we chose to do it. We're using redux and getting the scene graph to
adapt with manual subscribers - it's led to something that is very hard to
understand. That is why i'm writing the reconciler now.

The benefit of using tags (jsx) is that objects become fully managed. The most
clear indication to me is still effects.
[https://codesandbox.io/embed/mo0xrqrj79](https://codesandbox.io/embed/mo0xrqrj79)
You can drop that in and it works. Make it conditional and once it gets
unmounted the render loop goes back to normal. Almost magic, but then again,
react for the dom had the same benefits.

~~~
westoncb
Your tool looks very cool :)

> The benefit of using tags (jsx) is that objects become fully managed

I'm not sure I follow here. I assume that by 'become fully managed' you mean
that React will reconstruct as necessary based on what is declared in the jsx?

So maybe that is just the nature of writing React renderers specifically, the
only option for getting your objects managed is to map them to jsx?

> The most clear indication to me is still effects.

This does look much nicer than other examples I've seen of using effects. I
have only limited experience with them personally. They weren't needed for our
tool. Incidentally I am probably setting up some SSAO on another project today
though so I'll be likely be digging more into them later.

~~~
mlsarecmg
with managed i mean that the view is now merely a reflection of state.
Basically just like with the dom:

    
    
        {condition && <div>hello</div>}
    

The div is there as long as the condition is true. But of course it can now
also be a mesh, or a component that can clean itself up after the unmount.

We did this manually before, which usually meant watching out for objects to
remove themselves, they'll have to communicate up their hierarchy to inform
parents and so on. This wasn't straight forward in three, so there was a lot
of prototype hacking and terrible things like that, i'm so glad this will be
all gone.

------
rl3
This looks really cool. Although I'm not very familiar with React, I can
provide some insight into the history of mixing GL/HTML content for anyone
that's interested:

Three.js has had something called the CSS3DRenderer for like 5+ years now. It
uses CSS 3D transformation effects to proportionally scale/distort HTML
content to what the Three.js viewport is doing. It does most of the heavy
lifting for you in terms of matrix math, but you still need to write a lot of
glue code if you want a unified feel. I worked with it a lot in 2014, and it
felt like alien technology at the time. Ultimately I settled on purely
immediate mode though, because you can't apply custom post-process effects to
DOM content beyond whatever CSS effects offer.

Famo.us as it originally existed launched in 2015 and was all about mixing
WebGL/HTML content, but they pivoted shortly after launch. Some interesting
developments came out of that, however:

Their VP of Product, Zach Brown, ended up founding Haiku in 2016, and their
first product Animator[0] just officially launched for Mac. I view it as a
more capable replacement for Adobe Edge, which was a fantastic little tool for
animating HTML content. How flexible Animator actually is in terms of mixing
GL/HTML content, I'm not sure. It'd probably need to take the modern Adobe
Flash approach and decouple the tool output from the underlying client runtime
to succeed at that.

Following the famo.us pivot, community members founded a project called
Infamous with the goal of keeping the mixed content dream alive. While most
people fell away, one person (Joe Pea) did stay the course and has spent the
past few years since then developing Infamous[1], which is basically an entire
framework built off the premise of mixing GL and HTML content.

I'm not sure if it's ready for prime time yet, but it sure looks good and I'm
extremely impressed with Joe's perseverance. Just looking at the commit
history messages, you can kind of get a feel for all the little issues that
mixing GL and HTML content creates.

[0] [https://www.haikuforteams.com/](https://www.haikuforteams.com/)

[1] [https://github.com/trusktr/infamous](https://github.com/trusktr/infamous)

~~~
FarhadG
I was one of the core contributors to Famo.us' WebGL engine. Feel free to ask
any questions about "mixed mode" (i.e. WebGL interlaced with DOM)

------
mlsarecmg
PS. Full code for the demo:
[https://codesandbox.io/embed/ly0oxkp899](https://codesandbox.io/embed/ly0oxkp899)

------
omeid2
How do you read this thing? All I see is "React Three Fiber".

It put my laptop on Takeoff mode (fans full speed!) with Firefox.

~~~
outworlder
> It put my laptop on Takeoff mode (fan's full speed!) with Firefox. This is
> why all this movement to rebuilding a browser in browser is ineffective.

Non sequitur.

Your computer is doing more processing. There's WebGL, which will engage your
graphics card. If it is on your CPU, the CPU will consume more power.
Depending on your laptop, you could have a discrete graphics card, which will
be switched to and consume more power. There are animations, etc.

It is not only because of React. In fact, by avoiding unnecessary DOM changes,
it's probably an advantage (although this would have to be measured).

Unless you are telling us that you were playing fortnite or something and your
fans didn't spin up, but this thing caused them to spin up.

~~~
omeid2
You missed the point that it doesn't work at all, so it is obviously broken in
someway in Firefox.

I run multiple VMs and do 3d work (in fact, mostly direct opengl stuff) on my
laptop and it hardly ever goes to takeoff mode with just some simple
rendering.

~~~
nailer
It's broken in Chrome too - there's no content asides from 'Reach three fibre'
and the links. I would have expected at least a description to exist on the
page.

~~~
mlsarecmg
I ran it both in Chrome and Firefox, seems to be working. Could i bother you
for error logs?

~~~
shakna
Well, it appears broken for me... But I don't have any errors in the logs.

Firefox 65.0.1 (64bit), Linux (5.0.0).

There are a couple warnings about a particular extension:

> warning: extension `GL_ARB_gpu_shader5' unsupported in fragment shader

The rest just appears to be successful parsing messages.

(But it worked really well in Konqueror & Chromium, so that's something.)

~~~
mlsarecmg
I googled that, looks like an issue in FF:
[https://github.com/mrdoob/three.js/issues/9716](https://github.com/mrdoob/three.js/issues/9716)
Basically i just channel through regular three without creating much of a
dependency. If it doesn't work in react-three-fiber, then i doesn't work in
threee-js. But they're so big that usually these webGL bugs are either fixed
quickly by them, if it's their fault, or they're filling vendor issues.

~~~
shakna
three.js have marked it as a wontfix and referred it onto Firefox. It's
sitting in Bugzilla as a 6-month+ old bug.

Safari and Chrome both had to work around the same thing, so it seems like
there's some contention around who is to blame here.

Either way, three.js is (partially) broken for Firefox users.

------
gilbetron
This is really cool. When I learned React in 2016, it was fairly easy for me
to pick up the basic ideas because it reminded me of common patterns from my
years as an graphics (OpenGL) developer. I also love using WebGL, as I spent a
lot of time honing my OpenGL skills, so making apps in WebGL is second nature.
I looked for a package like this back then. Great job!

~~~
mlsarecmg
I do a lot of 3d stuff at my job, first openGL, then three, but it's too easy
to make a mess with the inflating approach and unmanaged variables everywhere.
I hope this will make a difference.

~~~
gatherhunterer
You could always implement a central store using an implementation of Node
streams. By building a custom store you can allow a React component to access
and adjust only the values it needs to render (hooks make this prettier), then
share the store with your rendering framework. In that scenario React and your
WebGL layer share a central “source of truth” with all of the protection from
concurrency issues that React states provide.

~~~
mlsarecmg
This we had in place, but without react driving the scene we used manual Redux
subscribes. Every change meant figuring out what part of the scene graph
needed to be added/removed or changed. Basically the same problem one would
face with jQuery and the dom.

------
mncharity
Oh, very pretty. Thank you for your work.

I used React for VR/AR, to interweave three.js (objects, assorted cameras and
shaders) with assorted DOM trees (left and right eyes, foreground (pixel
aligned) and background (high-bandwidth - video and such)). This looks
helpful.

~~~
mlsarecmg
That would be one of the things i would love to explore. Does three do this
automatically or is there something the app needs to specifically calculate?

~~~
mncharity
I'm on linux, and thus a custom stack, from tracking to UI. With three.js
providing only bare rendering. It's not a path I'd recommend.

Most current VR/AR exploration uses Unity (Windows) on vendor platforms
(SteamVR (Windows), ARKit, etc) and Vive/Rift/WMR/phone hardware.

Three.js is focused on WebVR. WebVR has been Windows-only. And very not
transparent about status (very "aspirational" reporting). I fuzzily recall
some year-ago VR work in a three.js dev branch. You might also look at
A-Frame[1] and AR.js[2]. But this is not an area I keep track of.

The three.js WebVR folks do aspire to "automatically". Historically, it very
_very_ hasn't been. Add more "very"s.

Now, big picture, I'd be unsurprised by the transition to AR/VR being
shockingly rapid. And unexpectedly soon. So there might now be some path that
"just works". And it could be worth starting to track, so if there's a new
niche you might like, you're aware of it.

But historically... VR, noun: A field covered in hirsute yaks. And the
trampled bodies of people's years, expended shaving them in hope of wonderful
fruit, while it was still unripe.

[1] [https://aframe.io/](https://aframe.io/) [2]
[https://github.com/jeromeetienne/AR.js/blob/master/README.md](https://github.com/jeromeetienne/AR.js/blob/master/README.md)

------
imetatroll
Here is a webgl project I've been working on that is written in gopherjs. You
will have to quickly register to play with it.

[https://imetatroll.com](https://imetatroll.com)

Map Editing

[https://www.youtube.com/watch?v=CE2Apyf-
KOE](https://www.youtube.com/watch?v=CE2Apyf-KOE)

Angled View of the Scene

[https://www.youtube.com/watch?v=ZePX5XPhlgQ](https://www.youtube.com/watch?v=ZePX5XPhlgQ)

------
xrd
Brilliant. Bravo.

~~~
jillesvangurp
Agreed. I just forwarded this to our designer. If this works well enough, we
might end up with a few slick effects on our new react website. We've been
using css transitions for a while and they add some playfulness to the UX. But
this is way more impressive.

~~~
mlsarecmg
Glad to hear that. If you need anything, ping me on github, i'm open to add
functionality.

