
Opening up cyberspace with Matrix and WebVR - vanburen
https://matrix.org/blog/2017/04/04/opening-up-cyberspace-with-matrix-and-webvr/
======
avaer
I'm huge on VR -- especially WebVR -- but this isn't what comes to mind when I
think of the metaverse opuses in the post. Telepresent chat rooms were never
an important part of these stories. That's not where the magic is.

My take is that the metaverse is simply a software system so deep that it
becomes part of everyday reality. And we've already built this, but our human-
computer interface technology (mostly the software) sucks too much for us to
see it. In the meantime we call it the web.

This looks like really cool tech, but I don't think the path to the matrix
involves a new protocol, or a new data layer. I think it will come about by
extruding our pre-existing software epic -- the web -- into our virtual and
augmented environments, in the same way the iPhone compressed the software and
ideas that already existed into our pockets.

~~~
Arathorn
The problem is that the web itself just doesn't have a standard realtime
component. (other than Matrix, or possibly XMPP or Solid). Sure you can
provide better HCI technology to expose the websites and webservices of today
to users via VR/AR... but they end up just being disconnected silos; the
equivalent of different tabs in a browser.

The thesis here is that you need some kind of fabric to weave these apps
together if you want them to feel remotely immersive in AR/VR. Rather than
each app being its own island, surely you want the ability to move your avatar
between them, and collaborate and communicate between services and apps, and
have some kind of common virtual physical metaphor that you can literally
build on as a platform - whether that's with bots or services or whatever.
This is the thought experiment we're doing with the VR side of Matrix.

~~~
avaer
Are Websockets and WebRTC not realtime enough? They need to bake more, but
they're literally standards.

I wholeheartedly agree that probably the biggest problem that needs solving is
that of siloed experiences. But this is also something the web solved long ago
with URLs and files on the front, and webby APIs on the back.

If we invent another protocol to solve this problem, are we not simply
creating new silos?

~~~
Arathorn
Websockets and WebRTC really are not enough _at all_ , and this is a very
widely held misapprehension. Websockets (and their HTTP/2 equivalent) are just
a dumb pipe for shifting data - just like a TCP socket, but tunnelled over
HTTP. Meanwhile WebRTC defines how to encode and transport real-time media
streams over the 'net... but deliberately avoids specifying how to discover or
signal the existence of those streams in the first place. In analog terms
WebRTC is like a phone line without a phone exchange.

I agree that the open Web avoids silos through URLs and ad hoc HTTP APIs. But
this is insufficient for a hypothetical VR universe of immersive services: you
don't want the only way for a user to be able to navigate between services to
be to 'click a link'. And it's arguably a failure that almost every single web
service ends up publishing its data via an incompatible ad hoc HTTP APIs - for
instance, given Facebook, Slack, HipChat and Skype all provide supersets of
precisely the same basic functionality, how come each has its own proprietary
custom HTTP API to do the same thing? Wouldn't it be better if there was a
standard open data fabric into which they exposed their data?

The idea here is that VR will suck if it's just a bunch of disconnected apps,
and instead you need decentralised primitives for sharing Identity, Avatar
properties (e.g. position & orientation), shared VoIP, shared Video, even
shared world geometry/assets/physics in order to stitch it together into a
coherent experience; both for users or developers.

The web was successful because HTML is so simple, forgiving and flexible - and
you end up with essentially the minimum successful viable hypertext system.
One could probably limp along with a VR Web which is just a bunch of webapps
which happen to have chunks of WebVR UI. But it could be _so much more_ ,
while still being flexible and forgiving and lightweight. At least this is
Matrix's hope :)

~~~
sidcypher
Apologies for nitpicking, but I find it rather important: Websockets are not
tunneled over HTTP, they are only negotiated over HTTP. Once you are
connected, there is no overhead of sending HTTP headers back and forth.

------
laumars
I want to find this exciting but I can't help feel a little jaded when I hear
Matrix's PR spin this as "the missing link" when we've already have countless
attempts at creating 3D worlds like this but which all have since faded into
obscurity. Heck, I even wrote my own "3D website" in the 90s using a long
defunct web markup language called VRML.

It will be interesting if/when we finally get there but I don't expect this
will be the breakthrough product. Though personally I think the real evolution
of this tech will be more along the lines of augmenting reality with web tech
rather than emulating virtual reality to behave like our physical world.

~~~
throwanem
Everyone's VR PR posits that they're unique and precious and indispensable.
Unicorns in goggles.

~~~
Arathorn
I'm not sure this is entirely fair, given the blog post clearly tries to
mention the other folks who are working in this space. Specifically for the
"missing communications/data layer" for VR, the only other options I
personally know of are High Fidelity (decentralised, but not yet trying for an
open standard or to use the open web), OpenCroquet/Cobalt (seems fairly dead
since 2010), OpenSimulator (not decentralised, afaik), and there have been
some talk of JanusVR + IPFS hybrids.

Empirically, there certainly is a missing layer here, and it doesn't seem
unreasonable to propose Matrix as a possible solution :)

In terms of "this will happen organically, rather than as a whole new virtual
universe appearing out of nowhere"... we see AR and VR as entirely the same
problem from a comms/data interoperability perspective. Both of them need
somewhere to store the data. The choice is simply whether you store it in a
decentralised DB like Matrix, IPFS, a blockchain(!) or whatever... or if you
store it in a silo like a typical VRML or WebVR app, or something like
Google/Facebook.

I don't blame everyone for being cynical though - this is the holy grail after
all :D

------
eropple
WebVR concerns me because of the garbage collector and the stop-the-world
behaviors you see in browsers. VR is _touchy_. If you drop frames, you don't
notice it just with your eyes, you notice it with your stomach. It feels
really, really bad. Getting adequate perf out of a browser for this stuff
seems really implausible.

~~~
Arathorn
that's what we thought, but you get a solid 60fps on a typical laptop in a 4
way webrtc video conference, or when showing 4K 360 degree video (plus a video
call). My Samsung S5 manages about 30fps in Cardboard mode, which is adequate
for Cardboard, and on a proper gaming rig it works flawlessly with a Vive or
Rift. Seriously: try the demo at
[https://matrix.org/vrdemo](https://matrix.org/vrdemo) in browser and see!

The only perf problems we've had are when using VP8 or VP9 for WebRTC video,
whereas H.264 (which the demo prefers) correctly decodes efficiently in
hardware... and on Firefox where unfortunately
[https://bugzilla.mozilla.org/show_bug.cgi?id=1246410](https://bugzilla.mozilla.org/show_bug.cgi?id=1246410)
slows down the 360 video significantly (to around 20fps for 4K video on the
gaming rig) relative to Chromium which manages a solid 30fps (which is the
source framerate).

The only motion sickness we've seen has been when editing the skiing videos -
the worst thing in the world is if the cameraman wobbles the camera whilst
getting up from the floor but your head doesn't move...

~~~
eropple
Video is one thing--you can offload a lot more of that out of JS-land, yeah?
What worries the hell out of me is the rest of WebVR. I mean--we see frame
drop in _Canvas_ on midrange PCs that's really hard to erase (sincerely, My
Last Two Days Of Free Time). The prospect of that sort of mess in VR has me
worried. I have a strong stomach so I may play with it once my replacement
Vive gets here, but...man. This stuff is so razor-thin on the margins that it
really worries me and will continue to so long as we're stuck with stop-the-
world GC (and to be clear, _of course_ you can write GC-friendly code...but
it's a lot harder in JavaScript, IME, than in Java) anywhere in this stack.

~~~
ngokevin
I'd expect 2D canvas to have worse performance than WebGL. I don't think
that's a fair comparison for WebVR.

Try out [https://aframe.io/a-painter/](https://aframe.io/a-painter/) when you
get your Vive in Firefox Nightly and try out the performance. It's native like
for me.

------
Arathorn
[http://www.geektime.com/2017/04/04/matrix-org-brings-
cross-p...](http://www.geektime.com/2017/04/04/matrix-org-brings-cross-
platform-open-comms-to-vr/) provides another take on this, fwiw, highlighting
that the battle for an open VR web has to happen right now if we're to avoid a
closed and fragmented future like we have today with instant messaging.

------
chriswarbo
My immediate thought was Cobalt/Croquet. It's nice to see that they mention
it, but only as "an ambitious project"; then go on to propose their own (part
of) an ambitious project ;)

There are some _really_ compelling things to like about Cobalt; not least the
fact it incorporates a load of existing protocols (XMPP for chat, VNC for
accessing "traditional" applications, etc.).

Unfortunately, its use of Smalltalk (which probably helped it reach so high so
quickly) seems to be holding it back from more contributors and adoption.
There are Javascript ports which have seen some limited usage, but they appear
to be more constrained. Perhaps bootstrapping Squeak on WebAssembly would
help, and maybe offering a layer of JS scriptability on top?

Whilst I wish Matrix the best of luck with their efforts, it would be equally
great to see their protocols/ecosystems integrated into existing systems too
:)

~~~
Arathorn
We (Matrix) have been tracking Cobalt/Croquet on and off since the original
Croquet project. Honestly, looking at
[http://www.opencobalt.net/](http://www.opencobalt.net/), things look pretty
dead currently though - no news updates since 2010? Should we be looking
somewhere else?

To be clear: Matrix tries to be pragmatic above all else, and right now we're
just saying "hey, use Matrix for open comms in VR/AR" (a bit like Croquet
advocated XMPP back in the day). There is nothing stopping one from using
Matrix to add comms into the Croquet metaverse however, and if it's alive and
well I'd really hope that someone would try the experiment :)

~~~
chriswarbo
> Honestly, looking at
> [http://www.opencobalt.net/](http://www.opencobalt.net/), things look pretty
> dead currently though - no news updates since 2010?

I agree it looks pretty dead, but a big bag of old, existing code still does
more than a project in the idea stage :)

> Should we be looking somewhere else?

I'm not involved in OpenCobalt, I'm just someone who's also been following it
for years. One of its major problems seems to be fragmentation: as far as I
can tell, the descendents include:

[https://en.wikipedia.org/wiki/OpenQwaq](https://en.wikipedia.org/wiki/OpenQwaq)

[http://www.3dicc.com](http://www.3dicc.com)

[https://www.krestianstvo.org](https://www.krestianstvo.org)

[https://en.wikipedia.org/wiki/Virtual_world_framework](https://en.wikipedia.org/wiki/Virtual_world_framework)

Maybe there's something salvagable amongst all this?

------
bhouston
I am not sure showing 2D videos in WebVR is actually useful or better than a
standard Google Hangout. I love having a web browser open while in a video
call so I can look up references, share content and other things.

Thus this feels like it is trendy (wohoo VR!), but not actually useful.

~~~
Arathorn
the point here is that if you view 2D video in a 3D environment with a plain
old web browser on a smartphone or tablet etc, it can feel way more immersive
when you hook up the accelerometers so you can look around the scene and tilt
the device to look from person to person and zoom to whoever's talking etc.
Obviously it makes no sense if they have goggles on, but in practice we've
found it a huge win over boring 2D VC. (disclaimer, i worked on this :)

------
verytrivial
Aside: To anyone finding the black-with-orange-stripes style weird/annoying,
I'm guessing it is a reference to [http://community-
sitcom.wikia.com/wiki/Dreamatorium](http://community-
sitcom.wikia.com/wiki/Dreamatorium) .

[Edit] Sorry, re: missed reference reference!

~~~
laumars
It's actually a reference to the Holodeck[1] in Star Trek (Matrix even has the
same arch design as the Holodeck but which doesn't appear in the
Dreamatorium). But you're not too far wrong as the Dreamatorium in Community
was also a reference to Star Trek's Holodeck.

[1]
[https://en.wikipedia.org/wiki/Holodeck](https://en.wikipedia.org/wiki/Holodeck)

------
Animats
At last, a portal site!

Oh, right.

High Fidelity is worth a try. It's like Second Life with higher resolution,
except there's nobody and nothing there.

