
Play GTA V in your Browser, Sort of - luastoned
http://phoboslab.org/log/2015/07/play-gta-v-in-your-browser-sort-of
======
moonchrome
I wonder how long before games start being developed for server side
rendering.

Stuff like MMO for eg. you dont care about latency as much, your payment model
already works with it and it solves several major issues :

* no hardware barrier to entry for high end graphics

* no instalation/play anywhere

* optimizations from shared state rendering, potentially advanced rendering techniques (world space lighting) - you can share computation and memory between multiple clients this way

* probably an order of magnitude harder to cheat

~~~
downandout
_> I wonder how long before games start being developed for server side
rendering._

A while ago I was working on a flappy bird clone (as a test bed for the
technology) that kind of did this. The app ran locally on the device, however
it created a text-based record of all objects and their movements on the
screen, and at the conclusion of each game, had the ability to ask you if you
wanted to upload a video of the game you just played to YouTube. If you
selected yes, the small text record of the game was uploaded to the server,
where it was used to create a video of the game as you played it and upload
it.

The idea was that if it were super-easy and used almost no mobile bandwidth to
upload video of game sessions where people liked their results enough to
share, it would go viral. The technology worked well in tests, but then flappy
bird popularity kind of died before I released it. Now I'm working on
implementing it in another game.

~~~
Dragonai
This is very reminiscent of two things for me:

\- the introduction of replays in Halo 3 (which worked the same way - by
saving the data of the entire session, one could freely move the camera around
the entire map and observe any part of the game at any point in time)

\- Super Meat Boy's level-end combined replays (which replayed all of the
user's attempts simultaneously, creating a pretty amusing sort of "heatmap"
effect)

I think this will eventually become standard for games where replays would be
valuable or fun to watch. But I'm not sure about actually rendering live games
server-side until we're at a point where input latency is unnoticeable.

~~~
spartanatreyu
Server side rendering for live games is a really bad idea. If there's a single
hickup with a packet, the whole game will stutter on the client's side and
they won't be able to do anything, not move, not look, not pause, whatever.

Server side rendering for replayed games on the other hand is totally fine,
not everyone has capture cards. They actually had this service on Halo 3 where
you could pay bungie for the number of minutes you wanted footage rendered out
at.

------
rasz_pl
As a nerd I love numbers, 'Latency is minimal' doesnt do it for me. Display
this [http://tft.vanity.dk/inputlag.html](http://tft.vanity.dk/inputlag.html)
or any other stopwatch, and record both screens. What is the bitrate and cpu
overhead at hd/fullhd resolutions 30/60hz?

Some very slow MVA monitors lag as much as full frame behind input, so in
extreme cases you could get equal delay on your laptop.

~~~
isaacremuant
> 'Latency is minimal'

Tell me about it, I've been trying to gather info about a solution where I can
record with a camera, probably 1080p (via ethernet or via hdmi to capture
card) and use the frame information as inputs in a simulation with a latency
around 20 ms and am unable to get hard data on the actual latencies. Nobody
cares about input lag because practically no one records as a real time input.

Incidentally, if anyone knows or has experienced with a similar setup I'd be
forever grateful.

~~~
frik
Gamer care about input lag. Pro gamer on PC don't use wireless input devices,
they prefer cable based ones for exactly that reason. The input lag is also
what (afaik) almost killed Gaikai. Sony recently bought it for some older PS3
titles.
[https://en.wikipedia.org/wiki/Gaikai](https://en.wikipedia.org/wiki/Gaikai)

~~~
scoggs
I never played at the pro level but I did play competitively and I would never
use wireless internet, mice, or keyboards. These days there are some great
mice, keyboards, and headphones but most people I know won't leave it up to
chance and always go with wires.

------
phantarch
This kind of technology is amazingly promising for being able to play games
anywhere on any hardware.

However, the thing that worries me the most is not latency or graphical
fidelity, but the way this will affect the actual design of AAA games. If I as
a developer/studio know that my AAA title is going to be remotely controlled,
possibly accessible only via subscription, and the end user's progression
monitorable at any point, the opportunities for monitization skyrocket. You
think DLC now is bad? Try playing Skyrim and being able to buy in-game gold at
any point via browser pop-up. Just died in Dark Souls? Respawn with no lost
souls and kill all enemies in the vicinity for only $1!

I'm not saying that all games will end up being this way, but a certain amount
of AAA titles may end up going with more mobile-oriented pricing models if
this sort of remote gameplay becomes mainstream.

~~~
TheGrimDerp1
I watched an extra credits video the other day that touched on this thought;

one option is to have AA studios / publishers fund teams of "indie" devs with
license to go nuts exploring new aspects of the medium.

Personally, since we're running full-tilt towards needing something to replace
"jobs," I'd like to hope that, at some point, that some places ( countries,
planets(?)) might institute a universal minimum income, along with a flat-rate
fair tax. Then folks could earn currency by playing games ( or creating art,
music, solving interesting problems, whatevs...

Could go a long way towards building space-ships & eliminating poverty /
starvation...

------
inglor
Possibly stupid question: Isn't this how the Playstation 4 plays Playstation 3
games (in a nutshell)?

~~~
KenanSulayman
Yes.

~~~
anc84
Interesting! Could you elaborate? Where do the frames come from?

~~~
Eric_WVGG
I believe it's a server farm consisting of hundreds of "bladed" PS3s.

~~~
fredsted
How does this work? Are the dvd images uploaded to Sony's servers? Do they
have PS3 datacenters in several locations?

~~~
Eric_WVGG
I have no first-hand knowledge, but it could be done like this…

1\. Take a pile of PS3s, throw out the cases, and attach the motherboards to
some custom-built rack arrays (these are the "blades")

2\. Build one huge SAN that contains all the games, and tweak the PS3 BIOS's
to read from the SAN as one huge internal HD. (BIOS tweak possibly not even
necessary if the SAN was good enough at pretending to be a simple local drive)

3\. Hook up the AV-out and the USB inputs to a series of XEON servers that
simply pump out live h264 streams (and pump controller input back in) from the
PS3 blades. One Xeon ought to support at least a dozen PS3s.

This is how it could be accomplished prior to Sony's acquisition of Gaikai and
OnLive. Since then, they'd have access to Sony's own knowledge of the PS3
platform, and possibly make PS3 VMs that would cut the hardware blades out of
the equation.

OnLive had five datacenters in North America, Gaikai ran on… 300? I imagine
many of them were running virtualized Windows machines, as Gaikai and OnLive
didn't start out as console streamers.
[http://www.engadget.com/2010/03/11/gaikai-will-be-fee-
free-u...](http://www.engadget.com/2010/03/11/gaikai-will-be-fee-free-
utilize-300-data-centers-in-the-us/)

~~~
dwild
The PS3 already support game streaming, you can play from your PSP and PS
Vita, you doesn't need another server for the actual stream.

------
punkrex
This looks awesome, and saves someone the headache of trying to setup the VPN,
but is not using the nvidia GPU's ability to use hardware encoding which is
much faster than going the CPU route in most cases. If someone could adapt
that, and open source it, that'd be the game changer that people are looking
for to build this sort of stuff.

~~~
phoboslab
Afaik nvidia's NVCUVID API only supports H264 encoding, which would require a
JavaScript H264 decoder. So far I haven't been able to get the one JS H264
decoder I know of (Broadway) to work reliably and fast enough for large video
sizes.

I wish the industry would just get their act together and support a common
video codec in browsers along with a JavaScript API that can deal with
decoding single frames. Currently native streaming support for the <video>
element in browsers is extremely poor and proposed solutions like MPEG-DASH or
HLS add around 10 seconds(!) of latency.

~~~
barake
Media Source Extensions[1] may be an option. It allows you to feed bytes to a
video element, sourcing those bytes from wherever you'd like. There is a
demo[2] that literally opens a video and split it in to parts arbitrarily.
Chrome, IE, and Safari all have support for MP4 with MSE.

[1]
[https://en.wikipedia.org/wiki/Media_Source_Extensions](https://en.wikipedia.org/wiki/Media_Source_Extensions)
[2] [http://html5-demos.appspot.com/static/media-
source.html](http://html5-demos.appspot.com/static/media-source.html)

~~~
TD-Linux
MSE is too high latency for this application.

------
vojant
This is really cool, after watching this demo I wonder how the future of
gaming might look like, e.g. we could pay subscription for a game and remote
machine to run game on. This way even with cheap laptop we could play AAA
games remotely. No more issues with hardware, drivers fps.

~~~
jaymzcampbell
This is very much what OnLive where doing a few years back - I had a
subscription for a while but never used it as much to justify it. They would
have some problems from time to time but for a casual gamer like myself that
only had an ancient and linux-only laptop it worked surprisingly well.

They're since defunct, Sony picked up their assets end of April this year
([https://en.wikipedia.org/wiki/OnLive](https://en.wikipedia.org/wiki/OnLive)).

~~~
vojant
I have never using OnLive, did they support playing via browser? Browser
solution is key to this as almost every device support modern browsers capable
of ruining OP solution.

~~~
WorldWideWayne
The browser is the key to what? Doing stuff in the browser?

It's certainly not the key to getting a game on every device. I can't think of
one browser game that people play on their devices. Even Words with Friends,
the simplest 2D game you could think of, is not used from a browser - most
people run the native app for that.

~~~
vojant
>It's certainly not the key to getting a game on every device. I can't think
of one browser game that people play on their devices. Even Words with
Friends, the simplest 2D game you could think of, is not used from a browser -
most people run the native app for that.

Because of performance, but streaming games solve this problem. Full screen
browser wouldn't be any different than native game client.

------
jchendy
Are there any advantages over Guacamole? Possibly higher framerate?

[http://guac-dev.org/](http://guac-dev.org/)

~~~
espadrine
Guacamole is a text protocol. The screen is sent as PNG images in base64. It
was built before websockets were compatibly implemented across browsers.

~~~
jchendy
Any idea what the actual performance impact of that is?

~~~
PhasmaFelis
For fullscreen 60FPS, it's probably going to increase the bandwidth
requirements by an order of magnitude, which will lock out a bunch of users on
less-than-ideal connections.

------
vernie
I'm surprised at the reactions in the comments. Both OnLive and Gaikai were
doing this around 2008 and nobody seemed to give a shit.

------
davewasthere
Well, it kind of makes a lot of sense. If you've got an ultrabook or something
similar. Laptop graphics cards are very underpowered on the whole...

Actually, I could see this being handy while travelling. Sure, at home I've
got a decent gaming rig. But it'd be nice to have decent quality gaming on a
low-end laptop...

I think it's cool

------
mahmoudimus
The latest round of articles encouraging streaming from the cloud to your
local client focusing on latency sensitivity is very interesting because it
might finally make things like high-performance, low-latency remote virtual
desktop for development solutions approachable.

Chrome Remote Desktop works OK and it's a sign of things to come in the field
of virtual desktops. Right now, the state of the art for on-demand cloud
remote development desktops to help facilitate intensive development
environments, like IntelliJ / Visual Studio / Mathematica, on underpowered
clients, (i.e.12" MacBook), is to rely on proprietary protocols that barely
work if the targeted remote machine is a Linux desktop.

Yes, I know about x2go etc, but I've had so-so experiences with it. Compared
to streaming _games_ , I wonder if there's a product in here somewhere.

------
marknutter
So I just bought a new Macbook which although is amazingly thin is also
somewhat underpowered. I also own a beefy Mac Pro (the trash can) and the
thought had crossed my mind that I might use remote desktop to utilize the
power of my Mac Pro from the convenience of my new Macbook. For gaming,
latency is a huge issue, but I could deal with 50ms latency when doing my work
if it meant not having my system lag in other respects. I only wish there was
a OS X equivalent.

~~~
gmurphy
You should try Steam streaming - you can run your full and regular desktop in
it (by using fake passthrough apps), and is how I often use Lightroom and
Solidworks from my Macbook in bed.

Coming from VNC and Remote Desktop, it's remarkable how low the latency is. In
many cases, because I can get a signifcantly higher framerate by remote
rendering, the latency is better than if I was doing stuff locally.

For games, it's worked pretty well with everything you'd be happy playing with
a gamepad (e.g. GTAV, but not CoD).

~~~
marknutter
I will do just that. Thank you!

------
0x0
How much work would be required to adapt this to use WebRTC with native H.264
instead of websockets with a javascript-based MPEG1 decoder implementation?

~~~
cypher543
Probably not much. The WebRTC reference implementation[1] by Google and
Mozilla is fairly easy to build and embed, assuming your Internet connection
can handle a complete checkout of the Chromium repository. Unfortunately it
only supports the VP8 codec for video. There is also OpenWebRTC[2] by
Ericsson, but it doesn't work on Windows.

[1]: [http://www.webrtc.org/native-
code/development](http://www.webrtc.org/native-code/development)

[2]: [http://www.openwebrtc.io/](http://www.openwebrtc.io/)

~~~
dharma1
There's VP9 too -
[https://plus.google.com/+JustinUberti/posts/3FLVapCvDaG](https://plus.google.com/+JustinUberti/posts/3FLVapCvDaG)

------
PhasmaFelis
How likely is it that we can _actually_ bring the bandwidth and latency down
to reasonable levels in reasonable conditions? High-def YouTube videos
sometimes skip or buffer for me, and that's on cable broadband in a medium-
sized American city, pulling from Google-backed servers. How do we make this
good enough that most urban Americans can enjoy it without constant
frustration?

------
fizzbatter
Anyone have any idea how to Stream audio with something like this? Would be
awesome if that same process could stream the audio

------
chadnickbok
Hey so I currently work on this technology - check out
[https://www.x.io/](https://www.x.io/) if you'd like to learn more. You can
sign up for free and get some free minutes to try it out.

We can run arbitrary windows apps and stream them to your web browser,
including videogames.

------
czk
The "vnc" suffix on the name could be a bit confusing to some because it looks
like this project does not use any existing VNC code or libraries using RFB
protocol. This could come in handy for streaming applications to my mobile
devices over LAN if I need to control something easily.

~~~
mahmoudimus
Yeah, that's what I thought too.

------
fla
Nice hack ! WebGL decoding looks pretty effective.

I think the major challenge with remote gaming is input latency.

Even here on a local network, it looks like there is at least ~200ms between
input and frame, wich is a blocker for many game types.

------
onion2k
This is awesome. I can see it being handy for demo'ing things. :)

One thing to note: It gets a bit confused about where your mouse is if you try
to stream one monitor in to a browser on a second monitor.

------
edc117
I'm rather curious if this could be used (or changed to support) two people
playing a local co-op game. Would open up a lot of games in my library for
play with a friend.

~~~
icebraining
It probably could be used as-is. Some people use screen sharing tools like
Teamviewer to do just that, by the way.

~~~
edc117
Interesting, I hadn't heard of Teamviewer as an option for this - will check
it out. Thanks!

------
xfalcox
Hey guys, what about doing this on a chromecast? Since it's a browser app, we
just need to know if chromecast can handle the decoding.

~~~
bshep
You would still need to send input to the app somehow, which possibly could be
done with a phone/tablet which is setup with just the controls and no video.

------
alyx
I highly recommend the Xbox One streaming feature that's coming with Windows
10.

It allows you to play your Xbox games on your PC.

------
Vekz
Sad to see everyone late to the party on this. There was a great company,
Onlive doing this for the past 5 years. Not browser based in their case, but
they were the thought leaders. They recently closed shop and were 'acquired'
by Sony. I believe thats what spurred this recent trend of articles.

------
chowes
Anyone have a sense of how well this setup would do w/ an Oculus Rift?

~~~
philo23
I imagine the network latency would be more than enough to induce motion
sickness as the display would not keep up with your movements.

~~~
cma
Yep, big time. The latest Rift SDK added depth-aware timewarp though, so if
you could stream the depth buffer as well you could get something maybe non-
sickening, but full of artifacts (transparent objects aren't even written to
the depth buffer; when a nearby object is occluding the background and you
peak around it in a warped frame it gets filled in a way that can stand out;
specular highlights are all wrong wrong in warped frames; lossy compression of
the depth buffer for streaming would cause glitching on the silhouettes of
objects).

------
bliti
\- This support one connection, right?

\- How feasible would it be to host multiple client connections (each running
their own instance of the hosted program) ?

------
mickanio
Bravo!

------
thegreatpeter
This is the coolest thing ever

------
citeguised
Clickbait much?

------
bobajeff
I'd like to see GTA III & Vice City (or a good clone) running in the browser,
client side via asm.js or wasm.

Edit: Not that they couldn't do it with GTA V I just prefer those games over
it. Saints Row the Third or Just Cause 2 are also acceptable.

~~~
TazeTSchnitzel
Well, we can emulate an x86 PC: [http://win95.ajf.me](http://win95.ajf.me)

~~~
bobajeff
Actually there so much more you can emulate:

[https://github.com/jsmess/jsmess](https://github.com/jsmess/jsmess)

[https://github.com/dreamlayers/em-dosbox](https://github.com/dreamlayers/em-
dosbox)

[https://bitbucket.org/tsone/em-fceux](https://bitbucket.org/tsone/em-fceux)

and others

[https://github.com/kripken/emscripten/wiki/Porting-
Examples-...](https://github.com/kripken/emscripten/wiki/Porting-Examples-and-
Demos#emulators)

~~~
TazeTSchnitzel
win95.ajf.me _is_ em-dosbox.

------
abluecloud
This is pointless and nothing new. Remote desktop in the browser has never
been an issue, so obviously playing games isn't an issue either.

The pointless part is that he's connecting to a "server" that's next to him,
not in a DC in London. I guess if you _really_ wanted to play GTA V on the
toilet, on your phone, it solves the issue - but really...

~~~
andybak
Nothing is new. It's simply a case of joining the dots.

1\. AWS can run Windows and therefore Steam. Nothing new. 2\. Steam home
streaming works over VPN. Nothing new. 3\. VNC can stream games. Nothing new.

etc.

But put all these pieces together and the first person to wrap it up and make
sure all the legal parts are in place might make a fortune.

~~~
abluecloud
What are you talking about, these pieces have already been put together
multiple times.

[http://www.pcworld.com/article/2359241/how-steam-in-home-
str...](http://www.pcworld.com/article/2359241/how-steam-in-home-streaming-
can-turn-your-old-laptop-or-windows-tablet-into-a-pc-gaming-force.html)

[http://shield.nvidia.co.uk/play-pc-games/](http://shield.nvidia.co.uk/play-
pc-games/)

[http://store.steampowered.com/streaming/](http://store.steampowered.com/streaming/)

This guy isn't running it from AWS though, like I said. He's running it from a
"server" that's right in front of him. By the way, have you ever tried RDCing
a remote Windows machine and done some trivial tasks in the GUI?

~~~
bshep
I actually have an AWS machine that runs Steam + VPN and I can stream games to
my home laptop, it works really well.

I also have non-steam games that I've added like Star Wars: The Old republic,
Starcraft II and Diablo III and they all work well using the same system.

The only glitch I've noticed is that if steam starts before the VPN, I have to
RDC in and restart steam so it picks up the VPN network. (If anyone has any
idea how to fix this I would be very grateful)

~~~
oellegaard
what AWS instance do you use?

~~~
bshep
I'm using one of OTOY/ORBX (ami-8ff2b9e6) instances ( I tried setting up my
own, but never got the graphics to work right )

~~~
bshep
I used the old version of this guide, the new one may work better but I
already have a working instance:

[http://lg.io/2015/07/05/revised-and-much-faster-run-your-
own...](http://lg.io/2015/07/05/revised-and-much-faster-run-your-own-highend-
cloud-gaming-service-on-ec2.html)

