
Google Stadia uses Style Transfer ML to change video game art in real time - pavel_tiunov
https://www.turingtribe.com/story/google-stadia-uses-style-transfer-ml-to-change-video-game-art-in-real-time-BGk226TY9zNP3tSpZ
======
lwb
To me this was the least interesting part of the Stadia reveal. State share
was the real innovation imo. Would any serious game studio use style transfer
ml to actually replace their visual artists? Didn't seem like the quality was
there.

That said, it is an awesome PoC. Just not something I see being practically
applicable.

~~~
dragonwriter
> Would any serious game studio use style transfer ml to actually replace
> their visual artists?

No, a serious (AAA) studio would not. OTOH, the platform might intend to
support producers that aren't AAA studios, and whose limited pre-release
resources are focussed on gameplay more than art assets, and those producers
might view it very differently than AAA studios.

~~~
cwyers
You still have to do all the work of building the geometries for the art
assets, are you really saving that much on not having to color/texture those
assets?

~~~
bduerst
Maybe? One of the hacks of adding more assets without having to do much work
is just recycling existing assets by assigning different color
schemes/textures.

~~~
candiodari
That's one thing, plus look at the result. This makes a very lowpoly game
quite impressive visually.

And this is just a trivial graphic filter through a neural network. A lot more
can be done with AI.

------
_bxg1
A neat tech demo, but as with so many things coming out of The Valley these
days, I don't think anybody actually wants this. No actual game is so bland
that you'd want the ability to overlay your own "themes" on it after the fact.
AI has lots of potential for assisting artists, and maybe part of that will
include post-processing effects, but putting such things in the player's hands
is pointless.

------
currymj
this is certainly just a neat gimmick, but it's interesting to me for a
particular reason: the Stadia box uses an AMD GPU, and per the Google
announcement, all the style transfer is done in real time.

this suggests that we may soon have better AMD support in TensorFlow.

~~~
make3
Tensorflow support on AMD is supposedly not awful anymore.
[https://medium.com/tensorflow/amd-rocm-gpu-support-for-
tenso...](https://medium.com/tensorflow/amd-rocm-gpu-support-for-
tensorflow-33c78cc6a6cf)

I wish someone who had >actual< practical experience in the domain would
comment.

It's an important question.

~~~
currymj
I haven't personally tried to do anything on an AMD GPU, but AMD actually has
a very active GitHub repository
[https://github.com/ROCmSoftwarePlatform/tensorflow-
upstream](https://github.com/ROCmSoftwarePlatform/tensorflow-upstream) .

If you go through the issues you can get a sense of what it's like from people
who do have practical experience.

~~~
make3
this is like bad selection bias though. people never write issues to say
things are going well. the general Tensorflow issues are also about only bad
things, by definition

------
roywiggins
Fun, but nobody will ever do it for more than 5 seconds before getting a
headache and turning it off again.

------
eof
Consensus in this thread seems to be that this is a neat gimmick but nothing
else.

Maybe because I have a friend working on a video game as a solo developer, and
putting a /lot/ of energy/money/time into art--I really see the potential in
this (type of thing).

This is not really for the player's benefit to put on whatever mods they want.

This is for fast iteration for artists and creators to automate a huge
(sometimes necessary evil) burden.

~~~
kllrnohj
> Maybe because I have a friend working on a video game as a solo developer,
> and putting a /lot/ of energy/money/time into art--I really see the
> potential in this (type of thing).

For that usage Nvidia's thing was way more interesting:
[https://blogs.nvidia.com/blog/2019/03/18/gaugan-
photorealist...](https://blogs.nvidia.com/blog/2019/03/18/gaugan-
photorealistic-landscapes-nvidia-research/)

That gets you usable texture assets super quickly, which is probably what your
solo developer friend spends the most time on. That or models. Neither of
which Stadia's style transfer helps with. They probably aren't spending much
time tweaking a shader-applied post-process effect, but hey, maybe they are,
and this style transfer ML thing is useful. Very, very unlikely, but maybe.

~~~
mikewhy
> For that usage Nvidia's thing was way more interesting

If you're building a photorealistic game. And there's already lots of tools on
the market to generate these sorts of textures.

I could see this Nvidia thing used for much better looking mockups / concept
art. But it doesn't seem suited for final asset generation.

~~~
kllrnohj
> But it doesn't seem suited for final asset generation.

Well yes, of course, and neither is Google's Style Transfer ML. These are both
just prototyping tools, not a magic wand to make production-quality assets.

------
andreygrehov
It could be that they are throwing it out so that the community try to find a
practical application of it. It is obvious that they understand that what they
are demoing is not the end product so to speak, but just the best way to
represent general idea.

------
adamnemecek
This seems like a solution looking for a problem.

------
kerng
I'm already excited to see what Microsoft will show at E3 with Xcloud
streaming service that they have been working on for a couple of years- lots
of great progress in that area right now.

------
SubiculumCode
I've seen a number of these impressive looking demos..for example:
[https://www.theverge.com/2018/12/3/18121198/ai-generated-
vid...](https://www.theverge.com/2018/12/3/18121198/ai-generated-video-game-
graphics-nvidia-driving-demo-neurips)

Impressive sense of realism in level of detail variety..show the
potential...BUT consistency, sharpness and realism of object boundaries would
not survive close inspection.

------
ohthanks
This is not a consumer product at all. I don't know how useful it will be but
it's designed to be development tool for early art direction visualization.

Even for 2d visualization it could be very useful if you could drop images
into your website mockup files and have it intelligently apply color palette
and texture information to your existing content.

It won't look good, but it might look _good enough_ to save you from having to
manually tweak the design 20-30 times to explore a range of styles.

------
theknight
this can potentially be a "mod" play... where individual gamers can change
game "skin" and stream videos on youtube...

------
summerlight
I don't think there will be heavy users of style transfer among high budget
games anytime soon, but the important part is that it's done in real time.
With an assumption that it has a reasonable level of latency, probably AA,
super resolution or any other neural net based quality improvement can be a
real thing.

------
alkonaut
The useful application for AI in streamed gaming seems like it would be for
predicting the next 30ms to avoid input lag.

This seems gimmicky at best. Prediction has to take place on the _client_
though - so the challenge is making it cheap.

~~~
nabergh
This is really cool to think about. In practice though I would imagine every
incorrect prediction to be incredibly jarring. Additionally my intuition is
that latency is easier to decrease than predicting the next 30ms. If there is
some level of latency that is very difficult to reduce (more so than training
an AI to predict the next 30ms with very high accuracy), I'd be curious to
know why.

~~~
alkonaut
Between e.g. central and west US you'd have a physical ping of ~10ms so 20
round trip just based on the speed of light (i.e. very hard to get rid of).
Meanwhile the input lag you want is one or two frames so on the same order.
But this lag is additive, when the game receives the new state it has to
compute and render the world, the display has its lag before it shows up etc.
So the new frame has to start computing as soon as the input is registered in
order to hit reasonable input lag.

Predicting 2-3 frames should't beyond what we can do at least roughly.
Especially for some important inputs such as turning in first person view it
should basically just start moving the screen. I belive John Carmack tried
some kind of rotational prediction (Not based on AI) for Oculus.

------
b_tterc_p
Silent hill concepts could be fun with this. Otherwise eh. I would like to see
Jojo palette swaps occur to highlight tension but I don’t see that being
implemented with something like this.

------
andjd
Did anyone else see this headline and think: Why would they use a customized
XML for this in 2019?

------
newobj
There is nothing more Googley than building something that's technically
impressive with absolutely no - or even antithetical - market fit

~~~
caprese
Onlive - required other people's infrastructure and a hardware play, and a
software client.

Playstation Now - required Sony to buy/use a cloud service, maintain it, and
expensive Sony hardware just as a force of habit.

Google - negligible instance running on their existing infrastructure,
maintained under existing processes, playable on chromecasts already in hotels
and homes everywhere, alongside browsers and mobile devices.

If Google doesn't shut it down within the 5 years for people to notice this is
even an option, they could focus more on titles and developers more than any
other entrant to this market.

And perhaps the gaming is just the application to get people's attention.

~~~
heavenlyblue
Are you implying Google doesn’t need to maintain an expensive cloud
infrastructure for it as opposed to Playstation?

The only reason they can do that today without specialised hardware a-la
OnLive is because Nvidia and co. now have specialised GPU circuits allowing
one to stream the encoded video output to memory.

When OnLive started they had to do it themselves.

I don’t believe you have any idea what you are speaking about.

~~~
muststopmyths
> Nvidia and co. now have specialised GPU circuits allowing one to stream the
> encoded video output to memory.

Could you clarify what you (and the comments mentioning NVENC below) are
talking about ?

As far as I understand it, NVENC and the ilk are solutions that capture
_video_ output and encode it to H264. So they are x264 encoding accelerators,
nothing more.

If you were running a datacenter this way, you'd be much better of having a
bank of Matrox capture cards (which support multiple simultaneous inputs) in
dedicated hardware converting your video game output to x264 streams for
broadcast. They even support capturing, x264 encoding and streaming to IP
addresses on your network for further distribution.

What would be interesting is if they bypass the video display output phase
completely and render the finished framebuffers to memory (or an encoder chip
via DMA over PCI) instead. I assume this would cut down manufacturing
(+licensing for HDMI ?) costs a bit.

I know AMD has DirectGMA that allows other devices on the PCI bus access to
limited chunks of GPU memory. There are signalling mechanisms in DirectGMA so
that devices can basically implement producer-consumer pairs.

As far as I know this doesn't exist in consumer chips though. You need
"workstation" GPUs. Which might explain Google's particular GPU choice, now
that I think about it.

~~~
heavenlyblue
You:

>> As far as I understand it, NVENC and the ilk are solutions that capture
video output and encode it to H264. So they are x264 encoding accelerators,
nothing more.

Me:

>> allowing one to stream the encoded video output to memory.

You're being pedantic. What I meant is that OnLive actually needed specialised
hardware because they could not provide their service without it (as in: there
was no capability in hardware, without changes to the codebase - to render not
to an external display). You, on the other hand - are speaking about the
conceptualised "perfect" implementation.

Google doesn't need that hardware in order to provide that service because by
now there's a hardware capability that allows you to capture the screen of the
device you are rendering to. They could try to be more efficient by using
other hardware, but that's not a prerequisite for their Gaming Service.

------
andy_ppp
Can't wait for ads to be inserted into games with this...

~~~
fileeditview
A few days ago I commented on the Stadia on HN and basically predicted the
same thing. Does not seem to be a popular opinion however..

~~~
andy_ppp
I do find HN to be against sarcasm of any kind, which in it's own way is
slightly ironic. I enjoyed your comment [1] and consequently upvoted it!

[1]
[https://news.ycombinator.com/item?id=19433773](https://news.ycombinator.com/item?id=19433773)

~~~
babuskov
That was sarcasm? I though he was serious and upvoted.

I can imagine walking down the street in Call of Duty game or something like
Cyberpunk 2077 and seeing ads overlaid on wall textures of buildings.

~~~
penagwin
A while ago PUBG plastered ads for their Berlin Invitational. While they were
ads related to the game, the incrementally added them EVERYWHERE!

First they're on a bus-stop wall, that's thematic okay. Then they're on random
walls - getting annoying. Then the loading screen and the parachutes, nows
it's annoying. Like, guys I'm from the US and I don't want to go to your
stupid competition!

------
smaili
Is anyone else concerned this will be yet another Google product that goes by
the wayside in a few years? This has essentially become my biggest fear now
with any new product they announce/release.

~~~
robotmay
It's pretty much a certainty with this project, in my opinion. Until everybody
somehow has gigabit lines to their home and to their phones, I don't see this
becoming popular. Like most Google projects, their actual success period is
most likely 10+ years away, and the project will be axed long before then.

~~~
chatmasta
5G deployments have already started rolling out. 5G has average latency in the
range of 1-2ms. Google controls a huge swath of the backbone after that. They
can realistically achieve < 5ms latency on 5G devices in the next 1-2 years.

IMO, a more valid argument against its success would be that any consumer who
is in a position to ensure they have fast enough internet for this service, is
also in a position to buy a gaming console. But the counterargument to that is
that there is no reason someone cannot have a gaming console and also use
Stadia.

I'm normally very skeptical of Google's motivations, respectful of their cloud
technology, and doubtful about many of their products. But I think this one
will be a success, and shortly followed by competing products from Amazon and
Microsoft.

~~~
robotmay
Seeing as I live just on the outskirts of a capital city (Cardiff), and I
can't even get 3G signal inside my house, and no more than 80/20 broadband;
I'll remain skeptical about the possibility of success for this outside of
very specific cities. There's also a lot to be said for reliability and the
requirement to always have a fast connection: one of the benefits to many
videogames is the fact that they are not online.

Isn't 5G on a wavelength that has poorer penetration than 4G? It might work in
the USA but signal is a constant issue in stone housing here in the UK (though
it may be better where you are based). I had to get a femtocell from EE in
order to get signal in my house :\

