
Integrating GTA V into Universe - gdb
https://openai.com/blog/GTA-V-plus-Universe/
======
squeaky-clean
I love GTAV so much, it's the most real single player world I've ever
experienced. I once rear-ended an NPC because they suddenly stopped in the
middle of the street. I got out of my car (because it's GTA and you have to
shoot everyone, duh) I saw the NPC had stopped because he hit a cat with his
car!

I'd be lying if I said OpenAI Universe / GTAV wasn't one of the primary
drivers in me trying to learn ML.

Anyone know what kind of hardware I will need to play with this? The AMI page
on github mentions an AWS g2.2xlarge (iirc a GTX 1060 is slightly better than
this?). And it seems like GTA is actually running on a different system you
use VNC to share the screen/input? Any estimates about whether a 1060 or 1080
on a beefy gaming PC could handle both at once?

~~~
dTal
Hmpth. Conversely, while playing GTAV in completely innocent explore-the-world
mode, I accidentally bumped a police boat with my jetski. Apparently this
warranted being shot in the head.

Most reviews I've read actually complain that GTAV is a step back in realism
from GTA4 (for example, body-part specific injuries were dropped), but let's
not beat about the bush - no GTA deserves the title of "realistic", because as
you say "it's GTA and you have to shoot everyone, duh". They're made to _feel_
realistic, but in fact they powerfully reward violence and punish being law-
abiding, in precisely the opposite way that the real world does.

~~~
toomanybeersies
I enjoyed GTA V because it was so hilariously over the top.

It didn't try to be realistic. The whole point is that the police response is
always massively disproportionate to the crime.

It's a pastiche of society, everything is exaggerated: they have a legalize
medicinal cocaine campaign, you can buy supercars online, rocket launchers and
machine guns at your local gun shop, you can launch your car at 100 miles an
hour of the edge of the highway and survive.

They even reference the outrageousness of it in a scene where Michael goes to
the therapist and comments that he may have even killed someone on the way
there.

~~~
Houshalter
I think it tried to be super realistic in many ways and deserves some credit
for it. Its like they made a realistic world simulation, and then tweaked a
few parameters so it would be funner to play as a game.

Like obviously no real person would be able to survive the injuries my
character gets every 10 seconds. But if I was dying all the time it wouldn't
be fun to play.

The one thing that really bothers me, as far as realism goes, is lack of
persistence. Leave a car somewhere, walk 10 feet away, turn around, and it
vanishes. Cars and NPCs vanish when you aren't looking. Damage you do gets
quickly reset as if it never happens. And a number of small details like this.
It's not just that it's unrealistic, it really limits what you can do with an
otherwise awesome sandbox.

Still, I think the game that deserves most credit for realism is dwarf
fortress. It will be a long long time before AIs master that.

~~~
wldcordeiro
One of the limitations of GTA that I've found most frustrating is that the
game is seemingly programmed to prefer red lights in the streets, I think this
is to force you to slow down a bit and let the loading occur more seamlessly,
but it's still annoying to just run into traffic jam after traffic jam.

~~~
restlessmedia
You adhere to the traffic regulations in GTA? :)

~~~
smcl
No but the NPC cars do, creating little hard-to-pass blobs of traffic

------
ChuckMcM
Ok, I am _so_ glad that we're not training AI to "win" at GTA V. That is not
an android I want to meet on the street :-)

The transitive property of teaching an AI to drive given a world simulation,
makes me wonder if you can train a network to recognize speech by feeding War
and Peace through a text to speech program and then sending the audio data
generated into an RNN. Where is the ground truth in such a system which you
can calibrate your results by?

~~~
semi-extrinsic
> Where is the ground truth in such a system which you can calibrate your
> results by?

Answering standard reading comprehension test questions? - "Why did Pierre
meet Natasha at the church?"

------
empath75
I'm not sure I would like to live in a world with autonomous vehicles that
were trained in Grand Theft Auto.

~~~
rawnlq
I don't think anyone is thinking of training on GTA then reusing it on a real
car. It's just a convenient way to work out the algorithms and architecture in
a controlled way that might then work out with the real world data.

But I do think GTA's unrealistic physics engine[1] will hurt it for simulating
the cases we care most about (accidents, lost of traction, etc)

[1] GTA is a regular submission on /r/GamePhysics which collects funny game
physics glitches:
[https://www.reddit.com/r/GamePhysics/search?q=GTA&sort=top&r...](https://www.reddit.com/r/GamePhysics/search?q=GTA&sort=top&restrict_sr=on&t=all)
[https://www.reddit.com/r/GamePhysics/search?q=GTAV&sort=top&...](https://www.reddit.com/r/GamePhysics/search?q=GTAV&sort=top&restrict_sr=on&t=all)
[https://www.youtube.com/channel/UCy332GE6lcMVP612IUs7ODw/vid...](https://www.youtube.com/channel/UCy332GE6lcMVP612IUs7ODw/videos?sort=p&view=0&flow=grid)

~~~
justifier
exactly, i saw a similar complaint as to the gp's in a thread about working
with classical computing to simulate quantum computing

my argument then was 'imagine prototyping on a machine that needs to be kept
at 17millikelvin'

here the argument is 'imagine prototyping on a two ton missile traveling at
lethal speeds through highly populated areas'

------
delegate
This is cool.

Next step is obviously teaching it how to kill people in the game and get away
from police.

I'd like to see the AI go on a rampage and win :).

That should be enough to wipe off a lot of smiles off our idealistic faces :)

~~~
smrtinsert
It's all fun and games until the Second Amendment is used to protect the
ownership of armed drones that can be used to assassinate anyone anonymously
simply by scratching the serial numbers off.

~~~
234tq3w4gfase
yeah that's definitely a scary possibility, though I would think explosives
would be the more realistic threat

anyways, electronic actuation of a firearm is already illegal in the US. if
you're firing a gun in any way besides actually pulling a trigger with a
finger, chances are good it's illegal (with some small leeway such as turning
a handcrank)

~~~
logfromblammo
If you're going to murder someone, you're probably not too worried about arms-
control laws. You're already planning on doing something _way_ more illegal.

But if you're going for some sort of obscure criminal achievement by only
breaking _one_ law at a time, then just make your lethal device something that
does not legally qualify as a firearm. A black-powder muzzle-loader or a
Bulgarian ricin-umbrella would do the trick, depending on whether you care
about noise or not.

Or, like the parent mentions, make a grenade, and deliver it via remote-
piloted aircraft.

Realistically, though, very few people on this planet would bother with any of
that when they can often just kick down the door of your home, kill you with a
tool purchased from a hardware or sporting goods store, and just leave before
police can respond. The clearance rate for violent crimes in the US is already
low enough, even for people who give no thought whatsoever to covering their
tracks.

I'll continue to assume that unmanned aircraft are carrying cameras, and
assume that if someone wants to murder me, they need a compelling reason to
have a more complicated plan than to just bash my skull in with a big rock
when I'm not looking.

------
zxcvvcxz
Who's ready for some "Deep Pimping"?

[http://gta.wikia.com/wiki/Pimping](http://gta.wikia.com/wiki/Pimping)

ABSTRACT: We explore the adage "pimpin' ain't easy" in the context of deep
reinforcement learning in the OpenAI environment GTAV. We train a Deep
Q-Network (DQN) to optimize player cash flows. Over the course of training,
our network learns to keep these hoes in line, and discipline misbehaving
customers.

------
captainarab
This is incredible! Huge fan of GTA V / Rockstar as well as Open.AI

Will be super interesting to see how all of the randomness in GTA V driving
influences autonomous driving AIs.

I could see them being a bit more "cautious" in order to deal with all of the
crazy drivers (and events) sharing the roads of Los Santos.

~~~
cr4zy
Thanks! Yes, the game is highly scriptable[1], so you can test the types of
scenarios you'd want. We've disabled violence against the driver, but could
enable[1] it on-demand if it's useful for training.

[1] [http://www.dev-c.com/nativedb/](http://www.dev-c.com/nativedb/) [2]
[http://gtaforums.com/topic/822314-guide-driving-
styles/](http://gtaforums.com/topic/822314-guide-driving-styles/)

~~~
rl3
What happens when a car hits a pedestrian at speed? Is it still a bloody mess
as in the normal game?

------
madez
I wonder if AI is where games can make use of many cores. If the game logic
can not be reasonibly parallelized anymore, the unused computing capacity can
be used to add "intelligent" agents into the game. They don't need to do
revolutionary things, just being more complex, more "reasonable" and less
predictable than the usual agents would be enough. I think that has the
potential to make some games more interesting.

Imagine a game like Warcraft 3 where the units actually seem to have a
personality and follow "their guts" in battles rather than following simple
patterns like attacking the nearest enemy.

Maybe I'm ignorant about the current state of the art and games already do
that. Do you know examples?

~~~
squeaky-clean
> Imagine a game like Warcraft 3 where the units actually seem to have a
> personality and follow "their guts" in battles rather than following simple
> patterns like attacking the nearest enemy.

> Maybe I'm ignorant about the current state of the art and games already do
> that. Do you know examples?

It's a really tough trade-off between fun and realism. It's cool that your
units have personality for a bit "Oh, look, those 2 dwarves are fleeing from
the troll fight early!", but not fun once they disobey your intentions "That
fucking troll has 3 hit points left. If my 2 dwarves didn't run away, it would
be dead! Now it's destroying my base!"

One game I remember striking a good middle ground with this was the original
Battle For Middle Earth. It changed animations, sounds and other aesthetic
things based on AI, while the game remained the same mechanically. If your
units ran up against a mountain troll, you would see the ones in front of it
cowering and holding up shields, while units behind attack furiously, but
every unit was really doing the same damage. Or if you left troops of Orcs
just standing around too long, they'd break out into fights and kill other
orcs, but the animation is done as a trick where as one orc model "dies", a
new one spawns in the group. So you didn't really ever lose Orcs, even though
there would be orc corpses strewn wherever you had been camping.

I think the main reason though is just that AI doesn't really sell. Graphics,
physics and voice actors sell. You can blame a failure on AI, but very rarely
is success attributed to good AI in a game.

~~~
alasdair_
>I think the main reason though is just that AI doesn't really sell. Graphics,
physics and voice actors sell. You can blame a failure on AI, but very rarely
is success attributed to good AI in a game.

The main reason that I avoided buying Civilization 6 is that the AI was
notoriously bad. For any kind of turn-based strategy game, I want the AI to be
superb.

~~~
shawn-furyan
I'll note that this actually makes the original point, since despite your
avoidance of the game, Civ VI is estimated to have been the top grossing game
on Steam in 2016. So, even in this case, the game has sold gangbusters despite
lackluster* AI.

*I didn't intentionally rhyme lackluster with gangbuster. Rather these are the words that sprang initially to mind, and it's a kind of satisfyingly weird combo, so I'm leaving it in.

Source: [https://galyonk.in/steam-sales-
in-2016-def2a8ab15f2#.nxpnuko...](https://galyonk.in/steam-sales-
in-2016-def2a8ab15f2#.nxpnuko8h)

------
lxe
It looks like it's not just using the camera/vnc images as input, but also
uses depth data to simulate LIDAR input
([http://deepdrive.io/](http://deepdrive.io/)).

I wonder, what makes it hard to rely on solely the 2 cameras to
calculate/guess the depth, similarly to the way us non-robotic drivers do it,
when it comes to machine sensing.

EDIT: found this:
[https://web.stanford.edu/class/ee368/Project_Autumn_1516/Rep...](https://web.stanford.edu/class/ee368/Project_Autumn_1516/Reports/Appiah_Bandaru.pdf)
on the topic. Is there anything else I should read up on?

EDIT: found lots of articles: [https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&e...](https://www.google.com/webhp?sourceid=chrome-
instant&ion=1&espv=2&ie=UTF-8#q=optical+flow+depth+estimation)

------
mysterydip
Open world games seem like a great place to work with AI. What has
traditionally prevented more intelligence, in my mind, was the ability of AI
to understand and integrate with its environment. For example, if I say I
threw a baseball, you can visualize that. In a virtual world, the AI can "see"
and potentially interact with every object to increase understanding. Great
experiment and I look forward to seeing how far you can take it!

------
uptown
Off-topic, but does anyone know whether it's the GTA engine used to record
these videos:

[https://www.youtube.com/watch?v=T5kooJeXGiI](https://www.youtube.com/watch?v=T5kooJeXGiI)

There's a ton of them on YouTube, but I can't figure out how they make them.

~~~
kowdermeister
It's just a normal stock game or some mod added.

Some use the GTA V. director: [http://kotaku.com/on-youtube-millions-watch-
shows-for-childr...](http://kotaku.com/on-youtube-millions-watch-shows-for-
children-made-enti-1788282618)

The creepy thing is that this video is targeted at 0-3 year olds who randomly
tap on mom's iPad. If you think like that now it's clear why the title is just
keyword stuffing, the lame music and the spiderman outfit.

80 million views in a month is a lot of money:
[https://www.youtube.com/watch?v=ipTJNNvW-
Gw](https://www.youtube.com/watch?v=ipTJNNvW-Gw)

Just checked their channel: 4,316,098,805 views

~~~
uptown
"80 million views in a month is a lot of money"

What would 80 million views translate to in dollar terms?

~~~
keypusher
I have seen $2,000 / million views, so $160,000 / month?

------
sytelus
This is super cool. Are there any good resources/how-to modify GTA V for
things like this? I'm looking in to GTA main website and wikies but info for
developers looks thin.

Here's few links I found so far:

[https://en.wikipedia.org/wiki/Modding_in_Grand_Theft_Auto](https://en.wikipedia.org/wiki/Modding_in_Grand_Theft_Auto)

[http://www.dev-c.com/gtav/scripthookv/](http://www.dev-c.com/gtav/scripthookv/)

[https://www.youtube.com/watch?v=wS8oJTHqf8Q](https://www.youtube.com/watch?v=wS8oJTHqf8Q)

[https://www.reddit.com/r/GrandTheftAutoV_PC/comments/38wct0/...](https://www.reddit.com/r/GrandTheftAutoV_PC/comments/38wct0/guide_to_modding_gta_5_safely_and_effectively/)

[https://steamcommunity.com/sharedfiles/filedetails/?id=55807...](https://steamcommunity.com/sharedfiles/filedetails/?id=558079253)

[http://gtaforums.com/forum/109-tutorials/](http://gtaforums.com/forum/109-tutorials/)

Also I'm reading that GTA folks went out of their way to prevent modding. So
may be everything you do with GTA is a hack and there is no supported APIs for
developers?

~~~
drivingmenuts
Rockstar's policy is that you're free to mod the single-player game, but any
modding whatsoever of the online game is forbidden, no matter how miniscule.
They detect it and it's a potential ban.

That said, they don't publish anything about modding and don't have an
official API available. Everything has been done third-party.

The modding has caused real problems in the online play - player in godmode,
handing out in-game money, etc. No matter the efforts of R*, they seem to be
unable to eradicate it for good. The group of players I previously played the
game with, including myself, have all become extremely disenchanted with the
game by the online modders.

~~~
yarou
It ultimately boils down to a cat and mouse game. What you end up with are
extremely intrusive anticheat tools that do all sorts of nasty Trojan-like
things; for example, injecting DLLs into a player's host machine to determine
if they have modified the memory of the game.

The anticheat process itself will hide and run in kernelmode. I don't know
about you, but I don't feel terribly comfortable having a closed source binary
running in ring0.

~~~
drivingmenuts
I'm more comfortable with an anti-cheat blob running in ring0 than I am game-
mod code running in any ring.

But, I run on a machine that was purpose-built for running games and if it all
goes to hell, I just wipe and reload. All the games are from Steam, so they're
easy to get back on there (and I only play a handful of those, so ...).

------
wrsh07
This feels like an important step in the democratization of AI.

We have algorithms, data, and computing power which are unevenly distributed.
Providing easy-to-access training data for a self-driving car feels like a
huge step in democratizing the data aspect of car AI.

~~~
drcross
With all respect I think "easy-to-access" is a bit of an understatement when
you are comparing a 3D game to real-world sensor data. They are barely
comparable.

~~~
hughes
If it is highly scriptable, we may see people simulating sensor data within
the GTA V environment. Data streams from cameras, lidar, and other types of
sensors could be simulated with as much noise as you want. This could be used
as input to train an AI using data that looks much closer to what you'd see in
the real world.

~~~
argonaut
This is way easier said than done. Even with added noise, the sensor data from
the real world is nothing like simulation.

------
celerrimus
Probably Uber should practice in GTAV, rather then on real roads:)

------
dsugarman
It would be really great to see OpenAI sponsor AI contests, the # of people
contributing / progressing on specific goals would be enormous.

This is really awesome, great work to those involved.

------
iagooar
So imagine I'd like to start doing cool stuff like this by myself. Where do I
start?

------
LeanderK
too bad GTA V is not open-source for researchers. I don't know how
customisable it is, but e.g. experiencing snow/ice or a failure like
spontaneous pressure loss in one of the tires would be interesting to
simulate.

~~~
cr4zy
Yes, it would be great to have source! The modding community does make up for
this in some ways. For example, mods provide access to things like finding
memory addresses for things like steering angle that weren't available in the
API for game scriptors working on GTAV:

[https://github.com/crosire/scripthookvdotnet/blob/dev_v3/sou...](https://github.com/crosire/scripthookvdotnet/blob/dev_v3/source/scripting/Vehicle.cs#L841-L856)

re snow: It is possible to lay fresh snow anywhere in addition to drive
through this area with snow burms, etc..

[https://www.youtube.com/watch?v=rOo2mtuPkKQ&feature=youtu.be](https://www.youtube.com/watch?v=rOo2mtuPkKQ&feature=youtu.be)

Traction is definitely affected in snowy areas.

re tires bursting: it seems possible

[https://github.com/crosire/scripthookvdotnet/blob/2b14a22ad3...](https://github.com/crosire/scripthookvdotnet/blob/2b14a22ad3095e6147289af45c0bf49797a3c39e/source/scripting/Vehicle.cs#L1650)

------
justifier
> artificially slowed to 8FPS

i'm sure this is to offer relief to the nn but does this further simulate
'reality'?

are our autonomous vehicles seeing at 8 frames per second?

~~~
alderz
As the average human reaction time is 0.19 seconds, I don't see 8FPS too bad
for a start. More FPS is always nice, of course.

~~~
will_pseudonym
Reaction times and flicker fusion rates[0] are orthogonal. 0 ms latency at 8
FPS is different than 190 ms latency at 120 FPS. E.g. If there is an event
that takes less than 125 ms, at 8 FPS there is a probability that you would
miss it entirely! At 190 ms latency with 120 FPS, you would be guaranteed of
seeing a 125 ms event, but processing of that information would take 190 ms
longer.

[0]
[https://en.wikipedia.org/wiki/Flicker_fusion_threshold](https://en.wikipedia.org/wiki/Flicker_fusion_threshold)

------
9erdelta
Just a heads up to anybody who tries to get this running, you may have to
request permission from Amazon to get the required instance going
(g2.2xlarge). Apparently they are making sure people aren't trying to set up
botnets and bitcoin mining. Just sent in my request, we'll see what happens.

------
hacker_9
The video shown is a bit underwhelming; the car moves at 1 mph with pretty
much just one car behind it the whole time.

~~~
ph0rque
How much real-time is needed to get 1k (or 1M) iterations, to improve it so
that it moves as fast as the speed limit allows while obeying all the laws?

~~~
cr4zy
Convergence on target speeds and steering is actually very quick (a matter of
hours on a modern GPU / AlexNet / 227x227 RGB) once you have the data.
Generating training data with the correct speed limit and providing speed
limit based rewards is the missing step. The training car already obeys
traffic laws.

~~~
ph0rque
Cool! You've proved my point :)

------
nojvek
This would make an interesting plot for a movie. Someday hacks GTAV ai into
Google self driving cars. Suddenly "do no evil" company becomes "fuck the
world up" and man tries to fight cars.

Anything that has wheels is dangerous to humans!

------
KennyCason
Imagine using this to train and to "unit" test a self driving car. With
realistic graphics and the ability to control events, I wonder how useful this
could be. (e.g. simulate near accident situations)

------
zitterbewegung
It looks like it has driving support but I read the post and can I make a
pedestrian agent? It doesn't appear that I could. This looks like something I
would like to play with and an awesome project.

------
deepnotderp
Damn, OpenAI is a house

------
IMTDb
Did anyone try to feed this AI with actual dashcam images and see if the
driving learnt in GTA would be applicable in real life situations ?

------
kuprel
Is it possible to return observations that a well equipped driverless car
would receive (e.g. lidar, sonar, multiple cameras, IMU, etc)?

~~~
kuprel
It seems like it would almost be easier to hire some of rockstar's game
designers and build a realistic self driving car environment from the ground
up, given the amount of funding OpenAI has

~~~
JabavuAdams
Nah. This is second-system syndrome thinking. Definitely easier to modify the
existing huge working system.

~~~
kuprel
GTAV is closed source

------
ex3ndr
Next Step: Build Street View map of GTA V.

------
fb03
I know this is totally unrelated, but i'd love to see a ML framework have a go
at NetHack!

------
9erdelta
This is super intriguing! Didn't have GTA V for PC but now I do!

------
cosinetau
I'm not sure how anti cheat methods will work now.

------
learnf007
looks like the blog post got deleted? why???? Sad face :(

------
AbenezerMamo
This is actually pretty exciting!!!!

------
flippyhead
I live in the future.

------
rubyfan
What could go wrong

------
vyacheslavl
now the interesting part: put it in front of ethical problem (take one from
[http://moralmachine.mit.edu/](http://moralmachine.mit.edu/)) and lets see
what it does

------
kobeya
I'm probably one of the most critical people of the new AI-safety craze, but
even I wonder if training AI on games like GTA is a good thing to do... if it
is something I'd be hesitant to let my kids play, should we really be training
our mechanical offspring on it?

~~~
megablast
It is not playing the game, it is using the game as a driving simulator.

~~~
kobeya
It's using the game as a driving simulator by playing it.

