
Python plays Grand Theft Auto V - adamnemecek
https://pythonprogramming.net/game-frames-open-cv-python-plays-gta-v/
======
sentdex
Wow, hello hacker news! I haven't seen pythonprogramming.net load this slow in
a ... ever.

Thanks for sharing my silly project!

Any ideas, pull requests, or critiques are welcome.
([https://github.com/sentdex/pygta5/](https://github.com/sentdex/pygta5/))

I'm currently working on PID control, and also contemplating switching the
model to a DQN, with a reward for speed (whether this is perceived, or read
from the game directly) IF we're on a paved surface. If you know about either,
don't be shy.

If you hate long loading, you can watch just the videos on youtube:
[https://www.youtube.com/playlist?list=PLQVvvaa0QuDeETZEOy4Vd...](https://www.youtube.com/playlist?list=PLQVvvaa0QuDeETZEOy4VdocT7TOjfSA8a)

edit: added link to github, as well as links to just the videos since
pythonprogramming.net is going slow from traffic.

~~~
stillsut
I'm always curious when someone shows up with a super impressive project, but
has never registered a user name on Hacker News.

Can I ask "where have you been all this time" \- Do you have some other
aggregator or blogroll that you follow? Do you try to avoid time wasted
surfing different articles and comments? Did you know about HN before and
avoid it due to its reputation, etc?

~~~
sentdex
I've just never felt the urge to post on hacker news, but it's in my list of
places I check frequently for news, I just lurk though. Maybe it's because I
dunno how to hacker news, but I just look at the front page. Usually, by the
time I am seeing something on HN, whatever I think or have to offer has
already been said, usually more eloquently than I'd come up with.

I have more of a presence on reddit
([https://www.reddit.com/user/sentdex/](https://www.reddit.com/user/sentdex/)).
The bar is lower there. ;)

For news, I have a bookmark dir with a bunch of sources in it, I right click
and open all whenever I want to check in and see what the world is up to.

~~~
cr0sh
> whatever I think or have to offer has already been said

I've only been on HN for about six months now, but from what I have seen in
this short time, I wouldn't let "fear" of saying something or posting
something you think has already been done stop you.

Case in point: Your first posting. Look at the attention it has gotten. I
enjoyed it, and others have enjoyed it. But guess what?

In my short time here I've seen other similar projects posted - and that's not
anything against you or them! Ultimately, it's how we all learn from each
other, and you shouldn't let prior work stop you from posting - there may be a
nugget of new work or something else interesting about it that makes it
worthwhile.

For instance, while your project isn't the first of its kind, I do think it
may be one of the first to be done in OpenAI's GTA V universe system; even if
not, it's one of the few that have been posted here - so it's relatively
"fresh".

Just my observations, of course - and thank you for posting!

------
StavrosK
Here's a faster way to get a frame using PyGTK:

[https://www.pastery.net/xafjmn/](https://www.pastery.net/xafjmn/)

Takes around 5ms per frame for me, rather than 50.

Slightly shorter code (still using the same method):
[https://askubuntu.com/a/400384/73044](https://askubuntu.com/a/400384/73044)

~~~
sentdex
Thanks for sharing this, will definitely look into it. With the ever improving
FPS, this AI's reflexes are going to be insane.

~~~
StavrosK
No problem, glad to have helped!

------
Keyframe
Self-playing games ought to be newer generation's Logo's turtle!

~~~
nojvek
There's something magical about creating a game AI. It's you're you're a god
and you built a divine being in your diving game world.

~~~
pacman83
To make an artificial player for a simulated world we'd built was part of an
assignment in Dr. Robert Harper's 15-212 class when I took it at CMU. On each
tick of time, each of them would randomly do one of the things it could do. To
see these characters randomly pick up change, put it into vending machines,
and get out guns, ammo, and more of themselves, and then collectively gun me
down, was a thrilling illustration of an AI control problem. I felt like a god
being killed by his creations. Mindchildren?

~~~
pacman83
By the way, in case anyone knows, I would LOVE to track down the origins of
this idea. A dozen years later, when I first thought to ask Dr. Harper, he did
not remember the source, probably one of his TA's circa Fall 1992.

~~~
cr0sh
It probably goes back further than that; one could argue that things like
Conway's LIFE implement certain aspects of the idea, as well as military
"tabletop" game simulations (which go back a long ways). Other kinds of
fantasy RPGs are similar, and then you have older video games on systems in
the 1970s that could be considered.

But I think military and other more "physical" simulations likely pre-date all
of this; these systems have their own rule systems for the various "pieces" on
the "board", and use things like dice and other random number generators to
determine how things progress.

There's also probably more than a few fictional stories which set up this idea
as well.

------
frik
I wish GTA V has the GTA IV vehicle physics engine settings.

Driving in GTA IV feels so realistic, with so many attention to details from
pedestrians that really react to small little things like misfiring of half-
broken car engine and so much more.

(it's the same engine (RAGE with bullet physics), just with dumbed down
settings for more noob-friendly/casual gameplay)

And not only the vehicle physics but also the damage model has been dumped
down - the GTA IV one was superb. (Only totally different and a lot of more
resource intensive damage models in RigsofRods/DriveNG are even more
realistic.)

They had to decrease the simulation level for GTA V as the PS3/Xbox360 weren't
capable to offer the richness of GTA IV on bigger scale. Sadly they never
added the details back with the PS4/XboxOne/PC HD re-mastered release.

So to get the most detailed simulation one has to use GTA IV and install mods
to drive around on the GTA V map. See YouTube videos, it's possible and a lot
of fun.

~~~
LordKano
IV was such a disappointment to me that I haven't even tried V yet. SA felt so
expansive and open to me that when I played IV, it felt like I was trapped on
story-telling rails.

Have they reversed that trend in V?

~~~
cr0sh
I'm not a gamer. I've only barely played any of GTA, but what I have I liked.

When GTA V came out, I couldn't believe the level of detail and the size of
the world, and how much could be done within it (especially in "sandbox"
mode). I was so stoked about it, I plunked the money down to build a dedicated
GTA V Windows box (and I haven't used 'doze at home in well over 15 years). It
was the first and only game to get me to do this.

Now - I'll level with ya here - I never put that box together; I still have
all the pieces (it was going to be a Mini-ITX with 16 GB, GTX 970, etc stuffed
in a ThermalTake Core V21) - I just never got around to it (I even bought the
game on Steam).

Now I'm thinking about using all of that for a new ML workstation to replace
my current aging system (but I can still use it for GTA V I guess).

I know this isn't a ringing endorsement, but if a game can make someone like
me that hyped about it - to drop money into parts for a complete new system
just to play the game, from somebody who isn't a gamer, nor a Windows user (or
fan) - well, that game has to be something special, ya know?

The only other game that moves me like this is currently near vaporware -
Cyberpunk 2077 (and I haven't heard anything recent on it, since I've been
heads down in my ML course).

------
pavel_lishin
A question for the author of this, or other authors who create step-by-step
guides like this: do you write it as you go along, or fiddle around and then
go back and write the blogpost?

Even with small commits, I usually have a hard time reconstructing the history
of what I've done well enough to write it up like this.

~~~
sentdex
Hey there, author here. It's just me who does the entire site of
pythonprogramming.net and the youtube channel.

The way I have gone about things has changed over time, but I have found the
worst way to do documentation is to write 100% of the code, the full series in
this case, then go back and document/write a tutorial on it.

I used to just do the videos, but had lots of requests to do write-ups too, so
I started that and really hated it at first, because I was just timing it all
wrong.

So now, I usually will do maybe 1-5 videos, and then make sure to do the
write-ups on them before continuing...otherwise the write-ups suffer
significantly. I also try to do the write-ups in the same day, and before I
personally progress on to the next topics.

It's just hard, because the last thing I want to do after I've made something
that I think is cool is document it.

Other times I will actually work locally, and just either save my scripts in a
step-by-step manner, or just work in an ipython notebook to save the steps I
took in development, and then film the video, and then go back to the notebook
to do the documentation.

Not sure that really helps much, it's an "it depends," but the main thing is
to not get too far ahead of any accompanying documentation.

------
sorenjan
Would it be possible to read the game's framebuffer directly to get rid of the
screen grab step? I'm thinking that might be faster.

~~~
bitL
With source code obviously yes, though it would need interfacing between
Python and C++ (likely used in GTA V), then internal screen resizing, and as
most of the time is spent in OpenCV anyway it might not give you the effect
you wish to get.

~~~
chatmasta
You should look into method swizzling/hooking, to hook into a c++ method
called on every frame, and add a line of code to save the frame buffer to
disk. You can either do this at runtime via some kind of code injection, or at
compilation time if you can manage to re-sign the game.

I don't know much about c++ reverse engineering but I've done this when
reversing iOS apps (and this is what "tweaking" means in the jailbreak
community). The same concepts should apply.

The first step is obtaining a class dump (what it's called with iOS) that
shows you all the header files and gives you the information necessary to
understand what's going on in a debugger so you can determine which method to
hook.

Edit: this should point you in the right direction -
[https://rafalcieslak.wordpress.com/2013/04/02/dynamic-
linker...](https://rafalcieslak.wordpress.com/2013/04/02/dynamic-linker-
tricks-using-ld_preload-to-cheat-inject-features-and-investigate-programs/)

~~~
anqh4
This being such a popular multiplayer game, I'm sure the memory offsets of the
position in the map/acceleration/etc are documented and you can just hook the
process and read/write these

~~~
liotier
Or, because it is a popular multiplayer game (and thus prone to cheating),
maybe some addresses are randomized & obfuscated.

~~~
anqh4
Just google "gta 5 trainer". I'm sure there are dozens, but it's possible that
none of them is open source.

------
krzrak
This is really cool. I love how those "stuping, time wasting" games turn out
to be - with a help from some smart individual - great learning tools.

------
mandeepj
This is also relevant - [https://github.com/ai-
tor/DeepGTAV](https://github.com/ai-tor/DeepGTAV) (A plugin for GTAV that
transforms it into a vision-based self-driving car research environment.)

------
omeid2
Anyone know of a similar game that you can run under Linux?

I want to play with this but rather not spend the time to setup a vm and go
through the whole pass-through dance or setup a dev env on windows.

~~~
bko
Check out openai. They have a lot of Atari games
([https://gym.openai.com/envs#atari](https://gym.openai.com/envs#atari)) and a
car racing game
([https://gym.openai.com/envs/CarRacing-v0](https://gym.openai.com/envs/CarRacing-v0)).

For the Atari/car racing games you get direct pixel output of the game so
using a neural network for image processing is required. But other games have
a much simpler environment where you can focus on reinforcement learning.

There's even classic doom.
[https://gym.openai.com/envs#doom](https://gym.openai.com/envs#doom)

~~~
omeid2
Thank you.

------
jackdh
Now this is is a great way to get people interested in coding. I'd of loved to
have something like this when I was learning!

------
coupdejarnac
Great tutorial, watched the first few when he did them a week or so ago. I was
one of the people suggesting he use PID controllers. While this is a ways off,
it is fun to think of a project like this evolving into an open source self
driving car package. pip install jalopy, or something like that.

~~~
sentdex
Any chance you know _how_ to implement PID control? I'm still throwing errors.
Last night I was looking through a pull request that threw a breaking error
and realized it was due to two uses of D (one for the key, one for the D in
PID). Still having issues though after fixing that.

If you have any ideas:
[https://github.com/Sentdex/pygta5/pull/3](https://github.com/Sentdex/pygta5/pull/3)

~~~
coupdejarnac
Yes, I do. Also, I was the guy who suggested using two PI controllers- you can
probably get by without the D term. I will look at it later tonight.

------
jprissi
The tutorial is great and the other website resources seem to be really
interesting. Glad it was shared here !

------
creo
I would love to have public API for Witcher3 or GTAV. Imagine all those crazy
things you could do.

~~~
falcolas
In a way, you do. The inputs are well defined (mouse and keyboard), and the
outputs (graphics) can be captured fairly easily. Interpreting those outputs
is a lot harder than interpreting a stream of "character A at point (X,Y,Z),
character B at point...", I agree.

~~~
creo
screen capturing technique also lacks things like find(X,Y,Z) or
events/triggers (ex. when collision happens)

~~~
falcolas
True, it fails to provide explicit notifications, but it does provide implicit
notifications which could be recognized.

------
roystonvassey
Wow, just wow. I am going to try doing this with my GTA SA.

I am amazed with how versatile and fun Python is!

------
cdnsteve
Very neat! I can't wait to take a look at this and give it a shot. The author
mentioned it would be better to offload the screen capture to the GPU, would
this be achieved through something like CUDA?

------
kyberias
Using python for all those individual WIN32 calls looks awkward. This
interfacing would be easier in pure C.

------
oculusthrift
i think the main contribution from this is just finding a way to get the
frames from the game into opencv. Would be interesting to see a more "real"
self driving ai method with this. I.E. a CNN

~~~
Houshalter
Could you run GTA5 and a CNN at the same time? I think that would fry your
GPU.

~~~
cr0sh
It can work - I won't re-post everything again here, but I've already
mentioned in this thread how I used my 750 ti for training a CNN based on
NVidia's End-to-End model to self-drive a virtual vehicle around a simulated
track (this was part of Udacity's Self-Driving Vehicle Engineer Nanodegree).

Udacity supplied the simulator, I set up CUDA and such to work with TensorFlow
and Keras under Python. I had to drop the res down on the simulator (windowed
at 800x600 IIRC), and training was done without the simulator (I used the
simulator to generate the datasets - this is built into the simulator). The
resulting model was actually fairly small (less than 300 kbytes); I scaled
down the input images (originally 640x480, scaled to 160x120) to limit the
number of initial parameters to the CNN, then applied aggressive dropout and
other tricks to further keep things in check thru the layers as I trained (I
could have used batch processing, too). The resulting model worked well with
the simulator afterward - and it had no problem with memory or anything else
in order to keep up.

------
cactus_frenzy
yo dawg! I heard U like GTA V and Python tutorials, so I put a tutorial inside
your tutorial so you can play GTA V while you're learning!

