
Full Autopilot in GTA Using TensorFlow - littlemtman
https://littlemountainman.github.io/2020/05/12/openpilot/
======
mistymountains
Let's all give this 12 year old kid some applause. Me water-cooling my xbox
360 (yes, and I wasn't alone in doing so) at 13 felt cool, but this is on
another level.

~~~
gibolt
The best part about being a kid is that you have nothing to think about other
than a little school work. You are free to scour the web finding all the
newest tools and put them to use for 'play'.

Not saying it can't be done as an adult, just harder to get that time and
unbroken/unlimited focus.

~~~
TheSpiceIsLife
I’d love to spend some time living in a serviced apartment with meals
included.

Or something similar to a retirement village or university student
accommodation, or be able to afford a staff of servants. You get the idea
anyway.

~~~
squnch
You can do this quite affordably in huge numbers of countries if you can earn
your income with a decent remote job in the US. The things to think about are
time zone, tropical disease, quality of food, and safety. There are a whole
range of places from Arkansas, to Costa Rica, to Panama, to Nicaragua.

The thing you'll lose out on is anyone you know, unless you can get them to go
with you, or can make new friends in your destination.

~~~
TheSpiceIsLife
I’m in Australia so I often half-joke: let’s sell everything we own and go
live like kings on a remote Indonesian island.

------
rvz
Grand Theft AutoPilot. Now that would be a great name for this project. :)

~~~
littlemtman
Yes that's true :)

------
tikej
And to me this is the essence of hacking and hacker culture, not “how do I
create MVP from my idea and become an entrepreneur” or “this and that in
company XYZ” mindset.

So glad to see people doing silly things that in fact require skill. This is
why I come to hackernews.

Keep up great work with this wonderful attitude!

~~~
smt88
Not sure why you need to put down entrepreneurs while praising this kid.

Hacking a company together is really hard. Like tech hacking, it requires
intelligence, creativity, and persistence.

There's no reason we can't read about and respect both on this site.

~~~
kick
The concern is in the appropriation of the term 'hacker'.

He's not putting anything or anyone down, he's saying that this is hacking,
and MBA posts of acronym soup aren't.

I think his implied conclusion is probably wrong-ish (both are arguably right
for HN, even if I vastly prefer this), but he's right in that this is hacking.

~~~
rapind
One could argue WeWork hacked unicorns.

~~~
toofy
One could argue anything they like, it doesn’t mean their argument (or
sophistry in general) leads to a better acquiring of knowledge through
discussion.

In my opinion, the gp’s discussion is one worth having.

------
wigl
What ever happened to the OpenAI/universe GTA sandbox? Is there any reason
Rockstar shouldn’t support research using their game?

~~~
numpad0
For some reason they really didn't like GTA being seen as a research platform
and C&D'd projects.

~~~
vhold
Perhaps because it could lead eventually to the most undetectable cheats yet
and undermine their Skinner box cash extractors.

------
numpad0
For inputs, roundabout but easiest ways I know is to either just get a Pro
Micro and use the gamepad library in Arduino IDE to make it a serial-to-
gamepad adapter, or use it in conjunction with a cheapest no-brand knockoff
Xbox 360 controller(which are easier than genuine controllers to mod as well)
and solder digital/analog out from Arduino to digital/analog sensing pads for
the same effect.

The latter allows real Xbox 360 to be used(don’t go online with it, that is
cheating). Xbox 360 is past EOL so I bet they can be had for cheap, steps to
capture is well explained by millions of wannabe streamers, and probably less
tedious to set up at the same time than a PC.

~~~
bluesign
Or full software solution for PlayStation is Remote Play.

------
hn_throwaway_99
This kid is awesome. I say that even though he's going 25 MPH in the left lane
:P

------
renewiltord
Great job, love it. Do commercial solutions use simulations and stuff? Sounds
like a great way to bootstrap is to use some of these more realistic games to
get some initial work.

By the way, I found this part funny:

> _...if it doesn’t make sense to you think about it and try to understand
> it._

We had a teacher once who'd say "If you don't understand, try to understand".
Haha!

~~~
dmoy
> Do commercial solutions use simulations and stuff?

Yes

Here's a random news article on it

[https://www.theatlantic.com/technology/archive/2017/08/insid...](https://www.theatlantic.com/technology/archive/2017/08/inside-
waymos-secret-testing-and-simulation-facilities/537648/)

~~~
renewiltord
Très cool. Thank you.

------
nagyf
Reading the title my first thought was someone is driving a driverless car in
Toronto, how cool is that. I was disappointed to see it's about the game.
(Great article though)

------
braunshedd
I'd love to see this done in Forza, although I'm sure that's optimizing for
the wrong kind of driving behavior (lol)

Amazing work!

~~~
mhh__
A'I' in racing games is already a cracked problem I think (unless you mean
forza horizon?).

The AIs in Rfactor2 are even pretty good wheel to wheel on some tracks -
definitely a lot better than online rookies who are quick but blind. The only
noticeable thing is that they don't learn over the course of a race. They seem
to be programmed to make mistakes realistically, i.e. if the car loses grip in
the wet they don't lock back onto the track.

~~~
bowmessage
Sure, but this is a different kind of agent - one that takes screen frames as
input, rather than one that has direct access to game state.

------
mendelmaleh
sentdex did something similar a year or two ago:
[https://www.youtube.com/playlist?list=PLQVvvaa0QuDeETZEOy4Vd...](https://www.youtube.com/playlist?list=PLQVvvaa0QuDeETZEOy4VdocT7TOjfSA8a)

~~~
littlemtman
Behavioural cloning, doesnt work in the real world ! I use path and lane
detection

------
Qworg
If you want to play with ML racing, AWS DeepRacer is free for May (and you can
compete against NASCAR drivers).

[https://aws.amazon.com/blogs/aws/join-the-
formula-1-deeprace...](https://aws.amazon.com/blogs/aws/join-the-
formula-1-deepracer-proam-special-event/)

------
CrumResearch
Howdy, I came here from [http://discord.comma.ai](http://discord.comma.ai).
littlemtman, what related source code is available? How about more details and
the code for connecting to the XBox controller interface, is it open?

~~~
cheschire
I'm not sure what OP used, but definitely look into FreePIE.

[http://andersmalmgren.github.io/FreePIE/](http://andersmalmgren.github.io/FreePIE/)

------
jaimex2
Reminds me of the the Mario Kart version someone made 3 years ago:
[https://github.com/kevinhughes27/TensorKart](https://github.com/kevinhughes27/TensorKart)

------
serf
this is pretty cool.

I used a similar scheme (two computers, one with HID control over the other,
and a webcam to observe) as a way to hide a simple aimbot as a proof-of-
concept for getting around on-computer anti-cheats that would search through
memory on popular FPSs. This was years ago on games with weak anti-cheats,
anyway.

It worked OKish. It had to rely on computer vision to do the aiming, so it was
significantly slower than examples that would rely on memory pointers or
straight pixel recognition from the screen itself -- but the point was proven,
the computer that was cheating looked fine as far as memory was concerned.

modern anti-cheats that look at things like pointer movement and player
randomness would definitely flag it now-a-days, but I had fun.

The hardest problems were room lighting, which at the time the cheapy webcams
had terrible issues with anyway, FPS, and resolution.

Major shortcuts had to be taken back then due to the resolution and fps
mismatches between the input and output, shortcuts by which reduced the bot
performance significantly, but I bet that'd be a lot easier to deal with
today.

Now, my point : I know that a majority of the memory structs are available,
documented, and deconstructed for GTA V, having written cheats for the game in
the past. Maybe you can exploit that to emulate some of the functions of an
IMU?

Lots of vehicle physics variables are fairly easily exposed. "GTA V Memory
Structs" google search will get you started on that road, should you care to.

Neat project. Good luck.

~~~
littlemtman
Thanks, yes maybe I will add proper sensoring to the project with emulating a
can bus.

------
savrajsingh
Great work! Now let's drive faster than 25 mph -- it's called Grand Theft Auto
for a reason :)

~~~
littlemtman
Yes I will add a speeding video !

------
jgalt212
The real challenge of AI vis a vis real world problems is how to create
_infinite_ examples, labelled observations, etc at _finite_ cost.

------
permalac
Can I buy stock of this company?

~~~
littlemtman
You can't

------
verelo
Can it drive a plane / the tank on the road? How do motorbikes go? So curious.

~~~
littlemtman
Yes a tank would be possible. Both only from the front camera view, the model
is not used to see the car it is driving otherwise !

~~~
pugworthy
Interesting. I have a World of Tanks twitch streamer's video playing on the
side as I read this, and also thought tanks.

------
z3t4
Since this is GTA, it would be cool to add pedestrians to the mix...

~~~
littlemtman
I tried it in the city center it was going nowhere, the bigger and complex the
models get the more they learn "intuitively" i.e. avoiding pedestrians,
avoiding crashes etc. Like the video where Autopilot predicted a crash 3
seconds before it actually occured.

------
tinus_hn
Wasn’t there a Twitch channel running this?

~~~
littlemtman
Yes but what he did was behavioural cloning, I do lane and path prediction
that's a whole different level

------
baron816
How about a self riding horse for RDR?

------
GekkePrutser
But can it do a drive-by???

------
muffin24
sick

------
bzb3
My heart broke when I saw it wasn't one of the renderware era GTA games.

