Hacker News new | past | comments | ask | show | jobs | submit login
Full Autopilot in GTA Using TensorFlow (littlemountainman.github.io)
486 points by littlemtman 13 days ago | hide | past | web | favorite | 111 comments

Let's all give this 12 year old kid some applause. Me water-cooling my xbox 360 (yes, and I wasn't alone in doing so) at 13 felt cool, but this is on another level.

Your praise notwithstanding, I think this kid "was" 12 in 2014 or 2015. From his "about" page [0]:

> I am currently a student working on programming projects since I am 12 years old. Back then I was learning HTML with a good friend of mine. We were trying to sell those websites we made but with fairly little success.Around 4 years ago I stumbled across Python. And I immediately fell in love with it. Later 2015 I was impressed by the Google Assistant...

[0]: https://littlemountainman.github.io/about/

Correct !

The best part about being a kid is that you have nothing to think about other than a little school work. You are free to scour the web finding all the newest tools and put them to use for 'play'.

Not saying it can't be done as an adult, just harder to get that time and unbroken/unlimited focus.

I’d love to spend some time living in a serviced apartment with meals included.

Or something similar to a retirement village or university student accommodation, or be able to afford a staff of servants. You get the idea anyway.

Imagine if jail fit that description.

you'd be surprised how far 600/month goes in south america

Just being retired is good enough for me.

You can do this quite affordably in huge numbers of countries if you can earn your income with a decent remote job in the US. The things to think about are time zone, tropical disease, quality of food, and safety. There are a whole range of places from Arkansas, to Costa Rica, to Panama, to Nicaragua.

The thing you'll lose out on is anyone you know, unless you can get them to go with you, or can make new friends in your destination.

I’m in Australia so I often half-joke: let’s sell everything we own and go live like kings on a remote Indonesian island.

Yes that's really the best part. School 5 hrs a day and then free time !

not 12. 15

I'm still jealous. At 15 I had a 10 MB hard drive and 4 color graphics available to me. While I am happy about who I am, I do wish sometimes my childhood had better technology available.

Frankly I think if I'd had better technology as a child I'd be even more content-addicted and boredom-averse than I am now. At least the way I am I can make it through a wedding without checking my phone.

The downside there is that I _had_ to make it through a whole bunch of weddings without a smartphone and social media to escape to...

I am impressed by this 15 year old.

His voice sounds a little well developed for a 12 yr but what do I know! :) Edit: after checking it seems he's 15. Still pretty impressive tho!

It's all fun and games until you start dealing with C++ thread issues.

Grand Theft AutoPilot. Now that would be a great name for this project. :)

Yes that's true :)

Your work here is done.

And to me this is the essence of hacking and hacker culture, not “how do I create MVP from my idea and become an entrepreneur” or “this and that in company XYZ” mindset.

So glad to see people doing silly things that in fact require skill. This is why I come to hackernews.

Keep up great work with this wonderful attitude!

Not sure why you need to put down entrepreneurs while praising this kid.

Hacking a company together is really hard. Like tech hacking, it requires intelligence, creativity, and persistence.

There's no reason we can't read about and respect both on this site.

The concern is in the appropriation of the term 'hacker'.

He's not putting anything or anyone down, he's saying that this is hacking, and MBA posts of acronym soup aren't.

I think his implied conclusion is probably wrong-ish (both are arguably right for HN, even if I vastly prefer this), but he's right in that this is hacking.

100% agree. Winamp was a hacker's application. iTunes is a company's application.

One could argue WeWork hacked unicorns.

One could argue anything they like, it doesn’t mean their argument (or sophistry in general) leads to a better acquiring of knowledge through discussion.

In my opinion, the gp’s discussion is one worth having.

hacking culture was not just about the skills it takes to participate, it was also about values. One of them, openness and sharing information. open protocols over closed walls.

entrepreneurship today in particular in tech often tries to capitalise on the latter rather than the former, at least most of the time. Adopting the aesthetics of hacker culture to further institutions that could not be more damaging to an open ecosystem is already more common than actual 'hacking'.

They seem to be antithetical ideas somehow.

Business inherently relies on secrecy (proprietary stuff). Hackers try to make secret things open.

What makes you think that business inherently relies on secrecy?

All business is based on some kind of upper hand right? Secrecy is usually the root of that upper hand.

Sometimes you can reveal your "secret sauce" but patents/copyright/regulation will protect it from being stolen. So the "secret sauce" is still what separates you from the other businesses...

It makes sense in my head, but maybe I'm rambling... I'll try and write a blog post to explain my thoughts a bit better.

Can you think of a business that is entirely transparent?

I think the framing is kind of off.

Being a successful business doesn’t mean that you must be secretive about your business operations. I don’t think it requires an “upper hand” either. Those may be key to out-competing other businesses in some way, but that is different than being successful. (Capital culture makes us think of a “successful business” as one that makes insane amounts of money, rather than one that achieves a goal.)

Of course, all businesses realistically will have some secrets, but those may often be separate from the business operations. For example, it would probably be very difficult to be public about all of your HR details, or expose all of your internal phone numbers/emails/physical addresses (humans just like basic privacy).

But many local businesses can probably operate just fine without being very secretive. There was a place in my hometown I loved that served shakes, cheese sticks, and cheeseburgers. I don’t think he did anything special that needed to be kept secret. People just like a nice diner with a decent burger.

Startup culture is basically the opposite of hacker culture. As kick hints to, the term has been appropriated. This site has been dubiously hackish from the start.

Hacking together a product takes intelligence, creativity and persistence. Hacking together a company takes empathy, honesty, respect, humility, etc. It's about the people and building a team. I don't associate this with "hacking" at all.

This is a distinction without a difference. When to start a new company, you hack together both a product and a company.

Comma.ai which is used in the demo is a for-profit company that makes a sellable product.

Yes, but the author is a student making use of their open source software. I think that's in the spirit of hacking.

The point is that the original topic is against "startups" where in this example, 99% of work was done inside precisely an organization like a startup, and only the resting 1% being work of a student who simply applied a ready-made solution to his situation. The point is, regardless of hacker culture, there is nothing wrong with startup culture, it often produces great products and technology.

Yes the comma two https://comma.ai/shop/products/comma-two-devkit. If you have a newer Honda, Toyota etc. you can try it in real life !

Won't be working on your Aviato car, I am sorry :)

There's plenty of hackers working on bringing this to old vehicles. One guy got it working on his old VW Vanagon/Syncro. https://github.com/Lukilink/openpilot/blob/071_OC_WATER/READ...

But please don't try it on a real road. Find a spot where it can't endanger fellows. It's possible to find those places.

There's hundreds of people using these on public roadways. Check out the official Discord and the various Community Discords centered around it.

Over 1,000 people just today. A couple thousand every week.

In addition to running on a laptop, openpilot also runs on a phone for easy mounting in your car.


Your FAQ says Comma "openpilot performs the functions of Adaptive Cruise Control (ACC) and Automated Lane Centering (ALC)" and my 2020 Corolla is supported-but it already has those features. I read through the site and I don't see the value proposition. Is it that Comma does a better job of it?

Thank you very much !

WSJ had an interesting article the other day about why there are fewer startups now - https://www.wsj.com/articles/six-theories-on-why-fast-growin...

tl;dr is that people are now more motivated by the idea of being an entrepreneur than by having a great business concept.

I think Scott Galloway has some very valuable input on this topic. I might do it a disservice by trying to to;dr it here but basically there is very little if any room for innovation left when the big monopolies are currently very efficient in “featuring” your whole startup and through economies of scale (not having to turn a profit) they’ll eliminate you from the market it acquire you if they like the talent.

I see your point and I agree with it. But I also want to remark that in many ways, building a company is also a silly thing that in fact requires skill.

I mean, I'm pretty sure I'd be way richer right now if I had just taken a job at some bigco. Or kept doing consulting.

I don't feel that what I'm doing right now (building a startup because I can) is fundamentally different from my demoscene past (coding computer graphics because I could). Sure, some startups make some people obscenely rich, but the vast majority don't :-) It's not too different from art or AI hobbying or gaming or game modding in that sense.

> But I also want to remark that in many ways, building a company is also a silly thing that in fact requires skill.

It is - Tikej's point isn't that it's not a skill - but rather that it's not the right place to share these. Think of the difference between a "Startup News" and "Hacker News". Hacker news used to be very deep on tech topics, now those deep topic have become more rare.

I remember an early pg quote where he noted that being a hacker isn’t supposed to be limit to coding or tech even, but all topics that hackers are drawn to and can ply their hacking in. Social hacking to bootstrap a userbase is a on-topic as far as that quote is concerned [edit: errant autocorrect]

Interestingly, Hacker News used to be called Startup News in the early days.

Indeed, we’ve come full (half?) circle. IIRC, it was renamed to Hacker News in part to discourage meta-discussion on intellectually stimulating posts, questioning why the front page had so many posts that weren’t about startups. Now we are instead discussing startup post fatigue.

You can see the reasoning here (Aug 2007): https://news.ycombinator.com/hackernews.html. Basically, pg got bored reading about nothing but startups.

Interesting the idea that the original plan was that your votes count for more if you vote for the right kind of stories. Would seem to create an echo chamber effect where what’s already popular stays popular and new stuff doesn’t.

It's impossible to say without trying it, and we never did. (Edit: actually I remember a conversation with pg where he said he did experiment with it, but dropped it. I don't remember why though. Maybe I'll ask.)

You could make the opposite argument: the "right kind of stories" includes being unpredictable (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...), so up-weighting votes from users who are good at picking those would lead to a less predictable, more interesting front page. Conversely, the median vote tends to be for the same few hot topics, leading to a more samey (as well as more sensational) front page.

My bet is that it wouldn't change much either way, because the problems with voting seem to flow from the voting mechanism itself, not from which users are doing it. Internet upvoting is the ultimate reflexive reaction, which excludes reflection, and reflection—the slower cognitive process which considers something before reacting to it and is thus able to see something new—is the quality that picks up on uncorrelated bits and makes for good taste. (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...)

If that's true, then instead of trying to squeeze more signal out of upvotes we should add a new mechanism that encourages reflection over reflexivity. Flagging is closer to that than voting is, so something like an up-flag might be worth trying: i.e. "this deserves extra attention because of how good it is".

Thanks for the thoughtful response dang. Perhaps a blend is useful - it's good to have quick takes on articles, but the current voting structure incentivises quick takes over thoughtful responses.

In many cases, writing a quick comment to get position in the discussion and then editing or self-replying might be the only way to get engagement from other thoughtful commenters. And a response that takes more than a few hours (or in some cases, minutes) to compose is likely to have very little engagement at all.

There is a comment feed, but it's heavily biased towards more recent comments and doesn't have enough context to be useful in many cases. Perhaps the "exalt / flare" system that you suggest could help to flag up more interesting comments in a slower stream.

At the moment, unless you check your own comments or look out for your karma and investigate any changes, you're likely to miss out even on replies to your own comments, let alone interesting comments on older articles.

I'm up-flagging the up-flag idea.

It's fun to think of what a good name for it might be. The word "exalt" occurred to me at one point.

The thing that immediately came to mind for me was 'flare', partially for the obvious "Oh wow it basically perfectly fits with the intention of it!" but also for alliterative reasons: flag / favorite / flare

Flagging is closer to that than voting is

Maybe I'm a terrible person but the experiment this suggests to me is a ranking where only flagging is allowed. A brutal agoge which only the best stories survive.

Hacker News has always been the message board of the most famous startup accelerator out there.

Very true, but it is hosted by ycombinator which exists to fund startups.

Hopefully I could bring you a deep tech topic !

I imagine the OP is just bored of the generic startup technology churn cycle. It can feel as though the majority of "tech stack" articles on HN are just Yet Another Web Application Tool. We have solved making web applications so hard that we have ascended to meta-tools, tools that manage our tools or use our tools for us.

It is refreshing to be reminded of the myriad of ways computers can interact with the world! I recently refreshed my love for technology by getting into 3D printing, it's nice to have a tangible result for my efforts behind the keyboard. The machine is surprisingly rudimentary as well, and reminds me of old school CNC machines, so it's relaxing in the way driving a classic car can be. You can understand all the parts and their purpose intuitively, as they operate in the real world.

What ever happened to the OpenAI/universe GTA sandbox? Is there any reason Rockstar shouldn’t support research using their game?

They got a cease and desist from the publisher [1]. The broad reasons are given in the article, but it's pretty much what you would imagine, IP issues. Also OpenAI seem to have taken down all their blog pages mentioning it without notice.. . Some are archived luckily [2].

I understand that they have to comply with regulations, however I was a bit disappointed that they weren't transparent about it.

[1] https://www.forbes.com/sites/aarontilley/2017/10/04/grand-th...

[2] http://web.archive.org/web/20170111195314/https://openai.com...

I was also wondering about this. There are two issues in OpenAI's GitHub repo related to the removal, with not much information:

- https://github.com/openai/universe/issues/126

- https://github.com/openai/universe/issues/160

For some reason they really didn't like GTA being seen as a research platform and C&D'd projects.

Perhaps because it could lead eventually to the most undetectable cheats yet and undermine their Skinner box cash extractors.

If there was a nice solution for it, I would have done it but there isn't. Also the action space for that would be enormous. Maybe I will be making a MuZero Bot drive in GTA, that would be fun !

Did some searching around and found a hidden fork, have you tried this?


Yes I have, but I would prefer a really clean Gym Env for just vehicles in GTA, with an understandable action space.

I mean. Can you imagine the shit storm that ensues if someone use GTA for autonomous training data and caused deaths IRL. I don’t even want to think about that news headlines.

For inputs, roundabout but easiest ways I know is to either just get a Pro Micro and use the gamepad library in Arduino IDE to make it a serial-to-gamepad adapter, or use it in conjunction with a cheapest no-brand knockoff Xbox 360 controller(which are easier than genuine controllers to mod as well) and solder digital/analog out from Arduino to digital/analog sensing pads for the same effect.

The latter allows real Xbox 360 to be used(don’t go online with it, that is cheating). Xbox 360 is past EOL so I bet they can be had for cheap, steps to capture is well explained by millions of wannabe streamers, and probably less tedious to set up at the same time than a PC.

Or full software solution for PlayStation is Remote Play.

This kid is awesome. I say that even though he's going 25 MPH in the left lane :P

Great job, love it. Do commercial solutions use simulations and stuff? Sounds like a great way to bootstrap is to use some of these more realistic games to get some initial work.

By the way, I found this part funny:

> ...if it doesn’t make sense to you think about it and try to understand it.

We had a teacher once who'd say "If you don't understand, try to understand". Haha!

> Do commercial solutions use simulations and stuff?


Here's a random news article on it


Très cool. Thank you.

Reading the title my first thought was someone is driving a driverless car in Toronto, how cool is that. I was disappointed to see it's about the game. (Great article though)

I'd love to see this done in Forza, although I'm sure that's optimizing for the wrong kind of driving behavior (lol)

Amazing work!

Seems like Forza would be easier to interface with too, example: this guy built a virtual cockpit[0] using VW parts -- the shifter, tach/speedometer all work

[0] - https://www.youtube.com/channel/UCSEqO_2kdxLtslBK9FLqtxA

A'I' in racing games is already a cracked problem I think (unless you mean forza horizon?).

The AIs in Rfactor2 are even pretty good wheel to wheel on some tracks - definitely a lot better than online rookies who are quick but blind. The only noticeable thing is that they don't learn over the course of a race. They seem to be programmed to make mistakes realistically, i.e. if the car loses grip in the wet they don't lock back onto the track.

Sure, but this is a different kind of agent - one that takes screen frames as input, rather than one that has direct access to game state.

Yeah... I bet that would make for some really exciting gameplay :D

sentdex did something similar a year or two ago: https://www.youtube.com/playlist?list=PLQVvvaa0QuDeETZEOy4Vd...

Behavioural cloning, doesnt work in the real world ! I use path and lane detection

If you want to play with ML racing, AWS DeepRacer is free for May (and you can compete against NASCAR drivers).


Howdy, I came here from http://discord.comma.ai. littlemtman, what related source code is available? How about more details and the code for connecting to the XBox controller interface, is it open?

I'm not sure what OP used, but definitely look into FreePIE.


Reminds me of the the Mario Kart version someone made 3 years ago: https://github.com/kevinhughes27/TensorKart

this is pretty cool.

I used a similar scheme (two computers, one with HID control over the other, and a webcam to observe) as a way to hide a simple aimbot as a proof-of-concept for getting around on-computer anti-cheats that would search through memory on popular FPSs. This was years ago on games with weak anti-cheats, anyway.

It worked OKish. It had to rely on computer vision to do the aiming, so it was significantly slower than examples that would rely on memory pointers or straight pixel recognition from the screen itself -- but the point was proven, the computer that was cheating looked fine as far as memory was concerned.

modern anti-cheats that look at things like pointer movement and player randomness would definitely flag it now-a-days, but I had fun.

The hardest problems were room lighting, which at the time the cheapy webcams had terrible issues with anyway, FPS, and resolution.

Major shortcuts had to be taken back then due to the resolution and fps mismatches between the input and output, shortcuts by which reduced the bot performance significantly, but I bet that'd be a lot easier to deal with today.

Now, my point : I know that a majority of the memory structs are available, documented, and deconstructed for GTA V, having written cheats for the game in the past. Maybe you can exploit that to emulate some of the functions of an IMU?

Lots of vehicle physics variables are fairly easily exposed. "GTA V Memory Structs" google search will get you started on that road, should you care to.

Neat project. Good luck.

Thanks, yes maybe I will add proper sensoring to the project with emulating a can bus.

Great work! Now let's drive faster than 25 mph -- it's called Grand Theft Auto for a reason :)

Yes I will add a speeding video !

The real challenge of AI vis a vis real world problems is how to create infinite examples, labelled observations, etc at finite cost.

Can I buy stock of this company?

You can't

Can it drive a plane / the tank on the road? How do motorbikes go? So curious.

Yes a tank would be possible. Both only from the front camera view, the model is not used to see the car it is driving otherwise !

Interesting. I have a World of Tanks twitch streamer's video playing on the side as I read this, and also thought tanks.

Since this is GTA, it would be cool to add pedestrians to the mix...

I tried it in the city center it was going nowhere, the bigger and complex the models get the more they learn "intuitively" i.e. avoiding pedestrians, avoiding crashes etc. Like the video where Autopilot predicted a crash 3 seconds before it actually occured.

Wasn’t there a Twitch channel running this?

Yes but what he did was behavioural cloning, I do lane and path prediction that's a whole different level

Maybe sentdex [0] streamed a session on twitch? He has a series on automatic driving in GTA iirc


How about a self riding horse for RDR?

But can it do a drive-by???



Context ?

My heart broke when I saw it wasn't one of the renderware era GTA games.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact