
Lane Following Autopilot with Keras and Tensorflow - yconst
https://wroscoe.github.io/keras-lane-following-autopilot.html
======
robinson-wall
This is nice work, but anyone wanting to try it for themselves should be
warned that you shouldn't unpickle data received from an untrusted source.

[https://blog.nelhage.com/2011/03/exploiting-
pickle/](https://blog.nelhage.com/2011/03/exploiting-pickle/)

~~~
wroscoe
I copied this method of loading datasets from Keras.
[https://github.com/fchollet/keras/blob/master/keras/datasets...](https://github.com/fchollet/keras/blob/master/keras/datasets/imdb.py#L43).
What's a better alternative.

~~~
Drdrdrq
Another serialization format which doesn't create objects, like JSON, XML,
CSV,...

------
nilkn
Was the track changed at all during the training? I'm wondering if there's
some subtle overfitting here where the car learned to drive along only this
specific track. It mentions this but I'm not sure what concrete actions were
taken to avoid overfitting:

> The biggest problem I ran into was over fitting the model so that it would
> not work in evenlly slightly different scenarios.

Regardless, a very cool project.

~~~
wroscoe
The method to avoid overfitting was to use the model with the lowest
validation loss, not training loss. I was able to change the track around my
house with reasonable success. I think would need many more example turns in
the training data to become robust.

------
zxcvvcxz
Great summary, I always think it's best when machine learning projects have
visuals and videos to showcase what is actually being learned.

This simple project is a good example of supervised learning from what I can
tell - the network will learn to steer "as good as" the human that provides
the training data. For a different (and more complex) flavor of algorithm,
check out reinforcement learning, where the "agent" (computer system) can
actually learn to outperform humans. Stanford's autonomous helicopters always
come to mind - [http://heli.stanford.edu/](http://heli.stanford.edu/)

------
grandalf
Consider the fairly massive changes to the competitive landscape ushered in by
the _combined_ factors of self-driving and electric vehicles:

\- For liability reasons, most of the algorithmic IP will likely be open
sourced. Either because it's required by regulators or because it's the most
efficient way for car makers to socialize risk of an algorithmic failure.

\- Electric vehicles have many fewer moving parts, which means that the
remaining parts are likely to be converged upon by the industry and used
widely. This breaks a lot of platform-dependency issues and allows for the
commoditization of parts like motors. As these become standardized and
commoditized, and easily comparable on the basis of size, torque, and
efficiency, there will be virtually no benefit to carmakers to manufacture
their own. The same applies to aluminum monocoque frames, charging circuitry,
etc.

Tesla currently differentiates its models based on how many motors and what
size batteries, but beyond that it's mostly just cabin shape, along with new
innovations like the hepa filter cabin air cleansing which will likely be a
standard part of all future models.

\- Battery tech works the same way as motors, with little competitive
advantage to be gained by automakers, especially since most of the IP in this
area is already spoken for.

Compare the number of patentable parts in a model T vs a 1998 Taurus vs a 2017
internal combustion vehicle vs a Telsa. Tesla is one innovator, and GM has
already likely patented many inventions relating to EV technology back in the
original Chevy Volt era.

All this is why Tesla acquired SolarCity and is attempting to make an
infrastructure play rather than a technology play. Only due to Musk's rare
ability to self-finance big risks is this even possible, since infrastructure
moonshots featuring $30K+ hardware units are hard to fund.

~~~
mulmen
How do you see car makers differentiating their products in a world where all
the parts including the frame are commoditized and the software is open
source?

Also, GM built an electric car back in the 90s called the EV-1. I wonder how
much innovation was in that car vs the Volt.

~~~
Drdrdrq
Interior design. Cars will evolve into mobile living spaces so the quality of
interior becomes more significant to exterior.

------
billyzs
Not to put down the OP's work (I think it's a great project), but I'm just
wondering what advantages might an ML approach have over "traditional" CV
algorithms. In a really well controlled environment lanes will be easy to
detect, and computing the difference between the current heading and lane
direction should be doable; maybe if we're talking about complex outdoor
environments and poor sensors then ML would have an advantage? Or if we're
teaching the robot what the concept of a lane is?

I think back to the days when I basically implemented lane following with an
array of photo resistors, an Arduino, a shitty robot made from Vex parts and
some c code. The problem is much simpler than the one presented in this
article, but then the computational resource used was order of magnitudes
less. At what point then, do you decide that "OK I think the complexity and
nature of the problem warrants the use of ML" or "Hmmm I think neural network
is an overkill here"?

~~~
wroscoe
Traditional CV approaches are much easier to debug as well. I chose the ML
approach with the assumption that it would be easier to build a robust
autopilot that would work in many lighting conditions. Actually my short term
goal is to get the car to drive around my block on the sidewalk (no lines).
From my experience CV approaches have many parameters that need to be tuned
specifically for each environment. While ML approaches also have parameters
that need tuning they stay constant between environments.

~~~
billyzs
I see, that makes sense. It'd be indeed worth it if we can apply a model
trained on controlled environment to a more challenging one with little to no
modification. Good luck with the project and keep us updated!

------
wroscoe
I updated this post with some of the great feedback from the comments. Also I
just ported the algo used by the last DIYRobocar race winner, CompoundEye.
Here's that post: [https://wroscoe.github.io/compound-eye-
autopilot.html#compou...](https://wroscoe.github.io/compound-eye-
autopilot.html#compound-eye-autopilot)

Thanks!

------
Cyph0n
Nicely done! But I'm assuming that this is more of an exercise rather than a
real-world application of ML? I say this because the task of keeping a car
between two lines is trivially done using control algorithms. Of course, the
CV part -- "seeing" the lines -- requires some form of ML to work in the real
world.

~~~
webaba
Obviously, "Lane Following Autopilot using my brain and controls theory" would
not make it to the top of HN. Welcome to the new era where Tensorflow replaces
Lyapunov and ML spares you the need of understanding hard problems... until
you need guarantees and safety... but but it's ok let's add more data.

~~~
gugagore
I agree with you. If you can leverage control theory from the 1950s to solve
your problem, what's the point?

However, I will state that using e.g. Lyapunov functions to prove the
stability of the system requires a model of the system. And even if you need a
guarantee for your system, that guarantee is only as good as the fidelity of
your model. For an inexpensive RC car, with slippage and saturation, without
torque control or inertial sensing, you're going to have a hard time doing
something that sounds as principled as what you suggest.

~~~
aub3bhat
You seem to be forgetting the entire vision pipeline that automatically
extracts "lanes" and that information gets incorporated in an end to end
manner requiring only true steering angles and nothing else. Its easy to
comment but its not as straightforward or trivial as one might assume.

~~~
webaba
Indeed, I was not really talking about the vision pipeline. But once you
decouple the problem (use ML for vision, planning for the trajectory, controls
for the rest), you'll get much more stability, guarantees and insight into how
to improve your problem. These kinds of end-to-end approaches are very hard to
evaluate, they have zero educational value, are not parsimonious and tend to
reduce people's analytical skills.

~~~
ogrisel
But to be able to decouple the vision pipeline you need a lot of manual
annotation work which is tedious.

~~~
argonaut
Tedious, and also solved for a decade already. Also, it's much easier to just
find lanes using traditional CV and simply using annotators to verify the lane
labels.

------
sja
I might be missing it, but I don't see instructions for installing
TensorFlow/Keras on the Raspberry Pi in the Donkey repo or in this blog post
(needed to actually run the trained model, it looks like). For TensorFlow,
there are pre-built binaries and instructions to build from source here:

[https://github.com/samjabrahams/tensorflow-on-raspberry-
pi](https://github.com/samjabrahams/tensorflow-on-raspberry-pi)

Note: I am the owner of this repo

~~~
wroscoe
Donkey runs a client on the Pi and a remote server that runs Keras/Tensorflow.

~~~
sja
A ha! Very cool- apologies for not seeing how it worked at first; I assumed
you used the server to control/collect data manually, and then loaded the
model onto the device. Thanks for the demo!

------
argonaut
Two major errors: 1) This doesn't seem to be controlling overfitting on the
right validation set. 2) There isn't a test set at all (separate from
validation).

Using Keras' "validation_split" parameter will just randomly select a
validation set. This is not the right thing to do when your data is image
_sequences_ , because you will get essentially identical data in training and
validation.

Because of this, the numbers/plot here might as well be training accuracy
numbers.

~~~
mhanus
Keras uses the end of the data set as validation, and only randomizes it if
the "shuffle" argument is set to True [1].

[1]: [https://keras.io/getting-started/faq/#how-is-the-
validation-...](https://keras.io/getting-started/faq/#how-is-the-validation-
split-computed)

~~~
argonaut
Except the second half of the data is the flipped of the first half (X =
np.concatenate([X, X_flipped]))

~~~
wroscoe
Well shit. Thanks for pointing that out. I'll revise.

------
feelix
Apologies if I'm being stupid, but I can't find the details on how to
physically connect the hardware together anywhere. Is this still on the todo
list? I'm interested in applying this tutorial and making an autonomous RC
car.

------
nojvek
What I would love to see is an end to end neural network soln. On one end
camera input comes through, on the other outputs for speed and steering angle.

But rather than a black box, it's explainable what the different layers are
doing. If neural nets are turing machines then we should be able to compile
some parts of the net from code.

Then the net is a library of layers. Some Layers trained with back prop, some
compiled from code.

~~~
argonaut
Almost all neural nets are not Turing complete. Only very specific RNNs are;
most RNNs aren't, including pretty much any RNN model used in the real world
right now ([https://uclmr.github.io/nampi/talk_slides/grefenstette-
nampi...](https://uclmr.github.io/nampi/talk_slides/grefenstette-nampi.pdf)).

Also, this is a useless fact, because so many other random things are Turing
complete.

------
ipunchghosts
X is in the range 0, 255. They don't show code converting it to a much saner
range for the network they've chosen. Is the full source somewhere?

~~~
cr0sh
Have you already looked at the pickled data? Because it looks like the model
is outputting a single label value out of 256 labels; depending on the
training data (steering angle) and how it is represented in the data (signed
float or integer?), each one of those 256 learned should (?) be similar - I
think.

Again, I'm not an expert. Or - maybe it is outputting a number 0-255, and then
taking that number and converting it (and maybe other operations) into values
suitable for the servo on the car (perhaps centered around 0 - so -128 to 127
or something like that - then scaled for servo PPM width or whatever values
needed)...

All guesses, of course.

~~~
wroscoe
The input values are image arrays 120x160 pixels with 3 channels for
red,green,blue. The values range from 0-255 and are not normalized before they
are fed into the convolution layer. I found this did not make a difference.

The output of the model is a single real number between -90(left) and
90(right). I believe a better approach would be to bin the outputs and use a
classifier. This way you'd know when the model was getting confused (ie,
approaching a perpendicular line.

------
ramshanker
Before even opening the link, I was thinking, which jurisdiction would it be
legal to program "personal" autopilot.

Awesome tutorial.

------
agumonkey
I remember seeing keras in commaai source. Who else uses it ?

~~~
wroscoe
Tensorflow anounced last week that it will incorporate Keras as its default
higher level abstraction. It's a pleasure to use.

