Cool project! Last year I built self driving robot for my bachelor's thesis. Instead of building end-to-end deep learning pipeline I used two neural nets: one trained with genetic algorithm to drive a robot based on ultrasonic sensors, another for object recognition and detection. Based on detected items (like road signs) robot took different actions.
Nice project and cool video! (our video is not so well produced :P). Our project was a summer project, we did it in this January. The evolution of the project will be something like yours, we didn't apply any object detection technique.
Video: https://www.youtube.com/watch?v=cUXh7iP3hoQ Code: https://github.com/kazepilot