
Watch Zoox’s autonomous car drive around San Francisco for an hour - pcshah1996
https://venturebeat.com/2020/04/17/watch-zooxs-autonomous-car-drive-around-san-francisco-for-an-hour/
======
Animats
Interesting. Watch at 1/3 speed or so to see it in real time. (Self-driving
car videos tend to be published sped up, so you don't see the mistakes.)

The key part of this is, how well does it box everything in the environment?
That's the first level of data reduction and the one that determines whether
the vehicle hits things. It's doing OK. It's not perfect; it often misses
short objects, such as dogs, backpacks on the sidewalk, and once a small child
in a group about to cross a street. Fireplugs seem to be misclassified as
people frequently. Fixed obstacles are represented as many rectangular blocks,
which is fine, and it doesn't seem to be missing important ones. No potholes
seen; not clear how well it profiles the pavement. This part of the system is
mostly LIDAR and geometry, with a bit of classifier. Again, this is the part
of the system essential to not hitting stuff.

This is a reasonable approach. Looks like Google's video from 2017. It's way
better than the "dump the video into a neural net and get out steering
commands" approach, or the "lane following plus anti-rear-ending, and pretend
it's self driving" approach, or the 2D view plane boxing seen from some of the
early systems.

Predicting what other road users are going to do is the next step. Once you
have the world boxed, you're working with a manageable amount of data. A lot
of what happens is still determined by geometry. Can a bike fit in that space?
Can the car that's backing up get into the parking space without being
obstructed by our vehicle? Those are geometry questions.

Only after that does guessing about human intent really become an issue.

~~~
emmelaich
> Watch at 1/3 speed or so to see it in real time...

The note in the top-right says its 2x.

~~~
harshulpandav
Top-left

------
carapace
It really _really_ bothers me that these folks are using a live city with
real, non-volunteer test subjects of all ages (little kids and old folks use
public streets) as a test bed for their massive car-shaped robots.

It's bad enough that people are driving cars all over the place, car
collisions have killed more Americans than all the wars we've fought put
together.

I'm one of those people who say, "Self-driving cars can't happen soon enough."
But I don't think that justifies e.g. killing Elaine Herzberg.

Ask yourself this, why start with _cars?_ Why not make a self-driving golf
cart? Make it out of nerf (soft foam) and program it to never go so fast that
it can't brake in time to prevent collision.

Testing these heavy, fast, buggy robots in crowds of people is extremely
irresponsible.

~~~
KKKKkkkk1
I think you're missing the narrative that the self-driving industry is pushing
here. They "solved the problem" and their fleets driving around "autonomously"
is being done in order to demonstrate this to the public. A golf cart is
obviously unsuitable for that purpose.

I think this narrative has run out of steam at this point, by the way. Waymo's
valuation has gone from $175B to $105B to $30B since 2018. Zoox specifically
is now laying off engineers.

~~~
yalogin
Wait, Zoox was valued at 175B at one point? Is this true? Can you point me to
any resources on this?

~~~
Xorlev
GP said:

> Waymo's valuation has gone from $175B to $105B to $30B since 2018. Zoox
> specifically is now laying off engineers.

Waymo.

------
xiaolingxiao
What is the general view on Zoox's progress relative to other non-waymo
playes. Such as Argo, Aurora and Cruise. There is the widely reported
disengagement per mile, but most robotics people know it is just smoke and
mirrors meant to make the regulators go away (disclosure, studied/researched
robotics in grad school).

~~~
IntenseChaos
The general consensus among my AV friends (who work at a bunch of different
companies) is that their AV driving stack is really good, but obviously not
perfect.

I have no idea about their business model and how COVID affects that, though.

~~~
Fricken
Relative to competitors Zoox's automous OS is doing quite well and doesn't get
enough respect. Relative to the objective everybody is fucked.

~~~
xiaolingxiao
Could you provide more context on the first part.

~~~
Fricken
They've been keeping abreast.

The co-founder, Tim Kentley Klay was somehow able to get Jesse Levinson on
board, and Jesse Levinson had no problem getting infinite street cred on
board. So they were able to attract a lot of key, original robotics talent
before the hype got out of control.

For a long time though, they were low on funds, so they did lots of closed
course testing, and it wasn't until they closed a large funding round that
Zoox began on public roads, and they performed quite well right out of the
starting gate.

Now Zoox and it's competitors are lost in an endless wasteland of testing,
development, and validation. It's futile to attempt to do a comprehensive
analysis between the different players, they all have their quirks, but Zoox
has built all the critical infrastructure needed to do full scale testing, and
they're eyeballs deep in it like everyone else.

However, Zoox has stormy waters ahead financially. They need another $2
billion to stay abreast in this never ending race. It's getting harder to
visualize scenarios where that happens.

What nobody can do well enough to build a competitive and scalable robotaxi
service is prediction in multi-agent scenarios. The AI for that just doesn't
exist.

~~~
matdehaast
Thanks for the info. Can you clarify what predictions in multi agent scenarios
are?

~~~
Fricken
Imagine an uncontrolled intersection. The Robotaxi is approaching from one
direction. In the opposite direction is a cyclist who intends to turn left
across the Robotaxi. There is also a pedestrian that may or may not cross the
street, and another vehicle about to cross in front of the Robotaxi from the
other direction. There are a huge number of ways this scenario can play out,
and any decision made by one agent can affect the behaviors of all the others,
compounding it's complexity. Humans can game out these situations intuitively,
but current AI cannot read deep enough into the matrix to deal with these
situations quickly and reliably.

~~~
TheSmiddy
In Australia there are no uncontrolled intersections (that I am aware of).
Every single junction clearly marks who must give way and we don't have any
4-way stops, instead using roundabouts in these situations.

It's possible that for self driving to work road systems will have to be more
formalised to remove the ambiguous situations you've described. I can't
imagine it working well in China or Indonesia where traffic flows much more
like water in a stream and lanes are merely just suggestions.

~~~
taneq
There are definitely uncontrolled intersections once you get out of the
cities. My understanding is that if unmarked, there’s an implied give-way at
the side road in a T-junction and all roads in a four-way junction.

~~~
fratlas
I imagine self driving cars won't be outside of cities for years and years.

~~~
ghaff
Outside of cities on limited access highways seem like the much easier
situation--and, frankly, a pretty significant win for both comfort and safety
once people give up their dream of having a personal chauffeur for their
entire lives. It's self-driving in e.g. Manhattan or Boston that I can't
really begin to imagine in less than decades.

------
vardump
Yawn. Good lane markings, no rain/snow or other bad weather, perfect road
surfaces.

Just like all other self-driving demos. I'd like to see a demo like this on
snow covered roads, with no lane markings visible. I think that would tell a
lot more about the system's ability to deal with an imperfect world.

~~~
chrisseaton
> I'd like to see a demo like this on snow covered roads, with no lane
> markings visible.

But humans can't drive well in those situations either. Why are you asking for
something better than humans can do?

~~~
draugadrotten
I was once driving on a road I could not see at all. It was at night, in a
blizzard on the road from Denver to Vail. It didn't take long until I was
following the two red lights of the bus in front of me. As a human, I knew I
could drive safely where the bus had been driving seconds ago. A self-driving
car would have... tell me.

~~~
rossjudson
Pulled over, like you should have done.

I've been exactly where you were, driving the Coq highway in British Columbia
at night in a blizzard, following two red dots in front of me. I had
(mandatory) snow tires on a rear wheel drive BMW. I also had my family in the
car.

It was probably the single stupidest thing I've ever done driving a car.

~~~
soared
You can’t pull over in that situation unless you want to get your car stuck,
be stuck in the cold all night, and potentially rear ended.

------
dehrmann
Watching this, it's so frustrating that we're 95-99% there on autonomous
driving.

~~~
phkahler
Remember that we've been at that level with voice recognition since the end of
the last millennium.

~~~
dnautics
No, voice recognition got markedly better in 2015-2016. Now, I regularly to
tasks with vr, which was not really a thing in 1999.

------
lolc
All in all I'm quite impressed with the demonstration. It was way more
thorough than previous videos I've seen. The main things the car is failing at
from what I see are the hard things: Object permanence and ad-hoc reasoning.
So no surprises.

Regarding object permanence: I was impressed overall with their detection.
Still, you could see kids walking close to parents blink in and out of
awareness of the car. Now I'm not saying humans are very good at tracking a
multitude of actors. So at some point the machines will be "good enough". But
that point seems way off when significant objects like kids can just disappear
from awareness when they pass behind a stroller.

And about the ad-hoc reasoning: They have the whole city mapped out! Including
traffic lights and turn restrictions. I'm not even clear whether they try to
detect the signs at all. I'd assume that they have an operations center that
hot-patches the map with everything cropping up during the day. So the cars
would send in unexpected changes to the road and they would classify those
changes and patch the map. Meaning the car is tethered to that feed and not
autonomous in the strictest sense. Sure, such a center would be marginal cost
given a large enough fleet. Still it's a subscription you'd need for your own
robocar.

They mention a lot of things they are prepared for. And I can't help but think
"oh they're really good" when they say "detect backed up lanes" or "creep into
intersections". But that always leaves the question what happens when they're
not prepared for something. When the rules don't fit. Can the car go over a
curb if the situation warrants it? Does it back out of a blocked off section?
Is it even able to weigh whether backing out is an option at this point?

so I'd like to see a "what we're currently stuck at" video. But I understand
one can't very well attract investors with such a video.

~~~
idavidrein
I agree with a significant amount of your point, but with regard to object
permanence, I would guess that they have prediction algorithms that don't only
rely on the current-time perception, so if something blips out of sight for a
second the system will still infer/predict it's existence (for a time -
obviously if something is hidden for a long time it won't continue to not
trust perception).

~~~
lolc
I'd be very interested to know how that works. But I don't think they have it.

The boxes they draw are very wobbly and dimensions expand and contract
directly with sensor input. Maybe they only show fused output (in itself an
achievement) and there is a later step they don't show. That would be weird
though because if they want to brag about their model they would definitely
show it if it was any good.

~~~
idavidrein
that's a fair point, but it seems reasonable to me that they would separate
the sensory input and the predictive/higher-level aspects of their modeling.
For example, we know for a fact that they must be doing tons of prediction for
both cars and people, so I think it's likely that different models (not
sensory) have the info that a person is probably still there.

~~~
lolc
Yes it's true they must have some form of persistence when they do
predictions. But expected trajectory of other vehicles and pedestrians was
missing from their video. A lot of other interesting feeds were missing too,
so I don't know what to read into it. I tend to think that that stuff would
look much worse. But maybe they just didn't want to clutter the video or show
how advanced they are already.

~~~
idavidrein
yeah, it's possible that the stuff doesn't look very good, but my guess (maybe
my hope?) is that it's too cluttering or through careful analysis could reveal
IP about their predictive algos

------
kevin_thibedeau
> Handling yellow lights properly, involves us having to predict how long they
> will remain yellow for

No. That isn't how yellow lights work in the US. If the light turns yellow and
you have enough space/time to make a safe stop you do it. There's no need to
predict the remaining time on yellow phase. We don't need robot cars bending
these rules.

~~~
travisporter
Not sure why you're being downvoted, but I think this is a classic example of
why self-driving is so hard. They're not bending the rules, just copying what
humans do. We also predict how long a light will be yellow for, but do it
naturally (if you just saw it turn from green, or it was yellow as soon as it
was in your line of sight).

------
NightlyDev
This is really cool, but the environment is also really simple and I think
we're definetly at least 15+ years out before self driving cars can handle
somewhat challenging situations as well as humans.

Just try to put one of this vehicles in a situation with varying road width,
no markings, snow with no sticks to mark the edges so you really have to pay
attention to where the road actually is. What would this do if you meet a car
on such a road? Try to figure out who should go back, and maybe go back to the
latest plase where its wide enough? Do random tests to check for grip every
now and then? It also needs to know whether the road is salted, understand if
the salt is working and so on and on and on...

~~~
vladislav
"we're definetly at least 15+ years out". Similar statements were made about
Go the year it was solved. AV is a vastly harder problem and requires new
techniques to get there, but AI can progress can happen any time.

------
m0zg
Such videos don't get published just because. They're either looking for more
funding or for an acquisition. Which is it?

~~~
vladislav
beggars can't be choosers

------
baby
This is super cool! I'm wondering how the car would react if:

* someone parked on the side opens their door too quickly and collide with the zoox car.

* there is a car not moving in front, and the zoox car cannot see what's in the other lane without backing to get more insight

I'm also super impressed at how it can understand where the lane is in this 5
lane intersection that crosses a tram line. Even I couldn't understand where I
would have had to drive!

~~~
wutbrodo
>I'm also super impressed at how it can understand where the lane is in this 5
lane intersection that crosses a tram line. Even I couldn't understand where I
would have had to drive!

This is actually one of those things that's easier for an AV than a human
since they have localization and full lane maps of the city.

------
yellowapple
The two turns (one left and one right-on-red) leading up to getting to Market
Street in the latter half of the video struck me as odd; the left turn looked
like a bit of a lane sweep, and the right-on-red looked dubious (is it legal
to turn right on red if you're not in the far-right lane?).

SF intersections are hard, though, and the computer seemed to handle them
about as well as I would've.

~~~
lolc
One thing they only mentioned casually towards the end is that they mapped the
city beforehand. So the car is starting from a position where it knows all the
intersections.

------
jasonv
I see these Highlanders (I drive the same model) parked in their lots in FiDi
almost every day. Glad to see what they’re up to.

------
Krasnol
I really appreciate the calm background music.

I think background music is important. Especially on such long explanatory
videos but often it becomes a reason for me to turn off a video if the music
becomes to aggressive.

------
dmitriid
Besides the sheer complexity of situations described in this video, I wonder
how these vehicles will deal with differences in traffic rules in different
countries (when even road signs can be different).

~~~
yellowapple
It sounds like it currently "cheats" a bit by already having driving rules,
maps (including signs), etc. baked in; it'd be akin to a human driver
memorizing the California Vehicle Code and a map of San Francisco word-for-
word and lane-for-lane.

Presumably Zoox deployments in other cities would work similarly, "cheating"
by baking in local driving rules and road maps. A consumer-owned self-driving
car would likely be able to do something similar by downloading the local
ruleset and maps on the fly, assuming one exists.

------
databus
Is there a way to know that this isn’t done with remote control, other than
the company says so?

~~~
chrisseaton
If you think people are just going to simply lie to you then how do you ever
get anything out of reading things on the internet?

~~~
yellowapple
By getting multiple opinions, like what the GP is presumably doing by asking
such a question on a forum like Hacker News.

------
stefan_
Casually starting the turn and not yielding to pedestrians at 10:21.

Companies actually put this kind of footage up without ever reviewing it?

~~~
hrishid
Are you referring to the pedestrian who's almost crossed the crosswalk on the
left side of the screen? This is still a proper yield as far as I can see. The
car just enters the intersection before that person has finished crossing.

~~~
stefan_
That's already the problem. Don't enter the intersection if you can not
speedily finish your turn. There is also already another ped on collision
course the moment they start moving forward.

~~~
dllu
While technically right, you'll never be able to get anywhere in a big city if
you drive like that.

~~~
cmendel
Immaterial. I don't want self driving cars driving like Bostonians or worse,
New yorkers. Self driving cars need to follow the law, drive defensively, and
be conservative. If that means that they take 10 additional minutes to get to
their destination so what? The alternative is that the car kills someone
because of impatience, that is to say an improperly weighted time value
function, which I would think no one wants.

------
vladislav
This demo is not informative as to the readiness for scalable L4 deployment,
for which it would be necessary to focus on the breadth/accuracy of perception
features under the hood of intent prediction and what happens at the tail end
with arbitrary situations that occur in urban driving environments.

~~~
anonymous_car

        This demo is not informative as to the readiness for scalable L4 deployment
    

does anyone make that claim?

~~~
vladislav
Presenting a subset of the information to let the uninformed jump to favorable
conclusions for the presenter is not a new marketing strategy. If there's no
indication about the true level of progress, what is the purpose of the demo?

~~~
chrisseaton
> to let the uninformed

The uniformed don't know what 'scalable L4 deployment' is, so they can't jump
to that conclusion.

~~~
yellowapple
No, but they're familiar with the definition (start-to-finish entirely
autonomous trip under somewhat-controlled driving conditions) even if they
don't necessarily know the lingo to describe it. Being able to get from point
A to point B without human intervention is what people expect when they hear
"self-driving car", and the video does little (if anything) to temper that
expectation (perhaps because it truly is ready for L4 deployment, or perhaps
because it's all smoke and mirrors).

~~~
chrisseaton
> under somewhat-controlled driving conditions

I don't even know what this means, so I doubt the uninformed know the
definition.

~~~
yellowapple
Meaning one can see the road, chiefly.

------
netsharc
Cheap criticism: the video starts with (I paraphrase) "This is 1 hour of
driving", the last thing I expected after the fade-out/in was to see a man
with a weird shirt... and then I notice the video is about 27 minutes long.

Edit to add: After that I started watching it, it's actually a video of an
impressive AI.

~~~
ygra
It's played back at twice the speed. Apparent from pedestrians waking twice as
fast, and if course, the 2x speed indicator in the upper left.

