
Self-Crashing Cars - devy
https://www.zachaysan.com/writing/2018-01-17-self-crashing-cars
======
csours
> _" Every single electrical and radiological device can be utilized to
> transmit data. What's the baud of opening and closing HVAC vents and reading
> them from space satellites?"_

[https://www.wired.com/2017/02/malware-sends-stolen-data-
dron...](https://www.wired.com/2017/02/malware-sends-stolen-data-drone-just-
pcs-blinking-led/)

Also reminds me of the stories of reverse engineers using, lights, motors,
speakers, etc to exfiltrate boot images.

> _" If hacking a Jeep is as straightforward as hacking a server, and servers
> are routinely breached, then where are all the hacked cars? It's a bit like
> the Fermi Paradox."_

One barrier to entry is the actual cost of acquiring a vehicle. You can buy a
lot of iPhones to hack on for the price of a Tesla.

> _" We think of Teslas as cars just like we think of an iPhone as a phone,
> but a more accurate account of reality is that they're both just
> computers."_

It's actually kind of worse than that, cars are actually rolling datacenters
with multiple computers and multiple networks. (CAN bus[es], Ethernet,
proprietary networks)

//Disclaimer: I work for GM, but not on any of this.

~~~
hinkley
I know things change over time, but the old advice on hackers was that most of
them are not as malicious as they seem. For instance, many viruses have been
accidentally more damaging than the creator intended.

Hacking a car to kill people is a lot different than kicking someone out of a
video game. Even people who SWAT don't anticipate their rival/victim being
killed.

They want the person to be miserable, not dead. It's cruel and vindictive, but
it's not homicidal. Or at least not most of the time.

~~~
cracell
There's been a number of suspicious reporter deaths with vehicles
malfunctions, possibly due to hack
([https://encrypted.google.com/search?hl=en&q=suspicious%20rep...](https://encrypted.google.com/search?hl=en&q=suspicious%20reporter%20vehicle%20deaths)).

The most recent high profile one being Michael
Hastings([https://en.wikipedia.org/wiki/Michael_Hastings_(journalist)#...](https://en.wikipedia.org/wiki/Michael_Hastings_\(journalist\)#Controversy_over_alleged_foul_play))

Could be paranoid nonsense. But white hat researchers have demonstrated a
disturbing number of attack vectors that could be used for this sort of thing.

~~~
hinkley
State actors have a myriad of options.

I was simply trying to answer the question of why this isn't happening all the
time.

If it is happening all the time (a point I won't dispute) then the question is
wrong.

~~~
jethro_tell
Do we know that it isn't happening all the time? We are already so conditioned
to driving being one of the most dangerous things yet we absolutely take it
for granted. It would be easy enough for it to more or less blend in.
Especially as I'd assume the current traffic investigation work force in the
field isn't going to have the chops to do this kind of work.

This may be happening, and just being written off as user error like all the
rest of the massive number auto accidents we live with everyday.

~~~
hinkley
Alan Turing was doing statistical modeling to make sure we didn’t blow up more
U boats than could be mistaken for bad luck. That was more than seventy years
ago so who knows.

------
bo1024
I think this is a really important issue and glad someone is working on
awareness. It extends far beyond cars but those are probably the most
'frightening' example. We need to

(1) Admit we have a big problem.

(2) Admit it will be difficult to solve and there aren't easy solutions.

(3) Start working on solutions.

Unfortunately, these steps never seem politically popular. But it will be
tough. Except in very rare scenarios (NASA), software security and correctness
aren't treated as important in the scheme of things, relative to functionality
and features. Nobody wants to invest in actually secure, carefully-audited
code. (They might claim they do, but their standards for "secure" and
"careful" would be litigation-worthy in any field but software.)

For example, I believe in freedom of software and ownership rights, which
seems to mean that I believe people should be allowed to run open-source code
on their own cars' computers. But this impacts public safety. How do we
develop reasonable regulations similar to those for physical "street-lega"
modifications?

~~~
sparkie
I have reason to believe that a competent hobbyist programmer is less
dangerous than the electronic engineer, pretend-to-be-programmers who wrote
the original firmware on the vehicle.

Based on years of working with electronic engineers' code. (Apologies to those
of you who are competent in both)

~~~
bo1024
I appreciate your point and am inclined to agree. But my agreement doesn't
matter. Your point is only one starting point for a challenging and nuanced
discussion over the legality and safety of people installing their own
software on their own cars.

My broader point is that this (among others) is a discussion we should already
be having and eventually must have -- if not amongst "ourselves", then with
legislators and laypeople. Yet the state of software security, awareness, and
incentives is such that it could be years or a decade until we do...

------
Animats
Real problem, clueless author.

The big problem is allowing firmware updates.

Many years ago, I knew some of the people involved with the Ford Electronic
Engine Control IV, the "EEC IV", used in 1980s Fords. This used a custom CPU,
an Intel 8061, which was an Intel 8051 with some extra timer features. The
program was _etched onto the CPU silicon_. There is no way to change that
program without replacing the whole unit. There's an external ROM with some
tables, different for each engine model, but it's a true ROM, not something
that can be rewritten. If you want to replace it, it takes a wrench.

The level of paranoia and testing that went into that program was very high.
Any bug meant bringing back hundreds of thousands of Fords in a recall. That
didn't happen, and there are still many EEC IV vehicles running today, 30
years later. That's what it was like in the days of hardcore embedded
programming.

I'd argue that safety critical vehicle software should not be downloadable. If
a fix is needed, the vehicle has to go back to the dealer for a new memory
module. This would discourage "shit early, shit often" software development,
and encourage manufacturers to keep the safety critical systems, like ABS,
totally disconnected from the entertainment system.

 _Autonomous vehicles should not communicate with each other._ Waymo's cars
don't. Most of the schemes for "car to car communication" are motivated by
marketing or surveillance, not driving. Aircraft systems don't communicate
with the ground much, and when they do, the uploaded data is very simple.
(It's common in commercial aircraft to hardware disable maintenance functions
unless the "weight on wheels" switch is on, indicating the plane is on the
ground and parked.)

~~~
reaperducer
My inner curmudgeon is on board with the idea of only allowing important
firmware updates to happen at the dealer. But the reality is that with so many
cars on the road, OTA update is a better way to make cars safer.

In my case, for example, I have a 2014 and a 2015 car which are both more than
a year overdue for a software updates. One is related to the braking system;
the other to a problem where if you stomp on the gas in an emergency (like to
avoid a crash), the car stalls.

The problem is that the dealers in my area don't have the capacity or the will
to do software updates. One dealership only has maintenance hours between 9am
and 4pm Monday through Thursday, and the ONE guy who is allowed to do software
updates is only in on Saturdays from 9-noon occasionally.

The other dealership doesn't take appointments at all. It's first-come, first-
served, 8am-5pm Monday through Saturday. I stopped going there because even at
7am, there's a huge line of people waiting for service. The final straw was
when I sat there from 7am until 3pm for a very simple non-engine part swap
that was covered under warranty.

The next nearest dealer is 138 miles away.

tl;dr version: OTA updates are good because there are places where you can't
get a software update done at the dealer in a reasonable time.

~~~
Bilters
If we’re talking about the self driving cars here like mentioned in the
article, the cars could go to a dealer / workshop on their own. You’ll only
have to tell the car “I don’t need you for the next x hours”. So you go ahead
and have the firmware updated.

Unfortunately there’s no quick fix about the current cars having the “guided”
self driving principle.

~~~
Piskvorrr
Unfortunately, the current climate is quite opposite, what with computers
randomly* deciding "you don't need me for the next couple of hours, I'll try
the install-revert-install-revert-install-revert with this week's update,
NOW."

(randomly, as in "where do you want to go today? meh, nevermind.")

------
schlowmo
While I agree with the author about the undervalued risks of autonomous
vehicles in the light of careless security practices his proposed solution is
just insane.

In my opinion the demand for a government controlled kill switch in every
piece of hardware that is somehow able to harm people is much more threatening
in so many ways than the insecurity the author is trying to reduce. Just a
few:

1\. Based on the asumption, that ones current government/state is for the good
of all people, what gives him the confidence that this will stay that way?
What if your beloved government goes rogue? That's a lot power for an
autocratic regime.

2\. Why even trust the government in the first place with that kill switch?
They are the same people which are careless about infrastructure critical
ITSec since decades.

3\. An univervsal security module which is highly standardized is a very
profitable target. While the author is aware that finding an attack vector of
one particular vehicle can mean that all vehicles of that type can be
compromised, he doesn't come to the conclusion that the same logic applies to
his security module.

~~~
3pt14159
I’m the author. I was nodding in agreement with you for your first two points.
I don’t think I’ve come to a perfect solution, but I want you to know that I
actually have many of the same reservations that you have and that I’m open to
changing my mind.

Here is where I still sit, however: the government had predator drones and
will soon have killbots able to take out individuals based on facial
recognition software.

If we can’t trust them to turn off our cars we’re fucked anyway.

~~~
nickodell
These safety modules sound really complicated.

Let's say a security vulnerability has been discovered. An attacker has wormed
their way into as many cars as they can. They send a 'go' signal from their
command and control servers. Across the world, cars start looking for
opportunities to kill their passengers and bystanders.

We send the shutdown signal. The safety modules wake up and take over from the
compromised computers. What do they do?

They could brake. However, the car might be in a turn, on a rainy day. Braking
could send the car into a skid and kill its passengers.

Maybe they brake, but slowly. However, the car might be behind someone who
just suddenly changed lanes and slammed on the brakes. Braking hard might be
the right move in that circumstance.

Maybe they do something conditional on the sensor input they get. However, if
the control computer can do something to 'blind' the safety computer, that
doesn't help. For example, can the control computer issue firmware updates to
the camera sensors? Can the control computer fill a sensor's bus line until it
can't respond?

I can't think of any solution to this that doesn't involve duplicating a
substantial part of the control computer.

~~~
vntok
1\. Have an emergency break button somewhere in the car that the user could
press at any time to have the vehicle break. Note that we already have those
in all trains, buses etc. 2\. Have a warning light up next to the button
asking the user to press the break button as soon as they feel safe to do so.
Note that we already have those service lights in all regular cars. Simply add
a buzzer as well for people asleep etc. 3\. ‎Have the circuit that breaks upon
button activation be sealed off from the internet (make it hydraulic). The
only action of the "good C&C" is to turn a light on.

------
LinuxBender
Unpopular opinion ahead. Having worked in the security industry for a few
thousand years (computer years), I can say I would never own a car that can
talk to the internet. I plan to move far away from cities very soon for this
and several other reasons. People will argue about this and meanwhile the
"impossible" will happen, repeatedly. I just replaced the engine and
transmission in my non internet vehicle and hope to get another 500k miles.

~~~
deeg
I understand your concern but right now 30k Americans die in car accidents.
Will (potentially) hackable self-driving cars be any more dangerous than
today? If you're that concerned about being involved in a car crash it seems
to me that you should never leave the house.

~~~
LinuxBender
Potentially, yes. Instead of 30k spread across one year, the very possible
opportunity for 30k in one mass-hack exists. I will defer to others to debate
this. Perhaps financial regulators, safety departments, transportation
regulators, insurance companies, etc. This is a very complex topic that would
quickly turn into banter here. Everyone will have to decide for themselves the
risk factors as it pertains to them.

~~~
paulcole
> the very possible opportunity for 30k in one mass-hack exists

It's fortunate that terrorists are both very incompetent and very low in
number. How else do you explain the fact that there's been exactly 1 very
serious and successful foreign terror attack on US soil (the highest value
target in the world) in the past 50-ish years?

Currently there's the very possible opportunity of a power grid/infrastructure
attack that could kill tens of thousands. But nobody should be truly worried
about it.

Terrorists just aren't that good at what they do. Why would they somehow be
better at hacking cars than they are at anything else?

~~~
brokenmachine
Why are you only talking about "foreign" terror attacks?

The Oklahoma bomber killed 168, for example. The first one that came to my
mind.

~~~
paulcole
Well our domestic terrorists aren't that successful or competent either:

[https://en.wikipedia.org/wiki/Terrorism_in_the_United_States](https://en.wikipedia.org/wiki/Terrorism_in_the_United_States)

Most news organizations report on foreign and domestic terror differently.
When the average American thinks of a terrorist it isn't a white guy with a
U-Haul.

------
stcredzero
_Since organizations do not observe an attacker 's failure, the market
generally does not reward extreme competence in cyber defense..._

 _Blackhats, in stark contrast, are sexy._

Bounty hunters are sexy. Why doesn't the government start paying bounties?
Paying bounty hunters to run honeypots would convert many black hats into
hunters of black hats. This could well create an ecosystem where the more
knowledgeable hackers directly prey on the script kiddies for fun and profit.
Taking useful idiots out of the ecosystem strikes me as desirable. Such a
program would also be useful for recruitment.

In a way, this is analogous to such bounties in the transition of the wild
west into a more normal society.

~~~
lucb1e
If you have a rat problem and start paying for rat corpses, people might start
breeding rats.

~~~
stcredzero
The analogy falls down a bit, because while rats are born rats, a hacker is
first a person. People can witness people becoming examples, and decide not to
do that. That said, problems can arise, due to national borders, limited
jurisdictions, and differences in access to socioeconomic resources.

------
vlucas
Easy hack to prevent remote control/hijacking of your car: drive a standard.
It will never be able to start or drive anywhere remotely.

~~~
eric_h
Excellent idea. Too bad it's so much harder to get a standard transmission car
in the US these days.

~~~
barsonme
I'd be curious to see the manual vs automatic accident rates per miles driven.
I have a feeling automatic are a lot higher. I mean, these days, if somebody's
driving a manual either they want to or had no other choice. I wonder if that
correlates with driving ability at all.

~~~
Wehrdo
That would be interesting to look into of one had the data. I would bet
insurance companies have investigated it.

The only evidence I could find was this study:
[http://journals.sagepub.com/doi/abs/10.1177/1087054706288103](http://journals.sagepub.com/doi/abs/10.1177/1087054706288103)

Although I can't access the article, the abstract says that teens with ADHD
had higher attention to driving when using a manual transmission vs an
automatic.

~~~
barsonme
That's funny you mention ADHD. I was diagnosed as a teenager and I've always
felt more focused while driving a manual than an automatic—it's not as easy to
zone out. I've also heard of doctors "prescribing" manuals for people with
ADHD. That's just second-hand, though.

------
bufferoverflow
Cars are different though, they aren't general-purpose computers (even if they
are based on them). So you can have a chip that checks the checksum of every
file, checks if it's all signed with the correct certificate, and shuts down
the car even if one bit is off.

That's quite different from your laptop or a smartphone, which can run pretty
much any code.

It would be very very very hard to hack something like that. As in, steal the
signing keys from Tesla hard without being noticed. I wouldn't be surprised if
their signing server is air-gapped.

~~~
beat
Cars are also different in that software updates are harder to pull off. There
are millions of cars out there on the road _right now_ that have internet
connections, and only minimal security between human-convenience internet like
map software, and the embedded systems that do things like steering and
brakes. And the average lifespan of a car is what, 15 years? Whatever the
vulnerabilities, they're going to be there for a long time.

And even if there's a firmware update mechanism available, the manufacturers'
abilities to maintain older software will also degrade, like legacy systems do
everywhere.

------
vermilingua
> What I learned was that there were not only no regulations there were no
> plans for them either. While I think my MP took my concerns seriously and
> did what she could, I came to understand that political will lags public
> outcry

This interested me more than the idea of car-bombs, are we moving towards a
post-law society? Thinking about it, I haven't seen a single piece of
legislation regarding self driving cars, cryptocurrencies, or shared computing
in my country; and are transport, currency, and communication not the
underpinnings of civilisation?

------
0xCMP
I looked and noticed this post has been submitted several times (not a
complaint):
[https://news.ycombinator.com/from?site=zachaysan.com](https://news.ycombinator.com/from?site=zachaysan.com)

Then I noticed he'd recently published an article on Zero Width characters
being used for fingerprinting which got some attention on HN in the last 30
days:
[https://news.ycombinator.com/item?id=16046329](https://news.ycombinator.com/item?id=16046329)

------
olivermarks
A witness to Michael Hasting's 2013 'car crash' describes what he saw
[https://youtu.be/fweyFCFKcp0](https://youtu.be/fweyFCFKcp0)

~~~
olivermarks
DARPA discussion [https://youtu.be/6OfcgJ-pl7Q](https://youtu.be/6OfcgJ-pl7Q)

------
anonu
tldr: Cyber attack is 1000x easier than defense. There are enough Tesla cars
on the road today that can be hacked into and turned into WMDs. Lots of ideas
added in on how to bolster defense.

~~~
Zigurd
Car bombs are bad, but they are not "WMDs." Nevermind that it is cheaper to
find a suicidal driver, or trick a driver into going on a suicide mission than
it is to buy a Tesla and hack it into a car bomb.

Many imagined terrorist threats are threats only in a vacuum. A rifle and a
tall building resulted in 50+ deaths and 500+ wounded. Any technologically
complex attack has to beat those numbers.

The only notable attack with an actual WMD was the sarin gas attack on the
Tokyo subway. It killed about one fourth the number of people as a man with a
rifle in Las Vegas. It took a secretive cult organization to implement. That's
why nobody uses WMDs for terror attacks.

Trucks and rifles do enough damage without even having to acquire bomb-making
skills. Autonomous vehicles that are programmed to avoid hitting pedestrians
are likely to make it harder to use vehicles as a weapon, not more deadly.

~~~
bzbarsky
The article is not talking about car bombs. It's talking about attacks like
"make every car of model X that's currently on the road accelerate to 120mph
and then ram the nearest wall". This would kill a number of people on the
order of magnitude of the number of cars of model X in existence (assume each
one only kills one person, the driver), but even that can easily be on the
order of hundreds of thousands to millions. For example, according to
[https://en.wikipedia.org/wiki/Toyota_Corolla](https://en.wikipedia.org/wiki/Toyota_Corolla)
there were 40 million Corolla's sold over the span of 47 years, which means
there are certainly model years out there with close to a million vehicles
sold.

~~~
Zigurd
More than 250 million Windows PCs are sold every year and the installed base
is over 1.5 billion. The largest botnets number in low single digit millions,
or less than 0.5%.

~~~
bzbarsky
This is an interesting data point, and I agree that it's hard to explain. Why
doesn't a newly-discovered 0-day in Windows lead to pretty much every
internet-connected Windows PC being infected?

(That said, there are issues around actually reaching these machines, because
a lot of them don't have routable IPs and may not load your attack web page.
Similar issues may arise for cars, which sure would be nice as a mitigation.)

Even so, 0.5% of 1 million is about 5000 people. Not quite WMD territory, but
also way out of typical "car bomb" territory... Maybe the mitigating factors
would mean the actual fraction would be even less. But it would be nice if we
didn't have to hope so.

~~~
Zigurd
Human drivers kill about a million people, worldwide, every year

~~~
bzbarsky
Yes, I am aware of that. One major goal of autonomous vehicles it to reduce
that number.

------
matthewowen
I know this is somewhat off topic, but what's the deal with "Look no further
than the Clinton email breach to see how much a single hacker can change the
world."?

To my knowledge, there's no evidence that Clinton's email was breached. The
DNC, sure, but the Clinton email controversy wasn't based on any actual known
breach. It's sort of alarming to see this weird retconning of history.

------
hueving
>because obscurity is a valid defensive measure. It's how passwords and
authorization tokens work.

This is extremely dangerous thinking. Passwords and tokens are not in the
'security by obscurity' category because you can observe how the entire system
works, review its source code, read its deployment configuration, and do
packet captures of the encrypted valid traffic and still not have a way to
gain access to the system.

Security by obscurity refers to hiding how the system actually works, and
that's a lot harder to keep a secret because you are one compromised device
away from revealing all of that and it can't be easily changed.

Do not call passwords and tokens "security by obscurity" or claim security by
obscurity practices (e.g. Running on non-standard port numbers) is on the same
level as passwords/tokens/keys.

There is a reason the strongest crypto is using public algorithms and only
private keys. Obscuring a system just means that your silly bugs don't get
found and exposed early.

~~~
Piskvorrr
Not a very strong measure, definitely not against a determined attacker, but
valid nevertheless. A mosquito net for opsec, so to speak.

------
mtgx
> _They rebutted that individual cars are much easier to hack and after they
> are first used in a terror attack we will get the political will to fix the
> problem._

Was that meant to be a joke? Sure, they can establish a new standard that will
apply to all the new car models coming out four years later. But who actually
expects hundreds of millions of cars that are already on the market, to
receive a software overall with a new architecture?

This is why it has always bothered me that almost no one seems to bring this
problem to the forefront - certainly not carmakers. They're all too focused on
how awesome self-driving technology will be and how it will save us from drunk
drivers. Thus, disregarding the fact that once we have 100 million to 2
billion self-driving cars on the road, that will be a _huge_ market for cyber
criminals, from ransomware and cryptojacking (hello powerful GPU computer +
free solar power charging!) to assassinations.

And before anyone says "how much harder it is to hack a car than a PC",
consider the fact that most cars today aren't actually connected to the
internet. And most of those that are, only have their entertainment systems
connected to the internet. Self-driving cars will be able to receive OTA
updates that will _improve their engine, steering, and brake performance_ =
the OTA software has access to _everything_.

Combine this level of access to the high level of recklessness in the name of
profits carmakers seem to be showing today, when they advertise features such
as "unlocking your doors through an app".

EFF's former chair and someone who worked on Google's Waymo, has some decent
ideas about how to protect self-driving cars, if only carmakers would listen:

[http://ideas.4brad.com/disconnected-car-right-security-
plan-...](http://ideas.4brad.com/disconnected-car-right-security-plan-
robocars)

~~~
3pt14159
Airgapping was my first instinct too, but the problem is we're dealing with
state-level actors. Airgapping doesn't work with them. They're patient, well
funded organizations. Trying to rely on never having a single type of car (any
of which could have a hundred thousand copies on the road) hacked is a fools
errand.

We need ways of disabling autonomous devices and detecting when they get
hacked, not trying to win an impossibly hard game.

~~~
bradtemp
The disabling system becomes another attack surface, and unless it is pretty
independent, is itself disabled by a sophisticated attack. But scared as we
are of external attack, allowing the government to shut off all cars is like
letting Mubarak shut off the internet in Egypt. That's a bigger danger than
foreign enemies in many countries.

------
deevolution
The human mind is also hackable... it just takes a considerably longer time to
hack it compared to installing software on a computer. Id argue that the
internet and social media has increased the speed at which the mind can be
hacked by a malicious actor. Im not so sure which is worse now, autonomous
vehicles or human drivers? I think its much harder to detect when a human has
been "hacked" than a fleet of cars. Cars could have subsystems build in for
detecting anomolies, axuilary computers and manual overdrive settings.
Detecting when a human has been "hacked" requires invasion of privacy,
constant surveillance, reliable friends and family, and a whole bunch of other
variables. Preventing a human from being hacked might require arduous and
expensive changes and experiments in legal systems, incentive mechanism,
censorship, education,etc...

~~~
deevolution
There's going to be casualties no matter what... that seems to be the nature
of technological advancement.

------
dfabulich
> _One of the problems I 've had over the past year and a half is how to
> communicate this idea without:_

> _1\. Sounding like a crank._

> _2\. Giving ideas to terrorists or hostile foreign governments._

#1 is this article's biggest problem. These problems are real, but I think the
author sounds like a crank. (Based on the way this article is written, I
suspect that the author actually is a crank--obsessed with security, but
fundamentally lacking in the relevant skills to do anything about it.)

If the author is serious about this, here's what you do.

1\. Become/join a non-profit organization. You want to be frequently quoted in
the press, but not as "well-known software security expert Zach Aysan" but as
"Zach Aysan, president of the Organization for Global Security."

2\. Build your reputation by finding and earning credit for security problems.
Do ethical reporting, but when the issues get fixed, exhibit them in a flashy
way.

3\. White papers, not blog posts. This "article" starts with a subtitle "with
apologies to Elon Musk", followed by a personal dedication to Zach's father.
This is not the tone of a white paper from a think tank.

The subtitle of the post should be the thesis statement of the article. "Self-
driving cars lack adequate security protections." The first paragraph should
be an executive summary of the argument of the post. Each paragraph should
support the thesis statement.

Have someone read your articles; update your work based on their feedback.
Thank them in the footer.

4\. Separate "how to" articles from arguments. This article is long because it
is both attempting to persuade the reader that we haven't invested enough
effort into securing critical systems and also to give a list of proposals.
These should be separate articles, with one article arguing why security is
important, and another article giving a list of proposals.

~~~
cornholio
> Separate "how to" articles from arguments. This article is long because it
> is both attempting to persuade the reader that we haven't invested enough
> effort into securing critical systems and also to give a list of proposals.

The solutions are also very very wrong, he proposes a state enforced kill
switch for every device capable of autonomous operation. Abusing that system
seems like a much greater source of havoc than individual compromises of
certain brands.

His main point is pretty insightful and well argued, giving life and death
powers to very fragile software on a massive scale will end in disaster. The
computer industry is woefully unprepared to deliver hardware, software and
systems with the provable integrity many of these applications require.

~~~
deevolution
The kill switch should simply be launching a rocket or emp at the infected
autonomous vehicle.

~~~
EamonnMR
The passengers would probably prefer a less violent approach.

------
woolvalley
Defense is harder than attack, but the attackers can be attacked back. Jail
isn't worth a few million dollars vs. working in silicon valley.

I think that is also a big reason why many potential black hats do not become
black hats. The only 'sustainable' black hats are ones associated with a
criminal organization or nation states.

------
skywhopper
I don't think the author is wrong about the threat. Unfortunately, his
apparent solution--a government mandated standardized "safety module" to
intermediate the Internet-connected bits of the vehicle from the non-connected
bits--is likely even more risky. Monocultures can be devastated by discovering
a single flaw. And there are guaranteed to be many flaws in any non-trivial
computer system. Imagine if a straightforward exploit for Spectre that totally
compromised a system leaving a hard-to-detect back door via some simple
Javascript had existed on January 1, and had been injected into a few popular
websites via ad networks... for that matter, how do we know such a thing
didn't happen?

------
Klasiaster
First step would be to stop public funding of 0day purchase and development in
NSA and co. People believe that more 'cyberweapons' would be good against
cyberthreats from 'others' but meanwhile all use the same systems. If these
systems do not get fixed because the NSA wants to use its purchased 0days then
everybody has a problem. What is needed is joint effort for secure open source
solutions.

Also, for computer security programmers need to overcome convenience practices
and start behaving responsible, but also people in management/politics should
not make technical decisions if they can't understand the implications.

------
icefox
> It's a bit like the Fermi Paradox.

During the late 90's and early 00's I often wondered why with all of the
Microsoft hate why someone didn't create a simple virus that did something
destructive like just format windows hard drives. There must have been other
incentives at play that caused this to never occur. Perhaps in the same logic
if you can infect every Tesla it is more valuable to not crash them, but
instead scrub the data and sell that back to [insert company] or something.

If there was ever something that would cause a formal programming guild to
sprout I would be willing to bet that it would form its roots around security.

~~~
sp332
There were plenty of destructive viruses. Some (e.g. CIH) would erase the MBR
of the disk making it unbootable. Others like ILOVEYOU would overwrite user's
data directly. Blaster specifically had a message about Windows' poor security
and tried to DDoS Windows Update.

Sometimes viruses are written to by self-limiting. MyDoom, which caused about
10% of all email traffic for a time, was programmed to deactivate on a certain
date. Also once viruses get to a certain level of infamy they get a lot of
attention. Blaster was mitigated in just days, so by the time it started its
DDoS it was already mostly wiped out.

~~~
marcosdumay
Even more relevant, Internet access wasn't common back then, and the only
large-range virus vector was the slow paced sneaker-net.

------
antirez
Still cars are already computers but so far nothing like that is happening...
In theory the fact of being autonomous should be a protection because you
could add some control system that can't be updated and can make choices
reagreless of what the other systems are telling to do.

------
whiddershins
Am I the only one who thinks it is ironic that Elon Musk gives so much money
to research regarding the existential threat of AI to humanity, while also
basing a business on putting AI in devices that can so easily kill humans?

~~~
daveguy
Pretty sure he has a guilty conscience. He wants to build an autonomous self
driving car, which will have to get pretty close to general AI and he wants to
put it in multi-ton robots. He knows this could be dangerous and scary. At
least he is giving money to research the problem rather than just forging
ahead with the potentially dangerous part.

Personally I think the robot uprising concern is at least 50 years premature.
But the "what if someone could take the controls" concern is a concern we
should have been already been considering yesterday.

~~~
Veedrac
The fact you said "50" and not "5000" is enough reason to _not_ consider it
premature.

------
King-Aaron
With the continued reporting of these possible attack vectors, it makes me
feel that there's a lot to be said about having a carburettor and points
running your car, instead of a computer with network connectivity.

------
bdamm
Some solutions are strangely misguided. UDP is insecure but TCP is somehow
magically secure? Certificates are not to be trusted? Can't agree with all the
conclusions but the specific paranoia is well founded.

------
kenning
> This article is geared towards people with a STEM background. For something
> shorter try this article in The Weekly Standard.

From personal experience I can tell you that people with STEM backgrounds can
also be too impatient to read an article this long.

edit:

> (Bio) After years of building startups and advising organizations large and
> small on data science and cyber security, I'm turning my focus to improving
> public literacy on technology.

Starting your article like this is a bad way to do it.

~~~
dcow
You may have a point or two, but you're being downvoted because of the
tactless way you've stated them. Also since when is a longer read only for
"STEM people"?

edit: Fooled me, looks like it's been corrected.

~~~
kenning
Yeah its the first line in the article

I imagine (some of) the downvotes are from people with a STEM background that
didn't even open the link

~~~
brokenmachine
Thus proving he was correct, lol!

------
carapace
FWIW, Open Kernel Labs was acquired by General Dynamics in September 2012.

