
Complex Car Software Becomes the Weak Spot Under the Hood - rectang
http://www.nytimes.com/2015/09/27/business/complex-car-software-becomes-the-weak-spot-under-the-hood.html
======
ihsw
ECUs have been notoriously badly programmed for _years_ , they're usually
contracted out rather than built in-house.

Across wide swaths of Fortune 500 companies, they abhor the idea of having
tech people on staff (for reasons I will not go into here) and _just now_ it's
starting to bite them in the ass _hard_.

Proper discipline and established standards are, well, not established, so we
get companies serving up product experiences to hundreds of millions of
individuals where they are held together by duct tape and super glue (with
varying degrees of quality but generally leaning towards overengineered+badly
implemented).

That said, there are _some_ standards in place for these devices, but
regulation is lax and punitive action rarely taken (it's an honor system).

Compensatory and reactive costs are going to continue to rise as the steady
flow of bad code borne from incompetence continues to accrue at exponential
rates as more and more companies need tech built but aren't willing to expend
the resources for it to be built properly.

Thankfully, as mentioned in the article, some auto companies are starting to
embrace the idea of maintaining a healthy staff of developers in-house. Tesla,
notably, has over-the-air updates pushed to vehicles directly rather than
requiring owners to bring their cars to the dealer.

~~~
titomc
Yes you are absolutely right. The connected car project which I worked for had
ECUs outsourced. The car manufacturer said "hey we are a car company". You
deal with the technicalities of ECUs giving you proper data.

The ECUs were giving me frames with bits turned on at the wrong places. We
used to call the telematics hardware/software provider which gives us an
abstracted API to talk to ECUs quoting the problem. Their reply was "thats the
ECUs messing up, we will "fix it" by giving you the bit turned on at the
correct position or you change your code, because changing the API needs an
OTA and OTA is not supported for that car manufacturer.

~~~
ethbro
Which I think is symptomatic of the bigger issue. Car companies look at
anything that goes into a car as a physical part, and they've gotten
extraordinarily efficient (through ruthless competition) at process for
dealing with physical parts.

Except... anything that contains code has orders of magnitude more test
dimensions than a typical physical part.

I would probably stop driving altogether if I knew exact figures on how many
auto manufacturers even have in-house test coverage of their outsourced ECU /
microcontrollers.

------
devingoldfish
From the article: "New high-end cars are among the most sophisticated machines
on the planet, containing 100 million or more lines of code."

Does anyone know where this number comes from and if its realistic? It's hard
for me to imagine unless you're counting all the LOC from Microsoft Windows:
Car Edition or something.

~~~
radiorental
Can't answer where they came up with that number but it does seem rather
larger. The only comparable machine where we do know the number is the Mars
Curiosity Rover running somewhere around 2.5 Million LoC. Now, given that
there are upwards of 200 micro controllers in a luxury car, I can see that
number growing exponentially, but 100 Million???

~~~
TeMPOraL
2.5 MLoC sounds extremely high even for a Mars Rover. I really wonder how it's
counted? Is it C code or ASM instructions? Does it include all standard
libraries that get referenced, even if they end up not being compiled into the
final binary? Or is everything just written in Java and configured with XML?

~~~
ethbro
_> 2.5 MLoC sounds extremely high even for a Mars Rover... Or is everything
just written in Java and configured with XML?_

Highly doubtful, Opportunity is at 4,000+ days of uptime...

------
radiorental
"One option for making auto software safer is to open it to public scrutiny.
While this might sound counterintuitive, some experts say that if automakers
were forced to open up their source code, many interested people — including
coding experts and academics — could search for bugs and vulnerabilities. "

This is simply not going to happen.

Firstly, the majority of automotive control systems on an ECU are running
generated code from toolschains such as CATIA Systems, Dymola, MapleSim,
OpenModelica, SCICOS, SimulationX, Vertex etc etc etc.

To validate against vulnerabilities requires in-depth knowledge of a
particular manufacturer's modeling practices & standards. The code is not
readable, the model is.

Which relates to the second reason, the IP in these models is very closely
guarded. Competitive advantage and product differentiations are pretty much
all software now. Even suppliers and manufacturers share black box models &
systems back and forth for fear of leaking IP.

In a world where say BMW hands Bosch a set of requirement, Bosch supplies BMW
HW and BMW validates in simulation and analysis with neither getting insight
in to the other's design. You're telling that needs to now be done in the
open?

~~~
ejdyksen
Well, what you're saying is that they wouldn't open source it _voluntarily_.
This is why we have regulations: to make people and companies do things that
they wouldn't do voluntarily.

~~~
radiorental
But isn't this analogous to asking Microsoft to open source their OS because
of vulnerabilities.

I do not disagree with you. These are safety critical systems, I just don't
see how companies could be forced to share IP.

What is more likely in my view is something similar to the validation
standards in aviation:
[https://en.wikipedia.org/wiki/DO-178B](https://en.wikipedia.org/wiki/DO-178B)

~~~
CWood1
This raises a point that has confused me for a long time. Aerospace
engineering is known for its track record of safety (not to say incidents
don't happen, but they're rare, and usually self-contained). Why, then, is the
same level of rigor not applied to the automotive industry, by government (or
even international) mandate? Just as much damange can just as easily be done,
and if it was possible to seize control of a Boeing 747 from your living room,
you can bet there would be people fired within the hour, and arrested within
the day.

That said, something tells me that, in light of recent events, there will be a
push for the automotive industry to start using things like DO-178, and its
companions for mechanical and electronic engineering (whose numbers I don't
recall).

~~~
raisedbyninjas
Air travel is inherently more dangerous than automobile travel. This is
intuitively and actually true. People learn from a very young age that falling
hurts and falling even 10 meters can be fatal. Falling, flying, great heights
tickle the fear centers in your lizard brain. Vehicles are more like
incremental improvements to things that are safe like running or riding a
horse. Additionally auto accidents are mostly isolated to 2 parties on the
roadway they're travelling. So we don't exercise the same caution with cars.

Practically, air travel is safer because they are built, maintained, piloted,
and continuously monitored by professionals. Conversely a 15 year old can
build a kit car and drive it on the highway legally. If they break a dozen
traffic laws they may or may not get caught and penalized.

If we could get people to assess and respond to risk rationally, we could make
a lot of things safer.

------
joesmo
Obviously open sourcing the software is the only solution and everyone here
knows that. Software security isn't a new problem, it's a many decades old
problem with simple, effective procedures that can lead to proper solutions.
Implementing those procedures and solutions isn't enough, as the most
important part of security is trust and there can be no trust without
transparency. Period.

Software should be required to be open source if the authorities that be can't
police it (and they can't). Let the people do for themselves what the
government can't. I can't think of a single, logical reason why it shouldn't
be open source (profit is not a logical reason when lives are at stake). Until
more people die, of course there will be no significant changes in this
industry. In fact, according to history, until enough people die to reach the
threshold for anger in the public at large, there will be no changes in this
industry.

~~~
aleh
Opensourcing is not a replacement for QA practices.

If it was, open source applications would never have critical bugs or security
vulnerabilities.

It may mitigate some issues in a long run (e.g. intentional backdors) but in a
short run it will create havoc as access to source code will make it easier to
create exploits.

And suddenly you have whole new problem of making sure that everyone keeps
their ECU software up to date or they risk fatal crash.

~~~
joesmo
"Opensourcing is not a replacement for QA practices."

I'm not suggesting it is, simply that public oversight (open source) is the
only way to ensure trust of a secure system. The system itself still has to be
secure and requires QA like any other.

~~~
kaftoy
The only people who will understand that code (assuming is not just generated
code) will be people from competitors... maybe. People outside the business
will have no clue. It will be a continuous source of false alarms. Will be
really hard to filter out the crap. Other people already talked about
exploits. Every day you'll live the fear of "do I run up to date SW on my
car???". This is not a phone.

First you need to know the system, then you can understand what the code
should do then you can have a fare chance of doing a code inspection.

I think people parallelize too much to web development or other environments a
lot more exposed to public development. People think that if you can read
someone else's C code for voice communication, they will also be able to
understand the C code behind a diesel injection system, air intake path,
exhaust gas treatment, fuel mass setpoints calculation. If you have never been
exposed to stuff like that, you cannot fully judge if the code is wrong. You
may find something obvious here and there (like a loop that may go beyond the
limit in theory but never in the field).

And believe it or not, there is still competition for IP. Technical strategies
of doing something in certain ways giving supplier X clear advantage in some
field. Why would X share his knowledge just like that? We are talking about
huge money. Never seen Google share the source of all they do.

------
stillsut
The old car saying goes: Fast, Cheap, Reliable. Pick two, you can't get the
third.

Requirement creep seems to be the real problem here: Now we want aggressive
fuel-efficiency targets, crash safety, and a host of other "nice to have"
modern features. And it's in the computing unit which ends up trying to
mediate these necessarily competing demands.

