
Tesla Gets Carried Away Hyping Crash Test Results, Has Hand Slapped By Feds - w1ntermute
http://www.forbes.com/sites/michelinemaynard/2013/08/22/tesla-gets-a-little-carried-away-has-hand-slapped-by-feds/
======
revelation
It says as much in the official Tesla press release:

 _NHTSA does not publish a star rating above 5, however safety levels better
than 5 stars are captured in the overall Vehicle Safety Score (VSS) provided
to manufacturers, where the Model S achieved a new combined record of 5.4
stars._

You can't blame Tesla for taking that number and running with it, given that
(1) modeling car safety from two seaters to SUVs on a range from 1 to 5
_stars_ is ridiculous and (2) the star systems suffer from the same problems
as contemporary game and movie reviews: 99% of all evaluated products land
between 3 and 5 stars, where 3 then becomes the new _horrible_ , compressing
the already low entropy of a star value further.

~~~
CatMtKing
I demand p-values!

sum(p(accident type) __* p(estimated of surviving corresponding safety test))

~~~
Strilanc
P-values aren't probabilities you should multiply or add.

At best you're computing odds according to the null hypothesis (probably not
what you indented). At worst it's as meaningless as adding meters to seconds.

------
robterrell
I really like Tesla and want them to succeed, but they've got some weird ideas
about good PR. The cost calculator that factored in the cost of pump time at
your billable rate; the over-the-top posture against the NY Times reviewer who
discovered that the battery heater could draw enough power to affect range;
and now this, where they made up their own methodology to convert VSS into
stars and found themselves to be the best. (I'm not even considering the weird
stuff like Elon Musk offering publicly to consult on Boeing's batteries,
although I think that points to the source.) Aren't the "best car ever made"
reviews enough?

~~~
lnanek2
You mean the reporter who drove around in circles trying to kill the car off
and lied about the heater settings he used, also in an attempt to kill the
car? I think you missed the whole week there where the actual drive data
proved he was a fraud.

~~~
stcredzero
I don't like that reviewer, because he's evidently a very sloppy reviewer with
a pretty poor understanding of physics and mechanics involved. However, I no
longer think he was gaming the system deliberately. Assume incompetence before
malice.

~~~
eridius
He may very well have been incompetent, but at the very least he was dishonest
too. He put a serious bias on what he "reported" in his article. It may have
been a malicious attempt to kill the car for a better story, or it may have
just been an attempt to cover up his own incompetence, but either way, it was
a pretty bad article and deserved to be trashed.

------
jcampbell1
"The agency has dominion over any part of a vehicle sold in the United States,
from the headlamps to the owner’s manual, but can’t regulate the speech of a
corporation or its employees."

True, but the agency gives legal cover. When there is a question in court as
to whether the safety systems performed "as advertised", the NHTSA is free to
say that Tesla didn't follow the advertising guidelines, and therefore cannot
comment on the performance of the safety system.

Drug companies can also advertise drugs outside the FDA guidelines, and would
win a court challenge on 1st Amendment grounds. None do it because it would
expose them to unbounded liability.

~~~
lvs
>Drug companies can also advertise drugs outside the FDA guidelines, and would
win a court challenge on 1st Amendment grounds. None do it because it would
expose them to unbounded liability.

No, no, no. This is utterly and confirmably false. It is illegal under the
Federal Food, Drug and Cosmetic Act to advertise a prescription drug for a use
that is not FDA approved, and it is within the FDA's mandate to recommend
investigation to the Department of Justice in such circumstances. The FTC is
authorized to oversee advertising for over-the-counter drugs under the
Wheeler-Lea Act. As a recent example, Pfizer agreed to settle with the DoJ
earlier this summer for half a billion to end an investigation brought under
these very circumstances[1].

[1]
[http://www.newstatesman.com/business/business/2013/07/pfizer...](http://www.newstatesman.com/business/business/2013/07/pfizer-
pay-491m-fine-label-marketing-rapamune-us)

~~~
jcampbell1
Jesus man. You seem awfully certain. Here is a ruling from December that
affirms 1st amendment rights to market off label uses:

[http://www.reedsmith.com/files/uploads/DrugDeviceLawBlog/Car...](http://www.reedsmith.com/files/uploads/DrugDeviceLawBlog/Caronia.pdf)

" Caronia argues that he was convicted for his speech -- for promoting an FDA-
approved drug for off-label use -- in violation of his right of free speech
under the First Amendment. We agree. Accordingly, -3- we vacate the judgment
of conviction and remand the case to the district court. "

It is uncertain whether marketing off label is protected by the 1st amendment.
The courts don't all agree.

~~~
lvs
> The courts don't all agree.

The one case you cite is very fresh precedent in a single Circuit, and it's
highly controversial. No other Federal Circuit has applied First Amendment
standards to FDA misbranding regulations before. Even within the Second
Circuit (three states), is it still illegal under law to misbrand a drug in
marketing materials. The decision in Cariona hinges on the question of whether
"misbranding" and "promotion of off-label use" are different narrowly defined
legal scenarios. The Second Circuit was concerned that, in the latter
scenario, doctors may be legally deterred from promoting off-label use, which
is a common practice in medicine and research. You should not take away from
this case that drug marketing is now a First Amendment free-for-all. Yes, I am
awfully certain.

------
etler
It just seems like the NHTSA is just reiterating that there's no official
higher than 5 star rating to protect themselves, rather than scolding Tesla.
Tesla did say that a rating higher than 5 stars does not exist, and that the
extra high rating should be taken with a grain of salt seemed implied to me,
but I guess they could have been clearer. The blog post was very in depth, so
it's not like they were misleading people with their wording.

~~~
marvin
Yeah. This is a non-story; it makes a good headline but no one has been
slapped. Tesla just noted that they received some papers which noted that for
the internal purposes of the ranking, it got 5.4 stars. Then the NHTSA made a
clarification that the official ranking actually doesn't go past 5 stars.

It's good for Tesla shareholders that Forbes decided to run with this story,
though. Free advertising. There is no way anyone will ever perceive a fight
about whether the car should have 5 or 5.4 stars as negative.

------
twoodfin
_Does it really hurt consumers if carmakers are bragging about doing well on
tests?_

I'd argue that it might: NHTSA wants to encourage automakers to build safer
cars. One way to do that is to provide them with detailed information on the
results ("scores") of its tests, so that automakers can benchmark their
engineering against each other and tune individual safety features as needed.
Another way is to aggregate those results for each model and publish a
consumer-friendly figure that automakers are motivated to be able to advertise
as a "stamp of approval".

What they don't want is automakers getting into NHTSA test "hot-rodding"
games, focusing their efforts on topping the individual tests by some small
relative figure in order to claim things like "safest car ever". In an
environment like that, NHTSA has to either stop publishing detailed results,
or keep the mechanics of their testing secret in an attempt to prevent
"benchmark gaming" of the kind we see all the time in technology. That sounds
like an NHTSA that is competing with automakers rather than working with them,
and I suspect that is how consumers could be ultimately hurt.

------
jhonovich
This is nonsense: "can’t regulate the speech of a corporation or its
employees"

Entities have rights over how they are used in promotions, i.e., the right of
publicity / personality rights
[http://en.wikipedia.org/wiki/Personality_rights](http://en.wikipedia.org/wiki/Personality_rights)

For example, Consumer Reports No Commercial Use Policy -
[http://www.consumerreports.org/cro/about-us/no-commercial-
us...](http://www.consumerreports.org/cro/about-us/no-commercial-use-
policy/index.htm)

------
smackfu
I wonder how many other cars got more than five stars by Tesla's methodology?
From the graph, it looks like some have very close ratings.

And clearly, the goal here is to stop every manufacturer from just coming up
with their own rules for interpreting the score that give them a different
rating from the one that the NHTSA gives. That's definitely worthwhile.

------
skue
It was easy to see that coming once the media story took off the way it did.
But this just generates more coverage of Tesla's results.

I doubt that anyone at Tesla is feeling the least bit chastised. Musk is
certainly not the type of person who is afraid to generate a little
controversy.

------
nonchalance
Official announcements on [http://www.safercar.gov/](http://www.safercar.gov/)
(image) and
[http://www.nhtsa.gov/Misc/HeadLines](http://www.nhtsa.gov/Misc/HeadLines)
(first headline)

------
Kylekramer
So a problem with precision and judgment, but not integrity?

~~~
nonchalance
__and __integrity:

> NHTSA does not rate vehicles beyond 5 stars and does not rank or order
> vehicles within the star rating categories.

So the following statement is flat-out untrue:

> SETS NEW NHTSA VEHICLE SAFETY SCORE RECORD

If NHTSA does not rate vehicles beyond 5 stars, the maximum safety score is 5
and thus 5.4 is irrelevant

I think if the headline was "sets new vehicle safety score record", dropping
the NHTSA acronym, then it would be fine.

~~~
invisible
The Tesla blog post makes it abundantly clear that the 5.4 is the report the
manufacturer receives (not the issued rating which is capped at 5). Why does
it stop at 5 if you can still die in those cars?

~~~
smackfu
I don't think it does make that clear. It says "safety levels better than 5
stars are captured in the overall Vehicle Safety Score (VSS) provided to
manufacturers" but the VSS value for the Tesla was 0.43 according to the blog
post. So where did the 5.4 come from? Extrapolation?

~~~
invisible
You're right - they didn't make it abundantly clear that the VSS report was
0.43. Regardless, the conversion they made isn't insane when you look at the
other VS scores. Plus - watch the videos of other 5 star rated vehicle versus
the Model S. It's like the comparison between a 1980s car and one from 2013.

------
hamburglar
ObHeadlineHypeComplaint: that's a "hand slap"?

~~~
asperous
I take it from the article, nothing but a statement from the safety agency
asking "car manufactures" (not even mentioning Tesla) to not misrepresent
their data to the public.

------
Aqueous
Better to ask for forgiveness than permission.

------
hauget
why is shit like this on the HN frontpage?

~~~
skue
HN isn't just about software/engineering, it's also about entrepreneurship.

Tesla is a startup with cool technology. They're taking on a market where
startups never succeed and are navigating significant challenges regarding new
technology, consumer bias, competitors, regulatory issues, business model,
press relations, and more. Some of us find it fascinating and enlightening to
watch them take this journey.

Edit: Reworded.

