
A Single Component Can Brick Older Teslas - pfortuny
https://www.thedrive.com/news/27945/a-single-component-can-brick-older-teslas-and-tesla-wont-fix-it
======
BluSyn
This issue is best explained on Rich Rebuilds' recent video here:
[https://youtube.com/watch?v=o-7b1waoj9Q&t=525](https://youtube.com/watch?v=o-7b1waoj9Q&t=525)

This has been a known issue for some time, and it's amazing Tesla hasn't fixed
it yet. As explained in the video, the issue is simply the base linux OS has
full syslog's still enabled, which over time burn own the eMMC chip used for
storage. Currently Tesla just replaces the entire media unit when this
happens, which is extremely expensive for them. In reality they only need to
replace a tiny part instead of tossing entire boards.

The GOOD news is this issue is almost fully solvable through a software
update. They just need to disable base syslogs, or at least only store logs in
RAM while car is running (if they need those logs for debugging purposes,
which I doubt). Long term, they should replace this eMMC with a proper SSD, or
removable SD card that's easily serviced when it goes bad (as is the case in
other parts of the car).

I think this must be an issue of visibility in the company, and the right
engineer just isn't aware of this issue yet. I hope this gets more visibility
and gets patched in a future software update.

~~~
joezydeco
"Replacing a tiny part" is trivializing the work necessary to do this, in my
opinion.

eMMC flash is typically a ball-grid array part and it's soldered down hard to
the motherboard of the processor. Almost every mobile device has the same
setup and it's tricky and expensive to rework this chip.

You need to carefully desolder it off the board, reball the part with the
correct masking stencil, then reapply it without torching the rest of the
board, missing a solder connection, or zapping any other component with static
electricity.

You can see a similar process done on an iPhone here:
[https://www.youtube.com/watch?v=y4M9uAZlbK4](https://www.youtube.com/watch?v=y4M9uAZlbK4)

You can understand why Tesla would rather replace/refurbish the board than try
to field-repair it.

~~~
solarkraft
Luckily Tesla isn't very space or money limited with this pretty ciritcal
component. An SD card would make life easier for everyone, as the logs could
be easily read out and the booting OS changed - yay for actually owning the
device you paid for. But wait: That's not something Tesla wants.

~~~
joezydeco
eMMC is literally the equivalent of an SD card soldered down to the PCB. It’s
more mechanically stable and also way faster.

In my experience, removable SD wouldn’t hold up to an application like this.

~~~
naikrovek
An SD card might work fine if it is mounted properly. The slot found on a
Raspberry Pi would not at all be sufficient.

If it were held in and held against some robust contact pins it likely would
be fine, even under automotive conditions. There are SIM card slots that are
used on heavy vehicle computers and controllers (the ones that run the engine
and so on) that are robust enough to hold a SIM card in place for years of
excessive daily vibration.

~~~
londons_explore
Whenever I drop my phone I see the "SimProcessor" service spinning up...

I suspect that designing reliable spring contacts isn't as easy as it seems...

~~~
naikrovek
Yeah, phones aren't designed to have robust SIM slots, they're designed to be
thin, so they use very low profile SIM cradles that are not robust to impact.

The trade-offs for a high vibration environment are very different.

------
stefan_
This is rather disingenuous. Of course Tesla doesn't replace the eMMC; that's
a soldered on chip. No car manufacturer in the world does chip level
electronics repairs. Instead you replace the entire board, and I have no doubt
you can get this repair from Tesla.

That's not to entirely excuse Tesla. If you have a modern operating system
running from flash that needs to work for 20+ years as people expect from a
car, it needs to be very carefully designed - all logging and runtime data
written only to a RAM disk, system and user data on entirely separate
partitions, etc.

~~~
pfortuny
You are right but at the same time one does not expect a component of the
dashboard to actually brick a car. I guess.

~~~
zrobotics
While counter-intuitive, I have actually encountered this issue with other
cars as well, specifically a Ford f150 where an internal problem in the APIM
module (accessory protocol interface module, essentially the dash touchscreen
controller) essentially bricked the truck by spamming the canbus with
erroneous signals. Remember that modern automobiles are rolling networks with
multiple interconnected controllers, some of which are required for the
vehicle to function.

Note that this doesn't excuse Tesla here, since the situation I discussed is
very rare, normally if that module fails the vehicle will still start and run.
Tesla engineers should absolutely have been aware of this issue, as pointed
out up thread there are multiple tutorials for ras-pi SD memory preservation,
and I have trouble believing a competent EE shouldn't be aware of life issues
due to eMMC. It also shouldn't brick the car, normally automotive electronics
are designed very carefully to avoid single points if failure, with fallback
routines and safety "limp-home" modes in case of problems.

~~~
pvg
_bricked the truck by spamming the canbus with erroneous signals._

Wasn't preventing this one of the design goals/selling points of CAN?

~~~
zrobotics
Yes, that is one of the strong points of CANBUS. Like I said in the original
post, this was a very rare failure. However, the APIM managed to spam the bus
in just the right way where it de-synced modules; when I initially connected
to the vehicle w/ a snap-on scan tool it was throwing codes for BCM and TCM
non-comm, as well as codes that implied the ECM was seeing different speeds on
the CKP & CMP (crankshaft & camshaft sensors). The CKP/CMP disagreement was
what caused the vehicle to be 'bricked', since the engine management had no
idea where the crankshaft & valves were in relation to each other.

Near as I could tell from my scope, the APIM was spamming the bus with exactly
the right frequency to interrupt the ECM during it's scan of critical sensors.
It was an extremely rare failure, and to Ford's credit they covered both the
repair as well as my shop's diagnostic time.

edit: To make it clear, I have seen 2 vehicles that still operated with a
direct CANBUS short to ground, as well as a vehicle that had CANBUS shorted to
12V+. In these cases, aside from expected failures (such as the BCM systems
not responding, or transmission limp-home), modules were able to fall back
into either safe states (limp-home, in the case of the TCM) or just a
dashboard warning light (in the case of BCM no-comms).

------
vajrabum
As is so often the case with articles about Tesla the headline is completely
misleading. No cars are getting bricked. Tesla fixes that problem by replacing
the bad board but they won't replace the bad chip on the board and they won't
sell the chip separately. Other manufacturers don't generally do that either.

~~~
vfclists
> they won't replace the bad chip on the board

There is no bad chip on the board.

A full chip is not a bad chip, and there is no reason why the board should
stop working on account of a full memory chip which contains a syslog that is
not used for anything.

Hacker News is riddled with shills and apologists for atrocious corporate
behaviour.

~~~
silversconfused
You misunderstand both the functional properties of flash storage and telsa's
blunder. Tesla's mistake was sending too much data to the chip, yes, but it
exceeded the WRITE ENDURANCE, not the capacity.
[https://en.wikipedia.org/wiki/Flash_memory#Memory_wear](https://en.wikipedia.org/wiki/Flash_memory#Memory_wear)

(Incidentally, not a shill or shareholder, but rooting for tesla due the fact
GM pushed for leaded gasoline and ford would rather litigate than fix
dangerous vehicles. We're rooting for the underdog here hoping for global
positive change.)

------
wcunning
The orphan car problem is a serious issue with non-Teslas, too, but I will
personally stick to repairable vehicles. You know what you'll never, ever have
trouble finding parts for? A Mustang or an F150. Corvettes. Carollas. All of
the ubiquitous cars will have parts produced for a seriously long time.

~~~
cheez
Isn't this more of a network effect than something inherent to Teslas?

~~~
whenchamenia
Nope, tesla will not allow you to order parts, or even look at a service
manual as a mere owner. Its crazy that people on HN of all places defend their
customer-antagonistic practices.

------
wil421
Any other examples of the big 3 US, German, or Japanese car makers having a
similar issue? Where a media center issue causes the car to not run and the
manufacturer refuses to fix the issue or even provide a part for it?

~~~
vajrabum
The article is misleading. The manufacturer does fix this. They just don't fix
the bad chip. The replace the card instead.

~~~
starshadowx2
Except they don't if the car is out of warranty, which (at least in Canada) is
4 years or 80,000 km.

I don't think this is just me but I believe cars should be able to last longer
than 4 years and not get bricked by fault of the manufacturer and then not at
least easily give me the ability to fix it myself if they won't.

~~~
slg
Do they not replace it out of warranty or simply don't pay for it out of
warranty? There is a huge difference between those two options. If it is the
former, I would imagine it is just a matter of time before a lawsuit. If it is
the latter, then isn't this functionally the same as a particular model of car
having a bad transmission or similarly critical ICE component that regularly
fails much earlier than expected?

~~~
lutorm
Powertrain components are required by EPA to have a 10yr/100k miles warranty,
I thought? So it's not exactly the same situation because fundamental parts of
the powertrain are covered by a longer warranty.

~~~
slg
That doesn't appear to be true [1]. It seems like a majority of powertrain
warranties are in the 4-6 year and 50k-70k mile range. Tesla's 4 year and 50k
warranty on this part would therefore be on the low end, but it matches the
powertrain warranties for Audi, BMW, Volvo, Mercedes, and several other
manufacturers.

[1] - [https://www.motor1.com/features/253277/comparing-new-
vehicle...](https://www.motor1.com/features/253277/comparing-new-vehicle-
warranties/)

------
CalChris
This affects pre-facelift MCU1 car, which is before about April 2016. So I
suppose a solution now is to upgrade the MCU to MCU2. Elon says that will be
possible.

[https://twitter.com/elonmusk/status/1105549053696327680?s=12](https://twitter.com/elonmusk/status/1105549053696327680?s=12)

------
nemosaltat
Model 3 owner here. Having same issue as back in March. No AP, no FSD and most
frustratingly, no cruise control. Multiple TS steps over phone with no
resolution. The earliest appointment was months out so I drove into a service
center. A frazzled, yet personable, tech assured me that the issue was NOT
from the recent FW update but rather a hardware malfunction with the left
repeater. I pressed for details- is it wiring, hardware, physical damage? He
couldn’t tell me, but insisted that it had nothing to do with the previous FW
update and that another FW update wouldn’t resolve it

Tech scheduled a service appointment for 4-13, much better than I could find
in the app. On 4-4 Tesla pushed a FW update, which I guess is pretty common
before a service appointment. On 4-5, a Tesla rep reached out and said the
issue WAS in fact, with the original FW update and that the more recent FW
update had resolved:

“Hello Sam,

We have reviewed your vehicle logs and alerts. At this time we do not see any
issues with the Autopilot system as well as you confirming that the system is
operating as designed. Since this is the case there will be no need for you to
come in for your scheduled appointment. While there is no way to say that you
will never have an issue again, at this time there is no issue that needs
correction. If you do have an issue in the future you are always welcome to
create another appointment with your concerns.”

I responded:

“Ok- I’d still like to understand what happened. It is generally acceptable
that the universe is getting more disorderly. It is perfectly acceptable that
something goes wrong and we don’t know why. It’s significantly less
understandable, that the issue mysteriously resolves it self. Does that make
sense?”

Tesla Response:

“Hi Sam,

It was found to be a software bug in the last update that was performed
causing your cruise control not to function. That was resolved in the last
update you performed.”

The rep then canceled the 4-13 appointment without further explanation or
communication.

The issue returned yesterday (5-13) and now the earliest appointment is mid
June.

Cruise control existed in 1958. I get that Tesla has “tight integration” but
the lack of a fall back/degraded performance mode seems ridiculous. This is
before considering the lack of NoAP and FSD, which adds insult to injury-
Those features represented nearly 15% of the vehicle cost in my case ($8,000
upfront).

Convenience in LA traffic was a primary consideration when purchasing the
Tesla. The lack access to purchased features is irritating, but I knew they
were bleeding edge. The further loss of a ubiquitous feature like cruise
control is even more irritating. Tesla’s handling of the situation feels
ridiculous.

~~~
zaroth
This report is both impressive and frustrating. I’m sorry your AP is not
functioning and their fix was not permanent, that must be incredibly
frustrating as a huge benefit of the car is the AP system.

At the same time, it is pretty amazing they are remotely logging into your car
to review system logs and diagnose the issue, and then pushing firmware
updates to try to remedy the problem.

There may be no choice but to drive back to the service center to try to get a
quicker response from Tesla. At some point if they can’t fix it there could be
a remedy under the lemon law — they only get 3 chances and a certain number of
days AFAIK.

I can understand how a single fault could disable the adaptive cruise and
likewise all of AutoPilot functionality. But Tesla is expected to be able to
fix the issue promptly.

~~~
fucking_tragedy
> _At the same time, it is pretty amazing they are remotely logging into your
> car to review system logs and diagnose the issue, and then pushing firmware
> updates to try to remedy the problem._

It really isn't if it didn't fix the problem and they used it as an excuse to
not follow up with their scheduled appointment.

This would be like if I was fixing a bug, SSH'd into a live instance, didn't
fix anything, then filed the ticket as "fixed" and refused to follow up on it.
That's not amazing, that's pretty sketchy.

~~~
zaroth
But that's not what happened.

nemosaltat didn't provide a complete chronology, but did say they had a
service appointment on 4-13, received a FW update on 4/4, and a rep contacted
them on 4/5 saying they reviewed the logs and the new FW fixed the issue.

Then nemosaltat responded to Tesla;

> _Ok- I’d still like to understand what happened....It’s significantly less
> understandable, that the issue mysteriously resolves it self..._

Tesla responded,

> _It was found to be a software bug in the last update that was performed
> causing your cruise control not to function. That was resolved in the last
> update you performed._

Then,

> _The issue returned yesterday (5-13) and now the earliest appointment is mid
> June._

So the issue was fixed, and then it returned. It's unfortunate but hardly
sketchy.

~~~
nemosaltat
I think gatherhunter hit it dead on. It’s metrics thing and they wanted to
close tickets. I sincerely doubt the FW update they pushed had anything to do
with the issue being resolved. It is common practice (according to Tesla
forums) to push FW before a service appointment just to be sure the car has
the latest version before it goes in.

The sketchy part for me is the inconsistency between the Service Tech and the
rep who canceled the appointment. The tech (HW flaw) was very convincing, and
the rep (SW flaw) was very vague. The functionality did not return immediately
after the FW update, and I think the timing may have been coincidental. It’s
even more frustrating that they weren’t/aren’t the least bit apologetic or
accommodating. I can’t seem to get a straight answer on why basic cruise can’t
be restored via a FW patch while I wait for my appointment. Is the same
hardware/software required for Cruise on Model 3s without NoAP and FSD?

~~~
zaroth
Presumably they know more about the root cause through looking at the logs. It
would be so great if they could be more transparent about it.

On the surface it doesn’t make sense that a FW issue would be causing an issue
with _your_ TM3. What about the other ~200,000 cars on the road?

But to answer your last question, yes, it’s layers of incremental software
functionality on the same hardware stack. Without knowing at which point it’s
failing it’s hard to say if cruise could be enabled without AP.

As a point of comparison, I had my TM3 wrapped in protective film. As part of
that process they removed trim including the side cameras to lay the film flat
with fewer cuts. The shop (not Tesla affiliated) left the side cameras
disconnected. Upon starting the car it reported an issue with side cameras. AP
worked but degraded — wouldn’t change lanes, and cruise still worked fine.
Some some hardware faults do not fully disable the feature.

~~~
nemosaltat
>left the cameras disconnected... AP worked but degraded

That’s very interesting, and closer to what I would expect.

The newest development, when I drove in this morning, the Service Tech said
it’s a seatbelt sensor malfunction.

There is a “driver seat buckled” interlock for cruise control, but the alert
for that explains the reason. I’m just getting “cruise not available” with no
explanation.

~~~
zaroth
Yeah I unbuckled once while AP was active to try to take off my coat and the
car lost its shit on me. Red flashing alarms like, “WTF you doing?!”

I think that service tech was grasping at straws, unless he actually examined
the logs and saw it written there. If it was the seat belt, wouldn’t you see
it indicated on the display that the seat belt was unbuckled? It displays the
unbuckled indicator for all seats that have occupants detected (weight
sensor).

Presumably the logs will contain the exact reason it fails? Next time they
text you maybe ask if they can tell you exactly what the log file says when AP
engagement fails.

------
TazeTSchnitzel
I'm sure the Model 3 having a single multifunction screen that does everything
(combining entertainment with critical driving features) will go _great_.

~~~
m463
Actually, every control in the model 3 is used for multiple functions (for
example drive selector also controls autopilot, scroll wheel controls many
functions, etc)

I guess you'll just have to drive with your phone.

------
arcticbull
The first half of the video is pretty cool, they're converting a Sprinter van
into a "Model 3"

------
vfclists
Rich Rebuilds zonks Tesla, again -
[https://www.youtube.com/watch?v=o-7b1waoj9Q](https://www.youtube.com/watch?v=o-7b1waoj9Q)

------
RickJWagner
Bummer. The value of a used Model S (they are becoming attractive) has just
dropped a few notches in my eyes.

------
BinaryResult
I have a 2013 model S with 55k miles. Hoping not to encounter this.

