
Burned-Out Flash Trips Up Older Teslas - JoachimS
https://www.eetimes.com/document.asp?doc_id=1335331
======
farazbabar
There are a ton of gremlins in Tesla just waiting to go off like time bombs. I
recently got a very expensive repair just outside of warranty where the car
simply refused to charge. Paid nearly 3000 for the diagnosis and repair for
the car to start charging again - we are talking about a car with no wear and
tear parts for the most part costing nearly 3K because a chip somewhere
between the charge port and battery failed. My whole reason for buying it was
that this would be the ultimate reliable car without ICE or brake wear,
leaving tires as the only consumable. Guess I was wrong.

~~~
Abishek_Muthian
Even in combustion engine cars if ECM buckles up it usually is expensive to
repair(at least in my part of the world). I guess with a car like Tesla, rich
with electronics & unavailability of parts to 3rd party service centers;
repairs can be very expensive.

~~~
Nition
Is any car company making a simple electric car without fancy features, or at
least one where they're optional? Even the Nissan Leaf now has all sorts of
high-end features. I'd much rather have a simple, cheaper, reliable car with
less points of failure.

Edit: Thanks for the suggestions!

~~~
wldlyinaccurate
Tesla fans will tell you otherwise, but pretty much all of the EVs from
"traditional" automakers are more reliable than Tesla vehicles. Cheaper &
easier to service too.

I've heard very good things about the Nissan LEAF, Hyundai Ioniq, Hyundai
Kona, and Kia Niro. It might be a coincidence but all of these cars are
packaged more like traditional cars, i.e. popping the hood allows you to
access many of the components.

~~~
pmorici
More reliable? Didn’t leafs of the same time period of the Tesla models being
discussed here have horrible battery degradation issues? Hyundai and Kia
electrics are too new to the market to know if there will be serious issues

~~~
mikestew
Early Leafs would apparently get their batteries cooked in Phoenix because no
active cooling. We’ve got one of the first ones that rolled off the line, and
eight years of Seattle weather has ours at ~12% degradation. We otherwise
haven’t done shit to that car except drive it. So from my POV, I read the
issues with Tesla’s and just shake my head.

------
ggm
The earlier models were premium not economy pricepoints. Why is the ECU
designed around low cost on-board, non-replaceable constructs like this? A
premium product, would not nickel-and-time the SD card, it would have
engineered it as a field replaceable unit, with an insert,format, wait (for
download) and go mode.

Replacing the entire ECU feels like an admission like Apple and increasingly
Lenovo (thinkpad), they have driven premium to the wrong place. _" please give
us soldered on memory and soldered on SSD"_ is not actually what we said, on
our feedback forms to Apple, or IBM (X1 Carbon).

Nobody asked Tesla to shave $1 on the ECU assembly and the incremental cost of
recall and fix, along with goodwill now outweighs the cost saving in any sense
I can understand it. If the authorities demand he fix for free, it becomes
another shit show of the TSLA share price. This wasn't a good choice against
all the other choices.

~~~
jchw
Lenovo still makes Thinkpads that are extendable. My P51 has 3! m.2 (2 of
which support NVMe,) 4 RAM slots, easily replaceable keyboard and more. But I
think most people want thinner laptops, and the thinner ones have less of
this. I don’t think this is a headphone jack type situation: people value
portability a lot on laptops, and the difference between a rugadized, user
expandable laptop and an “ultrabook” are striking.

~~~
jdnenej
My dell xps is super thin and it still has easily replaceable ram and battery.
The massive use of glue, rivets, proprietary screws and cryptographic
verification in macbooks is not for slimness but for planed obsolescence

~~~
Aloha
Unpopular opinion, but from where I sit the serviceable lifetime of a computer
is effectively about a decade, so long as all that non-replaceable stuff lasts
the serviceable lifetime of the object, then it doesn't matter how repairable
or not it is. Apple charges like 200 bucks for a battery and top case, which
is pretty competitive for a replacement OEM keyboard and battery combo.

~~~
ksaj
I think that's about the typical life span of a modern car, too. Anything
older than a decade tends to be quite an awful drive. Unless it was built in
the days before we decided everything needed to be so expendable.

~~~
kozak
A car can be maintained well as long as parts for it are available. That
people don't want to spend on maintenance of >10 yo cars is a different issue.

~~~
saalweachter
... if you live somewhere they don't salt the roads.

~~~
kozak
I think it depends on the quality of sheet metal. I live in a place where it's
all mud and salt throughout the winter, and somehow my car doesn't rust even
where the paint is scratched. I think it's because of a good-quality
galvanized coating under the paint (it's a boring normal non-luxury car made
in Korea, if someone's curious).

~~~
saalweachter
The frame is what I worry about more than the body. I'd be happy to keep
repairing my ~15 year old truck, but the frame is getting to the point where
I'm not sure how much longer it's going to pass inspection.

------
m0xte
There’s no excuse for this at all. It’s shitty short sighted engineering.
Simple as that.

The main firmware should have been on a separate physical device if data was
write heavy regardless of “enough space” being left. It would have cost a few
dollars to make this issue go away.

The more I hear about the engineering the longer I will avoid buying an
electric vehicle. Iterating big problems away on a tangible chunk of
engineering rather than a piece of software is stupid.

~~~
rapnie
A couple days ago I found this link [0] to a Reddit post on HN about the
software engineering practices at Tesla. Don't know how accurate the story is,
hard to tell, but it sure was worrisome.

[0]
[https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/form...](https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/former_tesla_programmers_anecdotes_about_problems/)

~~~
m0xte
Yeah I sent that to my father who is a model 3 owner and former software
developer :)

------
jve
There was once a guy that shared instructions on how to resolder that chip
(Get dump of software, root it, solder new chip with more memory). It was
instruction on how to root Tesla. But oh, he deleted his content and left some
comments:

> As a result I've taken down my howtos. I'm sorry about this but there is
> some kind of political problem which I don't understand.

[https://www.diyelectriccar.com/forums/showthread.php/howto-t...](https://www.diyelectriccar.com/forums/showthread.php/howto-
tesla-rooting-firmware-199457.html)

Pity the site is not available on waybackmachine.

~~~
chubs
I'd be surprised if it's any different to modding your BIOS in a PC
motherboard: Buy a SOIC-8 clamp, a USB interface and bob's your uncle. Unless
the ECU is potted in epoxy.

~~~
Scoundreller
Doubt it’s an SOIC. More likely a 40 or 48 pin TSOP or the like.

But if it was an SOIC... just lift the power pin, power the chip externally
and find some other points to tap into the clock and data lines and dump away.
Then solder the legs of the new one on one side and connect VCC on the other
side.

~~~
jve
Hehey, archive.is has that page!
[https://archive.is/kIZpA](https://archive.is/kIZpA)
[https://archive.is/Leqih#selection-775.0-779.271](https://archive.is/Leqih#selection-775.0-779.271)

------
pkaye
> Tesla’s firmware utilizes nearly 100% of that space available, so it’s
> almost impossible to use wear-leveling for the vehicle logs.

Its still possible to do this. You do wear level between the blocks containing
the firmware and the logs.

~~~
Akababa
Firmware doesn't update as often though so it'd have to be static wear-
leveling. I'm not sure of the safety of doing this with firmware files though
- could it actually decrease the life expectancy by moving critical blocks
more often?

In any case, if 90% of the space is filled with static firmware files then
wear-leveling becomes difficult. Probably best to turn it off and let the
physical blocks with the logs fail while keeping the firmware blocks intact,
or just stop writing logs after a certain point.

[https://en.wikipedia.org/wiki/Wear_leveling#Static_wear_leve...](https://en.wikipedia.org/wiki/Wear_leveling#Static_wear_leveling)

~~~
mlyle
> could it actually decrease the life expectancy by moving critical blocks
> more often?

No. You do normal, dynamic wear leveling. When you notice that the block you
want to write to is much more worn than the rest-- say 100 erases extra-- you
move a firmware block to it instead and claim that for use. Then that block
sits near-quiescent for a long time until it's time to put it back into the
rotation of blocks that get actively written. Static wear leveling can just be
a tiny, tiny share of write amplification-- a factor of 0.01 or even less--
while keeping all the blocks in play.

You only "lose" from static wear leveling if you end up moving something right
before it becomes active again. Whatever your strategy, there's always a
degenerate write pattern which will cause useless write amplification, but
even relatively simple schemes are robust against basically everything except
deliberately malicious write patterns, and you can bound this worst case.

Incidentally, one way flash fails is by dielectric breakdown increasing
leakage causing worn-but-idle-blocks to slowly accumulate error. Moving /
occasionally scrubbing static, important data actually helps.

------
tropo
This doesn't bode well for classic and antique car collectors of the future,
who will want operable Tesla cars.

There are Ford Model T cars that still drive. They were made 92 to 111 years
ago.

With undocumented and cryptographically signed interfaces between components,
it will be impossible to keep a Tesla running that long. The best you could
hope for is to graft the body panels of a Tesla onto the innards of a newer
car, merely dressing up the newer car to cosmetically look like a Tesla. Such
a hack would not be have vehicle title as a Tesla, would not drive like a
Tesla, and would not have an interface like a Tesla.

~~~
walrus01
A problem like this could be significantly alleviated by not having the flash
soldered onto the motherboard. This is one of my concerns with the current
generation of Apple MacBook laptops. Considering the cost of a Tesla as a
whole, surely it can't be ridiculously more expensive to put a m.2 socket and
discrete flash module on a motherboard.

~~~
oldgradstudent
> surely it can't be ridiculously more expensive to put a m.2 socket and
> discrete flash module on a motherboard.

It would be more expensive for Apple because people would buy cheap models and
later upgrade, instead of buying expensive models or buy new machines early.

------
sm4rk0
1\. Let's go back to the good old HDDs. There are even automotive industry
ones (I own one)

2.There's a difference (8-fold) between Gb (gigabit) and GB (gigabyte). Memory
chip capacities (and data transfer rates) are often expressed in bits (per
second), but memory device (RAM stick, storage,...) capacities and file sizes
are always in bytes.

------
ComodoHacker
With such extensive logging and monitoring it's hard to believe they didn't
monitor MMC wear and didn't see it coming.

------
agumonkey
SSD wear was known for a while, but that it would cascade into display then
charging malfunction.. that's class action worthy.

------
castratikron
Does anyone report on eMMC wearing out in phones, game consoles, TVs, laptops,
etc.? What makes Tesla special? Are there really no other car manufacturers
using eMMC btw?

Easiest hardware fix would be to replace the part with a larger one which
would only be a BOM change. But that doesn't help the units already in the
field. I'm sure they've figured something out by now.

>Tesla’s firmware size over the years went from a paltry 300Mb to a full 1Gb
over an estimated four year period, which consumes much of the storage space
in that Flash. Add on the constant vehicle log updates, and the failure rate
of those chips increased exponentially.

Does it really increase exponentially or does 'exponentially' just mean 'a
lot' now? Genuinely curious. Also I think I'm allowed to be pedantic when the
source is an electrical engineer.

~~~
sjansen
Most people replace their phones and laptops every 2 to 5 years. Too fast to
see any problem under normal usage.

Most game consoles and TVs are unlikely to do enough logging to wear out their
storage. But if they did, especially if they cost as much as a car, it
probably would make the news.

Cars tend to last a decade or two. They shouldn't stop working because of
something as silly as excessive logging.

~~~
novok
Most game consoles have a lot more free space, use hard drives and don't do
much when in sleep mode.

------
Tempest1981
The Model S and X have 8 GB of flash, according to the article. Does the Model
3 have more?

> The MCUv1 units supplied for the Tesla Model S and X, up until 2018, feature
> an Nvidia Tegra Arm-based SoC with 8Gb of eMMC integrated into the same
> board, which is hosted on a mainboard.

------
pbreit
This would seem fixable with software update. If not, then a cheap hardware
fix. Does anyone know?

~~~
jdnenej
There is no such thing as a cheap fix for an incredibly complex and
proprietary device. You will have to pay a highly trained expert and hope they
have stock left of the part.

~~~
SmellyGeekBoy
Sure there is. It's a mass-produced PCB. Just take the old one out and drop a
new one in.

Whether Tesla are willing to foot the bill outside of warranty, however,
remains to be seen.

------
Animats
It shows Tesla's priorities - collecting data on the customer has priority
over keeping the vehicle running.

~~~
t0astbread
Nowhere does this article state that the logs are sent anywhere. And I think
we can all agree that logging for the case of failure is reasonable.

------
huntertwo
I thought that Tesla engineers would have thought of this. Guess it’s hard to
be an innovator and inevitably you will fuck up pretty majorly. With cars
though these fuck ups are more significant than with phones.

~~~
rsynnott
> Guess it’s hard to be an innovator and inevitably you will fuck up pretty
> majorly.

Or, alternatively, hire people who know what they're doing and take their
advice? Flash wear isn't exactly an obscure issue, and it would certainly be
on the radar of, say, Bosch (who make a lot of this stuff for the industry).

~~~
huntertwo
Right, but not something you might consider until your OTA updates makes your
kernel go from 300MB to 1GB. How many car manufacturers do OTA updates? Still,
it seems like an extreme oversight and shouldn’t have happened even in the
most rushed conditions.

