I work in manufacturing and sometimes stuff like this happens despite controls in place. You can get technicians/assemblers who just take it upon themselves to fix a problem rather than notifying anyone. To them it is no big deal (i.e. doesn't warrant mentioning to engineering), so it must be "no big deal".
It's a design error from the start. The workaround shouldn't have happened, but is only one of countless ways this would have inevitably happened anyway. Glue has a lot of failure modes. Correct application can't be reliably tested non-destructively. Product variances are often very hard to detect. Degradation with age and physical use can't be reliably forecast.
Three pins on the back of that appearance plate that push into starlock style fasteners in the pedal are cheaper than the appropriate glue, faster to install than glue, more reliable, trivially verified, impossible to misalign, and that's why it's a common solution that auto manufacturers use in this exact application. This was a confoundingly stupid place to rely on glue.
Correct - a great design is also easy / automatic to build correctly. This is vastly easier said than done when you have a complex product with many components.
I remember reading about the original iPhone's manufacturing operations in China - how the Apple engineers spent a ton of time making sure that the right way is also the easy way for the factory workers.
Same, I work in manufacturing(not automotive but heavy construction equipment) and see things like this all the time. Workers think they understand/ don't think engineers understand or want to do it faster/easier than what they were shown.
I have no knowledge of Tesla but here would be my guess:
Assembly worker found pad hard to put on pedal in sub-assembly area and used a spray bottle with soapy water on the pad to slip it on.
Story time: Called out to final assembly, machine starts and runs but not moving. Troubleshoot and find brakes not releasing. further troubleshoot and find it is due to pressure not getting to brakes(configuration is such that brakes come on if there is loss of hydraulic pressure). Replace hydraulic line, machine is working. Remove contaminate from line, no one know what it is. Assembly pointing fingers and saying sabotage. I walk around the assembly area, I find that paint decided to use packing peanuts to mask holes that the hydraulic fitting go in instead of masking tape as directed. The packing peanut tore while being removed and the assembly working inserting the fittings did not notice.
Human nature. I run into this all the time. I've lost count of the number of times I've asked a user "Why did you not just mention this was not working right and you are working around it? We could have fixed this, but if you do not say anything it might be a while before someone on the dev team notices."
I think that devs often underestimate just how difficult it is for users to report problems. The most common problems are that the users feel ignored, like they're being a burden on the devs, or scolded (for not reporting it correctly, for "not holding it right", etc.). It's even common for there not to be an easy way to report such problems ("use Discord", "sign up for an account on this website and report there", etc.)
Even as a dev, I resist doing it because of how unpleasant it can be. If I can come up with a workaround without having to report the issue, that's what I'll tend to do. And if I have to talk to tech support rather than the devs? That's simply not going to happen unless I'm trapped into using the product.
We still haven't cracked this problem as an industry.
To make things worse the largest consumer tech companies, like Google and Apple, have a well deserved reputation for caring very little about customer feedback. It's a normal thing to lookup how to fix an annoyance or regression, finding hundreds of people posting about the same complaint, without ever getting any sort of response or reaction from the company.
Heck the only support Google offers for many products is a community forum that their own employees never post on, and I assume few even look at. People have largely been conditioned to think that tech companies don't care about their feedback.
This is a Support task, not a Dev task. Support should be working the tickets and reporting unsolvable issues with the code, so the Devs can address.
You've been dealing with bad support teams, because your experience is not how support is supposed to work.
Also, we Ops folks truly appreciate undocumented work arounds by the Devs. We love spending hours pouring over a given system, trying 107 different versions of some framework, causing lots of downtime, and working nights/weekends, just to learn that some UNDOCUMENTED cludge fucking bullshit is what's actually causing the issue.
Do better man. You're shitting on more than just the users.
With open source, you're usually reporting to devs. With commercial software, usually to tech support or to nobody.
> You've been dealing with bad support teams, because your experience is not how support is supposed to work.
Yes, I know -- but it is how the majority of support actually is, if there is even support available at all. In a whole lot of cases, there is none.
I'm talking about software meant for consumer use. Software for business use is much better on these issues, although you still do find them. At my workplace, we recently took a large financial hit (and almost lost an important customer) because of bad and unresponsive tech support from a supplier. It happens.
There's something difficult about notifying problems. You might make people angry, you might feel like a moron because you misunderstood, or guilty because you worry how they feel.
Agreed. For me it's often the time it takes to find the contact to notify, start a conversation, update the conversation, wait for the issue to be picked up, wait for the software to be updated. Now repeat for everything you notice.
If I did it for everything I encountered I wouldn't be doing my core work duty. It's more pragmatic the majority of the time to work around the issue and immediately get back to work.
Wait, don’t car assembly lines have a big red button you can push if you find a defect? Haven’t they for many years? Does pushing that button really make everyone angry?
That’s not how I had envisioned car manufacture at all.
I completely agree with you that some errors could be missed if no one noticed them. This seems entirely separate from the idea that people could become angry if the line were stopped, but of course you are right.
Have you tried thinking of the reasons? I can think of several:
* There's probably a small chance it actually would get fixed, and therefore a decent probability that reporting it would be a waste of their time.
* They needed a solution sooner than reporting it and waiting for fix to maybe eventually appear. Once the workaround was in place there was no need for a fix.
* Sometimes the people running projects you use can be hostile, which makes reporting stuff very unappealing and even stressful. Much better to avoid interacting with them if at all possible.
* They didn't know who to report it to, or how to report it.
Yeah our support people at my company do the same thing. Then you'll get a report 6 months later that "[big critically important feature] is not working" and you'll look into it and support has adopted a process that essentially disables that feature or they have a workaround for a bug that was fixed 4 years ago and because they never entered the conversation at that time they still do the workaround.
We had a big kerfluffle around our OTA update system at one point because they did a big round of updates and "none of them worked." And then I dug into the system logs for each of those components and 95% of what they claimed didn't work actually did. But meanwhile you've got product managers and other people wading into the conversation to try to tell you to fix something that isn't actually the problem.
We're never truly going to get away from this until we stop excluding people from the conversation about product problems. I'm just sitting here hoping we adopt a quality management system of some sort before the company's product implodes.
There was a poorly implemented customer support system that I worked with once that due to the way the app worked, support could run a query that would essentially scan the entire database, predictably it'd hit a proxy timeout. So what happened instead was they would open 10+ tabs doing the exact same query hoping one would get lucky and succeed, and we had to figure out why our database was getting ddos'd. Trying to explain that they were actually making the issue worse with the workaround was very painful, saying stuff like "well what did you change, it was working fine for months."
It's easy to portray it as arrogance, but in manufacturing, you run into small problems and ambiguities all the time.
By analogy to software engineering, do your bosses or clients give you water-tight, formal specs for the software you need to build? If they could do that, they wouldn't be needing you in the first place.
We zero in on situations like that and pretend that it's the worker's fault for making the wrong call, but we ignore that if they didn't make the right calls a thousand times before, nothing would ever get done.
In this case, if pedal cover is a friction fit and can slide off and get jammed in between panels, this doesn't sound like an assembly mistake but a pretty major design error, right? Your designs should be resilient. What if the owner sprays WD-40 on a squeaky pedal and the cover slides off?
Exactly - especially in a TRUCK of all things. The pedal area should be expected to get getting all kinds of crud and crap in it and be cleaned regularly and be extra durable.
I've always suspected it was apocryphal. But just think, if the workers had installed that fan beforehand, we might be reading a story about how important workers are at solving little production problems.
Everyone thinks this mentality of having workers fix problems is great, until they use soap to put glued parts on.
I don’t know anything about manufacturing, but the main thing that stood out when I toured the BMW plant in Munich was how ruthlessly efficient and regulated every step of the production process was. It’s hard to imagine there being any time for improvisation, or any ability for it to go unnoticed. Is Tesla’s production process just looser?
No... I've been a maintenance worker.
If you needed "Engineering" to help you fix every problem you faced everyday in a production line, the Engineer would need to come to work with you every day.
If you stopped the production line until "someone higher up" came down to approve your changes, you'd better make sure you have a strong reason to do so as the company will be losing millions while you wait :).
You just solve problems all the time, every day, and it's really up to the technician to know when something requires notifying Engineering or not. Notify too much and they'll get rid of you for being annoying... notify too little and shit like this can happen, but in the very large majority of cases, it doesn't.
I haven't been a maintenance worker, but I've worked as an SWE in a company with a large IT dept. Sometimes it's faster to work around them to find solutions to doing your job. Both sides have good intentions but the IT dept. cannot move nimbly.
I don't have a problem with technicians solving problems. But as an engineer I would like to codify the solution so that A) we're implementing a controlled process and B) if there's a better solution out there I can make that recommendation or fix the system. When you take it upon yourself then problems only happen if you don't communicate.
> If you stopped the production line until "someone higher up" came down to approve your changes, you'd better make sure you have a strong reason to do so as the company will be losing millions while you wait
Then what was the whole point of the Andon cable lesson that American manufacturers had to learn from Toyota?
Is "Irresponsibility" I feel -- without any true blame / shame though.
Complex work is hard.
Self-management is a big, under-appreciated part of that.
So Irresponsibility maybe not on the individual Worker's shoulders, but on all of us for under-appreciating the risky challenges of being a motivated worker in a complex job.
yeah IDK if the Worker is to blame, seems like an obvious design flaw, e.g. they should not rely on 'soap' to keep a flat pedal cover attached to another flat pedal.
I actually blame the engineering/design department for this one.
The soap revealed the issue, but why aren't the peddles a single piece? Why do they have a sticker on them?
Even without the soap step, what happens if the cabin gets too hot or the factory has too much dust in it?
If you look at your car's peddles (and I'm including mine, a Tesla model 3) you'll notice they are basically a single piece mechanically fit together. Not some sticker glued for style.
Right, it's always dependant on circumstances. I try to stress as much as possible that you always need to design things in such a way that even the dumbest, newest assembler will still be able to build it correctly. And often times we review drawings/instructions and find lots of poorly outlined procedures.
But sometimes you get something like "The blue wire ran out, but I still had a bunch of light blue, so I just used that instead". It can be a killer.
