> because it was written by under-experienced software "engineers."
I used to be an infotainment software engineer for an Auto maker. There’s this idea in the software industry that because the software behaves poorly on your car’s system, it must’ve been written poorly by someone who didn’t know what they were doing. In my experience it was almost always a matter of the hardware being stretched way past its limits. Your brand new 2024 car’s system has been in development for 4 years. Product people have been picking at the design and asking for changes way too late, and someone in accounting has asked the hardware contractor to cut more cost off the BOM.
My colleagues were tremendously talented. Some of the brightest engineers I’ve had the pleasure of knowing. The developers aren’t lazy or inexperienced, they’re just being asked to turn lead into gold and often get silver.
I'm willing to credit this type of feature creep for some of the problem, but only some.
Decades ago, we had hardware that was so much less capable than anything anyone has put in a car in ten years that the comparison is absurd. Nonetheless, those old systems were often very capable of doing useful things, and the UIs were often very, very fast by modern standards. And a lot of this was that the software stack was tuned for performance and latency and did not do things that hurt latency too much, and this came at a substantial modularity cost.
Back when a TI calculator with a whole OS running on a Z80 was an order of magnitude or so faster than your average new car, it didn't have a compositor (which still eats a good frame or so of latency, maybe more, and, as an industry we still don't know how to solve this). It ran an OS that was surely far harder to tweak than a modern car stack, but it had fewer layers and could be kept performant. An Apple ][ and its clones were, of course, much faster than a TI calculator, and we still can't match that level of performance.
A group of engineers that were willing to keep the managers away and make a simple, limited-purpose system, could surely get excellent performance out of any currently available car computer hardware, but it would come at a cost. Buzzwords would not happen :) Also, the software developers had access to full data sheets for the hardware, and when you told the hardware to put something in its framebuffer, it was there, and no blobs or firmware were in the middle.
I still remember when Windows 2000 launched with "layered windows": an application could opt in to being composited, and it would gain the ability to be translucent, but it came at a very obvious performance cost. Sadly, 23 years later, that performance cost is still there, but it's not opt-in any more.
The kind of development you’re describing happens in aerospace. I know some people who work in aerospace. They’re not any smarter than the people who work in automotive, but the people in aerospace have a much higher tolerance for dealing with bureaucracy, and management is much happier to get something less impressive but reliable.
automative main problem is the constant push to cut costs which leads to worse SoC being shipped. Aerospace profit margins are much higher (or don't matter for gov programs) so they don't need to cut costs that much on computer hardware
> management is much happier to get something less impressive but reliable.
^ this is accurate though. Management wants bells and whistles (animations) which are hard to build without abstraction layers
Even as someone who believes in the indefensible bloat of modern software, I'd say TI-OS on my 84+ (from using it in high school and college several years ago) is not exactly the fastest to use:
- Graphing a plot was slow due to slow CPU and high-precision math libraries (could take 3-10 seconds depending on function complexity and detail)
- MathPrint mode slows down printing expressions to an extent, IIRC even moving the cursor through history felt sluggish.
- The B&W LCD screen with slow color transitions adds more latency than a compositor does.
- Garbage collecting the flash memory could take dozens (IIRC) of seconds, and would happen unannounced when manipulating saved programs (IIRC not in regular calculations).
The main latencies in TI-OS (from memory) boil down to actual computation (graphing, solving equations, etc), the really slow LED, and those occasional slowdowns (GC).
Of course computation was slow on the slow, under-featured CPU, but that was to be expected. My car’s touchscreen system doesn’t do anything that is computationally complex — a computer from the late 90s could run a media player and do work at the same time, and my car’s CPU is faster.
The LCD is what it is. I’m talking about latency too when the pixels start trying to change. Anything with a CRT avoided this problem.
And GC is a fair criticism. But my car’s touchscreen is always slow. Oh, and it also often. takes a minute or two to start working when I get on the car.
Doom does very little work to draw each frame. It does a handful of small calculations to update the game state, and then traverses a BSP tree and draws some vertical blocks for walls followed by a flood fill for the floor and ceiling (although I suspect that was missing on the TI-84 ports because filling is slow). Calculating and drawing a complex graph often requires far more operations.
That's what's wonderful about Doom. The simplicity of what it does belies the effort put into achieving that on limited hardware.
When it come out it was mind blowing and I think it's a testament to what can be done when people actually care about the product.
I suspect the issue with software for consumer goods in general is a stake holder/project management one and not software engineering.
If the stake holder cares more about a tick list of features than how well the features work there's not much the engineer can do. In my experience stake holders are often much more interested in their burn down graphs and Gantt charts than in getting a really polished core which could later scale quickly with just config.
You need a product owner with real vision and understanding of what good looks like. It's almost a cliche now to reference Steve Jobs but he really did get that a few really excellent features are better than a laundry list of stuff that sucks.
I adapted the very old FM/AM, cassette, and external CD changer in my very old Range Rover to have Bluetooth because it doesn't have an aux input and my phone doesn't have a headphone jack anyway, and I want to listen to music or audiobooks off it when I'm driving.
I looked into reversing the protocol between the CD jukebox and the stereo, and using an Arduino to inject the appropriate commands - maybe even get the "next" and "back" track buttons working too - but in the end I just fitted a cheapy bare-PCB bluetooth deelie into the cassette deck and wiring it to permanently think it has a tape in.
It works pretty well, and it looks pretty standard. My phone sticks to the flat 1990s dashboard (remember when cars didn't have to have swoopy curves everywhere?) with a couple of bits of double-sided sticky Velcro, and plugs into a USB-C socket fitted where the lighter socket used to be.
I'm personally a fan of mushroom hooks (3M Dual Lock) besides the high price, because it's less spiky than Velcro and has more lateral rigidity (though unfortunately you need 3 pieces on each side in a triangle for a robust stable mount free of tilting wobbling). Though I actually wouldn't recommend it for mounting a phone; I added a piece to my phone case, but took it off because it kept catching onto my hand and surfaces, and ended up with more entangled dust than soft Velcro.
I actually stuck the velcro to a cheapy Otterbox rubber sleeve knockoff rather than the phone, which has some weird sticky-thing-repelling coating that the Velcro wouldn't stick to.
Not really though. You played a different game using a much simpler sort of engine (probably a raycaster, similar to wolf3d) with much simpler level geometry (probably square on a grid, and probably without stairs, windows through walls, elevating platforms, etc) There are several games like that on the z80 line of TI calculators, for instance Gemini: https://www.ticalc.org/archives/files/fileinfo/247/24742.htm...
But the real Doom isn't a simple raycaster, it uses an engine that exists in the space between a raycaster and a true 3d game like Quake. A so-called "2.5D engine". The level geometry had height data, but couldn't overlap itself (so no navigable bridges unless you fake it using raising and lowering platforms (or teleport the player to another part of the level, like the Build engine did.) Rooms could have arbitrary polygonal shapes, walls weren't all at right angles to each other, which complicated the process of rendering and necessitated the use of BSP trees to traverse the level geometry efficiently.
I know of one instance of such a game engine being created for the 83/84 z80 lines of calculator, detailed in a series of blog posts here:
if you make the UI software more performant the sourcing team just picks a cheaper SoC in order to cut costs. There is little incentive to bend your ass backwards to make performant software when cutting SoC costs doesn't come up on your team's KPIs for upper management
I believe that the engineers are competent at developing software, and think that a lot of the quality issues are due to engineers abdicating authority over the software to managers and never saying "No, adding in that feature at this point in the timeline is going to make this a steaming pile of shit and I refuse to do it and still call myself a software engineer."
I get that we have families to feed, but I've seen far too much of a mindset shift in fellow engineers into thinking that we're warcraft peons rather than professionals. "The business" has engineering feedback as a necessary input, and speaking individually with steakholders they expect this - they'll push until we push back.
There's no incentive for an engineer to do that. Saying yes and delivering crap gets you a bonus, hard truths get you shuffled around or made redundant. There's no real consequence for delivering crap, so that's what happens.
Contrast this with other engineering fields, where the engineer is truly responsible for the decisions they make. My civil engineer friends face losing their licenses, fines or jail time if they are found professionally negligent. The same is true of other high stakes professions - think doctors, lawyers, even accountants. It's probably not appropriate for most software engineering roles, but for safety critical systems it doesn't seem far-fetched to me.
The product team for the touchscreen control system scoffs at the engineering team’s concerns because “customers don’t care, they’re wowed by the touchscreen at the dealer lot.” It’s only after purchase that the regret sets in. Product teams know this and exploit it. The business side knows they’re selling a steaming pile to customers and don’t really care for engineering’s concerns. In most situations they’ll override these concerns forcefully. It’s a hard pill to swallow as an engineer in these companies.
"My civil engineer friends face losing their licenses, fines or jail time if they are found professionally negligent."
These standards ought to be applied more widely and done in conjunction with tightened consumer law. In many cases the quality of electronics equipment has gone to the dogs. I could give instances of appliances I use that can only be used in a hobbled mode—numbers of published functions simply don't work—because their firmware bugs are so bad.
These devices are so bad they wouldn't pass as early developmental mockups let alone early prototypes in a professional engineering establishment. I'm damned if I know why the hell consumers put up with the situation and haven't revolted, it remains a mystery.
I'm not a strong advocate of this, or of jail time in general for non-violent offenders, but as a thought experiment, suppose that Acme Auto release updates to their car's software which make the UI more laggy and less intuitive to navigate. After they do, there are a cluster of similar accidents - distracted driver hits a pedestrian when they should have stopped. These can be shown statistically to affect Acme models with the software updates significantly more than any other make of car, and more than Acmes which don't have the update. A class action lawsuit is started against Acme by both crash victims and drivers. In discovery, correspondence between software engineers is found. Engineer A writes to Product Manager B and says that they don't think the new build is safe, because they were forced to compromise latency performance, and button placement is now more surprising, having changed again. QA Engineer C chimes in and says that since the changes apply to features critical to driving such as de-misting, they won't be prepared to sign off on the change. PM B says that they have to go with the new version in order to meet internal targets on engagement with entertainment apps. They overrule A and C, as company rules allow them to do.
Do you think B should face any personal consequences within a public justice system? Or Acme is just liable for a big payout and then upper management decide who takes the blame?
"Do you think B should face any personal consequences within a public justice system?
Yes he should. Reason, because he now knows the consequences of proceeding if the problem not fixed first (he was told them by engineer A.).
Once aware, everyone has the responsibility to act. The Occupational Health & Safety laws of many jurisdictions are written exactly on this principle. Such laws don't just apply to managers and decision-makers, a floor sweeper who overheard the conversion would also be culpable if it were proven that he did not inform authority of the fact and or if he had good reason to suspect Management would do nothing.
Same for Engineer A, he would still be culpable if after telling Product Manager B the facts and he knew or had good reason to suspect Product Manager B or others responsible did not or would not act to fix the problem. Moreover, unlike the floor sweeper, Engineer A, due to his extensive knowledge of the facts and his senior decision-making position (as an engineer—even if not in charge of marketing or production), the Law would still require him to follow though with either senior management and or external authority until he was satisfied (to the level of his professional ability) that the problem was sufficiently in the hands of responsible others.
Whilst these laws vary between different jurisdictions the common themes are if one—and that's anyone, inside or outside the company—knows there's danger and or potential for someone to be harmed or killed then that person has to act, irrespective—full stop. Second, the more responsible or more knowledgeable someone is as to the consequences of something or some process going wrong then the more incumbent it is for that person to act (the floor sweeper in Boeing's factory would not be expected to know the wrong alloy had been used in engine turbine blades but the engineer would).
These laws were introduced to avoid problems like the Challenger and the Boeing 737Max disasters, and the Purdue Pharmaceuticals opioid crisis. Unfortunately, the US lags behind in either implementing them or making existing laws sufficiently strong.
You don't find the idea both kinda hilarious and somehow vaguely appealing even though it's a bit nonserious at the same time? I love it as a thought experiment.
I neither said nor implied this. To be clear, the products in question were sold under false presences, as they were sold with features that—as far as the lay consumer is concerned—don't exist (just because I'm a technical person and I know they are almost certainly software bugs and or are not designed as per specifications is immaterial). In essence, by deliberately selling a substandard product they've committed fraud.
Here's one of the many examples I could list but it's a clearcut easy one to understand. I have three identical PVRs/STBs (Personal Video Recorders/Set Top Boxes) of one brand and type—so the problem is not just a single faulty unit. These are the type that you add external storage via USB, 2.5" or thumb drives.
Advertised on the outside of their boxes is the statement that they will take external storage to 2TB in size, the scanty manual—if you can call it that—that's sealed in the box which you can't read until one unboxes the device makes a very clearcut statement that the maximum limit of external storage is ONLY 700MB drive (a rather strange limit methinks), and 1/3rd that published on the box. In practice, these units simply wiil not work with ANY external USB drive 2.5" rotary or SSD drives—even the lowest current SSDs of 120GB or smaller—which is in direct contradiction to what's stated on the box and in the so-called manual.
They will however work with thumb drives up to 128GB (I haven't tried bigger). Incidentally, have you ever seen a 2TB thumb drive? Right, I haven't either.
Thats not all, there are software bugs and an UNSTATED limitation that only six programs can be programmed at one time (this is an unheard of restrictive limit, I've never reached the limit on my other units although one type, which has other bugs and problems, says its limit is 32).
I also have three other PVRs but of a different brand (a well-known international mob). All three have the SAME identical model number but two have completely different electronics and their firmware operates in a totally different fashion to the first (clearly built by a different subcontractor), Even the boxes they came in are all identical.
I discovered this when the first unit failed and I bought two more of the same. Moreover, the first unit wasn't even out of warranty so the second purchase was only about six months on from the first.
To make matters worse, before the first unit failed and after getting nowhere with the local distributer I'd hunted around the internet for a firmware upgrade to fix the annoying bugs but couldn't find an ungrade (little wonder if different hardware exists for a given model). The so-called identical replacements are not only operationally very different but they have so many bugs that they are actually unusable. I'm still working on exchange/warranties and such.
Those two brands are not alone in having masses of bugs, I've three other brands—five all up with even more model numbers (yes, I've boxes of these damned things). The bugs in a third band are so bad that it allows one to program the same timeslot on different channels simultaneously—which channel takes precedent and is recorded is pot luck, at other times, about one in three, it fails to record the scheduled program, only a black screen (it switches to blank instead of a channel—but give credit where credit's due, it does switch to blank at the correct time)!
And believe it or not that brand/model has been on the market for several years and it still is without any firmware upgrages being available.
Here, I've presented only the tip of the iceberg—and that's only the PVR/STB story. Where else would you like me to start?
People should not have to put up with this shit, it wastes time and human effort not to mention wastes resources and the environment is clogged up with dead and discarded e-waste and other junk. A simple way around the problem would be to license both companies and their design engineers and threaten them with loss of license for producing junk. With importers, bring in junk and they'd lose their import license.
Implement these rules and most of the problems would soon disappear. In extreme cases where irresponsible designs threaten safety and life then loss of license and jail time would be a just measure.
You'd just get a lot less software as people instituted enough checks to make progress glacial. Not everything needs to be developed like it is a medical device or aerospace software.
Singapore has criminal liability for software malfunctions. I don't think they've sent anyone to jail for a software bug yet, but the law allows for it.
Rightly so, if justified by the consequences—to the extent of causing injury or death.
As with other professions, civil, chemical engineering etc., when the outcomes are the same (people killed or injured etc.) then the punishment should also be the same.
Software design should be no exception to any other profession just because it's common for programs to have bugs.
Moreover, the profession of programming now calls itself Software Engeering, if it wants to play with the Big Boys then it must face the same consequences when things go wrong.
The incentive is having a rewarding job where you develop products you are proud of. Once I have food to eat, this is by far the most important incentive for me and it greatly outweighs e.g my desire for promotions, raises and bonuses. If I can have both, great. If I need to choose one, it’s the fulfilling job and product pride every day.
Yep agreed. If you raise a flag, you'll be looking for work. Head down and build crap, and you have a job for life. I see it all the time. I've lived it.
Sometimes you have to decide, do I build a better system, or do I feed my family. The craft and world suffer, but...
It should be, at least on countries where Software Engineering actually means something, and not a title that one can easy peasy call themselves after a six weeks bootcamp.
I think it's the liability that matters, not the certification -- which usually translates to "X years in a government facility, pretending to learn something which may or may not be misguided and out of date."
I have a 2 month old baby, and got fired from my last job after 6 months of unpaid overtime...
I am currently trying (again) to get into embedded development. I would gladly take the job of the guy that refused to make shitty software, if that means I can keep feeding my baby. Unless it would be too unethical. (I refuse to do work that will kill people, for example I won't work for Palantir, companies that make sketchy software related to flight controls or medicine or other critical applications and so on... but the car media player? yeah, I am willing to make a crappy one if I get the job, I prefer to make a good one, but if my boss want a crappy one... then what I can do? overtime to get a promotion, clearly doesn't work ;) )
We know that phone using drivers perform worse than drink drivers. They regularly kill people because they're not concentrating on what's in front of them.
If you change the car's UI from something with low latency[1] to something with much greater latency[2] then you are definitely putting others at higher risk because drivers spend longer not concentrating on driving.
--
1. See button, move hand, feel large physical button, look back at road, press button, feel feedback click.
2. See screen, move hand, see screen pops up menu on hand proximity, see menu item, click menu item, miss-press try again, wait for animation, attempt to select feature but hand moves due to bump in the road, move hand again, try to select feature again, miss-press try again, wait for feedback animation, look back at road.
Then why is anyone skating for the decision to abandon buttons in the first place? The driver is responsible for paying attention to the road rather than fiddling with the radio.
> Then why is anyone skating for the decision to abandon buttons in the first place?
Cost.
My guess would be that manufacturers want touchscreens because they're cheaper to develop and implement than an inventory of individual physical controls.
Adding a new touchscreen widget to a car that's already in production is just an over the air software upgrade vs a very expensive redesign/recall for physical controls.
Drivers don't think through the consequences of the control system at the time of purchase or have it as a low priority compared to things like purchase price.
I think it's pretty easy to tell yourself that your not going to mess with a screen while driving but in reality it's much harder to fight that compulsion. If it wasn't then we wouldn't need the "I'm Not Driving" feature on phones.
I don't find it hard to resist the compulsion. If I'm driving, I don't look at my phone. Seems like a pretty simple rule to follow.
Anyway, I understand that they're doing it because of cost, but if you're going to start saying the guy who worked on the infotainment system has blood on his hands because he implemented some animation that takes half a second then surely the person who put a touch screen in the first place is more culpable.
We completely understand - and we would all
make the same decision.
Which is why (in the UK) I view the rise of strikes and unions positively - workers need to hang together - set standards that they won't go below, and also ensure the spoils are fairly shared out.
Recently the deputy governor of the bank of england said "inflation has made everyone poorer - get used to it not ask for wage increases". Which tone deafness misses the point that this is not about wanting a 10% pay increase - it's about how is society structured and how do we share the vast wealth.
The question is not how do I get more, it's how do I build a fair system that inwill be part of.
we are playing the sociopaths game as if it's the only game in town. There should be no game we all play where the rules include not feeding babies. That's a bad game
>"inflation has made everyone poorer - get used to it not ask for wage increases"
Which is absolutely horseshit, because the European countries that had the least amount of inflation (besides the super wealthy Switzerland and Luxembourg), were the countries where salaries are automatically indexed with inflation, like Belgium.
So it turns out that forcing companies to increase wages by inflation is a very effective deterrent against price gouging and inflation, yet those in charge try to convince us otherwise, that we should accept lower wages for our own sake lol.
Can you believe the nerve of these people? It's disgusting, but we get what we vote for. You want better, then vote better.
This doesn't help. Even if you were the guy that read all policies or whatever a candidate pushes, the vast majority if people are still just gonna vote with their emotions.
The industry works as follows: the OEM (Daimler, Toyota etc.) says they want a software spec implemented. Other companies (Tier 1 suppliers) bid to win the contract. The cheapest usually wins.
If you have a company and refuse to take the crazy deadlines and low quality and low pay then don't worry, there is another Tier 1 supplier across the street who will do it for you. OEMs know that you need them more than they need you.
And on top off that, some product manager at Mercedes might say they absolutely need let's say Atmos in the next release and ask the Tier 1 to implement it. The Tier 1 usually uses multiple Tier 2s. The Tier 2s say they need more memory, but that's not practical at all because that hardware was fully validated years ago and you just don't make incremental changes to automotive hardware, and there won't be a hardware refresh for another 4 years. It is in none of the tiers' interests to say they can't do it and lose out on a multi-year contract so they do the best possible job within the constraints.
That’s fine. But the worst case response time across the entire UI must be the first item on that spec! That spec is of course the responsibility of the OEM to create. Where this goes wrong I don’t know but competition, price sensitivity etc doesn’t explain it. Having soft and hard limits for response time seems obvious and someone either forgot, or they had a meeting where they (the OEM, after deciding on a solution and having it implemented) said “ok we can save $20 on the BOM by going for a cheaper SoC if we accept 200ms response times instead of 50ms” and that should basically be criminal due to the safety aspect.
It's not criminal so they will continue to lower the cost. 50ms quickly becomes 500ms response time when you realize that no tier 1 supplier can hit within your OEM budget and your boss is becoming impatient :)
Another thing that happens is when there is a 50ms response time mandated and at the start of the project you have 30ms and it slowly creeps up until you get to 55ms. Then the blame game starts. Like I said previously, each component is usually won by a different tier 1 supplier. So everybody puts the blame on somebody else. One tier 1 is building the linux distro, the other is making basic system libraries and the applications are written by another 20 tier 1 suppliers.
Then the deadline hits and the car needs to be sent to the showrooms. So whatever 50ms was chosen initially is changed to whatever it currently is so to say they are within spec and can ship the damned car.
I think this describes a lot of how software is developed (which just goes to show how immature that part of the industry is.). Similar contracts and agreements exist between manufacturers of the brake system as well but if there is an issue there where the ABS system doesn't prevent lockup in 1% of cases, then the problem is solved or the car launch is delayed, because that's what the system is designed to do. The problem here of course is that these things are seen as gadgets without safety aspects and there is no regulatory oversight.
I guess it's also up to buyers: don't buy cars with shoddy infotainment.
ah, there is some regulatory oversight and there's awareness of the safety aspects.
Not much, but, there is some.
After working on infotainment, I'm doing my best to avoid purchasing a car with an infotainment system. I had to compromise and got a Civic with an infotainment system for my wife, but I'll probably stick to late 2000s, early 2010s for as long as I possibly can to avoid them. Seriously hate touchscreens, and there's just not enough value in an infotainment system for me to be OK with what I'm losing by having one.
Something missing here is that the Tier 1 supplier will take one look at the spec and know that latency hasn't been mentioned or considered.
They will not tell the customer this because either the customer doesn't care, so why waste time and resources
...or when the customer realises they will open a Change Request to fix the issue. Change Requests are how you make an actual profit on unprofitable, low-balled contract and probably gain an extension on the unachievable timeline agreed to win the bid in the first place.
The "customer" is the person that signs the contract not the user of the product. They probably don't care that it's a pig to use. It just has to look OK in a presentation to their boss (...who isn't going to ever use it either).
If Tier 1 suppliers didn't behave this way they wouldn't be able to pitch bid responses cheap enough to win bids. The responsibility for crappy products lies entirely with the product owners. Only they know what's good enough.
From what I observed product owners know that the system sucks and it's laggy. They also know that the OEM management only keeps repeating "cheap, cheap, cheap", "with a renewed focust on costs" and other sayings like these.
That was true and it might be true for most of the production process, but as you may read in this article https://arstechnica.com/cars/2023/02/mercedes-ceo-tells-ars-... they are actively working to change and speed up the software development process:
Källenius explained the current system to Ars Technica like this:
"So some engineer in Sindelfingen comes up with a concept. You have to write that down. You have to send it to the supplier that needs to be quoted. Then procurement people need to negotiate with each other. Then that supplier goes to some sub-supplier in Eastern Europe and wherever they do. It goes back up the chain again. It gets tested and nine months later, you have actually changed something in your infotainment system. Now you go into the ESH [Mercedes' electric Software Hub]. To say, let's change this and you just do it."
Working on a product vs working on a project. The latter means that the usefulness of the thing produced is a non-goal because the customer is not a user.
It's very rarely a developer's role to say outright "no". It's our role to make trade-offs understandable by decision-makers, and to clearly articulate why we think something is a bad idea. Sometimes (usually...) the broader business has different goals that just delivering quality software, and I think part of being a professional is understanding that.
There is, of course, a time and a place for a hard "no". I've genuinely threatened to quit rather than implement a particularly user-hostile feature in the past.
I completely agree with you, and I think you've worded this better than I have.
I was trying to advocate for the middle path, where there's healthy communication from both sides, to the point that developers trust when "the business" makes a decision - not to the point of engineering completely blocking the, unquoted, business.
Yep. It's part of your job to tell managers "yes, it's possible, but it'll negatively affect performance in a significant way because of such and such hardware limitations, so it's not the most valuable feature to implement at this time".
And it's part of the manager's job to tell you "I agree but don't care. Do it anyway or I will find somebody else to do it. For cheaper." Always remember that you are replaceable.
When the newborn is crying and wants food you do what you've got to do. And there are many people like that.
Most of the time the hardware team doesn't get to choose their hardware. There is usually a budget and in automotive you can select only hardware that has some specific certifications for use in a car so that narrows the list even further.
And once that hardware is selected for a given range of vehicles, it's literally impossible to change. Updates to some parts of the software stack are also basically impossible at a certain point.
It’s your job to tell them if the requirements cannot be met and that something’s gotta give. It’s his job to decide what, and it looks like he’s choosing latency When push comes to shove.
I don't own the company. I will push back a bit, but if management insists on using the foot-gun, why would I get on their bad side? They are adults as well and if they don't agree with me, I do as they tell me.
I often see sentiment like this here but I think there's a disconnect between the sacrifice you're expecting people to make and harm caused by the work. It's admirable for someone to put their own job security second to obeying really harmful directives, but something that's going to make the music system laggier?
Even if the infotainment has been in development for 15 years, all new cars I've played with seem less responsive than a 1st-generation iPhone despite having a UI that is no more capable. Either the engineers aren't making the most of the hardware or the hardware is underpowered by a few decades.
I can imagine that customers are annoyed by laggy UIs but don't weigh that very highly when deciding to buy a new car.
Especially if the competition is roughly equally slow, the costs to engineer a responsive solution might be to high for its impact on sales.
Consumers, being used to feature rich phones, may indeed value feature count over quality (for the entertainment System). Despite what OEMs claim a car is first and foremost a means of transportation and a "software Plattform" only after that.
I think it's a growing factor and the linked article would indicates that manufacturers are receiving feedback about it.
It's not clear whether that feedback is from sales channels (i.e. potential customers are shunning particular products) or from service channels (customers are reporting issues that require additional development spend resulting in profits being eroded).
Personally, I've been looking for a new car for 3+ years and the quality of controls (which are nearly all touchscreen based these days) is a major factor.
This is also a reflection that trust in car manufacturers to address software related issues is low - they are used to designing and building cars that are churned out in 3+ year cycles. Moving to a scenario where the product is changing monthly is a huge change. My 13 year old current car is extremely solid and the controls don't change month to month - something that does happen without any customer choice in more recent times. :/
customers are annoyed by laggy UIs but don't weigh that very highly when deciding to buy a new car
UI is not something you can evaluate on a 30 min test drive under sales pressure. Sometimes there’s no way to test it (no rain, no snow, no free straight road, no immediate glitches). And there’s no reviews that cover it. It’s not a weighted decision, it’s a decision out of only parameters that are most obvious.
The same way you play a lottery with e.g. a washing machine, whether it beeps after unlocking the door or you have to go and sit near it like an idiot for a whole minute.
Seem is doing a lot of lifting there. For one, there is no generation of iPhone that is safe to use while driving. Such that you have a ton more tolerance for a phone than you would a car system. Mostly for good reasons.
It doesn't help that iphones from that time frame aren't really in use today. Car systems are.
Also, maybe the expected lifetime of something used to turn up and down the volume and show a map while being in a fixed case and not even having its own battery is different than a device running on battery, thrown around in everyday life and literally being pressed on 0-24. And even then an iphone from the last 5 years might well be expected to run for 5 more with a battery replacement at most. Mobile hardware went over a huge change in this timeframe.
I confess I am not a frequent driver. I do frequently get mad at my phone. And my computer. Though, not as often as it seems the average hn visitor does. :)
I did not mean my post to dismiss the idea that we should expect better. I'm just worried that the practices we put up with elsewhere in software would not, necessarily, be the answer.
There's a 0% chance 4 yr old hardware can't handle what GP is asking it to do.
> Product people have been picking at the design and asking for changes way too late
Sounds like classic everything-bolted-on-never-refactored-always-add-features you see in the vast majority of companies. Which is fine if you're making a social app or CRM but not okay when talking about heavy machinery. No company ever prioritizes making things snappy and it's ultimately always on the engineers to push back. Sorry, this one is still on the software developers to me.
That still just sounds like people not knowing what they're doing.
In no world would automakers say, "Hey you know, we just did all of this tire engineering with this rubber, but let's change the composition at the last minute and not retest all of its physical properties."
So, why should it be any different with software? Because some sucker PM didn't have the brains to say, "We can't actually do that."?
It feels like there’s some derision towards the infotainment software engineering since it’s perceived as not safety critical. It’s in the name, “infotainment”! Fun and optional! It’s not like you’re working on the PFD/MFD/EICAS on an Airbus. “So what, the seat heater button lags sometimes. Not gonna kill someone like the tires. Ship it” says some MBA higher-up.
But then the seats kick on automatically when the car starts and the driver tucks in with her morning coffee for a commute down the highway. The coffee kicks in, the cabin heats up, and suddenly that warm seat is like sitting on a griddle. She taps on the menu to bring up the seat controls, then lag, then has to hit the seat heat button twice to go from “high” to “low” to “off”, lagging each time. Oh, she hit it three times, now it’s on high again. Tap, tap, tap, tap… Meanwhile she has crossed three lanes of traffic and is staring down a jersey barrier at 70+ mph.
I know you're getting a lot of pushback here, but you're totally right.
Qt (among others) has been a thing in the embedded space for decades, and we've had snappy UIs on sub gigahertz CPUs for decades. I just don't buy that's it's a matter of under specced hardware.
I'd actually buy that it's more a matter of the corporate sausage making machine that turns ok hardware and talented developers with good intentions into an output of barely usable poorly performing crap. Ie: exactly how we end up with something like Spotify.
> I'd actually buy that it's more a matter of the corporate sausage making machine
This. The devs don't have a say in the tech stack. The decsion is usually outsourced to the lowest bidder. Other than Tesla and Rivian, I can't think of any OEMs that I've dealt with who have full control over control over their display. Almost everyone outsources it to the Koreans who have a lock on that market – i.e. OEMs send in their designs but it's actually implemented by LG, Samsung etc. From what I've seen, any changes or new additions to any on-screen components has a 2 year lead-time unless (not a typo) it's safety critical or catastrophic in some other way.
The corporate sausage making machine is merely only doing what it's customers want, for the price they'll pay, nothing more, nothing less.
A lot of these are good impulses but also feel like an unfunded mandate, like rear back up cameras being mandatory - obligating everyone pay a thousand dollars more every year.
The feel good comes from the DOT, the cost gets paid by the customer.
It's not that that the sucker program manager doesn't have the brains, or at least less so than PMs in other fields. It's that the PMs in automotive are not empowered to say that. There are 7-8 layers of management above them, all of whom are being shat on by the layer above them, and there's a greater disincentive to causing delays to a vastly interconnected program vs. staying in their lane and polishing the turd they've been assigned the best they can.
Another big problem is a very high employee turnover and there is no redundancy in roles and skillsets because of severe cost cutting through the ages. Tech is chosen by senior engineers but vision, knowledge and experience are lost and implementation suffers every time anyone moves companies. My company has been working on an automotive project for the last 3.5 years. I deal with around 10 people on at least a weekly basis. Out of those 10, only one was around at the start of the project. We're also working with another Tier 1, where only 2 out the 12 have been around for 3 years.
Dude Winamp was a clean, simple and wickedly responsive UI that allowed a multitude of real-time audio tweaks and supported playlists with thousands of files. This was in the late 90’s running on 166Mhz processors with maybe 16MB of system ram.
Are you saying modern car infotainment systems have fewer resources allocated to the UI?
You’ve just reminded me that I lived in a time when my desktop machine was not powerful enough to decode and play MP3s in real time. I had to decode to wav files and play those.
This was in a time when far fewer instructions were accelerated on the cpu, and my cpu at the time was something like a 66MHz 486-dx2, and even the (Frauenhofer) mp3 codec wasn’t as optimized as it was later.
>Are you saying modern car infotainment systems have fewer resources allocated to the UI?
Relatively speaking: Yes.
Response lag is always because the hardware is chugging to process all the bits in time (they're not in time). Between underprovisioned hardware and shittily written software, hell yeah fewer resources are allocated to the UI.
We need a small supercomputer's worth of resources allocated to bring the software of today up to snap like it was the 90s.
Nearly everything automotive now has a GPU. When you have a GPU, even a very low power one, graphics get far easier. simple operations like "fade from this screen into that screen at 60 fps" become easy. They would be impossible with CPU graphics, simply because a slow CPU can't update every pixel on the screen at 60 fps anyway.
Despite that, automotive graphics ramain laggy because they normally build something barely gpu accelerated using java or QT or something else laggy by nature...
At the highest resolution of the GP comment, 800×600, and a 240MHz CPU/MCU and 8MB RAM, 60fps smooth updates for things like scrolling, scaling, and drawing new buttons and icons should normally be possible with the CPU alone. Fading and blending large areas of the screen won't be, but small areas will be fine.
Basically anything that involves copying pixels or solid fills without per-pixel calculations. 60fps at 800x600 is 28.8Mpixel/s. Copying and filling at that rate should be achievable by a 240MHz CPU with some tight coding. The rest is 2d geometry, mostly axis-aligned, with relatively few edges. That's not demanding at all.
(I used to write 2d and 3d rendering engines for slower CPUs than that.)
A GPU is necessary for fancier effects like fades and shading, but the basics can be smooth on a CPU and the fancy stuff too if confined to smaller areas.
Also, for the car, people would be happy enough if it could do a consistent 30fps I think.
Those annoyingly slow GUI updates you tend to see on TV set top boxes are not slow because of the hardware... it's usually the software not being written with sleedy rendering in mind.
All that said, you're right that if there's a GPU or even just a Blitter, it should be used and then the CPU speed almost doesn't matter as the geometry calculations are so simple.
Automotive is different. They're selling a $30,000 car attached to every CPU, there's no reason to have to penny-pinch on the hardware, when an additional $50 would quadruple the specs.
It's great to get your perspective on this. Aside from the GPS/navigation features, aren't car infotainment systems extremely basic?
We've been controlling heating, cooling, audio source, and audio volume since the dawn of time.
How are technical requirements related to these basic functions changing such that their UIs haven't been absolutely perfected by now? Is the entire infotainment code being rewritten all the time so there's no chance to iterate and improve on the previous versions work?
Yes. The infotainment code is rewritten every time. By different devs. Because that is how automotive industry works. This time BMW wants infotainment and a third party wins the contract (because it's the cheapest usually). For the next BMW car another company wins the contract. Everything is rebuilt from scratch.
The components that make up the infotainment a lot of times are actually developed by different companies. So navigation is one company, media player another etc.
Also, the deadlines usually make no sense and there is zero time for any optimization work. It's not the devs, it's the way the automotive industry works outsourcing every little button on the dash to another (cheapest) company. They just don't understand that that mentality doesn't work with software. Most of them are mechanical engineers not software people.
I'm currently going through a quote cycle and one of my requirements for the component is a sub 8min boot, which the tier 1/2s are pushing back on as unreasonable. The insanity is real.
That can include booting after an update and self tests, but yes, nothing about the component should take that anywhere near that long. We just needed a number to design the manufacturing workflow and assumed it was a reasonable ceiling. They simply don't want to do the (potential) optimization work as part of their initial contract if they somehow manage to exceed it.
> Because that is how automotive industry works. This time BMW wants infotainment and a third party wins the contract (because it's the cheapest usually). For the next BMW car another company wins the contract. Everything is rebuilt from scratch.
Yeah, that's how the automotive industry works. For each car BMW will change the third parties manufacturing each piece and every model is completely rebuilt from scratch. They literally reinvent the wheels all the time.
You shoudn't read too much into the fact that you have a dozen or two of models listed under each version in https://en.wikipedia.org/wiki/BMW_iDrive because it's just a concidence that starting each time from scratch they arrived at the same thing.
I've never understood why that is happening. Are they trying to make sure there is no knowledge accumulation at a single tier 1 supplier? Kind of make sure nobody can create a competing product and just sell it to different OEMs?
I actually know for a fact that a new iDrive version was reimplemented from scratch even though it was just an upgrade from the previous version and there would have been enough code to reuse. Maybe he was sarcastic but this actually happens a lot.
That's very different from "This time BMW wants infotainment and a third party wins the contract [...] For the next BMW car another company wins the contract. Everything is rebuilt from scratch."
(And "code being rewritten" is very different from "no chance to iterate and improve on the previous version" when it comes to perfecting the UI - but it wasn't you who introduced that confusion. Rewriting the code could actually facilitate incremental improvements to the interface - and arguably that's what BMW tries to do.)
No. It's not "very different". You're just trying to be contrarian. 3 series and M3 has 90% the same infotainment. But 5 series and M5 competition is being developed by a completely different company. Even if they are 95% the same. Nobody is iterating on anything. The same climate control is implemented 20 times by different companies. It's just that the 4 variants of the 3 series usually are made by the same company.
"For the next BMW car everything is rebuilt from scratch" does sound very different to me from "the system is developed by a different company in each multi-year cycle that covers dozens of models".
Maybe you're trying to tell me that I should have understood that you couldn't really mean what you wrote because that's just absurd? In that case you may be right.
I guess that when you say "nobody is iterating on anything" you cannot really mean that either - because it's just equally absurd to think that completely independent systems were designed from scratch multiple times without any reference to the previous ones and it's just by chance that the first version had a dial only, the next "non-iteration" a similar dial and two buttons, later more buttons appeared, etc.
For what it's worth, at least in German auto companies it's becoming the norm to demand source code and build tooling, precisely to avoid lock-in and constant reinventing of the wheel.
>Most of them are mechanical engineers not software people
Ah yes, the trillion dollar industry (that's been writing software in many forms for decades) just doesn't have people that understand software.
Meanwhile, you have the "software" people at, say, Tesla, who has the worst driving interface of all and the iPads were melting out of the dash in the summer time. But at least the latency was low!
I wouldn’t necessarily the engineers, but I’ll certainly blame the designers and product managers.
I absolutely hate my Acura’s infotainment system. My “favorite” feature isGPS autocomplete.
You type in part of an address (which of course has a very slow response time), then you pick the completion off the list.
Sounds fine right?
But when you click it, you’re met with a SLOW ANIMATION OF THE COMPLETION BEING TYPED INTO THE SEARCH BOX. Motherfuck dude! What the hell is that? Why does anyone want to watch the keyboard flash for five seconds?
my experience working with industrial manufacturing is that there is ALWAYS a push to cut costs and the SoC that runs your software is usually the first thing to get to the chopping block. Given it is extremely hard to gauge how well software will perform before it is actually built and the SoC is chosen way ahead of time it leads to bad performance
And from the other side, there is a big push to add more and more fluff to make the UI design sleek, you could make it performant if you used DOS-style interfaces but no product manager is going to allow that and making animations from scratch (vs using a platform like Android or browsers that has a ton of abstractions to create fancy UIs) is extremely costly
The kind of manager I’d happily walk from. Either way, I won’t be doing pointless work that inevitably leads to failure - said manager can enjoy their low quality development team and failed outcomes.
When the managers bosses see the manager failed a project after firing all the people who said it would fail, it won’t look good for them.
Plus 400 layers of abstraction because the lowest bidder doesn't know any actual software engineering and every time they face a problem they add yet another library to the project that someone on SO mentions because the accompanying sample code in the answer has the least amount of lines.
This absolutely boggles my mind. I had to do some javascript recently, and every time I had what I thought was a very basic question the most upvoted answer was "just npm install this library, and use this one-liner". I honestly never experienced this before with any other language, it's horrifying.
The solution is quite evident is it not? Regulatory bodies to set response latency and limits with over-life degradation as an additional parameter.
Only reason "they’re just being asked to turn lead into gold and often get silver." happens is because automakers know they can get past all regulatory approval with lead/silver or whatever metal you prefer.
Yeah this is nonsense. If not for hardware considerations, probably some bureaucratic nonesense. Disputes between teams, or suppliers*, something fell through the cracks because it was nobody's problem; stuff like that.
Probably not because they lack 10x l33t coding skills, and have fat fingers.
* I'm not sure if people realize that many automotive dev teams are buying enormous stacks of software from outside parties that often have motivation to throw you under the bus... if it's not an external supplier, it's probably another team on a different continent... but maybe it's different in infotainment?
Unless the hardware can and will be upped from a $20 to a $200 SoC because it’s required to keep response times low, then someone isn’t doing their job. There has obviously been mountains of specs during these projects, but the most important one - the response time - was obviously overlooked.
I’m not saying like the GP that this indicates inexperienced developers, but it does indicate poor technical leadership and priorities. Further, good/senior developers would have been more likely to call this out.
A higher powered SoC might help make it lag-free... But the real solution is simply to put in the requirements that 90% of taps must be fully responded to within 100ms. Animations must complete within 350ms and be 60 fps. Dragging must have a glass to glass latency of under 50 ms. Booting from door open to usable must be done in 5000 ms.
While we're there, lets put in the requirement that a random joe from the street, with no training, when given the UI and asked to complete a task (eg. Play me some coldplay, turn on the window defroster, pair a bluetooth phone, get directions to the nearest gas station), must be able to figure it out within 20 seconds and 6 taps on average.
> In my experience it was almost always a matter of the hardware being stretched way past its limits.
I find this hard to believe. An Intel 486 from 30+ years ago could run Doom full screen with nearly instant key press response. A 6502/6510 C64 could repaint the entire text screen each frame. Why would today's auto hardware require 500ms to respond to touches on a static menu?
Car infotainment as responsive as a 2004 iPod would be earth-shattering. I refuse to believe that in 2019 the hardware to do this was a material expense on a $25,000 car.
We had responsive hardware/software 20 years ago, and car UIs are not doing anything out of the ordinary. To what do you attribute the low quality then?
My top of the line Android Auto compatable head unit is pretty ordinary. A RAM mount on the windscreen holding my phone is significantly more responsive!
I used to be an infotainment software engineer for an Auto maker. There’s this idea in the software industry that because the software behaves poorly on your car’s system, it must’ve been written poorly by someone who didn’t know what they were doing. In my experience it was almost always a matter of the hardware being stretched way past its limits. Your brand new 2024 car’s system has been in development for 4 years. Product people have been picking at the design and asking for changes way too late, and someone in accounting has asked the hardware contractor to cut more cost off the BOM.
My colleagues were tremendously talented. Some of the brightest engineers I’ve had the pleasure of knowing. The developers aren’t lazy or inexperienced, they’re just being asked to turn lead into gold and often get silver.