
The Quality of Embedded Software, or the Mess Has Happened - DmitryNovikov
http://www.viva64.com/en/a/0083/
======
ChuckMcM
I agree it is a bit sensationalist, and certainly disparaging of EE's, but the
truth that there is more and more software between you and an unpleasant
experience is absolutely true.

One of the things that really annoys me is that there isn't more openness and
sharing in the embedded space. Its not uncommon for me to go to a manufacturer
lecture where the only compiler choices are proprietary offerings and their
library functions and includes are riddled with proprietary licenses. In one
case I pointed out bugs in their (re)-implmentation of some basic libc
functions that were fixed in newlib years ago and pretty much everywhere else.

While the people around me fumble with their Windows [1] machines I pull up a
gcc cross compilation chain and begin porting the proprietary "workspace" or
"project" files into a bog standard make file. I've gotten to the point where
I can have their 'demo lesson' running in a fully open toolchain faster than
they can get the rest of the attendees windows and custom package install
issues sorted out. Of course I've already got the basic tools installed, and
generally the other students are installing an older version of the same
toolchain I'm using but into some incompatible set of directories that just
the vendor uses.

I went to Atmel, TI, and ST Micro chip seminars (looking for a processor for
my educational development board) and had I installed the proprietary tools I
would have ended up with three copies of GCC installed on my laptop!

I've been supporting efforts like the libopencm3 desire to build an open and
common ARM support library, but more of that needs to happen to build in some
good practices for these embedded developers to learn from.

~~~
nickff
> _One of the things that really annoys me is that there isn 't more openness
> and sharing in the embedded space._

Embedded products are time and capital intensive, which means that the cost of
a compiler or other tool-set is very small relative to the total cost of
product development. Because of this, the developers (and their companies) do
not mind paying a few thousand dollars for a new piece of software. Anything
the developer creates is often only useful for competitors to one's product,
so there is very little incentive to share.

> _I 've gotten to the point where I can have their 'demo lesson' running in a
> fully open toolchain faster than they can get the rest of the attendees
> windows and custom package install issues sorted out._

If you spend a year working on developing an entirely new hardware & firmware
product, you see that importing new files, finding libraries, and adapting
software from one IDE to another is not a very important or time consuming
part of the project. When you spend a few weeks discovering that there are
undocumented hardware glitches in the integrated circuit, that the errata is
inaccurate, or that you own hardware behaves unexpectedly, the IDE is of
little consequence.

> _I 've been supporting efforts like the libopencm3 desire to build an open
> and common ARM support library, but more of that needs to happen to build in
> some good practices for these embedded developers to learn from._

As an embedded developer, the open ARM library is of little interest to me.
ARM is proprietary anyway, and building open-source projects on top of closed
source systems does not seem to be a productive use of time.

------
pkolaczk
I remember during EE course we had an assigned project to write some software
for 8051 microcontroller, in assembly. I organized my code into small
"functions" and used a consistent stack-based calling convention with pushing
all the resisters on the stack and restoring them back on exit, just as if it
was generated by a C compiler. The teacher looked at this and said "huh, look,
this is not a CS lab. We do not do things like this in embedded software. Why
don't you just keep everything global and use absolute addressing instead
those crazy push/pop calls? And you should be reusing registers..." Anyways,
it was lot of fun.

~~~
robomartin
On a resource and computing-power constrained system used for a very specific
function it would not be too far off if one said something like that(with
caveats). Anything past that category and he would be dead wrong.

In other words, if you only have 2K or RAM to play with, a limited stack
depth, a CPU clock of, say, 12MHz and a 12 clock instruction cycle you have no
choice but to get very clever.

Pushing and popping a bunch of registers on every function call could rob you
of much needed performance and resources. In fact, on such a system that alone
could cause unintended side effects and introduce potential safety issues
(talking real-time systems with a requirement for deterministic response). You
could also have a situation where you overflow the stack and things go from
bad to worst very quickly.

BTW, this, among other things, is one of the reasons I abandoned such small,
8-bit, resource-constrained architectures for most embedded projects a long
time ago. The 8051 derivatives (Silicon Labs) have always been some of my
favorites. I might still use them for small projects such as small control
panels or simple sensor pre-processing. These days you can buy 32 bit
processors running in the tens to hundreds of MHz for $10 or less. These
provide you with the ability to write "proper" embedded software with all the
luxuries of greatly increased resources and operating speed.

~~~
hfsktr
I like the way you explained it. The way it was written above seems like he
was just told he's wrong but not any of the reasons why. Might just be my lack
of knowledge in the area but it made the teacher's response more
understandable.

~~~
pkolaczk
There's a huge difference between answering "beware of blowing up the stack"
and "in embedded we don't do like that".

~~~
makomk
Of course, Toyota failed to make sure they didn't blow up the stack _as well
as_ using a ridiculous number of global variables and poorly organised code.

------
kabdib
I know a good many bad firmware engineers. A few of my theories about why:

\- Many of them have a concentration in EE, not software. The software they
produce is what you'd expect from an amateur.

\- A lot of engineers do not spend time to think about the whole stack, from
decisions at the hardware level all the way to how the user interacts with the
product, and how the factory produces it. The more comfortable you are working
in these domains, the better decisions you will be making. Unfortunately, for
many firmware engineers their responsibility stops at (say) the USB protocol
they've hatched.

\- Firmware engineering is often seen as a lower-class of programming than
desktop. (To this, I say: Ha!)

I've sure seen a lot of crappy firmware, much of it from contract houses that
specialize in things like BIOS and board packages. They don't seem to hire
carefully.

If you run across that rare gem who can write desktop apps and UI, write host
drivers, write firmware, and use a scope, hang onto that person.

~~~
jmpe
I know a few good firmware engineers. A reason why they, too, produce "crap"
is:

\- Many of them come across hardware errors or suboptimal designs during a
later stage in the FW development. In most cases they hack a workaround that
doesn't involve a redesign to be able to move on with the next item on the
list. Folks higher up the chain of command see this as an excuse to avoid a
redesign: "we're not going to do a redesign for that, Mark fixed it in
software". Examples: pretty much every ARM System Board out there.

\- Someone picked a device that was available during the design phase, but
went EOL a few months later before production. It happens.

\- Someone found a cheaper device than originally planned but it doesn't
really do what it says on the box. Typically peripheral stuff like ethernet or
usb.

\- And here's the big one: !!Crappy Datasheet!! Some companies, some really
really big IC companies produce crap datasheets. Yo get accustomed to the fact
that this new Cortex-M µC will probably have an errata or updated datasheet
every few months.

\- the redesign fixed their EMC problem to get the certificate, someone fixed
the power supply. But now you get into trouble if you drive this and that
device because the voltage takes a dip.

But I'd like to add: I agree with your post.

------
diydsp
This is not about the state of embedded software. It is about one program, the
accelerator control of a Toyota.

The author of the article is linkbaiting and fearmongering to get you to
purchase his/her static code analyzer for C++.

As embedded software becomes ever more complex, it will benefit from
methodologies developed for other platforms, such as desktop applications,
etc. That is all that needs to be said. Please refrain from the "EEs don't
know how to program" comments. Thank you.

~~~
robomartin
> Please refrain from the "EEs don't know how to program" comments.

No, no. As an EE I can tell you that lots of EE's really suck at software
development. They never really took the time to study CS to a reasonable
depth. As a result you see code that is just horrid and really naive. I've
seen stuff that could blow up boards Hollywood style in a microsecond. Such
things as potential timing conflicts turning on two MOSFETs such that you have
cross conduction and draw hundreds of amps due to the direct short from +V to
GND.

Now, on the other hand, "desktop" programmers are equally --if not more--
dangerous than ignorant EE's. They have no clue and no experience with hostile
embedded environments. I mean, the simple startup configuration code of some
modern microcontrollers is the kind of thing you'd never see in desktop or web
development. Then you have a myriad of other issues: watchdog timers,
drooping/unreliable power supplies, static discharge-induced reset, load dump
in automotive, similar applications, or noisy signals, sensor and key-panel
data, potential single bit flip situations in aerospace/radiation
environments, thermal issues, etc. The list is huge.

These are issues that are far better understood by an EE. And so, the best
embedded software developer is probably going to be an EE with real solid
training of CS principles as well as an solid understanding of defensive or
safety-critical electronic and software design principles.

An example that comes to mind was when we designed a high-power motor
controller many years ago. This thing could handle thousands of Watts. Testing
included full output short circuits, zapping it with static electricity,
exposing it to RF via close proximity antennas and coils and other torture
tests. We discovered that under very rare conditions the output to go to full
power when power was first applied AND the power connection was, I'll say,
uniquely noisy. Imagine a corroded connection to a power strip on a marine
platform that is moving. When power is applied, if you look at it on an
oscilloscope, it looked like the most horrible noise you'd ever seen --except
this noise was actually powering the board. We had to build a custom power
tester using old motor brushes and an old commutator driven by a small motor
in order to simulate ugly power supply scenarios. We later replaced this with
a programmable digital system that could do the same. The embedded software
and the hardware was then modified to deal with these ugly conditions and much
worst cases as well. The thing was nearly bulletproof.

So, yeah, drop any EE in front of a compiler and you'll probably end-up with
really dangerous code. An EE with the right training and lots of scars would
be the best option. Someone with a bachelors or masters in CS and no EE
training and embedded world experience could be scary dangerous.

~~~
nickff
I agree wholeheartedly with this comment, as computer and embedded programmers
approach completely different tasks from completely different paradigms. The
embedded system may have to run for years on end, with no one pressing
control-alt-delete, performing a reset, or an update; it may also have to be
very robust against unexpected situations, fail-operational or fail-safe or
fail-deadly depending on the product, and protect itself from exposure to
dangerous signals and environments. On the other hand, computer software must
be easily maintained, updated to add new functions, work on new systems, be
easy for the end-user to operate, and limited in scope. Expecting people with
such different tasks to behave similarly is entirely unrealistic, and
foolhardy.

------
_pmf_
Ah, yes. Comfortable desktop developers theorizing about embedded software.
MISRA is more about compatibility with non-standard (or pre-standard) legacy
compilers from the 80s and miniscule improvements to the static analyzability
of code than about software quality; following MISRA to the letter would
actually decrease the maintainability of software significantly, and this is
the only valid measure of code quality any practitioner should care about.

> via an onboard network like CAN or FlexRay, a more complex superstructure
> over CAN

FlexRay is a completely different bus system.

------
pjc50
If you like this sort of thing, read comp.risks (
[http://catless.ncl.ac.uk/risks](http://catless.ncl.ac.uk/risks) ). On the
other hand, it might give you an omnipresent sense of fear about all the
computer-controlled safety critical devices around you.

------
RougeFemme
I used to work in a world with lots of embedded software. My experience was
that - regardless of domain or company -the "hardware" was regarded as just
that - simply hardware. And it was tested as that - simply hardware - not as
hardware that, oh by the way, had a helluva lot of critical embedded software.
So issues that _could_ have been caught at _some_ level of testing weren't.

------
ausjke
I read that report a few days ago. It's indeed shocking. One way to explain it
is that, the split of EE and CS in universities, most firmware engineers are
EE graduates, who knows the hardware to the register level, but not as good on
software, not at all. The life-threatening related products, such as cars,
medical devices, rockets,etc, probably some strict certificate on software
development is good, otherwise it's totally subjective. The so-called computer
engineering major is a nice try, students take courses in both EE and CS
fields.

------
memracom
This is why the world needs something like SEMAT
[http://semat.org/](http://semat.org/)

More and more of our environment is controlled by software and when it breaks,
it doesn't just shut down somebody's store but it does real damage to real
objects in the real world. Esepcially electrical engineers should be following
rigorous processes in building and testing software.

------
Mikeb85
Not that this exonerates Toyota in any ways, but if your car is accelerating
out of control, why not just take it out of gear (ie. put it into neutral)?

Or have they removed that ability from newer automatic transmissions? (my last
few cars all have manual transmissions)

~~~
tonyarkles
This is one of the reasons that I love manual transmissions. Sure, you'll
probably still blow up your engine, but that's way better than dying.

One of the interesting things from one of the other articles was related to
depressing the brake. Apparently one of the solutions to resetting the out of
control acceleration was to completely remove your foot from the _brake_
before trying to brake again.

This may sound curmudgeonly, but that is one of the reasons that fly-by-wire
makes me so uncomfortable. In my car, the accelerator is attached to the
throttle by a cable, and that's the way I like it :). I was driving my
girlfriend's 2010 Pontiac Vibe the other day (essentially a Toyota Matrix) and
stopped at a slippery intersection. When I went to get going again, the
traction control system detected that it was slippery and, as far as I can
tell, did throttle limiting to prevent the wheels from slipping. This is one
of the most uncomfortable feelings I've ever had driving a car... I pressed
the accelerator and there was _no response_.

~~~
Mikeb85
> When I went to get going again, the traction control system detected that it
> was slippery and, as far as I can tell, did throttle limiting to prevent the
> wheels from slipping. This is one of the most uncomfortable feelings I've
> ever had driving a car... I pressed the accelerator and there was no
> response.

Most decent cars let you disable the traction control (which is necessary
sometimes)...

But yes, sometimes the newer technology doesn't work as well as the older
technology, depending on the conditions and driver skill. I've had ABS kick in
a few times and it's really disconcerting, I almost prefer not having it.
Traction control is great for tiny patches of ice or rain, but terrible when
the roads are covered in fresh snow.

My old Saab 900 Turbo was the best winter car I'd ever had, could basically
plow through anything the front bumper could clear (and never failed to start
- even when it was minus 40 degrees). My new car is adequate, but not nearly
as good when it gets really ugly...

And fly-by wire IS scary, they should never eliminate physical connections for
certain functions (accelerator, brakes, steering).

~~~
tonyarkles
The ABS in my car lost a sensor a few years ago and shut itself off. I see it
as a feature :)

There's a way to disable the traction control, but figuring it out wasn't what
I had in mind while stuck at a 4-way stop.

------
AndreyKarpov
Don't be evil programmer. Use PVS-Studio Static Code Analyzer for C/C++:
[http://www.viva64.com/en/pvs-studio/](http://www.viva64.com/en/pvs-studio/)
:)

------
andrewcooke
[https://news.ycombinator.com/item?id=6636811](https://news.ycombinator.com/item?id=6636811)

------
speeder
I also saw a Toyota Corolla pull suicidal acceleration, I wonder if it is also
a software bug.

~~~
tjr
Software like that is where we need "software engineering", with comprehensive
requirements, verification, code reviews, sign-offs, extensive testing, etc.

It may also be good if such software was open source, so that anyone outside
the company could also review it, if they so chose. See also:
[http://www.tuxradar.com/content/karen-sandler-full-
interview](http://www.tuxradar.com/content/karen-sandler-full-interview)

~~~
JonFish85
See the thing is, you can't really do that all that easily in an embedded
environment. Pretty quickly you get into an "Uncertainty Principle" situation,
where it's difficult/impossible to test something without tainting the
results.

For all of the code reviews and testing you want to do, how are you planning
on having everyone test interrupt flows, re-entrancy, transients in interrupt
signals, defective/leaky grounds, watchdog timers, memory bus clock errors,
mutex/semaphores, pullups that are too weak or too strong, etc?

Code reviews are all well & good, to be sure, but it's not just software
anymore, there are physical things to think about. "Software engineering" is
usually too specifically "high level" to apply as well to embedded firmware.
How do you plan on writing unit tests to run? How do you test a signal edge
that seems to fall outside of a valid window once every few days?

Open source the code if you want (and if it's legal: a lot of times there are
NDAs signed for processors), but it does no one any good if they can't have
the same hardware, made with the same components, often times compiled with
proprietary compilers.

It's about so much more than just code, so be careful of mis-applying great
software engineering principles to a completely different situation.

~~~
tjr
I'm not very familiar with how car component software is developed, so I can't
speak to that.

In avionics, the software requirements are only one part of the system.
There's requirements for how the hardware is to behave, and tolerances for
acceptable behavior. All of these pieces are tested. The software is tested by
itself in a pure software environment, and it's tested running on the specific
piece of hardware that it needs to run on, and that software+hardware
combination is tested in various stages of increasingly complete avionics
systems, culminating in testing the entire airplane.

But I was just talking about the software, where we do employ requirements,
verification, testing, and "software engineering". That's only one piece of
the system of course, but still a valuable piece. If the software doesn't work
right by itself in an idealized environment, then it has little hope of
working right when the hardware is going bonkers. And even in the idealized
pure software environment, we simulate various system failures to do initial
testing on how the software responds.

------
wissler
Regarding the EE bashing, there's an element of truth but overall it is an
unfair characterization.

I think the true source of the problem lies not in the EE degree or lack of a
CS degree, but in a specific, weird distrust of abstraction by in the
institutional culture of embedded systems development. This opposition (and it
is opposition, not merely ignorance) isn't coming from the degree per se but
from the field as it is sometimes practiced.

My guess is that the reason for this culture is that long ago, there was a
history of "fancy newfangled compilers" screwing up the code, engineers
getting repeatedly burned, and learning not to trust such "fancy" tools. This
attitude may have made some sense a long time ago, but now it's an
institutional relic.

So the irony is: the engineers are avoiding abstraction in order to increase
reliability, but they wind up decreasing it because the code becomes
impossible to manage.

Anyway, this is not a defect in the way EE's are educated, it is a very
specific subculture of embedded systems development, which traditionally
happens to tend to hire EE's.

------
wissler
Is there a reason why something so fundamental as throttle control/braking
ought to be governed by a computer, instead of by foolproof old-fashioned
hardware?

~~~
HeyLaughingBoy
"Old fashioned hardware" is anything but foolproof. Linkages bind and flex and
change their behavior over time, hydraulics leak and piping must run in areas
that can accumulate rust (I had the not so pleasant experience of losing my
main brakes due to a rusted out brake line. Driving with only an E-brake is
not fun!), steel is heavy, etc.

Computers do add benefit: ABS, traction control and stability control would be
a nightmare to implement mechanically. There is no question that the software
solution is a better idea, but the design and implementation must be done
correctly.

~~~
vonmoltke
I think it is more fair to say that, being old and well-used, the engineering
behind old-fashioned hardware is more mature and its failure modes are better
understood. Embedded electronic systems of various types will reach that point
as they are more widely deployed and used for longer periods.

BTW, I have been through that exact failure scenario before myself (in an '81
Monte Carlo).

