
Exclave: Hardware Testing in Mass Production, Made Easier - lnmx
https://www.bunniestudios.com/blog/?p=5450
======
HeyLaughingBoy
_the original test specification was literally “plug the device into the USB
port of [your] laptop, and type in this AVRDUDE command to load code_

At the outfit I work for this is pretty much what they were doing for one of
the products when I started. It wasn't that they didn't know better (they had
other items in mass production); it was that the product used to be such a low
volume that typing in CLI commands to program and test it was (almost) good
enough.

I spent months automating that to the point of "plug cable into device and
watch for Pass/Fail" indication on screen. Now that we're shipping thousands
per month it make a huge difference :-)

Testing is a weird thing. The points you test are a function of what you
expect may fail in manufacturing. No point in testing something that's
extremely unlikely to fail, unless it's required by regulation.

e.g., FDA (Food & Drug Administration) Good Manufacturing Practices state that
if you don't build something using a documented and repeatable process then
you _must_ test 100% of product coming off the line. If you have a tight
manufacturing process, then you can decide what level of testing meets a good
balance of safety and cost.

People spend their entire career as Test & Manufacturing engineers and it is
quite a complex and interesting field.

------
my_username_is_
>Reputable factories will test 100% of every product shipped.

Frankly, this just isn't true. It depends on your expected defect rate, how
your defect rate drifts over time, the cost of doing 100% inspection, and
effects of product failures. If you're making safety-critical gear, yes you
will probably inspect 100%. If you're making injection molded clamshells for
connected consumer products, you're probably only doing inspection in batches.

~~~
kev009
I guess it depends on the industry but it is somewhat common for electronic
components. There are a lot of failure modes in parts, picking, soldering, and
assembly. The economics widely favor doing testing at manufacturing time
(versus something like at a distributor or dealing with customer RMA which may
come with other externalities like brand damage) because boards can be
reworked in the factory and you avoid other waste like packaging, shipping,
and tariffs.

Back in the day extremely elaborate and massive bed of nails in circuit test
rigs would poke the leads of a through hole circuit board to do continuity
testing and basic functions checks. Now a lot of that can be done with JTAG
circuitry, a serial bus many ASICs implement and chain together to provide
less invasive way to similar tests. Xray, thermal, optical scanners are often
used in various parts of the assembly. And of course plenty of humans. Test
rig design is a big part of any manufacturing process and may be used to
verify overall system function like button presses and connector fidelity.

As a separate but related thought, in VLSI: verification design, for both
manufacturing and BIST, can easily eclipse the rest of the design in some
ASICs in terms of cost/effort/complexity.

Software engineering is kind of the odd duck in that it is frequently a shoot
first ask questions later pursuit in most organizations (funnily, even
commonly in places like semiconductor companies that invent and progress new
formal methods). There are plenty of decent tools like Lamport's TLA+ that
aren't widely used in addition to more familiar aspects of code test.

------
snops
For another option, see OpenHTF from google, though this targets quite complex
devices:
[https://github.com/google/openhtf](https://github.com/google/openhtf)

~~~
markhollis
I am starting to use OpenHTF at work. I was not aware of Exclave. Seems like I
have interesting homework to do. This blog post by Bunnie was a very nice
read.

------
mips_avatar
I love Bunnies stuff. Reading about his reverse engineering work on the Xbox
was one of the reasons I chose to study computer science. Excellent article,
wish there were more people out there like him.

------
eltoozero
Fantastic, thank you Bunnie & crew, well done.

