
The Internet of Things – A Disaster - stock_toaster
https://gekk.info/articles/iot.html
======
noonespecial
With apologies to the late Mitch Hedberg, always design your device to fail
like an escalator, not an elevator. When an escalator fails, it becomes
stairs. When an elevator fails, it becomes jail.

Don't design a computer that is a smoke detector, design a smoke detector that
happens to also have a computer. A kernel panic or complete loss of computing
should leave behind a traditional smoke detector in a fancy looking case.

~~~
dechov
As an exercise of the limits of this advice, how does this apply to designing
elevators, short of "never design elevators"?

~~~
ouid
hard brakes that apply when doors are opened manually, and an emergency panel
behind which a small ladder is located.

~~~
jack9
The costing difference between an elevator and stairs is substantial.
Especially when you have to have stairs when you get an elevator by building
code.

------
com2kid
There is a battle between the old school people who want to do everything with
microcontrollers and who measure RAM in Kilobytes, and the new generation who
want to embed a full microprocessor + OS in everything.

When it comes to capabilities, the people arguing for entire OSs generally
win. Go with Linux, get an HTTPS stack, get XML parsing, get JSON parsing,
don't worry so much about RAM. And heck, the price different between a
microprocessor and a microcontroller is minimal now days, a couple bucks in
large quantity (especially if you have to outfit the microcontroller with more
chips to add functionality).

The microcontroller people are right in one way though. Their products are
simpler. Battery life will be better (it is easier to control power usage when
you start with nothing and build up, then trying to minimize an entire OS
down) and the code will be purpose built with less complexity to it.

The ESP8266, ESP32, and Arduino are one view of things, Raspberry Pi and
"Things with Android Running On Them" are another.

Of course against all of this, the analog EEs laugh at us and continue to make
things that just /work/.

~~~
wott
I was hired to design embedded electronics in my last job. Came a project. I
would have designed it with one microcontroller in the 50-120 MHz range, or
two of those to make it easy to deal with all interfaces, and called it a day
(on the electronic side) because almost nothing more was needed. It was
basically just a glorified RS485 extender + Web -> RS485 interface. But our
boss decided of the design: mandatory Linux and Python scripting, so we ended
up with a Big Mama Ubuntu on a 1 GHz dual core processor with 512 MB DDR2
RAM... Because yeah, on the software side, you 'just' have to throw ready-made
drivers and link pieces of ready-made software developed for the desktop.
Supposedly...

The board ended up with 1500 components, it took 1 extra year to get just half
of the features kinda working, the rest of the features was discarded. And the
power budget was blown, which probably led to interesting negotiations with
other involved bodies, but I had already quit at that time.

Obviously, 'embedded' meant very different things for me and for him.

~~~
joshvm
It's fun being on the other end of the scale - the last major commercial
project I did used an Atmel Atmega1284. Most other systems in this space go
down the route of FGPA which is hell to support. I was actually amazed we got
away with an 8-bit micro, but it was a great example of pushing the chip to
its limits and comfortably achieving the goal of the project.

------
_iyig
I worked at Nest. This blog post is full of BS.

For example:

"For the record, I read several stories about the Nest Protect going into
permanent alarm, and you know what my hunch is? The same thing I always
assume: "Dumb Linux crap." The culprit was probably some shell script that
opened /opt/smoked/detect and output 1 to it and then left the file locked so
nothing else could touch it or forgot to delete a pid file or whatever. This
is what I always assume when I read about Linux integrated devices screwing
up, and on the occasions I've actually heard what the cause was I usually end
up right."

As I mention in a separate comment, iFixit's teardown identifies the 100Mhz,
128kB RAM MCU which is the Protect's brain. Such a device does not run Linux.
Had the author done any research at all, they would have found this.

Instead, they waded into an unfamiliar topic with the knowledge of a novice
and the arrogance of an expert - precisely the accusation they level against
Nest.

Furthermore:

"The Smart device engineer does not begin by disassembling ten smoke alarms to
see how they work. They do not begin by reading papers written by fire chiefs
and scientists. They do not look at the statistics on fire-related deaths with
and without smoke alarms of different eras (although the marketing department
director does)"

All this due diligence and much more was done. The author's lazy speculation
insults me and my former co-workers.

This blog post gets lots more important stuff wrong. Suffice it to say that
today the Protect is very well-rated by consumers and safety professionals.

The IoT field as a whole is a mess, and deserves much of the author's
criticism. Nest, specifically, does not.

~~~
zkms
> All this due diligence and much more was done.

I don't have any connection with Nest Protect and don't own one, but the fact
that the Protect does not use an ionisation sensor (they suck because they
suck at detecting fires early, it has nothing to do with the harmless
radioactivity) and used a dual-wavelength photoelectric sensor gives them high
marks in my book.

All the hardcore smoke detection equipment -- both the aspirating kinds
("VESDA") and the open-space beam/imaging kinds -- use dual-wavelength
sensing. I cannot speak to all the "smart"/"connected" aspects of the device
but the smoke sensor seems far better than that of every other residential
smoke detector, and it would be very nice if other residential smoke detectors
ditched the ionisation sensor and went to a dual-wavelength design.

~~~
makomk
Most new smoke detectors are photoelectric rather than ionisation-based these
days because China can pump basic models out for $2-3 shipped in quantity one.
(In fact, the cheapest I've heard of them going for is £1 for a two-pack in a
UK discount store recently.) Nest got a bunch of flack because apparently they
screwed up their original single-wavelength implementation and had obnoxious
problems with false alarms.

------
komali2
I don't think a poorly designed and engineered product, that _happens_ to
connect to the internet (and to be fair is marketed as "IoT") means that the
entire concept of "connecting devices to the internet" is "bullshit."

I agree that there's a lot of marketing Bullshytt (to steal from neal
stephenson) regarding IoT, but when we get to the root of what the words
really mean (connecting stuff to the internet), I think it's the natural
progression of technology. Why _wouldn 't_ you connect your field of oil
pumps' valve readings to the internet (as well as having the manual needle-
gauge backup) so you don't have to have a dude drive out every day to take
readings? Why _wouldn 't_ you connect your thermostat to the internet so you
can override the schedule when it turns out you need to stay late at work?
Etc.

I don't get why everyone has to lump together the marketing bullshit that
obviously is overstated with IoT, and then go "hence, IoT will fail and is
crap."

~~~
rubatuga
The author mentions that he would like IoT for his washing machine, so I think
you’re misrepresenting his sentiment. Most likely, he’s angry with the current
engineering standards of IoT, more specifically Nest

------
3chelon
The author makes a very valid point about people replacing embedded systems
with Linux. I have a home automation system that has been running for over a
decade now, and has several generations of technology in it. The order of
failure is, from highest to lowest:- 1\. the desktop I receive status updates
on; 2\. the Raspberry Pi I added to monitor the remote nodes and email me
status updates; 3\. the batteries; 4\. the physical parts: motors, etc; 5\.
the embedded systems that read the sensors and control the motors; 6\. the
analog circuitry.

~~~
mcbruiser3
do you happen to have a writeup of your HA system and how you put it together?

~~~
ocdtrekkie
Not the parent, but I will probably be writing up mine soon. My choice of
hardware is heavily inspired by the notion that devices function independently
of the computer, and the computer functions independently of the Internet.

...My home automation software has bugs, and I've had plenty of problems with
it. But I've yet to have a problem with my thermostat, lights, or smoke
detectors. Despite all of them being linked to it.

~~~
mcbruiser3
OK, great. Will look for a reply with a link.

------
philipkglass
I first read about Internet of Things in contexts where it seemed to make a
lot of sense: monitoring bridges and pipelines for integrity, "smart grid"
coordination to optimize electrical grid services, networked sensors for
better preventive maintenance and higher productivity from existing capital
equipment (e.g. mining facility trucks, crushers, power plants...)

It's sad that the term has come to be dominated by ill-conceived consumer
devices. I still have to remind myself that people are picturing "waffle iron
that sends email" rather than "networked strain gages on bridges" when they
savage IoT as a concept.

~~~
azeirah
I've recently come to believe the same thing, always shrugged off the weird
consumer IoT stuff because of the marketing, security issues and uselessness
of a lot of the projected visions of the IoT future.

I lose confidence in my rejection of IoT when I think of smart energy grids
and cooperating sensor-networks. It's a useful and welcome addition to our
future.

------
ChuckMcM
An interesting read, a bit more inflammatory than I find informative but the
points are pretty solid. Realizing a new device, especially a safety critical
device, is something to be done with care (he rants about the Nest smoke
detector). And it is "easy" to get irrationally exuberant about adding a
computer to things that don't really need them. But I'd diverge a bit on
claiming the _practice_ flawed rather than simply the application of it in
this case.

~~~
Animats
That's what bothers me about IoT projects - little consideration of failure
modes. Nest thermostats have failed in modes that resulted in houses having no
heat, and the user couldn't override it locally.[1]

Combine this with active attacks, and it looks really bad. Over three weeks
after the attack, Maersk Lines is still struggling. Their big ports didn't
achieve close to normal operation until about last Monday. Their big automated
port in Rotterdam is running again, after being totally shut down for two
weeks, but there's much more manual paperwork than usual, and billing is
completely down, so they have zero revenue.[2] LA and Elizabeth NJ are finally
back up. Some customer-facing functions were completely re-implemented with
simpler web sites. Container tracking is still down. This is the world's
largest shipping line, 24 days after the event.

[1] [https://www.nytimes.com/2016/01/14/fashion/nest-
thermostat-g...](https://www.nytimes.com/2016/01/14/fashion/nest-thermostat-
glitch-battery-dies-software-freeze.html) [2]
[http://www.maersk.com/~/media/20170629-operational-
update/20...](http://www.maersk.com/~/media/20170629-operational-
update/20170714_5pm_nwc_faq.pdf?la=en)

~~~
pmarreck
The Nerves project ([http://nerves-project.org](http://nerves-project.org)) is
an IoT OS which basically boots a device into Elixir, and benefits greatly
from the battle-tested Erlang BEAM VM which is extremely fault-tolerant.

Do you have a link to the Maersk story? My dad used to work in shipping.

Also, my original Nests decided out of the blue one day that their power wires
were both disconnected (which is impossible); is this bug related?

~~~
willtim
Both Erlang anr Elixir, while safer than C/C++, are still far from ideal for
any critical system, since they lack any kind of static typing.

~~~
pmarreck
Citation needed, especially since 95% of the time your code is just dealing
with some combination of lists (arrays in OO langs), maps, strings, ints, and
symbols/atoms, and is easily unit-tested

~~~
Hasknewbie
It's interesting that you ask for a citation, and then immediately proceed to
essentially make stuff up ("95%", "easily unit-tested"). Maybe you could
provide a citation for these?

There is a long-running trend in the industrial sector to use statically-typed
languages in large and/or critical projects, so there must be reasons for
dynamic languages to be quasi non-existent there, and a change would require
more than "just easily unit-test it", no? (To be fair Erlang is in fact the
one notable exception, although it's -- as far as I know -- only used in the
telecom industry)

------
justinsaccount
> I can't buy something that just has a $5 microprocessor with just enough
> intelligence to connect to the internet and send me an email or a push
> notification if the buzzer on the washer goes off.

1000000 x this.

The only thing I'd ever want a smart device to do is have a way to configure a
mqtt or such endpoint that it should connect to to send/receive simple
messages.

Just no consumer actually wants this. They want to be sold a box that connects
to someone elses mqtt broker so it can work with an app that connects to
someone elses server.

It's just another front on the war on general purpose computing.

~~~
astebbin
> a $5 microprocessor with just enough intelligence to connect to the internet
> and send me an email or a push notification if the buzzer on the washer goes
> off

You have essentially described the Protect. Its brain is a cheap little MCU
with 128kB RAM. The author, had he read an actual teardown [0], should have
realized this. Instead, he made false and confusing claims about Linux.

Certainly other companies put grossly-overspecced hardware in their "Smart"
devices, but Nest's Protect isn't one of them.

[https://www.ifixit.com/Teardown/Nest+Protect+Teardown/20057](https://www.ifixit.com/Teardown/Nest+Protect+Teardown/20057)

~~~
wrs
Linux specifically is wrong, but according to ifixit, there are two general
purpose 32-bit processors, and two radio SoCs also containing 32-bit
processors. Which to me still seems like a _heck_ of a lot of CPUs and
software for a smoke alarm.

------
pishpash
"I need to preface this by saying that I have very little faith in the
worldliness or general sense of Silicon Valley hardware engineers. I have seen
a long history of extremely poor decisions from that part of industry, so I
will assume they made all the worst decisions."

Old school engineers designed bridges, dams, nuclear plants, transformers,
etc. They took a serious oath and got a license. Software "engineers" \-- and
let's be honest, most are closet software engineers, even the "hardware" ones
-- are not cut from the same cloth in approaching problem solving, to the
major detriment of society. I never trusted them.

~~~
ew6082
As one of those engineers, I often wonder. In my line of work, if you don't
have a stamp, you're not an Engineer. While NCEES offers a PE in software, how
many in software actually take the PE exam? Is Professional Engineer even a
thing in Silicon Valley?

~~~
astebbin
As a Silicon Valley software developer, I haven't seen many credentialed
engineers. The vast majority of my colleagues in software have a degree or two
in Computer Science, but no formal engineering certification. The few I've met
I would describe as "incidentally" credentialed, meaning they went to a school
(commonly Waterloo) where it was a requirement for graduation.

------
_wldu
A friend of mine likes to say, ___" The S in IoT is for security"_ __.

------
gumby
Nest is doubling down on the dumbness too. Their nest cams ship _everything_
up to the cloud for processing!

What happened to computing at the edge? Are Apple (of all people) the only
ones to still embrace this philosophy?

~~~
jff
Processors are cheaper than ever, and there are multi-core multi-gigahertz
processors with multiple gigabytes of RAM scattered throughout most homes but
yes, as you point out, nobody wants to use them, they want to ship them up to
"the cloud" or as we used to call it, "some servers on the Internet".

A charitable soul might say that by keeping local software simple, the end-
user's devices don't need to be updated as often, while the software running
on the cloud systems can be continually enhanced. A cynic like myself would
say they prefer to use the cloud:

1) because it's easier to gather serious data about the users for later
use/sale

2) because by tying users to an online service, they have something valuable
("2 million active users!") to offer when the startup inevitably gets bought
out

3) because you can get away with less efficient code if you're running on a
big timeshared server vs. on a small battery-powered device

4) because "the cloud!" is still an effective marketing gimmick

~~~
mahyarm
As an ios mobile dev, I get jealous of my fellow android dev's ability to roll
out updates incrementally and to publish updates immediately. With a cloud
model, you don't have the possibility of bricking your customer's device as a
failure mode.

1) You can actually gather more data on the device vs just a remote service

2) SaaS moats are attractive from a business model perspective

3) TBH pushing compute on your customers devices is cheaper for the cloud
owner, but harder to manage. On a per user basis most servers are actually
VERY efficient. Most peoples smartphones are relative supercomputers, only
some things will be very battery draining.

4) 'the cloud' does enable a lot of stuff that people don't want to manage
manually themselves

------
bsder
As much as I like slagging the Internet of Crap, smoke detectors are not a
great example because smoke detectors are remarkably complicated.

They have to work. For years. They have to detect disgustingly small signals.
Reliably. And, by the way, the chips have to survive in close proximity to a
radiation source. And generate an alarm loud enough to wake the dead.

If I remember correctly, Qualcomm nee NXP nee Freescale nee Motorola used to
keep an old 2" aluminum gate fab line around because every time somebody tried
to make a new smoke detector chip _something failed_. So, they kept the fab
open and simply printed money.

------
lxe
> Put it on top. Connect the I/O lines on your little PC to the output of the
> smoke detector chip. Let it do the heavy lifting, the stuff that you aren't
> sure you can do safely and which it has proven it can do for decades.

I worked on a network communications part (the little PC) of a device that
controls train signals (the smoke detector chip). This is exactly how it's
architected -- the safety-critical components are isolated from the non-
safety-critical ones. The busybox linux board does not go in between the
sensors and the vital control logic, but rather "on top"

------
jaxbot
Has anyone tried one of the newer revisions of the Protect and had any
experiences, positive or negative, with it? I was initially excited when I
heard about the product announcement. It seemed like everything I wanted in a
smoke detector: remote monitoring, ability to silence, starting with voice
before blaring and getting my adrenaline up, and, well, a smoke detector. But
Amazon reviews and the infamous video later, I backed quickly away. I've been
awoken by a false-positive smoke detector before and the experience is panic
inducing on top of outright annoying.

~~~
DIVx0
I just bought a house that has two 2nd gen protects along with traditional
style detectors, So far so good! No false positives so far. I reset both
detectors and had them go through their self test when we moved in and they
both report being fully operational.

Neither of the protects are near the kitchen or regular smoke producing spaces
though. However They are near bathrooms. I hear steam can set these off pretty
easily but I have not had that experience.

I might buy more protects to replace the other detectors but I want to wait a
bit and see if nest comes out with any other products.

~~~
pcblues
I would test them with smoke just to see how each behaved. Try with smoke from
a candle just put out, a piece of paper just put out, and a plastic wire just
put out.

------
ksk
I think it also requires a mindset that a lot of young engineers don't have
IMHO. Very few of them will ever ship code that will run 24/7 without
patching, and be stable out of the box.

~~~
ocdtrekkie
This really worries me when they start writing critical applications. I did
have to reboot an old school program at work today. Very dated piece of
software, pretty sure it predates Windows 7.

It had been running, over a (fairly complicated) network connection for 303
days. I can't think of many modern pieces of software that do their job 24/7
for 303 days straight.

And I'm pretty sure the reason it was 303 days, was because I had to move that
machine last year.

------
MarkMMullin
One thing I believe that's not being properly considered is the fact this is
really a multilevel problem - if you want to make a smart IoT device you still
want a reliable device. "On top of" is absolutely the best concept. Yes, the
biggest computational bang for the buck comes from stuffing linux or something
similar in, but who doesn't react with a smile when they see a multi year
uptime reported - it's not rare, but it's not anywhere near a majority either
- Start with the basic circuit, on top of that you add microcontroller logic
to mediate between the higher level ARM SoC or whatnot and the base circuit,
and just as importantly, you've also got a microcontroller based watch dog
that will kick linux if it falls over. My experience in building this
[https://hackaday.io/project/21966-quamera](https://hackaday.io/project/21966-quamera)
is to never stop asking "Quis custodiet ipsos custodes" (anybody proffering
analog solutions will be ignored, analog is hard and I am stupid)

------
jancsika
> First, begin with a smoke alarm. A tried and true design that you can buy
> for $10. Buy the exact parts that are already in use, and put them in your
> final product. The smoke sensor, the transistors, the through-hole
> resistors, the Fairchild IC, the 9V, the LED. Buy all of that and use it in
> your final product.

So... why didn't Nest do it this way?

~~~
debacle
Software people designing hardware.

~~~
jancsika
But why not piggy-back atop of the legacy hardware? Doing that would save time
and let the software people solve higher-level (and presumably more
interesting) problems.

Besides, software people I know are way to lazy to go sticking their fingers
in sockets.

------
nivramarvin
The mobile-friendly view, recommended on top of the article - is kind of a
usability disaster: I really like the authors approach using plain HTML. But
using the mobile-friendly view has many usability pitfalls like: No links, no
ability to copy text, no ability to search on page and switching to another
browser-tab will lose the current position where you were reading at. As said
in the article, sometimes its better to keep things which just work and are
used by the majority like that. Same goes for using some styles or a blog
template to provide a a fully mobile friendly read for the user, not a
compromise of two bad options (plain article or mobile-"friendly" view).

------
luord
All of these are excellent arguments against _most_ of the IoT practices, and
not even including the security issues. All in all a great read[1].

These are all the reasons for why I'm not interested in IoT/embedded. I just
don't have the hardware and low-level firmware knowhow and doing it the other
way, starting with an OS and going down, seems (and evidently is) horrible.
And I say this as a full-time software developer.

[1]: Even if there might be inaccuracies; the author admitted the possibility
and the point stands.

------
rubatuga
I’m just wondering, as we shrink our gates for CPUs more and more (the gates
are approaching 10nm soon), does it become less reliable? People always want
the next fastest and most power efficient device, but surely we must be losing
something with the shrink in CPU gate size! If that’s the case, are CPUs such
as the Intel 286s more reliable?

~~~
pacificmint
The vast majority of issues will arise from software defects, not hardware
defects.

So rather than switching to a 30 year old cpu, increasing the quality control
and testing for the software would be the better aproach.

Also, as some peope in this thread have mentioned, a smart smoke detector
should be designed in a way where if it fails it becomes a dumb smoke
detector, but the actual smoke detection and warning should still work.

~~~
sigjuice
_The vast majority of issues will arise from software defects, not hardware
defects._

Exactly. The last thing I want in my house is a smoke alarm with software
written by idiots like me.

------
rubatuga
A lot of the research into redundancy for computer systems has already been
done. Check out this article from NASA when they began to implement multiple
failsafe computers into their spacecraft.

[https://history.nasa.gov/computers/Ch4-4.html](https://history.nasa.gov/computers/Ch4-4.html)

------
kootenpv
The best article I have read in a long time.

------
frozenport
Tldr; build an analog backup into a mission critical device.

The argument about not reinventing the wheel might be misinformed, because
their custom chip could actually be a well known design with a tiny tweek.

------
grapeshot
If you're writing an article, is it really that hard to do a little research?
Most of this piece is just "I didn't even bother to look up a teardown photo,
but here's how I assume they must have designed it".

~~~
sgt
He explicitly says that while he didn't tear down one himself, the folks at
ifixit did and then he included a link to the photos.

