
Raspberry Pi microSD follow-up, SD Association fools me twice? - geerlingguy
https://www.jeffgeerling.com/blog/2019/raspberry-pi-microsd-follow-sd-association-fools-me-twice
======
AnthonBerg
I really can’t understand why SD cards are considered applicable as a main
storage device for a general-purpose computer. Reliability is the primary
bottleneck in everything, and IO speed second. SD cards are the single worst
experience I have had in both speed and reliability in any commodity computer
stuff for many years.

~~~
jrockway
The reason why rPi uses micro SD is that it's the cheapest possible storage
solution. You are not buying an industrial-grade workstation for $35. (For
those using the rPi in industrial settings, there is the Compute Module which
has eMMC. It adds to the cost, though.)

I am not really a fan of "embedded" devices that have to be shut down
properly. You can never assume that you won't lose power, and I think there
are a lot of flaky/bricked rPis as a result of SD not handing power loss well.
But it's cheap, and you just reimage the thing and go when your SD card gets
corrupted.

When I last worked on embedded Linux, our OS image was read-only and the
writable storage was literally initialized at boot with "mount /dev/mmc1
/storage || mkfs.ext4 /dev/mmc1". That is how you get a consistent state after
power loss. But throwing everything away at boot doesn't fit the Raspberry Pi
model of being half-embedded half-workstation. The result is flakiness.

~~~
m463
> I am not really a fan of "embedded" devices that have to be shut down
> properly.

the problem is the Pi is not an embedded device, per se.

It's running a desktop operating system that has to be shut down properly.
This is the main reason SD cards get corrupted - there is an unchecked flow of
writes to the SD card. This increases the chance of outstanding unsynchronized
writes and SD card burnout.

I've said many times that raspi-config should allow an option for mounting the
filesystems read-only, with an overlay ram based filesystem.

You can run openwrt on the pi, and the filesystem is set up this way. I run a
pi like this and have never seen a corruption problem.

~~~
jbuzbee
>>I've said many times that raspi-config should allow an option for mounting
the filesystems read-only, with an overlay ram based filesystem.

I've had success with this:

[http://wiki.psuter.ch/doku.php?id=solve_raspbian_sd_card_cor...](http://wiki.psuter.ch/doku.php?id=solve_raspbian_sd_card_corruption_issues_with_read-
only_mounted_root_partition)

------
ChuckMcM
Command queuing is such a dangerous thing.

This is the disk equivalent of 'buffer bloat'[1] but with disk data. The
question you have to ask is what happens when you lose power and there are 'n'
writes queued to the storage device. How has the kernel marked that data?
Historically once the disk accepted the write and returned kernels would say
"ok, data on disk" buffer clean now.

Of course if you lost power and that write never actually completed, well you
now had file system corruption (possibly silent if it was just a data block).

Another problem that cropped up is "ok we can re-use this data buffer" so it
gets fill with different data, and then the storage device says "oh hey, send
me that data you told me write ..." whoops.

The third problem that crops up is that the computer on the _storage device_
crashes (or watchdog resets) and it goes back and doesn't remember what
commands it had in the queue. So some time later your kernel better ask "hey
why haven't you acknowledged this write we sent 50mS ago?" and have the disk
say "What write?" and replay it.

One school of thought was "Command queuing in the storage device is always
bad, the kernel knows more, has more memory, and is ultimately responsible for
what is and what is not on disk." So the kernel would 'hold' writes to the
disk until the write was sufficiently aged or until enough writes had
accumulated to do a "streaming" write (multiple sectors on the same track).
But for non-spinning media there is no seek time so there is no advantage.
Except for flash, you have to erase a page which can be a _lot_ of data which
then has to be rewritten with the new stuff.

I enjoyed the writeup but it really needs to start with a highly fragmented SD
card (one with lots of random I/Os to it) so that it can measure the latency
hit for page rewrites.

[1]
[https://www.bufferbloat.net/projects/bloat/wiki/Introduction...](https://www.bufferbloat.net/projects/bloat/wiki/Introduction/)

~~~
throwaway2048
additionally SSDs use garbage collection algorithms so are fraught with the
possibility of a write taking an unbounded amount of time, even if nothing is
malfunctioning, which is pretty scary.

To head off replies about fixed time garbage collection, sure it exists, but
you cant guarantee it will reclaim enough space in a fixed timeslot (think of
a massively pathological case where 99% of every flash chip is full, but super
fragmented and data needs to be shuffled around perhaps hundreds or even
thousands of times before enough space can be reclaimed)

~~~
Dylan16807
> To head off replies about fixed time garbage collection, sure it exists, but
> you cant guarantee it will reclaim enough space in a fixed timeslot (think
> of a massively pathological case where 99% of every flash chip is full, but
> super fragmented and data needs to be shuffled around perhaps hundreds or
> even thousands of times before enough space can be reclaimed)

That's why you have spare space. It's not overly hard to clamp write
amplification to a number like 10x or 20x, even in the absolute worst case.

Note that the SD standard here requires a minimum number of writes per second
_inside a 256MB area_ , and even 1% spare space on a 32GB card would give you
a >256MB buffer.

------
m463
> (and this has become a bigger problem every generation of Pi—you need a good
> power supply or you'll have a lot of annoying problems)

I wish the pi just had a 12v barrel jack.

edit: it first dawned on me that the pi power situation could be improved when
I read this article years ago:

[http://www.bitwizard.nl/wiki/Reducing_power_consumption_of_a...](http://www.bitwizard.nl/wiki/Reducing_power_consumption_of_a_raspberry_Pi)

~~~
tonyarkles
That's one of the things that disappoints me most about the Pi, to be honest.
Not the power situation, but the "no way to remix it" situation.

While it's not quite as beefy as the Pi (especially the Pi 4), nor as cheap,
the Beaglebone family of boards are awesome. They come with schematics, and
you can order the processor in qty 1 from Digikey. I haven't yet remixed one
yet (a 400-BGA is going to be stretching my abilities...), but I have leaned
on the schematic quite a bit to figure out some clever things I can do with
it.

~~~
m463
When the Pi first came out I talked with a hardware engineer about it and he
said they made some weird choices.

One thing he pointed out to me was the pin design. Everybody else would design
it with female pins so that you won't accidentally bend the pins or short them
out.

He loved the beaglebone though and recommended it.

That said, I think the pi has a bigger community and more software. (I might
be wrong)

~~~
tonyarkles
> That said, I think the pi has a bigger community and more software. (I might
> be wrong)

Most definitely. From an "ease of getting started" perspective, the Pi is
absolutely amazing. NOOBS, or whatever the distribution is called, is pretty
painless to get going. You flash it to the SD card, plug in a monitor and
keyboard+mouse, and you boot straight into a familiar GUI with tons of tools
pre-installed. Plus, the $35 price point (well, more once you include SD card,
power supply, etc) is pretty hard to argue with.

Debian IoT (the "standard" distro for the BeagleBone) is cool in its own
right, but definitely not as approachable for someone who's new to embedded
Linux. Things like GPIO mappings need to get tweaked sometimes, you're dinking
around with device trees here and there, etc. For the stuff I've been working
on lately (robotics), the BeagleBone Blue has been an absolute godsend. Tons
of ports that speak different protocols (UART, I2C, PPM for driving servos,
GPIOs, etc), and librobotcontrol is really straightforward to start building
with.

I guess a great way to distinguish between them: I pretty much always use a
keyboard and mouse to get a Pi set up. I don't even know if the BBBlue has
video output onboard, because I've never looked or tried. On boot it exposes
itself as a network device over USB; you can SSH into it and start doing what
you're trying to do.

------
dev_dull
Even despite the issues around A1 and A2, the read speeds on SD cards are just
too slow, and will probably always be too slow.

What the RPI needs is a simple m.2 slot for a "real" hard drive, which are
getting cheaper and cheaper.

~~~
ThatPlayer
I think the goal is ease of access with the memory. How many people have a m2
dock or adapter ready to write the raspbian image on it? A memory card reader
is much more common.

~~~
rhinoceraptor
It seems like it would be fairly simple to give people a minimal USB image
that then installs to the M.2.

~~~
tonyarkles
The Beaglebone Blue has an onboard eMMC that comes blank from the factory. You
can:

a) write an OS image onto an SD card and just run off that if you want

b) set a flag in /boot. Upon reboot, the LEDs go into Cylon/Knight Rider mode
while it copies the OS image from the SD card onto the eMMC. When they're
done, pop out the SD card and reboot. Done!

------
Fnoord
You should mention very clearly which Raspberry Pi board you're using. I'm not
sure if CPU throttling can affect I/O but if yes then it might also matter if
you use a good case or use active cooling or heatsink.

Also, which OS version did you use? What Linux kernel version?

I'm more interested in longevity tests and industrial grade microSD card
performance.

~~~
geerlingguy
Raspberry Pi 4, latest Raspbian Lite image (2019-07-10 with kernel 4.19), and
did apt-get update and upgrade prior to test and rebooted, official case with
Fan modded in [1] to ensure it does not get anywhere near throttling.

[1] [https://www.jeffgeerling.com/blog/2019/raspberry-
pi-4-needs-...](https://www.jeffgeerling.com/blog/2019/raspberry-pi-4-needs-
fan-heres-why-and-how-you-can-add-one)

And everyone asks for longevity tests and industrial grade performance, but I
always answer the same: I have had four Pis running almost continuously (99.9%
uptime, occasional reboot for updates) for 3+ years, and have used the same
set of 6 Evo+ cards for 3+ years now, and have never had any problems with
corruption in any of those cards.

I also run a Kubernetes cluster with 4 Pis (see www.pidramble.com) from time
to time (right now it's running), and I have never had a corruption issue
either.

The main thing I attribute this to is the fact that I'm using reliable,
quality power supplies (either Pi Foundation official supply or some other
name brand or good PoE power supplies that support at least 2A).

------
toxicFork
Reading about SD cards reminds me of this idea I had. Is it possible at the
moment to have a wifi or 4/5g transmitter small enough to fit into a chip
within (or instead to bind to equivalent connectors of) sd cards to
automatically upload sd card contents to a hub or router that can sync with
the cloud? So that you can effectively have an unlimited capacity sd card?
Anyone working on this?

~~~
dmitrygr
Not only does it exist, but it runs linux too:

[https://www.amazon.com/Transcend-Wi-Fi-Class-Memory-
TS32GWSD...](https://www.amazon.com/Transcend-Wi-Fi-Class-Memory-
TS32GWSDHC10/dp/B00A659ILQ)

You can run custom code on it:
[http://dmitry.gr/?r=05.Projects&proj=15.%20Transcend%20WiFiS...](http://dmitry.gr/?r=05.Projects&proj=15.%20Transcend%20WiFiSD)

~~~
slowhand09
Love this: New (2) from $5,000.00 & FREE shipping.

Personally, I used an Eye-Fi card when I needed wifi.

~~~
mynameisvlad
What do you expect? It's a discontinued, niche product that probably didn't
sell a lot to begin with, so aftermarket prices are going to be insanely high.

------
ksaj
People expecting a $35 childhood educational computer to perform like a
mission critical embedded device get what they should expect from that.

Having said that, what is so hard about running regular backup images, and
simply burning a new one every time the current SD card _inevitably_ fails?
That's been working for me for years. Can't reboot... burn a new SD card from
the backup. Reboot. All the super-timely stuff is on github, so at most, I
might lose an hour or so every 6 months to a year. And anything that isn't
altered often just has a rolling updated image for me to dd if needed. It's
rare enough I just don't care about it.

I use rpis a __lot __, and since I work with MPI, usually there are quite a
few running at all times. I just back up regularly and don 't get alarmed that
they don't last like a desktop at _100_ times the price would. Of all the SD
cards and actual rpis that died, I heaven't come close to that price, and have
lost nearly nothing for all that I've gained in the experience.

------
z3t4
I read here on HN about someone who had a SD card show old data. He
reinstalled the OS only to reboot into his old OS. Same thing happened to me
the next week with a camera, the SD card showed _old_ pictures from before a
reformat. Using these things now is super scary. I wondered why some cameras
had dual sd slots, now I know why.

~~~
megous
Happened to me too. It's a good failure mode, IMO. :) If the card runs out of
total write capacity, it turns read only. It might be better for it to reject
writes, but whatever.

------
jokoon
The rpi is nice but I'd prefer a CPU 10x times slower if it had memory to
store the OS. The rpi's CPU is way too powerful.

The only one I would buy would be the RPI zero.

Having something reliable can matter.

------
altfredd
The description of A2 standard features sounds very similar to the new Linux
multi-queue block device framework.

From quick look at latest Linux source code, there are couple mentions of
"blk_mq" in drivers/mmc, but there seems to be no actual support for multiple
queues (unlike e.g. in drivers/nvme).

Does anyone know, if those are supposed to be same things? Does SD driver
actually need new firmware to support multiple queues or is it something, that
can be implemented on kernel side in software only? At the very least, it
might be possible to use a third-party reader as long as it's kernel drivers
are updated to take advantage of multiple command queues.

~~~
amluto
> The description of A2 standard features sounds very similar to the new Linux
> multi-queue block device framework.

Unless I missed something, it's not that fancy. It's just one queue, not
multiple queues -- the only new thing is that the length of the queue is more
than 1. A multi-queue device is a device with more than one independent queue.
This is useful for very high performance devices (NVMe, for example) that can
allow multiple CPUs to submit requests at once without interfering with each
other. If there's only a single queue, then all CPUs trying to submit need to
contend for access to the queue.

------
ggg3
this is like people putting premium gas on cars that ask for regular.

read your specs, use what is recommended. not the mopst expensive. it's just
engineering 101.

~~~
penagwin
At this moment in time 0 cars can utilize this "premium gas".

This isn't communicated well, the author points out that it's marketed as
better, but you have to dig into the technical details to find out the host
must support it as well.

------
geerlingguy
tl;dr: I tested an A2 'Application Performance' class card and found that it
was half as fast as it should be, based on the SD Association specs (minimum
4000 IOPS 4K random read, 2000 IOPS write).

Then some people dug deeper and found in the specs that 'Command Queue and
Cache functions' needed to be supported in device firmware and/or on the
kernel level for A2 specs to be reached—though I haven't yet found a way for
any consumer to get their hands on devices or software with this support... so
no way to achieve claimed A2 performance.

So then I bought an A1 card from the same manufacturer (SanDisk), and it is
actually much _faster_ than the A2 card (though for price/performance you're
still better off buying the not-Application-Performance-class-rated Samsung
Evo+ card).

And in the end, being able to use an SSD or mSATA drive would be even better
(the former of which is semi-possible today with the Pi 4 and USB 3.0—though
you can't fully boot without a microSD card yet).

~~~
duskwuff
> you can't fully boot without a microSD card yet

Not true! The Pi supports several different USB and network boot modes.

[https://www.raspberrypi.org/documentation/hardware/raspberry...](https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/)

This documentation doesn't cover the Pi 4, but USB/network boot is available
there too, cf.

[https://www.raspberrypi.org/documentation/hardware/raspberry...](https://www.raspberrypi.org/documentation/hardware/raspberrypi/booteeprom.md)

~~~
jack12
As your parent says, the Pi 4 doesn't support USB or network booting _yet_.
From your link:

> Support for these additional bootmodes will be added in the future via
> optional bootloader updates

I believe they need to write (bootloader) drivers for the PCIe host and for
the new USB3 chip before USB boot can be supported by the bootloader, while
netboot requires a driver for the new gigabit ethernet core.

~~~
andymal
You can use micro sd card for the bootloader and boot from USB 3 ssd on rpi4.

[https://jamesachambers.com/raspberry-pi-4-usb-boot-config-
gu...](https://jamesachambers.com/raspberry-pi-4-usb-boot-config-guide-for-
ssd-flash-drives/)

~~~
inetknght
That's not _quite_ booting _without_ an SD card though ;)

