I did a lot with RPi's back in the 2-3 era... My SDCards were constantly getting nuked by slightly noisy power sources when running timelapse photography over the course of days. My workaround was to use the SDCard for boot only, but running everything else off a USB drive. The sdcard then only ran for a couple minutes at startup, in read-only mode, and never had a chance to screw up a write.
This worked great, and allowed me to successfully run a four-month timelapse, where I didn't have access to the device. (You can see it here: https://www.youtube.com/watch?v=cyh0QJT0rAQ )
All the modern Pis can now boot solely over USB, and the Pi 4's USB 3.0 ports mean it actually has decent IO speeds for general use (and especially for things like transferring files over the network faster than a couple MB/sec!
Now that I have the HQ camera, I'm trying to figure out a good way to mount that, a Pi 4, and a USB SSD so I can have a much faster/nicer time lapse rig.
Based on some of your work (thanks) I've got 4xPi4-4gb booting from a fifth Pi4-2gb that is running dnsmasq (PXE only) and NFS for / and /boot. Boot order is 0xf412 - network, SD, Usb, repeat. The /nfs/tftpboot/<host_serial> is a soft link to the boot directory inside /nfs/<imagename>-<hostname> Each host gets its own directory - haven't looked into a CoW filesystem yet.
I've got a hacky Ansible script that will reboot a specific host from network, dd a fresh Raspios image onto the sd card, mount the root filesystem, copy over the ssh keys, reboot the pi, do the usual apt update/upgrade and a couple other tweaks, reboot from the network, dd the sd card back to an image and finally run Pishrink.sh on it. It's fascinating to watch in a Rube Goldberg kinda way.
I would add that net bandwidth seems quite respectable though not USB3 speeds. One issue i've run into is that K3S does not start - mostly likely due latencies with networked disks
That HQ camera is pretty awesome. I bought one to build a film scanner with. For those who don't now, it's a camera module for Raspberry Pi that comes with a 12mp sensor that supports attachable camera lenses(sold separately). It also comes with a really nice guidebook.
I absolutely concur with this setup. Power supply and SD Card can make or break your RPi experience. I've been running a Pi 2 for ~5 years in this setup i.e. Sony 2A power + Toshiba SD Card + USB drives for Git (SD card and USB drives encrypted).
It's been used only as a git server 24*7 for substantial amount of code, I reboot it once a year. Using it as a Git server started out as an experiment, but worked out great until the code became large and hung-up occasionally during commit due to CPU/Memory bottleneck (in-spite of adjusting relevant Git flags).
Recently I've started putting it to other usages, so not sure whether the reliability would be the same. My RPi3 needed official power supply and better cooling to extract similar performance/reliability ; Although it officially supports booting via USB I didn't have much luck in getting it work and had to use SD Card for boot and USB SSD as Arch Linux ARM/Btrfs root[1].
Having worked on firmware writing to SD cards, I can tell you that the big problem comes from allocating space for writes. That’s when the card disappears on you, as it looks for space it will potentially shuffle data around behind the scenes. Current spec I believe says max latency is 200ms. This is the operation that will abuse this part of the spec. If you want to see an SD card flail, just try to write one byte at a time. This will cause headaches no matter how wear leveling is implemented in the SD card. And will likely significantly shorten the lifespan of the card. If you can make sure you only ever write multiples of 4096 (probably card dependent) you’ll be much happier using SD cards. But even then, you might have a 200ms window where you can lose or corrupt data if your power to the card fluctuates significantly.
IIRC SD cards weren't made to handle the read/writes of running an operating system off them. Which is why you see them break so often when used with Raspberry Pis
My understanding - at least for the original Pi - was that there was a design flaw where the sd card was not protected from power irregularities in the same way the rest of the device was. There's a lot of argument over whether it's a hardware design flaw or a problem with sd cards generally (perhaps the usb spec has better buffered writes or something)... It's apparently much more a problem for long-running applications with high write frequency.
To avoid these problems I am using an overlay FS (just reads from flash and writes to memory) and after a reboot all changes are gone. That might not be suitable for everyone but works great for stuff like mqtt brokers or devices which pull their configuration on boot.
This is how I am using my Raspberry Pi (running Pi-hole) too: overlayFS. It means that I lost the statistics after a reboot, but it also mean that I don't care if the power in the house is suddenly lost either. I also enabled watchdog support so my Raspberry Pi is probably the most reliable part of my network.
Same here. I run temperature monitors on mine. Every five minutes a cron-job copies data to a /boot subdirectory which is still r/w. On power loss, the most I can lose is those five minutes.
I was going to ask if you had considered battery power, since a battery is basically the cleanest power source you can possibly get. But, your link makes it seem like basically a "set it up and check on it in X months" type project.
I'd been hoping that the recently released USB boot support for Raspberry Pi OS on the Pi4 would mean that Ubuntu for Raspberry Pi would also get USB boot support soon. Unfortunately it doesn't look like it will happen any time soon[1].
My main issue with USB boot is the abysmal driver support. I have three USB3 to sata adapters with three different chipsets and none work properly. You don’t get warnings or failures, you’re just left with a black screen. As often the problem with cutting edge Linux it doesn’t “just work” and this ends up driving people away.
Note that earlier in the beta there was a fairly annoying bug that affected all three of the SATA adapters I had. That bug was fixed just a week or so ago, and now all my adapters work out of the box. They're all cheap ones you can find on Amazon or at Micro Center. Here's the issue, in case you're interested: https://github.com/raspberrypi/rpi-eeprom/issues/151#issueco...
I got into learning Linux by building a remote booting network.
I was already doing that using Novell Netware and boot EPROMs or floppy disks, booting DOS over IPX.
I kept building remote booting networks, then I discovered https://ltsp.org/ which made it a lot easier. I could easily switch declaratively between configs, like rootfs on local disk or nfs.
I kinda miss those times...
ltsp was my first serious learning too after the joy of Windows 3.11, windows for workgroups.
I was running a PII in my roofspace and a diskless terminal in my living room using ltsp. I think sometimes you go through a season of fascination and learning. Luckily I ended up working in this arena as I was maybe less helpful around the house than I should have been.
I have a fond memory of pulling the cat5 cable through the roofspace floor. My son was little and couldn't climb into the roofspace so he was the cable puller. As he pulled the cat5 cable below through the hole in the ceiling he shouted up, "Daddy, is this magic?".
When I worked for a local government agency, I replaced a bunch of NT4 workstations that we used as kiosks for the public (with heavy-handed policies to disable all kinds of things) with ltsp. It ran like a dream on the Dell Optiplexes of that time period.
You can now boot directly from USB with relatively recent firmware. This means my only remaining gripe with Raspberry Pi hardware is not including an NVME slot.
Underneath the board, for instance, would be perfect, and although the glue logic might make it a bit more expensive, the 8GB model is already near €100 anyway, so... We’re already into Z83ii/mini-PC territory here.
You can get USB adapters for a few bucks that are already bootable with this solution, so I’d glue that to the USB 3 bus and be done with it. Should be fine until they change SoCs.
My experience is that normal SanDisc SD cards are the best. Other brands are unusable because of latency of write/read time! You need to power them when you buy them though, otherwise they actually break sometimes, and I don't mean data loss; I mean break!
This means you should try to buy them as fresh as possible and since sellers don't show production dates that means you have to wait for new models to come out and buy them then.
To me the most important metric is not speed of bandwidth but speed of time and stability! Does anyone have measures of how the SSD compares to SD card in terms of write/read latency?
I have only had problems with USB <-> SATA, the stability of USB3 drivers is a joke!
I've had three Pis running more or less continuously off a microSD for a few years, and none of those cards (same as original) have gotten corrupted. These aren't running zillions of writers per day or anything, but most people's problems stem more from bad power supplies or cheap cards than the Pi's use of microSD itself.
You should probably run iostat in your tests too, to have high bandwidth is completely useless if it has high latency. Just like a high framrate game is unplayable if the frames are instead delayed so the motion to photon delay is many frames.
Companies are now cheating in every way they can to extend and pretend, and now as the peak density on the flash media reveals that they break with time it's going to get messy!
This worked great, and allowed me to successfully run a four-month timelapse, where I didn't have access to the device. (You can see it here: https://www.youtube.com/watch?v=cyh0QJT0rAQ )