
UASP makes Raspberry Pi 4 disk IO 50% faster - geerlingguy
https://www.jeffgeerling.com/blog/2020/uasp-makes-raspberry-pi-4-disk-io-50-faster
======
floatboth
> Without UASP, a drive is mounted as a Mass Storage Device using Bulk Only
> Transport (or BOT), a protocol that was designed for transferring files way
> back in the USB 'Full speed' days

Wow, holy shit, TIL.. I was always assuming that USB drives were always SCSI —
they show up as 'da' (Direct Attach) on FreeBSD after all.

Looking at umass driver source:

* The driver handles 3 Wire Protocols

* - Command/Bulk/Interrupt (CBI)

* - Command/Bulk/Interrupt with Command Completion Interrupt (CBI with CCI)

* - Mass Storage Bulk-Only (BBB) (BBB refers Bulk/Bulk/Bulk for Command/Data/Status phases)

* Over these wire protocols it handles the following command protocols

* - SCSI

* - UFI (floppy command set)

* - 8070i (ATAPI)

* UFI and 8070i (ATAPI) are transformed versions of the SCSI command set.

Huh, they _are_ pretty much always SCSI, and "bulk only" is (unsurprisingly)
only a USB transport level thing that doesn't change the command set.

I guess "UASP" is some sort of marketing term for Command/Bulk/Interrupt
then..?

~~~
bakul
FreeBSD doesn't support UASP (protocol id 98). AFAIK neither do the other
three BSDs.

~~~
geerlingguy
Apparently as of 30 minutes ago FreeBSD _does_ support it on the Pi 4!
[https://twitter.com/FreeBSDHelp/status/1280062935327313920](https://twitter.com/FreeBSDHelp/status/1280062935327313920)

~~~
bakul
This is the PCI express driver for BCM2711. Good to see this but the UAS
support would be elsewhere! Either in the umass driver or a new uas driver
(just a guess as I am unfamiliar with the USB related kernel code).

~~~
geerlingguy
Gah, you're right. Totally misread that tweet in my state of non-caffeinated
stupor this morning. Still a useful improvement!

------
blattimwind
> I used a Satechi USB-C power tester and measured an 8% peak power savings
> using UASP. That means you'd get 8% more runtime on a battery if you do a
> lot of file transfers.

No no, even better! Peak power consumption is lower, but the same work is
completed much more quickly due to increased throughput, so the energy
required for the same work is decreased dramatically. Between the performance
increase and the lower power usage I wouldn't be surprised if this reduces
energy use by 50 %.

~~~
repsilat
Yeah. This kind of savings comes up in interesting places elsewhere -- if you
make your algorithm run twice as fast but using twice the memory, your time-
integrated memory use (measured in gigabyte seconds) remains the same. So _if_
you can make good use of that memory while your algorithm isn't running,
you're saving CPU and not really using more RAM.

For something like the Pi this is unlikely, but in data centers with big,
latency-insensitive distributed workloads "using memory for less time" can be
a real win. (The battery example with the Pi really is the perfect example
though, because you're usually interested in the time-integrated value more
than the peak draw.)

------
Ijumfs
The Pi 4 is a big revolution, fast enough to serve as a desktop computer for
most office people, students, and so on. It's pretty impressive and more
attention is being paid to it as a first-class computing platform as time goes
on.

Our desktop monoculture has led to serious security problems and massive
stagnation at least until recently. Even if you don't use a Pi you should be
thankful that they exist.

~~~
mixmastamyk
It should be and is decent, but when we tried we ran into numerous paper-cuts:

\- Needed to set up NTP to set the clock, it was always wrong.

\- It can't power down through software, so you have to shut down and then hit
the power switch like it's 1990.

\- Audio was a mess and had to spend hours researching how to write and edit a
bunch of config files to make it predictable.

\- The weird set up program means you need to do a lot of googling on how to
fix issues in a non-standard to linux way. It mostly works however.

\- Raspbian feels not quite "finished." Was much happier with Ubuntu Mate, as
they fixed 99% of the paper cuts like audio already, but it doesn't support
the 4 yet.

In short, it is a bit too cheap. I would have definitely spent another dollar
to see these hardware issues fixed. Or dropped the CPU speed a hundred MHZ,
whatever it takes.

In the end, a friend gave us a ten-year-old used iMac for free. I installed
Ubuntu Mate on it and it had none of the above problems. It's even a touch
faster.

~~~
sgerenser
A touch faster? The slowest 2010 iMac is the 21" with an i3-540 Dual core CPU,
Geekbench 4 is 2211 single/4466 multi. The Pi 4 is around 970/2000, less than
half the performance.

The Pi 4 has come a long way in speed since the lowly original Pi, but its
still not even close to a 10 year old desktop PC (or iMac).

~~~
geerlingguy
More people (even in 2010) used laptops than desktops, and the Pi is in line
with the numbers from the cheaper MacBook Airs at the time (I still have one
that runs... though it’s getting some weird green lines on the display).

But I can say that working on the 2010 MBA is not fun; one of my webcams
doesn’t even work with it (choppy at 30p), zoom is... slow at best, and web
browsing is a bit choppy even with ad blockers. Even then, it is a smoother
overall experience with that hardware and macOS than when I was testing 64-bit
Pi OS on the 8 GB Pi 4 a few weeks ago:
[https://www.jeffgeerling.com/blog/2020/i-replaced-my-
macbook...](https://www.jeffgeerling.com/blog/2020/i-replaced-my-macbook-pro-
raspberry-pi-4-8gb-day)

------
scottlamb
But does it work reliably?

I have a cheap Inateck dock which does support UASP (the FD2005). Two or three
times I had a bunch of errors like these:

    
    
        Sep 22 17:26:01 nuc kernel: sd 4:0:0:1: [sdb] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT
        Sep 22 17:26:01 nuc kernel: sd 4:0:0:1: [sdb] tag#2 CDB: Write(16) 8a 00 00 00 00 01 4d b4 c4 00 00 00 03 b0 00 00
        ...
        Sep 22 17:26:22 nuc kernel: usb 2-3: stat urb: no pending cmd for uas-tag 2
        ...
        Sep 22 17:26:22 nuc kernel: xhci_hcd 0000:00:14.0: ERROR Unknown event condition 34 for slot 3 ep 3 , HC probably busted
    

followed by filesystem corruption.

I found other folks complaining about similar problems, eg:
[https://unix.stackexchange.com/questions/239782/connection-p...](https://unix.stackexchange.com/questions/239782/connection-
problem-with-usb3-external-storage-on-linux-uas-driver-problem)

Then I disabled UAS. Haven't had any corruption since. It's not absolute
proof, but I believe the firmware and/or driver support for UAS with this dock
is buggy, and the older protocol works fine. I like speed as much as the next
person, but not if it means repeated filesystem corruption...

~~~
bsder
I'm all for blaming dodgy chipsets, but the Superspeed and Superspeed+ support
in Linux definitely has not had all of its rough edges filed off yet either--
especially the stuff that uses streams and asynchrony.

However, the article comparing to USB 1.1 is silly.

If you are on USB 2.0 (and I haven't bumped into anything mass storage in the
last 10 years that isn't), the bulk endpoints are quite happy to take
advantage of the USB 2.0 bulk packet size (512 bytes) and microframes (120us
turnarounds) with 13 packets per microframe.

That's going to get you in the 400Mbps+(40Megabytes/sec) range without a lot
of problem.

~~~
geerlingguy
That’s still less than half the speed you get from USB 3.0 even without UASP
from a cheap SSD. I drew the 1.1 comparison mostly to illustrate the fact that
the general protocol for MSD on USB was created in the late 90s for an
interface 400x slower than 3.0.

------
AnotherGoodName
>I found a USB 3.0 SSD was ten times faster than the fastest microSD card I
tested

I looked it up because i know enough about hardware that this shouldn't be the
case (even with a cheap SD Card). Seems it's a limitation of the Pi's SD Card
slot. [https://www.jeffgeerling.com/blog/2018/raspberry-pi-
microsd-...](https://www.jeffgeerling.com/blog/2018/raspberry-pi-microsd-card-
performance-comparison-2018)

~~~
aspenmayer
This is more up-to-date, and includes more models of Pi’s and SD cards. It was
linked in your post, but the post is now out-of-date relative to my link
below.

[https://pidramble.com/wiki/benchmarks/microsd-
cards](https://pidramble.com/wiki/benchmarks/microsd-cards)

~~~
geerlingguy
If you want to bookmark a page, that's the one that I update every year or two
(usually after a new Pi model is released). The blog posts fade a little in
relevance over time since I don't keep them updated.

~~~
aspenmayer
Thanks for all the work you’ve put in to this. I hope we see a Pi with NVMe
soon.

------
justanother
Excellent reading this; My Pi4/8GB cluster uses a 1000bT PoE hub for a
backplane, and / is an nfsroot from a node running a Sandisk UASP USB3 SSD.
Like the author, I'm blown away by the fun to be had with Docker and
Kubernetes on this cluster. Unlike the author, I'm having a great experience
with it as a Mac-replacing daily driver (thanks mostly to never running
chromium on the same node that runs gnome-session, to spread things out a
bit). Love to read about everyone's Pi cluster adventures.

~~~
geerlingguy
I think I could make it work, but the hindrance to my media (library of
65,000+ RAW photos from three different camera bodies and about 8 TB of 1080p
video projects (and counting)) workflows was what really killed it for me.

Even basic photo import / adjustments were a pain with any of the applications
I tested, and that was partly due to the Pi's slower GPU/CPU, the 64-bit ARM
paucity of desktop apps, and the lack of good (read: like Adobe/Apple/pro-tier
media apps) UI-based applications on Linux.

Most of the other issues (like window management and resolution problems) I
ran into with the 'Pi for a Day' project I could probably resolve with a few
more days' work.

------
totalZero
Friends, I have a question but I don't have a Pi 4 sitting around to test
this:

Is it possible to run a Pi 4 using UASP if I set it up to boot from a decent
(Sandisk/Samsung/etc) USB 3.1 pen drive?

Also, is this a reliable setup, or will the device be more stable if booted
and run from a MicroSD?

~~~
geerlingguy
I'm going to be testing a few different thumb drives in my next post / video.
I have a couple SanDisk models and an Arcanite USB 3.1 drive that I'll be
testing... but I don't believe they use UASP (haven't had time to confirm that
suspicion yet).

~~~
Legogris
Thanks Jeff, you’re doing a great service to the community here. I’ve tried to
hunt around for a good system drive and fell short so far; support for
UASP/TRIM in thumb drives is both rare, and seldom specified by manufacturers
even when present. Looks like the only way is to buy a bunch and try.

------
Abishek_Muthian
I've been using UASP SSD drives in Linux for ~ 5 years, the main issue I face
is not performance related but w.r.t monitoring.

S.M.A.R.T doesn't behave well with UASP, so not all Smartmontools's feature
work[1]. Especially, LifeTime of the SSD don't and they are crucial. I had to
resort to calculating remaining life of the SSD manually from the reported
lifecycles and negating it with the manufacture's MTBF if they're reported.

I've faced this with devices from varying price range [USB 3.0/3.1] -
MyDigitalSSD(OTG), Samsung T5, various ORICO USB SSD enclosures with Samsung
EVO/Crucial SSDs on host systems of different architectures(x86/ARM).

How are you all monitoring the life of your USB SSD on UASP in Linux?

[1][https://www.smartmontools.org/wiki/USB](https://www.smartmontools.org/wiki/USB)

~~~
elFarto
I have a USB 3.0 Orico enclosure with a Pi4 and I had to pass '-d sat' to
smartctl for it to work:

smartctl -a -d sat /dev/sda

Newer versions of smartmontools shouldn't need it, only the older one that
Raspberry PI OS has.

------
bhouston
How does a USB 3.x NVMe drive function on the RPi4? I notice those on Amazon
as well.

------
dmitrygr
What possible "controller feature" would be needed from the controller on the
hardware side to support just another USB protocol?

~~~
kevinoid
I was curious about this as well. The issue appears to be lack of scatter-
gather support. See:
[https://github.com/raspberrypi/linux/issues/875#issuecomment...](https://github.com/raspberrypi/linux/issues/875#issuecomment-482533041)

~~~
dmitrygr
which is not required to implement any usb protocol. one can easily use normal
DMA and still get a speed gain. since the bottleneck is _not_ CPU, making it
work a bit harder putting together those DMA descriptors is worth it.

------
jokoon
Why can't the RPi have at least 1GB or 2GB of flash storage on its board? My 4
year old, 80 euro android has 8GB.

It's not a secret SD cards quickly fail.

I've not bought the RPi for that specific reason. Even if the RPi is 5 times
slower, having reliable storage seems seems like a most important feature.

~~~
kingosticks
Because forcing everyone to pay a chunk more money for on-board storage which
not everyone needs is unnecessary. Their decision shows that cost is a more
important feature and I don't think that should come as a surprise, there's
plenty of stuff out there justifying the various decisions and it all comes
down to cost.

~~~
geerlingguy
I'm guessing if the RPi Foundation went that route with the Pi 5, it would be
similar to the Compute Module where there would be a cheaper 'Lite' version
without eMMC then more expensive ones with eMMC.

------
rgomez
Just did a "dd if=/dev/zero [etc]" in my Rapberry Pi 4:

4974456832 bytes (5,0 GB, 4,6 GiB) copied, 25,4896 s, 195 MB/s

Using this controller and a Samsung EVO SSD drive, I'm really happy with the
results: [https://geekworm.com/collections/raspberry-
pi/products/raspb...](https://geekworm.com/collections/raspberry-
pi/products/raspberry-pi-4-model-b-x825-2-5-inch-sata-hdd-ssd-expansion-board)

------
TaylorAlexander
Well, looks like I need to ditch the SD cards on my Pi4 based robot! Seems
like this would improve reliability over SD cards too.

------
neskiredk
I honestly thought this was common knowledge.

