Depends a lot on the memory pressure. If you can be fairly certain the data is (or will be) resident in memory, mmap is basically unbeatable. If you can't (because the data is larger than RAM or there's other stuff competing for RAM), mmap can have gnarly system-wide performance implications[1].
In my case: its a PROT_READ, MAP_PRIVAT map and i cannot get a SIGBUS since i tell the kernel to handle it all for me, thanks to liburing, instead i get a short send in that case.
To be more precise: i use that map to send assets out directly to clients from a zip file.
Its a new web server i am building and its the fastest way i could find out.
Just switching from epoll to liburing made the server ~45% faster too, its ridiculous. It can serve 10 gigabyte per second with a single thread, or around 10 million responses per second with h2 and 32 multiplexed requests.
I had to write a new http load generator for that since i couldn't find one which could generate enough load to saturate my server or be fast enough to withstand it.
??? The only configuration that will allow disk reads at 10 GB/s is if you're using PCIe 5.0. PCIe 4.0 or lower, and SATA will not drop out long before that.
He's very clearly hauling data straight from the page cache.
to not let the bytes i forward enter userspace, i serve from the page cache for the asset path, directly from a zip file.
i open the file, mmap it, close it and tell io_uring_prep_send which bytes from the mapping to send, this also saves me from a possible SIGBUS cause the access to the mmap happen inside the kernel, and when a SIGBUS would happen in userspace the kernel just reports a shorter send in cqe->res
The Linux-specific MADV_POPULATE_READ is much more reliable than MADV_WILLNEED if you want to implement read-ahead for a memory-mapped file.
In my opinion, the POSIX advices specified for madvise are useless or even dangerous.
On Linux, for precise control of memory-mapped files one should use only these 4 Linux-specific advices: MADV_COLD, MADV_PAGEOUT, MADV_POPULATE_READ & MADV_POPULATE_WRITE.
These should be used within io_uring, so that they will be executed asynchronously.
These have a well-documented meaning and using them carefully should be sufficient to reach optimum performance with mmap.
