On: The secret life of fsync

[mikepurvis]

How much are current database implementations hamstrung by the standard file semantics? If it's a lot, I'm surprised there aren't more databases that just mount a bare disk or partition and do whatever they want with it. Or ship a special kernel module they can talk to that implements a better interface to the underlying block device?

Or does none of this really matter in a world of virtualized-everything where your "block device" is still ultimately hitting multiple layers of actually-just-being-a-file on its way to hardware?

[hyc_symas → mikepurvis]

LMDB performance suffers when used on a journaling filesystem. Consequently, LMDB supports using a plain block device, to eliminate all FS overheads. And yes, all of the perf benefits go out the window if you're running on several layers of virtualization anyway.

[FridgeSeal → mikepurvis]

I was under the impression that MSSQL basically runs almost an entire FS/OS kernel inside itself to manage its own IO?

Scylla DB also goes to great lengths to optimise IO paths and talks directly to storage devices via DirectIO IIUC.

[meangrape → mikepurvis]

Thirty years ago it was standard practice to mount raw disks for Oracle to use.

[icedchai → meangrape]

~15 years ago, the last time I used Oracle in production, that was still standard practice. I'd be interested to hear from someone who's used it recently.

[needusername → icedchai]

That or ASM devices. To my knowledge io_submit and io_getevents only really work with raw partitions anyway.