Hint: If you really want to use your filesystem as a database (and don't mind the haters, there's a long and storied history of doing just this), make sure you break it up into many subdirs usually based on the first few characters of the uuid.
Example: ./jasondir/aa/bb/cc/aabbccdd
You won't like what happens when you put 100k files in one directory.
Even if it's running on a filesystem that deals with it well, standard utilities like 'ls' don't deal with it well, as the default behavior is to sort. You end up having to look up obscure options, like 'ls -U' to disable sorting.
You can also run into issues like "Argument list too long". ARG_MAX is larger on linux than it used to be, but it's pretty short on older kernels. I assume similar issues might exist on other operating systems.
I'm not sure that's so much of a problem, I just tried:
mkdir uuid; cd uuid
uuid -v 4 -n 1000000 |\
while read uuid; do
touch $uuid;
done
And ran out of inodes, but:
time ls uuid|wc -l
425621
real 0m1.796s
user 0m1.552s
sys 0m0.240s
Sure, it's not exactly stellar performance for a linear scan of ~400k keys - but it's not terrible (for various values and expectations of terrible).
This is in a hyper-v vm on a Surface 4 pro/i5.
The fact that the "uuid" program can quickly generate uuids make me wonder if maybe one approach would be to generate uuids to (a) fifo(s), and then let db thread/processes read uuids from the other end?
Depends on the system, and 400k is short of where things tend to go bad on mine. At 800k files I see it taking 7 seconds.
Even at 400k files, you see longer wait times if you've aliased ls to ls --color (pretty common), or use something like ls -F. Either runs stat() on every file.
Then, somewhere in the 1m+ range, it gets unusable.
For troubleshooting or statistical reasons. It's serving as the backing data store. So piping through grep, or to wc, etc. Specifically not sending the output to the screen.
As this is a standard Ubuntu install, ls is indeed aliased to "ls --color" - but afaik ls as standard detects pipes, and turns off color (so you don't get a lot of control characters if you do "ls --color | sort > file.txt". Unless you use --color=always if I recall correctly.
Just wanted to differentiate between "select * from documents" and "select count(*) from documents" being slow.
Example: ./jasondir/aa/bb/cc/aabbccdd
You won't like what happens when you put 100k files in one directory.