
No Space Left on Device - morecoffee
https://carlmastrangelo.com/blog/no-space-left-on-device
======
JonRB
During my young naïve programming days (which coming to think of it haven't
really ended) I thought to myself "Why bother using some third party database
when I could just store user information in text files, the OS handles that
sort of thing really well, right?". I looked into it as much as I could be
bothered and found no huge red flags.

After a few years of this server merrily creating a new file for each user I
found that my disk was full, as this poster did. After a lot of hunting
around, I found out that I had hit my inode limit, a concept I was completely
unaware of.

I know for sure I won't be making THAT mistake again.

~~~
bArray
"After a lot of hunting around, I found out that I had hit my inode limit, a
concept I was completely unaware of."

What was the number you hit?

I have a flat file system for something I'm prototyping. The files are in a
database with a random SHA256 filename associated with them. I was thinking
about doing what Git does with creating folders for parts of the string to
alleviate the number of files in one single directory. I'm sure that makes
file retrieval a prettier process.

Do all the files for have to be in the same directory for the iNode limit to
be reached or just there in general?

 __NOTE: __I ask because the prototype isn 't ever meant to go into
production, but my customer keeps getting confused and I'm concerned this very
important system may inherit (or even use) my design.

~~~
LukeShu
It's a global limit per filesystem. You can check what the limits are for your
filesysems, and what the current use is with `df -i`.

The thing that git does is for speed.

~~~
bArray
I've apparently used 1241165 out of 19005440, or about 6.53% on my development
machine. It's not inconceivable to have 19 million files, though, if you had
one per user. That's not great.

How is that number calculated? It's less than 2^25 but greater than 2^24, so I
imagine it's even awkward to store. I guess that has something to do with the
file table size for HDD - you can't have more nodes than you have addressable
space for those nodes.

I knew Git did it for speed, but I was wondering whether it was also about the
number of files that can exist in a single directory.

 __EDIT: __Corrected percentage.

~~~
LukeShu
It's a property of the filesystem that is set when it was created. In the case
of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but
by default, it is the size of the filesystem divided by inode_ratio (probably
4096).

~~~
caf
This seems to be one of those bits of old UNIX lore that's being lost - it
used to be fairly well-known in sysadmin circles that if you were going to be
storing a lot of tiny files, you should create your filesystem with a higher
than usual number of inodes (the classic case was an NNTP spool mount).

~~~
technofiend
I literally just solved an issue this weekend that was inode exhaustion: there
were years of snapshots and traces laying around and the DBAs were confused as
to why the database couldn't create a directory to start and was exiting with
"No space left on device" when there was 5% storage free.

That lead to the second discussion of why you don't fix it by piping filenames
one by one to rm, but either use find's inbuilt unlink/rm if there is one or
xargs to rm if not.

~~~
greggyb
Is that second discussion just a performance thing, or are there filesystem
implications for one or the other?

~~~
technofiend
Just speed: you don't spin up a new process for each file removed if you use
xargs.

------
cyberferret
Brings back memories of the old DOS days, when there was quite a low limit on
the number of files that you could store in the root directory. I can't
remember the exact numbers, but it lead to my early habits of creating a new
sub directory for every little thing so nowadays (over 3 decades later) nearly
everything I need is at least 5 levels deep! ;)

NB: My current working folders are now in Google Drive, but is essentially the
same folder structure as I started on our Novell server about 25 years ago...

~~~
jaredsohn
> when there was quite a low limit on the number of files that you could store
> in the root directory. I can't remember the exact numbers

512?

[https://en.wikipedia.org/wiki/File_Allocation_Table#Final_FA...](https://en.wikipedia.org/wiki/File_Allocation_Table#Final_FAT16)
(see the last paragraph.)

~~~
cyberferret
That sounds about right. It was a ridiculously low number, and even back in
the days of 10MB hard drives, it was easy to bust that number with just batch
and text files...

------
ajuc
I've had similar error in production. We had linux server with C++ program
doing many things, and in the process it created temporary files for a barcode
printer. The files were named uniquely, the server had lots of free space, we
left them there to allow easier debugging if sth wrong happened (it happened a
few times each year - we created the labels to print by glueing escape codes
and sometimes there were labels that didn't worked for one reason or another),
and only cleaned the old files when there was not much space left.

At some point there was a bug when sometimes users got "wrong permissions or
no space left" errors (sorry don't remember the exact error and IIRC our app
just tried to save a file and failed with unspecific error if it couldn't
write the new temporary file, so we haven't had the exact system error).

What was weird is - there was still lots of space on that disk, permissions
were OK, and I've tried creating new files there, both from our app and by
hand and it worked.

It turned out after a few days of frantic debugging, that when there's too
many files with similar names in one directory - creating a new file can fail
because of hash conflict, depending on the filename only. So,

    
    
        ls A1234.txt S1234.txt # no results
        touch S12345.txt   # fails
        touch S1234.txt   # works
    

:) It was one of the biggest WTF moments I had while programming.

------
anilgulecha
A good filesystem to use as pseudo-db is ZFS. It's designed to have
practically limitless number of files, and it's snapshotting/cloning come in
really handy for things like backups etc. Also you get full checksumming.

~~~
fnj
ZFS does have a hard limit of about 280 trillion files in a single directory.
It's actually just about the most confining limit it does have.

~~~
anilgulecha
Correct. Hence, practically limitless.

~~~
bbcbasic
No one will ever need in 280 trillion files. Ever.

~~~
liotier

        "No one will ever need in 280 trillion files. Ever." ~bbcbasic on Hacker News, September 2016
    

Found in someone's email .sig in the year 2046...

~~~
bbcbasic
That typo is immortalised :-o

------
rshm
No date on the post. Probably old and fixed now. Just checked with a random
folder, it happily supported 782292 files.

~~~
Xylakant
Since the root cause is a hash collision in the dir_index code you might just
have been lucky and the author of the article unlucky.

~~~
rshm
Probably lucky. File names are md5 of the content, i am not sure if it helped
on collision on 32 bit md4.

The linked post mentions dir_index default and tune2fs -l lists the dir_index
under Filesystem features only.

Not knowing this default behavior, i was considering enabling dir_index
explicitly to aid chocking rsync.

EDIT: I ran the reproduction code with 200K instead of 100K just to be sure.
There was no problem creating file as well as copying all the files to a
different folder. The disk was formatted about 3 months ago and mounted with
default options "ext4 noatime,barrier=0 0 0" . On original as well as copied
folder `ls | wc -l` still gives 200000.

------
Animats
Classic UNIX problem. Not enough error codes, and too many errors map to the
same thing.

------
Nux
Yes, this "silly" kind of thing bit us a few times as well.

It's good RHEL/CentOS 7 defaults to XFS now which doesn't suffer (as much)
from this problem.

[https://gagor.pl/2013/11/inodes-on-xfs/](https://gagor.pl/2013/11/inodes-on-
xfs/)

------
crgwbr
> _The technical reasons for this are boring and I really don’t care why; I
> just want to trust that my filesystem will do the right thing._

Statements like this make me trust a developer significantly less. It's our
job to care about "technical reasons."

------
caf
This does not seem like the exactly correct diagnosis.

ext4 doesn't care if there's collisions in the dir_index hash - they're
entirely expected. The dir_index hashes are just used to look up in a tree,
the lowest level of which points you to the directory block that contains all
the directory entries for names with hashes in a given range.

However, those hashes do get used for the directory position used by telldir()
/ seekdir() - and those are only 32 bits on 32 bit architectures. So I suspect
that something involving those calls is generating the ENOSPC.

------
jobigoud
I came to the same conclusion about sharding files from the get go. I had a
similar experience where my host wouldn't accept to synchronize a directory
with more than 5000 files in it. (Software producing the files is my own note-
taking application where each page is stored as an individual file to speed up
backup of only the pages that changes).

~~~
vog
Wouldn't 5000 files of note-taking easily fit into RAM? Why bother dealing
with thousands of small files? Also, rsync does quite a good job with
intransferring multi-MB (or even larger) files efficiently while copying only
what's actually changed.

~~~
jobigoud
I feel more secure by dealing with many individual files. For example in case
of a conflict during sync, I can visually diff between the two pages very
easily. This happens once in a while. Also for example if more than 5% of
files have changed, the sync is stopped and asks for manual confirmation to
prevent catastrophies.

------
Neil44
I thought this changed from ext3 to ext4? I know I've run into this issue
before and moving to ext4 was the easiest way to resolve it. Ext4 is much
quicker at doing operations on very large directories too.

------
guroot
Things like this is why I use XFS anywhere I'm not using btrfs, or ZFS.

~~~
lake99
I've had some bad experience with btrfs. When I had to hard-reset a hung
computer, some directories became unremovable and unrenameable. All else
failed, and I attempted to --repair, it did so much damage that I had to
reformat the disk. This was probably about 3-4 years ago. Have you had such
issues on your preferred filesystems in the last couple of years?

The reason I stay with ext4 is that it has proven amazingly robust in the
event of hard reboots.

~~~
guroot
In my experience XFS is rock solid. We use it almost everywhere at work. I've
never had an issue with hard reboots with it. Only recently started playing
with btrfs, so can't speak to it's reliability from personal experience.

~~~
lake99
I hard-reboot about once a week, caused by my carelessness and running out of
16GB of RAM. I always recommend against btrfs for system partitions, and if
you don't make frequent backups, against user partitions too. Read through
"man btrfsck", that should caution you sufficiently. I'll try out XFS on an
external HDD I'll buy soon.

~~~
guroot
You really shouldn't need to be hard rebooting just because you ran out of
ram... did you try a sysrq f (invokes the out of memory killer) first? If that
doesn't work you should at least be able to force a sync and remount read-only
before taking it down. If you have access to a keyboard and the kernel is
still responding to sysrq you should never just hard power off without at
least trying to regain control of the system.

Some reading if you're not familiar with sysrq:
[https://en.wikipedia.org/wiki/Magic_SysRq_key](https://en.wikipedia.org/wiki/Magic_SysRq_key)
Depending on your distro you might have to enable it, I believe ubuntu (for
some dumb reason) ships with emergency sync as the only command available.

Enabling it should amount to adding: kernel.sysrq=1 to /etc/sysctl.conf on
older machines or editing /etc/sysctl.d/10-magic-sysrq.conf on newer ones.

also if you are running out of ram that often you might want to consider a
swap partition. Or if you have one already try setting a more aggressive
swapiness value.

------
nikbackm
Huh, max 32000 files in a directory seems awfully low. Even crusty old NTFS is
supposed to support 4294967295 files.

Explorer will not be too happy displaying such a folder though!

~~~
fnj
ZFS, which is known for having crazy high limits, will fit somewhat over 280
trillion entries in a single directory. Curiously, though, you can have over
18 million trillion _file systems_ in a single storage pool.

------
vacri
"If you're working with huge files or a huge number of files, use XFS" went
the slogan.

I was hit by this recently - with our 500ish (!!) entries in /node_modules and
their dependencies of dependencies of dependencies, a standard 8GB VM
formatted with ext4 just simply didn't have enough inodes for us to have two
copies of our app (deployment swapover). Had to increase the size to 20GB,
just to get the extra inodes. XFS would have handled it much better (from
testing on a parallel machine). It was inode exhaustion, though the 'disk
full' error message was the same - confusing at first, as df showed 40% free
space...

~~~
fnj
You could have also tuned ext4 when you created it with mke2fs. You can
specify with -N the max # of inodes desired. Granted, it's not something very
many people consider when creating their filesystems; for one thing it's hard
to predict how many you will need.

~~~
watt
It would be great if it had switch that simply says "double the default number
of inodes". Per stories in comments, seems that on large file systems it's
easier to run out of inodes, the default must be too low.

