

Why Linux Doesn’t Need Defragmenting - plessthanpt05
http://www.howtogeek.com/115229/htg-explains-why-linux-doesnt-need-defragmenting/

======
sciurus
On ext2/3/4 filesystems, there are a couple handy tools for measuring
fragmentation that you probably already have installed

To see how fragmented a file is:

    
    
      filefrag -v filename
    

To see how fragmented your free space is

    
    
      e2freefrag devicename
    

<http://linux.die.net/man/8/e2freefrag>

<http://linux.die.net/man/8/filefrag>

To learn about the intricacies of how fragmentation can occur in ext3 and some
better tools for measuring it, see the fragmentation section of Oliver
Diedrich's article 'Tuning the Linux file system Ext3'.

[http://www.h-online.com/open/features/Fragmentation-746701.h...](http://www.h-online.com/open/features/Fragmentation-746701.html)

Extents, multiblock allocation, and delayed allocation make ext4 more
resistant to fragmentation than ext3, but the developers are still working on
a defrag utility.

<http://kernelnewbies.org/Ext4>

------
naner
How does ext* move files around to reduce fragmentation if the disk is full?
I'm glad you asked.

Take a look at your filesystem properties:

    
    
      # tune2fs -l /dev/blah
    

In there you'll see a property called "Reserved Block Count". That is space
reserved for kernel use to avoid fragmentation, continue logging, and for any
other tasks the kernel/root user may need to do on a full filesystem. Users
cannot write to this and your disk utilities will report your disk being full
even if this reserved space is still available.

I believe it is typically 5% of the partition size. If your partition is
hundreds of gigabytes then the reserved portion may be tens of gigabytes
(multiply the reserved blocks by the block size to get total byte size).

You can change the percentage of this portion with the tune2fs -m flag. I have
filled up a 500 GB backup drive with 0% reserved blocks (%5 reserved would
have meant that 25GB was unusable) with no issues. Obviously, you probably
don't want it set to 0% if you are constantly changing the contents of that
partition.

------
trentmb
"If you do have problems with fragmentation on Linux, you probably need a
larger hard disk. If you actually need to defragment a file system, the
simplest way is probably the most reliable: Copy all the files off the
partition, erase the partition, then copy the files back onto the partition.
The file system will intelligently allocate the files as you copy them back
onto the disk."

So... buy a larger hard disk, or get a second disk to copy everything to
instead of a defrag utility?

I think I'll take the Windows approach here.

~~~
toemetoch
I'm a bit puzzled by the second method (with the partition swipe and copy
back) .

So, linux scatters the files. My partition is almost full and fragments. Now I
copy all the files to a second location and wipe the partition. Now I copy
them all back. Won't that just scatter all the files and fragment all over
again ?!?

~~~
dredmorbius
In the first case, the file create & scatter may have occurred in stages,
leaving less optimal disk utilization.

By copying files back en mass, fragmentation / allocation will be more
optimized.

If you have insufficient disk overhead and continue the file
destruction/creation process, you may end up with higher levels of
fragmentation in future. This depends on your particular usage scenario.

If you're not underallocating disk, generally you won't have much of a
problem.

------
cabirum
1\. It's not hard to implement such file allocation strategy. If it was
superior, other OS'es would have adopted it long ago.

2\. Scattering files all over the partition makes the longest seek times. This
is actually the most inefficient solution for linux with its apps and sources
(usually) consisting of thousands small files.

3\. Windows does not need to be defragmented manually as it defragments itself
when PC is not in use.

~~~
bnegreve
> 2\. Scattering files all over the partition makes the longest seek times.

Yes but it isn't guaranteed that there will be less fragmentation on a windows
file system. It depends on when was the last defrag, on the way file are
created, and your disk occupation.

Also, when you scatter files on different disks it can be more efficient
because you can read several cluster simultaneously. But I guess Windows does
this as well.

------
Travis
Does this imply for certain contexts, the Windows model is superior?

Say you had a "write/update infrequently, read many times" system profile with
10% disk space avail. It seems like Windows would store these files closer to
each other on the drive, making reads more efficient.

Or would that be such a negligible difference that it's not worth mentioning?

~~~
gouranga
It's pretty negligible. I genuinely can't tell the difference between NTFS and
ext3 at least performance-wise.

However, the locking semantics on NTFS are shitty unless you know what you are
doing (i.e. use KTM/Transactional NTFS) which sometimes skews the statistics a
lot.

------
TwoBit
Windows seems superior to me. It performs better due to seek time reduction,
and it defrags and optimizes during idle periods. With Linux you have the
worst of both worlds: it's slower and when the disk gets near full you
supposedly have no choice but to re-do the disk.

------
rbanffy
"If fragmentation does occur, the file system will attempt to move the files
around to reduce fragmentation in normal use"

Can anyone give more details on this?

Oh... And the love in the comments on the original article is quite moving.

~~~
tedunangst
I can only speak to how BSD does it, but if you append 8k (any small amount)
to a file that doesn't have adjacent space available, the filesystem will
collect up to the last 64k and rewrite it all out in one block.

------
nitrogen
This might be more accurately titled "Why ext2/3/4 doesn't need
defragmenting." The advice given in the article:

 _Copy all the files off the partition, erase the partition, then copy the
files back onto the partition. The file system will intelligently allocate the
files as you copy them back onto the disk._

does not work for ntfs-3g. Whatever strategy it uses to allocate blocks, it's
terrible for streaming reads of large files (last time I checked about 1-2
years ago). So, if you're trying to use the copy-off copy-on method of
defragmenting an NTFS partition, don't do it from Linux.

------
thyrsus
In 30 years working with Unix, I've never encountered a situation where
"defragmentation" was the answer. File system too full? Dozens of times. So,
what usage pattern am I avoiding to live such a charmed life?

Anecdotally, I run a couple virtual machines on my laptop whose storage is on
an external drive that sounds like Geiger counter at Fukushima Reactor 1 when
the drive is busy. That is, when I've booted Windows 7. Not when I'm running
anything under Windows, Windows just needs to be booted. When I'm running only
a Linux VM, it drops to background level.

------
ars
The trouble is I did use a filesystem all the way to completely full, and now
it's severely fragmented (despite me resizing it larger) and I have no
practical way of unfragmenting it.

~~~
joejohnson
From the article: "If you actually need to defragment a file system, the
simplest way is probably the most reliable: Copy all the files off the
partition, erase the partition, then copy the files back onto the partition.
The file system will intelligently allocate the files as you copy them back
onto the disk."

~~~
gouranga
Isn't that a form of defragmenting? It also contradicts the point that it is
unecessary to defragment the volume.

~~~
dredmorbius
It's extremely rare to need to do this.

It's generally a sign that you're running your disks / filesystems
overutilized. Retaining 10-20% free space is generally encouraged.

If you do elect to do the "copy out, wipe, copy in" method, you can do this,
in general, while not completely losing the use of your system. There are some
tricks (union mounts, remounting filesystems) which will allow you to flip
between filesystems, in most cases. Specifics will depend on your partitioning
scheme, use case, and the filesystems under consideration (usually you'd be
dealing with user files under /home, spool or variable files under /var,
source trees, etc.).

As the article states, Linux systems generally experience very minimal
fragmentation, and dynamically resolve the issue for you, unlike Windows. You
can typically run a Linux system for years, given a 10-20% disk reserve,
without running into appreciable fragmentation issues. The gains from _any_
sort of manual defragmentation would be minimal compared to the cost of
performing the task.

------
yoasif_
Users can defragment files (copying files to a new location) automatically by
running something like pyfragtools:

<http://ubuntuforums.org/showthread.php?t=169551>

Other filesystems like ZFS and btrfs utilize copy-on-write to virtually
eliminate file fragmentation -- files are copied whenever edits are made to
the file.

~~~
drostie
Maybe ZFS/btrfs users don't care, but I would be really hesitant before using
a filesystem which does not respect the Unix `shred` tool. Like real paper-
shredders, I don't use them very much but when I do use them I really hope
that they work.

~~~
icebraining
Why doesn't shred work on ZFS/Btrfs?

~~~
drostie
They just said copy-on-write. I'm not entirely sure about the exact interplay
between shred and btrfs, but shred nominally just writes random data to the
file that was already there -- if you copy-on-write that, then you are just
copying random data to a different part of the filesystem, not overwriting
what's already there.

You could solve this by automatically shredding deleted stuff in the
background, but it would probably kill SSDs and other storage devices which
you don't want to write to too often.

~~~
dap
I don't know how shred works, but based on what you said, it assumes knowledge
of how the underlying filesystem works. If there's fault to be had (and there
probably isn't), it's not ZFS's.

You really consider being able to shred individual files a critical filesystem
feature? Over, say, not having to configure static partition sizes, having
snapshots, or being completely immune by design to large classes of data
corruption (including all those that require an fsck-like tool to repair)?

------
ams6110
What I haven't seen mentioned is that historically, Unix (which inspired
Linux) is a server/multi-user OS. All this thinking about "my" disk drives and
how "my" large files are allocated is pretty irrelevant when you have dozens
of users as well as system daemons accessing different files all of the time.
Your drives are seeking all over the place constantly anyway. Fragmentation in
such a scenario is much less of an issue.

------
ciupicri
XFS has some sort of a defragmentation tool - xfs_fsr:

    
    
        xfs_fsr [-v] [-t seconds] [-f leftoff] [-m mtab]
        xfs_fsr [-v] [xfsdev | file] ...
    

_xfs_fsr improves the organization of mounted filesystems. The reorganization
algorithm operates on one file at a time, compacting or otherwise improving
the layout of the file extents (contiguous blocks of file data)._

~~~
pyre
xfs also has a tendency to fill up the kernel slab space when stat'ing a bunch
of files/directories. This is really annoying because most "memory full" tools
don't delve into the slab space to see if the usage is something that will
free up automatically.

------
carterschonwald
this doesn't sound right... but maybe thats because I sort of assume any
"sufficiently smart file system" has sort of COW (copy on write) like append
semantics plus some sort of amortized compaction similar to a reasonable point
in the GC design space of copying vs mark-sweep. (realistically most File
systems don't do something like that, but its what I imagine "should be" the
case)

------
pooriaazimi
The comment thread on this article made me appreciate HN a lot more... It
reminded me of those silly schoolyard arguments.

~~~
nitrogen
I was surprised to see someone still commenting about how you can't use Linux
in the "real world" in 2012. So my world's not the "real world?"

------
wwrap
A lot of questions being asked here are already covered in this thread from a
day ago:

[http://dd.reddit.com/r/linux/comments/uc1o9/why_linux_doesnt...](http://dd.reddit.com/r/linux/comments/uc1o9/why_linux_doesnt_need_defragmenting/)

------
steve8918
So Linux DOES need to be defragmented, but the only difference is that it
defragments automatically, unlike Windows which requires an external
defragmentation tool.

So really the title should be "Why Linux Doesn't Need Defragmentation Tools".

