Back in the early days of the PC when storage was very limited (my first HDD was only 10 MB), managing disk partitions was critical. Today with 20+ TB HDD and even 4 TB SSDs becoming affordable, partitioning is still important but for different reasons.
I joined a little startup called PowerQuest that was developing a partitioning product called Partition Magic after I wasted several hours for the umpteenth time re-partitioning my hard drive so that I could test out multiple operating systems (DOS/Windows, OS/2). It was a big success, but as drives got bigger the need to re-partition because the partition was full, diminished. Eventually, Microsoft included free partitioning software with their OS.
Today, partitioning can still be very important. You might want to install multiple operating systems on a single computer, each with its own boot partition and file system. You might want to keep your data and applications separate for access or in case a bug corrupts the file system.
But the sheer size of HDDs can also cause a problem. A 20 TB HDD can easily hold a few hundred million files. With new HAMR drives coming out, we will soon see drives capable of storing over a billion files. File systems like NTFS have a file record (FRS) size that is quite large (4096 bytes per file). When you have a billion files on a single volume, that is 4 TB just for the file table. Loading that in and storing it in RAM is problematic (not too many systems have that much RAM). Searching for files, checking the file system, or doing other operations can cause the file table to be loaded in its entirety.
Having multiple partitions, each with a smaller number of files on them can be a way to deal with this problem. I am working on a new system (Didgets.com) that seeks to address this issue with a different approach.
Disk management is something you mostly avoid (well, it’s abstracted away) if you have a stateless containerized infrastructure and write your data and logs to API endpoints (including S3 and similar) and managed relational DB’s. Granted if you start to run things like Kafka at scale you’ll probably need to have someone with some Linux system chops around.
I use disk partitioning to logically separate my files. This helps reduce the sizes of my backups and excludes system files from indexing/searching.
For example, WizFile and WizTree are two very useful programs that work best with files organized by partitions.
My partitioning strategy looks like this:
- C: Windows/system
- D: data I want in backups (Windows desktop/documents/user directories have been moved here)
- V: variable data that is either temporary or does not need frequent backups (Dropbox, Google drive, etc). Also most program files (installed via scoop)
Other types of OS/FS may not need partitions to accomplish this.
Withe rise of larger and larger storage/disks, partitions are MORE necessary than ever.
Of course, you can cut down the number of partitions you use, but that makes your system extremely fragile. It then becomes a question of 'all or nothing'. I'd much rather have a situation where only one or two partitions are corrupted than have everything fall over.
I joined a little startup called PowerQuest that was developing a partitioning product called Partition Magic after I wasted several hours for the umpteenth time re-partitioning my hard drive so that I could test out multiple operating systems (DOS/Windows, OS/2). It was a big success, but as drives got bigger the need to re-partition because the partition was full, diminished. Eventually, Microsoft included free partitioning software with their OS.
Today, partitioning can still be very important. You might want to install multiple operating systems on a single computer, each with its own boot partition and file system. You might want to keep your data and applications separate for access or in case a bug corrupts the file system.
But the sheer size of HDDs can also cause a problem. A 20 TB HDD can easily hold a few hundred million files. With new HAMR drives coming out, we will soon see drives capable of storing over a billion files. File systems like NTFS have a file record (FRS) size that is quite large (4096 bytes per file). When you have a billion files on a single volume, that is 4 TB just for the file table. Loading that in and storing it in RAM is problematic (not too many systems have that much RAM). Searching for files, checking the file system, or doing other operations can cause the file table to be loaded in its entirety.
Having multiple partitions, each with a smaller number of files on them can be a way to deal with this problem. I am working on a new system (Didgets.com) that seeks to address this issue with a different approach.