Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Why isn't BTRFS the default FS in home-oriented Linux distributions?
37 points by develatio on April 22, 2023 | hide | past | favorite | 69 comments
I have been looking how BTRFS has been progressing. It supports CoW, snapshots, data deduplication, compression, data integrity, (...), a list of features that are unavailable in EXT4 (the current default FS in most home-oriented Linux distributions).

For granted, EXT4 keeps being faster than BTRFS in most synthetic benchmarks, which makes it a better choice for servers, but this difference in speed isn't noticeable in domestic workloads; domestic users would greatly benefit from the features offered by BTRFS. For example, Linux Mint has Timeshift, a backup solution that uses BTRFS volumes to make system snapshots.

This leads me to my question: Why haven't we switched to BTRFS as the default FS for home-oriented distributions? What features / reasons are holding us to keep using EXT4 as the default FS?




The rule of thumb is that a filesystem needs about a decade or so[0] to iron out the bugs and become reliable enough for regular use by regular users. Btrfs is 14 years old which means it's old enough to start getting adopted, but changing the default filesystem in a distro is not a particularly quick process, since most people simply don't care about most of the advanced features. Plus since data loss is such a feared problem, any anecdotes about it will cause a massive hit to the filesystem's reputation, regardless of the anecdote's validity. Right now Fedora and OpenSUSE use btrfs by default, likely because they're upstreams of enterprise distros where users will use those advanced features, so Red Hat and SUSE want to make sure it's as solid as possible before potentially rolling it out to paying customers.

Personally, I'm a happy user of btrfs installed by default by Fedora and haven't had any problems with it, but I'm also not using any of the advanced features.

[0] Source: casual conversation and poorly remembered anecdotes


It already is. OpenSUSE and Fedora both use brtfs by default now. It’s by far the most sensible fs for modern desktop usage and a killer feature of Linux.


To be precise about the "now": Fedora switched to btrfs by default a couple of years ago. OpenSUSE has been using btrfs by default for at least a decade.


Ubuntu Lunar Lobster now also uses BTRFS instead of ZFS by default with the new Flutter-based installer. It's also the suggested FS for Arch and Gentoo.


ZFS was never the default in Ubuntu


I was pretty sure it was, it was using ext4 as the default then I imagine? Anyways, the change then was that ZFS is not an option anymore instead of stopping being the default.


Just a friendly warning: BTRFS isn't anywhere near as robust as ZFS, which in turn has had a long history of problems.

There were some threads on HN related to BTRFS which made me look into its suitability as a RAID5 or RAID6 file system, and what I saw made me back-pedal very fast.

Its support for RAID1 or RAID10 (mirroring or stripe-mirror) is okay, but it has glaring issues in all other modes that have gone unresolved for many years. There are some fundamental design issues that might mean that these problems will never get solved, with data loss or data corruption being the inevitable consequence.

Storage is like multi-threading: you've either mathematically proved the correctness of your algorithm, or it is Wrong with a capital W. Storage is not like a web app where an occasional HTTP/500 is no big deal and recoverable. You stuff up something even a tiny bit, and the consequence is shredded or lost data and a very bad day for someone somewhere.

I'm just not seeing the right attitude from the team working on BTRFS. They've been very lax about data integrity issues, recovery from expected failures, etc...

My advice is: stay away.


In my experience, btrfs has one significant disadvantage over ext4. It's the behaviour once the partition is full. It's relatively easy to get stuck in a state where you can't delete anything because of the fullness. In some cases it's only recoverable by extending the partition, deleting, and then removing the extension.


I am using both. I've lost data on both, and near misses that required hair-raising metadata rebuild.

BTRFS documentation still greatly trail behind ZFS, imo. It's RAID1 implementation is subpar in terms of monitoring.


> I've lost data on both

Can I ask about the circumstances of losing data on ZFS? I've been viewing it as 100% rock-solid, so I'd very much like to know about any failure modes I'm missing.


I know of two instances, but they were short lived. One revolved around hole_birth, and the other around reflinks and sendfile... but that second one could have been a python/portage issue instead.

There was one instance of issues with encryption, but the feature was in testing at the time, and has long since been rectified.

As for failure modes... the first issue above caused issues with send/receive, the second issue caused empty/sparse files to be written instead of data, and the encryption issue caused issues with send/receive and/or garbage data.


iscsi target image file on ZFS RAIDZ zpool. I don't think it had much to do with iscsi though- the image file was affected because it's a TB-size file with lots of read/write. IIRC there were 2 other files that were also corrupted, but their sizes were in the range of teen GBs. I haven't seen any corruption from that installation since then.

The corruptions were identified by scrub after an unexpected restart, but the files were already corrupted before that (the system that was using the iscsi target behaved strangely whenever it accessed the drive.)

Server: Lenovo 3U server Xeon E5 v2 with ECC ram. I forgot what's the controller.


I lost data with Btrfs, whereas ext4 has been rock solid for many years. No thanks.


If you really care about your data you should be using a file system with checksums but in my experience btrfs is prone to fail if the hardware is not reliable.

About two years ago I tried it with second hand hdds thinking it would be good for checking the reliability of the drives. After writing some data in one of them and a reboot it failed to mount it again, total data loss while it would have been at most a partial data loss if using ext4 thanks to fsck. This happened 2-3 times as I was testing and didn't care about the data, on ext4 no issues at all. Still using btrfs as the root fs on a new ssd, the snapshots are really useful for backups, no problems in two years. But for reliability I would try zfs instead.


I personally switched to btrfs years ago due to it being (anecdotally) more resilient to power outages and forced reboots. With ext4 it would sometimes result in corruption but btrfs so far has been rock solid. Not a big deal of course but 0 corruption per year is proportionally infinitely better than 1.


I experience somewhat frequent power outages and my ext4 computers (3) have not had corruption afaicr.


i had many friends who were very vocal about btrfs. most of them lost their data at one point, with btrfs, due to data corruption.


I've only heard of BTRFS corrupting data in RAID configurations, but for a single disk laptop setup it seems perfect to me. Automatic snapshotting has already saved my ass several times.


If we're trading anecdotes, I managed to lose 2 root filesystems to BTRFS, on a laptop with a single drive, on OpenSUSE (which has been using BTRFS by default for ages, i.e. the one place I expected it to be the most stable). I think it was some problem triggered by filling the filesystem, because that system had too small of a root FS and snapshots piled up too fast, but I don't know for sure what happened, only that I couldn't find a way to recover it short of wiping root and reinstalling (thankfully, I had a separate home so that was actually okay). And to answer the obvious question, I don't know that it wasn't a hardware problem, but the home FS on the same machine, also on BTRFS, never gave me any problems, and honestly if the filesystem's reaction to a hardware error is to hose the whole filesystem that's still a hard pass from me.


Nope.

Sorry, it's the only FS I've lost data with. It seems to have a problem when disks start getting hardware errors and some blocks become unreadable. I mean of course any FS suffers in this situation, but most don't lose the entire volume as easily as btrfs seems to.


My question is why in 2023 isn’t there some better efforts to at least bring third party support for the various FSes to the various OSes?

Or is it a side effect of streaming everything? Ie who cares. Ie file systems are solved tech and nothing really new until permanent storage is as fast as RAM in all aspects, replaces it, and we get some new computing paradigm.


> My question is why in 2023 isn’t there some better efforts to at least bring third party support for the various FSes to the various OSes?

From the FS rather than the OS side, but I feel like support is pretty good?

ZFS supports all the big ones and many smaller options - Windows, Mac, Linux, FreeBSD, NetBSD, illumos (albeit with some slight incompatibilities depending on enabled features). The only OS that I kind of miss it on is OpenBSD.

BTRFS is, I grant, behind in this aspect, but it still has Windows and Linux, which is... a start, at least.


Windows is listed as alpha? Not sure about Mac support. https://openzfs.org/wiki/Distributions


1 - increased complexity, makes troubleshooting harder

2 - higher risk of losing data

3 - harder to explain and understand vs ext4.

4 - has more special needs. in some cases you need way more free storage for rebalancing stuff around... you don't need that for ex4.

---

now ext4 is surely worse than btrfs... but for the average home user btrfs is more of a problem than a solution, probably.


Personally I prefer ReiserFS, it has always been a killer file system in my view.


Yes, it has that one killer feature the other FSs so not have.


I'm not a fan of my wife so that feature is amazing!


I see what you did there! :')


I never used any of its advanced feature, and it makes managing volumes unnecessarily difficult


I would love to learn why ZFS and BTRFS are competing to a common-ish goal?


They were created by different people at different companies with incompatible licenses for different operating systems. In my personal and very speculative opinion, if BTRFS didn't have a reputation for eating data and ZFS wasn't licensed in a way that's not quite compatible with the Linux kernel, one of them would have won by now.


ZFS came first and Oracle started btrfs to try to clone it, before they acquired Sun anyway.

btrfs has never really matured. It has only a fraction of the features of ZFS and generally a lot more unstable.


> before they acquired Sun anyway.

This is the one that really burns me. There was a moment there where Oracle could have re-licensed ZFS, gotten it merged directly into linux, and had the best file system for their Linux distro. They even did it for dtrace! Instead, whether it was because they were so attached to their own pet project or other reasons, they just did nothing and now we get to live with the consequences forever because they don't own all the copyrights on OpenZFS so they can't even change the license on it now if they wanted. (Strictly, they could relicense their own Solaris code for ZFS, but that would be quite an effort to rework and leave us with two incompatible implementations.)


My raspberry pi with a slightly power supply got the filesystem into a state I couldn't recover it from when I tested it last.

I am using it again but on a NAS with a backup battery.


How long ago did that happen?


I've considered trying it several times, and each time failed to find any practical use case for my scenarios.


Have they fixed raid 5 yet? Last time I checked, they said "we don't care" and declared it stable.


Home distributions are very much not the right place to look for raid 5. That's small server territory.


Meanwhile, my desktop, running raidz2 (RAID6 equiv): works great.


Not everyone can afford a dedicated NAS/file server.


No one designing defaults for a home distro will tune it for the server use case by default, so raid 5 considerations are totally irrelevant to OP's question.


I hate to break it to you, but distros have been supporting lvm and mdadm for decades.

The only thing(s) that define a "server" distro are support contracts and ego. Ubuntu server is no different than standard Ubuntu when both use and can run the exact same packages.


I think we can both see that the question has been about defaults, which differ between Ubuntu server and standard.


What do defaults have to do with it? Like, sure, if I use a desktop spin of Ubuntu on a NAS I expect performance to be worse, but I don't expect it to lose my data. That's not a bad default, that's bad code.


OP's question:

"Ask HN: Why isn't BTRFS the default FS in home-oriented Linux distributions?"


It's not exactly a showstopper for btrfs, so that makes sense.

You can still use with with plain old MD raid.


How often are people reevaluating their file system choice? As excited as Linux users tend to be about the little things, the file system is so close to being unnoticeable day-to-day that it’s not worth the hassle when the current one works perfectly fine.


It's unnoticeable when you don't reevaluate your choices. Once you get into the habit of automatic snapshots, ext4's lack of snapshotting is very noticeable.


I'm not GP, but I'm curious. What am I missing out on? When would I benefit from having a snapshot on say my laptop?


I've lost track of the amount of times I accidentally deleted a file I shouldn't have or messed up a git project in a way it'd have taken ages to untangle, said oops, remembered I have hourly snapper snapshots set up and just copied the previous state over. It's been a massive time- and occasionally lifesaver for me.


You know, that's very convincing actually! I'm obviously missing out, I'll need to mess around with it more.


If you decide to dive into it depending on your filesystem of choice I can recommend either Sanoid [0] and httm [1] for ZFS or Snapper [2] for BTRFS as automatic snapshot solutions. Good luck with your endeavors!

[0] https://github.com/jimsalterjrs/sanoid

[1] https://github.com/kimono-koans/httm

[2] https://wiki.archlinux.org/title/Snapper


Rolling back or cloning the snapshot lets you get back deleted files, or reverting a bad update or config change...

The snapshot is also a point in time to derive backups from, and usually costs nothing to make, as opposed to making a dd image of the disk, or tarballing everything, etc.

Whether you think you need those is debatable, but it's quick and easy enough, works for me.


One thing the sibling did not mention is you can use btrfs send/receive to create dead easy, fast, and simple to restore incremental backups.


I’ve heard it lacks maturity and of a few cases of data corruption. Those cases are no doubt outdated now, but I haven’t heard any news to sway me that BTRFS is ready for prime time.

Ditto for ZFS btw.


> Ditto for ZFS btw

This is untrue. ZFS is ready for prime time. The problem here is that it is licensed under a license that is not compatible with the GPL, so it will never be imported into the kernel. Because it's an external module, most distros dont offer it in their installer etc.

If you want zfs, you can either install it during install in expert mode, or go BSD where it is a first class FS


apt install zfs works just fine though? I’ve set it up in Debian many times in minutes, am I missing something?


Yeah, like I said, after you installed it, it's available. As mentioned here, it's also not in the main repo.


On debian zfs is available only as dkms, and from contrib repo insted of main


ZFS is stable and solid. It's hold up is licensing. ZFS isn't licensed in a way that can be included in the Linux kernel distribution.

Plus, my understanding is that Linus Torvalds has a dislike of ZFS because it came from Sun Microsystems.


What problem did Linus have with Sun? Oracle I understand, but Sun? Was it something to do with UNIX?


I haven't heard of him objecting to the Sun origins of ZFS, but he has warned people away from ZFS on the grounds of Oracle's litigious reputation:

> And honestly, there is no way I can merge any of the ZFS efforts until I get an official letter from Oracle that is signed by their main legal counsel or preferably by Larry Ellison himself that says that yes, it's ok to do so and treat the end result as GPL'd.

> Other people think it can be ok to merge ZFS code into the kernel and that the module interface makes it ok, and that's their decision. But considering Oracle's litigious nature, and the questions over licensing, there's no way I can feel safe in ever doing so.

> And I'm not at all interested in some "ZFS shim layer" thing either that some people seem to think would isolate the two projects. That adds no value to our side, and given Oracle's interface copyright suits (see Java), I don't think it's any real licensing win either.

> Don't use ZFS. It's that simple. It was always more of a buzzword than anything else, I feel, and the licensing issues just make it a non-starter for me.


Could GPL be amended to be compatible with ZFS? Something along the lines of a legal shim. (Clearly Linus already said no to this in extension.)


In principle I guess, but not in practice. Linux is distributed under the GPLv2 without the "or later" clause, so you couldn't simply convince the FSF to release a GPLv4 that allowed this. Nor do contributors sign away their rights. To re-license the kernel you'd have to get the consent of all the contributors, or replace the parts for which that was impossible.


Facebook was deploying BTRFS 5 years ago.[0] I haven't heard that they've changed or had any data corruption issues. In fact, they've integrated zstd compression into BTRFS.

[0] https://engineering.fb.com/2018/10/30/open-source/linux/


Facebook runs servers a lot different than a home oriented distro is likely to be run. I was there through aquisition from late 2014 to late 2019, and I mostly avoided production operations in FB land.

I don't remember hearing about any power failures --- either UPS and transfer switches all worked 100%, node replacement covered any issues 100%, or I just wasn't in the loop. This isn't common at home; most people have at least a few brief power blips every once in a while, and those with many often get a UPS, but home grade UPSes often fail. I'd expect at least one unexpected power failure per 3 years in a home environment.

Most of FB production is built from ephemeral containers. If a container host crashes and restarts and the filesystem is corrupt, the container would be rescheduled somewhere else, and the host would be sent for repair (reimaging) automagically, and a product team like mine would not really know unless they really looked, maybe just that the container moved. That's not realistic for home users.

The containers themselves were short lived for many reasons. I'd estimate most of them lived for less than a month. A home user distro install often lasts much longer.

That said, I don't remember seeing disk corruption, but I do remember a series of issues where we couldn't write to disk because it was full, with disk usage of maybe 60%. As I recall, there was only one couple week period where that happened. This was resolved in the moment by replacing affected containers with new ones on other hosts, and permanently by a fix in the host kernels; broken filesystems were not fixed, they were abandoned. This isn't great for home users either.

There was a separate class of operations for persistent data storage; although there was work to containerize that as well, I had little visibility. Persistent storage machines were different and I don't know if they used btrfs or not.

This all doesn't mean it's not a good filesystem for home users, just that Facebook's endorsement doesn't transfer as the use case is much different.


zfs is way simpler to manage than btrfs.

also, there does not seem to be a proper guide to btrfs. only random of bits and bytes here and there.

zfs on the other hand is very well documented.


> zfs is way simpler to manage than btrfs

Not sure I agree. On ZFS I have to worry about an out of tree kernel module and all the implications of that, extra things like setting the correct ashift, making sure the recordsize is tuned individually in datasets to not destroy performance for specific worksets, ensuring ARC doesn't cause OOM behavior with applications which allocate a lot of memory at once (since it doesn't use the page cache), etc etc.

With BTRFS however off the top of my head the only thing I remember doing was making sure nodatacow was set for subvolumes containing databases and VM disks.

You are absolutely correct about the ZFS ecosystem containing superior tooling however.


this.

that said, lots of people do. arch and gentoo default install instructions suggest it just fine, telling you the pros and cons over ext4




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: