As the person who tested the latest ntfs3 patchset, and had tested
many of those iterations in the past, I would really like to see
this *finally* land in Linux 5.14.
However, I get the feeling it's not going to make it for 5.14 *or*
5.15, and it seems like Paragon became discouraged by the lack of
feedback on the latest revision.
I know that compared to all you awesome folks, I'm just a lowly
user, but it's been frustrating to see nothing happen for months
with something that has a seriously high impact for a lot of people.
It's a shame, because the ntfs3 driver is miles better than the
current ntfs one, and is a solid replacement for the unmaintained
ntfs-3g FUSE implementation.
Seriously, such a well funded, high profile project like Linux should be more receptive to key QoL improvements and drivers. NTFS is the de facto filesystem for portable data storage, much as I wish an open alternative was there. I hope they speed up the merge of ntfs3 into the mainline.
EDIT 2: Since many comments replied with the same thing, I'll reply here: I wouldn't consider read-only support as support for portable data storage, since that allows you only to send data in one direction. When you want to copy a file from a Mac to a Windows computer, NTFS is not a solution now (out of the box at least), therefore falls pretty short wrt "portable".
exFAT is probably better in terms of cross platform compatibility and addresses a lot of the limitations of FAT while remaining portable.
Since Snow Leopard OS X is able to read and write out the box.
Linux natively supported from kernel 5.4.
Windows supports it from updated XP and onwards.
> exFAT is probably better in terms of cross platform compatibility and addresses a lot of the limitations of FAT while remaining portable.
Still it is a shame that we're kept back literally decades when it comes to file exchange due to corporate culture polluting every corner of IT. Ensuring file export options plus file formats full documentation, should be among the most important requirements any software should satisfy to be even considered for professional use.
Alas you are right in that this culture is all over Corporate World (yes not just America). Vendor lock-in is something that's been with us since forever and I doubt it will ever go away. They might have gotten better at hiding it from obviously being spotted but that's about it and in some other cases it's still just as clear as always but the marketing people are still very good at selling it to customers.
Android is annoyingly hit and miss. It's dependent on whether the device manufacturer wants to pay Microsoft royalty fees for it. So some devices do support it just fine and others don't. Often if a device has a microSD card slot and says it only supports 32GB cards it doesn't support exFAT since SDXC made the default file system for 32GB+ cards exFAT.
Microsoft exFAT patents expired in 2019, I think. It's included in the Linux kernel since 5.7 or some version around then. Android devices support exFAT depending on which Linux kernel version they have.
Maybe that will change in the future but most Android devices are not using a 5.x kernel yet. All my Samsung devices support exFAT and use Android 11 with a 4.14 kernel. My old BB Priv also supported exFAT with Marshmellow which definitely did not have kernel version 5!
And Microsoft ACTIVELY went against Android manufacturers to get licensing fees a decade or so back, so those that didn't want to pay are much more likely to have a hard ban on exfat in place that will take time to overcome even after Microsoft opens it
Not disputing your facts, but I've used journalled file systems in flash drives since forever and never had a performance issue that was not directly tied to the hardware, and most have lasted many years.
I prefer having to replace a cheap flash drive more frequently than to lose data altogether.
Of course, everybody's use case is different and tradeoffs need to be considered.
Not really.
FAT32 is very simple and normally has a copy of each FAT table stored as well. When something goes wrong with the first FAT table, there's a reasonable chance you can still use the 2nd copy. [1]
ExFAT is much more complex and AFAICT it does not have a 2nd copy of all of its data. [2]
Anecdotal evidence is that I personally lost data on ExFAT file systems, but also have had many customers with problems when they figured to use ExFAT for my backup products. If I see ExFAT in a support bundle than -by default- my suggestion for fixing issues is to reformat to a different file system. More often than not, that's the end of the open support case.
What if a bit flips in your hash file and changes that? Now you need to store your hash a bunch of times too!
I kid, this is what I do. I've recently been backing up my family photos/videos to various media and I've been hashing each file and storing a hash file alongside the media to compare to in the future.
Just recently I brought a Mac-formatted exfat flash drive to a print shop, where it failed at being read on a Windows box. I doubt it that they're using WinME.
OS X/macOS used to have some serious issues with exFAT but they may have been addressed in the few years since I last used exFAT on a Mac. Something to double-check before using the pair to store critical data.
exFAT is HORRIBLE. Sorry for the caps. Interoperabiliy between Windows and Mac is hit-or-miss. Formatting on Windows and it's random if Mac can read it. Unplug without ejecting and expect to wait hours for Mac to analyse and fix, or plug-in and out and in and out of Windows and hope it fixes it. Many times a day. Zero problems with NTFS.
FAT32 doesn't support files bigger than 4GiB, that's a huge limitation these days. But I don't disagree that calling NTFS "portable" is a bit of a stretch, although hopefully that'll change if good quality open source drivers become available.
A bit of personal experience: I've needed to create a Windows 10 installer from a MacOS computer a few weeks ago. Without the MS helper program I've figured I should perform the following steps:
1- download iso
2- format usb with fat32
3- copy the data
Apparently you can't just dd the ISO file into the USB block device like every other Linux or BSD ISO. It need to be FAT32 to be able to boot through UEFI.
The issue however is that the .wim file containing the OS installation source is bigger than 4GB. I think its sources.wim but I'm not sure. I had to install some command line utility from brew that did the job of "optimizing" the WIM file and managed to reduce it to a file bellow 4GB.
What does optimizing means? Better compression? Cleaning up crap inside the image? If their ISO tool probably does this without user interaction then why include a file bigger than what the FS can handle in the first place?
I don't know if this counts as some kind of dark pattern or MS being really antiquated in terms of bootable flash drivers.
What I'm doing is just using GPT partitioning, creating one NTFS partition and copy all files there. All my computers were able to boot in UEFI mode from this USB drive and successfully install Windows. With Linux it was a bit more cumbersome, but most Linux distros are hybrid ISOs anyway, so it's not as big issue. I know that UEFI BIOS is not obligated to support NTFS partitions, but my little practice they do.
Good to know that this is an option but from MacOS I don't have NTFS write capabilities. I think ExFat is available so I might try two partitions one being the EFI ESP and the second containing the rest of the install files.
NTFS has journaling,compression, encryption, max 255 character filenames, max partition size is hundred of TB rather than 2GB, generally faster performance wise.
Trouble is , FAT32 has more OS compatibility.
Would be nice if Linux did have first class support for NTFS as it’s superior. But right now for maximum compatibility you need FAT32
You're confusing the filename length limit with the limitation on the entire path length in the Windows API. NTFS allows paths to be up to 32k characters long, but each path component tops out at 255 characters.
Anyone know why? Given ext2 has had a public domain implementation for more than a decade? Is it because windows & mac won't support ext2 even for free? Seems like a better lowest common denominator fs than any of the fats?
That's because Linux and OSX ignore NTFS permissions making it work more like FAT.
Plug an existing NTFS drive from one Windows system into another Windows system and you get a similar issue where opening a home directory requires you to take ownership.
> that depends, exFAT lacks permissions and symlinks.
The lack of permissions is why it works as a portable file system. If you've ever tried to use, say, ext for this it becomes clear why nobody does that.
IME most external hard drives and USB sticks are marketed as compatible with both Windows and Mac, and ship formatted with FAT32. But maybe I'm the odd one here.
Literally NONE of the 100s of Tb I have purchased over the past year in formats ranging from SD card to USB stick to SATA drives have been formatted as NTFS. None of them.
They do, but only to read, not write. I think the current up to date recommendation for cross-OS compatibility on external drives is to use ExFAT, though I believe the concept of the ‘external drive’ itself should be deprecated.
I think they’re problematic because they’ve reached ungodly amounts of storage and I don’t think most non-technical people fully understand how much of a loaded gun a non-regularly-backed-up 2TB external HDD is.
As a real world example, I’ve had to help recover my aunt’s 5-year-old 2TB HDD that she dropped several times and had no backup of, and by helping I mean confirming she probably lost a terabyte plus of data unless she wants to pay five digits USD to pay to a recovery company for a way less than 100% recovery. She is a highly educated professional (a MD) and yet, it did not even occur to her that the drive will not only might fail, but rather, will fail.
To quite a lot of non-technical people an external drive appears to be the safest form of storage, considering its being physically visible and movable. The closest analogy is storing money under the pillow versus a in bank, in all the good and bad ways.
> NTFS is the de facto filesystem for portable data storage
I've never seen it used that way, guess I'm weird.
One hopes that the people who do use it will work on it, but given the patent fiasco with exFAT, I wouldn't either look at or use the code, personally.
I dual boot between Linux and Windows on my laptop, and I use three physical hard drives: two SSDs, each dedicated to a different OS, and one hybrid HDD with more storage that's "shared" between the two (and also used as a simple way off passing data between them).
Maybe things have changed now but NTFS was the only reliable option for that last drive when I first set things up this way a few years ago.
Me neither. I'm not sure what's up with all the talk here about NTFS for portable storage. If "portable" is meant as in USB key, I've never encountered one formatted that way. If it's portable as in a disk in an enclosure, then I suppose I can follow the rationale.
Is it? The kernel has a billion users who pay $0 for it. Most of the work is done by volunteers still. Oracle and RedHat have a few people looking at it full-time, and then Linus himself. This patchset is maintained by a third-party, Paragon, who also have an NTFS-for-Macs solution.
Hence:
>> We simply don't have anybody to funnel new filesystems
Portable means that you can stick a USB pendrive in your car, speaker, TV, printer and be able to at least read the media on it. exFAT or FAT32 are the standard here, not NTFS.
Computers are easier to deal with, there might be obscure programs that can read and write any known file system format despite what the kernel supports.
Way back there was a contributor Con Kolivas, that made scheduler improvements to make desktop computing nicer. But he quit because they never gave priority to desktop stuff. I remember back then it was not uncommon for your mouse cursor to just freeze and then come back to life.
The kernel group has different priorities to end users
TDLR: Needful, worthy improvements require champions. A thankless role. And few of us have the resources and tenacity to serve the champion role for the long haul.
In trying to understand organizational psychology and apparent apathy, I keep thinking of Stephen Jay Gould's theory of "punctuated equilibrium" meets the attention economy.
Metaphorically, organizations (and people) are information processors. Attention is our scarcest resource. We're all doing triage, at every level. Just trying to cope with the torrent of information, discerning signal from noise. Trying to prioritize efforts.
So it's almost inevitable that needful, useful, worthy things remain neglected, undone. Even in the absence of pushback.
Until some kind of crisis. When the neglected thing suddenly becomes very important.
Cue the fire drill. Something must be done.
The champion must be watchful, ready, proactive. Seize that very brief opportunity to sneak in their needful improvement. Ahead of the all the spazs pitching their weird and wrong and ignorant reactive proposals.
The only winning strategy is to just keep pushing. Patiently, diplomatically. Be that useful thing that everyone's dimly aware of.
"Oh, wasn't Con Kolivas working on that mouse jitter thing? Ya, I heard it's pretty good. Ah, it's ready to go? Cool, let's do it."
If your needful thing is truly worthy, be confident that eventually it'll percolate to the top of the triage stack.
This game truly sucks. But I don't see how it could be any other way with our current decision making structures.
You'll need someone working almost full time on this (like Jens Axboe on io_uring). I'm wondering whether it's some emergent behaviour to make sure you'll maintain the thing on the long-term and not just dump the thing on a subsystem guy and see ya!
> NTFS is the de facto filesystem for portable data storage
No, that would be exFAT and FAT32. Some form of FAT is basically supported by everything, which is why if anything is the "de facto" for portability, it's FAT. FAT is the netBSD of FS.
As everyone seems to be fixated on pendrives and FAT, consider also that virtually every Linux-based consumer-grade NAS needs to support R/W NTFS. Typically it was through FUSE with considerable loss of performance/efficiency.
The parent said "de facto filesystem for portable data storage", NAS for Windows is a very specific use case. For data portability, compatibility and wide adoption matters, and that is undoubtedly where FAT and it's variants rule.
I'd guess not since big G has been on the push to abolish external storage - and I wouldn't blame them, really hard to put data (especially application data) on hotplug removable storage.
The camera world has been using exFAT since quite a while.
Ext3 sucks under BSD's, a lot of them can't write to the volume, doesn't matter either EXT2, 3, or 4.
UDF as I stated works fine with udfclient under OpenBSD and it could be "mountable" even with base tools (read /usr/local/share/doc/pkg-readmes/udfclient in order to correctly format it with udfclient).
I copied files back and forth on USB drives written in under OpenBSD and the volumes mounted like magic under Linux. And the performance is not bad at all.
I will give a try to UDF -esp if has comparable write speeds with ext3 , but for sure I had no issues to _write_ on an external ext3 with NetBSD + FreeBSD (my last desktop OS, before move to Debian Sid)
Linux is well funded in the sense that corporations put big money looking after their own interest in the kernel. Anything outside of their respective areas of interest? Good luck.
Paragon, for better or worse, is just yet another corporation. They shouldn't really expect other corporations or volunteers to care much about their code, other than the fact that their code may break others'.
Now that they have Linus' blessing, they can just send a pull request. The work paid off in the end.
Can you explain what you mean by "portable data storage"? I'm used to hearing the word "portable" used to mean "cross platform", but if it is not supported well under Linux, then given how widely used Linux is, I would think ntfs would not count as cross platform.
I'm pretty sure that the grandparent meant portable quite literally, as in you can take it with you. Things like USB sticks, external hard drives, sd cards etc..
That was the impression I was getting, hence my question. But since thumb drives typically come preformatted in either FAT32 or exFAT, I thought maybe there was a third possible use of the word "portable" that I am unfamiliar with.
External hard drives with larger capacities and potentially larger files might be a use case for NTFS. While it isn't fast, ntfs-3g has worked fine for basic use for me. Someone said in another comment that ntfs-3g is unmaintained, though, which was news to me, and obviously not a great thing if true.
I suppose exFAT might be a viable option for larger external drives as well due to its higher file size limit. None of the other commonly used file systems (including native Linux file systems) seemed easy to use from both Linux and Windows the last time I looked into it.
Honestly ntfs-3g is fine. It's slow, which I'd always blamed on NTFS since NTFS is also clearly slow for large directories of small files on Windows also, but apparently this new driver is better in that regard.
Hey can you please stop posting unsubstantive comments and breaking the site guidelines? We ban that sort of account. I don't want to ban you but we've had to ask you about this before.
> FYI: There is also another cross-platform filesystem (Linux kernel,
Windows NT kernel, Mac OS X kernel) suitable for hard disks too with
POSIX permissions about which people do not know too much. It is UDF.
As I was struggling to find a way to share data between Linux/Windows with my gpu-passthrough setup, I'm excited to use this.
I think I've read something about 9p support in the latest Windows versions and I think QEMU supports 9p so that might become a good solution, too.
There are several platform specific gotchas with UDF. If you want to use UDF on a drive, I suggest using the script at https://github.com/JElchison/format-udf. It takes into account the platform specific gotchas and formats a drive properly to be usable across Windows, Linux and macOS.
Any chance that there's a way to actually encrypt a device set up like that?
I usually use LUKS and EXT4 for all my drives but lately I've needed to move files to and from Windows systems on occasion, and it kills me that (from what I can tell) there are absolutely no solutions that even attempt to do something like this.
Does everyone just walk around with unencrypted portable media? I don't really want to float sensitive data on unencrypted drives, even temporarily, since that makes the data vulnerable to things like wear leveling analysis etc.
mkudffs (a.k.a. mkfs.udf) on the whole block device, by default it uses --bootarea=erase, and results in a device that automounts on macOS, Windows 10, and GNOME and KDE (at least).
There is also --bootarea=mbr which sounds similar to what this script is doing but I'm not sure why it's necessary; maybe for older versions of Windows.
9p is not a file system but a byte oriented file based RPC protocol for transferring data. In plan 9 the idea is files are objects in the programming sense which you create/open/read/write/remove/close. Example: your mouse is a file, /dev/mouse, which you open to read the coordinates and click events.
In plan 9 files are served by file servers, programs which speak 9p. The plan 9 kernel knows nothing of disks or file systems living on those disks. In fact, you don't mount disks in plan 9, you mount 9p connections. So to access files on a Fat formatted disk you need a file server program called dossrv(4) which opens the disk file and listens for 9p messages and performs the necessary Fat operations.
Fun fact: A common plan 9 newbie mistake is to mount a disk file directly as in Unix. This causes the kernel to write a 9p T_attach message to the beginning of the disk overwriting the boot sector. http://man.postnix.pw/9front/5/attach
I've used it before for a USB drive and honestly I have had nothing but problems with it. I don't remember exactly what kind of problems but I do remember that I was frustrated enough after some time to switch to something else.
> Although an implementation may use a journal to protect metadata integrity, this does not guarantee interoperability between platforms since it is not part of the standard.
I was just looking at the most recent v26 patchset from June yesterday and it was really depressing. Paragon has been plugging away at it for close to a year (2020-08) and there has been seemingly no interest in this at all. As a regular desktop user who spent years dealing with the very inefficient FUSE ntfs-3g implementation the disconnect seems very real.
It's also single threaded, i.e. it can only handle one read/write request at a time (fuse can do more but ntfs-3g doesn't implement it). That has kind of acceptable performance with hard disks but with SSDs it is really slow.
I haven't messed with NTFS in years but from what I remember the Linux kernel driver was read-only, right? I assume this new one will allow write access?
I vaguely remember having to use fuse to write to NTFS partitions in the past.
The existing kernel mode driver has a very limited write mode that is marked with flashing sirens and limited to things like (IIRC) "only modifying existing files without changing their size".
ntfs-3g is a FUSE driver with read/write support, but it's FUSE, and not bundled with the kernel.
This new driver is (based on?) the commercial NTFS driver Paragon sold, and does not suffer the limitations of either, AIUI, and I gather the objections have mostly been around the state of the code rather than functionality.
edit: Having looked, the comment in the Kconfig makes it clear that they believe the limited write support is quite limited, but quite safe.
The FUSE one (ntfs-3g) is fine - we use it for bulk editing Windows VMs when transferring them from VMware to KVM[1]. Unfortunately development has slowed to a standstill in the last few years. Which is a problem. At the moment it supports current Windows guests including relatively recent features like file compression, but we're only a MSFT new feature away from having to take over upstream development or look for an alternative. I doubt that an in-kernel driver will be noticably faster, but there's hope it might be more widely used and so better maintained.
Sure, I didn't mean to suggest there was anything fundamentally flawed about ntfs-3g, I've used it to basically universal success after the first year or two after it was released, just that it is FUSE, which means there's always going to be some overhead involved that wouldn't be present in an in-kernel driver (though they keep trimming away at how much that has to be).
I used NTFS with linux a year ago. It's read-write and suitable if you need to access files on Windows partition.
It was OK but not great: I used it to share media files between Windows and Linux and soon I noticed that if I'm playing mp3s from NTFS partition a lot of CPU used.
Only if fast boot is enabled. Which makes sense, because in that case Windows doesn't shut down entirely, the file system is still marked as mounted. You can break that lock, but you really shouldn't. Better to disable fast boot entirely.
