
exFAT in the Linux kernel - UkiahSmith
https://cloudblogs.microsoft.com/opensource/2019/08/28/exfat-linux-kernel/
======
pedrocr
I'll believe Microsoft loves Linux when I can install Microsoft Office in my
desktop Linux machine. Everything else is marketing.

Edit: to expand, the corporate world runs on Excel/PowerPoint/Outlook.
Microsoft milks them on Windows/Office licensing and is very aggressive
against organizations that try to do Linux deployments. They're a convicted
monopolist that got away easily when at some point the option on the table was
to split OS and Office into two different companies. If they want to continue
to use Office to keep Windows dominant that's their strategy. WSL and their
other "love Linux" efforts are all about making sure developers stay on
Windows. If you want to do all that fine but don't patronize us by then
claiming you love Linux.

~~~
tw04
Given that Microsoft is a for-profit company, what's the financial
justification for writing office for Linux on desktop? The market share isn't
even a rounding error and you're talking about them investing tens of millions
minimum into the development.

The fact they have office on Android, iOS, and osx proves it's about market
share and common sense, not some fabricated hatred or fear of Linux on the
desktop.

~~~
pizza234
> what's the financial justification for writing office for Linux on desktop?

Well, the same argument could be applied to their open source contributions.
Considering that [canonical corporate view is that] patented ideas are
expensive to produce, and that exFAT is patented, what's the financial
justication for open sourcing it (the specs or else)?

The argument is ultimately against "Windows love Linux". The thesis of the
parent is just that they're doing unsubstantial moves (or worse, manipulative
ones), and that if they "truly loved Linux", they would do something
substantial, which, for the corporate world, is to support Office on Linux.

~~~
close04
This argument can very easily be used against people who claim they love Linux
and FOSS but they never really seem to be willing to part with the same amount
of money other people are willing to give MS. Donating $300 for every copy of
Libre Office you use would go a long way to making MS Office on Linux
completely unneeded. Same goes for donating similar amounts for every Linux
you install. I guess everyone's love has limits and it's usually money.

As for MS's contribution to open source you have to consider the difference in
magnitude between the revenue that Office generates and the revenue that their
other open source contributions do (would?). Then consider also the
development effort involved.

It looks like OP just wants exactly what MS doesn't offer completely
disregarding the alternatives, and I'm pretty sure they would not be willing
to pay the markup for the extra development involved.

------
jaclaz
The IMHO extremely relevant point is in the very end of the actual
specifications:

[https://docs.microsoft.com/it-
it/windows/win32/fileio/exfat-...](https://docs.microsoft.com/it-
it/windows/win32/fileio/exfat-specification)

>26-Aug-2019 Seventh release of the Basic Specification, which includes the
following changes:

Updated legal terms pertaining to the specification, including:

Removal of Microsoft Confidential notice

Removal of Microsoft Corporation Technical Documentation License Agreement
section

Updated copyright notice to 2019

Till now the documentation wasn't AFAIK publicly available or it was anyway
"restricted".

~~~
userbinator
Skimming through the doc, it looks like a modification of FAT32 with some
(noticeably more complex) extensions, and not anything resembling what you
might envision a "FAT64" to be; a FAT is still a linked list of 32-bit
entries, there can still be two of them, and directory entries are still 32
bytes each (although defined in a rather complex way.) There's a separate
allocation bitmap, which seems redundant and a waste of space (and code) given
that the FAT itself already contains the information in previous versions.

I've written a FAT12/16/32 driver for an embedded system before; and oddly
enough, the documentation I had at the time was far easier to understand than
this one. I didn't expect exFAT to be this complex, or perhaps the doc is just
excessively verbose --- for example, the whole first paragraph of the 4th
section is saying nothing more than "there may be 1 or 2 FATs, one after the
other, and NumberOfFats is the field that says how many there are" except it
takes 5 sentences to do so.

That said, this spec is written in a style that's easier to understand than a
lot of the other "open specifications" docs that MS has released, but I
suspect that's because this one was once an actual NDA'd implementer's
document rather than the others which I've heard may have been released purely
for legal reasons related to the antitrust than anything else.

Now if only they'll do this for NTFS...

~~~
MuffinFlavored
With how slow modern day NTFS is for things like node_modules (lots of nested
directories), I can't help but hope some better file system will become
widespread on the next major Windows release.

~~~
kevingadd
Most of the performance issues are not NTFS so much as the prevalence of file-
system filter drivers. This is why WSL had so many file-system performance
issues despite the fact that it was using a custom FS. IIRC this also
motivated the move to WSL2 from what I read, since a virtual machine can use a
small set of files to store a whole FS.

EDIT: See here -
[https://github.com/Microsoft/WSL/issues/873#issuecomment-425...](https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829)

------
jdsully
The big deal here is the patent grant. FAT related patents were held over
linux like a gun in the Ballmer years. Its a reason Linux was stuck with 8.3
length filenames for so long.

~~~
taffer
Aren't the FAT patents about to expire anyway?

~~~
WorldMaker
Some of the earliest ones, surely, but presumably Microsoft also continued to
patent later innovations: all of the "extended" in ExFAT.

~~~
viraptor
This seems to be one of the patents involved:
[https://worldwide.espacenet.com/publicationDetails/biblio?CC...](https://worldwide.espacenet.com/publicationDetails/biblio?CC=US&NR=2008168029&KC=&FT=E&locale=en_EP#)

I'm not sure what's the expiry on 2008 submission.

~~~
loeg
20 years from invention? Which looks like ~2004 skimming the 2008 filing.

------
hnarn
I don't mean for this question to sound inflammatory but I can't help but
think: why does this matter? You can already use exFAT on Linux via FUSE, and
as pointed out by VentureBeat:

> To be clear, Microsoft isn’t open-sourcing exFAT — but it is making sure
> anyone building with Linux can use it. 'It’s important to us that the Linux
> community can make use of exFAT included in the Linux kernel with
> confidence.'

exFAT is still not open source, and still can't be distributed with the
kernel. So what changed? It will now be easier to integrate it into a _custom_
kernel instead of using FUSE? Why is this better than the current situation,
where you'd install exfat-utils for example? Are there benefits of using a
kernel module that I'm missing which makes this important news?

~~~
gregkh
It will be in the "real" kernel soon, here is a patch I submitted adding it:
[https://lore.kernel.org/lkml/20190828160817.6250-1-gregkh@li...](https://lore.kernel.org/lkml/20190828160817.6250-1-gregkh@linuxfoundation.org/)

And yes, this is _much_ better than using a FUSE interface to the filesystem.

~~~
ghostpepper
For the 99% of people reading who don't know the kernel as well as you
(seriously) can you explain why this is the case?

Does a userspace filesystem driver lose a lot of performance to context
switching, or is there something unique to filesystems that slows them down in
user space?

~~~
gregkh
> Does a userspace filesystem loose a lot of performance...

Yes.

Try the two versions out, and see for yourself if you are curious.

~~~
dekhn
I use a FUSE filesystem on linux- not ExFAT, but as an S3 interface. I often
see 1+Gigabyte/sec throughput over the wire from simple multithreaded IO
operations. FUSE can be very fast.

~~~
notriddle
For one thing, that's throughput, not latency.

For another, that's probably a desktop, not a smartphone.

~~~
dekhn
That's correct, my metric for performance is throughput I don't really think a
desktop vs a smartphone matters much at this comparison, it's just my
observation that when people say a fuse filesystem is slow it's just not well-
engineered, not that the fuse API is inherently slow

~~~
grosswait
Run a benchmark that measures more than throughput and you'll see the
difference. IOPs are usually poor.

~~~
codycraven
Latency is a nightmare if you perform certain types of tasks on FUSE.

------
rhn_mk1
Does this affect anything outside the Linux kernel? Will *BSDs be able to use
the specs without patent fears? What about microcontrollers, and the Arduino
community?

~~~
Conan_Kudo
No. The OIN promise only applies to Linux-based systems.

~~~
derefr
What defines a "Linux-based system"? Does Android count? Would VMWare's
VSphere ESXi hypervisor count? Would a unikernel running on a pared-down Linux
rump-kernel (if such a thing were made) count?

~~~
progval
According to
[http://en.swpat.org/wiki/GPLv2_and_patents](http://en.swpat.org/wiki/GPLv2_and_patents)
, GPLv2 includes an implicit patent grant

So my guess would be that the definition of "derivative work" would apply.

~~~
monocasa
That's super duper grey area, which is why GPLv3 went out of it's way to have
an explicit patent grant.

------
tambourine_man
OMG, I can't believe we finally have a cross platform read/write disk format.
At last. No more Fuse. I just need to know when it will be available for my
Raspberry Pi.

~~~
pwg
We already have one, it is just that few know we do:

Universal Disk Format
([https://en.wikipedia.org/wiki/Universal_Disk_Format](https://en.wikipedia.org/wiki/Universal_Disk_Format)).

I can confirm it works on Linux, and works on Windows (at least W10). The
wikipedia article indicates it works on newer MacOS'es, but I can't confirm
that fact.

~~~
tambourine_man
I thought it was for optical media only.

Yet, it seems there's no way to format to it out of box on macOS and support
looks dodgy. I'm not willing to bet my data on an unreliable file system
implementation.

~~~
pwg
> I thought it was for optical media only.

Nope, although it is most commonly used on optical media it is not restricted
to only optical media.

From the wikipedia article:

Specifications

The UDF standard defines three file system variations, called "builds". These
are:

Plain (Random Read/Write Access). This is the original format supported in all
UDF revisions ... Plain build

Introduced in the first version of the standard, this format can be used on
any type of disk that allows random read/write access, such as hard disks,
DVD+RW and DVD-RAM media. Metadata (up to v2.50) and file data is addressed
more or less directly. In writing to such a disk in this format, any physical
block on the disk may be chosen for allocation of new or updated files.

Since this is the basic format, practically any operating system or file
system driver claiming support for UDF should be able to read this format.

~~~
ldng
Have you used it IRL ? How does it fare performance wise ?

~~~
pwg
I've been using it for some number of years now on USB sticks that are larger
than 2G (to avoid the 2G file size limit of Fat32) for file transfer to/from
the work machine.

I have had zero issues with it from Linux or Win (I think I started using it
when the "work pc" was Win7, but am not 100% sure of that). It has just
worked, with no problems, for my use cases.

Performance wise, well, the performance is limited by the USB sticks (they are
USB2 sticks) so from what I can tell, UDF is not the limiter, the USB2
interface or the flash cells in the stick are the performance limiter.

------
mixmastamyk
Thanks MS. Would also like to see ext4 support in Windows and better NTFS
support under Linux. Then the turnaround (on the technical side at least) will
be complete.

~~~
wpdev_63
Bring btrfs to windows pls!

------
newscracker
When will we have a modern file system that “just works” for read and write on
Windows, Linux and Mac without the need to buy some proprietary product or
depend on some open source option that may not be maintained or 100% reliable?

Seems like going over the network is the easiest option for people who use
multiple OSes, though it comes with incompatibilities in permissions,
ownership and also comes with a speed hit.

I doubt if Apple would provide implementations of APFS for non-Apple operating
systems. So that probably leaves Microsoft to push NTFS (since ReFS isn’t a
common option anymore as per Wikipedia) if it desires. But Microsoft seems to
be on a cost cutting mode (cf the adoption of Chromium in Edge).

Are there viable and good options suitable for these three operating systems?

~~~
angrygoat
exFAT works on MacOS >= 10.6.5, Windows and with this announcement hopefully
shortly on Linux. I'd say it counts as a modern file system too - only
released in 2006.

~~~
shmerl
I'd count "modern", as something like btrfs, zfs or bcahefs.

------
buildbuildbuild
Too little too late, with no acknowledgement of Ballmer-era exclusivity,
bullying. It’s progress, but progress without awareness of baggage in the
room.

------
jasonhansel
"We love Linux so much that we would never let it be polluted by our horrible
desktop software!"

------
postit
MS wants a native kernel fs compatible with Windows so they can improve WSL.
Today's virtualized kernel pays an unecessary price to access ext4 FS

------
xvilka
Meanwhile, ntfs-3g[1] wasn't updated for years and looks like totally dead.
Last version is of March 28, 2017.

[1] [https://www.tuxera.com/community/open-source-
ntfs-3g/](https://www.tuxera.com/community/open-source-ntfs-3g/)

------
bubblethink
Any news on NTFS ?

~~~
benbristow
NTFS-3g is pretty good and support on Linux is definitely a lot better than it
used to be. Still, official (or officially supported/documented) support would
be really nice.

Honestly I hope even Windows abandons NTFS soon. It's the worst part of using
Windows. e.g. Try and develop any Node project on Windows and it's painful -
takes literally minutes to delete the node_modules folder whereas on Linux/Mac
it takes seconds. It only recently got long pathname support (260+ chars
long)...

~~~
GordonS
On the other side, file ACLs in NTFS are way more powerful than the Linux
owner-group-world model.

~~~
grumpydba
You can use posix acls whenever you want though.

~~~
amaccuish
Posix acls aren't particularly nice though. RichACLs is what I'd love but
they're not merged :(

------
The_rationalist
Almost every months Microsoft make a more or less major gift to the open
source community, this is beautiful. They are the new "benevolent" Google and
it's no surprise they're now the number one open source contributor of the
world.

~~~
TwoNineA
They already extorted Android OEMs using exFat patents for billions, might as
well try to look nice.

~~~
swiley
As much as I hate the patent bullying Android should have never used FAT for
_ANYTHING_. The stupidity around that continues to make everyone's lives hard
to this day.

~~~
lern_too_spel
If you want an SD card that is also readable by the digital camera ecosystem,
what choice do you have?

~~~
jdnenej
How often are people taking a microSD card from their camera and putting it in
their phone considering on a lot of devices this means pulling the battery
out.

Also they could support fat while not having the main system storage using
fat.

~~~
vetinari
Main system storage didn't use FAT; /sdcard did.

------
dang
Related:
[https://news.ycombinator.com/item?id=20821161](https://news.ycombinator.com/item?id=20821161)

------
acd
Thanks Microsoft and the developers there for sharing ExFat!

It is great to see the changed attitude and embracing of open source software!
This will ease the use of large external disks, sharing files between Windows
and Linux systems. It will help with larger USB memory sticks which no longer
needs to be reformatted. It will also make it much easier to deal with SD
cards from cameras and mobile phones.

------
shmerl
Finally. Next steps — MS can release active sync without patents, and also
start supporting Vulkan and drop DirectX 12 diversion.

------
ausjke
wan't Samsung providing a GPL-version of exFAT for Linux already? what's the
difference assuming this news means we will have a native exFAT driver in
kernel "soon".

Is exFAT only good for file storage considering it has no journal ling?
putting rootfs on it may have a dysfunction system relatively easily without
fsck I assume.

~~~
finchisko
I have extensive (bad) experience with exFAT implemntation on other platforms
than Windows. My external disk formatted as exFAT gets constantly corrupted
(need fsck), when used on macOS and Linux. On Linux I use this Samsung's
exfat-nofuse kernel driver on raspberry pi. ExFAT fs is mounted read-only and
still get corrupted.

So I don't share the excitement with most folks here. Until driver gets
better, exFAT is not rock solid option for sharing data between OSes.

------
dkh
Big deal! This is right up there with Adobe Premiere on Windows finally
getting a ProRes encoder at the top of my list of "Things That Would've Helped
My FilmTech Career Around 6 years ago".

1) We finally have a file system that works across operating systems, which is
a big deal if you ever do anything in a multi-OS environment.

2) exFAT is heavily used in the film production world, and the convenience of
knowing it'll mount properly on any OS so you can duplicate it a few times is
much more important than whether or not it's a "safer" file system in general.
It doesn't need to survive forever -- the card just has to make it from the
camera to the computer on the other side of the set, or survive a trip from
one office to another on a shuttle drive that by this point is not the only
place that data exists.

Regardless, it's still more mature in general than I remember it when it
started to appear in high-end cameras around 2012/2013\. When your Blackmagic
camera shoots exFAT but isn't capable of deleting files off it without a
computer...

3) Should something still go wrong (like, oh, the time I was almost
responsible for losing $60k of footage my second day on a job due to
unexpected use of exFAT + a truly unfortunate and odd-defying day of bad luck)
the fact that Microsoft is implementing it in the kernel should still be a
huge help -- it will be a proper implementation (not a reverse-engineered
sometimes-working mess), will perform much better than FUSE, and just
generally improve reliability. Trust me, when your card with irreplaceable
footage won't mount, you'd rather not have to fight that battle on two fronts,
with one of them being your Linux implementation.

~~~
userbinator
_fact that Microsoft is implementing it in the kernel should still be a huge
help_

If you're trying to recover video (which is one of the easiest types of data
to recover, due to it being large, sequentially written, and bitstreams
containing easily recognisable sync markers), and especially from a storage
device that's being exclusively used for video, the actual filesystem doesn't
really matter because practically all of the sane ones will store the file
data in a contiguous range of blocks. Depending on the exact codec, you may
even be able to dd the raw device into a decoder and it'll simply find the
first valid sync marker and start decoding from there.

~~~
dkh
Well of course, some file carving is what has gotten me out of a few messes in
the past, including the one mentioned. But in a situation where a volume won't
mount/read, it's nice to eliminate a poor implementation or FUSE issue as a
potential variable in determining if there's actually a problem. (Or worse,
introducing a new problem... I've seen that, too, especially in earlier days
of exfat-fuse/exfat-utils.)

------
Canadauni
Is it worth formatting my external flash media as exFAT at this point if I
find myself dual booting often? NTFS on Ubuntu is pretty well covered so that
has been what I have been using. But if exFAT support is improving I'm
wondering if it is worth switching over.

------
goatinaboat
Great news! But I would have delayed announcing it until I had the patches
ready to be merged.

~~~
mey
The real news here is the specification being opened.

------
wcchandler
I haven't spun up anything in Azure for quite a while. Does anybody know if
their default images use anything exFAT formatted? Doesn't UEFI/EFI binaries
usually sit on exFAT?

~~~
whereistimbo
By standard VFAT is required, but on most Windows PC, the UEFI partition is in
NTFS.

~~~
vetinari
UEFI partition is FAT32 (or EFI-FAT; EFI has their own specification of FAT,
frozen in time, with it's own identifier), if not for other reason, that most
UEFI implementations would not be able to boot from NTFS.

Windows installer creates several partitions: System (ESP), MSR at the
beginning of the disk and Recovery at the end of the disk.

~~~
whereistimbo
Most UEFI implementations are loaded on Intel Windows devices, which is most
likely to use TianoCore, reference implementation of UEFI which includes UEFI
NTFS driver by default. I have seen that some low ends from Lenovo and Acer
have its UEFI partition (ESP) formatted as NTFS.

~~~
vetinari
Are you sure? I had a look into edk2 and cannot find NTFS driver there. I can
see the FAT one.

The NTFS support depends on device. From devices that I have around, Intel
NUCs do support it, Asrock X399 board does not.

I have yet to see the Windows installer to create NTFS-based ESP partition. If
manufacturer does that for specific device (where it is guaranteed, that it
will support NTFS), that's different thing.

------
panpanna
> To this end, we will be making Microsoft’s technical specification for exFAT
> publicly available to facilitate development of conformant, interoperable
> implementations.

How about their FAT patents??

------
whalesalad
Sweet. I had to purchase something for my Synology NAS fairly recently to
support an ExFAT device. It’s great to see this contribution to open source.

------
mikorym
I remember having had problems with exFAT on linux. Did it not work at all, or
did you have to use user repos to get exFAT working?

~~~
paule89
You had to add it as a fuse module. I don't know if a user compiled kernel
module worked. But still even if it was used as a fuse module i myself had
often times problems with it. So i hope this goes away soon.

------
cosarara
In my experience, exFAT gets corrupted constantly, and then on linux you don't
have a proper fsck for it.

------
decoyworker
Will this allow WSL2 to access your entire Windows filesystem without the
performance hit?

If so this is really cool.

------
dis-sys
I will believe Microsoft loves Linux when they decide to drop the development
of their buggy NT kernel and starts to build windows on top of the mainline
Linux kernel.

Everything else is just cheap marketing.

~~~
rswail
"buggy NT kernel" is a stretch. There's nothing particularly wrong about the
NT kernel in its design or implementation, any more than other kernels.

MS don't have to drop Windows to support Linux. They can walk and chew gum at
the same time.

------
vkdelta
Does this mean it is royalty free now?

------
mmorearty
embrace, extend, exFAT

------
mda
So after abusing this bullshit for more than a decade to bully companies. How
gracious and generous of them. Oh btw, Satya is the CEO for 5 years.

~~~
jaboutboul
Stop shitting on them. We won the war. They’re mending their ways. It takes a
long time to right such a large ship with levels upon levels of bureaucratic
bullshit that’s been caked on for decades.

It’s a WIP and they just granted a patent license to the OIN for fuck’s sake.
Even 10 years ago this would have sounded like science fiction.

~~~
traverseda
The WSL looks a _lot_ like the embrace step of an embrace-extend-extinguish
doctrine. I'll believe them more when I can write user-facing apps against
their posix subsystem instead of requiring people to jump through many many
hoops to actually install it.

Right now it just looks like something they were forced into doing because
they were losing on the devtools market.

~~~
dhyft
>instead of requiring people to jump through many many hoops to actually
install it

Yeah the hoops of going to the Store, searching for "ubuntu" and clicking
Install

~~~
traverseda
Are they not still requiring this?

    
    
        Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
    

I admit I haven't used WSL in a while.

~~~
jdsfighter
You still need to enable the subsystem, but I don't see why most users need it
on by default. It seems like an easy enough feature to prompt the user to
enable (or enable for them automatically) if your application calls for it.

------
morekozhambu
Hmm.. Oops! now exFAT will bring the [A-Za-z] case in-sensitivities into the
GNU/Linux system.

~~~
craftyguy
The ext4 folks are working on adding case-insensitive support...
[https://lwn.net/Articles/784041/](https://lwn.net/Articles/784041/)

