
Why are there no true cross-platform filesystems? - Daegalus
http://yulian.kuncheff.com/blog/2013/02/21/why-are-there-no-true-cross-platform-filesystems/
======
seldo
I think the real, but unhelpful answer is "supply and demand". Most
filesystems are used by only one operating system at a time.

If two operating systems need to share files, they're usually on different
machines, so it's simpler to copy the files or provide a network interface to
them, rather than share disk-level access.

Getting different OS makers to standardize (and therefore slow innovation) on
something as performance-impacting as the file system isn't worth the benefit.

~~~
saulrh

      provide a network interface to them
    

If you're trying to use partitions off a single disk, implement this using
minimal virtual machines. OSX in single-user mode and only the necessary
kernel modules loaded, windows server core, suitably tiny linux. Not as good
as native fs drivers (in particular, VMs are memory hogs), but cheaper than
hardware and a lot faster than going through a cable.

~~~
blablabla123
This makes you happy until it fails. Windows, Mac OS X and even real-world
Linux write state information onto the FS. (Hibernate etc.) If stuff fails,
you may not be able to boot into your "native Installation" anymore.

Another thing is evolving features, Windows is particularly good at this: more
and more features are added to the FS, if you don't keep mini VM and native
Installation in sync, weird stuff could happen.

I use dual boot since many years, but I have stopped sharing partitions
completely. If I need interoperability I use the Cloud, USB sticks and "normal
VMs". It's fool-proof, safe and low maintenance.

------
chao-
I ran into yet another consequence of this situation today: In a Linux/Windows
setup, I lost two hours of minor Photoshop work (i.e. stuff that was simple
but annoying to recreate).

I have generally found it safe to read from my NTFS drives in Linux (I avoid
writing if at all possible). This time, however, when I went to load the
result of my work back in Linux, it could not read the folder (Thunar reported
some generic IO error). Going back into Windows to investigate, Window's
opinion was _"It smells like Linux has been here! I can't STAND that guy or
anything he touches, so I burned the folder just to be safe. You can thank me
later."_

As best I can tell, it had something to do with a combination of:

1\. Windows Update needing to do things upon shutdown and startup, but then
not booting back into it for a month.

2\. Linux having been in hibernation rather than halted. Might have had some
access bit set on the NTFS drive when it hibernated? I don't actually have the
domain knowledge to claim that is what might happen, so it's a complete hunch.

This was really just a rant to get out the anger at lost work. This article
was quite timely, and while I'm not glad to see people share my pain (because
it's still pain in the end), I hope we one day see improvements.

~~~
e12e
While technically hibernate (to disk at least) should unmount all filesystems
- I'd be very wary of using hibernate (on either os) in a dualboot setup.

Other than that - my experience of late has been that ntfs is safe both to
read and write from linux -- but as I now encrypt all my filesystems, that
doesn't really help (no bitlocker support). If I actually needed to dualboot
-- I suppose a separate disk/partition with truecrypt might do the trick.

My solution for the past several years have been to have all "interesting"
files on a separate linux file server -- and access it with cifs/samba and/or
ssh.

~~~
lloeki
> _While technically hibernate (to disk at least) should unmount all
> filesystems_

Not a chance. Hibernate is just like Suspend to RAM: freeze all processes but
core kernel, blit RAM to disk (usually stored in swap) and send poweroff
command straight to CPU. _Of course_ it doesn't unmount anything, hence there
are FS data structures live in RAM. Booting an alternate system and accessing
a filesystem open on the other side is like mounting twice the same filesystem
and writing to it. It _will_ blow up on you.

~~~
e12e
Ah, I was thinking of the Debian package for hibernate, that has custom
scripts for unmounting selected file systems, from the help text:

    
    
      "If you have network shares or external devices that should be unmounted before suspending, list them here."
      "Unmounts any filesystems of the given types. This is most useful for network filesystems such as smbfs and nfs."
    

And afaik it is perfectly possible to add ntfs mounts to that list - but I
don't think any program with open files on that mount would handle it
gracefully (I belive the script will figure out which processes (if any) have
open files and kill them prior to hibernation).

~~~
Shorel
You can't change Windows code to 'add ntfs mounts to that list'.

Windows 8 will in fact do hibernation when you think you have turned off the
computer.

Bottom line: You can't dual boot safely with Windows 8.

------
cbhl
What the author really wants is network-attached storage.

Find a cheap/old box, put your favourite flavour of Linux on it, add a few TB
hard drives, and run Samba. If performance is an issue, use a mobo with SATA 3
controllers and a few gigE ports, and add additional network daemons for more
"native" networked filesytems (NFS, AFS, etc.) as necessary.

~~~
timdorr
If he doesn't want to spend any money on hardware, perhaps he could do the
same with a Virtualbox VM? That can be hosted on all three of his OSes. I
don't know about performance, though.

~~~
cbhl
No, that's silly, because it presents a chicken-and-egg problem -- where would
you put the Virtualbox VM if you don't have a filesystem that all three OSes
can read?

I was very intentional in suggesting it be a separate physical device.

------
inopinatus
I've been using Paragon's native NTFS for OSX and it's been fast and robust
enough under the modest loads of small office file sharing.

The author of this article seems to have mistakenly dismissed their stuff as
FUSE based, which it doesn't seem to be. It's also cheap.

They also have HFS for Windows, haven't tried it even though it came free with
the NTFS for OSX driver.

Not associated with Paragon in any way, just a customer.

------
codeka
I think the author's use case is actually quite rare these days. I personally
don't dual boot any more, much preferring to run other OSes in a virtual
machine. Then I don't care what filesystems they use internally, all shared
data is shared via the VM's interface anyway.

I think that's the way of the future, not cross-platform filesystems.

~~~
Daegalus
While I try to do this often. My macbook even has Win8 and Ubuntu as VMs on
it. Running Xcode or Visual Studio in a VM can get pretty laggy, even when I
give the guest OS a considerable chunk of my host resources.

I have an i7 920 and 16 gigs of ram, and I have about 50% to the guest, and
Visual Studio and Xcode, along with simulators is pretty messy. Running a VM
within a VM can do that.

Its the only reason I do a triple boot like this. Otherwise, I would
definitely go pure VMs.

Though Parallels on my laptop does a bangup job with Visual Studio.

But then again, the main problem is huge files and lots of writes.

~~~
riobard
VM's IO performance is usually pretty bad. Try an SSD.

------
ComputerGuru
exFAT [0] is a great option. It's FAT32 updated with all the modern
requirements. Surprised there was no mention of it! It's by Microsoft and is
supported on XP and above (out of the box on Vista, update available for XP
and Server 2003). It's surprisingly fast, robust, and all around a great OS to
use for storing shared data.

There's a free implementation available for Linux [1] (actually in the default
apt/yum/whatever repos for many distros) and it ships out-of-the-box on recent
OS X installs [2].

0: <https://en.wikipedia.org/wiki/ExFAT> 1: <https://code.google.com/p/exfat/>
2: [http://www.macrumors.com/2010/11/11/mac-os-x-10-6-5-notes-
ex...](http://www.macrumors.com/2010/11/11/mac-os-x-10-6-5-notes-exfat-
support-airprint-flash-player-vulnerability-fixes/)

EDIT

Yes, it is mentioned in the article. The article sucks. It does not define
what "modern" features are of a filesystem, and the author clearly sets off
with his end goal in mind ("there is no cross-platform FS") randomly removing
FSes from the consideration for vague and hand-wavy reasons. This applies to
several of the entries in his list. I mean the guy thinks UDF will do the
trick, but exFAT is not good enough? Come on!

~~~
Yeroc
I think a lot of people would contend exFAT is NOT a great option by virtue of
the fact that Microsoft holds patents on the filesystem and has (in the past)
enforced those patents. So it's pretty unlikely that it's ever going to have
great support on Linux.

------
DeepDuh
About HFS: You seem to dismiss it a bit quickly. Trial versions of 'MacDrive'
on Windows have always worked very solidly for me - if I were still using
bootcamp I would pay for it.

However the bigger question IMO is: Do we really need native windows nowadays?
If we ban windows into VMs and only use OSX + linux natively, HFS is indeed a
solid option. The only thing hindering us so far from using windows VM-only
was gaming and audio/video rendering software. Audio is very niche and the
rest should soon be solved with increasing support for PCIe passthrough.

~~~
maratd
I prefer things vice versa. I know it's very hipster to hate Windows, but if I
stick OS X and Linux into VirtualBox, I can then use Samba on Linux and OS X
already supports SMB. Now everybody can talk to each other.

~~~
DeepDuh
Well, the thing about OSX is that it is heavily reliant on in house media
codecs and drivers that I find haven't been virtualized very successfully so
far. Furthermore, I find that the (paid) virtual host desktop solutions on OSX
are much better in terms of performance and integration than their windows
counterparts - probably because of higher demand, possibly also because of
architectural reasons.

But maybe you have found a way and you can then prove me wrong: Does iWorks
work in your virtual box VM? Does XCode including iOS simulator work well? Is
the performance acceptable? How about the iLive suit? How about unity
development? How about the clipboard - does it keep its 'PDF snippet'
functionality (no rasterization of fonts, original quality images)?

See, there's tons of OSX exclusive software that's quite demanding in terms of
media and hardware acceleration, simply because of the OS's highly integrated
character. Unless someone can show me a solid OSX-in-a-box I doubt that it's a
good experience.

~~~
maratd
> Does XCode including iOS simulator work well?

Yes.

> Is the performance acceptable?

It is for the above. Not so much for anything demanding.

You're kinda missing my point. If I wanted to work in OS X, I certainly would
never virtualize it. If I occasionally need to check a website or see how it
would look like on an iPhone, then virtualization is a good option.

------
dredmorbius
Let's clarify the author's use case: a multi-boot system for which he's
looking for a universally compatible filesystem.

There are various reasons why this doesn't exist, at least not beyond
compromise solutions such as vfat. A transfer partition and use of archive
formats would largely suffice.

The larger problem though is expecting to be able to manage the same storage
on a multi-boot system from multiple operating systems. It's one of the
reasons that multi-boot is a poor alternative to virtualized systems.

The author could configure his alternative OSes as VMs, and transfer files via
virtual networking, either via transfers (scp, etc.), or via mounted networked
filesystems (CIFS/sambafs, NFS). Compatibility issues are handled by the
networking protocols (or by using archive formats), and simultaneous access to
multiple environments is available.

------
doublextremevil
UDF is pretty cool. It is an open standard supported by Windows and Mac OS X,
but for some reason linux doesn't support versions higher than 2.01

Maybe somebody here has the skill and time to fix this.

<http://en.wikipedia.org/wiki/Universal_Disk_Format>

~~~
thristian
To be fair, the only versions higher than 2.01 listed on that page are 2.50
(which adds the ability to store all metadata on a separate partition... not
that useful) and 2.60 (which adds support for "partial overwrites" on
sequentially-recorded media).

Neither of those seem terribly useful for the "UDF filesystem on a shared
hard-disk partition" use-case we're talking about, so I don't think Linux's
limit of UDF 2.01 is very important in practice.

~~~
regularfry
It's important if it will refuse to mount a later version. I don't know if
that's the case, just guessing here.

------
tga
This is a problem, but triple booters are numerically an edge case.

Rather, what file format do you use on your _portable drives_ when you use all
three operating systems on different machines in different scenarios (say
Windows at work, OS X at home, Linux on servers)?

I'm sure that starting a virtual machine to share EXT4 drives over the host-
only network works (if you have the option of installing VirtualBox
everywhere), but I'd be weary of network file system limitations and issues
and overall bad performance.

------
bonzoesc
The two popular platforms are mostly closed to external contributions, that's
why there aren't any "true cross-platform file systems" that aren't decades
out of date like FAT32.

------
NelsonMinar
My use case is external USB drives. A lot of comments in this discussion have
said "use a network drive!". Which works in some scenarios, but brings a lot
of complication.= over a USB drive.

A $85 USB 2.5" drive carries a terabyte of data, fits in your pocket, needs
only a single cable to use (and no external power), and transfers data faster
than 100MBps ethernet. Or if you really care about speed, $40 gets you a 64GB
USB3 flash drive that's faster than gigabit ethernet. The problem is there's
no filesystem you can write on these drives that every machine can read.

~~~
jrockway
Link to the $4 64GB USB3 drive?

~~~
NelsonMinar
Oops, typo, updated to say $40.

------
shurcooL
I'm more upset there are no filesystems where I can organize files by applying
labels to them, similar to how one can apply labels to email in Gmail.

~~~
sdrinf
Not quite sure if you really want that: open your favorite shell, and count
the number of files in your data directory; then multiply it by, say, 2
seconds each -would you like to spend that time on organization, or rather
making money? (obviously you can apply the same label to multiple files, but
you still need to figure out what belongs to where).

Personally, I gave up on manual organization of my file system last year, and
learned how to love built-in search. This works great, as the rough index is
in your head anyways, so you can just take a single token of the filename,
type it in, and open it up; massively reducing both access time, and
eliminating organization entirely.

~~~
shurcooL
I only want to manually apply labels only to some files/folders.

I have some parts of the same project spread out through 3 folders: not in
Dropbox, in Dropbox, in Public subfolder of Dropbox. It'd be really nice to
have a unified view on those things. As in apply 'Dropbox' or 'Dropbox/Public'
labels to some files.

I know Dropbox has shared links, but I don't want to have to create and manage
those all the time. It's just an example.

Finally, I think a great way to organize files is by time created/modified. If
you exclude the system or program generated files. Quite often I just want to
find a file that I know I saved a few days ago (but no idea in which folder).

~~~
archivator
Symlinks. Put all three folders under a common parent. Then, put symlinks
inside your Dropbox pointing to the relevant parts of the project.

------
pjlegato
I'm surprised that more people haven't mentioned ZFS! It's a stable, cross-
platform filesystem, but that's just the tip of the iceberg.

There are stable implementations for Linux, Mac OS X, Solaris, and BSD (at
least). You can export a filesystem from any one of these, unplug the hard
drive, plug it into another computer, import the filesystem, and go. Doesn't
even have to have the same CPU endianness.

Besides being designed from the ground up to be operating system and CPU
endianness independent, it does a ton of other stuff that no other filesystem
does. It's a total rethink of the entire filesystem / volume manager / RAID
pool concept. All of these things are merged into ZFS, with a user interface
that's both radically simpler and far more powerful than managing all of those
things seperately.

You can easily add capacity to existing datasets (up to 256 quadrillion
zettabytes, it's 128 bit!), do mirroring and striping at the filesystem level,
and add hot standby drives. You can combine drives of different sizes into
storage pools. You can do rolling drive size upgrades in the future to expand
the filesystem size when drives become cheaper, without taking the filesystem
offline. It has built-in pervasive checksumming and self-healing data
corruption detection, there's a strong focus on data integrity despite flaky
hardware, writes are atomic, you can take snapshots and roll back to previous
filesystem states, there's an advanced adaptive memory caching system that's
far beyond simple least-recently-used caches..

Is that not enough? How about built-in encryption, data compression with your
choice of algorithms, deduplication of file data... You can add a small SSD
drive to your huge traditional hard drive based dataset to act as an L2 cache
for fast, durable writes, fully online maintenance and upgrades, builtin SMB
and NFS sharing, you can stream updates...

~~~
darklajid
As someone having a little over 6 TB of ZFS storage in my Linux home server:

How does that fit the 'One file system on Linux, OS X and _Windows_'
requirement? ZFS is useless for this scenario. No need to sell it if it cannot
even start in this competition.

Edit: On top of that, some features that you're selling aren't completely
available cross-platform (i.e. builtin NTS/SMB support on Linux, encryption
is(?) Solaris only iirc).

------
csense
What about network filesystems?

They can be quite fast if you aren't running them over a physical network,
such as between host and VM operating system on a single physical machine.

My preferred solution to your use case would be to run Linux in a virtual
machine, configure the VM platform separately in each host OS to give the
guest direct access to a dedicated raw partition in ext4 or any other Linux-
compatible format, then serve Samba or some network filesystem to the host
from the guest. (Of course, if Linux is your host, you can just mount the
dedicated partition directly from the command line or fstab.)

Back when Windows was my main OS, I liked Colinux [1], a port of user-mode
Linux [2] which allows you to run the Linux kernel as a native Windows binary.

But nowadays my first recommendation would be Virtualbox. You could also try
qemu, it might have better Mac support.

[1] <http://www.colinux.org/>

[2] <http://en.wikipedia.org/wiki/User_mode_linux>

------
Camillo
A friend of mine uses ZFS on OS X. Apparently it works quite well. I imagine
Linux support is good too, but I'm not sure about Windows. At any rate, I'm
surprised the author didn't mention it.

~~~
glabifrons
I thought ZFS on OS X was read-only.

Linux ZFS is still quite young, only supports a very early version of the fs,
and can't be updated to the newest without serious reverse-engineering.

As for ZFS on windows, I definitely would not hold my breath.

I ran into the same frustration as the author and gave up. My notebook triple-
boots Linux, Solaris 11, and OS X (I have no use for windows) and would be
beyond thrilled to have ZFS shared among them.

One other problem with the idea though... you'd have to export the shared fs
each time you shut down and import it into the next OS you booted, which would
be kind of a pain (of course you could force the imports, but that's ugly and
it only saves one step).

~~~
iso-8859-1
> Linux ZFS is still quite young, only supports a very early version of the fs

Which implementation are you talking about? See <https://launchpad.net/~zfs-
native/+archive/stable> . Zpool version 28 is old now? The only implementation
of 31-33 is in Solaris! See <http://en.wikipedia.org/wiki/ZFS#Comparisons>

~~~
glabifrons
<http://zfsonlinux.org/> They no longer list the versions on the front page.
Last I saw the page was only a few months ago (max) and you couldn't even
create a filesystem yet (you could create a zpool and zvols). I remember
reading on that page the zpool and zfs versions implemented at that time, and
they were way back.

And yes, I am using Solaris for comparison... I mentioned that I'm triple-
booting with Solaris in the post you replied to.

I'm glad so see the development has caught up, but it is still quite young.

------
jacques_chester
Filesystems are these days entangled with virtual memory.

That joins to the kernel's world at the hip.

------
richardwhiuk
The main reason I suspect is covered below, but the only time this is a useful
thing is dual booting - the rest of the time network attached storage is the
'right' answer.

Personally, I'm glad to have left the time of dual booting operating systems
behind me - it's a complete PITA. A Linux box with a bunch of disks under
samba/ext4/lvm/mdadm is much more practical IMHO.

------
wsh
I respectfully disagree with the author's conclusion.

There are at least two filesystems, intended for situations where several
computers share block-level access to the same disks over Fibre Channel or
iSCSI, that are truly cross-platform, allowing volumes to be used,
simultaneously if desired, from computers with different operating systems.

Quantum's StorNext File System is available for Linux, for Microsoft Windows,
and for several UNIX systems, including OS X, where it's bundled with the OS
and known as Xsan. SGI's CXFS ("clustered XFS") is similarly compatible.

Both of these filesystems are modern, in the author's sense that the
filesystem does not impose a practical limit on file or volume sizes.

------
nitrogen
I triple-boot my desktop and laptop (Linux, W7, XP), and have a pair of large
NTFS partitions to share data (including my Steam library using symlink
trickery) between all three operating systems. The downside is that the open
source version of ntfs-3g is slow (still faster than using a VM), and its
allocation strategy seems to cause extensive fragmentation.

To those wondering why VMs are insufficient, if you have work (or gaming) to
be done that requires maximum performance in multiple operating systems, a VM
is not fast enough. I develop software for my startup in Linux, write client
software in XP, and do video work and game in all three.

~~~
Daegalus
This is kinda the situation I am in. But I stated that in the first few
paragraphs of my post.

------
gizmo686
As others have mentioned, the cost of standardizing file-systems across OSes
does not seem worth it. But, is there any way to define a standard interface
so that a filesystem driver written for 1 OS can be easily ported to another.
Similar to how NVidia ships the same binary blob regardless of the OS. We even
have ndisswrapper, which can run many Windows wireless drivers on Linux
without modifying the binary.

We will obviously run into issues where different OSes have different concepts
of what the filesystem needs to do (such as differing security models), but at
the very least we should be able to get the common interfaces standardized.

------
tytso
The e2tools command line utilities should work just fine on OSX. I'm surprised
no one has bothered to make OSXFUSE work with an ext3/4 driver, but it should
be pretty simple. The libext2fs library which ships with e2fsprogs is pretty
easy to make work with userspace applications. The the e2tools package uses
libext2fs, and there have been FUSE drivers which use libext2fs. I can ask
around see if there are some FUSE developers who could update an ext2/3/4
driver that works with OSX and Windows.

------
wolf0403
Dare to use / test all different drivers on different systems from different
vendors to read / write single piece of disk with some valuable data is really
brave. Use NAS.

~~~
magnawave
Indeed vary brave. Filesystems ARE NOT and never have been an API to exchange
data between multiple OSes - they are a means to persist data efficiently for
a single OS. Now if you don't mind breaking a bunch of the FS rules, sure
compatibility hacks can always be written - but are you really sure each
implementation is FULLY honoring the rules. I can guarantee you, for anything
but the simplest filesystems, the answer will be "no". Hell, even FAT has
implementation stragglers out there in the wild and its brain dead simple.

Use an API for cross platform sharing layered on top of a filesystem: Either a
Virtual Machine sharing protocol (aka 9p virtio) or a real network protocol
AFP/SMB/NFS. Anything else is taking your data in your own hands and is
antithetical to the design of modern OSes.

------
kresh
I use a small console-only install of VirtualBox and use Samba to communicate
back with Windows. I think I only have it using 1 cpu and 256 MB memory. I can
easily run Autocad or Solidworks in the foreground.

I find that I only lose about 5% efficiency compared to booting up Linux and
copying without overhead.

------
RexRollman
One thing I love about Unix is the option to mount a filesystem read-only. I
would love to have the option for my Windows box, when noticing a new
filesystem, to present me a dialog box asking me how I want to mount the
filesystem.

------
regularfry
One solution that hasn't come up but might be worth pursuing: ext4, with a
umlwin32 ftp server to export it to Windows.

------
c3d
What about the native APIs? NTFS and HFS have their roots in non-Posix worlds.
The basic operations are somewhat similar in all platforms (open file, read
directory, ...) But the "modern" stuff is not standard, otherwise it would not
be modern.

This is probably the reason only FAT fits the bill: it's a lowest common
denominator that has so few features that it can appear "native" on all
platforms.

------
wbkang
Look how most cross platform GUI toolkits turned out :/

------
dschiptsov
Because files are "cross-platform".)

------
itsnotvalid
Just a side-note, if he's not using Apple hardware to run OSX, there could be
some licensing issues here.

------
drivebyacct2
>BTRFS

There's a pipe dream. It'd be great but I can't possibly imagine it happening.
I have to think that any spare FS effort is being spent on ZFS (OS X) or ReFS
(Win8/Server2012). I'm still rocking out with a 12TB RAID10 array and another
6TB RAID10 array that I gave to my parents. Haven't had any issues yet.

