
Apple File System Reference [pdf] - abkumar
https://developer.apple.com/support/apple-file-system/Apple-File-System-Reference.pdf
======
israrkhan
In past, I implemented HFS+ on an embedded rtos platform,. Technical Note
TN1150 [1] proved to be extremely useful asset. From a cursory look, I feel
TN1150 was much more detailed, and perhaps can be treated as a pre-req to this
document. At least it should have been mentioned in this document.

[1]
[https://developer.apple.com/library/archive/technotes/tn/tn1...](https://developer.apple.com/library/archive/technotes/tn/tn1150.html)

~~~
tambourine_man
You got me curious, can you explain why you needed to implement HFS+? Was it
read only?

~~~
israrkhan
I was working on a video recorder product that could record videos directly to
iPods, iPhones and other media players. iPods were formatted to HFS+, if you
connected them to a Mac out-of-box, and were formatted to FAT32, if you
connected them to a PC. The platform i was working on was an RTOS, and we had
to develop both FAT32, and HFS+ from scratch. It was not read-only, it
supported both read/write.

~~~
tambourine_man
Nice, that sounds fun, thanks.

------
chungy
This seems like it's probably enough to re-implement APFS on non-Mac
platforms; in fact, the about page (page 6) says as much.

Kudos to Apple for providing the information. It's a hell of a lot better than
reverse engineering the thing (see how many years it took to get NTFS down...)

~~~
Rondom
There is already one third-party-implementation in the works. They sure find
this helpful in their efforts.

[https://github.com/sgan81/apfs-fuse](https://github.com/sgan81/apfs-fuse)

~~~
atonse
This is user space – hopefully a kernel level one will come out as a result of
this.

Or does it matter, performance-wise? People that know more can chime in.

~~~
SSLy
Yes, FUSE is slower than in-ring0 implementation, even by the sheer overhead
of the syscalls.

~~~
ianai
Iirc, it’s relatively straight forward to migrate to a module though?

------
ken
At last! Apple's old APFS docs always had this mysterious note about Fast
Directory Sizing:

"You cannot enable Fast Directory Sizing on directories containing files or
other directories directly; you must instead first create a new directory,
enable fast directory sizing on it, and then move the contents of the existing
directory to the new directory."

but there was never any documentation on how to do this, and no Apple engineer
would say. The most common internet theory seemed to be that this feature was
purely automatic, and all mentions (like this) in the docs were just
incredibly misleading.

Now it seems we have an answer, in this flag: "INODE_MAINTAIN_DIR_STATS: The
inode tracks the size of all of its children."

------
mrpippy
The 'Fusion' section at the end is interesting. macOS 10.13 didn't convert
Fusion drives to APFS, but 10.14 will. Presumably this specific support for
Fusion drives is new in 10.14.

HFS+ had no knowledge about Fusion drives, the caching was handled entirely at
block-level by the lower CoreStorage layer (although later versions did add
some flags so CoreStorage could pin metadata/swap blocks to the SSD).

Now what I'm really interested to see is if they open-source the filesystem
driver along with the macOS 10.14 code drop. HFS+ (and its utilities) has
always been open-source, last year APFS was not.

~~~
arthurfm
The only bad thing about Apple supporting APFS on Fusion Drives is that it
gives them an incentive to continue selling future iMacs and Mac minis with
Fusion Drives.

Having had to replace failed HDDs in Fusion Drive iMacs at work, it's
certainly no fun. For all new Mac purchases I ensure they are SSD only now.

~~~
mrpippy
On that note, I am surprised that they added Fusion-awareness to APFS, rather
than just putting APFS on top of CoreStorage.

It certainly is better to have the filesystem aware of the Fusion situation,
but...measurably, significantly better? Would the experience have been
significantly worse without it? 10.13 betas allowed APFS use on Fusion drives,
presumably without any Fusion-awareness in the FS.

I'm surprised, but happy to see they did it.

------
Someone
There are several cases where the file system has monotonically increasing
integers that, when overflowing, are unrecoverable errors.

Those counters always are 64 bits, and won’t overflow in normal use (for
example, the text says: _”if you created 1,000,000 transactions per second, it
would take more than 5,000 centuries to exhaust the available transaction
identifiers.”_ ), but I can see people making ‘interesting’ disk images, for
example ones where writing to a specific directory is impossible or, depending
on how the implementation handles it, even panics the OS.

------
petecooper
One of my long-time favourite macOS applications -- iDefrag -- had support
withdrawn shortly after APFS appeared. Reasons cited were lack of an APFS spec
and increased System Integrity Protection.

I fear this is too little, too late to have iDefrag make a comeback. I
understand defragmenting an SSD typically does more harm than good [edit: and
I only defrag spinning drives), but nothing touched it for effectiveness on
spinning drives.

[https://coriolis-systems.com/iDefrag/](https://coriolis-systems.com/iDefrag/)

[https://coriolis-systems.com/blog/2017/9/what-works-
macos-10...](https://coriolis-systems.com/blog/2017/9/what-works-
macos-1013-high-sierra-what-problems-ex)

~~~
r00fus
I haven't defragged a disk in almost a decade. When is defragging an SSD ever
a good thing?

I only have spinning rust in my (older) NAS or as backup drives.

~~~
tinus_hn
Probably often but there is no way to tell really because the controller has a
mind of its own and they are all different.

~~~
forapurpose
> Probably often

Defragging an SSD is often a good thing? Why? It would seem to greatly
increase wear for no benefit.

~~~
akvadrako
If your fragments are really small like 16kb then you could see a significant
performance improvement, due to better predictive loading and packet overhead.
This is clear from SSD benchmarks.

But I don’t see how this would ever realistically happen.

~~~
forapurpose
What about the massive effect on wear?

~~~
tinus_hn
You don’t know if there is wear. The controller is a separate computer. It may
be clever and just remap blocks if you copy or move them around. It could be
that defragmentation allows the system to work in a way that is better for the
controller or allows that controller to place the blocks in a better way.

------
sneak
I’m still sad that this filesystem does not contain file data checksums. It
looks like we will be stuck with it for some years to come.

~~~
nicky0
Care to enlighten ignorant me why we would want that?

~~~
giobox
Simple, to help prevent “bit rot”. The problem is exacerbated further in that
many of us treat cloud sync services as backup, which they arguably aren’t -
they can inconveniently just spread the decay.

I’d also hoped that a next generation file system from Apple would have had
more to say on this topic, but it seems like features that promote their iOS
device agenda took front seat over less “sexy” features like data integrity.

In the days before iOS devices dominated OS level decision making at Apple
there was an assumption that Apple might adopt ZFS as their next generation
file system, which is apparently much better in this regard. There’s various
evidence of a cancelled MacOS ZFS project scattered throughout past MacOS
releases.

>
> [https://en.wikipedia.org/wiki/Data_degradation](https://en.wikipedia.org/wiki/Data_degradation)

> [https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-
> ap...](https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-apple-file-
> system-that-almost-was-until-it-wasnt/)

~~~
hollerith
Might it not be the case that combating bit rot is best done at higher layers
of the stack similar to how it is best done at levels higher than the IP layer
in a networking stack?

For example, data painstakingly entered by the user a character at a time with
a keyboard might deserve more redundancy than for example a movie downloaded
by iTunes.

~~~
giobox
I’m no expert, but my understanding is that bits can “flip” and introduce
errors due to things as unpredictable as background radiation etc, even on
files the system has had no interaction with, which is why it’s kind of
desirable to implement this kind of integrity check at the file system level.
A higher level check may be completely unaware of this kind of passive
background error.

Also, if I pull the drive and move it to another machine, again it’s kind of
nice if the data integrity features are tied to the drive format rather than
higher level software. I don’t think it’s too unreasonable to expect the file
system to make sensible guarantees that the sequence of bytes I record today
will remain the same until I next interact with them.

I’m not sure how appropriate comparisons with IP error correction is either;
it’s a markedly different class of problem really (you are not dealing with
long term storage issues at all).

------
cmurf
The EFI jumpstart is particularly clever. A straightforward recipe for
locating and verifying the file system driver, and then once executed the UEFI
pre-boot environment can fully navigate an APFS volume.

~~~
aasasd
Uhhh, personally I'd prefer that UEFI stay away from the OS particulars. But
anyway, afaik Windows' boot code had about the same feature―at least it
certainly did in regard to the chipset drivers, the result being IIRC that the
OS wouldn't boot if you moved the partitions a bit.

~~~
cmurf
The pre-boot environment needs to find the kernel and initramfs somehow. My
guess for how Apple is booting from APFS, now that it's all APFS, without a
separate recovery partition? They've got this minimalist EFI jumpstart code in
the firmware, it loads the EFI file system driver for APFS, and now it can
locate the bootloader, kernel, and kext cache.

For a long time Apple has had an HFS+ driver baked into the firmware. The way
APFS is implemented with EFI jumpstart, they've got much less filesystem code
in firmware.

------
saagarjha
It's nice to see that this is finally up; I know a lot of people have been
clamoring for a more detailed reference for a while and this should hopefully
make it easier for them to interact with APFS.

------
plg
fonts?

~~~
sawaali
Should be San Francisco.

~~~
sjwright
Every time I see the San Francisco font mentioned anywhere, I don't think of
Apple's rather lovely neo-grotesque, but rather Apple's original San
Francisco. (Which is also lovely, in an _oh cool, this 1984 era computer has a
variety of fonts_ kind of way.)

[http://luc.devroye.org/SusanKare--
SanFrancisco-1984.png](http://luc.devroye.org/SusanKare--
SanFrancisco-1984.png)

------
swingline-747
Good find. Wonder if there's as much doc of ZFS/OpenZFS.

~~~
chungy
Indeed there is:
[http://www.giis.co.in/Zfs_ondiskformat.pdf](http://www.giis.co.in/Zfs_ondiskformat.pdf)

It glosses over and assumes knowledge of XDR from an external source. That is
documented here:
[https://tools.ietf.org/html/rfc1014.html](https://tools.ietf.org/html/rfc1014.html)

