
APFS filesystem format - mkup
https://blog.cugu.eu/post/apfs/
======
xoa
Very, very cool exercise. For anyone who hasn't been following APFS much
though, worth knowing that Apple has stated in their AFPS FAQ [1] that they do
intend to directly "document and publish the APFS volume format specification"
themselves later this year. Presumably (as they do with other stuff they
develop) they want to be able to more easily make breaking changes during the
shakedown period of initial public deployment (first on iOS right now, later
on Macs, though maybe not a default option until 10.13 this Summer/Fall).
That'd be the most helpful in getting read/write into FUSE and other platforms
should anyone choose to do so.

\---

1\.
[https://developer.apple.com/library/content/documentation/Fi...](https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html)

~~~
avar
They said the same about Facetime and iMessages but never opened or documented
those, so let's see when and if it actually happens. Apple has a history of
false promises when it comes to opening and documenting their protocols & data
formats.

Edit: I misrecalled that iMessage was announced as something they were
opening.

~~~
scarface74
They never promised to open up iMessage. The story behind FaceTime is a little
bit more complicated. Steve Jobs said he would open it up without asking the
engineers and lawyers who were just as surprised as anyone else about his
announcement.

As far as Apple not open sourcing anything.

[https://opensource.apple.com](https://opensource.apple.com)

They also published the full git commit history of Swift from the initial
commit.

~~~
lobster_johnson
Apple's relationship with open source is pretty average. In the end, they use
open source strategically. If there's no benefit to open-sourcing something,
they won't.

Clang, WebKit and Swift are "proper" open source projects, done in the open.
But there aren't many of those. They have no open source apps, and most
frameworks such as Cocoa, AppKit etc. are completely proprietary.

Some of their open source efforts have been half-hearted; Darwin was
enthusiastically embraced by the community, then essentially shut down by
Apple. The XNU sources are APL, but there's nothing open about its
development. I/O Kit is closed, I believe.

~~~
madeofpalk
> In the end, they use open source strategically. If there's no benefit to
> open-sourcing something, they won't.

Wouldn't that apply for most companies? Let's not pretend that Facebook,
Microsoft et al do open source as a charity

~~~
exadeci
They don't but they both perform much better than Apple on the subject.

React is one of the leaders in JS dev and several other Facebook projects are
quite useful.

Microsoft is the company with the biggest number of open source contributors
and several of their projects like TypeScript are extensively used and they
are the biggest contributors to LINUX
[https://octoverse.github.com/](https://octoverse.github.com/)

Apple made Swift an open source language that runs only on their proprietary
devices and that's it...

~~~
bartvk
> made Swift an open source language that runs only on their proprietary
> devices

Incorrect. Besides Linux, like the other poster mentioned, it even runs on
Raspberry Pi:

[http://dev.iachieved.it/iachievedit/swift-3-0-on-
raspberry-p...](http://dev.iachieved.it/iachievedit/swift-3-0-on-raspberry-
pi-2-and-3/)

------
wolfgang42
For those who, like me, don't follow developments in this area closely:

APFS is Apple's new filesystem, intended to be used across all of their
devices (Macs, iOS, watchOS, etc). For Macs, it will be replacing HFS+, a
direct descendant of the Hierarchical Filing System which has been used since
1985. (The original Macintosh Filing System, from 1984, was suitable only for
floppy disks; among other things, it was a flat filing system and didn't
really support folders.) The main improvements HFS+ made over HFS were support
for larger files and Unicode; Apple has since added some more features but the
core filesystem remains a 30-year-old format which was designed for early hard
disks.

~~~
twoodfin
It replaced (as of iOS 10.3) HFS+ on the mobile devices as well, didn't it?

~~~
agildehaus
Almost correct. 32-bit devices that received iOS 10.3 did not receive APFS. It
was exclusively delivered to 64-bit devices.

~~~
orbitur
It's also worth noting that the first 64-bit iPhone/iPad were released in
2013, so they've been around for a while.

The iPhone 5c will have the shortest support window from its release, at 4
years.

------
mandeepj
If someone wants to know like me why Apple came up with a whole new FS -
[http://www.cultofmac.com/435718/apfs-new-apple-file-
system/](http://www.cultofmac.com/435718/apfs-new-apple-file-system/)

~~~
ianopolous
Great link! I'm very interested in the mention of per file keys, one for data
and one for metadata. I wonder what data structure they store these in. It
sounds a lot like cryptree [0].

[0]
[https://raw.githubusercontent.com/ianopolous/Peergos/master/...](https://raw.githubusercontent.com/ianopolous/Peergos/master/papers/wuala-
cryptree.pdf)

------
karmicthreat
Kaitai which was used as a parser for this looks pretty interesting. I need to
reverse engineer some of the Allen Bradley Ethernet Industrial Protocol and
its client communication. This seems pretty useful for documenting that.

~~~
contingencies
[http://kaitai.io/](http://kaitai.io/)

Also check out [https://ohmlang.github.io/](https://ohmlang.github.io/) and
view the maths demo in your local copy.

~~~
karlshea
Holy shit this is so cool. I will never have a need for this but it's really
amazing to poke around with. It kind of feels like Wireshark for files instead
of packets.

They also have a web version[0] with a bunch of example files if you want to
check it out without installing it locally.

[0] [https://ide.kaitai.io/](https://ide.kaitai.io/)

------
israrkhan
Apple should publish something like TN1150 for APFS. several years ago, I had
to implement HFS+ for an embedded system, and TN1150 proved very helpful,
saved me a lot of hassle.

[https://developer.apple.com/legacy/library/technotes/tn/tn11...](https://developer.apple.com/legacy/library/technotes/tn/tn1150.html)

------
jdmnd

      Timestamps are 64bit nanoseconds
    

As far as I can tell, this will overflow in just 11 years – 2028-06-15
09:33:27.3709551615 UTC

Why would this go into a new standard? Is there a practical use for such a
high level of granularity?

edit: off by a factor of 10 in my calculation... It's actually

2554-07-21 23:34:33.709551615 UTC

~~~
raldi
By my math, we're fine for almost 600 years:

[http://wolframalpha.com/input/?i=2%5E64+nanoseconds+from+now...](http://wolframalpha.com/input/?i=2%5E64+nanoseconds+from+now&x=0&y=0)

~~~
cimnine
But that should be 'from 1.1.1970', i.e. from the unix epoch. Still far out,
though.

------
fasterthanlime
Typo thread!

"this hight result in" => "might"

"and seams to be related" => "seems"

"sometimes spaceman" => "sometimes called spaceman" ?

Very interesting read, thanks!

------
tomsmeding
Awesome research, would love to be able to do that.

Some things that came to mind: I like that some of the block types are
"acronyms" of their meanings: 5 = S(pace man), B = B(-tree), C = C(heckpoint).
Also, I wonder why the volume superblock magic is "APSB", while the root
(container) magic is "NXSB"; maybe this refers to NextStep? Seems a bit
anachronistic, but I don't know.

~~~
kstenerud
Apple superblock, and next superblock (meaning "the next superblock", not
nextstep).

------
sandGorgon
Anyone know how Apple pulled off an in place filesystem upgrade ? Is the data
copied to your upgrade laptop and back ?

~~~
djrogers
No, nothing like that necessary. Data remains in the same spot, and during the
conversion new metadata and file tables/pointers are created in APFS format.
Then the volume is remounted using the new format, and the old hfs stuff goes
bye-bye.

------
twoodfin
Does anyone know or can speculate on the purpose of the 4 "alignment" bytes in
the node block header? Are they _for_ alignment (in which case it's strange to
have them at the beginning instead of the end)? Or are they _describing_
alignment (in which case it's strange to need 32 bits)?

~~~
cugu
I just called it alignment. It's just a bit that decides wether the following
entries are fixed length or variable length.

------
mappu
Is there a filesystem that can fragment a large, existing file to perform
arbitrary inserts?

~~~
tyingq
Common in the mainframe world, like VSAM. That would be variable length record
oriented files, versus stream files. Someone did do this for GEOS:
[http://justsolve.archiveteam.org/wiki/GEOS_VLIR](http://justsolve.archiveteam.org/wiki/GEOS_VLIR)
, but I've never seen that for anything remotely unix like.

------
tradersam
Ends a little abruptly (as in no closing) but other than that interesting
read.

~~~
mbell
It's a description of the format, not a review of it. What conclusion were you
after?

~~~
johnpowell
I wasn't even after a conclusion or anything. But since there isn't a footer I
just thought maybe my internet connection died and it only loaded a portion of
the page.

------
xaduha
Is there any reason to use APFS on desktop?

~~~
sargun
There are a couple features missing from HFS+. Some of these are implemented
in APFS: * Case sensitivity * Clones extents / ranges (reflinking) * Holes in
the middle of files * ACLs * Collapsing file ranges * Per file compression and
per file encryption

These are useful for apps like SQLite or other stores which implement a multi
user, log structured store.

~~~
mrmondo
FYI - HFS is capability of case sensitivity and is an option during
formatting. I believe it even used to be the default.

~~~
duskwuff
HFS and HFS+ were never case-sensitive by default on Mac OS. Case-sensitive
HFS+ was made available as an option in Mac OS 10.0, but was always glitchy
(especially for boot volumes), because many applications depended on case
insensitivity.

~~~
Tanegashima
I have a case sensitive HFS+ and everything works fine.

That was a problem with older versions of Adobe suite, but since lots of
people on Macs used Adobe Apps...

~~~
TheAceOfHearts
Isn't the problem still present? A few months back I tried installing the
latest version of Photoshop and Illustrator, and it didn't let me.

------
gigatexal
So it is copy-on-write. Interesting.

------
Tanegashima
Look, we get it, you bought an Android and you want to use iMessage and
FaceTime.

That would be stupid, Apple doesn't make money from users data, Apple doesn't
listen to conversations like Google does to sell ads.

So why would they support servers (and there are a lot more of android users
than iOS users) for people that didn't bought anything from them? For people
that bought Androids to like them? They don't need that.

It has nothing to do with open source, the open source community doesn't need
this and Apple couldn't care less about the GNU types after GPLv3
incompatibilizes with their business (while Android OEMs break the license
every single day, but the community turns a blind eye, because it's Google)

Also, they would lose a few of their product sales and lots of 30% shares on
sticker packs and other Apps for iMessage they sell.

~~~
dang
We detached this subthread from
[https://news.ycombinator.com/item?id=14175206](https://news.ycombinator.com/item?id=14175206)
and marked it off-topic.

