
APFS – A Backup Software Developer’s Perspective - 01001
https://macdaddy.io/apfs-backup-software-developers-perspective/
======
cyphar
If your backup tool uses rsync under the hood, which is just a normal user-
space process that uses the standard filesystem APIs, why does it matter what
the underlying storage is? Obviously filesystem bugs can cause issues with
this assumption, but filesystem bugs can break a whole host of other things in
your backup tool.

~~~
sprayk
This was what I was left wondering as well. rsync for sure does not have any
FS-specific code that I can find. If it uses the FS APIs provided by the OS,
just like every other program that touches files, why would we expect anything
other than boring old rsync doing it's thing?

~~~
feelix
rsync has a ton of HFS+ specific code. Making a faithful duplicate of a file
(specifically its metadata) is a feat on its own:
[http://blog.plasticsfuture.org/2006/03/05/the-state-of-
backu...](http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-
cloning-tools-under-mac-os-x/)

~~~
sprayk
Upstream rsync does not seem to have any special-casing for mac/hfs. I'd wager
Apple's extended-attribute handling is added in this patch[0] for the version
distributed with macOS.

Even here, there isn't special casing for HFS. Instead, a special library
function, copyfile()[1], is used to handle copying files and their associated
metadata.

It seems this function was introduced in Mac OS X 10.5, which was after the
article you linked. I'd wager copyfile() was introduced in response to the
unwieldy file copy mechanics.

After discovering how copyfile() is used in rsync, I am fairly confident that
rsync works so well on the new FS as a result of Apple implementing a fairly
solid copyfile() for APFS.

[0]
[https://opensource.apple.com/source/rsync/rsync-20/patches/E...](https://opensource.apple.com/source/rsync/rsync-20/patches/EA.diff.auto.html)
[1]
[http://www.manpagez.com/man/3/copyfile/](http://www.manpagez.com/man/3/copyfile/)

~~~
trapperkeeper74
You're looking in the wrong place. Install the homebrew version, it has 3
patches for macOS that aren't in the upstream.

[https://github.com/Homebrew/homebrew-
core/blob/master/Formul...](https://github.com/Homebrew/homebrew-
core/blob/master/Formula/rsync.rb)

------
eps
And this is how you do SEO-oriented content marketing, kids.

Rather light on the substance, few select links (with at least one pointing
out) and lots and lots of precisous keywords.

\--

Besides, the fact that a _file_ backup tool might need a near complete rewrite
with a change of a file system should raise some eyebrows. It might be
understandable for an imaging backup or if it was working directly with raw
file system, but this one doesn't.

~~~
Confiks
> And this is how you do SEO-oriented content marketing, kids.

And this in turn, kids, is how you assume bad faith without any reservation or
doubt.

~~~
eps
Who said anything about this being written in bad faith?

It's a really well executed example of content marketing. Whether it was
intentional is a separate question, but the post is really quite shallow in
technical terms and it exagerrates things here and there, so - yeah, it does
look like something written primarily for promo reasons.

~~~
_ix
Sure. There's nothing wrong with promotional material, nor is there anything
wrong with calling a spade a spade.

To be fair, though, your original comment did come across as negative — was it
supposed to?

------
dsabanin
"So, before installing 10.13, I started off by googling for any problems
experienced by people installing and upgrading filesystems during the process
using the public betas. To my surprise there were none."

Well, my quick visit to developer discussions forum at Apple website related
to macOs Beta quickly revealed a lot of people with the problems. It seems
like the author didn't look too far:

[https://forums.developer.apple.com/community/beta/macos-1013...](https://forums.developer.apple.com/community/beta/macos-1013-beta)

~~~
dpkonofa
That link shows discussions related to problems with the OS but it doesn't
list any problems with the filesystem upgrade. If it does, they might have
been bumped down by more recent posts but the only major issue (and the person
posting it admits that they changed some permissions) is a read-only drive.
That seems like pretty good odds.

~~~
dsabanin
If you go beyond the first page you will see more, including data loss after a
few days of usage and during setup.

I'm not trying to be negative, I was just pointing out the results of my own
research into this question when I was considering to try High Sierra beta.
It's not "zero issues" as the original author wrote.

~~~
dpkonofa
Mind linking to one? I can't find anything that has data loss in the subject
and most of the other threads are not about APFS. The only one I can still see
is regarding a kernel panic but it doesn't give any details, just an
assumption that the issue is with APFS.

~~~
dsabanin
Sure.

"I converted my Macintosh HD to the APFS and it will not let me boot it up
anymore."
[https://forums.developer.apple.com/thread/80784?q=apfs](https://forums.developer.apple.com/thread/80784?q=apfs)

"1TB APFS Container Corrupt, Then Reinit to Single Volume"
[https://forums.developer.apple.com/message/247781#247781](https://forums.developer.apple.com/message/247781#247781)

"APFS Brand new filesystem corruption"
[https://forums.developer.apple.com/message/145039#145039](https://forums.developer.apple.com/message/145039#145039)

"Re: Corrupted APFS disk map error"
[https://forums.developer.apple.com/message/254468#254468](https://forums.developer.apple.com/message/254468#254468)

"Device does not contain a valid APFS container"
[https://forums.developer.apple.com/message/235888#235888](https://forums.developer.apple.com/message/235888#235888)

"Re: Files & file changes disappear on APFS volume"
[https://forums.developer.apple.com/message/241082#241082](https://forums.developer.apple.com/message/241082#241082)

"Re: MacOs High Sierra Beta corrupted my disk"
[https://forums.developer.apple.com/message/244120#244120](https://forums.developer.apple.com/message/244120#244120)

I'm not really interested in debating this further, so if you are not
satisfied by these references let's agree to disagree :)

~~~
feelix
All of those, bar one, are from previous versions (not the latest GM) of High
Sierra, and then it's not certain that that was what they were using. So
granted, you have managed to find up to one case of what you were saying.

------
swyx
>Although too early to declare victory just yet, my anecdotal experiences
combined with my research on the web does lead me to think that the mass
transition from HFS+ to APFS is going to be another victory for Apple. This
opinion differs from all others I have read from well informed people, many of
who are predicting either catastrophe or at the very least some major
stumbling blocks. In this case, time will tell, but I am bullish where most
others are bearish. The only thing that bothers me is that I can’t understand
how they managed to pull it off.

Kudos to Apple!

------
the_common_man
I was hoping to find atleast one technical detail of why this is an
engineering challenge. But the author simply repeats the same thing over and
again about how it's hard, how competitor says it's hard, how others think
it's impossible. In the end it was flawless

------
jsjohnst
> uses Rsync under the hood

In that case, what difference does the underlying block storage format really
matter? If rsync didn’t work, general applications could easily be expected to
have issues too.

~~~
feelix
It's the metadata that's really important. Just copying a file, while
faithfully preserving metadata, is actually a feat on HFS+. That is why
popular tools such as BackupBouncer (to test the integrity of backups) came to
be.

~~~
jsjohnst
I think you missed my point, everything rsync does is via the very same system
calls any other app could use to originally get or set that metadata. As such,
the kernel file system layer abstracts the block storage away from you. This
is very different than say Shirt Pocket’s approach (linked to in the article)
where they are directly accessing/modifying/creating low level block devices.
I’d expect what they are doing to be SIGNIFICANTLY harder. The reason why OP
is generally positive is because essentially the only thing they had to do to
support APFS is a bunch of testing, unless there’s a part to this story they
didn’t tell.

~~~
feelix
I honestly have no idea what you are talking about. Shirt Pocket would not
"access/modify/create low level block devices". They would also be working at
the file layer. If you mean that you think that they are implementing their
own way of reading the filesystem through the POSIX node, I really dont think
that they are, and also it would be crazy to try to do that.

------
dunham
I'm really curious what is going wrong with Bear.app on APFS.

The theming engine breaks on APFS (all the themes look like one particular
theme) - but it works fine if you copy the app to a disk image formatted with
any other filesystem, including Case-sensitive APFS (so it only breaks on
case-insensitive APFS).

Nothing stood out to me in the dtruss output.

~~~
citruspi
> The theming engine breaks on APFS (all the themes look like one particular
> theme)

I'm not sure if I'm running into the same problem as you, but I just opened up
Bear on my Mac to check and applying the themes themselves works for me,
however the preview of each theme is the same.

This is on macOS 10.13 Beta (17A306f) and Bear 1.0.4 (3733).

Does each theme look identical when applied for you?

[0]: [https://imgur.com/dBswvky](https://imgur.com/dBswvky) [1]:
[https://imgur.com/jpHbS4i](https://imgur.com/jpHbS4i) [2]:
[https://imgur.com/LxUWWS3](https://imgur.com/LxUWWS3)

~~~
dunham
Yeah, same issue. Pretty jarring for me, because I like the white theme.

Make an HFS dmg in disk utility and move the app over and it will work fine.

Surprisingly, it works fine on "APFS (Case Sensitive)" too (I had thought
maybe it was a case-folding issue).

Edit: Oh, for me the theme works for the list of documents, just not the
"edit" panel. The two halves are stored in separate files on disk for some
reason. ([https://i.imgur.com/ZGS3Fp9.png](https://i.imgur.com/ZGS3Fp9.png))

------
SyneRyder
A few people here are disappointed that this post is about a tool that uses
rsync. But the "competitor" link is more interesting - it points to the makers
of SuperDuper!, a tool that makes bootable external backups. A couple of
highlights:

 _"...it's important to note that Apple still hasn't released any
documentation on the "proper" way to create a bootable APFS volume. An example
of what they have in mind was released for the very first time when the High
Sierra developer release came out a few months ago, but that's it. We
basically have to make an educated guess about what they want."_

and

 _" In particular, Apple has further tightened its System Integrity Protection
process, and is completely denying access to some files on the startup volume,
even when copying to a non-startup volume.... But since APFS is, again,
basically undocumented... what could that backup be missing?"_

[http://www.shirt-
pocket.com/blog/index.php/comments/news_on_...](http://www.shirt-
pocket.com/blog/index.php/comments/news_on_the_march)

------
realityloop
I haven’t tried the GM seed yet, but I’ve reported a pretty annoying bug with
inconsistent directory listings from host into guest with vagrant. A listing
in the guest will be missing entries, but you can CD into the missing folders.
This causes sites not to work as the application in the guest can’t see files
it requires. The only way to resolve it to date has been backup and reinstall
with without apfs.

------
jason_slack
Having written a backup app several years ago, I struggled to gain foothold in
the market. I sold the app off. After a few years of wanting more from my
backup, I started a new backup app. I utilize a number of commands under the
hood but it is an ncurses interface only right now (supports drag and drop
still into a terminal window).

I'd guess that if I wanted to put it out for sale I'd need to add a GUI on top
that.

Can anyone shed some light on why backup apps are struggling with preserving
metadata?

------
sigi45
Ah yeah okay tx.

I started to be disappointed as soon as i read that he is using rsync.

I was expecting someone who writes something like rsync and not a gui for
rsync.

------
sgs1370
thanks - this is a good heads up. I asked my team to back up anything not
checked into our git repo to some disk not controlled by OSX, and btw don't
rely on your time machine, who knows if some bug in the new FS might affect
that also. Better safe than sorry. I finished my note with a suggestion to
just wait a few months before upgrading to 10.13

~~~
tinus_hn
You can also keep your Time Machine disk disconnected during the upgrade so it
won’t be touched by the installer. In my opinion not a bad idea anyway.

