
Enduring file references in macOS - chmaynard
https://eclecticlight.co/2018/03/11/last-week-on-my-mac-enduring-file-references-in-macos/
======
jakobegger
Bookmarks are great in theory, but in practice they have a few issues:

1) The API is poorly documented, and behaves differently on different macOS
versions

2) Error reporting is really bad. Sometimes “opening” a bookmark fails, and
there’s no indication why. The error message suggests that the user should
change permissions in Finder, which is incorrect and misleading

3) It’s really easy to forget to “close” bookmarks, leading to leaked kernel
resources (which might be one reason for 2)

4) Issues with code signing or the keychain can cause bookmarks to silently
fail to open. There are no error messages that suggest this is the case, the
only way to fix these problems is to search Google and try random things until
it starts working again.

Bookmarks, just like the whole sandbox API, are a good idea in theory, but
they suffer from poor documentation and extremely poor error reporting, making
them unreliable. It’s no wonder developers are reluctant to adopt the sandbox
unless they want to distribute their app on the App Store.

~~~
kitsunesoba
I haven’t worked with these APIs for a couple of macOS versions but yeah, this
reflects my experience too. It’s enough of a frustration that if your
application works with arbitrary files often, it might be necessary to opt out
of sandboxing (and the App Store) if you want to ensure a consistent
experience for users and don’t have the time or resources to test your way
through all possible failures.

Maybe this area will get some attention with the rumored upcoming AppKit
replacement.

~~~
jakobegger
I strongly doubt that an AppKit replacement would fix the sandbox issues.

Also, to be fair, I think that quality of the sandbox has improved
significantly over the years. The remaining issues are in general pretty rare
(my apps have thousands of users, and I only get a handful of support requests
about these issues per year).

The problem is that when these issues pop up, there is nothing I can do to
debug them -- aside from asking users "have you tried rebooting your
computer?" (which does fix the issues occasionally)

Apple really needs to make better error reporting. When opening a security
scoped bookmark fails, the system should return a proper error message, or at
least log the reason for the failure.

And they must absolutely fix their error messages. When the sandbox denies
access to a file, AppKit shouldn't suggest changing permissions in the Finder
-- that won't help a thing.

------
joosters
You can’t seriously write an article extolling the virtues of inodes as a way
of locating files, without pointing out a very clear weakness: the inode of a
file can easily change when an application saves a new version of it!

Most apps don’t consider the inode as a user-visible (or user-significant)
part of file metadata and so will make no effort to preserve it...

~~~
ridiculous_fish
There's also the reverse issue: a deleted file may have its inode recycled.
Linux in particular aggressively recycles inodes.

------
derefr
My understanding is that Bookmarks are "capabilities", in the research-OS
sense: a Bookmark is a reference to a file, yes, but _having_ said bookmark
allows your process to access that file even if your app's sandbox doesn't
include it.

Notice that as of Sierra, apps can't just let you type in a path to choose a
save location, or a library file, or whatever else? Your workflow has to pass
through the File Open dialog. That's because the File Open dialog returns a
Bookmark—and as long as the app retains the Bookmark, it retains the
capability to read from/write to that folder (which it doesn't have by
default.)

~~~
jakobegger
The file open dialog does not return a bookmark. It just returns an URL, and
your sandbox is extended to get permission to use the file. You can then
create a bookmark, if you want to keep using the file across restarts of the
app.

------
PhantomGremlin
FYI for those trying to experiment with this ...

I tried using the command line to see what's under /.vol and nothing appeared.
Which confused me, because I found some old stuff on the web that said I
should see "a number" there. So apparently macOS no longer displays a volume
number there.

StackExchange shows the simple technique:
[https://superuser.com/questions/163941/what-is-the-vol-
direc...](https://superuser.com/questions/163941/what-is-the-vol-directory-
for)

You just stat a file to learn both its volume and inode. You can then use that
pair to access the file:

    
    
       $ stat /etc/hosts
       234881026 23347035 -rw-r--r-- 1 root wheel 0 1035 "Jul 15 10:11:35 2010" "May 30 20:55:10 2010" "May 30 20:55:11 2010" "May 30 20:47:32 2010" 4096 8 0 /etc/hosts
    
       $ head -3 /.vol/234881026/23347035
       ##
       # Host Database
       #

------
klodolph
The article hints at this at the end, but note that the Mac has worked this
way for a long, long time. You would use an “alias” to store the reference to
a file. The alias could be a file itself (via the “make alias” command in the
Finder) or you could just store the alias as data somewhere.

Mac apps have been working this way since the 1990s no problem.

~~~
kitsunesoba
Yep. I can’t find it now but Apple even had a video that marketed this as an
advantage of macOS compared to DOS and early versions of Windows. It handled
file moves, renames, etc on open files just fine while DOS/Windows would flip
out and sometimes eat your unsaved changes.

~~~
fiddlerwoaroof
I vaguely recall Windows NT 4 telling me to close the file before deleting or
renaming it in Windows Explorer

~~~
klodolph
This one is due to the major difference between the Unix and Windows approach
to files. Windows will enforce access locks (unlike Unix, which only has
advisory locks) and this is done by path (unlike Unix, which does it by
inode).

Windows now allows you to delete an open file, however, the way this works is
a bit of a hack. If you try to create a new file with the same name, the
operation will fail until the deleted file is closed.

------
g09980
_So long as we remain wedded to directory paths as a means of referencing
files and folders, much of our storage will contain duplicated content._

As a counterpoint, there are many situations where duplicating content may be
preferred. If I'm pasting something into a document, I want it to remain
immutable there without the risk of an external change (by me) altering the
document that I assume to remain static.

~~~
sly010
I would add that in general there is no one right abstraction. If you want to
be able to change the source file, you are best of using symlinks. If you want
to be able to change both files and keep them in sync, you should use a
hardlink. If you want to create a true copy and modify them independently,
just make a copy. Also worth mentioning that most COW file systems support
block level deduplication, so if enabled, two files with the same content will
be stored only once.

~~~
Firadeoclus
Rather than semantically splitting the original content (by either using
unidirectional links or getting rid of the connection entirely), it seems to
me that keeping the content in a versioned database with bidirectional links
is much preferable from the user's point of view.

The versioned database takes care of grouping related edits and lets you point
at a specific, immutable version. While bidirectional links let you
efficiently find all usage sites so that you can make an informed decision as
to whether they need updating.

------
nayuki
Quoting the article:

> This is reflected in the structure of our Home folders. We tend to put still
> images in Pictures, video in Movies, mail in Mail, then in Documents we
> collate content from those folders by duplicating it into finished
> documents.

> So long as we remain wedded to directory paths as a means of referencing
> files and folders, much of our storage will contain duplicated content.
> Bookmarks are a central part of a more advanced approach to structuring and
> organising content which can at last move on from the primitive constraints
> of MS-DOS.

> I think that we will see more, not less, use of Bookmarks as enduring file
> references in future versions of macOS. They don’t spell the end of inodes,
> aliases, links, or directory paths, but they need to be made much more
> accessible to users and sysadmins.

On the contrary, I think all of these approaches (duplication, bookmarks,
inodes, symlinks) are destined to fail. I believe the problem of file
organization cannot be solved adequately in a system of hierarchical mutable
folders/files.

With regards to duplication, I propose the use of hash references. For
example, you can put a picture somewhere on your disk, and let it be
retrievable by its SHA-256 hash. Then, when you create a document that
includes the picture, simply point to it by hash value - this avoids the need
to duplicate the full picture content into the document.

------
beautifulfreak
"Finder aliases have never been fully documented by Apple, but have been found
to contain within their opaque data structures inode numbers."

I was holding out hope that Apple would publish a tool for working with
aliases, or at least a specification of aliases so a 3rd party could design a
tool. I have a volume with hundreds of aliases that don't work anymore,
because the files they pointed to have moved to a new volume. I wish I could
batch edit their paths, using find and replace. Many Apple users have asked
for an alias editing tool, as I found when I went searching for one, to no
avail - not even a kludgy hack exists. I wonder what risk such a tool would
pose. Why doesn't Apple document the structure of aliases? Is it just because
they're messy, like photoshop psds, and they don't want anyone to see their
inelegance? Or is it because it would open Macs to new kinds of mischief?
Aliases are fantastically useful, until they break.

------
theamk
While I think bookmarks have their place, I disagree with the main premise:

> This is an extraordinary shortcoming. Imagine losing track of a stack of
> books when you move them from one room to another!

This is is how real life works -- there is no bookmarks in real life! If you
had a stack of books, and you move it into the attic, all you your memory
("Bradbury is located near the floor on the left") will not work anymore. If a
document was on your table, and you put it in the drawer, and then spilled the
ink on the table, then your document will remain unmarked.

When applied to files, directories have meaning, and magic tracking links, as
described in this article, would be surprising. When you delete old pictures
from Photos, do you want some Keynote presentation, which lives in a
completely different folder, break? When you move the file from
'current_info/' to 'outdated_info_do_not_use/', do you want old bookmarks to
be still valid?

------
chmaynard
See also this follow-up article on the various types of file references
available in macOS:

[https://news.ycombinator.com/item?id=16568103](https://news.ycombinator.com/item?id=16568103)

------
crest
I would settle for a decent CLI tool for dealing with macOS alias files.

~~~
kotharia
Back when I used OS X, I used this CLI utility for resolving HFS+ aliases to
actual paths.

[https://github.com/lipidity/CLIMac/blob/master/src/readalias...](https://github.com/lipidity/CLIMac/blob/master/src/readalias.c)

------
shakna
> Bookmarks are fine for app developers, who can make a macOS call which
> returns the current directory path to any given Bookmark.

I'm not fine with it. It's an extra function call, it's more overhead to find
and open a file, and it's much slower than placing a pointer to an inode.

~~~
klodolph
Not sure what you're suggesting but the comment seems wrong on multiple
levels. First, how do you "place a pointer to an inode"? I'm having trouble
dissecting that one. You can store an inode number if you like, but actual
inode pointers only work at the filesystem level (are you talking about hard
links?) Once you’re storing an inode number on disk, you’re basically
replacing the well-tested bookmarks system with your own little hack, and the
hack won't even work inside a sandbox (which bookmarks explicitly do). These
bookmarked files aren’t typically accessed often, usually only in response to
user input, so a small performance difference isn't negligible. So it's not
rational to complain about performance here.

Either your comment makes no sense at all or I don't understand what you're
saying.

