
macOS 11: copies of dynamic libraries are no longer present on the file system - jedisct1
https://twitter.com/mycoliza/status/1275305517066227712
======
oefrha
No more system dylibs _on the filesystem_ ; only loadable with dlopen (from
shipped dyld cache). The title is a shocker that sounds like system libraries
are no longer dynamically linked...

Edit: Since this comment is currently at the top, here's a better link to the
source: [https://developer.apple.com/documentation/macos-release-
note...](https://developer.apple.com/documentation/macos-release-notes/macos-
big-sur-11-beta-release-notes#:~:text=built-in%20dynamic%20linker%20cache)
(using scroll-to-text fragment[1] since there's unfortunately no usable anchor
to the specific entry; not supported in non-Chromium browsers at the moment,
in that case search for "built-in dynamic linker cache").

[1] [https://caniuse.com/#feat=url-scroll-to-text-
fragment](https://caniuse.com/#feat=url-scroll-to-text-fragment)

~~~
ohgodplsno
>not supported in non-Chromium browsers at the moment

Not supported in any-browser-that-follows-the-spec, really, considering
scroll-to-text is a google made, unilaterally implemented, nonstandard
extension ([https://wicg.github.io/scroll-to-text-
fragment/](https://wicg.github.io/scroll-to-text-fragment/))

~~~
oefrha
I never said it's a standard. Anyway, I'm interested in providing information
about dyld cache (and as an aside explaining why my best-effort link doesn't
work in some browsers), not encouraging more tribalistic Chrome/anti-Chrome
debate, so we can stop right here.

------
DCKing
Not a macOS dev but maybe one can clarify: is the title "No more system shared
libraries in macOS 11" actually true? The documentation reads to me as a
restriction on how you _read_ dynamic libraries (is using only dlopen
controversial here? also just trying to understand), not on which system
libraries are actually there. It seems dlopen with custom system dylibs would
still work if there is also a means to get your dylibs in the system cache?
Based on that piece of docs alone I see no reason to grab my pitchfork just
yet.

Anyway, I was expecting a few shitstorms to start happening over macOS 11.
Apple not announcing any major deprecations (edit: or removals) for this
transition - even OpenGL/CL is still there, depecrated - was suspicious.

~~~
frogblast
You're correct. For 3rd-party provided dylibs, it also isn't necessary to get
them into the system cache.

dlopen() will look for the dylib first (as today), and if the file doesn't
exist, it'll look in the shared cache.

So the only people this actually impacts are those who stat() system-provided
dylibs before calling dlopen(). They should just skip the stat(), and go
straight to calling dlopen().

Certainly not worthy of the end-of-the-world vibes in the twitter thread...

~~~
loeg
Classic linking uses a path to a library at link time. I suppose Apple may
have patched ld to work around this choice, but it still begs the question —
why change this? Who was it hurting?

~~~
saagarjha
> why change this? Who was it hurting?

Performance.

~~~
loeg
Can you elaborate beyond a single word? It is certainly not immediately
obvious to me how this benefits performance, or I wouldn't have asked.

~~~
saagarjha
Here's one optimization:
[http://sealiesoftware.com/blog/archive/2009/09/01/objc_expla...](http://sealiesoftware.com/blog/archive/2009/09/01/objc_explain_Selector_uniquing_in_the_dyld_shared_cache.html)

------
nly
Isn't this the model that has been encouraged on Windows forever with COM,
WinRT and .NET etc? When you consume an API in one of those environments you
don't have to worry about the location of the physical assets (library files).

I don't see the big deal as long as app local copies of e.g. curl can still be
used

~~~
gruez
How does it work on mac? Do you have to specify the full path of the library
to load it?

~~~
ratww
In practice it's transparent, like using a static library. The linker handles
it all for you during compilation: it stores the paths at a special section in
your executable, and a injects a _dynamic linker_ in your code.

The paths can be either absolute (for system libs) or relative (useful for
distributing Bundled apps).

The cool thing is that you can see those hardcoded paths using _otool -L_ and
you can modify them using _install_name_tool_. Useful for debugging
misbehaving closed-source tools.

------
0x0
This is mildly interesting but not much more than that. Who cares if the set
of system libraries are available in a series of separate dylib files, instead
of being packed together in one mega archive?

Most comments here on HN seem to misunderstand what this is.

~~~
asveikau
People writing third party linkers or debuggers is one.

I often dig at shared libraries using tools like nm to see what functions are
where.

~~~
0x0
I'm sure there will be a way to unpack the mega archive and run nm (or maybe
even run nm directly on the mega archive). Relax :)

~~~
asveikau
I never said I wasn't relaxed about it. I don't even use a Mac much.

But the platform does seem to have been in a multi-year process of technical
decline, and much of that does speak to a "we know better than you, and
everybody else" attitude.

~~~
prewett
A number of the changes that Apple has been making seem to be security
related. One reason to just dlopen() rather than stat() and dlopen() is the
race-condition, which an attacker could use to get a program to use the wrong
library. Similarly macOS' locked-down root partition. Checking hashes of
binaries is also for security, to make sure the binary you're running is the
one the developer thought they were giving you. So on the one hand the
platform is in "slow technical decline" yet we (HN) claim to value security.
It seems to me that the platform is doing well technically. We all remember
the mess Microsoft had 10 years ago with security vulnerability du jour; Apple
seems to be pre-emptively protecting its users. I submit that the "technical
decline" is really "macOS is changing from a freewheeling Unix system to a
secure system" and people here want to be cowboys and artists with their own
private filesystem canvas. I understand the frustration, but Linux still
exists. Apple isn't in the business of making a dev playground, but a computer
that Just Works for people. It seems to me there isn't a technical decline,
it's that HN is revealing itself to value the plasticity of traditional Unix
over security (now that security has gotten good enough). I expect that
SELinux has similar chafing restrictions, the only difference is that Linux is
more of a set of parts so you get to (must) choose how you put those tools
together.

~~~
asveikau
> One reason to just dlopen() rather than stat() and dlopen() is the race-
> condition

But dlopen() implies code execution all the time, and stat() does not.

A good way to close this race condition, by the way, is to have a dlopen-by-fd
API [FreeBSD has fdlopen], and a single call to open(2) resolves the race
condition inherent in referencing the filename multiple times.

But to your larger point... It seems there is a crowd that will kneejerk
dismiss any criticism as "it's for security". And any implementation bugs
(such as introducing massive delays when executing programmatically-generated
shell scripts because they need to be hashed and sent to a server, as was an
HN headline a while ago) are also justified. I think there needs to be greater
skepticism than this about how secure these efforts really makes the system,
or if they are just throwing buggy stuff at a wall and applying post hoc
justifications.

------
eugenekolo
If I understand correctly, this is just the same behavior as iOS's
"dyld_shared_cache" which is basically all the system shared libraries stuffed
into a single file.

[https://iphonedevwiki.net/index.php/Dyld_shared_cache](https://iphonedevwiki.net/index.php/Dyld_shared_cache)

~~~
saagarjha
Correct, all the system libraries can be now found in the dyld shared cache
can be found in /System/Library/dyld/dyld_shared_cache_x86_64(h) (it's moved
in Big Sur!).

------
coldtea
Are those guys in the twitter thread looking for an excuse to vent? It's a
minor annoyance at best, and with security/speed benefits...

~~~
john_alan
Yeah I mean what system libs do people even use?

Does this affect homebrew?

------
quotemstr
Good. Apple is basically the only entity seriously looking at dynamic linking
performance these days. We can do a _lot_ better in the ELF world. We should
not only copy dyld's closure cache, but also teach the kernel to do
relocations on demand, like NT does. A relocated data-segment page should be a
_clean_ page, not a dirty page!

------
icedchai
To think, the Amiga was doing something like this in the 80's. The
"OpenLibrary" call would transparently access shared libraries either in ROM,
already loaded into memory, or on disk.

~~~
sys_64738
Well ARM is a 1980s technology so seems apt.

~~~
messe
x86 is 1980s. POWER is 1990s. MIPS is 1980s. Sparc is 1980s.

What would you prefer they use? Itanium? RISC-V?

~~~
xenophonf
Itanium is 2000s. RISC-V is 2010s.

Apple could make a great leap forward by adopting the Mill, an ISA so novel
that it hasn't even been released yet!

~~~
icedchai
Too bad Alpha died. Those machines were like lightning (for the 90's, anyway.)

~~~
messe
IIRC even Windows NT could run on Alpha. I wonder if computing would've been
different had we gotten a modern 64-bit architecture with very little legacy
become commonplace in the 90s.

Makes me think of the Tanenbaum quote from 1992: "Of course 5 years from now
that will be different, but 5 years from now everyone will be running free GNU
on their 200 MIPS, 64M SPARCstation-5".

~~~
icedchai
Yes, Digital Unix, OpenVMS, and Windows NT all ran. You could also run Linux
and FreeBSD. I worked at a DEC shop for a while in the late 90's. Alphas were
speed demons. This was right before Compaq bought DEC. After that, I think
things started going down hill...

------
jlokier
I wonder how many years we've got where you're still able to _read_ the big
mega archive...

Someday it may sit in some kind of "protected enclave", mapped into your
process with "execute-only" permission for code, and maybe graded protections
for its data as well, so you can't debug, trace, instrument, reverse engineer
or otherwise get to really know the system libraries to see what they are
doing and learn from them.

The trend has been that way for a while. For the moment, that kind of
isolation is done at process boundaries, on devices that have enclaves.

There is no technical reason why that kind of isolation cannot extend inside
processes, for a company that designs its own CPUs. For the same kind of
"black boxes" as we already have inside mobile devices, but with faster API
calls and data exchange than doing it across processes boundaries.

------
pdimitar
Likely an unpopular opinion: not everything has to be a file. UNIX removed a
lot of obstacles and helped usher in the era of computing accessible to
individuals -- but we can't keep relying on old paradigms when it's now clear
(with the benefit of hindsight) that it's not a be-all-end-all solution.

Filesystems are awful databases and I believe everyone who was a programmer
for some time realizes this. The lack of ACID being at the top.

The sooner we move to specialized APIs for, well, everything, the better off
we're going to be.

\---

As one recent example, the PS 5 will just directly load 3D artefacts from the
SSD storage to the GPU when the corresponding API is called. This gives Sony
peace of mind that people won't be coming up with 100+ ways of doing it by
themselves which Sony will then have to support for the entirety of the PS 5
lifecycle.

We the programmers have demonstrated to ourselves, time and again, that too
much freedom can actually be a bad thing.

It's okay to move away from the "everything is a file" philosophy. If
anything, I'd love to see a Linux-like OS that utilizes sqlite3 for, say, the
entirety of the current `/etc` directory and uses sqlite3's backup/changeset
APIs to do incremental and point-in-time backups. Sure it's little less
transparent but you can't ever capture an inconsistent snapshot. That has to
be an improvement, right?

\---

EDIT: I suppose people can't find a link between my comment and the topic but
my point was that moving to dedicated APIs -- and not relying on the
filesystem -- for various services (like loading a dynamic library) is a good
thing because the "everything is a file" paradigm isn't as bulletproof as it's
made out to be (also because "stat" \+ "dlopen" one after another are race-
condition prone as others pointed out).

I'd love to hear the downvoters elaborate their reasoning.

------
Spivak
I think this is a good change. It's eliminates a potential avenue of
accidentally doing the wrong thing but not noticing because it "works on my
machine."

~~~
loeg
In what scenario does it eliminate this?

~~~
olliej
I have had to deal with bugs in the past caused by people using incorrect case
for paths to dlopen(), etc.

Lots (I wouldn't be surprised if _most_ ) osx devs use case sensitive
filesystems, but the default in case-insensitive, and so dlopen(path) fails if
you have (say) "/system/..." instead of "/System/..."

------
viraptor
Can anyone come up with a reason this would be preferable to having just the
index cached for the linker and handing out mapped memory / open files
instead? What's the benefit here over essentially ld.so.cache with bells and
whistles?

~~~
bdash
It reduces disk space.

Historically the dyld_shared_cache has been a pre-bound copy of most system
libraries that is mapped into memory once and then shared across all
processes. It was updated automatically by the system at various times, and if
a given system library was determined to be newer then the dynamic linker
would simply load it from disk. The result of this was that for most system
libraries you ended up with two copies on disk: one at the normal location,
and one within the shared cache. Now only the latter copy exists, and Apple
ships it directly rather than creating it on each machine from the copy of the
system libraries that they ship.

~~~
beervirus
Disk is cheap, and we're not talking about a huge amount of space. Surely
there's a better justification.

~~~
kolinko
On lower end laptops it isn’t cheap. Also these savings add up - a few hundred
MB here and there and you end up with a system that’s 10GB lighter on a laptop
that has just 128/256GB of space.

------
szc
Sorry, but the /usr model for compiling is out of date / flawed. It isn't
about "me" it is about "you" the target for running the executable.

For many years, macOS has supported being a cross compilation environment for
multiple platforms and multiple OS releases.

This means that include files and libraries for the target OS and platform are
not stored at the "host filesystem" paths that were used in the 1960's, 1970's
and 1980's... (an embarrassingly long list). Include files and references to
libraries are available in the SDKs. One should use the SDK path as a prefix
for the include and library paths - something that most GNU tools, and other
open source projects have supported for a very long time, but sadly a little
inconsistently. There are runtime tools and recipes that can be used to get
these paths, including for "myself".

All compilation on macOS should be considered as cross compilation -- because
macOS simultaneously supports multiple architectures and OS versions.

This really is a good abstracted model for all compilation.

This has benefits! Install time and disk space usage are reduced for most
customers and developers. Developers on x86_64 do not need the aarch64 content
in /System, /usr to compile. Install the SDKs for the targets you wish to
compile for.

------
sjnu
Frees up a gig or two of disk space. Good.

~~~
loeg
Frees up cheap disk space by moving the library contents to expensive RAM?
That doesn't seem like a good thing. And it also doesn't seem possible — you
want system libraries to survive reboot. Presumably these files exist
somewhere on disk, just hidden.

~~~
plorkyeran
The dylib cache is stored on disk. It's saving disk space by only having that
copy of the libraries instead of two copies.

~~~
loeg
What would the second copy be?

------
qppo
two questions

Does this break pkg-config?

What about fishing for dylibs in .bundle directories that may be installed in
multiple locations?

------
tobyhinloopen
So the title is wrong? The libs are just not directly accessible from the file
system

------
celeritascelery
I assume this is for privacy (an app can’t fingerprint you by looking at which
libraries you have) but macOS is starting to become a terrible dev
environment. Why are they locking down my device from me?

~~~
antishatter
When you say "Starting To" I presume you mean from about 2015 on. It's clear
Windows is trying to pick up slack and also be cloud developer friendly with
WSL integration.

Apple sticking with Intel and now reverting to a terrible proprietary cpu
architecture, locking software down more and more, releasing pro versions with
inadequate options and keyboard all acts to ensure they lose their status as
top developer environment

Before they dropped powerpc for intel, no one would have considered using a
mac for dev work.

It's clear they don't understand the developer market at all or are only
interested in Mobile because the revenue is insane.

They had a good ten years as the best dev environment but as they say "The
King is dead, long live the King".

~~~
coldtea
> _terrible proprietary cpu architecture_

Err, there are more cpus with ARM than x86 by an order of magnitude...

It's also less proprietary...

> _Before they dropped powerpc for intel, no one would have considered using a
> mac for dev work._

Huh? Where did you get that from? OS X Macs where quite popular for dev work
pre-Intel, popular for web devs, and with good Java support...

~~~
pritambaral
I'm guessing when your parent said "cpu architecture" they meant precisely
that and not "instruction set architecture". The ISA is ARM, sure, but the CPU
architecture is Apple.

------
afandian
How common is this technique for loading libraries? How about in the kind of
POSIX tools that are available, but not exclusive to, on mac os (sucha as
bash, termux, vim, git, apache etc)?

~~~
zaarn
The old code checks for the filepath to be present before attempting to dlopen
it. The new way requires you to dlopen the path. This is very meh because
dlopen already executes code from the library, so if you didn't want that,
good luck.

(glibc and some other core libs like openssl like to detect plugins by probing
the paths for libs and only loading them when necessary)

~~~
afandian
Yeah. So I understand it has been considered best practice to check the file
before opening it? Does that mean we can assume that most well-written
programs will follow the pattern and break on mac os?

I would have thought that even if Apple did make this change they could trap
the fopen or whatever and return something useful based on a heuristic? It
wouldn't be the first time that they put exceptions and heuristics in for
software compatibility.

~~~
chrisseaton
> So I understand it has been considered best practice to check the file
> before opening it?

Check it... for what?

~~~
coldtea
For presence...

~~~
chrisseaton
Why do you need to separately check for presence before you use dlopen? If
it's not there dlopen will give you a nice error message - no need to check
yourself.

Are you currently checking files are there before you dlopen them? How do you
avoid a race between the stat and the dlopen? What happens if the file is
removed between them? Seems broken and you shouldn't be doing this in the
first place.

~~~
zaarn
It is usually unlikely that a file will be removed between stat and dlopen,
such errors occur rarely. Even more rarely when the file happens to be a
systems lib or plugin lib. Some errors occur rarely enough that it isn't worth
worrying about.

------
ddevault
Is there a good term which distinguishes a general-purpose OS (Unix, Windows,
Plan 9, etc) from operating systems like Android, iOS, Windows RT, etc? The
term "operating system" is being overloaded to ship systems which are really
more of a graphical shell than a classical operating system. The moves to lock
down operating systems, ostensibly made for "security" reasons but
conveniently centralizing power and authority with the vendor at the same
time, are being marketed to consumers as the same kind of product as a
conventional operating system while in fact being profoundly different.

~~~
saagarjha
I think you're in the wrong thread.

------
messe
Pretty similar to what they do with kexts on iOS.

~~~
saagarjha
iOS does this for dynamic libraries too. Actually, it's been doing this for
many years.

~~~
messe
Didn't know that, I've never really done any serious iOS dev. Only knew about
the kexts because it came up in a job interview.

------
jtbayly
Somebody asked on Twitter, "doesn't this break all of Brew?"

Does anybody know?

~~~
unilynx
I don't see why, Brew just uses the standard compilation tools any other
software on OSX uses, so it'll work the way any other tool works

Brew either uses the system libraries (which still work) or compiles its own
version (which is not affected by this)

------
AlexandrB
Not to be paranoid, but with this change what stops Apple from silently
shipping libraries with backdoors/surveillance if requested by law
enforcement. How would a security researcher catch the change without being
able to examine the library files?

~~~
hellcow
They don't need to ship a backdoor because all files in iCloud (photos,
documents, etc.) aren't E2E encrypted anyway. According to Reuters they
readily share that with the government, despite all their FBI-thwarted-yet-
again posturing in the press.

Not to mention they could just backdoor the kernel, and since that's not open-
source, good luck detecting that.

~~~
chungus_khan
And of course they do, surveillance is the law in the US. They are expected to
comply.

------
stephc_int13
I don't see why it should be any different.

------
dfalzone
How will we write our own libraries, then?

~~~
sjnu
As before? This just means there are no longer two full copies of every system
library on disk.

------
dfalzone
How would we write our own libraries?

~~~
tannhaeuser
I guess just as before, by using install_name_tool to burn the install
location into a .dylib (with special vars), and then link against it? The
question seems rather, how to link against system libs if these are no longer
palpable files.

------
nojito
Clearly a change for privacy reasons.

You aren’t allowed to go fishing around the system anymore.

~~~
gspr
> You aren’t allowed to go fishing around the system anymore.

Sure you can. You'll now `stat` around for everything you did before, but now
you'll have to `dlopen` the system libraries instead. Surely no privacy win?

------
jonny383
For end users, this transition will be fairly seamless. For developers, let
the nightmares begin.

------
choeger
If I read this correctly, this is absolutely bonkers. Basically the "System
libraries" do not exist as files anymore? So they just deprecated normal
dynamic linking for a subset of libraries? But as I know the Mac folks around
there, they are going to swallow it. Because, uhm,..., It's Apple.

~~~
chrisseaton
> this is absolutely bonkers

Why do you care if the files are on the disk or not though? What are you using
that for?

~~~
choeger
For linking.

