Hacker News new | past | comments | ask | show | jobs | submit login
Dynamic linking (drewdevault.com)
408 points by scrollaway 16 days ago | hide | past | favorite | 244 comments



> On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies.

That's correct, but also very misleading and leads to the wrong conclusion.

The dynamically linked library has references to itself, externally visible or not. It would be wrong to claim that Application.run(); only uses a single symbol of a library.

> A good linker will remove unused symbols.

With LTO or -f{function,data}-secions + --gc-sections any linker will do. Without those options no linker is allowed to. I believe that this the reason why static libraries are usually shipped as separate object files (.o) within ar archives (.a), as those were only linked in on demand.


> On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies.

For more on this, I highly recommend these two posts, which show how modern symbol tables optimize for symbols NOT being found within a given shared object via the use of bloom filters.

1. https://flapenguin.me/elf-dt-hash

2. https://flapenguin.me/elf-dt-gnu-hash


This was a great look at something that in hindsight understand must exist but was totally unaware of. Thanks!


Also, how many symbols it uses is relevant for the time linking takes, but not for memory usage.

Worse, I think it would have been easier to measure the size in bytes of binaries, or, with a bit more effort, resident memory size.


> I believe that this the reason why static libraries are usually shipped as separate object files (.o) within ar archives (.a), as those were only linked in on demand.

Yep. One function per C/obj file for smallest static binary possible.


Can't you achieve the same result with -ffunction-sections?


Yes. And its twin, -fdata-sections.


Yes but that's modern fancy stuff. :-)


Yes, on the compilers that offer it.


> > A good linker will remove unused symbols.

Additionally, a linker not be able to remove symbols that are unused at runtime but are still (transtively) referenced statically. For memory mapped dynamic libaries those code pages will not be loaded unless needed.

For example, statically linking libstdc++ for any C++ program which uses iostreams (or anything else infected with std::locale) will mean including code to format and parse e.g. monetary amounts even if your program never uses this functionality.


Over half of your libraries are used by fewer than 0.1% of your executables

That's a very misleading reference and graph. First of all, what did they expect to find? As you add more executables, of course the % usage of a library will decrease.

e.g. say I have a networking library on my computer, and, in a perfect world, all my installed network tools link against it. But now I install Gnome, and my machine has hundreds more binaries. Not all of the binaries will do networking stuff, so the % usage of the networking library goes down. But that doesn't mean that the networking library is not being shared as well as it could be.

A much better metric would be to count, for each shared library on a machine, the number of programs that link against it. If only one program uses a shared library, then that means the 'shared-ness' is not being used. If more than one program use it, then the library is being effectively shared. But the actual count, whether it is 200 users or 20, doesn't mean anything more. That's why comparing all libraries against libc's usage shows nothing useful.


Yeah. Loading the GUI widget toolkit once rather than 30 times is pretty nifty.


Even better would be to measure sharing in the totality of programs you use directly and indirectly from boot.


As always, static and dynamic linking both have their advantages and drawbacks. The usual arguments for dynamic linking around brought up in the article, and as others have mentioned here, the analysis is a bit lacking so the conclusions aren't generally true. Static linking has its own, fairly straightforwards benefits as well. It's no surprise that those who push one or the other usually do so because of their specific needs. Sometimes we even see some interesting hybrid solutions: one recent one is Apple introducing dyld shared caches on macOS, which (while being a pain to reverse engineer) are basically all the system (dynamic) libraries all statically linked together and presented dynamically, with some linker tricks to make it appear seamless. Likewise, a lot of statically linked binaries are only partially statically linked, still using things like libc or graphics libraries. The moral really is to try both and pick whichever one is the one that's better for your use case, and perhaps even consider a mix of both to give you the most flexibility in which tradeoffs you'd like to make.


Nitpick: macOS has used dyld shared caches for over a decade. The recent change is just to remove the original copies of the libraries on disk.


Ah, you're right. Forgot that they were being used already :) I'm curious what would happen though if you modified a system library and didn't update the cache, though…


dyld would check the modification time and inode and avoid using the shared cache version if there’s a mismatch. (So I gleaned from the documentation of the DYLD_SHARED_CACHE_DONT_VALIDATE environment variable, which tells it not to do that.)


Specifically for C, static linking is a trap due to its poor semantics. Don't do it. Or fix the C link-editors, then static linking C will be fine.


What would be examples of poor static linking semantics? Thanks!



Static linking has been such a nuisance for the libSDL folks that they implemented dynamic loading of itself [0], controlled via an environment variable, as an escape hatch from executables w/libSDL linked statically.

It's understandable that games, especially proprietary ones, distribute statically-linked binaries ensuring any third-party dependencies will be present and be a compatible version. But the value of that decision tends to diminish with time, as those external dependencies are frequently the pieces interfacing with the system/outside world, which keeps changing, leaving such dependencies behind to atrophy at best or become vulnerable/incompatible/broken at worst.

I don't personally think it makes sense to approach this so dogmatically. Static linking makes sense in the right circumstances, so does dynamic linking. For general-purpose operating systems, it seems obvious to me that you'd want most higher-order userspace programs dynamically linked. I want my openssl updates to touch a single library and affect all installed ssl-using programs, for example.

Having said that, I do wish the average linux distro still statically linked everything in /bin and /sbin. It was nice to still be able to administrate the system even when the dynamic libraries were hosed. At some point it was changed to just a single static binary; sln for static ln IIRC, assuming you'd be able to fix your dynamic libraries with some symlinks if they were broken, if you happened to have a shell running and could navigate using just builtins. It was already an impossible situation, but even that seems to be gone nowadays.

It's a more nuanced issue, taking an "everything dynamically linked!" or "everything statically linked!" approach strikes me as just another form of ignorant extremism.

[0] http://hg.libsdl.org/SDL/file/2fabbbee604c/src/dynapi/SDL_dy...


> [...] Having said that, I do wish the average linux distro still statically linked everything in /bin and /sbin. It was nice to still be able to administrate the system even when the dynamic libraries were hosed. [...]

This argument came up back when Solaris 10 was in development and the project to get rid of static link archives for system libraries came up (search for Solaris "unified process model"). The disposition of this argument was that if your libraries are damaged (e.g., someone unlinked them or renamed them out of the way, or maybe ld.so.1 itself), well, the dependent utilities in /bin and /sbin themselves could have been damaged too, so you can't know the extent of the damage, and it's not safe to continue -- you have to use boot media to repair the damage, or reinstall. And, of course, the packaging system has to be safe, but that's not a lot to expect of a packaging system (is it??).

To my knowledge there were no subsequent customer calls about this.


Sometimes solutions give rise to new categories of issues, and it's difficult to connect the dots to the root cause. If you believe dynamic linking hasn't introduced an even broader array of difficulties for C coders needing to support both, then please read Ulrich Drepper's DSO tutorial which gives a pretty good rundown: https://software.intel.com/sites/default/files/m/a/1/e/dsoho... If I remember correctly, it was largely SCO Group that pushed UNIX vendors back in the 1990's to switch to a WIN32 linking model. I didn't find their arguments that compelling, to be honest, due to not citing alternatives considered.


Unrelated told your post, but please don't suggest that SCO Group was involved. The company in the 90's was simply SCO, and was a pretty cool company.

SCO Group was renamed as such from Caldera in the early 00's and was the people suing Linux users for copyright infringement.


Dynamic linking was already common outside UNIX (Amiga, Xerox, Atari, MacOS, MS-DOS overlays and extenders, IBM and Unisys mainframes, Oberon/Lilith) so what does Win32 even have to do with it?


There's nothing wrong with taking a principled approach to builds. For example, it's not just a question of static or dynamic. Linking of vendored/static/pinned/etc. sources/artifacts/etc. in general comes with the benefit of hermeticity. That enables tools to work better, since it gives them perfect knowledge of the things that exist. It also entails a certain degree of responsibility for writing your own scripts that can generate automated alerts should issues arise in the upstream sources.


It's really a decision you have to do per-library.

For something like SDL that has a stable ABI while providing user-visible improvements in newer versions, dynamic linking makes sense. SDL's solution isn't only for static linking but also for bundled dynamically loaded SDL copies. I am not really convinced that SDL's approach is perfect though as anyone statically linking SDL can just disable this functionality and this feature has already caused problems requiring you to manually substitute the loaded libSDL.

For libs that don't interact with the system, dynamic linking does not really provide an advantage for binaries that are distributed on their own.


> The total download cost to upgrade all binaries on my system which were affected by CVEs in 2019 is 3.8 GiB. This is reduced to 1.0 GiB if you eliminate glibc.

The upgrade problem has almost nothing to do with download size. The real problem is that you have > 100 binaries which depend on those libraries, and instead of having the library authors go and update the library, you need each team responsible for one or more binaries to go and take the new library and release a new version of their binary.

And then, when you want to check if your system is safe from Heartbleed, instead of checking if you have libopenssl > 1.0.1g, you need to check if bin1 > 1.2.56 or > 0.6.89h, bin2 > 5.76.1, or > 4.6.215,... bin100 > 1.67.89.

And of course, if one of them does NOT have a newer version compiled with the patched library, you need to fix it yourself, and maintain a patched version of the binary. Assuming that you even know that binary had been linked to the vulnerable library.


The real problem is that you have > 100 binaries which depend on those libraries, and instead of having the library authors go and update the library, you need each team responsible for one or more binaries to go and take the new library and release a new version of their binary.

This is a solvable problem. Package managers such as Nix and Guix rebuild packages if any of their transitive dependencies have been changed.

The difficult part are now language-specific ecosystems that use lock files to lock all their dependencies. Traditional C/C++ programs, either statically linked or dynamically linked, have a dependency graph such that e.g. each program that uses OpenSSL has the same package definition in their transitive dependencies. However, e.g. Rust programs may have different versions of the same crate locked.

(There are solutions to that, but there is still work to be done.)


I was talking about finding vulnerabilities on end-user systems and servers. I don't know about others, but I don't generally keep compiler tool chains for C, C++, Java, Go and maybe 1-2 others on my systems and on my servers in case I need to rebuild all of my binaries.


I was talking about finding vulnerabilities on end-user systems and servers. I don't know about others, but I don't generally keep compiler tool chains [...]

You don't have to, most NixOS/Guix systems use binary caches. So, their build clusters do the work for the packages included the nixpkgs/guix package sets. If your organization builds their own packages in top of that, you can use a CI plus your private cache (or something like Cachix).


> You don't have to, most NixOS/Guix systems use binary caches.

Yes, Nix/Guix and Spack are up to my knowledge the only systems that got that right. The centralised recipe repository (and their functional nature) make scratch recompilation reliable and easy.

Now good luck to get that with most lock-file based package managers like npm, cargo or pip with ~1000 packages in your dependency tree that hardcode their dependency number...

The distributed approach of some package manager often come with a security cost unfortunately. And that's a problem for static linking.


> Now good luck to get that with most lock-file based package managers like npm, cargo or pip with ~1000 packages in your dependency tree that hardcode their dependency number...

I don't know about spack or Guix, but nixpkgs has buildRustCrate, which builds each Rust crate dependency as a Nix derivation (it does not use Cargo). In this kind of setup it is possible to override specific crate versions across all packages that use a specific crate.

Unfortunately, currently most Rust-based packages in nixpkgs use buildRustPackage, which does not follow this approach [1].

[1] https://github.com/NixOS/nixpkgs/issues/89563


> instead of having the library authors go and update the library, you need each team responsible for one or more binaries to go and take the new library and release a new version of their binary.

Wait, so you're telling me dynamic libraries are a way for distros to ship their org chart?

Maybe it's time in 2020 for automated and deterministic builds to allow anyone to ship a library update and have dependent packages being rebuilt automatically if needed.


> Wait, so you're telling me dynamic libraries are a way for distros to ship their org chart?

I'm not sure what you mean by this. The libraries are not maintained by the distro, they are maintained by the library maintainers; the distro just distributes them when new versions are available, after testing.

> Maybe it's time in 2020 for automated and deterministic builds to allow anyone to ship a library update and have dependent packages being rebuilt automatically if needed.

This would be nice, but it is not there yet for most languages/pacakges; and you still need to solve the problem of who will actually run those automatic builds.


Say we have program X with dependency Y. X+Y is either dynamic or static. X can either have responsive maintainers or unresponsive maintainers. Y can either change to fix a bug or change to add a bug. (With Heartbleed, I remember our server was fine because we were on some ancient version of OpenSSL.) Here are the scenarios:

- dynamic responsive remove bug: Positive/neutral. Team X would have done it anyway.

- dynamic unresponsive remove bug: Positive.

- dynamic responsive add bug: Negative. Team X will see the bug but only be able to passively warn users not to use Y version whatever.

- dynamic unresponsive add bug: Negative. Users will be impacted and have to get Y to fix the error.

- static responsive remove bug: Positive/neutral: Team X will incorporate the change from Y, although possibly somewhat slower (but safer).

- static unresponsive remove bug: Negative. Users will have to fork X or goad them into incorportating the fix.

- static responsive add bug: Positive. Users will not get the bad version of Y.

- static unresponsive add bug: Positive. Users will not get the bad version of Y.

Overall, dynamic is positive 1, neutral 1, negative 2, and static is positive 2, neutral 1, negative 1. Unless you can rule out Y adding bugs, static makes more sense. Dynamic is best if "unresponsive remove bug" is likely, but if X is unresponsive, maybe you should just leave X anyway.


If you’re a Linux distribution and you have the source code to the software you distribute then the responsiveness of the maintainers doesn’t matter so much: any change to a library which is compatible with the previous version (in the sense that you could dynamically link to the new version instead of the old version) is going to be compatible under static linking too.


That's a really good point that I hadn't considered. All of the software I'm pulling from Pacman is being compiled from source, the authors aren't uploading binaries.

So the difference in effort between changing the static library and changing the dynamic library is... maybe not nothing, but not nearly as high as I was assuming.


> I remember our server was fine because we were on some ancient version of OpenSSL.

It's a side point, but I don't entirely understand the argument here since it's basically guaranteed you were vulnerable to a variety of other bugs due to not updating. It depends how old your OpenSSL version was I suppose, but still, it kinda feels like gloating you survived a hurricane by doing no preparation - I'm happy for you, but that doesn't necessarily make it a great idea ;)

> - dynamic responsive remove bug: Positive/neutral. Team X would have done it anyway. > - static responsive remove bug: Positive/neutral: Team X will incorporate the change from Y, although possibly somewhat slower (but safer).

You sure about that? That's basically your whole argument, and I really don't buy it. In my experience, any dependency (static or not) shipped with a program is rarely actively updated, and also never in any regular fashion - on the list of things to do it's typically near the bottom. You called this a neutral, but it's easily a positive for dynamic (and negative for static), and IMO is the biggest argument for dynamic linking and shipping libraries separately. And once you apply that change, it's just two Positives and two Negatives each.

But with that, there's another situation that you're ignoring which throws a big wrench into your table - you're assuming you already know what the dependencies of a program are, when in practice you usually don't. In a dynamically linked world, to address a bug in OpenSSL I just drop a fixed version of OpenSSL in `/lib` and reboot. If it's statically linked to some or all of my programs instead, I need to figure out which if any programs make use of a statically linked version of `OpenSSL` and either wait for a new version or attempt to recompile them with a new one. And if I miss one, then the bug is still there in some form.

Edit: It's a small point, but you're also assuming that the maintainers shipping the static library will catch every bugged version before shipping their program with it - and library maintainers typically try not to release versions with known unusable bugs, it's an accident. An issue like Heartbleed is always going to be found after it's already in the wild, making the `static responsive add bug` category a bit suspect. If a program maintainer was on top of their OpenSSL version and always shipped the latest version when they released, their users would have been vulnerable - they may have made a release in a few days with a new version (to go with the other category), but their users still got the bad OpenSSL version.


> I remember our server was fine because we were on some ancient version of OpenSSL.

It's a side point, but I don't entirely understand the argument here since it's basically guaranteed you were vulnerable to a variety of other bugs due to not updating. It depends how old your OpenSSL version was I suppose, but still, it kinda feels like gloating you survived a hurricane by doing no preparation - I'm happy for you, but that doesn't necessarily make it a great idea ;)

At the time, I think OpenSSL had several trees that were supported, Heartbleed was in the 1.0.1 tree, but not earlier trees, 1.0.0 and 0.9.8 trees were both still supported. OpenSSL versioning isn't very conventional, so one might not realize that 1.0.1 is a major version difference from 1.0.0, and 1.0.0 is a major version different than 0.9.8, but that's how they roll, minor versions within a series get letters after the name, and sometimes break binary compatibility (ugh). There's a reason a lot of people were slow to update to 1.0.1; unfortunately for me, my company had just updated to 1.0.1 about a month before the bug was reported, so we could get TLS 1.2 support. Would have saved a lot of headache if we had waited :(


You don't necessarily have to give equal weights.

Plus the whole reasoning tends to imply old versions are overall better (you don't have just one bug added or removed from time to time, you have to consider the overall bugginess over time, and if the project is well maintained, it will hopefully eventually decrease, especially if your usage scope remains constant)

But the reality is simply: it depends. On the libraries. On the application. On the platform. On the languages. On the tooling. On the maturity of all of that.

And even so, you may want (or be forced) to take an hybrid path (classic example: a program for Windows -- static is reasonable, but not for platform libs, and actually just not even available in this case)


I don't agree with this:

- static responsive add bug: Positive. Users will not get the bad version of Y.

On the contrary, it's negative because the maintainers of X will update Y and introduce the bug to their users (I'm assuming they don't thoroughly audit the source of Y each time they do a version upgrade).

Also, "unresponsive remove bug" should be given more weight since it's more likely to happen IMO.


Sure, but I think the most important reason people have in mind when they argue for dynamic linking is that they will receive upstream bugfixes. I don't think your eight cases are equally important.

You are of course right, though, and writing it down like this can be useful.


> Sure, but I think the most important reason people have in mind when they argue for dynamic linking is that they will receive upstream bugfixes.

Ironic considering that the package manager/repo model is largely implemented as a solution to issues with dynamic linking and yet makes getting up to date software from the developer much harder by introducing a third-party maintainer into the process.


Why would you not receive upstream bug fixes with statically linked programs? Assuming you are using an Apt-style package manager then the statically linked program would be rebuilt and updated too.

If you are not using an apt-style package manager then the program must include all of its dependencies (except ones that are guaranteed to be present on the platform, which is none on Linux and a few on Mac/Windows), and you will receive bug fixes when that program is updated, whether or not it uses static linking.

Static/dynamic linking does not affect how likely you are to get bug fixes in any way as far as I can tell.


The first hurdle is that you require source code to do any of this. Then you need to actually rebuild everything.


Apt-style package systems do have source for everything and rebuild everything.


Do any Linux/glibc or Linux/musl systems support static PIE binaries, yet? Without static PIE support you don't benefit from ASLR (at least not fully). This 2018 article seems like a good breakdown of the issues: https://www.leviathansecurity.com/blog/aslr-protection-for-s...

OpenBSD has supported static PIE since 2015; not just supported, but all system static binaries (e.g. /bin and /sbin) are built as static PIEs, and I believe PIE is the default behavior when using `cc -static`. The reasons for the switch and the required toolchain changes are summarized in this presentation: https://www.openbsd.org/papers/asiabsdcon2015-pie-slides.pdf

Also, simply checking the "static PIE" box isn't the end of the story. There are different ways to accomplish it, and some are better than others in terms of ASLR, W^X, and other exploit mitigations. It's been a couple of years since I last looked into and had a hold on all the issues simultaneously[1], but the basic takeaway is that dynamic linking in system toolchains and system runtimes is far more mature than static linking.

[1] static PIE issues are a nexus of exploit mitigation techniques, so if you want to deep dive into exploit mitigation or even just linking issues then chasing the static PIE rabbit is a good approach.


Most Linux/musl systems support it. On Alpine, gcc is built with `--enable-default-pie` so all static libraries can be linked into a static PIE.

On vanilla gcc (since version 8 when static PIE was upstreamed), `-static` means non-PIE static executable and there is a separate flag for `-static-pie` for static PIE. Alpine patches gcc so that `-static` and `-pie` are independent flags, so both `cc -static` and `cc -static-pie` will produce a static PIE.


I'm not sure I understand the first link you posted: if a binary is statically linked, why does it need a GOT? It's literally calling functions in its own binary…


But where in its own binary? To support ASLR of code (a ROP mitigation), especially fine-grain randomization, function call sites can't use static addressing, either absolute or relative, pointing directly to a function. To support ASLR of code you can either rewrite every function call site on load (kinda similar to DLLs on Windows), or use one or more tables that are updated at runtime (PLTs, GOT, etc). BSD and Linux environments use the PLT/GOT approach because 1) it's a better fit for ELF, 2) shares most of the preexisting instrumentation with dynamic linking, and 3) preserves the ability to share most of the mapped pages of static binaries (imagine if a static /bin/sh was effectively rewritten on every invocation). Trying to accomplish this safely and effectively is what RELRO is about. See https://www.redhat.com/en/blog/hardening-elf-binaries-using-... But RELRO poses some dilemmas regarding performance, and depending on your choices as well as the details of the implementation, RELRO might be ineffective or, worse, introduce exploits of its own. See https://www.usenix.org/system/files/conference/usenixsecurit...

It's difficult to achieve a design that provides all the desirable exploit mitigations without sacrificing startup latency and other features. OpenBSD added a new syscall, kbind (https://man.openbsd.org/kbind), so that they can have lazy binding without being susceptible to the RELRO exploits mentioned in that Usenix paper. (Unfortunately, the 2018 leviathansecurity.com article fails to mention kbind, even though kbind was added to OpenBSD in 2015.) There are other approaches. I think the PaX Team has written quite alot about their preferred techniques. But the point is that static PIE, which is desirable because of ASLR and other reasons (e.g. unification of code generation techniques), touches upon varying and distant components of toolchains and runtimes.


I know how ASLR/ROP/PLT/GOT/RELRO work, but I'm still not understanding what's going on here. When you statically link a binary, you get…one file. And that gets loaded into memory together, and to make it PIE you have all the jumps be pc-relative. Like, a straight up jump instruction to a fixed offset. So where the the room for a table like this?

(Unrelated, but since you brought it up: kbind is IMO not a very good mitigation. It seems that all you have to do to bypass it is leak a cookie and ROP to that one place in ld.so that is "blessed" and then you not only have the ability to scribble all over your read-only GOT but as far as I can tell you can overwrite any read-only memory, which means it opens up an extremely valuable exploit primitive…)


When I participated in NaNoGenMo 2015 [1], I ended up having to emulate a very small subset of MS-DOS [2]. That meant loading in an MS-DOS `.EXE` file into memory and running it. I had a 32-bit Linux system, so I was able to use the `vm86()` system call to run the 16-bit code natively. The `.EXE` file had basically two sections---the first the actual binary code (and data and what have you). This was followed by another section that contained offsets into the code that was loaded that needed to be updated with the proper address, and it took all of five lines of code to implement [3]. Once used, that segment can be discarded. You don't need position independent code to implement address space layout randomization (it makes it easier), you just need a table to rework the absolute addresses (downside---it may take some time).

[1] National Novel Generation Month

[2] https://github.com/spc476/NaNoGenMo-2015, specifically, https://github.com/spc476/NaNoGenMo-2015/blob/master/C/msdos...

[3] lines 337 to 343 of `msdos.c` [2]


This isn’t my area of expertise, but I think on Windows relocations are actually handled quite similarly (as a table of locations that the loader uses to patch with).


Relocation by patching code means you no can no longer share code sections between programs.

Turns out I was the confused one, not you. I thought it was at least possible to generate binaries with multiple segments, each with their own PLT, GOT, and .text sections, similar to putting .text and .data sections in separate segments. That way the relative position of all functions wouldn't be static. But it seems there's no way to generate binaries like this using the standard tools.


> but the basic takeaway is that dynamic linking in system toolchains and system runtimes is far more mature than static linking.

But that's a chicken-or-egg problem, right? The only way static linking tools will reach parity with dynamic ones in terms of maturity is if static linking replaces dynamic linking as the de facto method of choice.


Generally speaking static compilation has had 30 years of head start across multiple OS and hardware architectures, until dynamic compilation was even an option on mainstream computing, it was after all the only way, with dynamic only available in research platforms like what was going on at Xerox PARC.

So this widespread lack of knowledge that static is actually more mature than dynamic is kind of ironic.


Nixpkgs / NixOS does.


I imagine there's some way to, because the kernel is able to apply ASLR to the dynamic linker itself.


It's not a matter of possible, but whether the work has been done throughout the toolchain and runtime stacks.

When OpenBSD implemented static PIE, -static and -pie, the literal options and the code generation aspects more generally, were still mutually exclusive in GCC and clang. GCC required patching to enable both static linking and PIE generation. In fact, my local GCC 9.3 man page still says that -static overrides -pie; to build static PIE seems to require the special option -static-pie, in addition to -fPIE/-fPIC when building the object code. But it's not enough for the compiler and compile time linker to support it. libc and libstdc++/libc++ also need support, both internally as well as in the built libraries. And I think libdl might need support if you want dlopen to work from a static PIE binary. Likewise for libpthread. Supporting static PIE requires the cooperation of many moving parts. And that's just to support it. Making it easy, let alone the default behavior, so that it doesn't require many carefully coordinated, obscure flags without the risk of any misstep silently disabling some exploit mitigation, is yet another story.

It can be done. It should be done. But to what extent has it been done? I'm not sure, though some quick Googling suggests GCC and clang are mostly there, at least in terms of nominal support (which, again, is distinct from fully supporting all the same mitigation measures). glibc seems to have gotten some support (e.g. --enable-static-pie in glibc's build), though I'm not sure whether it's available in distros. And I would guess that musl libc support is pretty far along, and presumably more mature than glibc's given that Rich Felker started experimenting with static PIE several years ago, shortly after OpenBSD did their work. See https://www.openwall.com/lists/musl/2015/06/01/12


For sure. My comment was directed at Linux the kernel rather than Linux the ecosystem, but it's a totally valid point that for practical purposes that distinction can border on meaningless.


First we claim that dynamic linking does not provide any memory savings because the libc we used is small, and later on we use our lack of dynamic linking to justify having a small libc. Smart, very smart.


That's what happens when you decouple components and make decisions for each component separately rather than holistically for the system as a whole.


Dynamic linking provides encapsulation and security benefits. An application with a statically linked OpenSSL can be vulnerable if a CVE comes out for that version of OpenSSL, whereas a dynamically linked OpenSSL could be patched immediately without recompilation (which may be impossible if the software is proprietary). The vendor of the shared library can update the implementation without requiring all downstream consumers to recompile. This should not be downplayed.

Pure static linking makes sense when you are deploying code you control onto an environment you control.

While I like ease of deploying Go, where binaries are one big blob that just works, it makes it closer to the Java style than the UNIX modular-tool-does-one-thing-well tradition.


From distributions where you build everything from source (most cloud providers today, mobile phone operators, etc), such advantages are purely theoretical & never play out in practice. In such environments you're pushing out updates of the entire system on whatever your regular cadence is. Additionally, because the old library is still mapped & running you have to know to restart all the processes that have the old version linked in. Not an easy task.

About the only place it kind of matters is Linux distros where you are deploying closed source binaries linked against system libraries. I'm not sure that's a significant use-case.

The only performance benefit is memory. If you have a core infrastructure library that's widely shared (e.g. libopenssl) then it can have some benefit. In practice I much prefer the microservice model with a formal IPC API. Then the SW update is trivial to fix the exploit - just kill the 1 process providing the service.


Where are those mobile phone operators building everything from scratch?

Because from my telecommunications and mobile OS development knowledge I have hard time remembering at least one.

And binary deployments? They are done all the time.


When I said mobile phone operators, I meant more Apple and Google, not the telecom operators. I haven't worked at telecom operators but I wouldn't be surprised if that world is wildly different - IT is generally considered a cost center rather than as way to remove costs in other parts of the org.

Apple, Google, Microsoft, Amazon, Facebook all build everything from source. Source: I worked at 3 of those & have friends coworkers at the rest. For cloud users I don't have as good a knowledge of that space. I imagine the majority of them use off-the-shelf prebuilt libraries that come with the OS they run on.


Suppose 100 processes on the same box call the same function in glibc. If each process had its own redundant version of it mapped into memory, then that's 100x more cache misses compared to them all sharing the same page.

I get that only a handful of libraries are actually used, but that's a very fat tail and the performance will be very bad if that handful of libraries are always statically linked. This may be less important if the program is in a container (isn't sharing anyway) or launched as an AWS Lambda throwaway. I dunno, maybe dynamic linking isn't relevant anymore now that most Linux programs are meant to run in containers on an expensive cloud with tons of memory.

> In practice I much prefer the microservice model with a formal IPC API.

This converts a simple and solved problem, sharing memory, to a client-server distributed systems problem. There is no need. It's all on the same box.


I would highly recommend you shift your thinking & consider even something as basic as threads as a distributed systems problem. The distributed systems field has a much richer successful history of how to design robust systems at scale (not just in terms of compute resources but also in terms of # of people working on the problem & number of components). Sharing memory is actually not a simple & solved problem but distributed systems problem generally have lots of formal methods & tools for dealing with unreliability of any given component.

That's also ignoring that measuring the distributed cost of a component is fundamentally simpler than measuring it for a library mapped into 50 million processes.

Shared libraries can be useful but their value is often vastly overstated.


Back in the 90s we'd statically link the most frequently executed programs on busy servers for a significant performance boost.

Dynamic linking is not a performance feature, it's a decoupling feature.


It is a performance feature if you are memory constrained. Shared libraries are “shared” for a reason. On a server with high multi-tenancy the savings can be significant.


Not really. Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems.

As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything.


Not really.

Yes, really.

Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems.

Definitely not true. Every bit of memory that's not available that could be shared memory instead is a reduction in memory available for filesystem caches, etc.

As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything.

Cloud vendor computing models are not generally the computing model of the rest of the world. Comparison to their environment is not relevant to the general populace.

I worked for a "big iron" OEM vendor until late 2017 and the savings were definitely still significant then both for their customers and the vendor themselves.

There are numerous benefits to shared linking, that doesn't mean it's always the appropriate solution, but is not correct to claim that there are no performance benefits.

Especially on more memory-constrained consumer devices, the shared memory benefits of dynamic linking are still significantly beneficial.


> Cloud vendor computing models are not generally the computing model of the rest of the world. > Comparison to their environment is not relevant to the general populace.

The way that Google shares server compute and memory resources is by having a service oriented architecture. A single high scale service serves many different applications, usually colocated in the same cluster or data center. Each service is based on multiple instances of a statically linked binary.

At that scale, there is no point in try to use shared dynamically linked libraries to reduce consumption because you save more by either reducing increasing your own app's efficiency or relying on one of the major services rather than linking more functionality into your own app.


I'm aware, which is why I specifically said that cloud vendor computing models aren't really relevant to the general populace. Google's model in particular is (or at least was) very different from other vendors.


I don't disagree with your point that use cases vary, but when I hear people say that "only clouds need this" I think the speaker is underestimating the size of cloud facilities. The amount of the world's computers that are in Amazon's, Google's, Facebook's, and Microsoft's clouds is a huge chunk of the total.


FreeBSD statically links LLVM into Clang faster build times (2020). (The difference is pretty marginal, but it's at least a percent or two faster.) So my /usr/bin/clang is a 70 megabyte file.

The vast majority of the contents are r-x text (code) and r-- read-only data. I don't know if the kernel is capable of sharing read-only paged memory references or not.


> I don't know if the kernel is capable of sharing read-only paged memory references or not.

Any pages mapped from disk should be shared, as a consequence of the FreeBSD unified buffer cache; this is why if you write to a binary in-place with cp instead of unlinking first (like install), currently executing copies are changed, and usually crash :). Read-write program sections will be mapped so their changes are private though, so you get copy on write semantics. Trailing parts that aren't page sized will be copied into a full page, I think.


> this is why if you write to a binary in-place with cp instead of unlinking first (like install), currently executing copies are changed, and usually crash

This is not true, or no longer true: opening a running program's backing file with O_RDWR or O_WRONLY is prevented via the "writecount" mechanism (and vice versa: files being written cannot be executed).


Hmmm does that also apply to dynamic libraries? Now that I'm thinking about the fateful day, I pushed an updated library, not an executable. I'm guessing it was on FreeBSD 8.x, but it could have been 9.x.


Dynamic linking boosted performance for Solaris back around 2004.


A better way to do this analysis would be to build a Linux distribution with everything statically linked and compare to the normal version with dynamic linking, looking at disk space used, startup time, memory used, and time to launch specific applications both cold and hot.


When during Solaris 10 development the "unified process model" was introduced, and static link archives for all system libraries removed, boot times improved dramatically because all the programs that run at boot time were then dynamically-linked and started faster than their previous statically-linked selves because the C library and such were already loaded. Later of course we had even more dramatic boot time improvements via SMF (the Solaris/Illumos predecessor to Linux's systemd).

That was an apples-to-apples comparison of pre- and post-process model unification performance, and it was a win.

Now, this was back in... I want to say 2003 or 2004 -- before S10 shipped. And it's possible that the same experiment today would not have the same result.

I'm not sure how easy it would be to construct a distro with only dynamically-linked executables (at least for core libraries, like the C library) and then the same distro with only statically-linked executables. The S10 work was done by Roger Faulkner (RIP) and it was a huge change.


> A better way to do this analysis would be to build a Linux distribution with everything statically linked [..]

Here you go: Stali

https://dl.suckless.org/htmlout/sta.li/

"Stali distribution smashes assumptions about Linux"

https://www.infoworld.com/article/3048737/stali-distribution...


For a useful comparison you need the static and dynamic distributions to be otherwise the same, i.e. you want to pick a mainstream distribution and build it from scratch with both static and dynamic linking and compare.


I don't know of any mainstream distribution that doesn't make full-static-from-scratch builds gratuitously painful and difficult. Personally I gave up after trying to blunt-force-trauma glibc into linking correctly, though, so someone with more internals knowledge might have better success at it.


Depends on your ideas of "mainstream"; I expect nixos and gentoo are both happy to do such rebuilds for you. But, as you note, the real pain is that glibc really doesn't want you to do static builds... I wonder how gentoo and/or nixos support is for musl...


You going to have to disable PAM which basically limits the the stuff you can install. Also qt and Mesa are designed to be static libs so expect token stuff. I have some ideas to solve thisfor long time but any solution is going by in from multiple places.


>Do your installed programs share dynamic libraries?

>Findings: not really

>Over half of your libraries are used by fewer than 0.1% of your executables.

Findings: Yes, lots, but mostly the most common ones. Dynamically linking against something in the long tail is pretty pointless though.


> Dynamically linking against something in the long tail is pretty pointless though.

I disagree. Dynamic linking, in the context of an OS which offers a curated list of packages in the form of an official package repository, means that a specialized third party is able to maintain a subcomponent of your system.

This means you and me and countless others are able to reap the benefit of bugfixes and security fixes provided by a third-party without being actively engaged in the process.

In the context of an OS where the DLL hell problem hasn't been addressed and all software packages are forced to ship all their libraries that are shared with no one at all, indeed its pretty pointless.


> Dynamic linking, in the context of an OS which offers a curated list of packages in the form of an official package repository, means that a specialized third party is able to maintain a subcomponent of your system.

You nailed it, in my opinion. The biggest reason I favor dynamic linking is not because of any inherent advantage that I'm determined to believe in (in the face of purported evidence to the contrary, like this article), but because I fear the ecosystem changes that a major shift towards static linking would allow.

If everything is a static blob, you have an environment that is much more friendly to every dev shipping their app as a binary download on their website, like Windows "freeware". Or worse still, they only support AppImage or some other "modern" method of distribution. This cuts maintainers out of the loop. I want to continue using a distribution with maintainers. I think they solve some important problems: when a maintainer is the gatekeeper, it means that a dev has to convince a third party that their app is (a) worth including, (b) not malware, (c) open source - at least heavily preferred since the dev will be building it themselves. And plus having a maintainer means someone besides the original dev is responsible for making sure you get security updates.

Now of course you can distribute statically built apps that way, but there's a reason it's less common. There are so many Go apps out there where the only supported method to build them is using the Go toolchain and pulling in 150 Github repos.

See also: this defense of the maintainer-based ecosystem, from an Arch Linux dev. http://kmkeen.com/maintainers-matter/


> If everything is a static blob, you have an environment that is much more friendly to every dev shipping their app as a binary download on their website, like Windows "freeware". Or worse still, they only support AppImage or some other "modern" method of distribution. This cuts maintainers out of the loop. I want to continue using a distribution with maintainers.

So because that's what you want, let's make what other people want harder.

Personally, I can't stand the repo/maintainer model. I want to cut maintainers out of the loop! I like having a direct relationship with the developer of the software I use. If they update their software with a feature or bug fix I need, I want the update right now, not when some unpaid third party gets around to integrating it with the f'ing distro. When I report bugs to the developer, I don't want to have to involve the third party.

Shit like this is why I stick to Windows.


> So because that's what you want, let's make what other people want harder.

No? I haven't said it should be harder to statically link software. I just said I want to discourage it. I want to advocate that my Linux community resist these kind of ecosystem changes that would, in my opinion, harm the free software community.

> Shit like this is why I stick to Windows.

Seems like telling on yourself. Compare the results: the software available on Linux distributions vs. the Windows freeware market.

> I like having a direct relationship with the developer of the software I use.

And what if that developer isn't trustworthy in some way? (See the Arch Linux dev's post that I linked.)

> If they update their software with a feature or bug fix I need, I want the update right now

Just one example, but I've sometimes gotten updates for Firefox on Arch Linux before the official binaries got released. The maintainers seem to be on top of their game. And never mind the fact that if you're counting on automatic updates, you're assuming that they roll out to everyone simultaneously (often not true) and that automatic updating is always desirable anyway. (Plus a lot of Windows software doesn't update itself at all, so...)


> Seems like telling on yourself. Compare the results: the software available on Linux distributions vs. the Windows freeware market.

Obvious choice? WINE wasn't invented because Linux had a great software library.

> And what if that developer isn't trustworthy in some way?

Then you probably shouldn't use their software at all. But that isn't necessarily realistic, which is why I also advocate for sandboxed applications by default, something mobile got right.

> Just one example, but I've sometimes gotten updates for Firefox on Arch Linux before the official binaries got released.

Sure, maybe that happens sometimes for big and popular projects, but there are a lot of more niche projects where what is in the repo is years out of date. Hell, I have to get qbittorrent-nox from a PPA to be up to date and that isn't even very niche.


> Shit like this is why I stick to Windows.

So you can only use software in the Microsoft Store?


It can also cut the other way though. Bugs can be introduced, compatibility can be broken, users can not find the library in their package manager, or they may find too new of a version. The danger of this is smaller for popular libraries, but goes up as you move to the long tail.


But this also highlights the benefit of community packaging. Debian packagers often backport security fixes into older versions of libraries that are no longer maintained upstream. That's a big part of their job--not just to bang out a build and walk away, but to keep an eye on things. This is why it's important to only use distro-packaged libraries as much as you can, even when statically linking.

Getting off the treadmill of integrating interface-breaking upstream changes is one of the biggest practical reasons people prefer static linking and directly adding upstream source repositories into their build. It's at least as important, IME, as being able to use newer versions of libraries unavailable in an LTS distro. It can work well for large organizations, such as Google with their monolithic build, because they can and often do substitute the army of open source packers with their own army of people to curate and backport upstream changes. For everybody else it's quite risky, and if containerization provides any measure we're definitely worse off given the staleness problems with even the most popular containers.[1]

[1] I wouldn't be surprised if an open source project emerged to provide regularly rebuilt and possibly patched upstream containers, recapitulating the organizational evolution of the traditional FOSS distribution ecosystem.


Bugs & compatibility issues are still a problem with the statically-linked version, unless you want to stay on the version of the library with the security vulnerability, you have to upgrade. That means, for either static or dynamic, dealing with bugs and compatibility issues — which I'd argue is another form of bug; if you're practicing semantic versioning (which you should be, as it prevents exactly this issue), this indicates either someone accidentally broke compatibility (a bug in the library), or someone was relying on something outside the established API (a bug in the consumer). For major versions (i.e., where compatibility is intentionally broken), good package managers are able to manage side-by-side installation of well-behaved packaged. (E.g., Portage's "slots" concept.) I'd also mention Nix's design here; my understanding is that it allows you to upgrade the dependency for just some consumers, so you really can then choose between insecure & bugs.


Why stop there? Package every binary in its own container, too! Most programs only use a few system calls, so you're not really getting anything by sharing a single kernel for the entire system.


That's literally what RancherOS does. ls is a container. Blew my mind the first time I saw it. But they take the "cattle not pets" approach pretty much to its logical extreme (I suspect that the company's name arises from this metaphor)

It makes ops so liquid and convenient. I love it.

To really push this idea (and microkernel) to its logical extreme, it would be cool to see a null-kernel. You have socket drivers, that's it. Everything is either a network call or some sort of IPC. You might still need something to handle paging, though, and obviously your "peripherals" like storage would need to be more conventional in nature.


It makes sense from security point of view, but it definitely is much more resource intensive.

Imagine something like InteliJ or Eclipse, where every single IDE plugin is its own process doing IPC.


All containers on a single host very much share the same kernel. You're confused about virtualization vs containers.

Seems to be the general trend. A virtualized kernel per service.


Static executables are a sweet spot between the craziness of dynamic dependencies (where we come from) and the idiocy of shipping a whole virtual machine for each program (where we seem to be going to). If we could just stop at that spot!


I kindof agree, but I think the real sweet spot is that system libraries[0] should be dynamically linked and everything else should be static.

For Linux though, where the very concept of a standardized base system is loathed, static the sweet spot.

[0] libraries that are likely to be used by a large number of applications like the GUI libraries, kernel interfaces, networking, encryption, etc.


This is a crazy conversation! Dynamic linking allows the system to decide the UI. Static linking means that the UI cannot evolve.

Imagine statically linking UIKit or Android's UI library!


UI libraries tend to evolve by introducing new APIs or modes, because changing UI implementations has a habit of breaking apps.

For example, on Android even the switch to hw accelerated rendering was a mode app devs had to opt in to, that didn't even change the look!


Yeah but then you have CFLinkedOnOrAfter to negate that ;)


I believe the original reasoning for Dynamic Linking wasn't performance gains, but security gains -- someone described the driving story to me as essentially a found vulnerability in a very common library required updating and re-compiling everything on every system, scarring sysadmins globally and permanently; the space saving and performance aspects came up as later "bonuses".

I have little memory of the details of the story, and I'm not 100% sure it's true, but it's a much more satisfying and reasonable argument for dynamic linking than performance/space.

Of course, the more modern solution would probably be a good package manager -- if its trivial to recompile things, and track what needs to be recompiled, then dynamic linking seems to gain little, but bring in a lot of its own headaches (as we know today)


The original reasoning for shared libraries was reducing memory footprint -- back then memory was scarce. We're talking back in the days of SVR2, the mid-80s.

The original reasoning for ELF was that static link semantics suck. ELF's semantics are far superior (https://news.ycombinator.com/item?id=23656173).


Stallman and other GNU folks have claimed that they stuck with dynamic linking to foster cooperation between hackers. They knew that it wasn't a win performance-wise but they wanted to encourage hackers to help each other. The idea was that if someone found a bug in a library he or she depended on they would be "forced" to send the patch upstream rather than just fixing the bug in their local copy of the library. Thus they made dynamic linking the default in gcc and kept static linking as something of an after though.

I read it on a mailing list a long time ago so I don't have a source.


Don't forget that the principles of the LGPL (originally the "library GPL") were/are pretty much tied to the technical concept of a shared library object that an end-user can change, etc; IANAL so I can't say anything about the legal soundness of this interpretation.

Moreover, in a recent discussion about this topic on HN, someone said the introduction of shared libs into the Linux user space was mainly in support of porting X Windows to Linux (supposedly because of binary video drivers or to accomodate MIT-licensed code?), but I haven't found any supporting reference for that.

glibc's maintainer Ulrich Drepper also has pretty strong opinions on static linking [1]; no matter what you think about this technically, or Ulrich personally, his paper "How to write shared libraries" [2] is considered reference material on the subject.

Personally, I think that the over-use of shared libs in the Linux userland clearly serves no purpose if users flock to entire new layers of abstractions (eg Docker-like containers) to isolate their app delivery from the IMHO overengineered mechanisms in ELF and ld.so with their multiple RUNPATHs, configs, loader scripts, and versioned glibc symbols (on top of build-time libtool/autootols) that still doesn't seem to get to the point. While the LSB effort for more uniform Linux distros isn't dead, it doesn't seem to be taken seriously. Idk, but maybe the GNU folks also see the lack of binary compat for Linux apps as a desideratum, to frustrate any and all attempts to ship binary apps?

[1]: https://web.archive.org/web/20100527213559/http://people.red...

[2]: https://akkadia.org/drepper/dsohowto.pdf


Without shared libraries, the only way to do plugins is via IPC.

Sure it is more secure and probably preferable in modern times, but it also slower and requires more hardware resources, specially when one scales it with desktop software running hundreds of processes, each for their own plugin sets.

There is no free lunch as they say.


> Without shared libraries, the only way to do plugins is via IPC.

That's primitive superstition:

- dlopen() works in statically linked programs too.

- Even without dlopen(), you can load code dynamically. It's not magic, especially if the plugin is linked statically.

- Besides, how is IPC bad? I take fcgi over Apache's modules any day.


dlopen is useless in static linked programs when the idea is to provide plugins after the fact.

The idea of using plugins is exactly that various parts are able to ship them at various times during the lifetime of the applications.

Patching files compiled statically is obviously not what one wants from plugins.

Incidently plugins support is exactly why Go added support for dynamic linking into their toolchain.


One of us is confused; I honestly don't know which one.

Where is the problem in this scenario? My application (statically) linked provides a plugin interface. It works by calling dlopen() on the plugin and then passes a record of functions to the entry points. The plugin can be provided by a third party, and top make that easier, I provide an SDK (a header file).

No file is patched, and I have no idea what this has to do with Go. Enlighten me?


Your solution requires compiling everything from scratch in a single executable, that is not how plugins are supposed to be designed.

We have been there before with solutions like graphics drivers for Borland's BGI library for their MS-DOS compilers. Where a driver interface is provided, and then each "plugin" registers themselves on application startup.

Adding new plugins to an existing application in this scenario requires recompilation and updating the uses/#include being used for plugins, or patch the executable from an existing .obj file.

If you are using dlopen on a third party binary instead of your own, then you are already using dynamic linking by definition, by loading third party code dynamically and revolving the proper address locations for all symbols.

Go only had static compilation on the beginning as the only true way, but it failed short exactly in this scenario, to the point that eventually plugin package came to be and dynamic compilation support is now a feature of Go's toolchain as well, although many seem to still not be aware that Go compilers can also produced dynamic libraries.


> then you are already using dynamic linking by definition

"Well, technically..."

I proposed using the dlopen() mechanism (or a custom linker) from a program that is itself statically linked. Because that's what the argument is about: statically linking the stuff you always need. Now, technically, that's dynamic linking. And because I'm linking the plugins dynamically, I might as well link everything dynamically, right? No, absolutely not. Because the latter gives me DLL hell, and rpaths, and library maintainers who think that changing LD_LIBRARY_PATH is perfectly sensible. The former doesn't. Not equivalent. Not at all.

We're not talking about BGI or overlays or Go. We're talking about statically linked binaries calling the dynamic linker. You didn't actually know that was possible, did you?


Actually that was just how UNIX dynamic linking started out, by using models that patched a.out, and I still have the Slackware CD with the first Linux kernel that was capable of loading proper ELF files instead.

So my knowledge how these things work goes quite back in time.


That may all be true, but I guess GP was asking what that has to do with statically linking your main binary (or not), which of course you can do, yet still use shared libs as plugin loading mechanism. For example, ODBC drivers (with or without the various unixODBC/iODBC proxies) can be dlopen()ed from otherwise statically compiled main binaries just fine, can't they (even though for ODBC the typical use case is using a driver manager rather than invoking dlopen() directly)?


Sure, but that raises the question why bother with static linking if dlopen if being used anyway.

With dlopen the application can crash in very interesting ways, if it doesn't handle loading the errors in a proper way.

And there is no way to use something like dumpbin or ldd to actually find out what it is looking for, even something like strings might not help, because the path to the file being dynamically loaded in an explicit way might itself be dynamically constructed at runtime.


> Will security vulnerabilities in libraries that have been statically linked cause large or unmanagable updates?

This is maybe not an issue for open source packages which are managed by your distribution package manager, assuming they update all the dependent packages once some library gets updated (which would lead to a lot more updates all the time).

However, the maybe more critical issue is about other independently installed software, or maybe closed source software, where you will not automatically get an update once some library gets updated.


> However, the maybe more critical issue is about other independently installed software, or maybe closed source software, where you will not automatically get an update once some library gets updated.

If you care about security, you shouldn't be running closed source software anyway, at least not outside of a container.

Statically linked binaries for open source software, containers for everything else, and you're good to go.


I think the elephant in the room here is things like Flatpack and Snap, which are basically ugly bastardized versions of static linking.


Ugly maybe, but they're do kinda combine the advantages of both. They have the portability and reproducibility advantages of static linking, while still letting you take them apart and change the dependencies like with dynamic linking if you really want to.


And the disadvantages of both. These being the huge loading times (even for a calculator!) and the disk space required.

I think a more interesting analysis of "security vulnerability costs for static linking" would look not at just "how many bytes does the end user download" but "what are the overall costs to the distro to support a fully statically linked setup", looking at eg CPU costs of doing the rebuild or how much total-elapsed-time it would take to do a full rebuild of every affected package.


Not to mention that fixing a security vulnerability in, say, libm or libc becomes an amount of work equivalent to a distribution upgrade with all the associated risks.

Or am I the only one who has occasional problems when replacing all the binaries on my system?


That sounds like a separate issue; normally, having to do a full dist upgrade means that you changed versions of lots of core components (kernel, init, libc, gcc), which is indeed traumatic. If you replace every binary but the only change is bumping the statically-linked libc from x.y.0 to x.y.1, it should be just as boring as making the same change with a dynamically-linked libc.


This is an interesting analysis. For my part (anecdata), in my /usr/bin (Debian) I have 2,956 files under 9MiB in size, 1 of 13MiB and one of 39MiB. Most of the files are (much) under 1.0MiB.

On the other hand, I have a three statically linked binaries for CloudFoundry in my home directory. One for Linux, one for MacOS and one for Windows. They are each between 24MiB and 27MiB each.


Suckless has a project to get a fully static compiled Linux environment. Unfortunately I don't know how far that have come


I don't think stali has seen any activity in several years.

As far as I know, the only completely statically linked Linux distribution that is actively developed is my own project (inspired by stali), oasis: https://github.com/oasislinux/oasis


Do you static-link Linux? :-)


I'm not sure what you mean here. Are you asking whether I build my kernel drivers as modules or built-in? Personally, I build my kernels without modules, but I've never heard of that technique being called "static-linking Linux".


Yes, exactly. Loadable modules are a form of dynamic linking. I've never heard it called "static-linking Linux," either, but I think static / dynamic linking is well understood, and I'm happy you understood the meaning from context.



The thing that's missing is static link semantics that don't suck.


So let's fix the problem! How can we get better semantics? Have any hypothetical solutions in mind?


I've outlined it already, but let me do it again for you:

1) when making a static link archive (a .a file), include a .o with a static symbol(s) whose values(s) records the metadata that ELF would have recorded, which here is: a) dependencies, b) how to find them, c) the mapfile / version-script (at least which symbols are symbolic, protected, or demoted to local, and which, if any, are interposers.

2) on final link-edits only the direct dependency -lfoo arguments should be needed, and then link-editor should find the dependencies' dependencies... by looking at the metadata recorded in the .a files.

Really, it's very simple. libstool is a very bad attempt at layering this on top of the linker (recording metadata in .la files), but it doesn't work well. This functionality has to be in the linkers.


Is it literally called libstool…


To get these stats, Drew used 5 different scripts in 5 different languages. Awk, sh, C, go & python. Well, the C program isn't a script it's a test artifact. Drew must subscribe to the "best tool for the job" philosophy rather than the "use what you know" philosophy.


By far the most important reason for dynamic linking for C is semantics: static linking semantics are stuck in 1978 and suck (more on that below), while dynamic linking semantics make C a much better language.

In particular, static linking for C has two serious problems:

1. symbol collisions -> accidental interposition (and crashes);

2. you have to flatten the dependency tree into a topological sort at the final link-edit.

Both of these are related, and they are disastrous. They are also related to the lack of namespaces in C.

Besides fixing these issues, the C dynamic linking universe also enables things like:

- run-time code injection via LD_PRELOAD and intended interposition

- run-time code loading/injection via dlopen(3)

- audit (sotruss)

- reflection

- filters (which allow one to move parts of libraries contents to other libraries without forcing re-links and without forcing built systems to change to add new -lfoo arguments to link-edits)

- use of dladdr(3) to find an object's install location, and then that to find related assets' install locations relative to the first, which then yields code that can be relocated at deploy time (sure, "don't do that" is a great answer, but if you statically-link then you think you can, and now you just can't have assets to load at run-time)

- use of weak symbols to detect whether a process has some library loaded

and others.

C with those features is a far superior language -- a different language, really -- to C without them.

(EDIT: A lot of the semantics of ELF could be brought to static linking. Static link archives could have a .o that has metadata like depedencies, "rpaths", exported/protected symbols, interposer symbols, etc. The link-editor would write and consume that metadata. However, it's 2020, and the static link ecosystem is stuck in 1980 because no one has bothered, and no one has bothered because dynamic linking is pretty awesome. Still, it could be done, and once in a while I think I ought to do it to help save people from themselves who want static linking.)

> Do your installed programs share dynamic libraries?

> Findings: not really

> Over half of your libraries are used by fewer than 0.1% of your executables.

The C library most certainly gets shared, as well as libm and such. The rest, it's true, not so much, but it does depend on what you're measuring. Are you measuring C++ apps? Yeah, C++ monomorphization leads to essentially static linking. Are you measuring Java apps with no significant JNI usage? You won't find much outside the libraries the JVM uses.

> Is loading dynamically linked programs faster?

> Findings: definitely not

Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.

> Will security vulnerabilities in libraries that have been statically linked cause large or unmanagable updates?

> Findings: not really

Correct. But, being able to update libc or some such and not have to worry about updating consumers you might not even know about is a very nice feature.


You bring up some good points here. Here are some of my experiences with these problems when working on oasis (my static linux distro).

> 1. symbol collisions -> accidental interposition (and crashes);

I've encountered symbol collisions only twice, but both resulted in linker errors due to multiple function definitions. I'm not sure how this could happen accidentally. Maybe you are referring to variables in the common section getting merged into a single symbol? Recent gcc enables -fno-common by default, so those will be caught by the linker as well.

> 2. you have to flatten the dependency tree into a topological sort at the final link-edit.

Yes, this is pretty annoying. pkg-config can solve this to some degree with its --static option, but that only works if your libraries supply a .pc file (this is often the case, though).

I think libtool also can handle transitive dependencies of static libraries, but it tries hard to intercept the -static option before it reaches the compiler so it links everything but libc statically. You can trick it by passing `-static --static`.

For oasis, I use a separate approach to linking involving RSP files (i.e. linking with @libfoo.rsp), which really are just lists of other libraries they depend on.

> Besides fixing these issues, the C dynamic linking universe also enables things like: > - run-time code injection via LD_PRELOAD and intended interposition

Yes, this can be a problem. I wanted to do this recently to test out the new malloc being developed for musl libc, but ended up having to manually integrate it into the musl sources instead of just using LD_PRELOAD.

> - run-time code loading/injection via dlopen(3)

In particular, this is a big problem for scripting languages that want to use modules written in compiled languages, as well as OpenGL which uses dlopen to load a vendor-specific driver.

> Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.

But doesn't the dynamic linker still have to do extra work to resolve the relocations in the executable, even when the dependency libraries are already loaded?


> > 1. symbol collisions -> accidental interposition (and crashes);

> I've encountered symbol collisions only twice, but both resulted in linker errors due to multiple function definitions. I'm not sure how this could happen accidentally. Maybe you are referring to variables in the common section getting merged into a single symbol? Recent gcc enables -fno-common by default, so those will be caught by the linker as well.

No, this comes up all the time. Try building an all-in-one busybox-style program, and you'll quickly run into conflicts.

If static link archives had all the metadata that ELF files have, then the link-editor could resolve conflicts correctly. That is the correct fix, but no one is putting effort into it. The static linkers haven't changed much since symbol length limits were raised from 14 bytes!

> > 2. you have to flatten the dependency tree into a topological sort at the final link-edit.

> Yes, this is pretty annoying. pkg-config can solve this to some degree with its --static option, but that only works if your libraries supply a .pc file (this is often the case, though).

pkg-config alleviates the problem, but it's not enough. Among other things building a build system that can build with both, static and dynamic linking is a real pain. But more importantly, this flattening of dependency trees loses information and makes it difficult for link-editors to resolve symbol conflicts correctly (see above).

> > Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.

> But doesn't the dynamic linker still have to do extra work to resolve the relocations in the executable, even when the dependency libraries are already loaded?

It's still faster than I/O. (Or at least it was back in the days of hard drives. But I think it's still true even in the days of SSDs.)


> But doesn't the dynamic linker still have to do extra work to resolve the relocations in the executable, even when the dependency libraries are already loaded?

For dynamic linking there's usually only a single reference that needs to be fixed up in the PLT or GOT for each referenced symbol and the fix-ups are all localized, so that part is typically a very small cost. But this means every call to a dynamically linked library goes through an extra stub function, which adds I$ and BTB pressure compared to static linking.

(If you do things manually with dlsym there's also an extra indirection but it's a little different mechanically since you do an indirect call of a function pointer rather than doing a direct call to a forwarding stub. The advantage of the forwarding stub is that even if there's a BTB miss for either of the two direct calls, you don't suffer a branch mispredict but just a few front-end cycles waiting for the instruction decode, which VTune labels as a "branch resteer". The direct call option also leads to smaller code size for each call site which helps with I$ pressure. Basically it's the difference between CALL rel32 and MOV tmp, [RIP + disp32]; CALL [tmp].)


In a containerized world, static linking will generally be faster, unless the OS does page de-duping for text/rodata. So doing the work to make static linking semantics not suck seems likely to be important.


Most linkers support archive "groups" at the command line, so you don't need to do the topological sort. Most linkers also support relocatable objects, so you can trivially solve the symbol collision problem by just linking together a subset of your app and then stripping that symbol before linking the conflict. And like, it frankly just sounds like you have never heard of libtool, which adds most of the stuff in your edit--like dependencies and configurable linker settings--to the static link process, and which has not only existed but has been widely used among most marquee projects for over two decades. :/


I know all about libtool, thank you, and I'm quite aware that it writes dependency information into its .la files. I use libstool every day. It sucks. This functionality has to be built into the linker.


Making the linker--which is architecture specific and generally "just barely what is required to munge together object files"--into some fat stack of code (that I guess you would expect to work the same for every single linker) because you don't believe wrappers are appropriate for some reason is a disagreement over "clean reusable architecture" and not that the ecosystem as a whole is somehow "stuck in the 80s" :/.


Part of what has to be done can be done in wrappers, it's true: the part that has to do with recording additional metadata. But the other part -using that metadata to resolve symbol conflicts- cannot be done in a wrapper -- it has to be done in the linker-editor. IMO this is a very strong argument for putting all of this in the linker-editor.


> - use of weak symbols to detect whether a process has some library loaded

RTLD_NOLOAD

> Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.

RTLD_LAZY, though it's one less attack vector to have RTLD_NOW and mprotect your GOT and PLT as readonly.


During development of a large system static linking is hell. The final link just takes too long if you are only working on one object file. Also during debugging you have to load all symbols in gdb, which does not scale. With dynamic linking you only load the debug info for the library you are working on.


GDB handles binaries in the hundreds of megabytes fairly well.


Static linking allows LTO with aggressive inlining and is therefore able to achieve far superior performance beyond just the startup time. Arguing that dynamic vs. static has better RAM utilization or not is pointless because nowadays we have plenty of RAM but single core performance is stagnating for almost a decade already. Moores law might give us more transistors but single thread performance is still more or less bound by the clock and transistor switching frequency. Sooner or later static linking will become the only way to move forward and the conveniences dynamic linking offers will not be worth the costs.


> Arguing that dynamic vs. static has better RAM utilization or not is pointless because nowadays we have plenty of RAM ...

Using more RAM means having lower cache hit ratios. If dynamic linking means using less RAM, you win. But it's not a clear-cut thing -- it will depend a lot on the surrounding ecosystem.

In any case, for C, the problem with static linking is about semantics. Until those are fixed I'll be resolutely against static linking for C, and for everything else, well, do whatever performs best.


lol, you don't want aggressive inlining everywhere.

Here's a good example. Firefox does even cross-language LTO for stuff that matters — all the DOM/CSS/etc components that are tightly bound together. But it happily allows you to dynamically link to a system libjpeg, and you won't lose any performance because the libxul<->libjpeg boundary is crossed.. about a couple times per downloaded jpeg. Heck, what would you even inline between the jpeg decoder and the browser? Absolutely nothing!

The world at large is already using static linking where it matters.. e.g. with C++, header-only libraries and anything that uses templates results in a lot of static linking. Even if you link to libboost_something.so, you probably already have most of that something inlined into your code because templates :)


static linking != LTO

LTO requires source, or at least IR.

Static linking involves object files; there's been a loss of fidelity at that point that prevents inlining.


Either something is wrong with the testing script or my computer is way faster than I thought:

    ./test.sh | awk 'BEGIN { sum = 0 } { sum += $2-$1 } END { print sum / NR }'
    -698915


I’m curious about the supposed memory advantages of dynamic linking: on average how many different executables share each page of memory? What about when memory is in high demand? How high does that average become? What is the probability that a page of a shared library is already in memory (cache or otherwise) when it needs to be loaded, and in particular the probability that it is there because another program loaded it).

My guess is that apart from eg libc, the average is pretty low (ie 1 for the pages that aren’t free).


I think this depends heavily on your platform and how often libraries are reused. On macOS, for example, most libraries aside from libc are loaded R/O or CoW quite literally hundreds of times, because every app shares AppKit and WebKit and Security and the dozens of other platform frameworks (and their private dependencies!) that are basically "free" to use and ship with the system and so have very high adoption. On more "coherent" Linux distributions I'm sure things like GTK, glib, OpenSSL, zlib are used by a lot of things too. Sure, there's going to be a lot of one-off dynamic libraries too, but there's a lot of duplication with the popular dependencies and then a long tail.


What are the counter arguments?


Run htop or similar, sort by "shared memory" column and see how much more memory you'd need per process if shared linking did not exist.

I think the author's using a wrong method to make a point. Dynamic linking feels out of place for most long-running server-side apps (typical SaaS workload). One can argue that in a mostly CLI-environment there's also not much benefit.

But even an empty Ubuntu desktop runs ~400 processes and dynamic linking makes perfect sense. libc alone would have to exist in hundreds reincarnations consuming hundred+ megabytes of RAM and I'm not even talking about much, much heavier GTK+ / cairo / freetype / etc libraries needed for GUI applications.


Go executables are statically linked. It makes deployment a breeze.

I think you overestimate how much saving you get from dynamically linking libc. Each executable uses only a small portion of libc, so the average savings is going to be in the handful of kilobytes per executable.


In theory yes. However, in practice static linking with glibc pulls in a lot of dead weight, musl comes to the rescue though:

test.c:

  int main(int argc, char **argv) {
    printf("hello world\n");
    return 0;
  }
Dynamic linking (glibc):

  $ gcc -O2 -Wl,--strip-all test.c
  $ ls -sh a.out
  8.0K a.out
Static linking (glibc):

  $ gcc -O2 --static -Wl,--strip-all test.c
  $ ls -l a.out
  760K a.out
Static linking (musl):

  $ musl-gcc --static -O2 -Wl,--strip-all test.c
  $ ls -sh a.out
  8.0K a.out


Static linking (https://github.com/jart/cosmopolitan)

    jart@debian:~/cosmo$ make -j12 CPPFLAGS+=-DIM_FEELING_NAUGHTY MODE=tiny o/tiny/examples/hello.com
    jart@debian:~/cosmo$ ls -sh o/tiny/examples/hello.com
    20K o/tiny/examples/hello.com
Note: Output binary runs on Windows, Mac, and BSD too.


I hope you can forgive me for asking, but what exactly is this (cosmopolitan)? It looks interesting but I can’t really tell what it’s trying to be.


True, but a couple github imports and 10 lines of code generates a 100mb binary. But, to be fair, I guess we're okay with shipping huge binaries now a days because we're literally shipping whole environments with docker anyway.


To be fair, you're not really supposed to be shipping a full system in a docker image if you can help it; you're supposed to layer your application over the smallest base that will support it (whether that's scratch, distroless, alpine, or a minimal debian base). Of course, I'll be the first to agree that "supposed to" and reality have little in common; if I had a dollar for every time I've seen a final image that still included the whole compiler chain...


In my experience, most who build fat images with the compiler chain and everything do so because they're simply not aware that multi-stage builds are a thing now.


Yep. And just general awareness of how Docker works and how to best use it. And that's not a knock on them; it's a whole topic unto itself, and lots of these folks are developers who are just trying to get their stuff working without having to go off and learn all about Docker, and I have a hard time blaming them for that. Which, of course, is why they keep a handful of us sysadmin folks on staff to help tidy up;)


> Go executables are statically linked. It makes deployment a breeze.

But what are we comparing to what then ?

A few custom Go applications, compared to whole classic Linux distros, on which deployment is both not really the same thing, but still a breeze ?

So yeah, to different needs, different tools.


> Run htop or similar, sort by "shared memory"

The top two entries are 80 and 36 kB respectively. RES is 2.3 giga-bytes (over four orders of magnitude larger) between them. Even multiplying the top SHR by your ~400 processes gives 32 MB (still two orders of magnitude off). That is not a counter argument; that is a agreement that dynamic linking is useless.

Edit: RES, not VIRT.


That is an extremely misleading figure. Shared memory is page-aligned entire libraries dropped into RAM. Statically linking would, as the article shows, only use on average about 4% of the symbols available from the libraries, and the majority of this would not end up in RAM with your statically linked binary. And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks.

Put the cold hard numbers right in front of someone's face and still the cargo cult wins out.


> Put the cold hard numbers right in front of someone's face and still the cargo cult wins out.

Come on.

You did not actually measure the figure GP mentioned and which you are disputing. Your methodology and assumption — that 4% external symbol use translates into 4% size used — is a plausible guess, but you haven't supported it with data.

Even if you had measured the figure you're accusing GP of ignoring, the tone of your remark is just aggressively condescending and inappropriate. Tone it down.

To address your other claims:

> Shared memory is page-aligned entire libraries dropped into RAM.

There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary.

> the majority of [the small fraction of static linked library used] would not end up in RAM with your statically linked binary.

Huh? Why do you claim that?

> And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks.

I think that claim is plausible! But it wasn't an option presented on your blog post, nor was it discussed by GP. Prior to that comment, discussion was only around 100% vs 0% dynamic linking.


> There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary.

But most code isn't performance critical. Thus trying to align functions to page boundaries is just wasting memory. Even in performance critical code, aligning to cache line sizes is enough and aligning to page boundaries doesn't provide any advantage.


Not only is it wasting memory, but I wouldn't be convinced it'd not be outright hurting performance by increasing cache line aliasing, increasing TLB overhead and misses/invalidations! Avoiding page alignment can be a performance gain! https://pvk.ca/Blog/2012/07/30/binary-search-is-a-pathologic...

Like, sure, mapping your executable's code section in at a page boundary is probably fine, but I think trying to align individual functions to page boundaries would be a counterproductive mistake as a general strategy.


> I think trying to align individual functions to page boundaries would be a counterproductive mistake as a general strategy.

Yes — which is why no one does that. I don't know where bjourne came up with the idea.


> But most code isn't performance critical.

Wasting TLB slots on your unimportant code still pessimizes your hot code.

> Thus trying to align functions to page boundaries is just wasting memory.

No one aligns individual functions to page boundaries; you align the entire loadable code segment to a page (or preferably, superpage) boundary.

> Even in performance critical code, aligning to cache line sizes is enough and aligning to page boundaries doesn't provide any advantage.

This is a different kind of optimization (avoiding cache line contention) than I was talking about (optimizing TLB slot use).

Yes, forced page-alignment doesn't help cache line contention anymore than forced cacheline-alignment. But that's irrelevant for code, generally: (outside of self-modifying code, which is extremely uncommon) code doesn't share a cacheline with memory that will be mutated and thus doesn't contend in that way.


Then I don't know what you're getting at. Clearly the most efficient approach would be fitting your performance critical loop into one or at most two consecutive pages making the TLB a moot point.


That is an extremely misleading figure. Shared memory is page-aligned entire libraries dropped into RAM

First of all, I seriously doubt it (haven't looked closely, but if library-loading is similar to mmap, it should only count actually used segments).

But that shouldn't even matter, as most of these libraries are fully utilized. All of libc is used collectively by 400+ processes, that is also true for the complex multi-layer GUI machinery. The output of ldd $(which gnome-calculator) is terrifying, run it under a profiler and see how many functions get hit, you'll be amazed.

Put the cold hard numbers right in front of someone's face and still the cargo cult wins out.

The coldness of your numbers did not impress. And calling a reasonable engineering trade-off "cargo cult" doesn't get you any points either.

Static linking is better than dynamic linking. Sometimes. And vice versa. That's how engineering is different from science, there are no absolute truths, only trade-offs.


@ddevault: performance is only part of the story. The poverty of semantics of C static linking is atrocious -- it could be fixed, but until it is we should not statically link C programs.



Are these also issues for languages like Go and Rust?


No, not for Rust, and not for Goland, IIRC. The reason is that they have proper namespacing functionality in the language, and generally modern languages record dependency metadata in the direct dependents.

This is strictly a C problem.


And the worse part is that other system languages back in the day (PL/I, Algol, Mesa and Pascal variants) already had some form of namespacing support.


Of course not. It's not even a problem for C++ if you use namespaces pervasively. This is purely support for C.


Would this distinction would go largely away if OS would have method to share read only memory pages or larger regions of memory between all processes?

I suspect you would need to compute signatures for static libraries for binaries located /bin/, /lib, ... to make load times faster.


ldd won’t get you all the programs that dynamically load with dlopen... but these are still interesting results.


Not convinced.

First, this analysis was done on Arch Linux, a source-based distribution. Since you know at compile time what your environment is, I would expect the benefits to be smaller. And of course, this means you're willing to do a lot of recompiles. I'd like to see analysis done on more traditional (& common) binary distros.

Second, the arguments seem a little cherry-picked. "Over half of your libraries are used by fewer than 0.1% of your executables." is cute. But modern systems have a lot of executables, so 0.1% > 1, so that still matters, and what about the other half.

Finally, we're already having serious problems getting containers to upgrade when a security vulnerability is found. Requiring recompilation of all transitive users is not likely to win any update speed contests. If it's completely automated then it would work, but any rocks in the process will leave people endlessly vulnerable. See the Android ecosystem, etc.


> First, this analysis was done on Arch Linux, a source-based distribution. Since you know at compile time what your environment is, I would expect the benefits to be smaller. And of course, this means you're willing to do a lot of recompiles. I'd like to see analysis done on more traditional (& common) binary distros.

Arch Linux is not Gentoo. And AUR is only a secondary method of installing software. So I'm not sure what you mean.


It's not Gentoo, but in practice, many frequently-used packages remain in the AUR and an AUR helper to make that transparent is standard kit for anyone using the box as a workstation. It's much more "source-based" than it may appear at first glance. I have ~15G of self-built packages sitting in my PKGDEST right now, if that counts for anything.


Have you checked if they have official packages yet? I was surprised to find that half my AUR packages already were in official recently. Font packages and Android Studio can easily take up 5G. Also, you don't need an AUR helper.


Agreed; I main Arch and I very rarely install packages from AUR. Rare enough that I've never felt the need to install a helper, it would just be unnecessary clutter.

I suspect many users will need to dip into AUR once or twice for unsual packages, but I'm mildly skeptical that most users need to dip into AUR often enough that compile times would be a serious argument against static linking.


> an AUR helper to make that transparent is standard kit for anyone using the box as a workstation

Fwiw, I've been using arch as my daily driver continuously for 14 years and haven't ever really felt the strong need for an AUR helper.


How do you ensure that all of your AUR packages are up-to-date? The makepkg workflow makes installation easy but I didn't update my AUR packages very consistently until I used a helper.


> First, this analysis was done on Arch Linux, a source-based distribution. Since you know at compile time what your environment is, I would expect the benefits to be smaller. And of course, this means you're willing to do a lot of recompiles.

I run Gentoo as my Linux box. It takes a long time to compile stuff as it is (and I have a 22 core machine). Being forced to compile a lot more because everything is linked statically would be a nightmare.

This is definitely not a plus for source based distributions.


Running Gentoo and complaining about long compile times is like cutting off your leg and complaining that walking hurts.


> Running Gentoo and complaining about long compile times is like cutting off your leg and complaining that walking hurts.

I fail to see the relevance of this comparison.

It's more like wearing fitted clothing and (rationally) objecting to requiring a visit to your tailor every time you change your socks.


I'm not complaining, and I can live with the compile times I currently have.

I'm pointing out that saying this will benefit source based distributions more is nonsense.


If compilers weren't so pathetically slow this wouldn't be that much of an issue. If this became widespread it might have the positive impact on projects like LLVM and get them to pay some attention to compiling time, not just optimization.


The problem with compile times is mostly not the compiler's fault. It is with the project setup.

See http://www.real-linux.org.uk/recursivemake.pdf for a classic explanation.


But we haven't learned from this as modern, non-make based build systems still suffer from terribly slow compiles. Take Rust for example. I don't know a single project that uses make to build (they all use cargo), yet Rust suffers from extremely slow compile times. Much of this (as far as I understand) comes from the LLVM compiler, which is why I was picking on compilers.


I have not used Rust very much.

But in https://doc.rust-lang.org/cargo/reference/build-scripts.html I find comments like, It is recommended to carefully consider each dependency you add, weighing against the impact on compile time, licensing, maintenance, etc. Cargo will attempt to reuse a dependency if it is shared between build dependencies and normal dependencies. However, this is not always possible, for example when cross-compiling, so keep that in consideration of the impact on compile time.

So it looks like cargo can have the same issue that make does with recursive dependencies.


This is talking about very different issues than the ones identified in that paper. Most of that paper, if not all, does not directly apply to Cargo.


Thank you for the insight. I know you know Rust very well.

Do you have an explanation for why compile times would be an issue for Rust?


There’s a lot of factors. https://endler.dev/2020/rust-compile-times/ talks about some of them.


Rust's slow builds come from rustc, which is based on LLVM. Other LLVM frontends, like clang, are not as slow. This is in part because borrow-checking has a cost.


I've seen several LLVM backed languages that suffered from Slow compile tiles (eg. a recent post on Crystal mentioned this) and thought it was generally attributed to LLVM. And I hadn't heard anyone talk about clang as being fast and even you seem to be saying it is still slow, just not as slow as others (damning with faint praise?).


I don't have good data for this, but Clang is much faster compiling C than C++, for example. I don't know if it's fast or slow in any absolute sense; it's not too far off from GCC time-wise and produces approximately-as-good optimized code for my purposes.


It helps that in C and C++ worlds we gladly use binary dependencies, which isn't the case for Rust and cargo based builds, unless you throw something like scache into the process.


(But only a very, very small part, generally.)


How is arch Linux a source based distro?


What do you mean by "source-based distribution"?


One without packages shipped as prebuilt binaries: Gentoo [1] would be a good example. You compile every binary yourself (sometime you can use prebuilt binaries for the biggest software, if you are so inclined and the distro allows it).

[1] https://www.gentoo.org/


My Arch install doesn't compile anything locally unless I'm installing from AUR.

Maybe it's doing it behind the scenes or something, but I'm doubtful, because otherwise I think my upgrades would take a lot longer.


Doesn't arch use AUR / pacman for binaries / package management? I wasn't aware that like Gentoo you had to compile most software


No, he's wrong. Arch Linux is primarily a binary distribution.


This comment hits all of the boxes for everything that is wrong with Hacker News.

"First, false statement. Since you know false assumption, I would false conclusion. And of course, this means false conclusion. I'd like to see analysis done on what you did them on."

"Second, the arguments seem a little cherry-picked. "Quote from article about W and Z" is cute. But modern systems have a lot of Z, and Obviously You Didn't Consider This."

"Finally, we're already having serious problems getting Thing the Author Almost Always Rags on for Sucking to upgrade when a security vulnerability is found. Requiring recompilation of all transitive users who the author doesn't care about and who the author has already told are wrong is not likely to win any update speed contests for a use-case the author thinks is invalid. If it's completely automated then the perceived invalid use case would still be viable, but any rocks in the process will leave people with perceived invalid use case endlessly vulnerable. See Notoriously Bad Ecosystem That Isn't Relevant to the Article, etc."


I prefer that comment to your reply. By a lot. It attempted to apply reason. You only mocked the form the arguments took. I am firmly of the opinion that content is more important than form.

That comment did make a major mistake. Arch Linux is not a source distribution. But the mistake notwithstanding, it was otherwise well-reasoned. Your response was not.

You can mock the point that there is still a benefit in sharing a library a few times, and the article ignored the few libraries that get shared a lot. But said point remains true.

You may think that Android isn't relevant to the article's point. But the tradeoffs between static and dynamic linking are true across operating systems. The challenges that Android has had because of static linking are therefore worth paying attention to. Refusing to look for parallels to inform your intuition from is simply refusal to learn.


[flagged]


This is quite possibly the stupidest thing I have heard this year. How could you possibly have come up with something that brain-dead?

That is an opinion. It is your opinion. I happen to have a very different opinion. It may be surprising to you, but surprise doesn't mean that it is trivially wrong.

Experience has taught me that if you have good content, then you have something worthwhile to say if I can get past the form. But, by contrast, someone who only has good form has nothing worthwhile to say. The more I you dig in, the less value that there is to find.

Obviously having both form and content is better than only having one. But if I have to choose one, I will choose to surround myself with people who have good content. Every time.

Is this how you normally are, or did you have to reach deep into your mind to find the most inane thing you could possibly say and pollute Hacker News with it? Goodness, I hope so. I don't even want to think about the possibility that you are this clueless in real life. I pity those that have to interact with you.

If you're curious, you can always find out more about me. My online profile goes back quite far. A few random things that I've written include https://www.perlmonks.org/?node_id=59345, http://elem.com/~btilly/effective-ab-testing/, and the answers at https://stackoverflow.com/users/585411/btilly?tab=answers.

Or you can go with your already formed opinion. It is no skin off my back.


I have read a lot of your writing, much more than you've linked here, simply by being active on this site for just over three years. While I don't really think I could say I know you personally, I think I have at least a little bit of context about you, what your interests are, and what you like to talk about. Your comments have been consistently useful and relevant, and I have no complaints there.

But back to the topic: form and content are not a zero-sum game. You can be eloquent and polite while also conveying useful information, as you mention yourself. You can find people who do this, and it's not a big deal if you're trying to do it yourself. Honestly. If you think about it, the reason I'm replying to you (and why I stick around in some threads and leave others) is both the content and the way it's dicussed: I usually stop interacting when people start hurling insults, because it's just exhausting to deal with it for extended periods of time. It's part of the reason why I occasionally leave Reddit frustrated, why I'm not on Twitter, and why I let other people pull information out of Slashdot and 4chan for me. Really, if I can find one person who says something useful but surrounds it with vicious barbs, and another who can say the same thing but without the acid, why wouldn't I want to go with them? If I can only find the latter, perhaps I'm with the wrong people.

And again, I do personally make some concessions for people I truly do think have exceptional content–but again, these are some concessions. Even Linus–who is usually the poster child of the "who cares how you say it?", who is exceedingly bright and talented, and who is basically a "equal opportunity abuser"–could probably tone it down some. When he is right it's just extra verbiage I need to scroll through and ignore, and when he's wrong (and he is for some things) it just makes him sound extra rude.


I agree that form and content are not a zero sum game, and try to present with a combination of form and content.

But when I consume information, I prioritize looking for content over form.


> This is quite possibly the stupidest thing I have heard this year.

I doubt any comment in this thread could possibly be the stupidest thing any of us has heard or read in 2020.


Probably not, but considering my comment was confusing to at least four people here perhaps it might be one of the stupider ones.


You're being unnecessarily rude.


The point of the first half of his comment was very obviously pointing out that "form is less important than content" is wrong; Saagar wasn't doing it for no reason, and wasn't doing it in bad faith. That's why it was in an entirely different tone than the rest of the comment, and just about every other comment he's ever made on this site.


Oh, I see. Sarcasm on the internet can be tricky. (There are many people who actually write like that, or much worse.)


I tried my best to make it really over-the-top, but sadly there are people who write that way unironically on the Internet and so Poe's law applied :(


the article ignored the few libraries that get shared a lot

The author didn't ignore this, because it's pretty much false. Every package people insisted was relevant, he shot down before he published the article. Example:

https://cmpwn.com/@sir/104406644780241359

Hitchen's razor.


> Of the 2188 packages I have installed, 70 of them depend on OpenSSL.

This must not be counting packages which depend on openssl by a chain of dependencies, some number of which contain binaries or libraries which link libssl. On my desktop Arch system of 1448 packages, 42 depend directly on openssl and 672 depend on it directly or indirectly. (pactree -ur openssl | wc -l, then subtract one for openssl itself)


I know it's a popular Hacker News trope, but when done well I actually appreciate comments of the style "the author does a comparison and has some results, but do also keep in mind xyz that is non-obvious". I know there is a problem with people who go through articles and try to pick things that could affect the results and treat that as invalidating them, but more often than not simple comparisons like these do have more to say about them than what is presented and I don't want to discourage those.


This one in particular was done terribly, and is imitating ones in the same style. I don't dislike the act of disagreement, nor even disagreements in this vein, I dislike the act of being confident, and further condescending ("is cute"), while making so many obvious mistakes that could be corrected with even a speck of knowledge about what you're talking about.


Actually, this is a very serious topic, and TFA is wrong in a number of ways. HN commenters being able to say so is everything that's right with HN. My own commentary on TFA is all over the comments on this HN post. I put more effort and detail into my comments on this post than GP, but GP is not wrong.

Commenting that a comment is exemplary of all that's wrong with HN is... popular, I'll grant you that, but it grates, and it's not really right. It's a "shut up" argument. Just downvote what you don't like, upvote what you do like, and reply with detailed commentary where you have something important to say. No need to disparage commenters.


HN commenters being able to say so is everything that's right with HN.

This would be correct, if the objection were right. But the commentator was objectively incorrect, in a way that could have been avoided with something as small as a web search, and the comment had the audacity to do so in a condescending way. I didn't knock anyone who had a genuine objection to the post, I replied to an incredibly low-effort and low-quality comment pointing out that it was low-effort and low-quality.

Commenting that a comment is exemplary of all that's wrong with HN is... popular, I'll grant you that, but it grates, and it's not really right.

This is the first one I've ever made, and I post pretty frequently. I don't think it's wrong whatsoever, and I'm standing by it.

It's a "shut up" argument.

It absolutely is a "shut up" argument. Comments should not be written when they can't get the simplest of facts straight, and authors of them should think before they post.

No need to disparage commenters.

I didn't, I disparaged their comment. I don't think doing so was wrong, either, because the comment was practically equivalent to replying to a well-considered post with a PNG of a meme.


I saw nothing wrong with u/dwheeler's comment. I've addressed issues with static linking in more detail up and down this thread, so perhaps you won't tell me to "shut up", but really, it seems all you've got is "shut up". Maybe I could say something clever about comments like yours being representative of all that's wrong with HN, except I don't believe that -- no, I'm glad you get to say "shut up" -unpleasant though it is-, and I'm glad u/dwheeler can express skepticism without going into as much detail as you'd like even though you'll be there telling him to "shut up".


The first graph looks like a power law. It's fascinating how many processes follow power laws.

As a side note, you are almost always better off (also on this case) by plotting power laws on a log scale.


The main problem I have with dynamic loading is that my app crashes without any sensible error message on runtime, if I misspelled a single symbol.


With NetBSD, it is relatively easy to compile the entire system as static. Is it easy to do this with Arch Linux? (Maybe Void Linux?)


I appreciate the brevity of this article.


these benchmarks dont seem to take into account page cache


Paging Laura Creighton!

Your time is now.


What should I search for to get a clue as to what you're referring to?


Her name, and USENET. She was the most vocal opponent of the move to dynamic-linked libraries, back when it happened. Her expectation was that programs that once worked would stop working as the libraries rotted.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: