Linux usually (by convention) provides 1 file and 2 symlinks per lib: liba.so -> liba.so.x -> liba.so.j.k.l.
The first one is to make the linker (ld) happy: -la will look for liba.so. The linker puts the SONAME (liba.so.x) in DT_NEEDED.
The second symlink's filename corresponds to the SONAME, so that the runtime linker (ld.so) can locate the library by SONAME in rpaths.
The third one is the actual library, which can be updated while keeping the same soname & same abi.
Now, it would be great if the linker had an option to not only copy the SONAME into DT_NEEDED, but also register the path in which the library was located as an rpath.
Cause the situation on Linux is absurd! You pass some flags -L and -l to the compiler/linker, the linker links something and nobody knows what. Then when you run your executable it has to locate this something again, and you can only pray that your libc and binutils/llvm agree on search paths & order. In most cases this does not work, and you must manually pass -Wl,-rpath,/some/path to add a search path. Nobody guarantees that what the linker links is what the runtime linker uses.
Of course there are many edge cases:
- linking during make without relinking during make install will make your executables register rpaths to build directories instead of install dirs
- sometimes you link to a stub lib that should not be used at runtime
But still, some guarantee that what you build with is what you run with would be a major user experience improvement for linux.
> Cause the situation on Linux is absurd! You pass some flags -L and -l to the compiler/linker, the linker links something and nobody knows what. Then when you run your executable it has to locate this something again, and you can only pray that your libc and binutils/llvm agree on search paths & order. In most cases this does not work, and you must manually pass -Wl,-rpath,/some/path to add a search path. Nobody guarantees that what the linker links is what the runtime linker uses.
The issue you're missing is that the build directory is usually not the location of the final binary objects. The actual absolute path of libfoo.so may well be in /builds/runner/foo-package-4df78af0/build/prefix/lib/libfoo.so, which is unlikely to exist on anyone other than the CI's machine (and even on the CI machine itself for too much longer). The actual location will usually be /usr/lib64/libfoo.so, but the library that is linked against may well not be there at the time of linking (particularly in the case where a package is building both a library and an executable that depends on said library in the same package).
What you really want is for the relative path to the library to stored in the executable. Unless what you want is to actually use the globally-installed library and not one you're building at the same time. There's no single solution that fits every use case!
> The actual location will usually be /usr/lib64/libfoo.so
Usually indeed, for distro's that pretty much support one single version of every library. But this is no longer true for Nix, Spack, Gentoo Prefix and Guix, all these package managers/distro's have in common that there should be no default search paths where all libraries are dumped into.
How about `--copy-link-path-as-rpath` and `--copy-link-path-as-rpath-ignore=/build/dir`, so that ld continues to copy the soname to dt_needed, and registers rpath of non-build dirs. Then Nix, Spack, ... can simply use these flags in their linker wrapper.
I think most people designing build systems (I'm one of them) would prefer to explicitly set the paths at each point rather than have gcc ferry values between inputs behind the scenes.
My build system will have already resolved all these paths. It's very easy to interpolate these paths into the command to call the compiler.
Of course you can't guarantee that the library will get installed to the directory you think it'll get installed to, since there's no unified installer system for all Linux distributions. So even a relative path doesn't necessarily work. The best you can do is hope that the Freedesktop filesystem hierarchy is being followed, but that forces installing software for all users at once instead of per user, despite Linux supposedly being a multiuser OS.
Your complaints are reasonable, but gcc already does this. Here, I'll show you how:
> You pass some flags -L and -l to the compiler/linker, the linker links something and nobody knows what.
I agree, it would be really nice to be able to specific exact shared object paths instead of using -L and -l. Build systems typically already know the full paths to all the objects and the abstraction here is often unhelpful.
This could be remedied fairly easily by allowing (for example) -l to take an absolute path to an object rather than searching -L paths. But gcc already does this - you can just put the shared object on the command line directly like so:
Change this: gcc -lfoo bar.c
Into this: gcc bar.c /path/to/foo.so
The effect is the same, but more explicit. The foo.so.x object will be linked and added to the SONAME list.
> Nobody guarantees that what the linker links is what the runtime linker uses.
This part, however, is by design. We explicitly do NOT want what the linker links to be what the runtime uses. This is how we update shared objects between minor versions to fix bugs without reinstalling every binary on the entire system!
Letting libfoo.so.1 link to libfoo.so.1.x is a huge feature. Locking in an explicit minor version would defeat the entire purpose of dynamic linking.
> Letting libfoo.so.1 link to libfoo.so.1.x is a huge feature. Locking in an explicit minor version would defeat the entire purpose of dynamic linking.
My suggestion is to continue copying the SONAME into DT_NEEDED, and record the dir the lib was found in as an rpath.
Well, you complained about ambiguity in the search process during build, and referencing by path is how to fix that.
Regarding runtime, we absolutely do not want to implicitly embed build paths. As others have said because it's unlikely we will put them in the same place, and it's unlikely we will build and run on the same systems.
The runtime library management system is going to have a structure for where to place libraries. It may be as simple as tossing them in /usr/lib, or it may be somthing where we have different paths for each application. We can't do this if the compiler implicitly dictates universal linker paths.
One aspect I think you may also be missing is that shared objects themselves have DT_NEEDED and rpaths. You would quickly run into very confusing conflicts between binaries built on different systems, or with different build environments.
It's hard to see a problem here, since adding an rpath is very easy. You appear to be asking for implicit, hidden behavior in the compiler which doesn't fit the vast majority of use cases.
The first one is to make the linker (ld) happy: -la will look for liba.so. The linker puts the SONAME (liba.so.x) in DT_NEEDED.
The second symlink's filename corresponds to the SONAME, so that the runtime linker (ld.so) can locate the library by SONAME in rpaths.
The third one is the actual library, which can be updated while keeping the same soname & same abi.
Now, it would be great if the linker had an option to not only copy the SONAME into DT_NEEDED, but also register the path in which the library was located as an rpath.
Cause the situation on Linux is absurd! You pass some flags -L and -l to the compiler/linker, the linker links something and nobody knows what. Then when you run your executable it has to locate this something again, and you can only pray that your libc and binutils/llvm agree on search paths & order. In most cases this does not work, and you must manually pass -Wl,-rpath,/some/path to add a search path. Nobody guarantees that what the linker links is what the runtime linker uses.
Of course there are many edge cases:
- linking during make without relinking during make install will make your executables register rpaths to build directories instead of install dirs
- sometimes you link to a stub lib that should not be used at runtime
But still, some guarantee that what you build with is what you run with would be a major user experience improvement for linux.