1) "Shared libraries are bogus."
2) "Anyone who likes normal-looking user interfaces rather than plain boxes with text in them is a poopipants."
Both of these propositions are contentious to say the least, but what bothered me was that two propositions were mutually supporting while being (to my mind) orthogonal. The obvious and most compelling example of a shared library on the Unixen at the time were all the various UI libraries (Motif and various other abominations; all of them huge and unwieldy). It seemed necessary to accept that these libraries were obviously Completely Unnecessary to buy into the Plan 9 idea that shared libraries didn't do anything worth mentioning.
I'm sure it's possible to design a better UI library (or maybe even a wacky user level file system for user interfaces; in fact, my honours project in 1993!) but at the time the way people made interfaces that looked vaguely like what other people expected computer programs to look like was to use big-ass shared libraries on Linux.
This was also the way (dragging in one of those godawful blobs like Motif etc) that anyone might have exerted themselves to port across a (not completely pitiful) web browser, but the degree of aggressive disinterest in supporting Other Peoples Code was extreme (Howard Trickey's "ANSI POSIX Environment" or "APE" didn't get much love as far as I could tell).
It was quite undignified to watch people struggling with text-based web browsers or firing them up on non-P9 boxes because of the inability to support the web at the time.
Are you implying that "shared" = dynamically linked? Was there not a version of Motif that could be statically linked?
In the 90's, the most popular UI library for Windows was MFC (Microsoft Foundation Classes), which could be linked either statically or dynamically.
Visual Basic ate a considerable chunk of Delphi's lunch in the enterprise area, where price was less of an issue, and large dev shops would buy an MSDN subscription anyway. But it was never more popular than Delphi for desktop apps.
Worldwide, no idea.
The GP seems to be confused about the meaning of "shared library", and seems not to be aware of the existence of static libraries -- which would (as you point out) be perfectly usable in the case of Motif.
'Perfectly usable' is a bit of a reach given how huge Motif was at the time. I shudder to think about the size of a directory with a bunch of X applications that each snarf in all their dependencies; remember we're talking 1990s.
No, GP is “grandparent”, that is, the post two posts upthread from the post in which the term is used (or, in some contexts, the user who posted the post two posts upthread.)
Similarly, GGP is “great grandparent” (3 posts up), and so on.
OP is “original post” or “original poster”, which is (in the general case) a different thing than GP.
Though I could be wrong, as my exposure to internet message boards is weirdly eclectic, but far from comprehensive.
Prior to that, the system dirs were searched first, which was the trigger for the DLL hell issue.
But not even Microsoft's own products respected that in practice.
Also Aix uses the same model as Windows.
Finally, .NET Assemblies are versioned, and Microsoft has introduced Application Manifests around Windows XP.
So it a matter of actually taking the effort of specifying the versions that the application works with.
But then, you still have to put up with defining potentially hundreds of <bindingRedirect>s in config files to even get an application to run correctly, even though they mostly don't share their DLLs.
The only thing it takes is Nuget dowloading a newer version of some dependency another library depends on, even if it's backwards compatible with the old API. Which, of course, it should always be if you haven't switched to a new major version.
Edit: Also, I find it very disturbing that the resulting errors always happen to be crashes during runtime instead of up-front. If that happens with a plugin or through reflection: fine. But for most of .NET code, dependencies should be able to be loaded during startup.
The real distinction in my view is between tightly coupled components and loosely coupled ones. Loosely coupled as a design paradigm is harder to design and maintain for the programmer and in some cases has a performance impact, but a lot nicer for the admin and feels ‘cleaner’ to me, at least.
This is one of the reasons the web eclipsed native applications. The web only provided the most basic common widgets, so designers were forced to reimplement the UI toolkit, but also given the freedom to have their own take on it.
I personally would prefer to see a UNIX-like UI library made of many composeable parts. With idependant codebases.
In that world, having a single giant dynamically linked UI blob doesn’t help.
I’m not saying standardization is bad, just that forced standardization at the architectural level is bad.
The web has been terribly violent to this idea. Native UIs are expected to support keyboard navigation, keyboard shortcuts, a menu bar, type select, drag and drop, accessibility, scripting... And in any given interaction there are advanced modes: multiple selection, modifier keys, etc.
Hardly any of this works on the web; even if it did you wouldn't think to try it. Does type select works in GMail's custom context menu? Can you command-C copy a message in one folder, and paste it in another? Would it even occur to you to try those things?
That stuff is ancient I know, and it would be one thing if the web were pioneering new UI vocabulary that displaced the old. But it's not. There's nothing new that has taken hold, no replacement for what is lost. Gmail has its janky, app-specific keyboard shortcuts, which is at least something, but there's no mechanism for it to spread.
We're in a Dark Age. Every web page now has its own custom CSS-style menu that only supports the most basic hover-and-click, and the bar for good UI is just lying on the floor.
(take my experience with a grain of salt, I've only used React on side-projects, never anything complicated or that I was forced to develop on because of work, and never ran into any perf issues)
- Standard layouts
- Standard input and display components
- Standard event and message handling
- Standard user interactions
- Standard color themes
Which is great for developers to set up an initial UI, but makes every application look the same and takes a lot of work to customize in a way that's consistent with the defaults.
On the other hand, building a complex, feature-rich UI requires a lot of skill both in terms of implementation and design, which is less than common. So that Qt UI will be functional and predictable to users, even if it was not very well thought out, and the broken web UI will be useless to everyone.
Even after reading lots of UI guidelines, and developing several tools and applications, it's taught me that I'm terrible at it, and should let an expert do that part while I stick to the other parts!
Not actually UNIX UI libraries.
OK, and those parts ought to be shared libraries.
Using it primarily? I would find that very difficult with today's dependence on the web, especially with how complex the web has become today.
that's my desktop in wikipedia
or as relevant at the time: how did you communicate via msn/aim/yim/icq? I hope you're not one of those condescending Unix types that only had online interactions on irc :P
Googling for the web browser options online, it seems Links and Abaco are the best options right now. There are others listed in Wikipedia.
I wonder if there's someone out there that has put in the effort to improve their Plan9 environment. I wonder how difficult it might be to port something like Chromium, Firefox, or Webkit to Plan9. Maybe it's not as tremendously difficult as I imagine it, but maybe I'm dreaming thinking this.
Source: I worked as an Inferno developer.
Ah man, that's really cool.
So what's your favorite operating system? Is there one you find ideal or do you like different ones for their different trade-offs? What do you wish was the most widely used OS in the world?
I ask these questions thinking of the OS foundations and ignoring that OSes like Plan9 and Inferno are lacking many things only found in more popular OSes by virtue of being popular.
"Everything is a file" as an abstraction enables you to build really powerful through composition. e.g. running TCP/IP over a serial connection just by mounting files in certain places.
Your end user program doesn't need to know anything about networking, it just reads from files.
this is an irc bot written in shell script
It's not perfect, of course, it had problems with throughput - we were forever discussing at conferences how to improve the protocol to implement streaming.
I describe it as a racing car, not everyone can drive it, you need to work to keep it going, you might kill yourself but when you're out front, it feels amazing.
Basically, suckless's ii seems to imitate that.
To have a functional, modern IRC client, you have to not only implement what there is in the RFCs (which incidentally, no IRC daemon follows anymore, not even the original ircd2) but you also have to pull in IRCv3, SASL, ISUPPORT parsing, CTCP parsing, a TLS library, ... the list goes on. And for a good experience you also want scripting and the ability to abstract away the differences between services implementations (Atheme and Anope, mainly). Not to mention you have no clientside encryption option on IRC short of any number of buggy OTR implementations.
IRC is not great. It might be okay, but by no stretch is it great.
Curious because I actually haven't seen anyone that disliked it beyond the loading screen's silliness.
Some of these are already in place, eg. IRCCloud can render simple markdown, supports push notifications, can show inline images, supports display pictures, Slack-like threads, emojis, etc...
An example: https://twitter.com/IRCCloud/status/971416931373854721
Hopefully longer term we won't need to use IRCCloud/a bouncer to keep chat history. AFAIK an IRCv3 server should be able to offer that natively (to an IRCv3 client).
As for unencrypted TCP... a lot of IRC networks seem to offer TLS these days.
Not saying this is perfect -- far from it, but people are working on improving IRC!
>no support for threads
That one's just wrong. You can make a threaded client if you want one, although it's not totally clear why you would since the CPU load of IRC is going to be very low in any reasonable situation.
Edit: Oh I see, threads as in forum threads.... just open a new channel. Otherwise, the concept seems to map poorly to ephemeral real time discussion. This is the first I've ever heard of AIM,etc of having forum threads.
Opening a new channel wouldn't work for fairly obvious reasons (nobody would be in it, you'd effectively have to do it for every single message, etc.).
IrcCloud is an example of a client that tries to solve pretty much all the issues and you can even plug it into a desktop IRC client and get most of the functionality including history. It's not necessarily better than modern chat or anything, but I love having a modern chat experience while being able to discuss things with people and groups who virtually only use IRC.
If you aren't a fan of hosted services you can run your own BNC. Some IRC servers have built in BNC services that give you things like better security and channel history.
Let me reaffirm that I am not of the position that IRC is perfect or modern. But you are misrepresenting it a fair bit, even in 2006 I was connecting with TLS (sometimes self signed certs though.)
No channel history, no threads, and no images are features.
There's formatting: bold/italic/underline and colours.
yep, IRC, if I was at the computer I had a window with FreeNode and QuakeNET.
And a condescending Plan9 user, "Not only is UNIX dead, it's starting to smell really bad", as we say.
Did you see the part where I was a website devops for 10 years using Plan9?
What? There's a Linux emulation mode in Plan9? O_o How well does it work? Could I run anything Linux in Plan9?
Starwman, nobody said that. Shared libraries make sense when a program wants to load additional code at run time. A classical example is loading and unloading plugins.
For everything else, not so much.
Most people I know use a tiling window manager with terminals. So plain boxes with text in them seems to make sense for many people.
"For everything else, not so much".
Most people I know also use the web and menus and buttons and dialog boxes and so forth and expect all this stuff to look vaguely like other computers do. The fact that the Plan 9 folks were trundling over to non-P9 machines to read the web seemed to suggest that they also liked seeing things that weren't just text in plain boxes.
The point remains that these two propositions are both at the least contentious, and only by combining these unrelated points could anyone really seriously take Plan 9's approach to shared libraries (at the time) seriously.
Your "most people I know" comment is bizarre: even hardcore terminal users need to browse the web, and it's been years since lynx or w3m could be claimed to be adequate for most web tasks.
But the Plan 9 folks consciously made it difficult to write or port web browsers and other consumer software. I think I remember Tom Duff stating that writing a web browser was a "fool's errand", which I guess captures the sense of heightened seriousness, privilege, and lack of concern for the average mainstream user that informed Plan 9's design.
Well, he wrote a browser for Plan9 (Mothra), so if he said that, it's not for lack of experience.
I tracked down the original message in the archive of the 9fans mailing list. The fate of Mothra is discussed.
There's more discussion in this file between Howard Trickey (who refers to himself as an "erstwhile fool", having written Charon for Inferno), Duff and other Plan 9 enthusiasts.
The displayed level of knowledge and engagement with the requirements for building a web browser isn't exactly off the charts. I think it's fair to say that Duff didn't have a lot of patience with the project. He saw it as a moving target that probably couldn't ever be hit. For Plan 9 the classic web browser is a culturally alien type of software: complex and to some degree necessarily monolithic.
Here's what Duff said:
> how come mothra bit the dust?
I wrote it. It was not in good condition when I left
Bell Labs. Understantably, nobody else wanted anything
to do with it, so it died.
Its biggest shortcoming (other than its general
internal hidousness) was that its document imaging
model was fixed before <table> entered the picture.
Deep down it believed that documents were running text
with embedded line-breaks and indent changes, meaning
there's no good way to get tables or frames to work.
Also, if your browser doesn't closely match Netscape
and Microsoft, people will believe that it just doesn't
work, regardless of how good a job you do of meeting the
On the other hand, I still think its idea of how to handle
navigation (mostly the panel with an LRU list of pages
visited) was better than anything else I've seen.
Writing a web browser is a fool's errand. The
specification was changing faster than I could type, and
> Surely somebody must be trying to port Mozilla to Plan 9?
While that would be better than no browser at all, Mozilla
is just the sort of stand-alone monolith that we're trying
to argue against. And of course, everyone else in the
world is trying to turn their system into a giant web
browser. I'd rather see web access be one aspect of a
system in which the pieces work in concert.
They had the luxury of being able to choose (effectively) to reject the web user experience and they took that option. Tom Duff probably wouldn't claim that he was speaking about the intricacies of web browser development with great authority!
There's more discussion in the 9fans archive. And see also Abaco, another attempt at a graphical browser for Plan 9: http://lab-fgb.com/abaco/
As my human-computer interaction prof said back in undergrad: You are not normal. There is a reason that no modern operating system outside the "Other Linux/BSD" bucket ships with this as a default.
Plan 9 has an alternative method for sharing code among processes, namely the 9P protocol, and consequently never needed -- and never used -- DLLs. So for example instead of dynamically linking to Xlib, on Plan 9 a program that wanted to display a GUI used 9P to talk to the display server, which is loosely analogous to a Unix process listening on a socket.
Edit: It does of course get you extra functionality, like being usable over a network connection. Trade offs.
Yes, it's less efficient.
The better question might be "did it matter in practice?".
I'm not advocating for removing DLLs from our OSes, BTW. Nor am I advocating for Plan 9.
What has happened? Two things, apparently. One is that when I created my custom patch to the window system, to send mouse clicks to
Emacs, I created another massive 3/4 megabyte binary, which
doesn’t share space with the standard Sun window applications
This means that instead of one huge mass of shared object code running the window system, and taking up space on my paging disk, I
had two such huge masses, identical except for a few pages of code.
So I paid a megabyte of swap space for the privilege of using a
mouse with my editor. (Emacs itself is a third large mass.)
The Sun kernel was just plain running out of room. Every trivial hack
you make to the window system replicates the entire window system.
CPU caches aren't that big so it still matters today, at least for desktop applications.
You'll notice that those fallback paths do not include any location relative to the application. It's actually pretty difficult to get the object code to lookup libraries relative to its own directory. Explanation here:
The program sets LD_RUNPATH_SEARCH_PATHS which bakes a list of paths into the binary. This can include relative paths as well as @loader_path and @executable_path. eg a plugin bundled with an app can specify @executable_path/../../Frameworks to reference the parent bundle's Frameworks directory. Libraries can also add search paths to help locate their dependencies. @loader_path will expand relative to the library that actually has the dependency.
Any linked library with a DYLD name of @rpath/ will be searched in those locations.
At build time the linker checks each library for its DYLD name and bakes that name into the final binary. Making sure all of your non-system dependencies are @rpath and relative to your app bundle is what makes it a "Relocatable" application.
One could just use a wrapper script that sets the DYLD_FALLBACK_LIBRARY_PATH relative to the binary's directory and run it.
2. Even in an absurd world where each coreutils executable required 100mb of libraries, a busybox-like delivery would already shave ~10GB off of that. Other improvements can be made: binary deltas for security updates, performing the final link step at package install time, probably others.
3. libc updates have introduced security issues; shared library updates in general break things. I can take a statically linked executable from 1998 and run it today.
Lastly, this is totally unrelated to the question because 971 statically linked command line applications will be well under 1GB, but a 250GB drive? The last time I had a drive that small was a 100GB drive in a PowerBook G4. Linux (and p9 from TFA) are OSes used primarily from the developers (at least until the mythical year of linux on the desktop). Springing $200 for a 512GB SSD upgrade seems well worth it if you are being paid a developers salary anywhere in the western world.
The largest executable is ptx at 272kb vs 72kb for the Ubuntu binary.
For the smallest, false is 48k statically linked vs 32k for the Ubuntu binary.
If all 970 executables in /usr/bin average out to 100kb of extra space, that's less than 100MB overhead.
Stripping the binaries decreases the size to about 7MB total or byte sizes of 34312 vs 30824 for a stripped false binary and 251752 vs 71928 for ptx.
For download times, a tar.xz is a good measurement and it's 819k for the 105 statically linked files or 1015k for the full installation of coreutils including the info files and manpages.
Some proponents of static linking talk about performance, I think it's a negligible win, but as I have it handy I thought I'd measure:
10000 runs of a dynamically linked "false":
Lol, you have to do some kind of stripping or GC sections or what not for this to be a fair comparison. A proper version of false is 508bytes on my machine.
Our tool, allmux, merges independent programs into a single executable and links an IR-level implementation of application code with its libraries, before native code generation.
I would love to go into more detail and answer questions, but at the moment I'm entirely consumed with completing my prelim examination. Instead, please see our 2018 publication "Software Multiplexing: Share Your Libraries and Statically Link Them Too" .
If you'd manage to implement that too then it seems that really big projects could be packed together.
Or if you prefer google groups: https://groups.google.com/forum/#!topic/comp.os.plan9/x3s1Ib...
The headline could use a "(2004)" suffix.
If you report an iOS or Chrome bug where you tried to revert a library upgrade and something broke, they'll just mark it "Won't fix: omg never ever look at this".
The dependency graph when everyone isn't updating all at once is brutal. Half of Unix life is/was "well I need to update X, but can't because Y depends on old X. Now we'll just create this special environment/virtualenv/visor/vm with exactly the brittle dependency graph we need and then update it, um, never."
We complain about One Version/Evergreen, and should, but it's got huge advantages. And might be an indicator that testing surface is the real complexity constraint.
One Version's success a good indication that Plan 9 was at least not totally wrong.
In the case of an irreconcilable "X requires Z v1 but Y requires Z v2" they fork package Z.
In practice it is far easier for a lot of software to distribute Windows-like binaries where all but the most basic dependencies are included (e.g. Flatpak or Snappy). In that case dynamic linking doesn't help at all.
Modern Ubuntu systems rely on Snap or Flatpak for a lot of software. What these systems do (as I understand it) is package a large amount of the dynamic libraries that would be provided by the operating system, and stick them in a compressed file (or virtual file system, whatever).
So what you essentially get is a 200MiB 'binary' without any of the benefits of dynamic linking (being able to swap out a library given a vulnerability without recompiling) OR static linking (a single file, with the extraneous code removed, etc. etc.).
And then you pray that the interface and behavior have not changed enough to break things that depend on it.
> Some systems use dynamically linked libraries (DLLs) to address these configuration issues. The problem with this approach is that it leaves security code in the same address space as the program using it. The interactions between the program and the DLL can therefore accidentally or deliberately violate the interface, weakening security. Also, a program using a library to implement secure services must run at a privilege level necessary to provide the service; separating the security to a different program makes it possible to run the services at a weaker privilege level, isolating the privileged code to a single, more trustworthy component.
The paper goes on to explain how the various cryptographic services are exposed as a file server. This is the Plan 9 way of doing things: have lots of small programs that talk to one another.
- You can upgrade core functionality in one location
- You can fix security bugs without needing to re-install the world
- Overall, they take up less disk-space and RAM
- They can take much less cache, which is significant today
Because it's a separate process, so when it crashes your application can put up a "SSL failed, reconnecting" message and carry on. Also when your application has read-something-somewhere security vulnerability, it can't compromise your SSL keys.
Nitpick: it's not one process, but one executable; you can have multiple SSL daemons using the same (read-only) binary, so a attack on one only gets one set of keys. (The same attack will probably work on every instance of the same version of SSLd, but shared objects don't fix that.)
Think about it: If you were to force absolutely everything to be statically linked, your KDE app would have to include the ENTIRE KDE library suite, as well as the QT libraries it's based on, as well as the X window libraries those are based on, etc etc. You'd quickly end up with a calculator app that's hundreds of megabytes.
But let's not stop there, because the linkage to the kernel is also dynamic, which is a no-no. So every app would now need to be linked to a specific kernel, and include all the kernel code.
Now imagine you upgraded the kernel. You'd have to rebuild EVERY SINGLE THING on the system to build with that new kernel, or a new version of KDE or QT or X or anything in between.
The kernel membrane is a form of dynamic linkage for a reason. Same goes for IPC. Dynamic linkage is useful and necessary; just not to such a microscopic level as it once was due to size constraints.
The key is not to eliminate dynamic linkage, but rather to apply it with discretion, at carefully defined boundaries.
That is false. Static linking only links in the the parts that are actually used, which would be a small fraction of the total.
KCalc would for instance pull in HTML rendering for the Help dialog, which would also add all available picture format plugins and also PDF writing support etc.
I just don't think "I need to let the user open an HTML document, though not as any core part of my program's functionality" is a strong case for "well I guess I'll have to embed an entire HTML rendering engine if I statically link my deps".
It allowed some interesting things. For example, you could write applications that could read, display and modify the contents of Excel files, except they didn't do it directly; they delegated all the actual work of opening, reading, and modifying the *.xls file to Excel itself.
I wouldn't personally consider consuming a COM interface like that to be a form of linking.
That said, getting back to the broader context, I've no idea how you'd make something like that work for a GUI toolkit. But WinRT is supposedly based on COM, so maybe they got something figured out?
That being said I agree that even with that you'd still end up with a massive binary because of all the code that's actually used in one way or an other and all of the code that the compiler and linker can't statically mark as dead. In particular anything being called indirectly through a function pointer is almost certainly going to end up in the final binary regardless of whether it's used or not.
And even if a lot of the code gets removed it's a bit silly not to reuse the shared code in RAM instead of allocating pages containing the same thing over and over again. It's wasteful and cache-unfriendly.
no, because we had working LTO for a long time - every KDE app would contain only the exact code that it needs, down to the individual member function. Sure, all KDE apps would have a copy of QObject / QString / QWhatever - but most C++ code nowadays is in headers anyways .
> You'd quickly end up with a calculator app that's hundreds of megabytes.
The DAW I'm working on (340kloc), which can be built statically linked to Qt, LLVM, libclang, ffmpeg and a few others, is a grand total of 130 megabytes in that case.
It's actually heavier when distributed as set of dynamically linked things (unless shipped by linux distros of course, and most of that size is useless LLVM things that I haven't found how to disable yet) :
only accounting for the dependencies I have to ship, it would be 176 megabytes - plus 17 megabytes of the actual software. I'd argue that in that case dynamic linking actually takes less disk space overall because most people don't have LLVM installed on their machines and don't need to.
Java is like that too. The smallest possible program (just a single line to exit) requires things like java.time.chrono.JapaneseChronology. Because argv is a String array, and making the String array requires calling a function or two that can throw exceptions, which requires the Throwable class, which has a static initialiser that requires... the chain is long, but at the end something has a SomethingChronology member, and so AbstractChronology.initCache() is called and mentions JapaneseChronology.
A friend of mine tells a story about how how he accidentally FUBARed the tests, and then discovered that running one unit test gave 50% line coverage in Rails.
We have big libraries nowadays, and we use them...
Unless you have build servers capable of rebuilding all Qt, WebKit etc. and performing an LTO link (which pulls in all build artifacts in form of bitcode archives/objects) in a reasonable amount of time (big reason buildlabs exist - it takes a long time), LTO is not likely to be suitable, it's an extremely expensive optimization that essentially defers all real compilation until the link step at which the linker calls back into libLLVM/libLTO and have them do all the heavy lifting.
At the very least you need a workstation grade machine to be able to do that kind of stuff on regular basis, you really can't expect everyone to have that. And there's a reason libLLVM.so is usually dynamically linked, it cuts a massive amount of time spent on builds, which is especially useful while developing and it's a middle ground between building all LLVM and Clang libraries as shared objects and having to wait for static linking of various LLVM modules into every LLVM toolchain binary (which tends to result in the toolchain being much much bigger). The build cache with shared libLLVM.so for Clang/LLVM/LLD builds is around 6-7GB (Asserts/Test builds). Statically linking LLVM modules blows that up to 20GB. God forbid you actually do a full debug build with full debug information with that.
That's a terrible argument against dynamic linking. That's not to say static linking is bad, in fact, recently it's been making a comeback for exactly that reason - LTO and next-generation optimizers. But saying LTO makes static linking viable for everyone including consumers is somewhat far fetched.
I of course do not argue doing LTO while developing, it seemed clear for me that the context of the whole thing is about what's released to users.
If you look at the transitive dependencies and then only link in the code that is actually reachable, I doubt that static linking has a significant impact. (I don't know to which degree this is possible with today's toolchains, but I suspect that many libraries have some inefficiencies there due to missing dependency information). As an uneducated estimate (from someone who has written compilers and GUIs entirely from scratch) I'd say 100K of additional statically linked object should be enough for almost any desktop application.
> the linkage to the kernel is also dynamic
Not an expert here, but you don't link to the kernel at all. The interface is via syscalls. At least on Linux, to help decrease inefficiencies at the syscall boundary, there is some magic involving the so-called VDSO, though - which is indeed dynamically linked, but shouldn't be very large either (On my system it seems the [vdso] and [vsyscall] mappings (both executable) both have size 8K. There is also a non-writeable and non-executable [vvar] of size 8K and I guess it links to live variables written by the kernel).
This is completely misguided. For a modern UI library it is very very hard to figure out what code is actually used.
Say we load the ui description from an xml file. What controls get used in the app then? Maybe the xml has a webview control in it? Then we can't drop the webview dependency.
Then your app needs to load an icon, which typically goes via some generic image loading API. What types of loaders are there? Can we get rid of the gif loader (and dependencies)? If any single image file reference is not hardcoded in the binary, again we can't.
Want to script your application in lua/python/js, and let it call into the UI library? Then we have to assume that basically any code can be reached.
Static linking like this just is not realistic.
Maybe it's possible, though, if we remove a little flexibility and flatten the dependencies, such that the application hardcodes a little more of what actually happens (e.g. specificies the file formats for image loading, the format of the ui description, etc).
This is the way I've done my own GUI application, where I simply coded what should happen, and created widgets as I needed them, and all the important decisions were known statically. But maybe this amount of specification is not what people expect from a standalone GUI library.
For instance you end up with things like ui designer tools that produce manifest files, where you just expect that some non-programmer can edit it and load it into your app without having to rebuild the toolkit.
In your example, the "OS" is a program on it's own, it's controlled by a single vendor and released/updated at the same time. So dynamic linking it fine there
For a downloadable program statically linking Qt would seem prudent however, otherwise an incompatible version on the OS would render the program unusable.
You see this design decision nearly everywhere where legacy is not too big a concern: Go and Rust statically link its runtime and dedendencies in executables, JVM consumer apps are usually bundled with a specific JVM, containerization etc.
 edit: masklinn correctly pointed out that usually the kernel is still shared between containers and the host OS
Containers specifically do not statically link the kernel, the entire point is to have multiple isolated userlands use the same underlying kernel. That's why they're so cheap.
This reminds me of early web servers and CGI's. A CGI can be relatively cheap to run as long as you write it in C and don't link in too much; not so much for a scripting language.
Redundancy can happen at multiple layers, so deduplicating at one layer might not be enough, depending on the situation.
So maybe not kernel, but the entire userland/OS of the app is still statically linked with containers.
If the host OS is different than the container's you need either an intermediate hypervisor / VM (pretty sure that's what happens for docker on OSX, and why it's much more expensive than on linux) or for the host to provide an ABI compatible with the VM's OS/expectations (WSL, smartos's branded zones).
Either adds even more "dynamic linking" to the pile.
> So maybe not kernel, but the entire userland/OS of the app is still statically linked with containers.
Yes, a container's userland should be statically linked.
Ironically, this is what Plan9 does out-of-the-box. And even cheaper.
While it's practically not possible to compare performance, just the fact that this is a part of the initial design and a core of Plan9 vs adhoc feature (actually ported from Plan to Linux) is enough to speculate on performance.
>Linux namespaces were inspired by the more general namespace functionality used heavily throughout Plan 9 from Bell Labs.
It really doesn't. For all we know, because the plan9 version has to be more general it is much slower than the ad-hoc version.
Shipping prepackaged images with self-supplied libraries is functionally equivalent to shipping a blob linked statically. But the implementation is different, in a container, each program still loads the self-supplied libraries using the loader, and this difference matters. Only dynamically-loaded libraries with PIE executables can enjoy the fully benefit of Address Space Layout Randomization. But statically-linked executables cannot, and they leave fixed offsets that can be exploited directly by an attacker.
> Security measures like load address randomization cannot be used. With statically linked applications, only the stack and heap address can be randomized. All text has a fixed address in all invocations. With dynamically linked applications, the kernel has the ability to load all DSOs at arbitrary addresses, independent from each other. In case the application is built as a position independent executable (PIE) even this code can be loaded at random addresses. Fixed addresses (or even only fixed offsets) are the dreams of attackers. And no, it is not possible in general to generate PIEs with static linking. On IA-32 it is possible to use code compiled without -fpic and -fpie in PIEs (although with a cost) but this is not true for other architectures, including x86-64.
Unless we stop writing major programs in C one day, and/or formally verify all programs in the system and provide a high-assurance system that most people would use, using exploit mitigation techniques is still the last line of defense to limit the damage created by a memory-safety bug.
After thinking for a while, I think should change my stance, and support containerized applications such as Docker or Flatpak despite it's an ugly blob. A buggy and ugly container with prepacked applications linked dynamically to vulnerable blob with PIE and ASLR, is less of a evil than a buggy and ugly executable, linked statically to vulnerable blob with fixed addresses.
Static linking would never cross process boundaries, would optimize for modules that are actually called from the program that the libraries are being linked to and has absolutely nothing to do with IPC.
Dynamic linking is useful, but not always necessary and not always appropriate and does not result in 'calculator apps that are hundreds of megabytes', it might be 10Mb or so, which is still large but not comparable.
I really think that the linked post throws the baby out with the bath water, library versioning issues can be handled by the package manager to become effectively non-issue. DLL hell is a thing on Windows because it lacks a proper package manager. On a case-by-case basis applications that can't be packaged properly (proprietary binaries for instance) can be statically linked.
In my experience that's how it works on Linux and for the BSDs and as far as I'm concerned it hasn't caused many issues. Maybe I've just been lucky.
If I have this wrong, someone please correct me.
This is not true in general.
For this code:
By using -ffunction-sections -fdata-sections -Wl,--gc-sections (per https://news.ycombinator.com/item?id=19170667) I get this down to 804 KB. Still not quite what you suggest.
 Clang might just be using the same linker as GCC here, and you might blame including the extra stuff on the linker.
$ musl-gcc test.c -O2
$ ls -laB a.out
.rwxr-xr-x 15,776 roblabla 15 Feb 16:49 a.out
$ musl-gcc test.c -O2 -ffunction-sections -fdata-sections -Wl,--gc-sections -static
$ ls -laB a.out
.rwxr-xr-x 17,864 roblabla 15 Feb 16:48 a.out
It's true that without proper care, you might end up importing a whole lot of useless stuff though.
FWIW, I believe (but could very well be wrong) the reason that this works this well with MUSL is that every function is defined in its own translation unit. I doubt that Qt and KDE do that.
uClibc and musl libc have much better support for static linking, and you will be able to make a much smaller binary with them than you would static linking against glibc.
$ gcc -O3 foo.c -flto -static -o foo
$ du -sh foo
$ strip foo
$ du -sh foo
Question: In the right system, does discretion even matter? Eg, NixOS - where linked libraries and dependencies are defined by content addressing. Should we use discretion with linked libs in a system like that?
> Now imagine you upgraded the kernel. You'd have to rebuild EVERY SINGLE THING on the system to build with that new kernel, or a new version of KDE or QT or X or anything in between.
You could though, that's pretty much what unikernel systems do.
You'll still likely have dynamic linkage of sort to the hypervisor or (ultimately) the hardware.
And when I say nice, I mean absolutely essential. Ive had glibc go south before... And then nothing works. Its a "hope you can boot with rescue media and hope and pray the old glibc is in your apt/rpm cache".
But having the rest of the system being dynamic linked makes a great deal of sense for the reasons you stated.
like containers? surely this is the same thing as you suggest, and is a good solution.
> The symbol versioning breaks assumptions users have about how shared
libaries work -- that they provide a link to one version of a function and
if you replace the library all the programs get fixed. I've seen this
problem in practice, for both naive users and very non-naive sysadmins.
I have also designed it that way to do live coding in C and I have made a similar library for live coding python .
I am addicted to DLLs , send help :D !
I think a lot of code highlighting is basically ineffectual, though, given that it (almost always) highlights based on keyword-based regex, rather than the semantic meaning of the code.
I contest that they highlights based on keyword-based regex too. I haven’t formally analyzed TextMate grammars, but I believe the scoping behaviors take them into context free grammar territory.
I don't think I would care one bit if my syntax highlighter stopped making keywords blue and type names green. But damn, would it get frustrating not having dumb quoting mistakes stand out right away. I like fixing those based on immediate visual feedback as I type, vs looking at a surprise compiler error next time I try to build.
Which of the below string expressions (C# as example language, many are similar) has a mistake?
"Most strings are " + easyOrHard + " to read and " + concatenate + " with variables
"Some, like \"" + onesWithQuotes + "\" are trickier"
"Your friend \" + friend.Username + "\" has joined the chat"
@"In C# multiline string literals denoted with @, \ is a legal character on its own and """" is used to escape the "" character"
@"You're probably a jerk like my buddy """ + name + "" if you mix syntaxes like this"
> printf("this is a test\n"
> "this is \"another\" test\n"
> "hello world\n");
You're right though, I do have to pay more attention in strings. Paying more attention doesn't seem like a negative thing to me, though.
Could be correlation rather than causation. Presumably, you're a more experienced programmer now than you were three years ago and you might be making fewer errors and paying more attention regardless of whether you turned off syntax highlighting.
Seems like the Plan 9 guys haven't heard of these? If done properly, dynamic linking is almost always better than static -- key word here is "properly", which doesn't always/often happen.
Also, this is the definitive classic take on Dr. Epper.
Symbol versioning does not break things in this way. If you've replaced your only libc.so with a fixed one, there is no other libc.so for dynamically linked executables to link with at runtime. If the new one brought a needed fix, the bug is fixed everywhere using libc.so.
It's not like the new library bundles old bits to fulfill old versions of symbols, that's madness.
The only way I can see someone getting into a situation implied by the article is when they install multiple versions of a shared library simultaneously, keeping vulnerable versions around, and have executables continuing to use the old copies. This is an administrative failure, and says nothing of dynamic linking's value.