
About Musl - peter_d_sherman
https://musl.libc.org/about.html
======
npx
The musl community is one of my favorite things on the internet. It's obvious
that many members of it could be making a generous salary at any number of
companies but they're just hanging out on IRC because they don't give a shit
and they love software.

I don't get the sense that they think they are infallible, just that they are
right far more often than not. You can't achieve the sort of spartan aesthetic
that you see in the musl codebase by handling things like everyone else. The
ability to distill things to their essence betrays a very high level of
understanding.

As they say in the streets, the dope sells itself.

------
stefan_
The libc situation is pretty unfortunate. You can go with glibc for
compatibility, but ideally you don't actually use the upstream version but
copy the one from Debian that has crucial fixes that most of the world uses
that they didn't bother to upstream because for a while there, glibc was
maintained by the biggest asshole in all of software engineering.

On the other hand you have musl, and we can all appreciate its design goals,
but then it is similarly maintained by people that believe they and their work
are utterly infallible. Hence why to this date they have not made it possible
to _detect_ musl and would rather cite you a POSIX standards meeting from 1980
than fix an incompatibility _everyone else handles differently_. Just ctrl+F
musl on [http://landley.net/toybox/](http://landley.net/toybox/) for some
insight.

The last time I felt like smashing the desk because of musl was when I
realized you can't stacktrace a thread in a syscall because they refuse to
annotate their 10 lines of assembler doing the syscall with the required CFI
directives. They would rather wait for someone to write a 200 line AWK script
that does it ([1]) than fix debugging on platforms like arm64.

1: [https://github.com/bminor/musl/blob/master/tools/add-
cfi.x86...](https://github.com/bminor/musl/blob/master/tools/add-
cfi.x86_64.awk)

~~~
ludocode
The inability to detect musl is indeed frustrating.

One example of this problem is in detecting C11 thread support. This is a
feature of the C library, not the compiler, but it's the compiler that sets
C11 feature flags. For threads, you can only detect that they aren't supported
by __STDC_NO_THREADS__. But of course since this can't be defined
retroactively in old versions of musl, you can't use it to detect C11 thread
support in musl. And since you can't detect musl at all, you can't even just
assume the version of musl is recent enough and turn it on unconditionally.

The musl team seems to advocate guessing at supported features by detecting
with test compilation (a.k.a. configure scripts.) They don't believe in
depending on specific versions of musl because features can be backported or
disabled. Okay fine, but then why not give us explicit means to detect it? Why
not just define MUSL_C11_THREADS or something so we can detect it at compile-
time instead of during a slow, fuzzy, non-portable configure step?

Does anyone know of a workaround for this? Does musl define any kind of
feature macro that can be used to detect C11 thread support?

~~~
tlb
Do you have a real system that uses threads if available, but can also work
without them? What does the architecture look like?

~~~
toast0
BEAM, the Erlang virtual machine, could be compiled with or without threads
until release 21 (from 2018).

------
moomin
Biggest problem for musl isn't actually a problem _with_ musl at all, it's
this: the C stdlib isn't an ABI. What this means is, something compiled for
glibc doesn't necessarily work with musl and will just crash. This problem
goes much further than you'd expect. For instance, if you use .NET 3.1 with
Alpine, you can't use Google's gRPC client (thankfully, you can use
Microsoft's).

~~~
ur-whale
This actually goes further than that as different version of glibc aren't ABI
compatible with one another.

~~~
bonzini
They are backwards-compatible (which is all you can do of course).

~~~
ur-whale
Are they ?

If seen my share of old x64 binaries refusing to run on modern linux distro,
spouting some sort of undecipherable glibc error message on startup (something
along the lines of "version `GLIBC blah' not found)

~~~
floatboth
That message happens when a new binary is run on an old glibc, not the other
way around.

~~~
ur-whale
Yes, you'd think so indeed.

Unfortunately, I've seen the exact opposite in production: older binary
running atop newer glibc and crashing with that kind of message.

glibc abi compatibility is and has been a joke.

------
peter_d_sherman
Excerpt:

"Attention to correctness

musl was the first Linux libc to have mutexes safe to use inside reference-
counted objects, the first to have condvars where newly-arrived waiters can't
steal wake events from previous waiters, and the first to have working thread
cancellation without race conditions producing resource-leak or double-close.
All of these are requirements of the specification that were ignored by other
implementations, and getting them right was a consequence of careful reading
of those specifications.

musl's entire development history has been a process of reading
specifications, seeking clarifications when corner cases aren't adequately
covered, and proceeding with extreme caution when implementing functionality
that's underspecified."

~~~
jfkebwjsbx
So basically a lawyer view of software engineering.

It is very good to adhere to specs when possible, but not always.

~~~
Koshkin
So, as Einstein could have said it, "We should follow the specs as far as
possible, but not any further."

------
buserror
I had some problems with musl last year and had to back out using it for an
embedded system. Typically for example, 2 threads listening to their own
socket bound using REUSEPORT wouldn't work; half the connection would arrive
on a thread and the second one wouldn't get anything. That scared me a bit as
I was on a deadline to deliver, so I had to quickly recompile my distro with
glibc, and the problem disappeared.

I really wish it worked as I like the lean&mean&precise approach they've been
using!

~~~
wahern
Other than #define'ing SO_REUSEPORT, libc has no role whatsoever in its
behavior. You can see this yourself by grep'ing for SO_REUSEPORT in the glibc
and musl source code. And both glibc and musl implement 1:1 threading, so it's
the kernel making all the thread scheduling and inbound connection queueing
decisions.

Your problem lay elsewhere, unless you were using a really old version of musl
that lacked the SO_REUSEPORT definition.

~~~
buserror
I do not have an explanation either. I'm more of a kernel guy so I realized it
"should" have worked but it didn't. I suspect perhaps a thread wakening issue
of some sort, as the connections were definitely 'queued' on the listen
socket. I know that that same code worked perfectly with glibc!

------
ddevault
Alpine Linux uses musl libc, and SourceHut runs Alpine on all hosts and VMs.
It's lean and mean and fast, and together with Alpine makes for a wonderfully
simple and auditable system. I wouldn't dream of using anything else in
production.

~~~
driverdan
Dependency management can be a pain. Want to install numpy? You have to use
alpine's version because compiling it from pip requires glibc.

~~~
ddevault
Using anything other than distro packages is a pain. pip is an anti-pattern
and one giant security risk.

~~~
takeda
Disagree, although pip should assume --user option by default which installs
the packages in user directory instead of system directory (some distros (was
it Debian?) modify it to work that way).

The best way of using Python though, is to create a virtualenv (since Python
3, venv module is built in which makes it straight forward, python -m venv
<directory>) and install packages inside of it. This gives you control to use
exact versions of packages you need in your applications and makes your
application not tied to the OS, so system upgrades and Python version updates
are much easier to perform.

~~~
dnautics
> This gives you control to use exact versions of packages you need in your
> applications and makes your application not tied to the OS

Is this true for drivers, too? like the coupling of cuda versions to
tensorflow versions?

~~~
takeda
Unfortunately that's one weakness, but unless you package your application
using system packager (which comes with its own set of issues[1])

If you have any dependencies that depend on system libraries there are two
options:

\- the libraries can be compiled statically, they call them manylinux wheels,
this generally works well, except if your dependencies are overlapping with
python or other packages' dependencies. Most commonly this happen if package
depends on openssl. If the compiled-in openssl is different than the version
on the system in certain circumstances python might crash. This is for example
why psycopg2 was initially distributed as manylinux but now they opt for the
second method (they still provide psycopg2-binary but they discourage its use;
most time it works fine but it is a problem on certain distros)

\- make python create bindings on installation, this makes installation
process do small compilation to create bindings between system library and
python, this makes the packages robust, but tied to the OS. In my experience
this is not an issue, but it can be annoying, because you still don't have
100% control of dependencies.

This issue is what made me investigate Nix[2] package manager, because it
gives you full control of all dependencies down to libc making everything
fully reproducible, so you can control the exact version of python to use and
all system and python dependencies.

[1] in one of my previous jobs on a team that I joined they were running their
applications on CentOS 5 that was already EOL, because rebuilding RPMs to
CentOS 7 (at the time the newest version) was a lot of work, another issue was
that they were bound to python package version that came with the system, they
could create their own RPMs, but no one did it because maintaining that was
adding more work. I spent time converting the python code to be packaged using
setuptools. Once that was done switching the OS was trivial. We finally also
could use the latest versions of many of our dependencies.

[2] This is IMO good article which describes tooling that makes Nix much more
enjoyable to use: [https://christine.website/blog/how-i-start-
nix-2020-03-08](https://christine.website/blog/how-i-start-nix-2020-03-08)

------
tyingq
_musl, pronounced like the word “mussel” or “muscle”_

Oh. I've been saying mew-sel this whole time.

~~~
moomin
I've been saying muzzle.

------
api
Alpine Linux is by far my favorite distro and I wish it had more support. Not
only is musl nice, but it lacks a huge amount of obsolete cruft and the
package manager is nice and simple and not over-engineered. It also lacks
systemd.

It probably lacks a lot of the "enterprise" cruft in CentOS/RHEL or Debian,
but that's a good thing for everyone else.

~~~
fmajid
I use Alpine as the OS on my home server (it used to be SmartOS/Illumos) and I
compile my entire stack myself using my own cross-platform (Solaris/Linux/OS
X/FreeBSD/OpenBSD) build system similar to BSD ports. While there are some
portability gotchas, by and large it's manageable.

------
Aissen
I've been using musl on embedded systems for a while, and its simplicity is
indeed quite a feature.

My biggest issue is the lack of ASan, or ever HWAsan. valgrind is nice, but
quite slow. I know Rich has been working on a new allocator that will make
ASan easier to implement, but it's still a few months away.

------
random3
We were using Alpine Docker images. Every few months the docker build would
fail with some random glibc / musl glibc-compat new issue.

The docker file is full of comments like the following:

# libc6-compat needed by outdated grpc-tools # we get # sh: node_modules/grpc-
tools/bin/protoc: not found # can be removed if we remove grpc-tools and
assume an existing protoc for dev # i.e. developers would independently intall
protoc in order to generate # grpc-web # we install separate from the packages
above to avoid a failure noticed on # 2019-11-01 # apk update && apk add
protobuf grpc gcompat libc6-compat # ERROR: unsatisfiable constraints: #
musl-1.1.20-r4: # breaks: libc6-compat-1.1.24-r0[musl=1.1.24-r0] # satisfies:
musl-utils-1.1.20-r4[musl=1.1.20-r4] #RUN apk add libc6-compat

Bottom line is that it's awesome to have something lean like Alpine, but that
can hardly justify the effort to maintain it. It feels like Linux two decades
ago.

~~~
MuffinFlavored

        # libc6-compat needed by outdated grpc-tools 
        # we get 
        # sh: node_modules/grpc-tools/bin/protoc: not found 
        # can be removed if we remove grpc-tools and assume an existing protoc for dev 
        # i.e. developers would independently intall protoc in order to generate 
        # grpc-web 
        # we install separate from the packages above to avoid a failure noticed on 
        # 2019-11-01 
        # apk update && apk add protobuf grpc gcompat libc6-compat 
        # ERROR: unsatisfiable constraints: 
        # musl-1.1.20-r4: 
        # breaks: libc6-compat-1.1.24-r0[musl=1.1.24-r0] 
        # satisfies: musl-utils-1.1.20-r4[musl=1.1.20-r4] 
        #RUN apk add libc6-compat
    
        

formatted that for you

~~~
random3
Thank you! I missed this doc
[https://news.ycombinator.com/formatdoc](https://news.ycombinator.com/formatdoc)

------
floatboth
Huh, nice domain name, is that new? It was musl-libc.org with a dash before..

~~~
saagarjha
It's fairly new, I remember there being a tweet about it but I can't seem to
find it…

------
dang
Related from 2012:
[https://news.ycombinator.com/item?id=4058663](https://news.ycombinator.com/item?id=4058663)

2014:
[https://news.ycombinator.com/item?id=7434554](https://news.ycombinator.com/item?id=7434554)

2015:
[https://news.ycombinator.com/item?id=9791709](https://news.ycombinator.com/item?id=9791709)

2019:
[https://news.ycombinator.com/item?id=19284648](https://news.ycombinator.com/item?id=19284648)

------
kbumsik
I've heard a lot of libc but not stdlib from other languages. I assume that
most of languages that depend on C (such as Python, JVM) depend on libc as
well. Is there any languages implements a runtime in C but implements its own
low-level interfaces?

~~~
ksherlock
some k-family languages do that.

[http://kparc.com/b/A.S](http://kparc.com/b/A.S)

~~~
smabie
But why?

~~~
yjftsjthsd-h
The k family likes to be the whole stack; they would view libc as an
unnecessary huge[0] dependency.

[0] To k, every other language/library/program is too big.

~~~
smabie
Seeing that libc is undoubtedly already loaded in memory as a dynamic lib on
all systems, that doesn't make much sense. Moreover kdb+/q links to libc, and
it has very good performance and memory usage.

------
rustybolt
The bad thing is, it's pretty hard to compile Linux with musl. I don't
remember the specifics, but it had to with the #define mess that glibc uses.

~~~
friend-monoid
But the kernel itself doesn’t use glibc?

~~~
alexdowad
Definitely not. The kernel has its own implementations of any standard C
functions which it needs.

~~~
einpoklum
Is that because of circular dependencies of glibc on the kernel, or something
else?

~~~
ur-whale
>Is that because of circular dependencies of glibc on the kernel

That but also because lots of code in glibc would simply not work in kernel
space

------
Chris2048
Alpine (the docker distro) uses musl, but this means that you can't use many
python wheels because they aren't musl based :-/

~~~
jpgvm
Alpine is not "the Docker distro". It predates Docker by quite some time and
was used in lots of lightweight systems.

Python wheels compiled using the manylinux infrastructure actually do work on
Alpine + musl.

That said I wouldn't recommend running Python on musl as I have seen
incredibly weird behaviour including corruption of Python's low integer cache
resulting in some truely awful consequences. i.e amounts of currency suddenly
being 10x what they should be.

~~~
Chris2048
Presumably those manylinux wheels do not have c bindings, when they do they
are glibc based.

~~~
Conan_Kudo
manylinux wheels are exclusively binary wheels with compiled parts, so they're
all glibc-based (as the manylinux ABI is glibc).

~~~
Chris2048
Then, I can't see how they work on Alpine w/ musl.

@jpgvm Do you mean they work if you manually install them? b/c pip won't
install glibc wheels on alpine.

~~~
greenshackle2
There's no guarantee that they do and I certainly don't expect any random
manylinux wheel off pypi to work on alpine. I maintain one that definitely
doesn't. We build a musl wheel too but there's no musl platform tag so we
don't have a good way to distribute it on pypi.

