
Rich Felker of musl libc comments on Google's LLVM libc proposal - maximilianroos
http://lists.llvm.org/pipermail/llvm-dev/2019-June/133308.html
======
ajxs
> A corporate-led project does not have to answer to the community, and will
> leave whatever bugs they introduce in place for the sake of bug-
> compatibility with their own software rather than fixing them.

This speaks volumes very clearly. This highlights an immense hazard.
Enterprise scale companies contributing to open-source is a fantastic thing,
but enterprise scale companies thrusting their own proprietary libraries onto
the open-source world is not. I'm already actively avoiding becoming beholden
to Google in my work as it is already, let alone in the world where important
software uses a libc written by Google.

If you're not concerned by this, refer to the immense power that Google
already wields over the extremely ubiquitous web-standards through the market
dominance that Chrome has.

~~~
jchw
(Disclosure, Googler here, not working on projects related to any of the
things being discussed, speaking in a purely personal capacity.)

I hate to disagree with Rich Felker. I think most people have a lot of respect
for musl and the amount of effort it took. But, I disagree with the basic
conclusions.

For one thing, I don't think it's fair to compare this to Chrome, an end user
product. libc is a fundamental piece of architecture, like (edit: as in,
fundamental in stacks that they are used in, to clarify) protobufs, gRPC,
Kubernetes, etc. I heavily sympathize with folks uneasy about the Chrome
situation - like I've mentioned maybe hundreds of times now I use and
recommend Firefox personally, but I feel this particular discussion will
become more productive when that stops being a focal point.

Their other points are not really related, but I disagree with them also. I
don't think there's a good motivation to keep bugs for backwards
compatibility: Google runs non-Google software, too; keeping subtle
incompatibilities could be a disaster. Also, Google has a pretty good corpus
of code to test their libc on, so I think they have a unique opportunity to
fast-track a pretty battle-tested libc (I did not know of this project before
this Hacker News post, so this is purely a hot take, but for now I stand by
it.)

No idea what their plan is, but open governance would probably work wonders
here. After all, LLVM has plenty of corporate contributors, and it seems to do
fine.

~~~
gumby
The problem was stated in Felker's response: "...corporate development teams
are uniquely qualified to utterly botch a libc, yet still push it into
widespread use..."

This can come innocently as well. For a simple non-code example: google has a
code style guide. For C++ it says "don't use exceptions" and even (or used
to?) explains that the writers consider exceptions to be a _good_ thing but
they have enough legacy / third party code base that doesn't use them that
they can't switch. Lots of people use the google style guide because it
exists, rather than write one themselves, or rather than spend time hashing
one out. So they end up shackling themselves to a constraint entirely
resulting from an environment _they are not in_.

I see this indirectly in your own comment, where you casually referred to gRPC
or protobufs as "fundamental piece[s] of architecture". Well they aren't
standards, they are technologies that google likes. Protobufs is a perfect
example of a technology in, well not widespread use, but is used by a bunch of
projects, and as discussed elsewhere contains a number of decisions that
reflect the google environment, probably not resulting in a great fit for a
number of places where it is used.

There's nothing malign in that; the whole point of writing code is to meet
your needs. But when you're an 800 pound gorilla, don't presume your every
step is what everyone else needs. They are what _you_ need and without
proactive effort may not help many others....but can crowd out what could.

~~~
jchw
>I see this indirectly in your own comment, where you casually referred to
gRPC or protobufs as "fundamental piece[s] of architecture". Well they aren't
standards, they are technologies that google likes. Protobufs is a perfect
example of a technology in, well not widespread use, but is used by a bunch of
projects, and as discussed elsewhere contains a number of decisions that
reflect the google environment, probably not resulting in a great fit for a
number of places where it is used.

I am not suggesting they're fundamental to the world, rather just fundamental
pieces of architecture where they're being used at. That is to say,
fundamental in a similar (though not identical) way to a libc would be. This
is more of a miscommunication than a disagreement I think.

>There's nothing malign in that; the whole point of writing code is to meet
your needs. But when you're an 800 pound gorilla, don't presume your every
step is what everyone else needs. They are what you need and without proactive
effort may not help many others....but can crowd out what could.

So, I'm not in the know, but... To quote the original post:

> So, what do you think about incorporating this new libc under the LLVM
> project?

That would, to me, imply under LLVM governance, not Google, and I think that's
the primary thing people should focus on if they are worried. Kubernetes is a
good example of where I think this ended up going extremely well; Kubernetes
is under control of CNCF and not Google.

Moving governance elsewhere is surely beneficial for the project's health, and
since it's a low level piece of architecture rather than an end user product,
I see no reason why there wouldn't be motivation to do that. It appears it is
the intent to do that in the first place.

~~~
chandlerc1024
Hi, full disclosure, it's my team working on this at Google FWIW, and the
technical lead for much of Google's contributions to LLVM.

When we asked the community if they were interested in us developing this in
the open as part of LLVM, we meant as part of LLVM. We're very committed to
the LLVM open source project, and think it would be great for this to be
developed within that framework (technical, project, community, the whole
works). I'm hopeful that the community is in fact interested.

But we also were asking -- we have some specific technical goals that we'd
like to make sure we can accomplish. It really is the community's decision
whether this makes sense. =]

~~~
andrey_utkin
Hi Chandler!

Is it just me, or the original Googler's posting on llvm-dev does not explain
their motivation beyond the vague "doesn't quite address"? I guess some solid
list of problems hard to resolve could help.

Also, as you say, it's a community decision now, what will be a plan if
community NACKs the proposal? A BoringLibc from scratch? Commitment to
contribute hard fixes to glibc and/or mail?

~~~
chandlerc1024
I think it's hard to predict what a community will be interested in deep-
diving to understand. And I'm not sure that HN is super representative of its
interests either. The email was aimed at LLVM folks, not HN.

I generally trust the LLVM community to ask for any details they need to
reasonably evaluate a proposal like this, and I also trust the folks on my
team to work to address those requests to the extent we can.

I don't think it makes sense to try and speculate about what option will make
the most sense if LLVM says "nope". Generally, I plan to encourage the team to
see if there is a good way for us to address concerns the LLVM community has
while still getting the technical things we need. IMO, it would be somewhat
surprising if there were _no_ reasonable path where this could both be
reasonable for the LLVM community and Google. Doesn't mean it is impossible,
but having detailed and precise plans don't seem like a priority. IMO, the
priority is finding a good way to work with the LLVM community here.

On a more meta level, I also think it would be good for lots of folks (HN,
twitter, etc.) to be a bit less harsh in their criticism of initial posts
proposing new efforts/projects. I've seen this several times recently (ranging
from this to the V language stuff). I'd suggest folks maybe ask questions and
give people a chance to flesh out their thoughts and provide missing context
rather than hammering in feedback. In many cases, I think the feedback is
actually good, but the method of delivery makes it much harder for people to
learn from and respond to constructively.

Anyways, enough meta...

~~~
majewsky
> I've seen [harsh criticism] several times recently (ranging from this to the
> V language stuff).

It's probably not a good idea to put yourself on one level with a scammer.

Seriously though, any effort made by Google will _of course_ be scrutinized
much more vigorously than the same effort made by a smaller org or a single
person. If I decide to build my own libc, no one will care because _they
probably don 't need to care_. But big companies like Google have the power
and inertia to just shove decisions in people's faces. I think for most people
here, a large part of the dayjob is spent dealin with and working around the
stupid decisions of third-party hardware/software vendors.

~~~
gorio
There are also many example of Google doing just that to various degrees when
it comes to open source projects, security research, their own products,
dealing with vendors, customers or employment among other things. This of
course isn't something unique to Google. What is unique however is their
position.

They are probably the most powerful company when it comes computers, software
and the Internet. They have the culture, means and goodwill to do a lot more
damage than almost anyone else. I am sure that is at part of why people want
to work for Google. Not to do damage of course, but because you can make a
difference. Unfortunately making a difference for Google frequently isn't the
best for rest of us.

------
eqvinox
His 3rd argument is absolutely and utterly compelling and a total knockout
against Google's approach to this IMHO:

> Some folks have told me that I should press LLVM to make musl the "LLVM
> libc" instead of whatever Google wants to do, but that misses the point:
> there _shouldn 't be_ a "LLVM libc", or any one library implementation
> that's "first class" for use with LLVM while others are only "second class".

If they want to create a great libc - or fuck up another libc - they're much
welcome to do that, but they need to go do it _outside of LLVM_.

Dumping a libc on LLVM is just (1.) appropriating credibility from LLVM, (2.)
sticking others with the maintenance bill, and (3.) horseshit.

~~~
qalmakka
This, and I also strongly argue that compiler implementations should be untied
from standard libraries, and this also applies to libc++/libstdc++. A compiler
shipping its runtime is a bad idea that totally screws up compatibility (just
look at how painful it is to use MINGW + C++ on Windows because it can't link
with VS's C++ Runtime - I understand that MS headers are crap to parse, but
with a bit of effort clang now does it fine), and ties them together and
prevents reuse. In an ideal world shipping a runtime should be the OS's role,
not the compiler's, and the compiler should make use of whatever the platform
ships, instead of having N compilers shipping N different, incompatible
libraries. The interface between the library and the compiler would then
become a true language standard, instead of "library for compiler version
X.Y.Z"

~~~
nly
Raymond Chen wrote about this in his 2014 blog post "Windows is not a
Microsoft Visual C/C++ Run-Time delivery channel"

[https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12...](https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=1273)

Stable ABIs are good, but the ability to break it and iterate/improve is even
more so. GNU platforms are currently stuck with a shitty std::string
implementation (32 bytes but only 15 bytes of SSO capacity) because breaking
ABI means recompiling your entire Linux distro.

~~~
Asooka
As he notes there, Windows does ship a C/C++ runtime, but you're not allowed
to use that one - it is only for internal Windows stuff and only ever has to
be compatible with whatever software comes with Windows. Any Windows update
can make it binary-incompatible with a third-party program. User axjs notes in
another top-level comment that a Google libc(++) will always just serve
Google's products, including keeping bugs that a Google product relies on -
somewhat similar to the Microsoft C/C++ runtime that comes with Windows, which
you are strongly encouraged not to use.

~~~
Crinus
That ship has sailed a long time ago, anything compiled with MinGW links
against MSVCRT.DLL and did that from the beginning. Considering they had to
backpedal changes in that DLL because it broke some program, as mentioned in
the linked blog, i doubt this DLL will ever be removed as that will break
thousands of applications that have come to rely on it.

~~~
xeeeeeeeeeeenu
>anything compiled with MinGW links against MSVCRT.DLL

It has changed recently, MinGW-w64 supports linking with UCRT instead of
msvcrt.dll. Unfortunately, it's almost undocumented[1].

[1] -
[https://sourceforge.net/p/mingw-w64/mailman/message/36621319...](https://sourceforge.net/p/mingw-w64/mailman/message/36621319/)

~~~
Crinus
Considering that it isn't the default and it is only available on Windows 10
(and that only since 2015) i think that it is very safe to say that for all
intents and purposes "anything compiled with MinGW links against MSVCRT.DLL"
is valid and will be for a long time.

Though even if MinGW switched to UCRT tomorrow, it would still leave a ton of
executables since 1998 (or whenever it started using it) relying on MSVCRT.DLL
(which AFAIK is included by default in Windows since Windows 98 so it has a
slightly wider support range than UCRT :-P).

------
cpeterso
I'm surprised the proposed initial scope would include x86-64 but not ARMv7 or
AArch64. That suggests Google's motivation here is their server software, not
mobile devices running Android or Fuchsia.

Also, portability and consistency is an important attribute of a libc.
Supporting multiple architectures from the beginning would help shape a
portable design. For example, the original target processor for Windows NT
development was the Intel i860 specially because it wasn't the standard i386
architecture. Fun fact: the Intel i860 was codenamed "N10" aka "N-Ten", which
may have been the origin of the Windows "NT" moniker.

~~~
floatboth
> mobile devices running Android

They already have Bionic!

> or Fuchsia

They already have a weird fork of musl that Rich described in the email!

typical Google "Not Invented in this _Team_ " approach..

~~~
pjmlp
Bionic was a very hard child birth.

Check NDK roadmap to see how much years (yes years) has taken them between
Android and NDK releases to provide a proper libc.

------
nemothekid
What does “LLVM implementation of Libc” mean here? Does that just mean an
implementation of libc maintained by the LLVM project?

~~~
noncoml
Sounds like an LLVM "recommended" libc?

Loved the third point in Rich's reply:

"there is tremendous value in non-monoculture of libc implementations, or
implementations of any important library interfaces or language runtimes.
Likewise there's tremendous value in non-monoculture of tooling (compilers,
linkers, etc.). Avoiding monoculture preserves the motivation for consensus-
based standards processes rather than single-party control (see also: Chrome
and what it's done to the web) and the motivation for people writing software
to write to the standards rather than to a particular implementation. A big
part of making that possible is clear delineation of roles between parts of
the toolchain and runtime, with well-defined interface boundaries. Some folks
have told me that I should press LLVM to make musl the "LLVM libc" instead of
whatever Google wants to do, but that misses the point: there _shouldn 't be_
a "LLVM libc", or any one library implementation that's "first class" for use
with LLVM while others are only "second class"."

~~~
wyldfire
> there is tremendous value in non-monoculture of libc implementations

> any one library implementation that's "first class" for use with LLVM while
> others are only "second class"."

I agree with Zachary here. LLVM's really good at mixing and matching w/target
libs IMO (and linkers too, while we're at it). FWIW lld and libc++/abi have
been a part of the llvm project for a long time now and BFD ld and libstdc++
are still the defaults for the linux clang driver.

~~~
bigcheesegs
Correct. clang is always going to default to targeting whatever the platform
ecosystem is. I don't see linux distros moving away from glibc any time soon.

The notion that a LLVM libc would in some way make other libc's "second class"
is completely unfounded. libc++ vs. libstdc++ is proof of this.

------
rjvehn
Well glibc kind of proved that individual maintainers can be just as bad for
stewardship[0].

[0]:
[https://news.ycombinator.com/item?id=7914022](https://news.ycombinator.com/item?id=7914022)

------
xvilka
Quite relevant then: "Why I'm donating $150/month (10% of my income) to the
musl libc project"[1].

[1] [https://andrewkelley.me/post/why-donating-to-musl-libc-
proje...](https://andrewkelley.me/post/why-donating-to-musl-libc-project.html)

------
new_realist
Why not start with musl?

~~~
gpm
This is a common complaint about companies starting new open source projects,
and it really annoys me.

Anyone who works with software knows there are tons of very good reasons why
you might want to start a new project from scratch instead of contributing to
an existing one. You might find the existing architecture limiting. The trade
offs taken to be non optimal (for your use case or in general). Consider the
existing code to be low quality. The code contribution process to lax
(introducing bugs) or strict (slowing work). Find the community toxic. Want to
utilize existing internal code bases. Etc.

There are tons of valid reasons to make a new project, and for most of them
you don't want to say it out loud. Saying "your code sucks" or "we don't like
you (i.e. toxic community)" is rude. Saying "we don't like the processes" is
asking for a long political fight, and likely to be taken as rude by the
people who invented them. Saying that they want to utilize internal code bases
could be problematic internally. Etc.

In this case I get the feeling that quite a bit of it is disagreeing with the
tradeoff's musl took (see: Wanting to make libc more like a library), which
has to be about the only reason you are allowed to publicly say. Maybe there
are other reasons as well, maybe not.

Sorry to jump down your throat, it's just a common type of comment that I
wanted to give a general reply to. (Edit: To be clear I'm not involved with
this project, or a google employee for that matter)

~~~
nine_k
This is reasonable.

A simple answer like "we are not starting with musl because ..." would be
quite welcome, though.

~~~
yc12340
I assume, that Google's answer would sound like this:

"We are not starting with musl, because we want to own all our software the
same way people own hamsters or dogs: care for them, feed them, train them,
single handedly decide whether to castrate or kill them. We don't need the
risk of owning something we don't fully control."

As far, as I am aware, musl isn't owned by public company or foundation.
Google can't exert desirable degree of control over musl project, because
there is no board of directors to buy their way into, — it is all maintained
by one guy (and chattel slavery is outlawed in US...)

~~~
AstralStorm
Bribing or hiring the dev can still work.

------
Sir_Cmpwn
Copy-pasting a comment from "cup" on lobsters, because I think it's on-point:

>1000 times yes. If Google wants their own libc that’s their business. But
LLVM should not be part of their “manifest destiny”. The corporatization of
OSS is a scary prospect, and should be called out loud and clear like this
every time it’s attempted.

[https://lobste.rs/s/dgmoyi/response_libc_llvm#c_hg8dvg](https://lobste.rs/s/dgmoyi/response_libc_llvm#c_hg8dvg)

~~~
favorited
There has been huge corporate involvement and investment in LLVM from its
earliest days as anything beyond a research project.

When Apple open-sourced Clang, or their C++ standard library implementation
and contributed them to the LLVM project, was that something to be afraid of
as well? It seems to have gone pretty well.

~~~
svnpenn
Fair point, but Google is not Apple. Google has a history of railroading
standards to suit their needs, to the point where the standard becomes "what
Google is doing". See the Chrome project.

~~~
Conan_Kudo
There are plenty of examples on Apple's side, too. To name a few: Floppy
disks. CDROM. USB. Firewire. WebKit.

------
mehdix
These are the highlights IMO:

> Avoiding monoculture preserves the motivation for consensus-based standards
> processes rather than single-party control

> there _shouldn 't be_ a "LLVM libc", or any one library implementation
> that's "first class" for use with LLVM while others are only "second class".

------
catern
One of the replies is this email from an LLVM developer working at Google:
[http://lists.llvm.org/pipermail/llvm-
dev/2019-June/133315.ht...](http://lists.llvm.org/pipermail/llvm-
dev/2019-June/133315.html)

I can't say it looks particularly good for someone at Google to be using their
moderator powers on this unrelated-to-Google list to tell Felker not to
include parts of his message which reflect badly on Google.

~~~
scarface74
I see nothing wrong with this. The last thing you want in a technical
discussion are petty personal attacks. The moderator sounds a lot like
something you would see dang say on HN.

~~~
hyperpape
Felker's comment was that a big corporation might release botched code that
contained hacks, but their influence would mean developers would be stuck
coding against it.

If that's insulting at all, it's several levels less insulting than the sort
of thing I see dang moderating.

~~~
tinus_hn
The argument would be that he is dragging in incidents and companies, which
could derail the discussion unnecessary. That’s reasonable but this is not:

> I don't think this project poses any risk there, much like Zach points out
> in his reply. Google is specifically discussing [...]

Now he’s using his moderator rights to push his opinion which unsurprisingly
defends his employer.

He should have sent two mails, one moderating and one participating in the
discussion.

------
rurban
Rich is of course totally right. There's not only the botched musl in Fuchsia,
also Android Bionic, which is a libc written in C++. It's utterly unusable.
They had a better, older, safer libc implementation based on some HP code, but
that is gone.

Some of Rich' points are moot though. Google already have their own libc's, in
Android and Fuchsia with enhanced security measures. Similar to Huawei or
Intel or Cisco. Better than Apple's freebsd libc. Now reaching out to the
community to outsource their development efforts under llvm is a good thing.
There's obviously a need, musl, bsd and glibc cannot address. I'm sympathetic
to that.

~~~
wbl
And the need is? I read the thread and I still don't know.

------
kbumsik
Doesn't google use newlib already?

~~~
altmind
Wait, weren't they using Bionic on android?

~~~
murderfs
Google is too big for the left hand to know what the right hand is doing.
Various parts of Google use bionic, newlib, musl, and multiple forks of glibc.

~~~
hinkley
So does that mean all that propaganda about them having a monorepo is
bullshit?

~~~
murderfs
Android (bionic), Chrome (forks of glibc and newlib in NaCl), ChromeOS (glibc
with patches), and Fuchsia (musl) are all in separate Gerrit repositories. The
monorepo for Google proper probably contains all of these, plus the libc used
for Google.

------
gok
This would have been a much better email without the needless stone throwing
from one libc maintainer's glass house.

------
m0zg
This sounds like Android NDK development will get quite painful in a few
years. It previously included a botched libc++ missing a bunch of features.
This is libc, but I imagine subtle feature disparities and incompatibilities
there will cause plenty of pain as well.

~~~
mrpippy
I don’t think the Android team is the interested party here, given the
original poster’s interest only in x86_64

~~~
m0zg
Google doesn't do "only x86_64". Even when I was there (years ago) at a
minimum all software had to run on PowerPC as well, and non-portable
assembly/intrinsics required explicit approval from "Eng Elders" and a
portable fallback. I wouldn't be surprised if ARM compat is also required now.

~~~
saagarjha
That might be true, but the email posted here explicitly lists x86_64 as the
architecture they are targeting initially.

------
jasonhansel
Filed under "This is why we need the GPL"

~~~
taneq
How does the GPL help with this? Google could build their own libc
replacement, and if they push it hard enough to gain widespread community
support, it could become the (or at least, _a_ ) new standard. At that point
Google is free to do whatever they want with it as long as they make it useful
enough that the community won't abandon it. The GPL just means that it could
be forked later, but if Google is still putting significant resources into
developing the main branch then no-one will switch to the forks.

------
fourier_mode
I think its primarily because Chris Lattner[0] is at Google now. I wonder even
if they pull out this effort how much of that would still be maintained if
Lattner changes jobs.

[0]:
[https://en.wikipedia.org/wiki/Chris_Lattner](https://en.wikipedia.org/wiki/Chris_Lattner)

~~~
favorited
Chris Lattner said that he has nothing to do with this project

[http://lists.llvm.org/pipermail/llvm-
dev/2019-June/133274.ht...](http://lists.llvm.org/pipermail/llvm-
dev/2019-June/133274.html)

------
_pmf_
Make vs buy decisions work differently for companies with virtually unlimited
buying power. The absolute control that doing it yourself provides cannot be
bought. And Google has to care only the platforms it wants to support (as some
guy at BSDcon said, you have Windows, Linux Intel/AMD/ARM and rounding
errors).

Do you think the 100000 cookie cutter game developers for Android will stop
developing for it because the platform libc has subtle issues? They rightfully
don't give a shit.

------
lone_haxx0r
Google are going to make it with or without LLVM's cooperation.

If it's done with LLVM, it's highly probable that LLVM won't let them _extend_
, _extinguish_ libc. Google's code will be carefully evaluated by a relatively
neutral party: LLVM.

If Google does it alone, they will write whatever code they want. They'll be
free to extend, extinguish libc.

~~~
saagarjha
Why do you think Google has the ability to EEE libc by themselves but somehow
be voluntarily reigned in by LLVM?

