
Building a Better Go Linker - rjammala
http://golang.org/s/better-linker
======
cpeterso
"The original linker was also simpler than it is now and its implementation
fit in one Turing award winner’s head, so there’s little abstraction or
modularity. Unfortunately, as the linker grew and evolved, it retained its
lack of structure, and our sole Turing award winner retired." :)

~~~
luuio
For those out of the loop, who are they talking about here?

~~~
p7IDD243
Perhaps Ken Thompson?

~~~
rjammala
Kenneth Lane Thompson is an American pioneer of computer science. Thompson
worked at Bell Labs for most of his career where he designed and implemented
the original Unix operating system.

Books: Queen and Pawn Against Queen, Roycroft's 5-man Chess Endgame Series

Awards: Turing Award, National Medal of Technology and Innovation, Japan Prize

------
slx26
I love to see that at the end the document the plugins package is mentioned.
Currently it doesn't have support for Windows (among other issues), so it's
really good to hear that the work in the linker might eventually improve the
situation with plugins too, and that at least it's being kept in mind. On
golang, interfaces being implemented implicitly is not everyone's cup of tea,
and I can understand why, but when you combine that property with plugins, you
can make really powerful plugin systems that use very simple code to work.

~~~
nappy-doo
Plugins are hard, and very OS dependent. There's plenty about the current
system that is fragile, and likely to cause problems. I think it's likely the
linker rework needs to happen before plugins can be properly addressed.

~~~
jart
This doc offers great depth on why dynamic shared objects are so hard:
[https://software.intel.com/sites/default/files/m/a/1/e/dsoho...](https://software.intel.com/sites/default/files/m/a/1/e/dsohowto.pdf)
and that's just for Linux & co. I'm surprised the Go team chose to add support
for DSOs.

------
jstimpfle
I don't get it. A linker's task should be straightforward. In essence, it
looks up addresses from strings, whose number is bounded by the lines of code
written by humans. I think that there must be a lot of incidental complexity
if that task somehow becomes a bottleneck.

And how can it be that a binary called "cmd/compile" has 170k symbols (that's
like, global definitions, right?). Not that that's a huge number in terms of
today's computing power, but how many millions of lines of source code does
that correspond to?

Still, 1M relocations, or 34MB of "Reloc" objects, as indicated, shouldn't be
a huge issue to process. Object files should have minimal parsing overhead. Is
there any indication how long this takes to link? Shouldn't 1 or 2 secs be
sufficient? (assuming 100s of MB/s for sequential disk read/write, and < 1us
to store each of the 170k symbols in a hashmap, and < 1us to to look up each
of the 1M of relocations).

\- I don't think mmap should be used if it can be avoided. It means giving up
control over what parts of the file are loaded in memory. And from a semantic
point of view, that memory region still must be treated differently, since on-
disk and in-memory data structures are not the same.

~~~
wahern
> that's like, global definitions, right?

No, not just global definitions. Closures need linking, too. But the linker is
doing much more than linking function entry points. Many automatic (on the
stack) variables need linking so the GC can (a) trace the object graph and (b)
move them when resizing the stack. Likewise, type definitions require metadata
generation for GC tracing. And then there's all the debugging data that needs
to be generated, which basically involves everything.

~~~
jjtheblunt
What do you mean by closures, and by linking of closures?

[I've implemented a Scheme before, in C, for context, so I'm wondering if I'm
reading what you wrote with common vocabulary.]

~~~
wahern
The OP was referring to global symbol definitions and the first thing that
came to mind were closures, which are generated as implicit global function
definitions taking a context argument. Because Go supports lexical closures as
first-class objects, they're not uncommon in typical Go code. (Though my
experience is limited as I don't use Go regularly.) Point being, there can be
hidden, effectively global function definitions. I'm sure that only accounts
for a small fraction of the total symbols; it's just the first thing that came
to mind, and one of the easiest to understand if your notion of a linker comes
from what a Linux runtime linker does for C ABI-based object code.

------
derefr
> its implementation fit in one Turing award winner’s head

What a great unit of measure :)

~~~
ChrisSD
Wait, how many London buses fit in a Turing award winner's head? And can they
all fit in to Wales despite the number of Olympic sized swimming pools already
there?

~~~
ggm
Convert to Sydney harbours and add a banana for scale

------
AlexB138
Any recommendations on resources to better understand the build process for
compiled languages in general, and Go in particular?

~~~
ori_b
[https://www.amazon.com/dp/1558604960](https://www.amazon.com/dp/1558604960)

~~~
journalctl
I just bought this! It’s fantastic and definitely still more than relevant for
understanding the subject.

------
Solar19
I like it. What would also be cool is another Go compiler and linker,
developed by a different team not at Google. It would be good for the
ecosystem and performance if there was more than one compiler and runtime.

I think there would be a market for a proprietary compiler and maybe an IDE to
go with it — if the performance was better than the open source one. I think
this is achievable because as good as Go's performance is now, there's still a
lot of headroom. Google isn't exploiting modern CPUs very well, and the linker
is not doing extensive LTO.

The biggest constraint is the blazing fast compile times. A compiler and
toolchain that was able to take some time for optimization might deliver
markedly better runtime performance.

~~~
ksherlock
gccgo perhaps?

~~~
sdrothrock
I don't think that would be appropriate since Golang isn't (to my knowledge)
under the GNU license.

Maybe gocc?

~~~
jerf
I think you may misunderstand the nature of the post:
[https://golang.org/doc/install/gccgo](https://golang.org/doc/install/gccgo)

It's a thing that exists, not a proposal.

~~~
sdrothrock
I did, in fact! Thanks. :)

I thought it was just spitballing an alternative name.

------
benesch
What I would give for the developers of the Go toolchain to have spent the
last decade improving GCC or LLVM instead of their own bespoke toolchain.

In many ways Go seems like an excuse for Google to fund the continued
development of Plan 9. Three of the five most influential people on the Go
team (Ken Thompson, Rob Pike, and Russ Cox) were heavily involved in Plan 9.
And it shows. Go's toolchain is a direct descendant of the Plan 9 toolchain;
in fact, the Go language is really just an evolution of the special C dialect
that Plan 9 used [1]. Indeed, for a while, the Go compiler was written in this
special dialect of C, and so building Go required building a C compiler (!)
that could compile this custom dialect of C, and using _that_ to compile the
Go compiler [2].

By all rights, Plan 9 was an interesting research project, and seems well
loved by those familiar with it. (I'm not personally familiar; it was well
before my time.) But it never took off. What we ended up with is Linux, macOS,
and Windows.

Go very much _wants_ to be Plan 9. Sure, it's not a full-fledged operating
system. But it's a linker, assembler, compiler, binutils, and scheduler. All
it asks of the host system is memory management, networking, and filesystem
support, and it will happily replace your system's DNS resolution with a pure
Go version if you ask it to [3]. I wouldn't be surprised if Go ships its own
TCP/IP stack someday [4].

This is, in my opinion, craziness. What other language ships its own
_assembler_?! [5] To make matters worse, the assembly syntax is largely
undocumented, and what is documented are the strange, unnecessary quirks, like

> Instructions, registers, and assembler directives are always in UPPER CASE
> to remind you that assembly programming is a fraught endeavor. (Exception:
> the g register renaming on ARM.)

> In the general case, the frame size is followed by an argument size,
> separated by a minus sign. (It's not a subtraction, just idiosyncratic
> syntax.)

> In Go object files and binaries, the full name of a symbol is the package
> path followed by a period and the symbol name: fmt.Printf or math/rand.Int.
> Because the assembler's parser treats period and slash as punctuation, those
> strings cannot be used directly as identifier names. Instead, the assembler
> allows the middle dot character U+00B7 and the division slash U+2215 in
> identifiers and rewrites them to plain period and slash.

The excuse for the custom toolchain has always been twofold, that a) LLVM is
too slow, and fast compiles are one of Go's main features, and b) that the
core team was too unfamiliar with GCC/LLVM, at least in the early days, and
attempting to build Go on top of LLVM would have slowed the speed of
innovation to a degree that Go might not exist [6].

I've always been skeptical of argument (b). After all, one of Go's creators
literally won a Turing award, as this document not-so-subtly mentions. I'm
quite sure they could have figured out how to build an LLVM frontend, given
the desire. Rust, for example, is quite a bit more complicated than Go, and
Mozilla's developers have had no trouble integrating with LLVM. I suspect the
real reason was that hacking on the Plan 9 toolchain was more fun and more
familiar—which is a very valid personal reason to work on something! But it
doesn't mean it was the right strategic decision.

I will say that (a) is valid. I recently switched from writing Go to writing
Rust, and I miss the compile times of Go desperately.

That said—and this is what I can't get past—the state of compilers would be
much better off if the folks on the Go team had invested more in improving the
compile and link times of LLVM or GCC. Every improvement to lld wouldn't just
speed up compiles for Go; it would speed up compiles for C, C++, Swift, Rust,
Fortran, Kotlin, and anything else with an LLVM frontend.

In the last year or so, the gollvm project [7] (which is exactly what you'd
expect–a Go frontend for LLVM) has seen some very active development, and I'm
following along excitedly. Unfortunately I still can't quite tell whether it's
Than McIntosh's 20% time project or an actual staffed project of Google's,
albeit a small time one. (There are really only two committers, Than and
Cherry Zhang.) There are so many optimizations that will likely never be added
to gc, like a register-based calling convention [8] and autovectorization,
that you essentially get for "free" (i.e., with a bit of plumbing from the
frontend) with a mature toolchain like LLVM.

There are not many folks who have the knowledge and expertise to work on
compilers and linkers these days, and those that do can command high salaries.
Google is in the luxurious position of being able to afford many dozens of
these people. I just wish that someone with the power to effect change at
Google would realize that the priorities are backwards. gccgo/gollvm are where
the primary investment should be occurring, and the gc toolchain should be a
side project that makes debug builds fast... not the production compiler,
where the quality of the object code is the primary objective.

[0]: [https://dave.cheney.net/2013/10/15/how-does-the-go-build-
com...](https://dave.cheney.net/2013/10/15/how-does-the-go-build-command-work)

[1]:
[http://doc.cat-v.org/plan_9/programming/c_programming_in_pla...](http://doc.cat-v.org/plan_9/programming/c_programming_in_plan_9)

[2]:
[https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdw...](https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdwTuF7WWLux71CYD0eeD8/edit)

[3]: [https://golang.org/pkg/net/](https://golang.org/pkg/net/)

[4]: [https://github.com/google/netstack](https://github.com/google/netstack)

[5]: [https://golang.org/doc/asm](https://golang.org/doc/asm)

[6]:
[https://golang.org/doc/faq#What_compiler_technology_is_used_...](https://golang.org/doc/faq#What_compiler_technology_is_used_to_build_the_compilers)

[7]:
[https://go.googlesource.com/gollvm/](https://go.googlesource.com/gollvm/)

[8]:
[https://github.com/golang/go/issues/18597](https://github.com/golang/go/issues/18597)

~~~
enneff
> the state of compilers would be much better off if the folks on the Go team
> had invested more in improving the compile and link times of LLVM or GCC

It never would have happened. How do you motivate people whose principal
frustration is the state of C++ to work on a large C++ codebase?

Heterogeneity is a huge benefit to any ecosystem. Improving existing things is
great, but building new things is also very important. Go would simply not
exist today if it were built on LLVM or GCC.

~~~
benesch
Let me refine my point a bit. Apologies; my original post had gotten a bit
long.

I agree that enthusiasm is important! And indeed, for the Go creators, their
particular leanings might have been such that they couldn't get excited about
building an LLVM/GCC frontend, and adapting the Plan 9 toolchain is literally
the only way those three could have Go gotten off the ground. As a member of
the Go team, you'd certainly know better than I.

But Go is long past a personal passion project. Go is over ten years old. Go
likely has over a million developers [0]. Go 1.0 has been stable for about
seven years, and the first meaningful changes to the language are just now
being talked about. In my opinion, it is several years past due for Google to
start investing seriously in a Go toolchain based on a mature compiler stack.

I realize the audacity of this claim and I don't make it lightly. But if I had
the money to spend on a team of developers, I would spend it making llvm-as
and lld fast enough and stable enough to be Go's assembler and linker, and
abandon the custom Plan 9 ones.

> It never would have happened. How do you motivate people whose principal
> frustration is the state of C++ to work on a large C++ codebase?

Well, for one, once the language gets off the ground, you can write the
frontend in the new language. Rustc manages to be almost entirely Rust, for
example.

> Heterogeneity is a huge benefit to any ecosystem. Improving existing things
> is great, but building new things is also very important.

I agree, and I think Go is an interesting contribution to the P/L
landscape—essentially it proved that stripping away a good deal of complexity
(generics, inheritance, etc.) results in a very useful, highly productive
language. But I don't think Go's custom assembler and linker are meaningfully
contributing to the ecosystem. They're useful presently in that they improve
Go developers' productivity with ultra-fast builds, but they're not suitable
for use by anything but Go. Improvements to Go's linker and assembler benefit
only Go. Improvements to lld or gold can benefit practically everyone using a
compiled language.

[0]:
[https://research.swtch.com/gophercount](https://research.swtch.com/gophercount)

~~~
tylerl
> it is several years past due for Google to start investing seriously in a Go
> toolchain based on a mature compiler stack.

That's a bit presumptuous, prescribing specific implementation details based
on the fact that it's "several years past due" that they replace their tech
stack with one you would like to see improved.

Remember there already is a mature C compiler alternative for Go: gccgo.
There's also already a first-party LLVM based Go toolchain created by Google
(gollvm). Whatever hypothetical benefits you might presume would emerge from
this kind of synergy already exist. But the community mostly isn't interested.

Also, Google also already invests a massive amount of resource into LLVM. In
fact, the principal author of LLVM and Clang works at Google. But even when he
was at Apple they were already shoveling resource into the project.

~~~
benesch
> Remember there already is a mature C compiler alternative for Go: gccgo.
> There's also already a first-party LLVM based Go toolchain created by Google
> (gollvm). Whatever hypothetical benefits you might presume would emerge from
> this kind of synergy already exist. But the community mostly isn't
> interested.

I suspect the community is uninterested because it’s hard to be interested in
compilers that a) compile slower than gc, and b) produce slower code than gc.
That's a worse compiler on all fronts!

For gccgo or gollvm to be useful, they need to provide some benefit. I suspect
we'll see (b) fixed within a year. GCC/LLVM have far more optimizations than
gc, and so it's mostly a matter of plumbing enough information from the Go
frontend into the LLVM optimizer to unleash its full power.

I don't expect we'll see (a) fixed, unless something changes at Google.

> Also, Google also already invests a massive amount of resource into LLVM. In
> fact, the principal author of LLVM and Clang works at Google. But even when
> he was at Apple they were already shoveling resource into the project.

Yes, but Chris Lattner is not actively working on speeding up LLVM. His big
project lately has been supporting TensorFlow via MLIR.

------
NieDzejkob
The article often mentions how the long-lived objects are straining the
garbage collector. Wouldn't a generational GC solve this?

~~~
remus
Potentially, but then the linker is just one use case for the GC and GCs are a
game of trade offs so switching to a generational GC would impact other use
cases.

------
warent
For anyone having problems loading the doc due to a traffic overload, here's a
copy: [https://docs.google.com/document/d/1y3k3nLlPpoMf_RAfi-
FCpZpt...](https://docs.google.com/document/d/1y3k3nLlPpoMf_RAfi-
FCpZptDZfrv9Y8W7OzQjfhYJs/edit?usp=sharing)

------
miki123211
Anyone else is getting a "you can't view this page because you're offline"
error? I don't know why this is happening, how does it even load when I'm
supposedly offline?

~~~
warent
It's probably being overloaded with too much traffic. The link is a Google
Doc, and singular google docs aren't really great at handling traffic spikes.

------
jacobush
I hoped they would fix how the linker uses private OS calls.

~~~
nappy-doo
Which private OS calls. Can you site what they are?

~~~
kccqzy
I'm guessing GP refers to the fact that Go binaries use raw system calls even
on operating systems where the syscall boundary is considered a private
implementation detail (which is the case on Windows and macOS).

~~~
4ad
The Windows port never used raw system calls.

The Solaris port never used raw system calls.

The MacOS port doesn't use raw system calls since go 1.11.

The remaining ports for Linux/BSDs/Plan 9 have a stable syscall interface.

------
nn3
Who would have thought that statically linking hundreds of MB worth of
binaries every build causes problems?

Maybe some of it can be fixed with a lot of engineering effort, but the fact
remains that it is a bad idea to redo so much work every time.

Maybe the people who invented shared libraries had a point after all.

Of course the real problem is software and dependency bloat, but that is
unlikely to ever get fixed.

~~~
dullgiulio
Besides the fact that static linking has a lot of benefits, shared libraries
need to be linked too: just this process is called "loading" and the linker is
called "loader".

Worse, this needs to happen at every executable invokation.

