
How a fix in Go 1.9 sped up our Gitaly service by 30x - rfks
https://about.gitlab.com/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/
======
Animats
_Each Gitaly server instance was fork /exec'ing Git processes about 20 times
per second so we seemed to finally have a very promising lead._

What's really wrong here is that they're apparently spawning processes like
crazy. Do they spawn a new process for each API call? That's like running CGI
programs under Apache, like it's 1995.

~~~
CapacitorSet
I wonder how much it would speed up if they were using libgit2 directly.

~~~
sytse
I'm at GitLab but not on the Gitaly team. I think we are using libgit2 but
that it doesn't contain all the calls we need.

~~~
CapacitorSet
Reading through an old issue (can't link right now, am on mobile), it seems
that the main reason for not using libgit2 in Gitaly is performance, since it
would read too many unused files.

------
jorangreef
We currently have the same problem in Node, where fork is still being called
synchronously from the event loop instead of asynchronously from the thread
pool.

Calling exec() or spawn() in Node is therefore not asynchronous and can block
your event loop for hundreds of milliseconds or even seconds as RSS increases.

[https://github.com/nodejs/node/issues/14917](https://github.com/nodejs/node/issues/14917)

~~~
alecthomas
That looked like a very frustrating exchange.

------
jsiepkes
"Recompiling with Go 1.9 solved the problem, thanks to the switch to
posix_spawn"

I never understood why so many people use fork() instead of POSIX spawn(). For
example OpenJDK (Java) also does this as the default for starting a process.
Which leads to interesting results when you use it on a OS which does do
memory over committing like Solaris. Since the process briefly doubles in
memory use with fork() your process will die with an out of memory error.

~~~
nitwit005
I thought of creating a fix myself way back, and the issue was that Go made
use of system calls directly. You basically have to re-implement posix_spawn
in Go. If you look at their change, it includes updates to chipset specific
files, and the fix only seems to work on a CPU that reports as amd64.

~~~
ithkuil
I must say I didn't go to look at the sources of the patch, but what you say
sounds so odd that I'll take the chance and suggest that perhaps the fact that
in golang "amd64" is, for historical reasons, the name of the architecture
more neutrally known as "x86_64", is the source of confusion (I.e. it doesn't
just work on AMD or on CPUs that claim/report having a specific model/maker
etc).

Low level syscall ABI is architecture dependent.

~~~
linkregister
amd64 is the original name of the instruction set. Intel did beat AMD to a
64-bit instruction set: that of the Itanium processors, IA-64. Itanium had
performance issues and lots of errata. Most importantly, IA-64 was not
natively backwards-compatible with x86 instructions. amd64 became the
standard.

x86_64 is a common name for the amd64 architecture, and is a way to describe
both the AMD and Intel implementations. In my opinion, amd64 is a less
ambiguous name and is more historically accurate.

[https://en.m.wikipedia.org/wiki/X86-64](https://en.m.wikipedia.org/wiki/X86-64)

Yes, I am aware that my point is undercut by the fact that the article title
is x86-64, but I stand by my statement.

~~~
cesarb
It's not just the article title. Follow the two footnotes in the "History"
section of that article, to the press releases from AMD announcing the new
ISA. They consistently call it "AMD x86-64" or "AMD's x86-64" or just
"x86-64". The oldest snapshot I could find of the x86-64 web site
([https://web.archive.org/web/20000817014037/http://www.x86-64...](https://web.archive.org/web/20000817014037/http://www.x86-64.org:80/))
also calls it x86-64. The most recent snapshot of that site, however, calls it
AMD64; it seems to have changed sometime in the middle of April 2003.

That is, both x86-64 and AMD64 are historically accurate (2003 was early
enough in the ISA's lifetime), but x86-64 is the earlier name.

------
kevincox
It's often to remember the other point of view when you see huge performance
differences.

> A bug in Go <1.9 was causing a 30x slowdown in our Gitaly service.

~~~
0003
The author does say fix.

------
empath75
Related post from a few weeks ago:

Fork is not my favorite syscall:

[https://news.ycombinator.com/item?id=16068305](https://news.ycombinator.com/item?id=16068305)

------
zebra9978
Are you guys planning to migrate gitlab to golang ? I think the biggest
feature that everyone wants is better performance.

Is the migration path that tough ?

~~~
romanovcode
They have so many features that I don't see it happening ever.

~~~
carussell
They wouldn't need to stop the world and do a full rewrite. It would be
feasible if they stop writing new components in Ruby and began replacing the
existing parts piecemeal.

------
tuna
Pretty sure that using stdlib and trying to limit shell script in Go would
help performance. Case in point, forking "du":

[https://gitlab.com/gitlab-
org/gitaly/blob/master/internal/se...](https://gitlab.com/gitlab-
org/gitaly/blob/master/internal/service/repository/size.go#L24)

------
stonewhite
> Having solid application monitoring in place allowed us to detect this
> issue, and start investigating it, far earlier than we otherwise would have
> been able to.

Yet apparently nobody either caught or investigated the latency spike after
the previous deployment.

------
haikuginger
The article linked to a set of posix-spawn benchmarks[1] that seemed to
indicate that while fork/exec time scaled linearly with resident memory on
Linux, it did not scale at all on macOS.

First of all, I was somewhat confused by that due to the availability of copy-
on-write; I wouldn't have expected fork/exec time to scale up that way.

Second, I was surprised that there wasn't an attempt to explain the behavior
difference between the two systems. Can someone familiar with either or both
point towards an explanation for why that's the case? It seems very odd.

[1][https://github.com/rtomayko/posix-
spawn#benchmarks](https://github.com/rtomayko/posix-spawn#benchmarks)

