
Rewriting a large production system in Go - qdie
http://matt-welsh.blogspot.com/2013/08/rewriting-large-production-system-in-go.html
======
lbarrow
It's interesting to see that more and more internal projects at Google are
getting redone in Go. However, this sentence stands out to me in the article:

"Before doing the rewrite, we realized we needed only a small subset of the
functionality of the original system -- perhaps 20% (or less) of what the
other projects were doing with it."

I'm guessing that a lot of the benefit of the rewrite came simply from the
simplification of the core logic and dropping extraneous functionality. That
said, having written a little bit of both C++ and Go, I can completely see why
the author found that Go was both far more readable and more maintainable than
C++.

~~~
mdwelsh
OP here. It is absolutely true that we did not rewrite the entire original
system in Go; I tried to be very explicit about that in the blog post. But, I
feel confident that it could be done, in much less code, with greater clarity
and modularization. The Go language by itself does not force good software
design. A rewrite in _any_ language would have been better than the original
system, but our decision to use Go turned out to be fortuitous in that we
managed to do so in record time and with much greater programmer productivity.

~~~
lbarrow
I'm glad to hear it.

Was there anything about the language that you found particularly _forced_
clarity and modularity? I'm giving a talk comparing Go and Ruby next month and
I'm curious as to what people with experience with larger Go programs find to
be most helpful.

~~~
mdwelsh
The Go module system for sure. Also I really like Go's interface model (as
opposed to Java or C++ classes) as it is more flexible and in some ways more
precise. Note that I don't know Ruby so I can't compare Go to that...

~~~
danieldk
Thanks for the article, it was a very interesting read! Since this always
comes up in Go discussions: did you miss generics in this project or was the
lack of generics a non-issue?

~~~
mdwelsh
I haven't found that I needed generics so far, though I can see places where
they would be useful.

------
sytelus
The number of strongly typed garbage collected compiled languages is surely
shrinking. I can understand someone's disgust towards Java considering
dwindling velocity in adding next-gen features and now sleeping in same bed as
lawyer run Oracle. That pretty much had left C# in the arena until Go came on
the scene. I was honestly hoping Go would give us head-to-head battle with C#
but from initial looks Go pretty much feels like C# 1.0. No generics, no
partial classes, no linq... C# 5.0 even has async and dynamic types. I like
minimalism but these features are something we take for granted in languages
that were created post-2000 era. So it all boils down to people choosing Go
for singular reason: It's not from Microsoft! If licensing for C# wasn't a
barrier and let's stretch our imagination by thinking C# was created by some
other group in Google, would Go have a chance?

~~~
pjmlp
Yep, Go would be hardly mentioned here if it wasn't being developed at Google.

Having said this, Go can be a nice replacement for many use cases one would
use C for.

~~~
scott_s
Go seems to be gaining traction. Does it matter if that's because of where it
came from? Maybe in a cultural, I-want-to-think-about-why-some-languages-get-
adopted sense, sure, then it can matter. And I think it's worth having those
discussions. But I get the impression that people think there's some absolute
injustice in the idea that people are adopting a language, and it's partly
because it's from Google. Consider:

"Yes, C would be hardly mentioned here if it wasn't being developed at Bell
Labs."

~~~
pjmlp
> "Yes, C would be hardly mentioned here if it wasn't being developed at Bell
> Labs."

Fully agree, most likely if UNIX hadn't been adopted at important American
universities, C wouldn't have spread as it did.

------
bane
> So programming in Go is making me soft.

Something similar happened to me years ago when I started picking up Perl. All
the BS C++ and Java made me go through to get anything done seemed like such a
huge waste of time I ended just writing lots of stuff in Perl until I had
pretty much forgotten how to C++ and how to Java.

Go might be a new Perl in that sense.

I'd also add that as Google moves more and more production systems onto Go, it
will literally become responsible for handling billions of dollars of
business. That and because Google is using it (and everybody wants to work at
Google so knowing some Go ahead of time will be useful) will lead universities
to start covering it and before long this virtuous cycle will result in Go
everywhere.

I've also noticed a number of posts here over the last few months extolling
the virtues of moving off of slow frameworks built around slow scripting
languages in terms of huge reductions in infrastructure costs. For each system
that Google brings on-line with Go, even if it took a team of 10 6 months,
they've likely saved themselves millions of dollars in new capex
infrastructure acquisition. Imagine serving 40x the requests off of the same
old hardware. Alternatively, you might be able to use slower, but more energy
efficient hardware to serve the same number of users and save on power and
heat management. All this combined with Moore's law means this is a fantastic
idea at this kind of scale.

~~~
npalli
"For each system that Google brings on-line with Go, even if it took a team of
10 6 months, they've likely saved themselves millions of dollars in new capex
infrastructure acquisition"

I thought the old systems were in c++. So, how did the infrastructure savings
accrue? In fact, I don't recall anyone doing analysis on that front.

~~~
dgacmu
Matt linked to my course, but also: I did a tiny analysis of this with my pi
searcher ([http://da-data.blogspot.com/2013/05/improving-pi-
searchers-s...](http://da-data.blogspot.com/2013/05/improving-pi-searchers-
speed-by-moving.html) ) -- it's a toy compared to the kind of system that Matt
described, but my experience was extremely positive. Rewriting it in Go made
it easier to architect the system right to take advantage of persistence. It's
hugely faster.

We also have a paper accepted to SOSP this year (the academic weenies will
understand that) where the system - a Paxos variant - is implemented in Go. We
also implemented about four _other_ Paxos variants in Go to compare against
them, a task that would have been absolutely grueling in C++. In Go it wasn't
too bad. Our performance isn't super-blazing, but it turns out to be one of
the faster publicly available Paxos implementations anyway. I'll have to write
up a bit about our experience with it -- I'm with Matt 100% on this one.

~~~
mdwelsh
We've done lots of load testing and the CPU and memory footprint of the Go
version is better than the C++ version. Not surprising since we reduced the
code size so much, but at least using Go did not involve significant bloat.

~~~
continuations
What about throughput and latency, how does your Go version compare to the C++
version?

~~~
mdwelsh
Identical. The bulk of our system's latency involves making calls out to other
services, so that is not the bottleneck in this case.

------
zzzeek
> If I could get out of the 1970s and use an editor other than vi, maybe I
> would get some help from an IDE in this regard, but I staunchly refuse to
> edit code with any tool that requires using a mouse.

So you've shown that Go is appealing to rigid curmudgeons. Personally I'm
still hung up on the "every function (edit: that does _anything_ which might
itself return an error code, which in large scale code is _quite a lot_ ) must
return an error code" thing. Just like not understanding the purpose and
usefulness of graphical editors, the creators of Go seemingly (despite their
legendary status...) seemingly not understanding the purpose and usefulness of
exceptions (waving them off as "they result in convoluted code" with no
explanation) is keeping me pretty skeptical.

It just requires I take in some well written, idiomatic Go code so that I can
finally "get it", how I'd live without exceptions, but I'm too busy being
productive in Python (and enjoying Pypy's ever increasing speedups) to get
that interested.

~~~
BarkMore
It is not the case that every function most return an error.

The Go creators do understand the purpose and usefulness of exceptions. They
chose not to use exceptions with this knowledge. See
[http://golang.org/doc/faq#exceptions](http://golang.org/doc/faq#exceptions)
for their reasons.

~~~
sgift
I don't know. The quality of this FAQ entry is in my opinion narrowed by their
inclusion of FileNotFoundException, which it seems they did not understand.
The use case for this exception is _not_ to allow lazy programmers to use it
instead of checking that a file exists, but to signal to a programmer that his
world view of the state of his program may be _wrong_ despite his best
efforts, e.g.

    
    
      if(file.exists())
      {
        //do something
      }
    

There's a race condition between the if and do something which invalidates the
programmers world view (someone can delete the file between these two
statements). And this is an exceptional situation the program has to deal
with. Error codes _may_ tell the programmer this, but it is quite likely that
the programmer will just ignore it, because "I've already checked that it
exists - what could possibly go wrong?". Exceptions, especially checked
exceptions (in my opinion the only good exceptions for "normal" program
code)(1), force the programmer to deal with this problem. Or to say -
deliberately - "Hey, program, I don't care for the stability of my software.
Just explode if this happens!".

(1) Languages which have only unchecked exceptions do, in my opinion, cave in
to the laziness of programmers: "but, but, it is so much WORK to deal with all
of this. Can't it just go away? Please?" \- the result is code which can
explode everywhere. This is even worse than no exceptions. With return codes
you know at least that you have to check the code yourself very, very
carefully all the time.

~~~
enneff
> There's a race condition between the if and do something which invalidates
> the programmers world view (someone can delete the file between these two
> statements). And this is an exceptional situation the program has to deal
> with.

It's not an exceptional situation. It's a bug in your program. And it's not
lazy to open() and check for existence. It's actually the _only_ sane way to
check that the file exists, if you plan to open it.

> Error codes may tell the programmer this, but it is quite likely that the
> programmer will just ignore it, because "I've already checked that it exists
> - what could possibly go wrong?"

The following code is obviously wrong because of the race condition you
mention. Nobody in their right mind would write code like this.

    
    
        if exists(file) {
            f = open(file)
            // do something with f
        } else {
            // handle "file not found" case
        }
    

This code is correct, to some degree:

    
    
        try {
            f = open(file)
            // do something with f
        } catch (file not found error) {
            // handle "file not found" case
        }
    

As is this Go code:

    
    
        f, err := os.Open(file)
        if os.IsNotExist(err) {
            // handle "file not found" case
        } else if err != nil {
            // handle any other error that might arise
        }
        // do something with f
    

The thing is, the above code is not me being extra careful about checking
errors. It's just bog standard Go code. Checking error values is the only way
to write even half-decent Go code, so everyone does it all the time.

No reasonable programmer would write Go code like this:

    
    
        _, err := os.Stat(file)
        if err != nil {
            // handle "file not found" case  
        } else {
            f, _ := os.Open(file)
            // do something with f
            // (or explode if file doesn't exist)
        }
    

To my Go programmer eyes, the underscore (where I'm ignoring the error) in the
os.Open line sticks out like a sore thumb. You wouldn't write it, and when
reading you would certainly notice it as bad code (or at the very least
extraordinary code).

------
derefr
You know, every time I see some Googler shocked at the effectiveness and
various advantages of coding in Go, I wonder why Google never adopted Erlang.
They could have been getting all these same advantages (and then some) a
decade ago :)

~~~
zaphar
(full disclosure: I work at google and also like erlang)

Erlang has fantastic facilities for robustness and concurrency. What it does
not have is type safety and it's terrible at handling text in a performant
fashion. So if you don't care about either of those things and only care about
robustness and concurrency then Erlang is great. There were internal
discussions about Erlang here but the upshot was. We had already basically
duplicated Erlangs supervision model in our infrastructure, only we did it for
all languages and Erlang didn't offer any benefits in performance for us. It's
only benefit would have been the concurrency model. That's much less benefit
than Go gives.

Go gives you Erlangs concurency model, a similar philosophy of robustness,
type safety, all batteries included, and performance. Equating the two
languages works in 1 or 2 dimensions but not on all the dimensions google
cares about.

~~~
derefr
Interesting, thanks for that; it's pretty much what I guessed (especially the
bit about the supervision tree and hot-code-upgrade advantages being mooted by
your infrastructure.)

On a tangent, though:

> What it does not have is type safety

I've tried to work this out before (I'm designing a new language for Erlang's
VM), but as far as I can tell, type safety is in practice _incompatible_ with
VM-supported hot code upgrade.

If you have two services, A and B, and you need to upgrade them both, but you
can't "stop the world" to do an atomic upgrade of both A and B together
(because you're running a distributed soft real-time system, after all), then
you need to switch out A, and then switch out B.

So, at some point, on some nodes, A will be running a version with an ABI
incompatible with B. In a strongly-typed system, the VM wouldn't allow A's new
code to load, since it refers to functions in B with type signatures that
don't exist.

On the other hand, in a system with pattern-matching and a "let it crash"
philosophy, you just let A's new code start up and repeatedly try-and-fail to
communicate with B for a while, until B's code gets upgraded as well--and now
the types are compatible again.

It's an interesting problem.

~~~
laureny
> type safety is in practice incompatible with VM-supported hot code upgrade.

That's not true.

First, it's very easy to hot reload changes that have been made to the code
that are backward compatible. The JVM spec describes in very specific details
what that means (adding or removing a method is not backward compatible,
modifying a body is, etc...).

This is how Hotswap works, the JVM has been using it for years.

As for changes that are backward incompatible, you can still manage them with
application level techniques, such as rolling out servers or simply allow two
different versions of the class to exist at the same time (JRebel does that,
as do other a few other products in the JVM ecosystem).

Erlang doesn't really have any advantages over statically typed systems in the
hot reload area, and its lack of static typing is a deal breaker for pretty
much any serious production deployment.

~~~
derefr
> As for changes that are backward incompatible, you can still manage them
> with application level techniques, such as rolling out servers or simply
> allow two different versions of the class to exist at the same time (JRebel
> does that, as do other a few other products in the JVM ecosystem).

Neither of these allow for the whole reason Erlang has hot code upgrade in the
first place: allowing to upgrade the code on one side of a TCP connection
without dropping the connection to the other side. Tell me how to do that with
a static type system :)

~~~
lenkite
Out of curiosity, where/why would such an exotic feature be needed in today's
internet architectures where you always front a group of servers with a load
balancer ?

~~~
butterflyhug
Not all Internet protocols are HTTP. If you're running a service where long-
lived connections are the norm, "simply fronting a bunch of servers with a
load balancer" can require a pretty smart load balancer. E.g. IMAP connections
often last hours or even days, and are required to maintain a degree of
statefulness.

------
jussij
> Go's type inference makes for lean code, but requires you to dig a little to
> figure out what the type of a given variable is if it's not explicit. So
> given code like:
    
    
       foo, bar := someFunc(baz) 
    

> You'd really like to know what foo and bar actually are, in case you want to
> add some new code to operate on them.

The gocode utility (which works with several editors) does exactly this.

------
rgarcia
Re: figuring out the signature of functions or the definition of structs, I've
found godef useful: [http://godoc.org/code.google.com/p/rog-
go/exp/cmd/godef](http://godoc.org/code.google.com/p/rog-go/exp/cmd/godef)

The Go emacs mode has godef-jump (C-c C-j) and godef-describe (C-c C-d):
[http://golang.org/misc/emacs/go-mode.el](http://golang.org/misc/emacs/go-
mode.el)

------
kailuowang
It disheartens me a bit when people compare Go with C++. It gave me the
impression that Go was created to be better C++ (or Java) mostly to make life
easier for the thousands of developers at Google who are stuck with those two
languages.

~~~
mdwelsh
OP here. What do you think we should be using apart from C++ or Java?

~~~
azth
Scala. Rust also looks extremely promising, though not mature yet. It is
getting there quickly though.

~~~
JulianMorrison
If you use Scala in a way that has not a whole lot of extra cognitive load,
it's a slightly prettier re-skin of Java. If you use it like it's Haskell, it
has a cognitive load that's as bad as Haskell (or worse). And if you inherit a
codebase from someone else, you can bet they've been tempted to be oh-so-
clever. That's why I'd never let Scala near production.

Go has almost no cognitive load beyond the complexity of the algorithm itself.

~~~
cgag
It sounds to me like you're lumping things like writing for loops and mutating
state in with the inherent complexity of the algorithm. I think reducing the
ability to create abstractions like generic map/filter increases cognitive
load. Reducing cognitive load is the whole reason I'm learning Haskell and
currently use Clojure.

------
dodyg
Go is a nice language with awesome features except that it doesn't fucking
allow unused imports/variables. It makes exploratory programming with Go
annoying as hell.

~~~
im3w1l

        don't {
            print x for {x : unused variables}
            x() for {x : function from unused imports}
        }

------
lucian1900
Interesting that Go's lack of HM type inference is that problematic. In, Rust
or Haskell, one could simply use that value further on with some type and have
it inferred correctly.

------
orenbarzilai
Can you share your new system benchmarks comparing to the old c++ system?

~~~
mdwelsh
Unfortunately, probably not. Hopefully one day.

------
grannyg00se
"So given code like:

    
    
        foo, bar := someFunc(baz) 
    

You'd really like to know what foo and bar actually are"

Wouldn't your editor, mouseless though it may be, provide for that? Does ctags
not work with vi?

And why use vi over vim?

~~~
mdwelsh
OP here. I do actually use vim but I have yet to adopt most of the fancy
plugins that vim provides -- put me in a time machine back to 1977 and I would
be very capable of programming on any UNIX system that you'd drop me in front
of, provided it had vi installed. I'm not defending this choice of lifestyle;
it's just how I learned to program :-)

~~~
lifebeyondfife
Even in the 2010s I've found proponents of the UNIX console style of
programming because, as you say, they can be dropped in front of any similar
system and off they go.

However, tools improve and you're denying yourself productivity benefits. One
of the advantages of a statically typed language is that the computer _knows_
the type of the arguments. I love environments that allow me to 'mouse over' a
function or variable etc. and tell me exactly what it is (and allow me to
navigate to its definition). You don't have to use a mouse but what's the
disadvantage in having alternatives to keyboard shortcuts _where appropriate_?

------
pvdm
Unless it is open sourced, I can't validate the claims.

~~~
mdwelsh
I would love to open source this system, but even if we did I'm not sure we'd
be able to convince you that using Go was better than some other language in
terms of developer productivity. How would having access to the source help?

~~~
pvdm
You mentioned in the article that you read a section of code by one of the
lead engineers and immediately you became convinced that using Go is the "way-
to-go". Well, I would like to experience that eureka moment as well.

~~~
mdwelsh
Good point. Maybe you could check out one of the many other open source Go
projects out there as an alternative. I'm not sure reading our code would give
you that eureka moment, since it only made sense to me since I was so familiar
with the old code :-)

