
Docker and Go: why did we decide to write Docker in Go? - sylvainkalache
http://www.slideshare.net/jpetazzo/docker-and-go-why-did-we-decide-to-write-docker-in-go
======
rdtsc
For me it is Go vs Erlang. I see both solving similar set of problems (for
me!).

The slides said Go is easier than Erlang. That is true in some aspects, namely
syntax (Elixir to the rescue here!). Go is also easier because of static
typing. Speed of compilation is awesome. Building to an executable that only
depends on libc -- yes please! love that about Go.

I am scared about safety and fault tolerance. Ability to share memory is good
for performance but a shared heap in a large concurrent application is scary
to me. Not sure what I think about multiple channels and running select over
them. Then it pseudo-randomly choosing one channel among those that can
receive. Erlang's focus on processes/actors vs channel and each actor having a
single mailbox somehow makes more sense to me. Having a shared heap mean
faster data processing but GC cannot be fully concurrent.

Now passing channels inside other channels is cool, but also kind of scary to
me. You can pass channels, inside channels, inside channels, can make some
kind of a lazy computation chain.

Also I have been wondering, but couldn't find it by doing a quick search, is
it possible to build supervision trees out of goroutines like one builds
supervision trees in Erlang's OTP. Say I want to have a chat client goroutine
to keep the chat up. But I want to monitor it in case it crashes (panics?)
then then spawn another one perhaps. Is that possible?

~~~
MetaCosm
As someone who has shipped production software in both Erlang and Go. They are
very different beasts. Regarding error handling, Go is far closer to C than
Erlang. If Erlangs slogan is "let it fail" and gives you the power to avoid
writing error handling for every single thing, Go's is "don't let it fail" ...
and you really need to write error handlers for everything, reducing the
"unexpected" footprint.

There is no real way to grab a goroutine from the outside and tell it what to
do -- you have to use its channels, which if bad things have happened are
probably broken.. there are some systems built that catch exceptions and send
messages (this is from inside the goroutine of course) on the channel, but I
haven't used them.

That might come across an unduly negative of Go, but Go has major upsides.
Great tooling, easy to understand after a tiny amount of time, a culture of
explicit (even at times non-DRY) code, amazing deploy model, and generally
just very understandable even to newcomers.

~~~
rdtsc
Interesting, thank you for explaining. The lack of monitoring of goroutines is
certainly different.

> which if bad things have happened are probably broken.

It reminds me of Joe Armstrong's quote about how it is hard to perform surgery
on yourself. In other words having the component that is failing trying to fix
itself. I guess I would have to read more about design patterns. I saw a
presentation about concurrency patterns in go, but it is about very simple toy
examples, I am more interested in a larger concurrent applications, handling
multiple connections for example.

~~~
MetaCosm
Yeah... so far my Go stuff has had exceptional reliability, I verified I
handled all error known conditions with errcheck
([https://github.com/kisielk/errcheck](https://github.com/kisielk/errcheck))
and just generally took the slow and plodding approach of handling everything
explicitly. My app handles hundreds of thousands of concurrent connections,
each one often using and/or spawning 4 or more goroutines.

~~~
rdtsc
Thanks for the errcheck link, I will be using that. Would Go authors consider
building that into the compiler? That would make quite a bit of sense.

~~~
MetaCosm
As soon as I first used errcheck, I thought the exact same thing... "this
should be included as warnings"

------
leokun
I love Go and want to see it succeed, but as long as "why I wrote X in Go" is
a novel thing, Go maybe hasn't made it yet. There are probably thousands of
wonderful Python libraries and sure sometimes people decide to talk about why
those chose Python, but all of these "Why Go?" posts feel a bit like a forced
explanation to the question "Why did you write X in some new language that
almost nobody else is using?"

But that isn't true anymore. Lots of people and teams are using Go. You don't
need to explain yourself. Just write stuff in Go. Maybe even write about the
process of how you wrote your app in Go, but stop making it about the
language. It's there, it works. No need to apologize, evangelize or explain
yourself.

~~~
yeukhon
No, it is totally relevant. Sure you can write docker in Perl in Bash or even
in lolcode if you really want to. Each language has its strength and weakness.
If everyone is language agnostic why do we bother to invest in Rust when C++
is just good enough in the first place? Why do people here keep inventing a
new language? Why Javascript if we could just use Java back in the 90s?

Does the language provide some facility that allows docker developers to make
docker easier to write and easier to scale than writing docker in Python?
Totally relevant. People experiment with new language and hearing Go is
scalable and Go is about concurrency - what do these things mean to docker
inventors? As a library writer I want to understand what makes Go their choice
when Ruby, Python, C and Java seems to the primary languages people use to
write system tool.

If I were to write a similar tool like docker in Go, what patterns can I learn
from reading docker?

------
stretchwithme
Wouldn't it be great if audio of a presentation were also available and the
slides were played in sync with it?

Imagine if presentation software generated a coded tone for each transition
that could be used later on to merge the recorded audio of the speaker and the
slides into an audio slideshow.

Then having a video of your presentation would just be a side effect of your
presentation. And it would focus on the material you wanted to present, not
your image.

Although videos of speakers are nice, often the material the speaker is
referring to isn't visible. It really should be the visual focus.

~~~
prewett
I wish people would post their notes and not their slides. If I can understand
your talk from your slides, your slides are way too detailed, and it might be
more beneficial to just give everyone the text and not bother with the
speaking, since it's so much slower than reading.

And if there's a video, I'm going to immediately go somewhere else. I don't
have time to be watching hour-long videos when the content could be summarized
in something I could read in 5 minutes.

~~~
elehack
Not everyone has notes. When I lecture, I do so from notes, but I usually do
other presentations without them. I'll likely have created an outline, but it
has less information than the slides.

------
shurcooL
Comments on slide 28:

"go get, can't pin a particular revision"

\- There's an elegant solution by now, see
[https://github.com/kr/godep](https://github.com/kr/godep)

"must deal with private repos manually"

\- go get can get them via ssh using key auth, just do
[https://gist.github.com/shurcooL/6927554](https://gist.github.com/shurcooL/6927554)

~~~
jpetazzo
Thanks!

godep is definitely cool. But it's a bit sad that it has to be an add-on.
Don't get me wrong -- it's a great thing that go gives us a good build system,
a good test system, a good dependency system, etc. But if we end up using
another build system, another test system, another dependency system, because
the stock ones aren't powerful enough, it kind of ruins it :-)

Regarding private repos: IMHO it's much simpler to go to your
$GOPATH/src/github.com/blahblah and `git clone git@github.com:blahblah/repo`.
But if you know a difference between both approaches I would be happy to hear
it!

~~~
shurcooL
> it's a great thing that go gives us a good build system, a good test system,
> a good dependency system, etc. But if we end up using another build system,
> another test system, another dependency system, because the stock ones
> aren't powerful enough, it kind of ruins it :-)

That's the thing, godep isn't another build system. It's built on _top_ of the
existing Go build system. It augments it, doesn't replace it or replicate
what's already there.

Go has tools for building at master; godep adds a file that keeps track of
explicit commits and creates a sandbox GOPATH with those revisions checked
out. Nothing you couldn't do by hand; godep simply automates the task, making
it reproducible.

The reason Go doesn't have everything is because the developers haven't found
a very good solution for everything. So instead of providing something crappy,
they just don't include it... Until a later time. This gives the community
time to experiment and find a good solution in the meantime.

Honestly, I'm very glad they're taking this approach.

------
sandGorgon
if you are running latest versions of Ubuntu or Linux kernels, it is a PITA to
get Docker running because of its dependence on AUFS - which is deprecated on
the kernel side.

The better (but unknown) way of running it is to install Docker 0.7-RC4
through _curl[http://test.docker.io](http://test.docker.io) | sudo sh_.

Just FYI.

~~~
nfm
Why would anyone downvote this?

~~~
phpnode
probably someone who doesn't like the whole "blindly execute untrusted shell
scripts from the internet as root" thing

------
viraptor
Devs not aware of where and how is their software running, while ops are not
aware what they're running? That sound like a recipe for a disaster. You don't
need to know everything, that's why there are separate teams... but the way
it's described in the slides is slightly scary to me. I've seen people caring
only about their own area and lobbing things over the wall for other teams to
take out over. It never ended well.

~~~
city41
I think it was more "I want to write my code without how it ends up packaged
or deployed having influence on how I write it", and vice versa.

We definitely don't have that at my job. We have to be conscious of the dev,
test, build and production environments when coding. Ops has to be more
conscious of what they are deploying than they'd like. It's not that either
team doesn't care about the other's work or the big picture.

~~~
alex4nder
If how it's deployed doesn't influence the way you write software, you're
either: a) developing something trivial, or b) deploying something that's not
going to work well in the long run.

~~~
city41
I can't really argue with that. Just giving my interpretation of the slide. I
still don't know Docker very well, maybe it's not as revolutionary as it seems
to be.

------
JulianMorrison
The solution I've found to the "package main" problem is to put the "main" as
a _sub_ package.

    
    
      -src-myapp-main1
                \shared_code
                \main2
    

That way main can import "myapp/shared_code". Yes, the packaging standard is
being broken for the main programs, but that's ok, nothing needs to import
them.

~~~
cmccabe
Maybe I missed it, but I didn't find anything on
[http://golang.org/doc/code.html#PackagePaths](http://golang.org/doc/code.html#PackagePaths)
that seems to forbid what you're doing. In fact, it says "in practice you can
choose any arbitrary path name, as long as it is unique to the standard
library and greater Go ecosystem." But rather than main1 and main2, surely it
makes sense to name those directories after the binaries you're building?

I actually kind of like Go's scheme here. One thing I never quite liked in
C/C++ was huge directories full of source files where it wasn't clear what
code was part of what binaries. If something is only part of one binary, why
not make that obvious by putting it in another package?

~~~
JulianMorrison
Yeah, main1 and main2 are not real names, just a way of showing on my
ASCIIgram where the mains are relative to the libraries they import.

------
yeukhon
What do they mean by Go is neutral?

~~~
IbJacked
If they chose ruby, python coders might not like it. If they chose python,
ruby coders might not like it, etc.

By using Go, it avoids a lot of existing biases. I think that's what that
slide is referring to.

~~~
snogglethorpe
Er, but those people tend to be for "their" language, and suspicious of
others, regardless. Avoiding traditional rivalries may give them some
breathing room, but Go is not somehow outside of the tribal fanboy ecosystem.

Indeed, the "OMG GO!1!" tone of many HN stories and comments on the language
suggest that it's diving in headfirst...

~~~
RBerenguel
I don't think so: I'm guessing a lot of Go programmers where Ruby/Python guys
before. My main language before Go was Python (every time I needed to do some
non-trivial bashy stuff I went for Python) and now unless I need some
esotheric library only available in Python (or Lisp) I go straigt for Go. And
still like Python, no big deal. Won't bash a language I still like!

------
millstone
Slide 24 says "Why Go? Multi-arch build without preprocessors."

My understanding is that Docker is strongly tied to Linux by virtue of using
Linux containers. So how does Docker benefit from the multi-arch build
facilities of Go, if it only runs on Linux?

~~~
jamescun
LXC does require Linux and x64 is the recommended architecture to run it on,
however LXC can be built for other platforms (namely x86, x86-64, IA-64,
PowerPC, SPARC, Itanium and ARM [1])

And just today there was a breakthrough in getting Docker running on the
Raspberry Pi:
[https://news.ycombinator.com/item?id=6708565](https://news.ycombinator.com/item?id=6708565)

[1] [https://en.wikipedia.org/wiki/LXC](https://en.wikipedia.org/wiki/LXC)

~~~
millstone
But the Go compiler only has backends for three out of the seven architectures
you listed. So this would seem to be a strike against Go.

It looks like Docker provides binaries for only x86-64 Ubuntu. So slide 24
seems to be more of "we think this is cool in theory" rather than "this really
helps us."

~~~
justincormack
Yes and I don't think there are image repositories for other architectures,
though perhaps the pi thing may mean an arm repository turns up.

It would be cool if IBM sponsored more powerpc ports of stuff. They do give
free access to ppc machines, but they should have people who eg write go
backends for ppc64.

------
rwmj
I notice that "security" doesn't figure in the devops slide at all. Bundling a
whole load of untracked, old libraries together and then exposing it on the
web might not be the best idea in the world.

------
ksec
I am more leaning on OSv rather then Docker at the moment. Although OSv still
isn;t ready yet.

------
a8da6b0c91d
I don't really know anything about docker. Do you actually have to write Go to
use it effectively?

~~~
cschmidt
Not in the least. It has a nice command line interface to use it. No Go
knowledge is required, or even useful.

------
jsoo4
It's painfully obvious these guys have never created a proper distro package
in earnest.

~~~
millstone
Can you please elaborate? What do you mean?

