
Linux Namespaces and Go Don't Mix - _lm_
https://www.weave.works/blog/linux-namespaces-and-go-don-t-mix
======
vishvananda
Author of the go netlink library here. I've run into this issue a number of
times. There has been conversation in the past about adding some kind of new
runtime command like LockOSThread to prevent new threads from being spawned,
but it didn't gain any momentum.

Even though I am a big fan of go, I've personally built two container runtimes
in other languages do to the namespace clumsiness.

Personally, I think rust is an excellent alternative for namespace utilities.

EDIT: there is more information and links in the issue in the netns library:
[https://github.com/vishvananda/netns/issues/17](https://github.com/vishvananda/netns/issues/17)

~~~
bogomipz
>" I've personally built two container runtimes in other languages do to the
namespace clumsiness."

Can you share any details on those other container runtimes?

~~~
vishvananda
One was a simplified runtime in c, similar to a stripped down version of
systemd-nspawn. I can hopefully share the other one in a few weeks. I'm going
through an open source approval process for it.

~~~
bogomipz
Thanks. It's an interesting topic. I hope you post it on HN then.

------
bhawks
I think the proper title is: Linux Process and Threads Don't Mix.

The Linux syscall interface exposes certain functionalities that are much more
easy to reason about at the process level such as namespaces, capabilities,
seteuid and so on. However these syscalls all operate on the thread level
(since the kernel treats threads pretty similarly to processes). Therefore in
order to perform these operations safely you need some sort of process wide
mechanism to apply the operation on every thread (and don't forget error
handling!)

This is _not_ just a golang problem or an M:N threading problem as many
comments suggest. The kernel really needs to provide new syscalls for these
features that operate at the process / thread-group level. The current
syscalls are extremely difficult to use correctly in any multithreaded context
in any language. When you consider the security implications of these features
it makes the problem even worse.

Check out [https://ewontfix.com/17/](https://ewontfix.com/17/) for a really
good analysis of the difficulty musl libc has faced making a multi-thread safe
seteuid on Linux. There are also many bugs in glibc related to this as well.
Linux makes userspace responsible for patching up the leaks in the kernel's
process abstraction and that's really not a job that userspace is in the right
position to take on.

~~~
jeremyjh
The M:N problem in Go is that you cannot control which thread runs which code.
So yes, you could wish the OS exposed different APIs, but presently you can
manage this situation in languages that let you manage threads.

~~~
jchw
The person you're replying to has already made that clear. It's, in fact, also
possible to manage the problem in Go with some finagling. But like Go, if you
have multiple threads, it's difficult

I've hit this exact problem with multithreading in C and setuid and just
because it _can_ be managed in C doesn't make it easy or straightforward.

Therefore, I mirror the sentiment that there needs to be a way to operate on a
process level, even if that has some interesting consequences.

(P.S.: In C, if you're using glibc, it DOES actually patch this issue up on
its own using one hell of a nasty hack.)

~~~
cyphar
> P.S.: In C, if you're using glibc, it DOES actually patch this issue up on
> its own using one hell of a nasty hack.)

I too love reading the code for nptl(7). It's a riot. :D

------
SwellJoe
This leads to go code being roughly as messy/clumsy as C (or whatever else)
code in the sections that need concurrency and also need to change namespace.
That's unfortunate, but I don't know that it really "raises a few eyebrows".

I mean, what's the better alternative to Go for this work? Maybe Rust? It is,
at least more controllable at a lower level...but, not as easy to pick up for
people coming from a C/Python/Perl/Ruby systems and ops background. I'm not
saying one _should_ use Go for containers/namespaces programming, but a lot of
people are with some success (probably also banging into the namespaces issue
now and then), I'm just saying it's not obvious to me what the better
alternative would be.

~~~
scriptkiddy
> I mean, what's the better alternative to Go for this work? Maybe Rust? It
> is, at least more controllable at a lower level...but, not as easy to pick
> up for people coming from a C/Python/Perl/Ruby systems and ops background.

You know, it's interesting. I've been programming with Python for about 6
years now. I've also picked up Javascript, SQL, bash, and PHP along the way.
I'm always gaining a little bit of C knowledge here and there when writing C
extensions for my Python applications. I'm a fairly experienced programmer at
this point. To the point:

I tried picking up Go one day because I was hearing so much about how it could
replace Python as network glue code with better performance and reliable
concurrency. I can't really validate or invalidate those claims. That said, I
found Go to be sort of difficult. The syntax is really simple. Compiling is
really simple. Concurrency is even simple. However, need to do something in a
different way than Go decides is correct? Well, you can't. It won't even
compile. The difficulty in Go is in learning about what the compiler thinks is
OK. I don't really like that. You don't really know if your code will work
until you compile. Basically, I just think Go isn't really flexible enough for
modern programming. I find that Nim can do Go's job better than Go can for my
use cases anyways.

~~~
holydude
Yes but the problem with languages like Nim is lack of support and maturity.
Go is more versatile and at the same time more mature than anything out there.
It is a different design and it excels in what it does (considering all
tradeoffs now).

Will Nim be as versatile and solid as Go in the future? Hard to predict but i
would say no. You need a solid financial backing and certain amount of
adoption where people actually write software that makes them money.

~~~
scriptkiddy
> Yes but the problem with languages like Nim is lack of support and maturity.

Agreed, that's why I don't use it for anything super important yet. Nim is
approaching a 1.0 release soon. Go has a similar problem in that it is
maintained almost entirely by Google who has a history of dropping projects
without warning.

> Go is more versatile and at the same time more mature than anything out
> there.

This is objectively incorrect. In fact, Go aims precisely to be non-versatile
for the sake of simplicity. That is why Go does not have generics for
instance.

> It is a different design and it excels in what it does (considering all
> tradeoffs now)

I don't think it's design is all that different. It looks like a stripped down
version of C and it's definitely not the first PL focused on concurrency.

> Will Nim be as versatile and solid as Go in the future?

Nim is already leagues above Go in the versatility(I assume you mean
flexibility?) department. As far Nim being as "solid" as Go, I'm not entirely
sure what you mean. If you're asking about stability, I believe that Nim can
reach a similar level of stability as Go, yes.

> You need a solid financial backing and certain amount of adoption where
> people actually write software that makes them money.

I agree with this. However, it's not always a quick process. The only reason
Go is as popular as it is is because of Google's size and reputation(edited).
Every programmer on Earth heard of Go within a few days of it's official
release. Nim is taking a slow roll approach. Look at Python. It took almost 15
years before it started getting really popular.

All that said, I didn't come into this thread to argue about Go vs Nim. I've
been accused of shilling Nim in the past. I'm sorry that I like talking about
PLs I enjoy using.

~~~
zellyn
It's more than fine to be enthusiastic about programming languages. However,
some of your points about Go are dubious.

The reasons Go doesn't (yet) have generics are practical rather than
philosophical. And well-documented.

I also don't believe Google's marketing budget contributed much if anything to
supporting Go. The _reputation_ of Google and the Go authors was far more
important. (Personally, when I saw people like Brad Fitzpatrick try Go and
rave about it making programming fun again, I decided to try it.)

You can happily accuse me of shilling Go if you like :-)

~~~
scriptkiddy
No, I encourage you to shill in fact.

> The reasons Go doesn't (yet) have generics are practical rather than
> philosophical. And well-documented.

Can you give an example? I guess I fail to understand why a statically typed
language would choose to forgo all of the advantages generics provide. Does it
have something to do with Goroutines?

> I also don't believe Google's marketing budget contributed much if anything
> to supporting Go.

I didn't necessarily mean their marketing budget. I should probably edit that.
I meant that anything they do is news.

~~~
zellyn
> Can you give an example? I guess I fail to understand why a statically typed
> language would choose to forgo all of the advantages generics provide. Does
> it have something to do with Goroutines?

Like I said, the objections are practical, not philosophical:

They published four past design docs for generics in Go that simply didn't
pass technical muster
([https://github.com/golang/proposal/blob/master/design/15292-...](https://github.com/golang/proposal/blob/master/design/15292-generics.md#proposal))

Also, rsc stated he plans to understand generics better in 2017:
[https://research.swtch.com/go2017#generics](https://research.swtch.com/go2017#generics)

Also, bradfitz said recently on the GoTimeFM podcast that doing Generics and
Go2 together makes sense.

> I didn't necessarily mean their marketing budget. I should probably edit
> that. I meant that anything they do is news.

I feel like the comments from others about Dart give the lie to this one. Go
is fantastically, dramatically, massively more popular than Dart, which is
also from Google.

~~~
unscaled
> Like I said, the objections are practical, not philosophical

Russ Cox and Rob Pike keep repeating this mantra, but I don't buy it. No, I
don't believe they object the idea of generics itself. And yes, generics pose
practical complexities and a plethora of issues that have to be resolved.

But so does every other language feature. The features you choose to add
reflect your philosophical values and priorities.

Go chose to bake some hitherto very niche features that could have been as
libraries into the language. Channels in particular, are a language construct
only because Go doesn't allow operator overloading and generics for user
types. But channels get them, because channels are demonstratively important
for Rob Pike[1].

There's nothing wrong with that of course, but that's a philosophical
decision. Why channels can avoid the vagaries of interface{} boxing, but not
sets, linked lists or queues?

The reality is that every modern statically typed language except Go has
generics, and they all implemented them very well. The ML languages and Ada
were already doing it in the 80s, and OCaml and Eiffel managed to combine
generics and polymorphism back in the 90s.

It seems to me that when originally designing Go, Pike, Griesemer and Thompson
just didn't think generics are worthwhile enough for the effort it takes to
properly research them.

When Go was started, its authors mostly looked to languages which implemented
generics later in their lifecycle (namely Java and C++) and their
implementations suffered from problems due to other, rather obvious, design
defects:
[https://research.swtch.com/generic](https://research.swtch.com/generic)

I'm happy to see that this attitude is changing, and other languages are
looked at.

[1] [https://swtch.com/~rsc/thread/](https://swtch.com/~rsc/thread/)

~~~
zellyn
> The reality is that every modern statically typed language except Go has
> generics, and they all implemented them very well. The ML languages and Ada
> were already doing it in the 80s, and OCaml and Eiffel managed to combine
> generics and polymorphism back in the 90s.

> It seems to me that when originally designing Go, Pike, Griesemer and
> Thompson just didn't think generics are worthwhile enough for the effort it
> takes to properly research them.

Are you arguing that it's easy, and they just haven't done it? Or are you
arguing that it wasn't important enough at first, and so the implementation
developed in directions that preclude straightforward implementations now? I
can believe the latter, but after reading the multiple, detailed proposals
from Ian Lance Taylor, I don't believe the former.

------
liveoneggs
this is just a more complex version of the issue where go can't safely do the
daemonize dance for a privileged port

~~~
brandon
There's a simple-ish workaround for the privileged port issue if you can't
just use CAP_NET_ADMIN:
[http://play.golang.org/p/dXBizm4xl3](http://play.golang.org/p/dXBizm4xl3)

The namespace issues are unfortunately a lot tougher to address.

~~~
vbernat
It's only reliable if none of your dependencies are spawning goroutines during
initialization. If this is the case, some goroutines (yours' or the
dependencies') can end up with increased privileges.

~~~
brandon
The workaround functions by opening the privileged socket and re-executing the
binary as an unprivileged user with access to the filehandle. Any background
goroutines would exit with the privileged parent.

Also: please don't spawn goroutines during init(). There's generally a better
time and place, and in the event that you justifiably need a package-level
background routine you can spawn it on demand with a sync.Once.

------
sreque
I've ran into this problem before. IMO this issue has nothing to do with go
and the solution is straightforward. Simply create a sub-process whenever
entering a new namespace because the operation isn't concurrency safe within a
process.

Note that you'd run into this bug within any multithreaded process, whether
the code was written in go, Java, c, or whatever.

------
ghthor
This is discouraging considering go was initially designed as a systems
programming language. I wonder if there is another way for go to handle
blocking syscall such that this use case would become reliable.

~~~
pritambaral
I think Go was designed as a server programming language more than a systems
programming language.

~~~
stouset
Go was retconned into this role, but originally it was marketed as a systems
language.

------
jankedeen
I read this crap in language design and despair. This is why C is my only
recourse for systems programming in *nix environments.

~~~
Safety1stClyde
This is what I cannot understand about Go: how did a newly-designed
programming language end up as such a muddle?

~~~
YZF
Because this isn't really a language issue. It's an OS issue. Go assumes that
all threads are equal and therefore any Goroutine can run on any thread and
threads are all equal. Linux got late to the thread party and its threads are
kind of like processes that haven't decided if they want to be processes or
threads.

It's true the Go runtime could give users more control over goroutine and
thread scheduling but that would kind of defeat the purpose of not needing to
know about it and having Goroutines as the only flexible unit of concurrency.

I think the kludge here is on the Linux side. Having some magic properties
bestowed upon threads doesn't make sense. The property should be accessible
via a handle that can be shared amongst all threads.

~~~
valarauca1
>Go assumes that all threads are equal and therefore any Goroutine can run on
any thread and threads are all equal

This isn't true in Linux or Windows so I don't see why Go would make this
assumption other than poor design.

>The property should be accessible via a handle that can be shared amongst all
threads.

It literally is. The namespace information is shared with all threads in a PID
group globally available in the /proc fs.

\---

What OP is doing is effectively having 1 program run part of itself in 1
container, and another part outside of that container.

Go isn't a systems programming language so this level of fine grain control
isn't possible. Hell its pretty difficult in a C/Rust/C++ environment.

~~~
YZF
What's an example of it not being true in Windows?

I think it's mostly true in Linux and the situations where it isn't true are
subtle and require expert knowledge in some specific areas. It's not like
there's a big sticker on Linux that says threads are generally not symmetric.

I get the bit of half the process being in one space and half in another. I
just find it odd. Can half the process run as root and half the process run as
another user? Maybe yes? Can a file be open in half the process and not open
in the other?

At any rate, I think it's not black and white. Green threads do make sense in
general and introducing different thread types and more granular control makes
things more complicated.

I think you're missing my point about APIs and handles:

    
    
      handle = open_namespace("test")
      syscall(handle, "yes")
    

if handle can be used across threads then this sequence will run correctly
regardless of what thread is executing it, it doesn't rely on anything
bestowed on a thread.

(EDIT: thread local storage I guess is an example of asymmetry between threads
but it's intentional/clear asymmetry designed for specific purposes and
something you don't need to access in Go e.g. because you don't use threads
directly).

~~~
the8472
Maybe your assumption of symmetry is just backwards? If you look at the clone
syscall then threads are more akin to processes that just happen to share
virtual memory and some other things such as IO priorities, file desciptor
tables, cgroups, .... Many of those things can be shared individually. In
other words they are considered orthogonal features that, when taken together,
happen to function as threads.

------
cat199
alternate title: developers who don't understand user level threading don't
understand user level threading.

