
On Go, Portability, and System Interfaces - zdw
http://garrett.damore.org/2015/09/on-go-portability-and-system-interfaces.html
======
geofft
This is confounding two separate issues: using the system call interface
instead of the platform C library, and static vs. dynamic linking. It's an
easy thing to confound, because (as the author states) on many platforms,
including not only Solaris but also OS X, the only interface to the libc _is_
dynamic linking. But you can certainly dynamically link to libc.so without
dynamically linking to anything else, and without using dynamic linking within
your own language community. (Rust takes roughly this approach.)

There's one more subtler problem with the "defined to exist on POSIX" thing:
strictly speaking, what's defined to exist is a C-language interface. Some
interfaces may be defined to use macros, and some platforms may make things
work when compiled through the C compiler but not through the dynamic-linker
interface. One example that I've run into recently, when binding things in
Rust, is the cmsg API, which is primarily defined in terms of macros to walk a
heterogeneous array, and has to be reimplemented in Rust (with platform-
specific code): [https://github.com/carllerche/nix-
rust/pull/179](https://github.com/carllerche/nix-rust/pull/179). Another
example is Android, which before 5.0 (Lollipop) exposed a handful of signal-
handling functions as inline functions in a C-language header file, so they
were not actually dynamically linkable: [https://github.com/rust-
lang/rust/commit/a8dbb92b](https://github.com/rust-lang/rust/commit/a8dbb92b).
In both of these cases I would have loved to just #include the C header, but
cross-language, that wasn't an option.

It does seem that cgo has some ability to just #include the C header, as
demonstrated here. I'd be curious to know if it's powerful enough to handle
these two cases.

(Also, the author is totally right in saying that this doesn't belong inline
in random application code. If you find yourself writing code like this, it
_should_ end up in an abstraction library so that porters have a single place
to look, and any application calling `tcsetattr` gains the portability
benefits.)

~~~
pcwalton
> Some interfaces may be defined to use macros, and some platforms may make
> things work when compiled through the C compiler but not through the
> dynamic-linker interface. One example that I've run into recently, when
> binding things in Rust, is the cmsg API, which is primarily defined in terms
> of macros to walk a heterogeneous array, and has to be reimplemented in Rust
> (with platform-specific code):

The worst offender I've seen here is Xlib, which has to recreate the internal
layout of large structs like Display that have haphazardly grown fields over
the decades in order to deal with macros that reach into them:
[https://github.com/servo/rust-
xlib/blob/master/src/xlib.rs#L...](https://github.com/servo/rust-
xlib/blob/master/src/xlib.rs#L1101)

~~~
tedunangst
There are C binding equivalents for many of the macros in Xlib (having played
this game before and recently). This may be slower than a macro, but you can
usually cache the result. But really, shouldn't you be using (XML/)XCB? :)

~~~
pcwalton
Last I looked into it GLX didn't work well with XCB. You can use XCB for some
stuff, but you can't drop Xlib.

It doesn't look like this has changed, unless this page is out of date:
[http://xcb.freedesktop.org/opengl/#index5h1](http://xcb.freedesktop.org/opengl/#index5h1)

~~~
geofft
Fascinating! I was wondering why nobody had done an XCB-based set of X
bindings for Rust, given that XCB was basically designed for this purpose
(well-typed, high-level metadata to generate good bindings from).

If I'm reading that page correctly, the problem is that GLX is specified as an
API + implied ABI, not a wire protocol, and part of that API/ABI contract is
"I was compiled with <X11/X11.h> and -lX11, and you'd better give me
structures compatible with those"? And half of the GLX implementations are
closed-source? Sigh.

------
smegel
> Basically, the Go folks want to minimize external dependencies and the web
> of failure that can lead to.

What is he ranting on about? The reason Go doesn't link against system
libraries is that Go in binary-incompatible with C, as in it's calling
convention is different. There is a way to call C through a shim but it's
extremely slow which is why they implemented their own syscall interface for
Go. Static linking is just a side-effect of this.

Edit: the Go team want to support dynamic linking and it is on the roadmap but
not fully implemented as of 1.5 [https://docs.google.com/document/d/1nr-
TQHw_er6GOQRsF6T43GGh...](https://docs.google.com/document/d/1nr-
TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit#)

~~~
tedunangst
I doubt go to C calling conventions are going to be the slowest part of
calling tcsetattr(). Using ioctl() is being difficult for the sake of being
difficult.

~~~
pcwalton
> I doubt go to C calling conventions are going to be the slowest part of
> calling tcsetattr().

Well, by "C calling conventions" it really means "performing a stack switch to
get out of M:N, allocating a big stack if necessary or taking a lock to check
one out". It's quite a lot of overhead and I wouldn't be surprised if it
dwarfs the cost of the SYSENTER/SYSEXIT pair in many cases.

~~~
4ad
It's not necessary to allocate a large stack, there is always one available
because threads have their own stack, and you can switch to it at all times
because goroutine threading is cooperative. In other words, if you are
scheduled to run on a thread, you and only you have access to that thread
stack.

Stack switching doesn't take more than a few machine instructions.

The problem with tcsetattr is that it's a library function, so you
(generally[1]) need the target library in order to create a program that calls
it. This is bad because in Go we value cross-compiling a lot, so we can't
depend on having other people's shared libraries available in the build
environment.

Syscalls, however, we know how to do them and we know what they are, so we
don't need any special target library in order to generate programs for the
target.

[1] On platforms like Solaris and Windows, where we are forced to use target
shared libraries, we cheat a little and encode the target symbol names that we
need, and we rely on standardized ELF/PE symbol resolution mechanism creating
a scheme that's essentially equivalent to having import libraries available.
But this is very complicated and takes a lot of code in the linker, so it's
not at all unexpected that we avoid it whenever possible.

~~~
tedunangst
There's theory and there's practice. Saying that using ioctl is the right
choice because it helps cross compilation rings a little hollow when the cross
compiled binary obviously won't run due to the target system not having said
syscall.

~~~
4ad
I'm sorry, but what?

If the system call doesn't exist, then the program won't _compile_ , because
the system call stub won't be defined for that platform.

If the system call is ioctl, and the user tries to call with an request that
doesn't exist, the program won't _compile_ because the request will not be
defined.

But we don't even export ioctl! First of all, it's not type safe. We only
expose ioctl wrappers that are type-safe and use requests defined by the
target platform, not arbitrary integers passed by the users.

Of course if the _user_ does an indirect system call, and mucks with unsafe,
and passes it arbitrary data, than that's a user problem, not a Go problem.

Using ioctl _internally_ by Go is the right thing.

 _Users_ indirectly calling ioctl and breaking type-safety with unsafe is just
bad code, but that doesn't have anything to do with Go. Go won't prevent you
if you want to shoot yourself like that.

~~~
tedunangst
What is the linked post about if not code that could have been portable but
wasn't? I'm having a hard time determining if you're saying the code in
question was right or not.

~~~
4ad
Well the code in the linked post doesn't compile on Solaris because SYS_IOCTL
is not defined on Solaris. So that was my point. You don't get code that
compiles but doesn't work, unless you try really hard.

But the code is bad. If you have to call system calls through the indirect
system calls and use unsafe to pass parameters, you either do something wrong,
or we did something wrong. In this case the author of that code is excusable.
It was our fault. We didn't expose the necessary type-safe wrappers, so the
authors was forced to write code like that.

This is changing however, just yesterday I committed support for termio-
related stuff for Solaris. I assume Linux, BSDs and the other systems will
follow very soon.

------
laxk
The method proposed by the author does not work for packed structs. The btrfs
library has a lot of them.

main.go

    
    
      package main
      /*
      struct packed {
        unsigned char a;
        unsigned long long b;
        unsigned char c;
      } __attribute__((packed));
      */
      import "C"
      type Packed C.struct_packed
    

go tool cgo -godefs main.go

    
    
      package main
      type Packed struct {
    	A		uint8
    	Pad_cgo_0	[8]byte
    	C		uint8
      }
    

We lost packed.b field. Go doesn't support packed structs. More info here:
[https://groups.google.com/forum/#!topic/golang-
nuts/UX5srUMt...](https://groups.google.com/forum/#!topic/golang-
nuts/UX5srUMtSSw)

------
pcwalton
Using the syscall interface is a lot faster in Go than calling to libc,
because the system libc is going to expect large stacks and so you incur the
overhead of a stack switch when you switch out of M:N threading into the C
world. I assume that's why Go calls syscalls to begin with.

~~~
teacup50
However, calling directly into the syscall interface is illegal on most
platforms other than Linux.

I'm not sure Go _can_ fix this without getting rid of their M:N threading.

~~~
4ad
On Windows and Solaris, Go doesn't do its own syscalls.

~~~
teacup50
How about OS X?

(and, how does that work when calling into userspace APIs that assume fixed
1:1 thread/stack correlation)

~~~
4ad
On OS X Go does its own system calls, ignoring the OS X guidlines.

Go uses M:N scheduling, but you still have N threads with (relatively) large
stacks. When calling into C code, Go switches stacks, so it runs on the
(relatively) large stack where C code can run. Of course there's some overhead
associated with this, but system call overhead is large on its own compared to
this anyway.

During the system call the thread (obviously) can't run any _other_ Go code,
but that's ok, the runtime makes sure there are always threads available to
run user Go code (that's one reason why the runtime stands between user code
and system calls).

