
The Cost and Complexity of Cgo - orangechairs
http://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/
======
lazyfunctor
Interesting! I wonder if someone can help me with a problem I'm facing with
cgo. I'm trying to include go runtime as apython module (.so) and the code
appears to work fine as a standalone program. But when I try to run it from a
pre-forked process manager (say Celery for python) the code gets stuck at
runtime.futex (from strace). I guess some interplay of python and go runtime
w.r.t goroutines. I feel Go is not meant to be embedded in Python runtime.

Code here:
[https://github.com/lazyfunctor/pymosaic](https://github.com/lazyfunctor/pymosaic)

Edit: Added last line

~~~
jerf
"I feel Go is not meant to be embedded in Python runtime."

I suspect that it is going to ultimately be your core problem. Even if you
make it work with specific versions or specific modules, it'll always be
fragile.

Like many modern languages that embed an event loop into themselves one way or
another, the Go runtime expects to fully own the target process. It makes it
difficult to play well with others, because the "default" C-based runtime
provides so few guarantees that it's difficult for two otherwise C-compatible
runtimes to "play nice" with each other. It's part of why the JVM and .Net are
so interesting for cross-language development; the VM provides a richer,
better-specified "base layer" that multiple languages can use, enabling much
easier interop.

It's a death-by-a-thousand-cuts sort of thing; hypothetically nothing really
_prevents_ this, but in reality you're probably not going to want to take the
time to make it work which may very well involve custom modifications of
either runtime, when you should probably just go ahead and set up some RPC
connections. (JSON's a nice default to prototype quickly, but depending on
your needs you may want something like protocol buffers or the half-a-dozen
competing libraries.)

------
nothrabannosir
The "C managed memory" problem is definitely ugly. Some libraries and APIs
allow you to protect yourself from this by giving you the option to supply the
memory yourself, instead.

If your library can be used completely with you managing the memory, you can
use Go's allocator. E.g. libopus:

    
    
        size := C.opus_encoder_get_size(C.int(channels))
        mem := make([]byte, size)
        p := (*C.OpusEncoder)(unsafe.Pointer(&enc.mem[0]))
    

This way, all memory is on the Go heap.

Not every C library will support this, but those who do make CGo a lot more
bearable.

------
namelezz
Thank you for sharing. I think you mean 1000 not 100 "100 goroutines come
pretty much for free in Go;"

~~~
orangechairs
[cockroach labs employee here] Post corrected. Good catch. =)

------
jaekwon
Thanks for the pointers, cockroachlabs. Your posts really help developers
building real applications on Golang.

