> I do not think Nim code can run "in a goroutine".
It does. Don't forget that this is gccgo so it is possible to use plain C functions as goroutines. Nim is translated to C and with the help of a macro I convert Nim functions with an arbitrary number of arguments into ones with a single void* arg that gccgo wants for its 'go' keyword implementation:
Yes, but when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it? If it doesn't, then nothing prevents foreign code from blocking other goroutines. How does the Nim code yield the thread for other goroutines, does it have to register a callback?
> when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it?
No.
> How does the Nim code yield the thread for other goroutines, does it have to register a callback?
There are no callbacks. Yielding happens automatically when launching another goroutine, when sending, receiving or selecting on a channel. You can also yield explicitly with go_yield() - the better named equivalent of Go's runtime.Gosched().
It's easier to understand if you realize that all those operations with goroutines and channels end up being done in the Go runtime.
> > when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it?
>No.
Are you sure? Once a thread enters cgo it is considered blocked and is removed from the thread pool according to these sources [1][2]. I previously found a thread where Ian Lance Taylor explained it more explicitly but I couldn't find that now. Is that not what is happening though when __go_go invokes your function pointer?
I do not understand how the Nim code can live in the segmented stack of a goroutine, nor how the Go runtime could know it is time to grow that stack.
Yes. See the chinese whispers benchmark with 500000 goroutines and a maximum resident set size of 5.4 GB on amd64. It has the same memory usage (and run time) as the Go version compiled with gccgo.
> Once a thread enters cgo
This has nothing to do with cgo. It's a different mechanism specific to gccgo.
> I do not understand how the Nim code can live in the segmented stack of a goroutine
Good thing you asked. I just ported to Nim the peano.go benchmark described as a "torture test for segmented stacks" and... it failed. The fix was to pass -fsplit-stack to gcc when it compiles the C code generated by nim.
> nor how the Go runtime could know it is time to grow that stack
From what I can tell it's done in __splitstack_makecontext() and friends from GCC's libgo.
It does. Don't forget that this is gccgo so it is possible to use plain C functions as goroutines. Nim is translated to C and with the help of a macro I convert Nim functions with an arbitrary number of arguments into ones with a single void* arg that gccgo wants for its 'go' keyword implementation: