> Go garbage collector requires that all threads are stopped when running a collection and that memory must be in a consistent state.
People often ask about how Go differs from Erlang. That one's a fairly large difference under the hood. Erlang does GC on a per process (Erlang process, not OS process) basis.
I hadn't looked at that issue before, but have definitely been bit by it.
On our main codebase, we experienced some major issues when moving from Go 1.0 to 1.1 from this exact issue. We had a goroutine that was doing some remote calls wrapped with a timeout, and the call was consistently timing out even though the remote service was perfectly fine (it was another service on the same box).
We found the cause was another goroutine running something in an for loop that didn't do anything that would allow a pause in execution for another goroutine. So, the scheduler just obsessed on that goroutine, running none of the others until that one was done, and by then the timeout on the remote call had expired.
We fixed up that case, but also found a few others were we simply added a very small sleep call... for no other reason than allowing the scheduler to evaluate other goroutines. Meh. It made sense when we finally tracked it down, but was one of those things where we had to pause and ask "really?"... and adding a sleep call with comments "yes, I am really calling sleep".
While it would be nice if the programmer never had to worry about this situation in the first place, when confronted with something like this I use runtime.Gosched() rather than a sleep call to yield. It more directly performs what you're attempting to do and is much more clearly self-documenting for situations where you really don't need to sleep for whatever period of time but do need to yield.
I hadn't seen runtime.Gosched() before, will take another look at it. Mentioned it to a coworker and they already knew of it, so maybe it was always switched to call it instead of sleep. :)
I had what I think was a related problem with file descriptor retention in timeouted socket code when writing a simple portscanner in Golang; I had to implement simple time-based flow control to give the runtime time to collect file descriptors, or I'd run out of them.
If you use Go's time.Sleep(), the goroutine will yield to all other goroutines while it is sleeping. Unfortunately, as georgemcbay says, using the system's sleep impedes Go's runtime as it cannot use that thread to run any goroutines in that time.
But yeah, I would say that is one of the quirks of Go at this time. You definitely need to be aware of how the scheduler works if you're using it in production.
This doesn't make a whole lot of sense to me. As long as you have GOMAXPROCS set > 1 this shouldn't happen, right? Go doesn't make a secret about only supporting cooperative multitasking above the OS thread level. Are you saying that one spinning thread was locking up all the others?
EDIT: Okay, saw your reply below. Just to reiterate, Erlang has to make a lot of throughput compromises to support pre-emptive multitasking. Just being compiled pretty much takes Go out of that conversation entirely. I'm happy with the tradeoffs for the kinds of things I need to do. And you can always set GOMAXPROCS to a _multiple_ of the number of cores on your machine to get OS-managed pre-empting.
So, does that mean it's safe to say that Erlang must copy the memory of values sent as messages between processes? Or, is there some kind of locking of such (aliased) message data???
Messages are deep-copied in Erlang. There are other techniques possible such as uniqueness types, which allow "transfer of ownership", Rust uses those but Erlang does not.
> So, does that mean it's safe to say that Erlang must copy the memory of values sent as messages between processes?
Theoretically not since Erlang data structures are immutable, but outside of binaries which seat in a shared global heap the benefits was never considered to outweight the complexity costs (aliasing analysis and others), so messages are copied indeed.
I would think so unless it stores all message bodies in shared memory and protects them with an enforced COW policy to begin with, or better yet does this with virtual memory with some clever COW mmapping.
https://code.google.com/p/go/issues/detail?id=543
Also, from the article:
> Go garbage collector requires that all threads are stopped when running a collection and that memory must be in a consistent state.
People often ask about how Go differs from Erlang. That one's a fairly large difference under the hood. Erlang does GC on a per process (Erlang process, not OS process) basis.