I had what I think was a related problem with file descriptor retention in timeouted socket code when writing a simple portscanner in Golang; I had to implement simple time-based flow control to give the runtime time to collect file descriptors, or I'd run out of them.
