Cory Benfield's PyCon talk last week, "Building Protocol Libraries the Right Way" (https://www.youtube.com/watch?v=7cC3_jGwl_U), makes the argument that a large number of problems can be traced to not cleanly separating responsibilities of actually physically doing I/O and making semantic sense of the bytes. His primary worry was about reimplementing things like HTTP many times, once for each I/O framework (why do Twisted, Tornado, and asyncio all have their own HTTP implementation?). But it seems the same problem can be seen here: every single part of the code thinks it knows how to actually retrieve data from the network, so it interacts with the network on its own, causing nested polling and similar awkwardness. If every part of the event-processing code thinks it knows how to do network I/O, you have many more opportunities for getting network I/O wrong.
If xterm were designed so that e.g. xevents() had only the responsibility of fetching bytes from the X socket and do_xevents() and everything else had only the responsibility of handling bytes from an buffer, there would be no temptation to poll in two different functions. Only one function would even know that the byte source is a socket; the rest just know about the buffer.
One of the nice things about Go is that the io.Reader and io.Writer interface being written into the base libraries means a lot of code gets this right, and only expects a stream rather than "a socket".
The takeaway here is not that Go is awesome; the takeaway is a lesson on the importance of getting a very early release of a language and its stdlib correct. The vast majority of modern languages today could trivially-to-easily do the same thing, but they don't in the standard lib, so the first couple of libraries end up string based, so the next libraries that build on those end up based on strings, and before you know it, in practice hardly anything in the ecosystem is implemented this way, even though in theory nothing stops it from happening. (Then around year 3 or 4, a big library gets built that does this correctly, but it's too late to retrofit the standard library and it only ever gets to about 10% penetration after a lot of reimplementation work.)
The more I see such problems the more I like erlang. Most socket handling libraries split handling into protocol handling layer and application layer. Protocol layer ensures there is full message available and application layer handles only full messages. Most of the time it's the simplest and most natural way to do anything in erlang.
Erlang really gets this right. Abstract out all the generic server stuff and have it coded up by experts, then have the application programmers concentrate on the application. A bit like programming a plug-in for Apache but then extrapolated to just about anything you could do with a server. Erlang is a very interesting eco-system, the more I play around with it the more I like it and the way it is put together. If it had a shallower learning curve it would put a lot of other eco-systems out of business. But then again, the fact that it doesn't makes it something of a secret weapon for those shops and individuals that have managed to really master it.
> If xterm were designed so that e.g. xevents() had only the responsibility of fetching bytes from the X socket and do_xevents() and everything else had only the responsibility of handling bytes from an buffer, there would be no temptation to poll in two different functions.
X is an interesting special case. The X protocol has some special cases where you have to make sure you read before you write, or vice versa; doing the wrong buffering or blocking operation can result in a deadlock between you and the server.
I certainly enjoyed that PyCon talk, and I agree with the conclusion; however, there are some special-case protocols like X where integrating them into your main loop requires some special protocol-specific care.
I think you can solve this by reporting an "I can't read unless you write some more" event, or allowing a "I can't write unless you process some events" return code from the write function. You need some protocol awareness (you can't completely abstract every protocol as bytes -> JSON and JSON -> bytes), but it doesn't rise to the level of letting application code directly have access to the underlying file descriptor.
I believe both SSL and SSH have similar issues, where the state of the protocol client requires that you order reads and writes in some way to avoid deadlock. I guess TCP also has the a similar risk with window sizes going to 0, and in practice, hiding TCP behind a UNIX file descriptor and a relatively constrained socket API works fine; client apps don't need to care about the exact state of the TCP implementation.
If xterm were designed so that e.g. xevents() had only the responsibility of fetching bytes from the X socket and do_xevents() and everything else had only the responsibility of handling bytes from an buffer, there would be no temptation to poll in two different functions. Only one function would even know that the byte source is a socket; the rest just know about the buffer.