If you have to pass a target buffer every time you call mp_read, that ends up using a lot of memory in a scenario where there are a lot of connections open but traffic is sparse. Since data might arrive at any open connection, you'll have to have at least one "pending" mp_read operation for every socket at all times. This quickly adds up in terms of memory usage: if you were to read into 64k-sized buffers (that's somewhat arbirary - but using smaller buffers tends to be very bad for througput) you’d need 625mb of memory for just these buffers alone to handle 10.000 connections. In a C10M scenario (http://c10m.robertgraham.com/p/blog-page.html) the same would need a staggering 625GB of memory. The good old select model lets you allocate these read buffers "just in time" before you read data into it.
This was a major issue when implementing libuv (node.js) for windows. Windows overlapped I/O has exactly the same problem (and so has that new RIO thing).
A better model would be to have a "buffer pool" that is kept reasonably full by the user-mode application. The kernel could then take a buffer from the pool as soon as data actually comes in from the network.
You can probably keep buffers very small with megapipe, since syscalls are batched together, so they can't impact throughput very much in case of many concurrent connections. You still get a lot of memory taken out of kernel per syscall, but spread across many connections, if I understood megapipe correctly.
Well sure you could do that. But I don't believe that reading say 50 bytes at a time is very efficient even when you factor in automatic syscall batching.
But this is all speculation. I didn't any consideration for this problem in the paper at all so I don't have the impression it was on the researchers' radar.
If you have to pass a target buffer every time you call mp_read, that ends up using a lot of memory in a scenario where there are a lot of connections open but traffic is sparse. Since data might arrive at any open connection, you'll have to have at least one "pending" mp_read operation for every socket at all times. This quickly adds up in terms of memory usage: if you were to read into 64k-sized buffers (that's somewhat arbirary - but using smaller buffers tends to be very bad for througput) you’d need 625mb of memory for just these buffers alone to handle 10.000 connections. In a C10M scenario (http://c10m.robertgraham.com/p/blog-page.html) the same would need a staggering 625GB of memory. The good old select model lets you allocate these read buffers "just in time" before you read data into it.
This was a major issue when implementing libuv (node.js) for windows. Windows overlapped I/O has exactly the same problem (and so has that new RIO thing).
A better model would be to have a "buffer pool" that is kept reasonably full by the user-mode application. The kernel could then take a buffer from the pool as soon as data actually comes in from the network.