
Poll vs Epoll, once again. - jacquesm
http://www.jacquesmattheij.com/Poll+vs+Epoll+once+again
======
bandushrew
interesting.

The discovery of the edited source is a nice, solid addition to the
discussion.

Ultimately though, AFAICT you and Zed are looking at - and generally agreeing
with - the same facts, ie. that epoll and poll perform better depending on the
situation.

If I am understanding you correctly, where you both are disagreeing is that
you are determined that no webserver will actually reflect the usage pattern
that would make it worthwhile for Zed to implement this 'superpoll' feature,
while Zed is thinking that it will be worthwhile.

Which makes you seem like two reasonable people agreeing on the facts and both
coming to a different conclusion, unfortunate but it does happen (and always
fascinates me).

I wonder if part of the reason for the differing conclusions is differing
expectations for Mongrel2? you seem very focused on 'expected real-life
scenarios for existing websites', but I do have the feeling that Zed is
anticipating/hoping/expecting that Mongrel2 will have a broader set of uses
and be used in other ways in addition to the standard webserver.

~~~
abstractbill
_I wonder if part of the reason for the differing conclusions is differing
expectations for Mongrel2? you seem very focused on 'expected real-life
scenarios for existing websites', but I do have the feeling that Zed is
anticipating/hoping/expecting that Mongrel2 will have a broader set of uses
and be used in other ways in addition to the standard webserver._

I can't speak for him of course, and I may have misunderstood some of the
details of these posts[1], but it seems like jacquesm is saying that the cases
in which poll beats epoll turn out to not actually be _high-scale_ use cases
in any interesting sense (and his arguments seem compelling to me). In which
case it does seem safe to not bother optimizing them.

I certainly wouldn't discourage Zed from _trying_ though. Who knows what might
come of it?

[1] I write a lot of Twisted code that _uses_ epoll - I wrote the justin.tv
chat server for example, which has scaled to half a million concurrents, and I
believe I get the basic concepts of epoll/poll/kqueue, but I've never actually
tinkered with the details.

~~~
bandushrew
He says "Also, you should take under consideration the workload of your
system, if you are sure that you will have a very large portion of your fds
active most of the time and that your worker pool is sufficiently large
compared to the number of connections that you have that there is no reason
not to use poll (think media/file servers), whereas if you are fairly sure
that the majority of your fds are inactive and/or that the size of your
workerpool is substantially smaller than the number of fds that you have open
(think web servers) that there is no reason why you shouldn't use epoll.

That's the precise reason those two calls exist in the first place, and which
one is 'best' (for whatever bit of extra throughput you can squeeze out of a
machine because of this optimization) depends on your traffic."

To me it sounds as if overall he is saying roughly two things: (1) "There are
server conditions where epoll works better, and server conditions where poll
works better,"

and (2) "I dont believe that web servers specifically under real-life
conditions are likely to be improved much by poll"

Clearly he and Zed agree on (1), that being the point of the related Zed post,
and (2) might well be true, but I dont think it necessarily speaks directly to
the use cases that Zed is hoping will eventuate for mongrel2.

Zed used a web server to test the differences between epoll and poll, and that
has clearly focused the discussion somewhat, but from what I gleaned by
reading his ongoing 'The Coming Of Mongrel2' series, he has a wider range of
possible uses in mind for Mongrel2, and I am wondering whether that is where
his and jacques conclusions are differing.

[1] I mean, clearly they are _both_ entirely insane in a very real sense, but
I do love to pretend that those around me are rational actors and enjoy
projecting rational explanations for their behavior upon them.

~~~
bandushrew
sorry? why am I being downvoted? surely if you think Ive missed something in
the above summary it would be more constructive to post a reply?

~~~
jacquesm
It may have to do with calling Zed and me insane. I don't know what you base
your knowledge of me on but in defence of Zed he does not at all come across
as insane to me, though he definitely does seem to have a pretty rough edge.

------
wglb
Do you have any hard evidence that _I think (and have observed in real life)
that servers spend more time doing other stuff than they spend time polling,_
is true?

~~~
bandushrew
The call for great quantities of hard evidence that is occuring on both sides
is both entirely reasonable and utterly unrealistic.

These are two good developers who are drawing on their real life experiences
to develop their opinions, backed up with some simple tests performed on local
machines.

While it would be pleasant if they could provide hard data for a wide range of
webservers to 'prove' their opinions it is hardly a realistic option for
either of them without investing a degree of time and effort that is far
greater in scope than the work being discussed.

I would prefer to see less of a cry for hard evidence of real life usage
patterns from either side, and a greater acceptance of the fact that this is
simply an interesting discussion of technical possibilities by some good
developers.

~~~
jacquesm
I've given what data that I have easy access to (from a single threaded
asynchronous home brewn server that sees an awful lot of traffic) in that
other thread. Now that's a special enough case that I think that the
conditions under which normal servers operate will be skewed considerably in
the _downward_ direction compared to that one and even there I never observed
an active-to-total ratio of more than 40% over a span of 5 minutes during a 2
hour run.

Instrumenting varnish or apache would be a lot harder to do because I don't
know those code bases well enough to know for sure that I'm doing it just
right so all I can rely on there is the data reported by the 'status' modules
of those programs, and for now neither of those suggest otherwise.

Zed attacks that data with a bunch of handwaving but to date has yet to show a
single instance of a server which actually has an active-to-total ratio that
is higher than 60%. I suspect such servers do exist, media servers, file
servers and other other servers with relatively few connections, a large
worker pool compared to the number of active connections and lots of long
files being shipped across those connections.

But your typical web server is more likely to be stuck in an endless cycle of
accept-read-write-close where both the 'read' and the 'write' can block for
significant amounts of time compared to the number of bits shipped, and the
backend can do the same thing once more. So idle is the norm.

------
grandalf
Uh, Zed stopped being inflammatory quite a while ago now.

~~~
dschobel
Are you being facetious?

If so, read the comments here: <http://news.ycombinator.com/item?id=1570694>

------
bandushrew
wow. the downvoters are strong in HN tonight.

an actual discussion would be an interesting thing.

sure, its _easier_ to just downvote someone as you go past without actually
engaging sufficiently to understand or discuss the issues, but ask
yourself...is that really who you want to be?

~~~
abstractbill
<http://ycombinator.com/newsguidelines.html>

"Resist complaining about being downmodded. It never does any good, and it
makes boring reading."

~~~
bandushrew
Its not the downmodding I am pissed at, its the lack of decent conversational
skills.

