
Apache Bench and Gnuplot: you’re probably doing it wrong - bradleyland
http://www.bradlanders.com/2013/04/15/apache-bench-and-gnuplot-youre-probably-doing-it-wrong/
======
zimpenfish
It's been a long (long!) time since I touched Apache Bench or Gnuplot but
isn't that first graph "number of responses at this response time"? i.e. about
3000 of your responses were ~100ms or quicker; 4000 were under ~150ms; etc.

Your scatterplot has just unaccumulated that data - but it's the same data.

~~~
bradleyland
I definitely agree that it is the same data, and I plan to do some follow-up
to this piece because of shortcomings in my evolving view on what I just
learned. By "doing it wrong", I mean that many people believe that this data
is response time on the y-axis, and chronological time on the x-axis.

I hate to single out anyone, but Philip is obviously a smart guy, and I think
he's competent enough to not be hurt by a simple oversight like this. If you
look at his write up here, he uses the oft circulated gnuplot template (check
the comments):

[http://rubysnippets.com/2013/04/10/rails-4-live-streaming-
ve...](http://rubysnippets.com/2013/04/10/rails-4-live-streaming-versus-node-
dot-js/)

> On first sight, we immediately see from the graph that the response time
> using Puma at the end of the 10000 requests is pretty bad with 100
> concurrent requests, with the longest request taking around 60 seconds. I’m
> not entirely sure why this happens or what happens near the end, but here’s
> one plausible explanation: > When the benchmark starts, 100 concurrent
> requests are sent to the web server. A maximum number of 16 threads, and
> thus 16 requests, are allocated by Puma at once. The 17th request will block
> until one of the 16 threads currently in use is finished. However, since
> we’re executing 100 concurrent requests, there will be 84 requests waiting
> (100-16). Looking at the requests in the generated puma.dat file (generated
> with ab -r -n 10000 -c 100 -T 'application/x-www-form-urlencoded' -g
> puma.dat -p ../live_streaming/post <http://127.0.0.1:3000/messages>), we see
> that exactly 84 requests have been waiting for execution. These are the
> requests that were issued first, but have never been allocated to a thread
> by Puma. As a result, they have been waiting for the entire benchmark. I’m
> not sure why Puma would behave like this.

That protracted explanation is predicated on the inference that the data is
ordered chronologically. Many, many people make this mistake (google "apache
bench gnuplot"). I always thought it was as well. I don't know why I never
looked at the starttime or seconds columns of the data.

~~~
zimpenfish
Yeah, I see what you mean from his explanation. I wonder how/why the
misreading happens? Maybe it depends how much statistics you got beaten into
you at school or something.

~~~
bradleyland
I think two factors have caused this confusion:

1) Most of the people using the gnuplot template really don't understand what
it's doing

2) We all assume that the output of `ab -g plotfile` is a serial log

------
markild
I'm no statistics wizard, but I find it interesting that you conclude that
we're "doing it wrong" while at the same time suggesting that a scatter plot
where a huge percent of the points overlap is the way to go.

I appreciate that you try to up the resolution to counter this, but it still
strikes me as the wrong presentation.

~~~
leephillips
Well, he does suggest, at the end, that there are probably better
representations.

~~~
markild
Very true, but the whole idea behind the blog post seems to be that the
representation of data is done wrong.

I would rather the author go into a bit of a discussion about what the data
represents and what different ways this could be presented. I can't help to
get the feeling that the choice of a scatter plot is more or less arbitrary.

~~~
bradleyland
I agree with your criticisms. I wrote this last night and submitted this
morning, but after reflection, I even made some edits. For example, I started
by saying that the first graph was "wrong". Upon reflection, it's not. I think
more accurately, it just doesn't represent what people think it does.

If you Google search for "apache bench gnuplot", you'll find a very similar
gnuplot template that has been circulated for a very long time, but everyone
seems to think that the resulting plot is response time over time.

I'm definitely going to follow up on the issue. I'm a mediocre programmer, so
gnuplot is pretty hard for me, but I keep having these "ah ha" moments. My
next post will look at each graph in more detail and try to explain just what
is being shown in each.

~~~
markild
Very good to hear! I really didn't intend to come on as overly critic, and
look forward on read your follow up on the issue.

~~~
bradleyland
I really appreciate your feedback. I didn't think you were overly critical at
all :) I post my stuff to Hacker News so I can get tough reviews from smart
people.

------
dhruvbird
A graph which plots the time taken for the fastest 5%, 10%, 15%, 20%, 25%,
..., 100% of the requests would definitely be useful since it would give an
idea of the median and 3rd quartile requests, etc...

