For average commodity hardware I found something like 400 reqs/s/node was normalish, even sustained. Yours looks like about 2 minutes in it dies. Come to think of it, could you have your open file descriptors limited in the OS settings? That looks just like pattern I'd expect to see from that.
Might be unrelated but common pitfalls I had were:
- Using the HTTP proto. Protobuf is way faster.
- You can tweak the r and w values to get less read and write consensus when you can afford to, depending on the task and data.
- ulimit open file descriptors might be too low.
In any case, if you were to do a short writeup, I'm sure the basho guys at the mailing list would be interested.
I was monitoring with iostat and a couple of other tools. It was certainly very heavy on io, with 80% util, 20% iowait, and that increased as the currency went up.
I was using protobuf, and a w value of 1, so I was out of things to optimize.
When I was inserting objects already in Riak's cache, it ran about 3 times faster, but of course that's not possible with new objects.