
Performance Tuning Apache Storm at KeenIO - neom
http://highscalability.com/blog/2016/3/8/performance-tuning-apache-storm-at-keen-io.html?second_pass
======
dzello
Great post! Makes a lot of sense to move I/O to a thread pool. Do you control
max topology I/O with the thread pool size now, in addition to Storm settings
like # of executors? Another q - do you monitor how long each bolt blocks
waiting for a thread to become available? (as a measure of pool saturation)?

~~~
manum
Hi

The max thread pool size definitely affects the maximum I/O we can do and
we've tuned that based on perf testing the memcache/cassandra clients in
isolation.

However the bottom line is that we are still limited by the max spout pending
or the max number of queries we can run concurrently. There will be short
spikes that might use up all the I/O threads but they are short lived because
there are other CPU intensive operations that the topology is doing at the
same time.

In terms of blocking for each bolt, we don't block, the bolt tries to execute
the operation in the current thread if the I/O thread pool is saturated. This
way if there are pending tuples they queue up in storm's receive queues and it
creates a back pressure when we have too much work outstanding.

Hope this helps! Manu

~~~
dzello
Cool idea for how to still let Storm apply its own backpressure, thanks for
sharing!

