Ask HN: Do you do periodic perf/scalability test for your service. If so how? - petrohi
======
cimmanom
Yes, but we run them in production.

Kind of kidding. Our service is strongly seasonal, and there's a specific week
every year when our traffic can double or triple compared to the low points of
the year.

In the past, that week reliably revealed performance bottlenecks. We would
then make it a priority to fix the most recently discovered bottleneck before
the next year.

This past year, we finally had a crunch week with zero performance hit.

We've also in the past had trouble with bottlenecks when hit by ill-behaved
bots. That problem is also finally solved - we haven't had a bot-related
outage in several months.

Based on the load and performance profiles of our infrastructure at throughout
levels that have caused outages in the past, and our current load at weekly
peak, we now have room to grow traffic 10x without hitting another ceiling.
And we're a slow growing business, so that gives us at least a few more years
before it becomes a concern.

Note that this strategy is not an awesome one and is in fact VERY bad if you
run a service with SLAs; with traffic patterns that include extreme spikes; or
where high availability is critical. But it's worked out just fine for us.

