> Here are a few numbers: StormForger did a total of 1,213,583,187 requests in over 50 load test runs to the Quizduell system and a total of about 2.21 TB of data was moved. The error rate was at about 0.000000216% (1 error every 4,624,616 requests).
Sounds useful for doing end-to-end from a cloud provider to another cloud provider, but there's more lessons to learn when you have 20 different ISPs and a couple dozen different clients sending you traffic in different ways.
For example, for free services without customer support, usually you don't notice the bugs from all these different devices trying to access your service under load. Once you start charging and provide support, suddenly people are calling you with all kinds of edge cases that affect a minority of users, and you have to invest time in tracking them down or refund the money (which is a bad idea since that builds a reputation that you'd rather dump customers than help them).
Impressive. It's not their fault and they seem to be good and talented, but I can't stop thinking about a possible marketing line: "DDoS as a Service".
this is especially challenging actually. just because you own the domain name, doesn't mean you own the infrastructure you are pointing your DNS records to!
So adding DNS records is pretty useless to provide proof that you own the resources under test.
Something I have been really interested in is being able to replay the actual traffic our site is experiencing. So, we press record and everything that's going to our server is recorded for playback at a later date. Does anyone know any solutions like that?
There are a couple solutions that do that - none of them I find sufficient. If you want to clone live traffic, that's probably okay, but recording traffic and replaying it in a usable fashion is a non-trivial task. e.g. systems tend to be dynamic so your recordings would need to be dynamic too in order to be realistic.
The api looks alot like tsung. I'm almost willing to bet that either they got inspiration from it or are using it on the server. Anyway, tsung is great and these services are great too.
if you look at the 'your data - your results' piece of the carousel, it has a log which has 'pid' as a column and, e.g., '<7038,69,0>' as a value. So you're probably quite right.
I can't recommend Redline13 (https://www.redline13.com) highly enough. They act as a service that coordinates load testing using instances within your own AWS account.
Really easy to use, and I can run tests that used to cost upwards of $5,000 for under $5. I don't know how they aren't better known amongst DevOps types.
I saw basti's talk on stormforger at last month's interactive cologne – and I think he switched to JavaScript for the test definitions. Maybe the website is a bit outdated. :)
Heroku started with just Rails. It's better to start with one language and do it well and then expand than half-ass it with everything from JS to Erlang imo. The important part is the quality of the service and the information it provides, not the DSL.
WRK is a nice tool that can bring you quite far, for sure. But what about the orchestration of complex sessions? As I understood that blog post mentioned in another comment StormForger was actually able to play a game. A simple one, but there is still a lot of complexity involved.
But even if you can build something like this with WRK (which should be possible, due to the Lua support) you still need to setup your test environment (reproducibly), collect all the logs from the machines involved, monitor the test generators, create useful metrics from the test results and make it possible to interact with them. And I'm pretty sure I forget a whole bunch of stuff that needs to be done.
And I know, that I don't want to do all this stuff by myself. With a service providing all this, I can concentrate on the real work: Interpreting numbers and fixing problems ;-)
BTW: I'm pretty sure under the hood there is some load testing tool deployed. Cannot imagine someone runs such a large scale test without such thing.
