

Ask HN: please review our online load testing service - loadimpact.com - rlonn
http://loadimpact.com

======
vaksel
I think your pricing is pretty much way off.

Why would I want to pay you $40/mo, if I'm paying $3/mo for shared hosting
that I know can handle 250 users?

And its the same way through out. Even at the highest level, it'd be cheaper
for me to lease a second dedicated server to spread the load, than it is to
pay you for your service.

And I don't think this is something that should be a monthly service. I only
want to check my load once in a while to see if its doing fine, so I think by
asking so much money on a monthly basis, you are driving users away. I think
you'll be better off doing it as one off results. i.e. the $499/mo option,
should be a $29.99 one time fee for 3 reports

~~~
teej
I disagree with you on both points.

First - the people on $3 /mo shared hosting are not in the target market for
this service. If you are on shared hosting, you don't need loadtesting.

Second - I think subscription is fine. As a subscription, I recognize that I
-should be- load testing on a regular basis. As a-la-carte, I am much more
likely to say "umpteen dollars?I can do my own loadtest for cheaper then
that."

~~~
mixmax
I'll second that. For personal use it may seem like a lot of money, but for
the target groups that needs this kind of service it's cheap. It's not just
about the cost of servers, it's much more the time spent doing it.

------
huhtenberg
Welcome to my IP blacklist.

But, seriously, this should be _strictly_ an opt-in service. Only if there is
a loadimpact.txt on the server, you run your tests. Otherwise - refuse and
explain that the server admin has not consented to the stress testing using
your service.

I'm sure you have some DoS provisions in place, but in the end it really boils
down to the fact that if _you_ screw up, then it's _my_ server that gets
DoS'd.

~~~
volida
this is true. Their servers shouldn't test if they don't find a specified
file, like Google does with Webmaster tools.

~~~
timf
Exactly, "claim your site"

------
mechanical_fish
You passed the first test: I asked myself "wait, how is this not a one-stop-
shopping service for aspiring, technically incompetent DOS attackers"? And I
found your FAQ and answered that question inside of 45 seconds:

<http://loadimpact.com/forum/viewtopic.php?id=7>

~~~
andrewljohnson
You failed my first test. I tried to load test my start-up -
www.trailbehind.com - using your free service. However, it said I had already
tried to load test my site and it failed.

I went through your registration process, and got nothing. Very disappointing
- the type of thing that will make me never look at your website again and
scoff about it to my friends should they bring it up.

And a couple other nits:

1) Your registration page where I choose my plan type is not intuitive. Maybe
it's because green is not a good color to call attention to stuff.

2) Your Register and Cancel buttons should be different colors. I clicked the
wrong one and had to redo my registration.

3) Don't ask for number of employees and industry. They're not required and
they will hurt your conversion rate slightly. And it indicates to me you might
sell my data.

~~~
rlonn
Sorry to hear you were disappointed. We will try and improve the registration
procedure, and also the feedback to users as regarding why load tests are
denied (often, it is because some resource used on your site have been used by
a lot of other sites aswell, and our system doesn't want to strain that
particular resource/object - but we have to be clearer about this).

Employees and industry is there for us to learn who our customers/users are.
This is a new type of service, and we need information on who are target
audience are, because we frankly don't know that yet. These values are
optional to fill in, though. Should we be more informative, maybe, and tell
people that we will absolutely not under any circumstances give out
information about them to a third party - would that reassure you?

Finally, your confirmation email might have been delayed as there was a DNS
issue yesterday that caused outgoing email to be queued for a while. Apologies
about that too and thank you for the feedback. It's very nice of you to give
it even though you weren't happy with the service. Negative (but constructive)
feedback is the most valuable of all for us.

------
jasonkester
Nice effort! After a few weeks of mediocre "review my site" submissions here,
it's nice to see something come through that I would actually use!

You did a good job of making the reports and graphs pretty while still keeping
a clean feel to the whole site. There seems to be a good amount of polish and
attention to detail there too. I don't feel scared to click anything while the
test is running for fear of breaking something or redirecting me to a FAQ page
and canceling the test.

I look forward to seeing how this pans out.

~~~
jasonkester
Ooh. Spoke too soon. Registration after finishing a test is really really
painful.

\- shelled off to the Pick a plan screen

\- pick the free plan

\- nothing happens, so click it a few more times

\- scroll down to see if I missed something and find WAAY too many required
fields

\- click register

\- screen scrolls up to the top, but nothing else happens

\- click register again

\- "this email is already registered"

\- find login box, type in email an password

\- get the homepage and a "Account not confirmed" message

\- open my email client, find the mail, click the link

\- get a "confirmed" page, and I STILL need to go type my email and password
in again.

Guys, you're killing me. Such a good product, with such a terrible
registration experience. It's completely unnecessary.

Ask for a username and password, and let me get on with trying your thing. If
I like it I'll engage with you further. That's Startup Usability 101. Fix it
and maybe you've got a shot.

~~~
hesdeadjim
Yea the registration process was very confusing. Few thoughts:

\- I had to scroll down on the plan selection page to even see there was a
form to fill out. This is on a 1680x1050 screen btw.

\- I expected to be taken somewhere when I clicked on the plan I wanted,
instead it was highlighted and with the previous issue in mind, I was left
confused about what to do next.

\- The most confusing part for me was after I fill the form out, hit submit,
and the page scrolls to the top. It took me a few moments to realize that a
"Registration email has been sent" box had appeared. I would recommend going
to another page after you submit the form.

That's all so far, I'm just starting to use the service so I may have more
comments, but already I can see it being a very useful service for the small
startup I work on.

~~~
hesdeadjim
Okay, actually ran your service against our beta site. The only (minor)
complaint that I have is that when I initiate a test there is a long delay
before I get to the page that is monitoring the test itself.

I assume there is a setup process that is happening behind the scenes that is
taking some time, but it left my finger itching to hit the 'refresh' button
while I was waiting for this to complete. If it were possible to have some
kind of ajax-y thing telling me what is going on here that would be great.

Otherwise I was very pleased with your service. I am definitely going to sign
my company up for it, as I can see it being invaluable for tweaking the
performance of our site.

As far as suggestions, it would be great if there was some way to work in
extended metrics into the load testing. E.g. allow pre/post-test hooks between
each test run that allowed me to associate arbitrary data such as load
averages, memory usage, etc.

------
jey
I tried to test my site (NSFW: <http://fapseek.com>) and got the message "This
configuration contains addresses that has been tested too many times the last
24 hours." I assumed someone else had tested the site and that I would be able
to test after logging in and verifying ownership of the domain. Even after
creating an account, logging in, and creating the loadimpact.txt file, I'm
still getting the same error message about my address having already been
tested too many times.

~~~
rlonn
That's our somewhat restrictive security features that notice some object
being loaded as part of your webpage, an object that other sites also use. For
example, many sites load urchin.js from google-analytics.com. This means that
the daily load our system allows itself to subject google-analytics.com to
will quickly be reached. Anyone trying to start a new test for a webpage that
loads urchin.js will then be denied to run their test.

We have a list of URLS/sites that are exempted from these checks, but this
list needs to be populated. So far we only have a few entries on it (like
google-analytics). It seems your site loads things from googleapis.com so I
added an exception for that address now. Try again and see if it works.

We probably need to work on the information to users whenever a test is denied
also. If you know what the offending URL is, you can remove it from your load
script and run the test without it.

~~~
tlrobinson
I'm getting the same error: <http://tlrobinson.net>

I have GA, Disqus comments, TipJoy and Reddit badges, and a few other little
things.

Maybe you could have an option to restrict it to your own domain name.

------
chops
Maybe I'm just retarded, but I couldn't find the answer to my question on your
site. Does your service just pound the exact page that's targeted, or does it
follow links, submit forms, etc to try to simulate actual users?

~~~
rlonn
The simplest test you can run will just load a singe page, and all its
dependencies (images etc) over and over again. However, if you register an
account you can create your own load scripts that load different pages, with
pauses in between. Paying users also get access to the recorder feature which
allows you to record a session that you run in a new browser window - you surf
around your site and click on things, POST forms, etc and when you're done you
have a load script automatically generated that will let your simulated
clients do the same thing.

We don't currently have any "crawler" mode though, where the system tries to
follow links etc on the site automatically.

~~~
imp
Wow, the recorder option sounds really cool. I didn't see that when I first
scanned the site. Maybe highlight a bit more somewhere.

~~~
rlonn
Yeah we only allow paying users to even see the recorder, which is probably
something we need to change I think. We want to give paying users extra
features, but we also need to show people what extra features they can get,
otherwise they won't know if it's worth to start paying. I guess more
help/documentation would help, but it would be nice if there was a way for
people to test a feature before they sign up to pay. Suggestions are
appreciated.

------
lazyant
I see a potential legal problem with the way the stress test is "authorized".
The thing is that web server access (putting the "loadimpact.txt" file in the
server) is not exactly the same as web site ownership.

My paranoid mind can think of an scenario like this (INAL so I maybe be off):
a web master asks for the load test, this is done and the site goes down for a
while, incurring a monetary loss (loss in sales for example). The owner sues
the testing company for a 'DoS' and the testing company has no way to prove
that it was authorized (what text file, there's none!).

I personally wouldn't do a stress test on a server without written
authorization of the owner. The way I would implement this on an online
service is by asking the target web site and the email address, where this
address has to be in the public whois information. Then I'd send a special
message to that email address and by replying the owner gives authorization
(this way I'd have something from the official email address). This is not a
'written authorization' but comes as close as it can and the whois email
address is used for domain transactions etc so it has the same level of
protection/legal procedure behind.

------
vladimir
Of course you should permit to run such tests only if you are sure you are
dealing with site owners or their representatives. You can use the system that
is used in Google webmasters tools - users can confirm their ownership by
adding specific document or meta-tag. Also - registration system's usability
is not perfect. But the idea is great - I would use this service.

~~~
rlonn
Yeah, this is something we have thought about also. We already have this
security feature that uses a special file (loadimpact.txt) on the target
system(s) to verify that it's OK to run large-scale tests. We should probably
improve on that and allow people to bypass the other checks if they have this
file. It is the safest way we have to verify that a test is legal, so we
should give it more priority before other checks.

------
agotterer
You guys should offer a daily or limited time use rate. I would think many
sites dont need to load test every month. But instead when they are first
building an infrastructure or making upgrades.

One of the sites I am working on now is moving to a new codebase. We wanted to
load test for a few hours but found everything we looked at to be too
expensive.

~~~
rlonn
That is actually on its way also. We're going to offer a "single test" option,
where a user gets to use the system for an hour. But maybe we should make it
24 hours instead?

~~~
agotterer
Maybe charge a similar way to how you are doing monthly... 1-5 tests are X
dollars 5-20 tests X more dollars. A single test is only useful if you want
know what your system can physically handle. If you are trying to tweak
settings to get the most optimal configuration, multiple tests will obviously
be needed.

Can you handle very large sites? I assume your system is in the cloud and
scales as needed?

~~~
rlonn
We don't want to make the different subscription offers too complicated, which
is why we've stayed away from having people pay per number of tests executed,
or similar. I agree that people will want to tweak and run several tests, but
a 1-day subscription sounds pretty reasonable then, I think, no?

We currently simulate max 5,000 concurrent users on a site and that is
something our own servers can handle, so right now we're not using cloud
services. We do have support for running the backend on a cloud though - we've
been testing it and it works very well. If customers start asking about
running larger tests, then we will probably implement a cloud backend mode
that will feature a much, much larger traffic-generating capability (but also
be more expensive).

~~~
agotterer
I think a 1 day subscription makes sense. But from your own usage standpoint,
theres a significant difference between running 1 or 2 tests and running 100
in a day. 5,000 seems like a good start. I'm looking for bigger :)

Good luck!

------
pierrefar
How is it different from the freely available ab and httperf?

Refs:

<http://httpd.apache.org/docs/2.0/programs/ab.html>

<http://www.hpl.hp.com/research/linux/httperf/>

~~~
rlonn
It depends in what way you mean. It generates traffic and measures performance
in a similar way as ab/httperf. The biggest difference is that it is an online
service, which means you don't need to own any servers yourself to run ab or
httperf on. You don't have to install or configure ab/httperf, and you get
results nicely formatted as graphs with no extra work on your part and test
configurations and results are always available to you, on the web.

If you have the time, knowledge, interest and a server to spare, then ab or
httperf (or I would recommend the Grinder - <http://grinder.sourceforge.net/>
) is definitely worth a try however.

------
modoc
I really like both the idea and the site.

One feature request that would have me signup in an instant is if you could
provide web service load testing as well.

I'd also like to know more about the session recorder and if I can feed
parameterized data into the scripts.

~~~
rlonn
Thank you, and while I'm at it: thank you very much everybody for all the
great feedback. It is very appreciated!

The session recorder is pretty cool, we think. We have managed to make it
zero-configuration (i.e. no user configuration necessary) while still being
able to handle client-side javascript. There are other HTTP session recorders
out there, but to be able to handle javascript they're usually implemented as
a true HTTP proxy that you have to configure your browser to use, or they're a
browser plugin that you have to download and install.

The recorder outputs a load script that you can edit afterwards, if you want.
However, the load script is currently more of a "list" of things to load, with
pauses in between. It has no logic, conditional statements etc. and does not
support parameterized data. This is something we intend to change soon though.
It just needs some thought so we don't open up new possibilities for
abuse/overuse of resources when we let users execute their own load script
code on our systems.

------
wesley
So, what can I learn about these results:

[http://loadimpact.com/result/www.improvingtheweb.com-b75533d...](http://loadimpact.com/result/www.improvingtheweb.com-b75533d03c65a3892b30ce9dbb9051f9)

~~~
rlonn
The graph is pretty flat, which indicates the test is not really putting any
stress on your server. The load added does not make much of a difference for
improvingtheweb.com. One thing we need to put more effort in, I think, is to
communicate these things. What can you learn from a graph, what are the
prerequisites for getting good test results (in this case, more load would be
a good thing).

I think there is a lot we need to do to explain things better. That is the
most difficult part of all.

~~~
yan
I think there's also a lot of value in personalized recommendations.

i.e. Load increased with the following stress pattern, which can indicate an
improvement is needed in areas A, B, and C.

~~~
cedsav
second that. Maybe something similar to ySlow?
<http://developer.yahoo.com/yslow>

------
asnyder
This is a good start, though not as comprehensive as we're used to. Currently
we use WAPT which is significantly more comprehensive while still maintaining
the ease of use that most other comprehensive tools lack.

That said, it would be great to incorporate some of their features such as
different scenarios, load patterns, particular series of pages, random, etc.
It would be great to not have to have a dedicated resource to run the stress
tests, it's also tricky to know if the local bandwidth can properly emulate x
number of machines.

------
jrockway
Very nice. Are you guys using any off-the-shelf tools for the load testing, or
is it all custom? (and if so, what language? Erlang?)

Also, is there a way to simulate more than 5000 users?

~~~
rlonn
It is mostly custom. The load generator is written in C and using libCURL -
<http://curl.haxx.se/libcurl>

5,000 users is what one single instance of the non-threaded load generator can
simulate today. We have some optimizations we can make that should improve
this though, and we are also going to implement multi-source/multi-program
load generation that means we can distribute the load generation for a single
test over several processes and several load generator hosts. Using only our
own infrastructure I think we can scale the system quite a lot, but obviously,
if we want to run really large simulations with several hundred thousand, or
millions, of simulated users, then we need to buy cloud server capacity.

------
rubentopo
Works well, i did like it, however i have two comments: -Please be able to get
<http://www.mywebsite.com> , www.mywebsite.com , mywebsite.com ,
<http://mywebsite.com> and lastly the good case <http://www.mywebsite.com>

-I believe that you'd have much more success if you charged on a test basis, your monthly billing might scare people away.

------
leftnode
I'm interested in how many paid registrations posting here on HN resulted in.
It seems like a pretty positive response here and a great tool! Great work!

~~~
rlonn
Thank you. We have had a ton of visits and a good number of registrations,
although only a handful of paid accounts as of yet. But that's OK, it's
natural that people want to try before they buy. We hope they'll convert once
they have tested the service a bit.

If you ask me, however, I'd say that the feedback we are getting here is
what's really invaluable. It will help us improve the service a lot.

------
quickpost
Under the "Need help interpreting your graph?" section all the graphs lead to
the same explanation! Need to have a different explanation for each one.

------
epi0Bauqu
On the free results page at the bottom there are some FAQ graphs, but they all
link to the same place.

~~~
rlonn
About the graphs, we're going to add more info about different graph types but
for now we just have one help text explaining "strange" graphs.

------
jusob
How does it compare to <http://litmusapp.com/> ?

~~~
timf
Another online service is <http://www.soasta.com> which is $1k/hour (cloud-
buzzword heavy but slick UI)

~~~
rlonn
Litmus is more about functionality/compatibility testing, unless I'm mistaken,
while our service is all about performance testing.

Soasta does essentially the same thing as we do although they are a lot more
expensive and aim for larger clients.

Traditionally, load testing has been both difficult and expensive, which is
why mostly larger organizations have been doing it. Soasta I think has
therefore aimed for this segment while we're betting on a larger, but so far
unproven, market among small- and medium sized companies.

~~~
timf
Best of luck to you, I think it's a good strategy. I'd be surprised to find
_any_ small business that would pay Soasta at those prices. $1k/hour is the
kind of money where I would hire someone instead to build a in-house solution
via EC2 etc.

------
paraschopra
I get this message on my personal domain: "This configuration contains
addresses that has been tested too many times the last 24 hours"

Does this mean somebody is DOSsing my site through your service?

~~~
rlonn
No, it is most likely because your site is loading some object - like
urchin.js from google-analytics.com - that a lot of other tests have been
loading also. That means that our security checks will think someone is trying
to DOS google-analytics.com and so stop your test. We have a list of
exclusions that the system should not bother checking (such as google-
analytics.com) but the list is not complete so people run into this problem
frequently right now.

I am going to fix so that we inform people what the offending URL/object is
(rather than just saying "one of the systems involved in the test..."), so
they can remove it from their load scripts (if you're a registered user you
can edit your load scripts before starting a load test) and also contact us
and ask us to put new systems into our exclusion list.

------
raamdev
Here's the test on Google.com:

[http://loadimpact.com/result/google.com-398c66a059f3ab3c096b...](http://loadimpact.com/result/google.com-398c66a059f3ab3c096be74fd5d2dab1)

~~~
rlonn
Strange, 50 simulated clients don't seem to stress Google much :-)

------
luminousbit
I love the site, HATE the pricing. I'd really like to pay for the service, but
what I REALLY need is:

1) Run several low user tests (200-300) to tweak server settings, app code,
etc. 2) Then run one or two big tests (1000+) just once to see the results.

I MIGHT need this at most once a month. That just isn't worth $200 per month.
I'd much prefer a subscription for the lower levels of consistent use, and
then be able to purchase an additional once-off very large test for maybe
$10-20 each.

------
sh1mmer
You are testing from Europe somewhere by the look of it. Many people have
scattered servers now. It would be good to be able to test from non-European
locations.

~~~
rlonn
We have two load generator nodes right now. One in Stockholm, Sweden and one
in Chicago, US. The default node is the Stockholm one, however, and only
paying users are allowed to choose what load generator they use. If people
want more locations we'll be happy to set up more nodes in the future.

------
brandinfection
Cool service! I tried it out and might get an account for testing with more
users.

------
omarish
Nice site, but how is this different from siege?

[http://www.linuxhaxor.net/2007/12/06/siege-an-httphttps-
stre...](http://www.linuxhaxor.net/2007/12/06/siege-an-httphttps-stress-
tester/)

~~~
rlonn
Siege is just another load testing software. The new thing here is that we
offer load testing as an online service. No need to install, configure or
maintain any special software or hardware. Test configurations and results are
always accessible, on the web.

If you're interested in running your own load testing software I would
recommend the Grinder - <http://grinder.sourceforge.net> \- which is very nice
and flexible. If you need to generate a lot of load, curl-loader might be of
interest - <http://curl-loader.sourceforge.net>

------
caustic
What would happen if I tried load testing loadimpact.com?

~~~
kevTheDev
I tried it, and it wouldn't let me as it had been tested too much in the last
24 hours

~~~
there
it let me run a test on <http://127.0.0.1/>

~~~
rlonn
Oops, I'm filing a bug report about that right away. Thanks.

~~~
huhtenberg
You may want to exclude entire 127.0.0.1 - 127.255.255.255 range, all of which
is reserved for the localhost.

~~~
rlonn
I've filed a ticket about excluding 127/8, private network IPv4 address space,
and multicast address space. Anything I forgot?

------
simplegeek
I'm interested in giving it a shot but site isn't opening. Is it just me?

~~~
asmosoinio
<http://downforeveryoneorjustme.com/loadimpact.com/>

