

Ask HN: How do you test your web applications? - unignorant

I'm an undergraduate who happens to be taking a graduate seminar in Programming Languages. Recently, we read a paper on testing web applications, and the (perhaps cynical) notion among the grad. students was that "no one tests web apps," which I felt was wrong.<p>So to possibly vindicate myself, I'm curious as to how many here do test, and if so, how?<p>Edit: Thanks for the responses! To be more clear, we were discussing testing more along the lines of "finding bugs and vulnerabilities" and less along the lines of "usability."
======
cschneid
Testing is a vague term. There are lots of types of testing, which other posts
go over quite well, but I'll give a rundown.

* Requirements testing - take the napkin you scribbled your idea on, and mentally run through it a few time. A feature that is 100% correct code but does the wrong thing, is still a "bug".

* Unit testing - low level, typically at the database and business logic level. Does this function return what I think it does?

* Integration / Frontend testing - automated scripts that run through and 'click' on elements, and verify content. Webrat and Selenium are a great tools for this.

* Usability testing - throw some real users at it, find bugs in your design and workflow.

* Penetration / security testing - attack your site from the outside. Alternately, audit the site from a code-level point of view. Find security holes, figure out ways that people will abuse your site.

* Load testing - once you get a site up, will it stay up after traffic hits. This is a rather deep topic in and of itself.

You get the idea. There are probably more that I didn't hit. But basically
every step along the way to having the final app can and should be tested.

Testing is more than finding bugs, it's finding flaws in thinking. Whether
it's the 'business analyst' aiming for the wrong target, the designer having a
hard-to-find button on the mockup, or a coder using the wrong import()
function override.

------
tptacek
Since I do this professionally, I have a lot to say about testing web apps,
but I'm going to keep my recommendation short and precise.

Go here:

    
    
      http://portswigger.net/suite/
    

Drop the $200 or so for a copy of Burp Suite Pro.

Burp is the industry standard tool for testing web apps; probably more than
half of all web app penetration testers use it. It's like Firefox Tamper Data
on steroids; it gets between you and your request, breaks out all the
parameters for you, and gives you tools to manipulate and replay the requests.

You in particular want to understand the Burp Intruder, which is Burp's
fuzzer. The fuzzer takes a request, tries to figure out where all the
parameters are (you can help it with that), and then feeds malicious inputs
through them and records the responses.

You want to fuzz every page in your application at every code drop. We've seen
entry level QA people trained to do this, so delegate the task if you're not
sure you can make yourself do it reliably. You want to collect every error
response, and you want to grab your application logs and look for every
exception or abnormal event. You want to know how the responses are different
from the last run each time you ship a new version of the app.

Burp is so extremely effective that I'm amazed every web dev team doesn't
already have a copy. I think it's because it's billed as a "security" tool.
But it's obviously more than that.

Webdevs have a lot to say about unit testing and cloud testing and integration
testing and test-first development. What I can tell you is that I've seen no
correlation between test methodology and pentest outcome. The TDD teams fall
just as fast as the seat-of-the-pants teams. I want to strongly recommend that
you don't waste time strategizing on how you'll test or validate architecture
or whatever. Burp is a no-BS tool. Get it, run it, and you will find bad
things. Work out your effective bill rate and the amount of time it takes you
to adopt or extend a test suite, and it's the cheapest quality enhancer that's
probably available to you. Do whatever you want on the testing side, but do
Burp first.

~~~
jrockway
s/Burp Suite Pro/QuickCheck/.

Basically, there are two stages to a web application; understanding the
request, and processing the request. The understanding part is a parser, so
you are going to have to ensure the correctness of that with test cases -- you
can never _prove_ correctness, you can only show that it works on a lot of
input.

The rest of your application, however, will be modeled on a finite number of
possible states that this data structure represents. This means you can use
better tools (things like Catch) to ensure correctness; you can literally
prove that there is never a case you don't handle, for example. This gets you
reliable code more quickly than random testing does.

Anyway, to answer the OP's questions, web apps should be tested like
everything else. A web app is a pure function from Request -> Response, so
test it like any other pure function.

------
tbgvi
Selenium is what I'm using, Selenium IDE (firefox plugin) makes it easy to set
up tests. <http://www.seleniumhq.org>

UserTesting.com is great for usability testing, they have people follow some
of your instructions and record their screen/voice as they go through it.
Feedback from this has lead to a lot of critical improvements.

Another article you might be interested in is about how software is developed
for the space shuttle (they test like crazy). Someone posted the link on HN
last night and it's fascinating:
<http://www.fastcompany.com/magazine/06/writestuff.html>

~~~
nethergoat
On a related note, one of the authors of Selenium IDE founded a startup called
BrowserMob (<http://browsermob.com/>) that uses Selenium recordings for load
testing and monitoring.

Great guys there, and a fantastic service. I've been a customer for over a
year - it's one of the few companies I'm impressed with enough to evangelize.

~~~
hugs
Shinya Kasatini is the original creator of Selenium IDE, based on my version
of Selenium (which was simply called "Selenium Core"). Shinya is also famous
for creating the "Pocket Guitar" iPhone app.

Patrick Lightbody is founder of BrowserMob and is the co-author of Selenium
RC. RC is the Java-based Selenium server for driving browser testing from any
programming language.

I also do lots of stuff with Selenium at my startup, Sauce Labs. Sauce focuses
on making your functional/acceptance testing cycle go faster by providing
hosted "browser in the cloud" infrastructure where your tests run in parallel
for a huge speed boost.

------
felideon
Here's a guide on penetration testing you might find useful. I would assume
some frameworks handle some/most issues "out of the box" but it's still a good
guide to follow.

[http://www.owasp.org/index.php/Category:OWASP_Testing_Projec...](http://www.owasp.org/index.php/Category:OWASP_Testing_Project)

Other than that, testing a web app is just like testing any other software.
Look for edge cases, try different combination of things, etc. to say the
least. But here are a few things to look out for off the top of my head:

* Don't be too restrictive on validation on fields like names, addresses/zip codes (if there are international customers), e-mails, etc. However, make sure people can't enter numbers in an SSN field or letters in a date field either.

* Make sure weird characters in different fields don't break anything. For example, entering O'Hara as the last name should be stored correctly in the DB and then rendered correctly to the user whenever you need to show it.

* Make sure your text box max lengths are not greater than the DB's column definitions. You don't want someone to enter a 50 letter name and have the field in the DB defined as varchar(20).

* Make sure the web app handles session time outs gracefully. You don't want a user filling out a registration form to get a phone call and then come back to complete the form, only to be redirected to the starting page when he clicks Submit.

There are also automated frameworks you can use to script automated tests,
such as Watir/WatiN. (Please don't ever waste money on QTP.)

And as others have mentioned there's always load testing and stress testing
you need to do at a certain point.

~~~
coolnewtoy
Thanks for posting this. I will definitely add this to my ever-expanding lit
review for my thesis on SQL Injection Attacks/Web App Development Frameworks

------
nandemo
I've been in a few web projects and all had a testing phase just as any other
projects: creating test plans based on the requirements and specification,
doing functional testing, integration testing, load testing, and so on.

For what is worth, most of these applications were internal apps for large
companies (users are the employees, not customers) and might as well have been
developed as desktop or client-server apps. I see no reason why the choice of
architecture/deployment would affect the QA policies. It might affect the
particular way of testing (tools, etc) but not whether to test or not.

I can't see quite see why is this even a question. Perhaps your colleagues are
assuming that all web apps are like the typical apps they might use: B2C, free
or "freemium", etc. Other things being equal, these are more likely to be
released without careful testing.

------
MicahWedemeyer
Rails comes with automated testing baked in, and it's one of the best
unit/automated testing packages I've ever seen.

Then...people went and made it better with things like RSpec, Shoulda, Factory
Girl, and other enhancement packages.

I've given up trying to convince others to do automated testing, but for me
and my projects, there's really nothing that beats what Rails and the Ruby
community provides.

------
vital101
There are many types of testing, almost all of which are useful. From a client
perspective, usability testing is probably the most important. Having a
testing group come in, use your app, and then give you immediate feedback is
invaluable.

There is also penetration testing to check you web apps security. Could you
maybe be more specific? I think it would help you get some great answers from
the community.

------
CoryMathews
One of the many web apps I have worked on do something similar to the
following:

before any feature rolls out each feature (from a list) is tested with
different (but not every possible) set of inputs. This catches 90% of
problems.

The other 10% that get past our testing are either sent in by the user through
a feedback from, or by the system sending messages (errors warnings etc)

------
Shamiq
I've used Selenium to run unit test along with various flavors of Watir. Watir
requires much less overhead and is programmable in Ruby. Selenium has
libraries in all your favorite libraries, but requires running a Java proxy
server. However, you can automate tasks by having Selenium record your actions
on a page, which is super nice.

~~~
hugs
Selenium 2 does not require the Java proxy server. That was one of the top
complaints with Selenium 1, so we got rid of it. :-) The Selenium 2 Ruby
bindings are very good these days and have just as much overhead as Watir
(that is to say, not much).

------
UmYeah
Jmeter for load testing. Selenium for regression testing. Penetration testing
is also a pretty vital aspect if there web application has access to sensitive
data. Web applications should be tested like any other applications.

Graduate students stating that "no one tests web apps" are showing why they
are still called students.

------
imgabe
I've just started using Django testing framework to do unit tests for my app.
So far it helps catch a lot of errors that would have required more tedious
manual testing, which is what I did previously. It sounds like Selenium would
be a good way to test the javascript aspect.

------
BornInTheUSSR
IMO Biggest bang for your testing buck - cucumber with webrat

<http://railscasts.com/episodes/155-beginning-with-cucumber>

------
erlanger
I use QUnit to test my JavaScript and whatever's on hand to test backend code
(unittest in Python, Test::Unit in Ruby). I'd use Selenium for point-and-
click-style testing but I haven't gotten to it.

------
mdg
click, click, click, mouseover, click. So tedious.

~~~
jdee
sounds like you need the Selenium Firefox plugin <http://seleniumhq.org/>

~~~
mdg
... I had no idea! This is awesome (Its like discovering Firebug for the first
time)! Thanks!

~~~
johndunham
Um, and if you like that, try the free enhanced version from Sauce Labs called
Sauce IDE which records videos of your tests running in cloud space. Makes
troubleshooting way easier. (I cofounded Sauce with @hugs)

