

Ask HN: What do you use to test Frontend code? - outside1234

At our startup, we are trying to sort out the best way to handle testing against our supported Browser matrix. We want something that is repeatable and automated. We've looked into QUnit and Selenium but have this nagging feeling that we are doing it wrong.<p>Our application is primarily one page and uses a lot of jQuery to both animate things on screen and to work with the backend via AJAX.<p>Any hard won tips out there?
======
sgrove
I always feel frustration at this question, I can't believe that sauce labs
hasn't taken off. Use capybara as the driver layer, and a proper test
framework that separates the different layers, and sauce labs to hit _any_
browser matrix you want. Tie it into a CI system and never worry about this
again - it's a solved problem, and it was solved by Sauce Labs.

disclaimer: I used to work there, and put my blood, sweat, and tears into
building it out just for situations like this.

------
dbalatero
If you're in the Ruby world, Capybara is a great addon to Cucumber and RSpec.
It solves the animation problem by adding a timeout to wait for elements in
the DOM. A step like this:

When I click the link that does a 4000ms animation I should see "#mydiv"

Capybara will wait for #mydiv to appear for some timeout period (30 seconds).
If it appears before then, it keeps going.

This way you can have complex animation and still assert final state of the
page. Capybara allows Selenium, headless WebKit, and other JS drivers. It's
one line to swap.

If you aren't in the Ruby world, you might at least steal some ideas from that
lib to make your Selenium tests better.

Also, you might drop the folks at Pivotal Tracker an email. They have a fat
one-page JS-heavy product, and I bet you anything they automatically test it.
They might have cool stuff to share.

~~~
schrieaj
Capybara works pretty well for any web based front end. I'm using it to do
some basic automation of websites (scheduling tasks and the like). You write
the tests in ruby but honestly it is really simple:
<http://pastie.org/2476405> is my script for logging in to google voice and
sending a text message. If I recall it was simply gem install capybara, a gem
install selenium-webdriver and that was it.

~~~
dbalatero
Good point, I forgot that you could just use it standalone!

------
JesseAldridge
I'm not using full on unit tests for my current project. I just break the code
into a few independently functioning layers, and make a single html file to
test the code in each layer. I run the corresponding html file manually as I
develop code in a given layer. It may be heresy to say this, but I've found
that automated unit tests are more trouble than they're worth in a lot of
cases.

The second thing I do is use my tool every day. I use the unstable branch on
the unstable server whenever possible. Problems usually become obvious very
quickly. I don't push to production until I've used the unstable server for 24
hours without incident. I find this combined with the testing practices
described above to be the best compromise between reliability and getting-
stuff-done-ability.

------
jph
We use SauceLabs with Selenium, Jellyfish (very fast), Capybara (very
flexible), and Jasmine. Yes, Selenium may feel hard to work with at times yet
the payoff is enormous when you aim to support many browser for many years.

Edit: I have a great outsourcing team that builds Selenium for me, and runs in
SauceLabs. PM me if you want details.

------
timruffles
We use jsTestDriver at Picklive to test our game, alongside Sinon.js for a
mock implementation of our game server. It's very easy to test simultaneously
on multiple browsers, run tests on the command line, and we run it headless
for continuous integration.

In comparison with Selenium, I've found unit and integration tests using this
setup fast and resilient.

------
barnaby
I use qUnit all the time because I wrote lots of jQuery plugins for everything
that we do at my small startup. Then we have a QA team in India that are
always clicking through the site and finding new bugs (they also have
extensive selenium scripts that go through the whole site every night, but
oddly enough none that run the unit tests... grr).

------
BerislavLopac
Admittedly, Selenium's standard interface leaves a lot to be desired; however,
I've had great time using it in combination with a Python wrapper, where you
write Selenium test cases as Python unit tests.

~~~
zzleeper
This. Just write the Selenium base code in the Firefox addon, and then click,
export as python :D

------
simplegeek
I guess Selenium is fine, out of curiosity, why do you think using Selenium
isn't the right choice for you?

~~~
outside1234
I'm not sure that it isn't. I'm worried about the maintenance costs of keeping
it up to date - but maybe that is the price you pay.

~~~
otaku_coder
If you're worried about keeping it up to date, why not outsource the test
runner to something like saucelabs?

~~~
notJim
I believe outside1234 is referring to the maintenance cost of keeping _the
tests_ up to date.

~~~
jtheory
I'm using Selenium on a major project, and from the start refactored out
common tasks. I record new tests using the Selenium IDE formerly, now the
Sauce Builder plugin, then export to code and replace quite a lot with calls
to our own much-simplified API (often cutting out huge sections of test code &
replacing with one call).

Updating the tests even when we make significant changes to the website is
must easier because of this approach.

One issue we ran into quickly... tests need to verify _just one thing_ as much
as possible, not long strings of actions that wander through the whole site.
The problem is that there's considerable setup before testing that one thing
-- creating users and getting them to the appropriate state, especially (worst
of all when test cases require 3+ users to interact).

To address this I've set up some helper requests Selenium can make -- actions
that aren't included in our production build -- to create/destroy test users
of different types & in different states directly. I'm also thinking of adding
a page that'll simply return a full dump of info on a given test user, so we
can test that the end result of actions in the UI have resulted in the
expected data.

It's all very much work in progress, but I'm convinced that with a little
extra elbow grease added as pain points show up, we can keep a full suite of
Selenium tests alive and useful.

------
tdavis
For Javascript testing, I really like JSTestDriver for automation and Sinon
for mocking/stubbing.

I found bulky solutions like Selenium to be somewhat hit-or-miss and overkill
for most tasks. It's very rare that with proper unit testing I really need a
whole additional layer just to click around the site and make sure stuff is
working as expected. Maybe some basic sanity checking, but it's really not my
preferred way to think about testing--which of course is just a personal
preference.

~~~
pmorel
Same, I feel that if there is a good coverage from unit testing, then you
should only need sanity checking. Furthermore, if you are using jquery (which
itself is well testing on all the browsers), then you should have more
confidence that your javascript will work (providing you're not using non-
cross-browser javascript on top of jquery)

~~~
pmorel
For reference, the list of tools and framework mentioned on this page can be
found at <http://www.romku.com/list/343858/web-testing-framework>. Feel free
to rate them and add any missing information.

------
mpd
Having dealt with the Selenium and its frustrations in the past, I tend to
outsource this sort of testing if at all possible. Some things can only be
truly tested with eyeballs.

One tip I do have is to disable jQ animations during your test runs so you
aren't having to jump through hoops in your test suite just to wait through
animations -

jQuery.fx.off = true will do that.

~~~
arbrandes
This! Also, try to mock any setTimeouts and setIntervals so that they execute
callbacks synchronously (i.e, timeout = 0).

------
pacmon
Could try watir (<http://watir.com/>)

~~~
tombot
Watir is what Facebook use I think, it has plugin support for most of the
major browsers and tests are fairly easy to write.

------
qixxiq
I use a custom selenium test runner that runs them in parallel (php). I plan
on neatening it up and releasing it at some point if theres interest.

It also has a 'live mode' which is useful for writing up new tests. It
basically follows a file and whenever there is a change it begins running the
commands from that line onwards again. If it hits an exception it'll pause
there and wait for you to fix the broken test/site before resuming.

~~~
jtheory
I'd be interested as well.

I wrote a custom test runner as well, though not addressing the same issues;
mine is specifically for _non_ automated testing. It's a Java GUI that lets
you choose from all our existing tests, tinker with parameters, and run the
test live in your own browser against our test server with a few extras (like
stepping through the test, go/pause, run all tests in sequence, etc.).

It's partly to let our CEO really interact with our testing -- i.e, see what's
being tested, and rapidly explore the site as a whole -- but also is pretty
useful as a robot; i.e., when I want to register three new users to try
something out, I don't have to do it manually.

------
ByteMuse
Some things require manual testing. Animations and a lot of visual stuff needs
to be analyzed holistically; there are sometimes visual quirks that are near
impossible to write tests for but can easily be detected from someone quickly
viewing the site.

Have different environments set up. After every big push, run through a simple
user flow to make sure what the user will actually see is what you intended.

~~~
MartinCron
_After every big push..._

One way that I minimize my front-end testing liability is to do as few "big"
pushes as I can. The safest change to make is the smallest change possible.

------
gudmundur
Check out Ghostbuster (<https://github.com/joshbuddy/ghostbuster>).

------
douglasmiranda
Hmmm, I'm using Splinter (<http://splinter.cobrateam.info/>) to do tests. It
lets you automate browser actions and much more, like execute javascript, find
elements returned via AJAX, interact with forms, links and more... it has
drivers for chrome, firefox... You shold to test it :)

------
mfolnovic
You could try with Jasmine, although it's not for this specific purpose...

And I agree with simplegeek, selenium should be fine in your case. Although
can't imagine how you could test animations... :)

~~~
rchiniquy
Jasmine isn't what you're asking for, but you should use it in addition to
what you're asking for.

------
riffraff
on a side note, do you people test graphics? I mean, it's kind of easy to say
"I get a div #foo". But how do you check if it has the right shade of $color
and a rounded corner and the text content is not overflowing the containing
box?

I live in constant worry of changing css and totally screwing up a completely
unrelated bit of the web app, and the usual thing I get told is "well, it's
very fast to check if it still works", which reminds me of me not
understanding why unit testing is useful.

~~~
hopeless
Asking a computer to check aesthetics is start of your descent into madness.
Use Selenium to check that links do what you expect, elements are loaded etc.
Don't use it to check if a button graphic has the right icon loaded.

Computers are great at functional testing but you still a Mk1 eyeball to check
aesthetics. Luckily, functionality is what your customers will really care
about. No one will shout at you if an element is misaligned for a while before
you catch it with manual testing

~~~
jtheory
Side note -- I definitely agree about not using Selenium to check for
alignment, images loading, etc..

But... it's pretty cool to have a single Selenium test that does nothing but
zip through your site and show all of the unique views at rapid fire speed, so
as soon as the page is loaded it has already filled in any relevant form and
is loading the next page.

Make sure someone with good concentration sits and watches that test run once
in a while; anything seriously wrong will pop out, and they can watch it zip
along while taking a coffee break.

------
wlievens
For algorithms or application logic, even rather simple cases, unit tests are
an absolute must.

For user interfaces, it's a lot more effort for a lot less return. YMMV

------
hiwaylon
Take a look at JsTestDriver, for hooking in to your CI system.

------
long
you might want to check out SIKULI (<http://sikuli.org/>) - it's like python
on top of a computer vision stack, so you can write code like click([part of a
screenshot]).

------
ltamake
Selenium works fine for me.

------
juanbyrge
zombiejs?

