

Browser testing sucks, it's time to start doing it right. - mikecotton
http://saucelabs.com/blog/index.php/2011/05/testright-browser-testing-done-right/

======
akeefer
Internally here at Guidewire, we have a declarative widget-based web framework
for our apps, and we test by automatically generating a Page Object model
based on the declarative widget definitions (using metaprogramming techniques
in Gosu). The rigidness of defining your application in terms of declarative
widgets is not for most applications, but the resulting improvement in
testability is probably the biggest benefit. Separating out a logical model of
"widgets" on the page and writing against a Page Object model, whether or not
your web framework has widgets, is clearly the right way forward to me.
There's nothing worse than debugging hundreds of test failures because no DOM
element with id "User_Search" was found on the page. Did the page not render?
Did the test run too fast? Did someone change the ID? Did someone move the
search button to a different page? Did they hide it behind some div I need to
expand first? Did the page just break? There goes two days of development time
. . .

Prior to that approach, we tried everything: we actually wrote our own test
framework that used javascript to drive the browser from inside the browser,
then we moved to using selenium, but they all suffered from that same problem
of incredibly fragile tests. Over time, test maintenance is easily 10x the
cost of initial test development, so it's a huge, huge problem.

So I see this as a promising direction, but building and maintaining that
(essentially) Page Object model by hand is going to result in its own set of
maintenance problems. How do you know when the model is up to date? If someone
changes the model, how do you know which test that affects if you're in a
dynamic language like Ruby? It solves one set of horrific problems, and
replaces it with another set of problems that are hopefully, with enough
developer discipline applied (and guidance from the framework), ultimately
slightly less horrific. Being able to automatically generate the POM via
metaprogramming by statically analyzing your application is awesome, but it's
not a technique that really works with any standard web framework that I know
of.

Web app testing really still has a long, long, long way to go.

~~~
sgrove
There are some _awesome_ strides about to made for generating/maintaining POM
(or WOM, in our case) - sadly, marketing has shot down all of my suggested
names :(

That said, it's going to be a lot of fun, and I'd love to have someone with
your experience and insight join in the conversation. I'll have to think of a
way to do it - a Test::Right convore, or mailing list, perhaps?

~~~
akeefer
My personal e-mail is in my profile, so I'm happy to participate in whatever
manner you deem is best. Just send me an e-mail and let me know (or if you're
in the Bay Area, you can catch me in person at SuperHappyDevHouse on
Saturday).

------
cwp
I think the goal of "impossible to write bad tests" is unreachable. It's like
"idiot-proof": impossible because the idiots are too resourceful. I'll be
really happy, though, if Test::Right accomplishes "easy to write good tests."
Right now, nothing out there does.

I especially like the decoupling of intentions from mechanics. It lets us
improve the mechanics without breaking or rewriting all the tests. Maybe the
mechanics are implemented with JS DOM-traversal code, maybe it's CSS
selectors, maybe it's SIKULI-like image recognition (<http://sikuli.org/>).

Looking forward to trying it.

~~~
sgrove
> I think the goal of "impossible to write bad tests" is unreachable.

We all sat down a bit ago to reflect on 16 month of non-stop in-the-trenches
browser testing for all kinds of people, to integrate it all. We talked about
how we wanted to make things better for the people out there who know _what_
they want to do, but are continually screwed over by the _how_.

The goal "impossible to write bad tests" floated out, and it was almost a
joke. But we started to wonder it would take, and it grew more serious. It's a
mantra now.

And we have so much cool stuff flowing out form that mantra in the next few
weeks/months from that attitude, it's insane.

And excellent point about intentions/mechanics - Jason Huggins is _all over_
Sikuli. We're watching very patiently :)

~~~
cwp
Well, I love big audacious goals. It's just that "impossible to write bad
tests" could lead to a system where it's also impossible to write _good_
tests. If you start shooting down good ideas because they could be abused to
write bad tests, you'll end up with mediocrity. At the extreme, just prevent
people from writing tests at all. Mission accomplished! Now, I know you guys
are smarter than that, so I'll just leave it be.

Thanks for the work you've already done on this, and will no doubt do in the
future.

------
andrewdavey
I see a big problems in web testing caused trying to test drive from outside
of the browser. I'm currently working on a solution that has tests written in
javascript that run _inside_ the browser. The test runner puts your web app
into an iframe, so the test code can use jQuery, for example, to easily
automate and assert over the UI.

JavaScript also makes creating a very clean testing-focused DSL easy.

I'm hoping to open source the project soon. If anyone would like to know more
give me a shout (@andrewdavey).

~~~
sgrove
This is the approach that Selenium 1 took, and there be many dragons down that
path. It's workable (we've done a ton of cool stuff at Sauce Labs making it
work), but it takes an usual mix of skills and insight to get it to _really_
gel together. Selenium 2 was built with all that mind.

That said, it's not the important part. Selenium (and watir, windmill, etc.)
is an _execution_ tool, it doesn't help you avoid some very, _very_ large
pitfalls waiting for you down the road. Too often your tests are tightly
coupled with the dom, data and selectors are strewn all about, you have manual
pauses and timeouts set by someone who left years ago, and you have so many
that the whole thing takes _two weeks_ to run (yes, I've seen this a number of
times).

Basically, I don't care about the tool to execute the tests anymore. They'll
all become interchangeable with Selenium 2 anyway. The _much_ more valuable
piece here is in creating tests that are maintainable and a joy to work with.

Certainly best of luck, please be sure to share you experiences and insights!
There's always more to be learned.

~~~
andrewdavey
Reducing tight coupling is definitely a concern I'm looking into. I can't see
there being any silver bullets, but a good framework should certainly
encourage a sustainable way of working.

I've developed a rather nice asynchronous, sequential, execution model for
test code. So they avoid the callback-hell you'd see with multi-step async
tests. The spin assert idea mentioned in the article is very useful, and I'll
be borrowing it for sure! :)

I'm wary of running tests in parallel, due to concurrency issues. However, I
guess spinning up multiple web servers and in-memory databases would work
fine.

------
geuis
I'm having a hard time pinning down the feeling, but there's a lot more
declaration here rather than evidence.

In other words, shut up and show me. (This isn't an attack on saucelabs)

The github page is sparse on documentation, and the wiki page is basically a
copy of the README.

So don't make vacuous claims. Write an article demonstrating how this system
works, how it compares to X, and why it's better than X.

~~~
sgrove
Code's there, example application with tests is there, go and try it.

We err'd on the side of getting this out (several developers were chomping at
the bit even with just a bit of editing on the post) - rest assured that we'll
have more coming. In particular, we'd like to have some more documentation,
examples, and best would be quick screencasts talking about and demonstrating
the techniques/benefits going on in here.

No worries about 'shut up and show me' - but the counter-challenge is 'dive in
and try it' :)

------
heyrhett
tldr; "I hate browser testing. Did I fix the problems? No. I started writing
opinionated software though, because my opinion should matter. I wrote it in
Ruby because I know Ruby."

~~~
sgrove
Hmm, that's not really a considered review.

Straight off, no on at Sauce Labs hates browser testing, that's just a stupid
thing to posit.

Test::Right is a really cool mental model of testing, and it's going to _make
people's lives better_. Robust tests, parallelism, immediate cross-browser
compatibility (this is an issue people without a lot of experience don't
_really_ understand), a lot of cool shit.

I love the promise that if I write my tests in this pattern/framework, my
tests _will always be good_. It's not there yet, but it's still damned cool.

We also have several volunteers who're excited to port this to other
languages, but we started with ruby because it's a fertile playground, and the
important piece here is the pattern, not the language/framework itself. It'll
end up being _the_ flagship browser testing framework de facto, but that's not
the goal here.

I played a big part in putting some pieces of Test::Right together, full
disclosure.

~~~
JonnieCache
_"ruby ... it's a fertile playground"_

I like this a lot.

