
Selenium IDE alternatives for UI regression testing - Ost_Goth
http://screenster.io/selenium-ide-alternatives-for-ui-regression-testing/
======
wildpeaks
I'd recommend coding the tests instead of using the IDE, much less
frustrating.

 _Protractor_ ([http://www.protractortest.org](http://www.protractortest.org))
is a good wrapper for Selenium: despite the name, you can totally use it with
non-Angular pages (I'm a Preact person myself). It's convenient because it
handles installing and updating webdrivers (which was especially handy on
Windows), and seems well-maintained. And if you're using only _Chrome_ and
_Firefox_ , it even has a direct connection mode to skip having to run a
Selenium server (example: [https://github.com/wildpeaks/threejs-examples-
screenshots](https://github.com/wildpeaks/threejs-examples-screenshots)).

Otherwise if you just want to run tests in a real browser environment but
don't want to test in multiple browsers, you could run them in NW.js,
Electron, SlimerJS, or PhantomJS.

~~~
Sacho
Do you know of a list of wrappers like Protractor?

~~~
jumasheff
Nightwatch-Cucumber [https://github.com/mucsi96/nightwatch-
cucumber](https://github.com/mucsi96/nightwatch-cucumber) is a killer combo
for me.

Lately, I was playing with it. Here is a starter project:
[https://github.com/jumasheff/nightwatch-
cucumber-e2e-testing...](https://github.com/jumasheff/nightwatch-
cucumber-e2e-testing-example)

------
nnq
On the more technical side: _does anyone know of a real alternative to
Selenium?_ ...cause everything seems to be _based_ on it even if you're not
using it directly.

I like my web app acceptance tests to be code, readable and maintainable, not
UI/OCR thingies... but I've seen so many bugs like "this element is visible
and clickable, but Selenium says it's not, because the 3d-css animation + dom-
manipulation voodoo of the tested app got selenium completely confused"... and
I'm not even mentioning web code with events-on-SVG-nodes or webgl here...

Any non-standard and animation-heavy web-app/page will utterly confuse
Selenium to the point that you end up testing by injecting js into the browser
(so now you can have bugs in tests only, or tests that pass despite bugs)
mingled with test-runner code and heuristic timeouts everywhere...

(I know the "real solution" is "use a frontend framework and write front-end-
unit-tests instead", but usually you're writing the acceptance tests for
something that already exists).

~~~
mhd
I do wonder about the accessibility of pages like this. If it confuses
Selenium, can screenreaders cope?

~~~
nnq
Ugly and simple truth: no. In real life, in a customer centric approach, you
can choose to ignore even 50% of your users anyway, from a marketing pov, if
you decide those are not the users you care about (hint: you care about those
willing to pay way way more - like when you make an overpriced iOS app but
completely ignore Android to give another example). Why care about... less
than 1% of users, when you already chose to ignore an even larger percent in
your other approaches? Fair? No. Makes sense? Yes.

Expose all data trough a clean and documented API. If screenreader
accessibility matters, someone will eventually pay you to develop an alternate
interface using same API, or an open-source will be developed at some point...

Solution to accessibility problems is usually: (1) expose your _data_ though
an API, (2) wait to be paid to develop a truly accessible alternate "light" UI
(if it ever happens), but don't cripple the "main ui" (well, actually more
like "have it like the customer wants it" ;) heavy animations and fx are
"crippling" in my personal view too...).

~~~
mhd
I'm just saying that ignoring users and ignoring testers has considerable
overlap in this regard.

(And apart from general and professional ethics, local laws might disagree
with the assumption that you're free to ignore users for that reason.)

------
dhpe
Since this thread hits my expertise... We have developed Usetrace
([https://usetrace.com](https://usetrace.com)) for a few years targeting teams
that want non-techies to automate GUI testing. (Disclaimer, I am a founder).
So far the feedback has been positive and we're getting healthy revenue. It
seems that teams are desperate for faster solutions than coded UI tests. But
they will only use a tool that keeps up with frequent UI changes
(automatically or semiautomatically) and is really painless and quick to use.

Some lessons learned that we're solving with the product:

    
    
      - creating basic tests must be doable in 1 minute
      - testing is a team effort (testers&devs) - must have collaboration tools built-in
      - must be extendable with code (js)
      - sync/timing issues must be handled automatically (when possible)
      - UIs will change - generate locators and algorithms that adds robustness
      - debugging must be speedy: must be able to debug test steps live with the browser context (html5 remote connection to the browser)
      - randomness is needed - need an easy way to introduce random data (like random emails)
      - testing emails must be as easy as testing a web page - inbox must be an integrated feature
      - must handle frames transparently - user does not care if the element is inside a frame or not
      - must be pluggable to CI
      - must have good, clear reports
      - some tests will be flaky - must be a way to keep builds stable & green
      - no installations / plugins, just a browser
      - fast to update tests when UI changes - the system must do global replacements when it sees duplicate stuff
      - duplicate test step/code will cause a major maintenance burden - must have reusable/parameterizable components
      - people will want to reuse functional tests for stress testing and monitoring - allow that too
      - people need to check for dynamic content - must have a quick and codeless way to use variables
      - people want to have quick builds - tool must offer parallel testing
      - people will want to integrate to their own systems - must offer APIs
      - html pages are responsive - must have support for different screen sizes

~~~
HenryTheHorse
While your premise is indeed sound - and let me say I'm not taking away any
credit for what you and your team have built - but we did evaluate Usetrace
last year and found several basic issues. The system simply wouldn't record
several events and user actions and it fell so far behind in a test case that
we concluded it was probably not the right tool for us at the time.

But I'll be happy to re-evaluate the tool.

~~~
dhpe
Thanks for the feedback. We have come quite far since last year. For example,
now you can record html5 drag and drop events. Give it a try, we'd love to
hear if there's a webpage with issues, so we can fix Usetrace.

~~~
HenryTheHorse
Thanks - will certainly have another go at it.

------
all4rage4sure
The product might be good, blog post might be right. But for a vendor locked
alternative to say an opensource alternative as "doing more harm than good"
makes me rage.

~~~
pryelluw
Yup, this is obviously marketing. They didnt even go as far as posting it
somewhere other than their blog. Funny how they promote 2 competitors in the
post and make them look more favorable. This company needs marketing help ( if
you are reading this email me and Ill help you improve).

~~~
ig42
I'll think about it, thanks

~~~
pryelluw
You are welcome. Best of luck with the project.

------
mherrmann
Everybody interested in web automation should also check out Helium [1]. It's
a wrapper around Selenium, thus inherits its stability, speed and browser
support. But instead of requiring you to write
driver.find_element_by_id("btn356").click() you can just say
click("Download"). This alleviates the need for record and playback tools,
because scripts are so easy to write. It also avoids the problem that scripts
created via R&P are virtually unmaintainable. Helium is commercial and I'm one
of the authors.

[1]: [https://heliumhq.com](https://heliumhq.com)

~~~
nnq
I went with CodeceptJS [1] instead (Seleniu in nodejs, with most of the async
madness abstracted away) for the last project, but, as thangalin says it best
"It is a lot of effort to automate user interface testing with Selenium,
especially for web applications with complex user interfaces, concealed
options, and so forth.".

Advantage is that testrunner code mingles seamlessly with injected-into-
browser-js. Problem is... complexity, and twilight-zone-class bugs in the
testing tools themselves: there comes a point where the logic of the
acceptance tests is so convoluted, and they are so undebuggable and
unmaintainable, that fixing and extending them eats the time of _the best and
most expensive developers_ , instead of being a great tasks for juniors...

Eager to try helium next, hope it sucks less :)

[1]: [http://codecept.io/](http://codecept.io/)

Edit+: don't get me wrong, CodeceptJS is amazing compared to the madness of
seleniu-webdriver/webdriver.io used "raw", with promises and callbacks, so
kudos to its devs. But its far from optimal... though this maybe because
selenium itself is far from optimal.

------
defied
A lot of companies still use Selenium IDE to create and run tests, even though
it's basically been deprecated for a couple of years now in favour of
WebDriver.

The reason a lot of people still use it, is because it's very easy to create a
test; you basically record the things you want to test and Selenium IDE
generates the test for you, without requiring any programming knowledge.

( Source: I work for a company where people upload their Selenium IDE tests to
run them on a large grid of browsers in the cloud,
[https://testingbot.com](https://testingbot.com) )

------
scriptmonkey
A Selenium UI recording tool is a good starting point to help people learn.
Quickly though it becomes pretty limiting and specialized testers need to
switch over to using code.

The Selenium IDE I like using at this point is Oxygen. It's Atom with a
Selenium recorder built in. It records and gives you code with details about
the XPath and CSS Selectors provided in comments above each recorded action.
It's the best tool I've found for quickly getting off of record-playback
dependency and into working with code. The Oxygen IDE provides intellisense
and real debugging. It strikes the right balance of allowing you to record
when you need to, but then getting out of the way and providing you with a
real IDE so you can work with the code.

[http://oxygenhq.org](http://oxygenhq.org)

I did a blog post not too long ago about some of it's quirks on Windows...
[https://scriptmonkey.wordpress.com/2016/06/03/oxygen-for-
sel...](https://scriptmonkey.wordpress.com/2016/06/03/oxygen-for-selenium-on-
windows/)

------
gowan
> Selenium IDE makes you handle timeouts manually which is a major
> inconvenience.

you can set the implicit wait setting. this will work in most cases but will
make the tests slow when you need to test negative conditions. the behavior
around implicit wait varies by driver implementation which can be annoying. in
general it's better to use explicit waits.

> Selenium IDE only works with Firefox and may not work for sites or web apps
> that have slightly different DOM structure in Chrome, Safari, and IE.

selenium ide has webdriver playback[1]. if you don't like that you can
generate webdriver scripts in c#, java, python, or ruby and run them against
selenium server.

[1] [http://blog.reallysimplethoughts.com/2013/02/18/webdriver-
pl...](http://blog.reallysimplethoughts.com/2013/02/18/webdriver-playback-in-
selenium-ide-is-here/)

~~~
koollx
Screenster support for timeouts is much more extensive. It automatically
determines the time it takes for the server to process each user action, it
uses the visual baseline to wait for the right state instead of timing out or
waiting too long.

Obviously you can do everything with WebDriver, Screenster uses it too. The
point is that you can get a lot of benefits out of the box when you are using
a highler level platform like Screenster vs hand coding to a low level API
like WebDriver.

There's some description of Screenster features here
[http://screenster.io/selenium-alternatives-for-testing-
autom...](http://screenster.io/selenium-alternatives-for-testing-automation/)

------
thangalin
Where I find such IDEs lacking is their inability to control playback timing
and animate mouse movement. It is a lot of effort to automate user interface
testing with Selenium, especially for web applications with complex user
interfaces, concealed options, and so forth. It should be trivial to generate
an instructional video showing how to use the application by re-using the
efforts involved with writing the web tests.

A while back, I combined Java's Robot class with Selenium and a simple
scripting language to create Scripted Selenium.

[https://bitbucket.org/djarvis/scripted-
selenium/](https://bitbucket.org/djarvis/scripted-selenium/)

[https://www.youtube.com/watch?v=tGvcm0Rlm-0&](https://www.youtube.com/watch?v=tGvcm0Rlm-0&)

------
mhd
I really don't like pricing levels that aren't just per user, but per test.
That encourages thriftiness at a level where you shouldn't be thrifty.

Never mind that I'd be seriously worried that a lot of these small time
players are out of business the next year and I'll end up with useless tests
(of course this is better if the software itself is just helping recording
tests and it all ends up as Selenium scripts in some human-readable language).

------
wswoodruff
Has anyone used
[Nightmare]([https://github.com/segmentio/nightmare](https://github.com/segmentio/nightmare))?
It's built on top of Electron -- the Atom code editor is built on Electron.

~~~
kevining
It's great, but I can't stand the chaining syntax. I'm a bit biased, but I
think that some suite with async/await will be the future. Here's something
that I've been working on:
[https://github.com/KevinGrandon/ghostjs/](https://github.com/KevinGrandon/ghostjs/)

------
mavelikara
If the authors are listening, I think the tool you have in mind is called Sahi
(not Sachi). Also, you lump it with proprietary solutions unfairly when you
say:

> Finally, Sachi, RFT, and UFT are expensive enterprise solutions that require
> local installation, certain servers, runtime licenses and lots of config.
> Being difficult to learn, hard to maintain, and costly, none of these
> solutions beats the open-source and free Selenium. End of story.

Sahi is open source and is hosted on SourceForge:
[https://sourceforge.net/projects/sahi/files/](https://sourceforge.net/projects/sahi/files/)

~~~
ig42
Thanks for pointing this out, I should've done my homework better with Sahi
pricing. I've fixed the issue

------
rstormsf
if we go with simplicity, then I didn't see anything simple than
[http://sikulix.com/](http://sikulix.com/) \- you can automate anything you
want there.

~~~
koollx
Sikuli is simple but from what I know about it you can't write maintainable
tests with it. Relying on the pictures to be always the same and driving your
test is guaranteed to cause false positives.

I think the future lies with platforms like Screenster that have recording,
collaboration, built-in browser execution, visual verification and lots of
other magic, and not with low-level frameworks like Sikuli.

But both have their place :-)

------
zabil
Although a recorder makes it easy to record tests with a growing test suite
you'll find it frustrating to re-use or re-factor the tests.

Plus all the data that you record is tangled in the generated code making it
hard to locate or change.

We created gauge [http://getgauge.io/](http://getgauge.io/) (It's open source)
to solve this problem by making it easier to author and write tests in your
favorite tool, without a recorder.

The tests can be authored in mark down and wired up to re-usable selenium
code.

------
kgodoy
I've been out of the QA role for a few years now but there is a nifty, albeit
not maintained, tool that I used to use for generating UI objects for
Selenium. It's not really an IDE, but it did allow me to quickly generate
files needed for using the PageObjects model.

[https://unmesh.me/2013/08/29/pageobject-generator-utility-
fo...](https://unmesh.me/2013/08/29/pageobject-generator-utility-for-selenium-
webdriver/)

------
regecks
Maybe slightly unrelated but what do people use for browser driven stress/load
tests? I have been using PhantomJS + phantomherd[1], but boy, PhantomJS sure
takes a lot of CPU.

\--

1\.
[https://github.com/nmeans/phantomherd](https://github.com/nmeans/phantomherd)

~~~
manyxcxi
Definitely not browser based but I've had good luck with Bees with Machine
Guns[0] (Python EC2 load test swarm) for load testing. It's not as scriptable
as a PhantomJS test, where you can have every actor follow their own script or
anything like that, but it can sure generate a lot of requests a lot cheaper
than some of the alternatives.

One thing we've done in the past is to generate a large background load with
BWMG and then run a few PhantomJS/Selenium tests in a smaller test rig. It's a
lot easier to get the load that way vs. trying to swarm Phantom or Selenium.

0:
[https://github.com/newsapps/beeswithmachineguns](https://github.com/newsapps/beeswithmachineguns)

------
martinald
I'd be curious how these test recorders handle mouseover events (eg: you have
a multilevel menu which you hover through). The only way I got this to work
was record every mouseover event, which quickly made an enormous brittle test,
or just don't test stuff that has mouseover.

~~~
dhpe
Having developed a test recorder, I found out that the best solution was to
make mousehover an explicit action (the same as in webdriver). Better to have
a robust test than try to do something too fancy.

------
lwhi
Hmm .. This is kind of pointed towards the host product. At least include
links to the competing products if you want to give an impression of
objectivity.

------
WhiteSource1
Have you looked at Applitools for visual testing?
[http://applitools.com/](http://applitools.com/)

