
Our team's troubles writing automated UI tests - twynsicle
https://medium.com/@steven.lemon182/our-teams-troubles-with-hand-written-automated-ui-tests-cb189cbbff90
======
GvS
I used to have troubles with UI tests too, but then I discovered Cypress
([https://www.cypress.io/](https://www.cypress.io/)). You can still run into
an issue once in a while, but it's way better than anything else and it's
getting even better with each release.

~~~
mavelikara
After reading through the website and watching their video, I still don't
understand how it works out what it's value proposition is. Is there some demo
where I can see and understand how Cypress provides value over the state of
the art?

~~~
Macha
It solves the same problem as Selenium without actually being Selenium.

The one time Selenium was reliable in my company was when our team had a
dedicated, massively overprovisioned grid.

The biggest pro is it just has less random "Oh, guess that timed out?" issues
than when we used Selenium.

The biggest negative is it's chrome only.

~~~
karldcampbell
Have you looked into TestCafe[1]? I've only used it for a PoC, but it has
better browser support than Cypress.

[1][https://devexpress.github.io/testcafe/](https://devexpress.github.io/testcafe/)

------
zmmmmm
I think this is one of the "dirty secrets" of current software development ...
one of those things that pretty much nobody does very well (at least, anywhere
I have seen).

My view is that we won't make real progress on proper UI testing until there
is a paradigm shift that moves responsibility for creating applications that
are fundamentally testable to developers. This is not dissimilar to how the
idea of "devops" finally recognised that managing deployments and
infrastructure is actually first class development activity.

The problem is that testing a system that has not been designed for it is
almost fundamentally intractable. I have seen toolkits that auto-generate
random ids for every element on the page so that it is almost impossible to
hook onto anything within the page as a reliable anchor to identify testable
points within the page. Even when those things do exist it is usually only
incidental and not an agreed contract b/w developer and test automation
engineer. So they will always break unexpectedly etc etc.

~~~
jankerz
It’s not just developers that need to be more involved in UI testing but scrum
masters, stake holders, management, etc. Automated tests should be viewed as
shippable code, and their creation and maintenance should be factored into
every story.

Without those “above” developers on board, you end up with a hodgepodge suite
of tests because developers/QA are forced to focus on delivering features
instead of a testable product and reliable tests.

------
doublerebel
Really excited for the adaptable UI testing being done by test.ai , which
solves many of the listed troubles. They use ML in 2 main ways:

1) to understand common UI flows and iconography, then automate test
generation for all/many possible paths through the UI

2) to adapt the test path to UI changes or panel rearrangement

Ran into a presentation by them at the Big Data AI conference in Seattle this
spring.

------
malkia
I've found this while looking last two days -
[https://visualregressiontesting.com/](https://visualregressiontesting.com/)
\- but in my case I'm more into validating 3d rendering from a tool

~~~
malkia
Also this - [https://github.com/mojoaxel/awesome-regression-
testing](https://github.com/mojoaxel/awesome-regression-testing)

------
benologist
I solved my UI tests by scripting a ton of screenshots for my documentation
using Puppeteer, going through each thing the user can do and proving they can
do it at multiple device resolutions. It was a lot of work but now it is quite
simple -

    
    
        await owner.click('Create plan')
        await owner.screenshot('Fill out form')
    

One thing I did wrong that I have to revisit is I don't confirm the
screenshots hold any data I'm expecting. It would have been much easier to add
the asserts when I was writing them but I was manually inspecting everything
very closely to address responsive issues so it didn't seem important until
the responsive issues were sorted.

~~~
lostintangent
I’d love to hear more about this! Do you have automation that runs as part of
your CI builds, and dynamically captures the screenshots for your
documentation using Puppeteer? If so, that is so cool :)

Have you had much trouble with the automation scripts breaking as a result of
app churn?

~~~
benologist
So far I still manually trigger a regeneration but it's only a matter of time
till it just runs automatically on any update.

The main problem has been with Puppeteer, there is a chance it will just
randomly crash at any point and it feels like there are a million little race
conditions where you need to wait or keep retrying to access an element after
it says a page has loaded so I have a lot of ugly code in my puppeteer-helper
like:

    
    
        while (true) {
            try { 
                element = await active.$('#element')
            } catch (error) {
            }
            if (!element) {
               sleep(100)
               continue
            }
            break
        }
    

Another issue only pertained to getting the simulated device screenshots,
switching the viewport configuration reloads the page which for me makes any
state like "yes, you just deleted x" be an "x does not exist" error so at the
moment I get around that by doing each device independently which takes a fair
bit longer where slow APIs are involved.

This is one of the 'recipes' -

[https://github.com/userdashboard/userdashboard.github.io/blo...](https://github.com/userdashboard/userdashboard.github.io/blob/gh-
pages/administrators/stripe-subscriptions/applying-coupons-to-
subscriptions.js) with some glue for Puppeteer v1 so a bit WIP still -
[https://github.com/userdashboard/userdashboard.github.io/blo...](https://github.com/userdashboard/userdashboard.github.io/blob/gh-
pages/screenshots.js)

The end result I think is really cool:

[https://userdashboard.github.io/administrators/stripe-
subscr...](https://userdashboard.github.io/administrators/stripe-
subscriptions/revoking-subscription-coupon)

In the final incarnation people using my software will be able to point the
screenshot generator at their own website and regenerate all the screenshots
so they can reuse the documentation.

------
nitwit005
This is largely a general discussion of the problems of UI automation instead
of their specific issues, as the title would suggest.

------
ethbro
Repurpose a graphical process automation tool. You're going to have a lot
better time.

[https://www.uipath.com/](https://www.uipath.com/)

~~~
arjun27
That is very interesting! What benefits do you see from using UiPath over
testing frameworks?

~~~
ethbro
In graphical interface automation, compatibility is the limiting factor.

Compatibility has an extremely long tail.

The only companies incentivized to implement support for that long tail are
those who derive revenue from doing so.

Testing frameworks, by and large, don't derive revenue from that. Process
automation frameworks do.

Ergo, if you're sick of "Oh, well that isn't supported", use the latter, even
if for the purposes of the former.

------
swsieber
One approach that's been appealing to me WRT to page models is to generate
them from the page/component templates. It doesn't work with JSX, but it does
work with other stuff like Vue or Angular templates:
[https://samsieber.tech/posts/2019/06/type-
safe-e2e-testing-d...](https://samsieber.tech/posts/2019/06/type-
safe-e2e-testing-dsls-in-kotlin/)

My other comment is: you absolutely need data management methods if you're
going to be writing UI tests. And maybe spinning up/down a testing environment
on demand.

The one thing i havent fogured out (personally) is testing google integrations

------
boozang
Hi guys,

Not intending to drop yet another test tool in the mix, but if you are worried
about flakiness due to timeouts or dynamic classes/id you can take a look at
boozang.com.

It has many similarities to Cypress, but element selectors are based on
natural language so doing automation on top of changing ids/classes is
trivial. Also it supports Chrome, Firefox, Safari, Opera and Chromium Edge.

------
fadzlan
Copying the comment I made on the site :-

The thing about writing test, either unit test or functional UI test, is
repeatability. Back then I had this problem too, to which I had worked on
getting a script to drop the database, and recreate it back again,
repopulating it with consistent test data.

Then, docker-compose came along. It is so easy right now to just have an
environment up and running consistently. And once the test is done, the whole
container set of the docker-compose can be easily removed, and created again
on a whim.

For most database containers, you can just attach your sql files into the
volumes that the database engine reads, and you will have consistent test sets
overtime.

This will solve :-

1\. Running test again and again, will make certain elements go of page
because of pagination. 2\. Creating new data with the same data to ensure
uniqueness and testing it, can be done, because it will be a clean slate
everytime. 3\. Previous data causing inconsistent state with the current test

This does not solve every issue with functional UI testing, but in my
experience, having consistent data every time is huge! No more second guessing
when you write the logic in your test in fear of having existing data or any
manual testing data breaking your test.

------
legulere
At the project I’m currently working on we evaluated WinAppDriver (for WPF)
and found it to be pretty buggy, slow and low featured. In the end we decided
for a commercial product on around the same level as it, but if you’re on
windows you can always use the ui automation Framework.

~~~
arjun27
Any recommendations of commercial products that worked for you?

~~~
legulere
We’re using testleft from smartbear, but I can’t really say how it compares to
other commercial products.

------
jchb
Article should really be called “our teams troubles writing automated UI tests
_for native windows apps_ ”.

The article mentions “If you follow the Page Object Model pattern, then for
each page and control in your application, you create models so your tests can
find and interact with the elements on that page or control.“.

Not being a windows programmer I am not familiar with this model in detail,
but a major issue seems to be that the UI toolkit doesn’t have any built in
support for UI automation.

For example on iOS and macOS, the accessibility system (which is what e.g the
screen reader for blind users use to drive the application) is repurposed for
UI tests. This means that if your app is accesible, which it pretty much is by
default where you use standard UI controls, with minor code changes such as
assigning unique identifiers to UI elements where needed, it is also UI-
testable.

~~~
pjmlp
WPF and UWP support UI Automation, there are even VS project templates for.
Forms does not, beyond the typical Win32 SendMessage().

However there are good UI tooling that does a good job in spite of it,
assuming that one is willing to pay for them.

[https://www.ibm.com/us-en/marketplace/rational-functional-
te...](https://www.ibm.com/us-en/marketplace/rational-functional-tester)

[https://www.ranorex.com/](https://www.ranorex.com/)

[https://eggplant.io/](https://eggplant.io/)

[https://www.tricentis.com/](https://www.tricentis.com/)

[https://www.microfocus.com/en-us/products/unified-
functional...](https://www.microfocus.com/en-us/products/unified-functional-
automated-testing/overview)

