

Ask HN: Are you doing automated UI testing – why or why not? - atiffany

I have worked in several companies that have had goals of automated UI regression test suites, but I&#x27;ve never worked at a company that pulled it off successfully.  I can name off a case-by-case list of reasons why these efforts have failed, but I am curious if the community is seeing similar trends.
======
atiffany
I'll add one example from a past company I worked for:

\- We had a manager and one internal developer assigned part-time to an effort
to seed an automation effort.

\- The front-end of the application was Flex.

\- Most QA analysts at the company had never written code before.

\- We attempted to use Selenium.

As you can imagine, this was a crash and burn for several reasons. The company
couldn't get enough buy-in internally to dedicate more developer time to seed
the effort, and the long-term approach was not well thought out. A few tests
were deployed, but they were brittle because they relied on DOM elements. This
was partially because QA analysts were expected to be able to use a Selenium
"capturing" tool to create tests without writing any code.

Writing this out now, I think the root problem is that at the management
level, the sentiment was just that we should "do automation" and no one really
understood how much that would involve. There doesn't seem to be any turn-key
solution out there that is designed to fit in with the rest of a company's
development processes.

When there isn't a clear path for solving a problem, it becomes a much more
expensive problem to solve. In this case, the cost far out-weighed any benefit
that the company would see.

With that said, I am still optimistic about the potential for effective UI
automation testing and I'd love to hear more stories.

------
jfella5
\- We use selenium, test cases are written in python.

\- We use the webdriver, recording never worked

\- Followed the page model

\- Had to build in retries on getting elements to make tests reliable

\- You need upper management buy-in. Took us 8 months to start getting it
right

\- I would say our team members are intermediate with python

\- Devs currently do not write tests. But they asked to start writing which is
a good sign

\- Yes some days you just spend fixing broken tests. But you get experience on
how to write them better over time

Tips:

\- Work with devs to get build the page with proper selectors (use ID if not,
then us CSS selectors, don't use xpath)

\- Use APIs to seed data for your tests

\- Get to the page you need to test as fast as possible by hitting the urls
directly, using APIs to seed what data u need. Manually injecting cookies to
get to the state you need.

Costs:

\- 2 full time engineers only working on automation for 8 months.

\- Once you have your framework in place writing tests become a lot easier

------
jags-v
I think the question is incomplete(at least I think so).

Its not about automating the UI tests for regression testing,but to ensure
that regression testing is effective.Most of the times,this is driven through
the UI, which could be flaky.

As the test pyramid paradigm states , having a lot of UI tests, depending on
the functionality & complexity could be an anti pattern. Essentially , a lot
of the functionality can be tested using by writing unit tests & integration
tests and the end to end business use cases should be automated through the UI
just to ensure it works.

If it is possible to drive the state of the application using an API or URL,
to the target state and then we assert elements to check the UI.

This enables faster feedback and is less flaky.This is how we approached
testing in our team.Depending on the objective , we would use Selenium ,JUNit
, Cucumber or corresponding ones in a given programming language.

------
jfella5
Yes, I along with my team members have helped my company come from zero
automation to decent coverage with UI tests. We have tests that run reliably
for Web, iOS and Android.

It has certainly improved our deployment time.

If you describe some of the your problems. I can tell you how we overcome
them.

~~~
atiffany
Thanks for the response. I have several questions.

What is the division of labor between developers who work on the tests and
(perhaps non-coding) QA analysts?

What software are you using?

How large is the application, and would you mind describing the nature of the
application?

Do you have any clear cost metrics on the effects of automation testing? E.g.,
are you able to operate with a smaller QA team, measurably lower incidence of
bugs discovered in production, etc.?

Thanks again.

