

Ask HN: How do you manage automated test workflow and view test analytics? - jonstjohn

During the past several years, my company has gone from running a handful of automated tests on a single server to running thousands of automated tests.  These include unit tests, integration tests and Selenium tests.  The Selenium tests are particularly slow, and although we can parallelize them, the builds still take upwards of an hour to run.  We typically don’t run the full regression build on every push, but instead use them to validate code prior to merging a branch to master and tagging.<p>We’ve come up with our own workflow and hodge-podge of tools to generate test analytics, but we’re looking a tool or product that we could integrate into our build system to streamline the workflow and give us insight into identifying slow and intermittently failing tests (typically due to Selenium stability).  We’re currently using Jenkins and CircleCI (different projects).<p>What options are out there?  Have other developers encountered challenges managing their automated tests infrastructure?  Have you developed in-house tools to address this?  Or are there any software tools that you’d recommend?
======
trcollinson
It's great that you have embraced testing in your product development! And
rest assured it is not uncommon for you to run into issues when your test
suite becomes large.

Unfortunately there is no silver bullet to make your tests run faster. I have
used both Jenkins and CircleCI and they are great for most environments.

To combat this speed issue I have added a timer to all of my individual test
runs. In Ruby with rspec, for example, this is relatively simple by just
adding --profile to your .rspec file. Every time your test suite is run, this
will show the top 10 slowest running tests. Each testing framework has a
similar mechanism or add-on which will give you a report. Next start
refactoring those long running tests. At first it will seem like a slow
processes, but working on it a bit each day will quickly show significant
improvement in your testing suite speed.

Next, watch out for tests, particularly integration tests and Selenium tests,
which either test the framework and not your code, or which are redundant. A
number of times I have walked into clients offices and have been told that
their test suite is as "tight as it can be" and yet still runs slow. I look
through it and find many areas where engineers in their desire to test
thoroughly were unfortunately testing areas they don't need to. A bit of
refactoring again showed massive dividends.

Finally, don't be afraid to segment your tests to run only portions that are
effected by the code that was changed. It's true that running the entire suite
is important, but you're correct in only running the whole thing at certain
times. My current project has segments for API, various Front End resources,
and service integrations. There are a number of sub segments as well. Again I
think of this as a test refactoring exercise, but it will pay off if designed
well.

So, unfortunately I don't have a software solution that will solve your
problems. But I can say that a few refactoring activities can make all of the
difference.

~~~
jonstjohn
Great tips - thanks! In addition to some slow-running tests, we do have a lot
of test duplication. It's probably time to step back and evaluate what tests
are overlapping functionality.

Do you use any tools to facilitate your workflow on fixing tests? Typically,
we run a regression build on a branch and may find a half dozen failures. With
4-5 developers on a distributed team, we've built a utility for 'checking out'
and marking a test as fixed (to be verified on the next build) but haven't
seen any similar functionality in the CI systems we use.

~~~
trcollinson
Nothing slows down a test suite like test duplication! I would definitely
suggest a refactor, it will help. But remember you don't eat an elephant all
at once, take it one bite at a time. A small amount but consistent level of
test refactoring will go a long well.

It sounds like you have a good workflow to me but obviously you feel you have
a problem or you wouldn't be posting here. I've working in environments where
the failing tests are assigned out on an individual basis, where there was a
bouncer who took on all broken tests, and where each individual engineer would
run the tests often enough that they would just fix what they broke. The
latter method is my favorite. But it is not always feasible.

I guess the answer really depends on your environment. I would be very happy
to give my advise (for what it's worth). If you would like you may email me.
My address is my Hacker News username at googles very large and famous free
email service.

------
pbiggar
I work at CircleCI, so I can take a stab at some of these questions.

Selenium tests are a pain, and we come across a lot of customers who have
unstable selenium tests. We're building a lot in the next few months to help
with it.

Firstly, one feature that we've just put into beta is to show you failing
tests. If you're using RSpec or Cucumber or another framework that can
generate junit.xml files, we should be able to show you which tests failed,
which is shown on the build page, in chat integrations, etc. (Also available
via API if you want to consume it via hubot or at the command line or
similar).

We're planning to use this to track flaky tests. Someone on our team has the
objective to build it this year, so if things go as we optimistically guess
that they might, we might see it this year. Otherwise, it should be easy to
use the API to build that yourselves.

It seems like that might help you - let me know!

~~~
jonstjohn
Thanks! I'll have to check that out. I really love the CircleCI service -
incredible time saver in getting continuous integration setup and so many
features out of the box. Would definitely recommend it to anybody looking for
a hosted CI solution (good price, too!).

------
forgottenpass
I have a similar problem but unfortunately the conventional wisdom to long-
duration tests is that we're just doing it wrong. I'd love to say there is a
good system available for having a pile of infrastructure in a corner
somewhere running tests, but I haven't found anything with out of the box
support for a more complicated execution model than running latest tests on
latest build artifact.

The best I've been able to accomplish is a set of parametrized Jenkins jobs
that default to running latest tests on latest code, and report results to a
custom external application. The test monitor has a status console and can
automatically schedule more Jenkins jobs using build parameters to run the
tests with specific versions of application and tests to bisect failures. It's
ugly, but is tolerable for what we need.

~~~
jonstjohn
Agreed about the conventional wisdom. Unfortunately, I think many companies
aren't 'doing it right' and end up with somewhat bloated systems. From what
I'm hearing, most companies roll their own solution to that problem and don't
use any specific tools to address it. I'm seen some rudimentary statistics in
Sauce Labs, as well as built into Jenkins.

