
Ask HN: How often are you frustrated with flaky tests? - ublaze
Are you looking for better tooling to debug&#x2F;catch&#x2F;auto retry flaky tests? What kind of tooling do you already have, and what alternatives have you looked for?<p>Is there some tool&#x2F;service you wish existed?
======
quickthrower2
I've worked somewhere before where there have been flakey tests and they had a
system to detect them and list them, and they could be assigned to people.
Pretty nice.

I'd like it to exist as a service yes, if it can plug into any testing output,
and accurately know that yes something is a flakey test. The telltale sign is
a retry on the same branch fails then passes with no source code changes.

I am not sure however if any company I worked for would PAY for such a system.
That's a different question. But as a developer I'd like to use one if it is
well honed.

------
photonios
Just fix them? Where I work we explicitly do not automatically retry failing
tests. Otherwise the incentive goes away to fix them. At some point somebody
will be frustrated enough to fix the issue.

Note: we have run hundreds of little browser-based tests, visual snapshots and
thousands of unit and integration tests on each PR. Flaky tests get annoying
real quick and get fixed quickly as a result.

------
ComputerGuru
Write tests to be deterministic from the very start (mainly: learn to use
synchronization primitives to synchronize events rather than trying on timing
coincidences) and then you'll realize there's no such thing as a flaky test.

~~~
quickthrower2
Have you written Selenium tests before, pointing to a staging or production
build?

~~~
ComputerGuru
Yes, I have and it's IMHO a very poorly designed API (or rather, abstraction)
for reproducible tests, but yes, the premise still stands. Instead of crossing
your fingers and hoping a navigation has occurred/completed, test
deterministically for the current state (inspecting browser navigation state,
page url, and window events) to decide whether a certain action can be taken
or is valid at the time. Basically, you should build your own framework upon
which real tests can be run.

------
cpufry
not to be too dramatic, but they are bane of my existence

