
Ask HN: How do you do software testing? - lawverena
How do you as developer test your codes? How does your QA team involve in testing? What tools do you use in testing? What are your challenges in testing?
======
tcbasche
It depends what type of ticket we'll be playing. If it's a bug, we'll probably
write or augment an existing test that tests that "X does/doesn't happen". If
it's a feature we'll start with the unit and integration tests to show it
works, and then write the code that does that thing.

Our team is all Python so we rely on pytest and mock etc. For testing AWS
related stuff we typically use Moto, which is an AWS mocking library, but
sometimes to have it run locally with localstack, and then against the dev/uat
environments once the code has been pushed.

End-to-end tests exist as well that cover the major user stories but these can
be a little bit flaky - and this is the main challenge right now.

------
ohiovr
I find I get great results with UI testing by observing someone use the
software and not tell them how to use it. Then refine away all the bumps.

------
sethammons
My team's main software is a backend network service that feeds off of a queue
and sends data out into the wider internet. It is the tail end of a pipeline
of work with another team responsible for the incoming work. We call out to a
couple dependent services as part of our flow.

We write unit and unit-integration tests to cover the error cases and success
cases. These make sure that our code flows the way we expect, handles errors
talking to a dependent service or data source, ensures retries happen, errors
and events are logged, etc.

We have integration tests. These are written in a test framework and leverages
docker compose. It spins up all our dependencies and mocked/stubbed sinks for
data we send out to the world, and we have utilities that populate our
incoming edge with data. These tests are black box: given this input, we
expect this output. These tests ensure that known payloads produce known
artifacts.

When developing, we write the tests. Not all stories/tickets need integration
tests, but most do. Unit tests are universally expected to be written (we
don't strive for 100% line coverage, but coverage needs to be reasonable). For
QA, another developer of the same team reviews the code, the tests, and
validates in any other way that the code meets our definition of done, meets
its acceptance criteria, and generally does what it is expected to do. This
developer will often manually execute the code to be sure it does what the
code and tests say it will. They are responsible for asking the "what if"
questions (what if $foo fails? what if assumption $bar changes? do we have
telemetry in place to debug this in the future?) and ensure that the code is
good to go to production.

The ticket/story pull request(s) are blocked from merging to master until the
automated tests pass. After they are merged to master, the automated tests run
again. The master branch tests create any artifacts needed for deployment.

We leverage buildkite for our builds. After master is green, buildkite plugins
allow us to automatically create canary deploys, deploy the whole cluster,
and/or roll back a deploy. Deploys start by going into our staging
environment.

Our staging ecosystem is where all team's latest code lives. We have a set of
acceptance tests that are run here as a matter of course. These run tests as
if a customer initiated an action, and ensure our system behaves as expected.
If these tests do not pass, we do not roll the code to production.

Anyone on our team can deploy any time, though we try to keep that during
business hours and not on Fridays. Deploys are logged/announced in a slack
channel in case something later goes wrong, and someone needs to do some
detective work to find out what happened when.

For the production roll out, as mentioned earlier, we will deploy canary
nodes. We then observe logs, alerts, and metrics. Depending on the scope or
risk of a deploy, canary deploy nodes may be long lived or not. The deploy is
rolled through the rest of the cluster and that is that.

Tools: Go, docker, docker-compose (redis, mysql, other services), dnsmasq for
simulating the outside world, buildkite, kubernetes, argo cd, and more.

QA team: as alluded to, we don't have a "qa team." We separate the QA activity
on our team. As we are responsible for the live monitoring and running of our
services, we are in the best position to more rapidly test our own software.

Challenges: we had to build a lot of tooling, but it was very much worth it.
We can deploy multiple services multiple times per day. We still have more
things we would like to have under automated test and automated monitoring so
we can do better "less high touch" deploys. Our biggest challenges arise due
to scale. We cannot generate enough load in dev or staging environments and
can only see some classes of problems when we go out to the production
clusters. Customer's can do all kinds of strange things with the data they
give us, and we cannot attempt to capture the combinatorial complexity they
can throw our way. Sometimes, an error case has to build for days or weeks to
even be noticeable in our metrics. Some of our caches can be very long lived
and sometimes a cache invalidation bug might slip through; those are the
worst. Any time a bug is discovered, it is added to one of our test suites to
guard against regressions.

Overall, I like where our team is from a testing perspective. There is always
more you can do, but you have to strike a balance between providing value to
the org and protecting ourselves from ourselves. You can go too deep into
testing land for marginal benefit. I think we are at a decent balance where we
can refactor ruthlessly when needed and have good confidence in anything we
release. I would like to see more from our team on performance testing /
monitoring over time. We have telemetry on all this stuff, but not tied to our
build system.

I kinda just slammed this out stream of conscience style; hope it rolls ok.

