I've always felt the best approach for (automated) testing is:
Unit test style tests should test the specification. That is, you test that good input creates the results that the specification states it should and you test that the known failure cases that the specification says should be handled are, indeed, handled as the specification states. This means that most of the tests are generally happy path tests with some testing the boundaries and known bad cases. The goal is not to find bugs, but to prove in code that the implementation does in fact meet the requirements as set out in the specification.
Regression tests. Any time a bug is found, add a failing test to reproduce the code. Fix the bug and the test should pass. This both proves that the bug is actually fixed and prevents the bug from creeping back in again later. Again, the goal is not to find bugs, its to prevent reoccurrence and is completely reactionary.
Finally, property-based generative testing. Define properties and invariants of your code and schemas of your data, then generate random test cases (fuzzing essentially). Run these on a regular basis (overnight) and make sure that the properties always hold, for good input data, and that error states are handled correctly, for bad input data. You can also apply this to the overall system, by simulating network failures between docker containers [1]. The goal of this is to find bugs, since it will test things you won't have thought of, but since it generates random test cases, you aren't guaranteed that it will find anything. Its also notoriously hard to write these tests and come up with good properties. I don't often do these tests, only when the payoff vs effort seems worth it. Eg for a low impact system, its not worth the effort, but for something that's critical, it may be.
For human QA, I think it makes most sense to test the workflows that users do. Maybe sure everything works as expected, make sure the workflow isn't obtuse or awkward, make sure everything visually looks ok and isn't annoyingly slow. Test common mistakes due to user error. Stuff like that. I don't think we can expect this to be thorough and its unrealistic to think that it will find many bugs, just that it will make sure that most users experiences will be as designed.
So, test for what you expect (to prove that its what you expect), test known bugs to prevent regression and to prove that its fixed. Then, only if your software is critical enough to warrant the effort, use property-based testing as necessary to try and weed out actual bugs. Most software can skip that though.
IMO a key thing for human QA testing is to test unhappy paths and error handling. What happens if the user does something unexpected. The happy path is generally well tested by developers during development, but testing the unhappy paths is much more difficult and time consuming, and that's where the QAs come in.
Sure, agreed. That’s why I said “Test common mistakes due to user error. Stuff like that that.” My main point is that it’s unrealistic to expect human QA to find bugs, but what humans shine at that computers don’t is workflow issues, visual issues and “does it feel slow/bad/whatever”. I suppose leaning more heavily on the mistakes side of the workflow makes sense though, yes.
Unit test style tests should test the specification. That is, you test that good input creates the results that the specification states it should and you test that the known failure cases that the specification says should be handled are, indeed, handled as the specification states. This means that most of the tests are generally happy path tests with some testing the boundaries and known bad cases. The goal is not to find bugs, but to prove in code that the implementation does in fact meet the requirements as set out in the specification.
Regression tests. Any time a bug is found, add a failing test to reproduce the code. Fix the bug and the test should pass. This both proves that the bug is actually fixed and prevents the bug from creeping back in again later. Again, the goal is not to find bugs, its to prevent reoccurrence and is completely reactionary.
Finally, property-based generative testing. Define properties and invariants of your code and schemas of your data, then generate random test cases (fuzzing essentially). Run these on a regular basis (overnight) and make sure that the properties always hold, for good input data, and that error states are handled correctly, for bad input data. You can also apply this to the overall system, by simulating network failures between docker containers [1]. The goal of this is to find bugs, since it will test things you won't have thought of, but since it generates random test cases, you aren't guaranteed that it will find anything. Its also notoriously hard to write these tests and come up with good properties. I don't often do these tests, only when the payoff vs effort seems worth it. Eg for a low impact system, its not worth the effort, but for something that's critical, it may be.
For human QA, I think it makes most sense to test the workflows that users do. Maybe sure everything works as expected, make sure the workflow isn't obtuse or awkward, make sure everything visually looks ok and isn't annoyingly slow. Test common mistakes due to user error. Stuff like that. I don't think we can expect this to be thorough and its unrealistic to think that it will find many bugs, just that it will make sure that most users experiences will be as designed.
So, test for what you expect (to prove that its what you expect), test known bugs to prevent regression and to prove that its fixed. Then, only if your software is critical enough to warrant the effort, use property-based testing as necessary to try and weed out actual bugs. Most software can skip that though.
[1] For example https://github.com/agladkowski/docker-network-failure-simula... or https://bemowski.github.io/docker-iptables/ I've personally successfully used https://github.com/IG-Group/Havoc to test a fault tolerant distributed system I worked on a few years ago, using Clojure's generative testing to create test cases and failure scenarios and Havoc to handle injecting of network errors.