Very cool! I already can see a lot of "this is already solved by playwright/cypress/selenium/deterministic stuff" in the comments.
Over nearly 10 years in startups (big and small), I've been consistently surprised by how much I hear that "testing has been solved", yet I see very little automation in place and PMs/QAs/devs and sometimes CEOs and VPs doing lots of manual QA. And not only on new features (which is a good thing), also on happy path / core features (arguably a waste of time to test things over and over again).
More than once I worked for a company that was against having a manual QA team, out of principle and more or less valid reasons (we use a typed language so less bug, engineers are empowered, etc etc), but ended up hiring external consultants to handle QA after a big quality incident.
The amount of mismatch between theory and practice in this field is impressive.
> yet I see very little automation in place and PMs/QAs/devs and sometimes CEOs and VPs doing lots of manual QA
Because software is a clownish mimicking of engineering that lacks any real solid and widespread engineering practices.
It's cultural.
Crowds boast their engineering degrees, but have little to show but leetcode and system design black belts, even though their day to day job rarely requires them to architect systems or reimplement a new Levehnstein distance but would benefit a lot from thoroughly investigating functional and non functional requirements and encoding and maintaining those through automation.
There's very little engineering in software, people really care about the borderline fun parts and discard the rest.
Thanks for sharing your experience! Completely agree - there's often a huge gap between the perception that testing is "solved" and the reality of manual QA still being necessary, even for core features. We recently had a call with one of the largest US mobile teams and were surprised to learn they're still doing extensive manual testing because some use cases remain uncovered by traditional tools. It's definitely not as "solved" as many might think.
I work in the field and built a tool that has way less flakiness than deterministic solutions.
The issue is testing environments are always imperfect because (a) they are stateful and (b) there's always some randomness in actual production software. Some teams have very clean testing environment but most don't.
So being non-deterministic is actually an advantage, in practice.
Jevon's paradox though?
Planes are so much more efficient now than the first prototypes, yet usage is so much higher that resource consumption due to airplanes have vastly increased.
Same goes with generative models.
Over nearly 10 years in startups (big and small), I've been consistently surprised by how much I hear that "testing has been solved", yet I see very little automation in place and PMs/QAs/devs and sometimes CEOs and VPs doing lots of manual QA. And not only on new features (which is a good thing), also on happy path / core features (arguably a waste of time to test things over and over again).
More than once I worked for a company that was against having a manual QA team, out of principle and more or less valid reasons (we use a typed language so less bug, engineers are empowered, etc etc), but ended up hiring external consultants to handle QA after a big quality incident.
The amount of mismatch between theory and practice in this field is impressive.
reply