I work for a Danish municipality and we buy quite a lot of development from various software houses. Being the public sector we track and benchmark almost everything, and we actually have a dataset on automated testing that’s been running for two decades.
It’s hard to use the data, because we’re comparing different projects, teams and suppliers but our data shows no advantage in choosing the companies that are very test-focused.
They are often slower, more expensive but have the same amount of incident reports as the companies which tests less or doesn’t do automated test at all.
Interesting. My intuitions (as a developer of 25 years experience) are that testing can be a big tax on development velocity; perhaps 60% of time is spent on writing tests, especially in a more complicated code base where the choice is between tests with lots of mocks (=> brittle against code change), or tests with lots of data setup (=> brittle against schema change), where the tests are more verbose than the code under test.
I also think testing compensates for communication inefficiencies when developing with a larger team with developers with less tenure and experience. I'm not sure this tax is avoidable as you try and scale up organizationally.
Code architecture also has a significant effect on the cost of writing tests, and TDD was originally propounded as a way to ensure that code is testable, but with object-orientation it tends instead to lead towards dependency injection and pluggable interfaces purely for testing rather than reconfiguration, and you end up with overly abstract software.
More functional code, and architecture which increasingly represents control flow as data flow, is a better direction to aim in, IMO. That argues towards streaming and messaging techniques more than interfaces and APIs.
TDD is great for fixing bugs, less great for software design. I think the key reason is that the best design is in tension; it's a solution being applied to several problems, and triangulating that point starting from a single problem is both (a) further away from the solution, and (b) unlikely to be close to the final design unless you were really lucking in choosing the test problem.
One question: are they somehow more efficient in answering to an incident report?
My conjecture: I am not so surprised at these having the same % of failures - after all the TDD will not magically lead everyone to think of all the possible ramifications; you write tests for the case that you can "imagine" just like a non-TDD developer will implement logic to cover cases they can "imagine".
But on the other hand, if after fixing the bug you need to prove that you haven't caused some other side effect, then having a full test suite in place should be more efficient than retest everything again (or less risky than "ok, fixing module W should not impact X, Y and Z, so we test X only and ship...").
I guess these are good points. The last line especially.
Yet it's obvious to me that correctness and stability should be features, and more fundamental than all others. It's not as if a non-extreme amounts of testing is all that expensive.
That hasn’t been my experience at all. Test-oriented firms and test-oriented projects have had far fewer incidents and the incidents they have had have been resolved much faster. The data has been so conclusive that our exexutives are mandating that we test. Maybe the difference is automation? Automating the tests is key.
This may hold some of the answer. IMO, tests are basically automated debugging, that you can run as many times as you want. To take full advantage, you should run them basically whenever you can, in the background of editing, forced to succeed upon merge, etc.
I like this approach and it was used heavily in a previous role by both the devops and development teams in a company I worked for.
However it can come with a cost if the tests are not targeted enough or there are simply too many tests to run within a reasonable period. At one stage tests were taking 20-30 minutes to run, so you could be waiting an hour or more sometimes to find out whether the change you pushed went green or red. Often this meant people pushing before their lunchbreak or at the end of the day and effectively leaving the build broken for many hours, with others unable to push unless they included a fix for someone else's breakage.
I see a lot of teams/projects where the tests take 20 to 30 minutes. Like that's the threshold where you can just barely get anything done, so things grow to that size, development velocity grinds nearly to a halt, and they stop growing. So everything you see is about that size.
Personally I need tests to run in a few seconds. I work on a large project that does it so it's definitely possible.
I am lucky enough in my current role to have a lab environment that is almost 1:1 with our production environment, so we use testinfra to test all our VM's in parallel, with tests tailored for each VM to the task it performs (database, application, load balancer, DNS, LDAP, SMTP etc). We can run tests on an entire environment in under 2 minutes.
Far more holistic than the 20-30 minutes I described in that previous role, where the whole set of tests was run in a single vagrant VM, usually recycled to save time on spinning up a new VM for each run.
There's a lot of great stuff you can learn from devops, but there also seems to be a large amount of cargo culting where people have cut & pasted certain solutions from a blog or something without understanding how to adapt it to their environment and make it work properly for them.
You are self-selecting for companies that would be willing to go through the bureaucracy to be vendors for you. Consequently, the type of people these companies would hire may not be of high quality. This may or may not be true for you, but is true in my experience.
So the type of customer you are attracts those type of devs.
Do you do any verification on how test focused they are? IME automated testing is like agile, the companies that talk the most are awful at the execution.
I'd completely believe that the automated testing companies were slower and more expensive, but I'd be inclined to blame the quality of the tests themselves. I've seen so many code bases where the tests slow down development and rarely catch errors, where you'll often have to spend more time decoding what the tests were supposed to be testing than the actual development. Even otherwise good developers can see testing as a necessary compliance item and will write awful tests just to get coverage up.
In your dataset are there any standout companies with low incident reports which do/don't have automated tests?
That's an interesting dataset and it aligns with my gut reaction: TDD (and testing in general) are ideal for bug fixes and code refactoring, which would be less useful when the software is going to be sold at a specific point in time (and therefore frozen or mostly frozen).
Companies that sell themselves on TDD (among other things) are part of our dataset. Maybe “automated tests” isn’t the exact literal description of that, but I didn’t mean it as such.
It’s hard to use the data, because we’re comparing different projects, teams and suppliers but our data shows no advantage in choosing the companies that are very test-focused.
They are often slower, more expensive but have the same amount of incident reports as the companies which tests less or doesn’t do automated test at all.