A lot of tests don't survive implementation changes, that doesn't make them "bad tests by definition". It means their value came and went. Think of it like scaffolding. You need it for a time, then the time is up, and it's removed. That doesn't make it bad, it was still necessary (or at least useful) for a time.
When there's an implementation change you'll likely end up discarding a fair number of unit tests and creating new ones that reflect the new implementation details. That's just natural.
A lot of tests, especially unit tests, are just change detectors and get updated/go away when change happens, that is just WAI. It is fairly hard to write non change detection tests, it requires for you to really reason abstractly about the contract of your module, or to write integration tests that are moving a bunch of things at once.
small, fine-grained black box tests can be really good for this. in my last project, a type checker, the vast majority of the test suite was code snippets and assertions about expected errors the checker needed to catch, and it was an invaluable aid when making complex changes to the implementation.
Anything that transforms or processes text, like a compiler or type checker, is pretty easy to test. You get into trouble with user interfaces, however.
1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges.
I'd love to have better programmers show me how I can make my tests better. The code is out there.
2. Even if I have good tests "by definition", a radical rewrite might make old tests look like "assert(2x1 == 2), assert (2x2 == 4)". Tests exist in a context, and radically changing the context can change the tests you need.
---
This is not in OP, but I do also have a problem of brittle tests in my editor. In this case I need to test a word-wrapping algorithm. This depends intimately on pixel-precise details of the font. I'd love for better programmers than me to suggest how I can write tests that are robust and also self-evidently correct without magic constants that don't communicate anything to the reader. "Failure: 'x' started at x=67 rather than x=68." Reader's thought: "Why is this a problem?" etc. Comments appreciated on https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t.... The summary at https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t... might help orient readers.
>> If your tests break or go away when your implementation changes, aren’t those bad tests by definition?
> 1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges.
Good and bad are forms of judgement, so let's eschew judgement for the purposes of this reply :-).
> I'd love to have better programmers show me how I can make my tests better.
Better is also a form of judgement and, so, I will not claim I am or am not. What I will claim to do is offer my perspective regarding:
> This is not in OP, but I do also have a problem of brittle tests in my editor.
Unfortunately, brittle tests are the result of being overly specific. This is usually due to tests enforcing implementation knowledge instead of verifying a usage contract. The example assertions above are good examples of this (consider "assert (canMultiply ...)" as a conceptual alternative). What helps mitigate this situation is use of key abstractions relevant to the problem domain along with insulating implementation logic (note that this is not the same as encapsulation, as insulation makes the implementation opaque to collaborators).
In your post, you posit:
> Types, abstractions, tests, versions, state machines, immutability, formal analysis, all these are tools available to us in unfamiliar terrain.
I suggest they serve a purpose beyond when "in unfamiliar terrain." Specifically, these tools provide confidence in system correctness in the presence of change. They also allow people to reason about the nature of a system, including your future-self.
Perhaps most relevant to "brittle tests" are the first two you enumerated - types and abstractions. Having them can allow test suites to be defined against the public contract they provide. And as you rightly point out in your post, having the wrong ones can lead to problems.
The trick is, when incorrect types and/or abstractions are identified, this presents an opportunity to refine understanding of the problem domain and improve key abstractions/collaborations accordingly. Functional testing[0] is really handy to do this fairly rapidly when employed early and often.
