Although I agree that you must test optimized code for correctness, I think it should be approached very cautiously, as tests can become checkpoints that constrain optimization.
I had a memorable experience of that, actually - for a long time we had a Trello card titled “faster to operate on the bits instead”, and the team looked at that card fondly during development. After we started getting traction, it was finally time to optimize! We conscientiously wrote tests to ensure correctness, and then began an incredibly enjoyable informal hackathon. We blazed through every step, replacing the logic piece by piece with bit operations. This step was 2x faster, that step was 5x faster, and so on. A few steps were slower, which was a bit weird, but we were having so much fun shouting out our gains to each other across the room, racing to improve. By the afternoon we had optimized each step and gotten an order of magnitude improvement.
“Great stuff, team.”
“I thought it would have been faster.”
“Yeah, same.”
We did some evening profiling and noticed a lot of time being spent in a dozen or so functions with names like “toRepr”, “reconstitute”, stuff like that. Actually it was like most of the time - what!? And then we saw, at each step we converted to raw bits, performed the equivalent logic with fast bit operations, then converted back to the representation for handoff to the next step. Wait, why we are converting to and from bits like thirty times each run? That can’t be right. Realization strikes - right before most handoffs there was a commented-out line that ran out tests on the representation to ensure it was still correct.
Someone pointed out we wrote that test before we had done any investigation of performance (I remember it almost word for word actually, “When we wrote the test, all we knew was it would be faster in bits, but not how. And the test doesn’t even look at bits!”). So we ripped out all the conversions and stayed in bits from step to step. Without changing the bit logic it was now almost three orders of magnitude performance gain, basically running in milliseconds instead of seconds. Two OOM that we had earned, but left on the table!
Tests introduced arbitrary checkpoints and imposed irrelevant constraints on the problem, without us noticing until afterwards. We caught the problem pretty easily that time, but I do wonder how many other times we (or others) didn’t because it wasn’t as obvious. The service that accepted the output maybe should have been accepting raw bits instead of the representation. The service that gave us the input definitely should have been staying in the bits.
So, do test your optimizations for correctness, but be really careful. Tests can easily and subtly constrain your optimization space in major and arbitrary ways!
I haven’t used TDD much but when I did I found I had that issue, that I would imagine the code I wanted to write and then write tests to check that I had written that code (obviously not the intent of TDD). I got a little better when I thought about testing requirements. I actually used an old trick I’d learned from the head of a web dev shop - when he was struggling to elicit requirements from a client in a meeting, he would hold up a pencil and say “does this fix your problem? why not?” and then inverting their objections would produce requirements more fluidly. (Obviously you don’t stay on the pencil for long, but it puts the client in the right mental state of “looking for lack of capabilities in proposed solution” rather than the more usual “bullet point shopping list of desired delivery levels, SLAs, etc”).
I had a memorable experience of that, actually - for a long time we had a Trello card titled “faster to operate on the bits instead”, and the team looked at that card fondly during development. After we started getting traction, it was finally time to optimize! We conscientiously wrote tests to ensure correctness, and then began an incredibly enjoyable informal hackathon. We blazed through every step, replacing the logic piece by piece with bit operations. This step was 2x faster, that step was 5x faster, and so on. A few steps were slower, which was a bit weird, but we were having so much fun shouting out our gains to each other across the room, racing to improve. By the afternoon we had optimized each step and gotten an order of magnitude improvement.
“Great stuff, team.”
“I thought it would have been faster.”
“Yeah, same.”
We did some evening profiling and noticed a lot of time being spent in a dozen or so functions with names like “toRepr”, “reconstitute”, stuff like that. Actually it was like most of the time - what!? And then we saw, at each step we converted to raw bits, performed the equivalent logic with fast bit operations, then converted back to the representation for handoff to the next step. Wait, why we are converting to and from bits like thirty times each run? That can’t be right. Realization strikes - right before most handoffs there was a commented-out line that ran out tests on the representation to ensure it was still correct.
Someone pointed out we wrote that test before we had done any investigation of performance (I remember it almost word for word actually, “When we wrote the test, all we knew was it would be faster in bits, but not how. And the test doesn’t even look at bits!”). So we ripped out all the conversions and stayed in bits from step to step. Without changing the bit logic it was now almost three orders of magnitude performance gain, basically running in milliseconds instead of seconds. Two OOM that we had earned, but left on the table!
Tests introduced arbitrary checkpoints and imposed irrelevant constraints on the problem, without us noticing until afterwards. We caught the problem pretty easily that time, but I do wonder how many other times we (or others) didn’t because it wasn’t as obvious. The service that accepted the output maybe should have been accepting raw bits instead of the representation. The service that gave us the input definitely should have been staying in the bits.
So, do test your optimizations for correctness, but be really careful. Tests can easily and subtly constrain your optimization space in major and arbitrary ways!