I built my own. It’s IMO necessary, but it quickly turns into a job by itself. I’ve been benching K3.1 all weekend. Even when you get a good test suite, you need to do multiple runs, not for variance, but because the model behavior changes so much between releases you often need to retune to get a real score if what you’re after is actual optimal (or closer to it) performance on your workload.
Example: I could barely induce K3 to use subagents. I had to force it via instructions and then, in my tests to try to get it to use Qwen, it didn’t work well. 3.1 lost 6 hours thrashing overnight because it voluntarily called subagents and the box can only run one Kimi at a time. My own fault for not looking up the new Moonshot system prompt (that they changed also because of this behavior), but one example of how different even point releases can be.
Example: I could barely induce K3 to use subagents. I had to force it via instructions and then, in my tests to try to get it to use Qwen, it didn’t work well. 3.1 lost 6 hours thrashing overnight because it voluntarily called subagents and the box can only run one Kimi at a time. My own fault for not looking up the new Moonshot system prompt (that they changed also because of this behavior), but one example of how different even point releases can be.