We run benchmarks against all of our endpoints, in production. That first chart that the author shows is in fact our live benchmarking data. If providers underperform, we kick them out of the routing pool. That is why we run those benchmarks. Performance
And errrr...yeah...that auto-exacto performance chart is both 100% useless, and totally unclear. We will get that fixed. But under the covers it is doing a lot of valuable work! https://openrouter.ai/docs/guides/routing/auto-exacto
Oh wow someone from the team! I'm really heartened by this response -- as you can tell, I've been something of a fan, so I felt kinda betrayed by the idea that this was a risk. Thanks for clarifying, and for taking my frustrated hyperbole in stride.
Submitting an application to your product role now. "All of the models are at least okay" is heartening, but there's a whole bunch of fun places to take this.
And errrr...yeah...that auto-exacto performance chart is both 100% useless, and totally unclear. We will get that fixed. But under the covers it is doing a lot of valuable work! https://openrouter.ai/docs/guides/routing/auto-exacto