Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.
BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.
> Providers probably serve quantized versions without disclosing it
It should be OpenRouter's responsibility to protect you against it, by regularly benchmarking providers and giving you the control to avoid bad providers.
In fact, that's a big opportunity for them, since it justifies their place as a middleman between users and inference providers.
100% agree. The key issue is that users think they are getting results from a specific model configuration and they are clearly not, which is fraudulent.
If OpenRouter wants to succeed as a business, they need to be auditing the providers they connect to (i.e. benchmarking) and removing fraudulent ones from their service.
It's more than just quantization. The middleware the provider is running matters a lot even to the point of exactly which version they are running due to defects being introduced / resolved.
In my coding agent harness I've included 25 open weight providers mainly because I keep having to find new ones when what was previously a great combination of model and provider becomes pretty bad. vllm has defect that causes reasoning to get dropped much of the time for the GLM family of models. sglang has a defect that causes the elements of array args to get dropped for the deepseek family of models. Some providers need some very specific additional config passed through for reasoning to make it back to the model.
I've not tried OpenRouter as adding yet another layer will just make it that much more difficult to get a model and provider combination working well.
I suspect people's bad experiences with open weight models have a lot to do with these headaches. Finding a good model and provider combination is pretty tedious and so far its been a never ending process. I'd really like to host my own models but it isn't economically feasible for one person for the open weight models that work well (i.e. the 300B+ ones).
I mean that is addressed as well later, even when they do declare the quantization, doesn't mean you'll get better performance than the lower quantized one. I'm guessing they're doing something similar to what stadia was doing, saying you are playing games in 4k, because you're getting a 4k stream, but the game itself is running in 1080p
Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.
BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.