Likewise, I switched over about 20 frustratingly buggy llm judgement calls that were using Luna (should the model use X tool, should it trigger deep research, what topic is the question etc.). I had spent days tuning the prompts to get somewhat decent results - meaning correct maybe 70% of the time.
I told an agent to test switching those all over to Jev. Came back a couple of hours later, thinking of this as an experiment, no major expectations, to find that they were all giving 100% accurate results in 800ms median time.
So it got adopted in a couple of hours work, it's live now, working very well, and I haven't had to think about it for the past week. That's the definition of a successful product.
EDIT: I should note two things
- there was actually one non English case that didn't have such high accuracy, now that I think about it.
- I assume a big part of why I got such good results was that I'd already spent ages tuning the prompts for Luna
OpenAI has a decisions API now so you didn't even have to migrate to Jev at all. Personally I'm using the open weight D3 decision models which work great and work locally on everything from phones to beefier computers due to the range of models they have.
It's still a pretty new field but Jev handily beats Luna-decisions on the Artificial Analysis Decision Index on every metric (price, latency, and accuracy)
I already have openrouter wired in so it was about as easy either way. Anyway I'm always happy to switch even a little bit of the stack away from OpenAI.
