> It is not my fault if Claude outputs something like "*1*, *1*", adding markdown highlighting, when most other models respect the required format correctly.
Yuck. At that point don't publish a benchmark, explains why their results are useless too.
-
Edit since I'm not able to reply to the below comment:
"I want structured output from a model that supports structured output but will not enable structured output, nor ask for an existing format like XML or JSON" is not really an interesting thing to benchmark, and that's reflected in how you have Gemini 2.5 Flash beating GPT-5.4.
I really hope no one reads that list and thinks it's an AI leaderboard in any generalizable sense.
Why not? I described this in more detail in other comments.
Even when using structured output, sometimes you want to define how the data should be displayed or formatted, especially for cases like chat bots, article writing, tool usage, calling external api's, parsing documents, etc.
Most models get this right. Also, this is just one failure mode of Claude.
Interesting benchmark. It is notable that Gemini-3-Flash outperforms 3.1 Pro. My experience using Flash via Opencode over the past month suggests it is quite underrated.
Needless to say, benchmarks are limited and impressions vary widely by problem domain, harness, written language, and personal preference (simplicity vs detail, tone, etc.). If personal experience is the only true measure, as with wine, solving this discovery gap is an interesting challenge (LLM sommelier!), even if model evolution eventually makes the choice trivial. (I prefer Gemini 3 for its wide knowledge, Sonnet 4.6 for balance, and GLM-5 for simplicity.)
It’s worth also comparing Qwen 3.5, it’s a very strong model. Different benchmarks give different results, but in general Qwen 3.5, GLM 5, and Kimi K2.5 are all excellent models, and not too far from current SOTA models in capability/intelligence. In my own non-coding tests, they were better than Gemini 3.1 flash. They’re comparable to the best American models from 6 months ago.
Not really related, but does anybody know if somebody's tracking same models performance on some benchmarks over time? Sometimes I feel like I'm being A/B tested.
Yeah, good tests are associated with cost. I'd like to see benchmarks on big messy codebases and how models perform on a clearly defined task that's easy to verify.
I was thinking that tokens spent in such case could also be an interesting measure, but some agent can do small useful refactoring. Although prompt could specify to do the minimal change required to achieve the goal.
What about images, links?
Formatted text like bold or underline?
I also prefer plain text, but in most of my emails I talk about technical stuff, or I send transactional emails that require actions, in which case showing buttons is a much better user experience than plain text.
Every MUA I've used allows the reader to set a font size, so changing font sizes is 100% a feature of plain-text emails. Then they get the link the size they need to read it correctly and it's absolutely easy to read. This here comment is pain text. Is it hard to read this link:
I think the OP app is meant for creating transactional emails (or bulk-send emails like newsletters).
Those templates should account for all types of people and accessibility levels (including things like ADHD, where you need a big red button to click, otherwise you get overwhelmed by a block of text).
Using a URL shortener obviously. But you are right, if they only send plain text, they won't be able to include those 1x1 images at the bottom to track whether you have opened the email. Any sane email client blocks images by default, but whatever.
Yeah, I get it, I unfortunately live in the real world too. I like to keep it plain text whenever possible but it's extremely useful sometimes to have inline screenshots and stuff like that.
I didn't mean to be sarcastic but it's just that to me, philosophically, email is a plaintext technology that had HTML bolted on to it kicking and screaming, and it's always been kind of crap. People like me hate things that are fundamentally ugly and crap even if they are useful. The web was designed for HTML from the start.
Not sure if AI can have clever or new ideas, it still seems to be it combines existing knowledge and executes algoritms.
I am not necessarily saying humans do something different either, but I have yet to see a novel solution from an AI that is not simply an extrapolation of current knowledge.
Speaking as a researcher, the line between new ideas and existing knowledge is very blurry and maybe doesn't even exist. The vast majority of research papers get new results by combining existing ideas in novel ways. This process can lead to genuinely new ideas, because the results of a good project teach you unexpected things.
My biggest hesitation with AI research at the moment is that they may not be as good at this last step as humans. They may make novel observations, but will they internalize these results as deeply as a human researcher would? But this is just a theoretical argument; in practice, I see no signs of progress slowing down.
This is my take as well. A human who learns, say, a Towers of Hanoi algorithm, will be able to apply it and use it next time without having to figure it out all over again. An LLM would probably get there eventually, but would have to do it all over again from scratch the next time. This makes it difficult combine lessons in new ways. Any new advancement relying on that foundational skill relies on, essentially, climbing the whole mountain from the ground.
I suppose the other side of it is that if you add what the model has figured out to the training set, it will always know it.
We call that Standing On The Shoulders Of Giants and revere Isaac Newton as clever, even though he himself stated that he was standing on the shoulders of giants.
The difference people are neglecting to point out is the experiences we have versus the experiences the AI has.
We have at least 5 senses, our thoughts, feelings, hormonal fluctuations, sleep and continuous analog exposure to all of these things 24/7. It's vastly different from how inputs are fed into an LLM.
On top of that we have millions of years of evolution toward processing this vast array of analog inputs.
Jokes aside, imagine you give LLMs access to real-time, world-wide satellite imagery and just tell it to discover new patrerns/phenomens and corrrlations in the world.
It means extending/expanding something, but the information is based on the current data.
In computer games, extrapolation is finding the future position of an object based on the current position, velocity and time wanted. We do have some "new" position, but the sistem entropy/information is the same.
Or if we have a line, we can expand infinitely and get new points, but this information was already there in the y = m * x + b line formula.
Shameless plug: if you think about switching from Google Analytics/Hotjar, check out my project[0] (built in EU, started in Romania, now working on it remotely in Netherlands).
If not, happy to hear any criticism or the alternatives you decided to go with instead.
I've been building my project[0] for 14+ years, but I'm still looking forward to the day I won't have to worry about next month's rent. Customers are happy though. Last week I got messages from two customers that purchased more than 7 years ago, because they wanted to install/use the product again. It was fun to revisit our email conversations and to see how much the product has evolved since. What's funny is that the feeling is the same after 1, 5 or 15 years. There's always stuff to optimize and improve. Unless it's a standalone experience (e.g. game or a movie), most software and tools do need constant updates and improvents to keep up-to-date with the current world.
What's funny, is that I had many other projects attempted over the years, but many rise and die quickly, yet the one that lasted the longest is also the one that is likely to last longest from now.
I tried it, it was fun to get started, but after 3-4 sessions, when you want to start getting good and learning stuff, it's not fun anymore.
I tried playing with Playtomic, with better players, and everyone expects to play the same, go and play at the net as that's the main goal, but I dislike staying close to the net and playing the reflex game, while trying to not get hit by the ball. I have a lot more fun playing back court, and volleying the ball, but apparently that's not the way to play this sport.
Also, the wall part of the game, once you play against better players, is not fun. Most of the balls go in the corner, and you have to wait for the ball/predict and avoid hitting the glass or hit from an uncomfortable position, which is not as fun as the open play game.
https://aibenchy.com/compare/minimax-minimax-m2-7-medium/moo...
reply