Hacker Newsnew | past | comments | ask | show | jobs | submit | seunosewa's commentslogin

I believe the AI labs are weakly motivated to train strongly against cheating when it helps with benchmarks.

Does it help with benchmarks? Are you saying there are examples of benchmarks where the models have solved the problem by cheating?

Hundreds, at this point? Every benchmark is flawed as shit, written by clowns. DeepSWE, They're given the full git history (the solution is in it), others don't even bother to verify if the code is the right one and just the output, they've modified the test harnesses, injected code to make all tests pass, etc. The entire benchmark galaxy is just clowns propping eachother up and are regularly talking with the big AI labs.

It will get into the hands of people who just want to burn the world down.

Meta is awfully close.

lol! Good one...

Went from years behind to months pretty quick.

In my research it's a solid 10 - 20% shorter on average, free of charge. That's nice to stack on top of other optimisations.

It's not a general trend. It's only Opus 5.

Sonnet-5 does the same

Fable 5 is the same

They are cute but nasty. They eat their prey alive in the most painful way.


i don't have a source but there's something about their mouth or teeth that make it very hard for them to suffocate prey the way a lion or hyena would. They kill their prey through shock and blood loss from disembowelment which means basically eating it alive like you said.


Oof, I'm glad I didn't witness that (not that the zoo would feed live animals to them). It certainly would've changed my opinion on them, even though it's just their nature.


From the benchmarks, the single core performance is close but lower.


Unfortunately?


$100 Claude and $100 ChatGPT Pro is the best value for $200/month. They can see each others' mistakes.


Could you provide a practical example?


There is one in the quickstart:

    mathcode -p "prove that the square of an even number is even"
https://math-ai-org.github.io/mathcode/#quickstart - if you look very closely, the screenshot at the top actually shows the output (and the solution).


so ' a problem' here is just preexisting math theorems ?


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: