Ask HN: Are agent benchmarks the new credit ratings?

[shreyassudhanva]

I did some talking to engineers at new-ish startups and agent companies are buying benchmarks to prove reliability to their customers. I'm theorizing that it looks a lot like pre-2008 credit ratings: the party being rated pays for the rating.

If you were to buy, or even approve an agent product, how much do you care about these benchmarks? What gets it over the line for you?

[uliseis]

I don't really think benchmarks are very useful than for wide marketing... If I were a customer that wanted to get some AI agents in my workflows, I would prefer the vendor to provide a small PoC delivering "real" results than looking to (possibly) cooked results in benchmarks (as you say, that one who pays gets the rating)...

[blinkbat]

Tests and community are far better indicators