On: Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

[freakynit]

This is the first set of benchmarks which match my observations around gemini-3.8-flash perfectly. This model is a true hidden gem.

[thefourthchime → freakynit]

Really? I gave it a trivial HTML job, and it went off for fifteen minutes. It did eventually did a do a decent job, but I can't wait that long.

[freakynit → thefourthchime]

I'm using it through antigravity cli .. and in every single run (100's by now), this model was fast, and the outputs were of good quality. Of course, not at Astra or Fable level, but, close to like Sol-low level.

[sureMan6 → thefourthchime]

I gave it a trivial HTML job and it messed it up in several ways including being lazy and lying about results

It's probably very hit or miss like everything with LLMs but I was really surprised it performed that badly