I feel like I’m an outlier based on all these comments. This is still working through my personal benchmarks, but it’s spending 30% more time and performing worse than the previous 4.1 flash did.
Benchmark is historical PR reviews. Models are put in a sandbox with a git and db snapshot as it existed at the time. PRs are a selection of changes some bugs (major and minor), some with things that aren’t bugs that have tripped up models in the past and some no findings that they should approve and move along.
4.1 flash already wasn’t that strong at this task, but the speed edged out v4 pro for a spot in the roster.
Benchmark is historical PR reviews. Models are put in a sandbox with a git and db snapshot as it existed at the time. PRs are a selection of changes some bugs (major and minor), some with things that aren’t bugs that have tripped up models in the past and some no findings that they should approve and move along.
4.1 flash already wasn’t that strong at this task, but the speed edged out v4 pro for a spot in the roster.