On: Why is everyone trying to replace Software Enginee...

[taylodl]

> I am developing an evaluation suite so I can keep watching the progress in a systematic way..

Sounds like something that should be published on github

[pona-a → taylodl]

Open benchmarks are vulnerable to saturation. I think benchmarks should have an embargo periodic, until which only 3% of the question-answer pairs is released, with an explicit warning not to use it 3 months after being released.