There are hundreds of verifiable tasks in AI development as simple benchmarks on the models themselves, such as
1. Performance engineering ability
2. Ability to reconstruct visuals in code alone
3. Ability to orchestrate cheaper models towards a given task
4. Pretty much any verifiable engineering task
The loop is pretty clear, have AI's work on pre and post-training, then benchmark their outputs against these tasks to determine how to steer AI progress. The non-verifiable stuff isn't on the critical path towards these accoutrements anyway.
1. Performance engineering ability 2. Ability to reconstruct visuals in code alone 3. Ability to orchestrate cheaper models towards a given task 4. Pretty much any verifiable engineering task
The loop is pretty clear, have AI's work on pre and post-training, then benchmark their outputs against these tasks to determine how to steer AI progress. The non-verifiable stuff isn't on the critical path towards these accoutrements anyway.