An interesting idea and a good example of the EA/rationality community's unfortunate obsession with putting numbers with little practical basis on everything. The scores may as well be generated from rand(3,8).
The conclusions drawn in the "Discussion" section don't even make use of the scores! It's all qualitative. Embrace that. Sometimes words suit one's purposes better than numbers, but I guess the desire to dress things up with the authority of a number is too tempting.
I'm critical because I believe in EA and think the AI control problem is a worthy cause, but we can do much better.
3. Change both the models and the data collection methods in response to changing understanding of the problem.
4. Loop through 1-3 until the data collection methods and models stabilize.
5. Draw conclusions.
You're looking at a process very much at step 1, and concluding that no models can be drawn from the data, and concluding that we should abandon the process. I think that's a very premature conclusion.
> For one case, the PQCrypto 2006 conference, we were unable to find sufficient information after 45 minutes of investigation to provide an evaluation.
The conclusions drawn in the "Discussion" section don't even make use of the scores! It's all qualitative. Embrace that. Sometimes words suit one's purposes better than numbers, but I guess the desire to dress things up with the authority of a number is too tempting.
I'm critical because I believe in EA and think the AI control problem is a worthy cause, but we can do much better.