On: Calling the AI bluff: Adding "Do not guess" cut made-up claims from 71% to 20%

[westurner]

So research methods like the scientific method are still subjective in AI implementation according to the AI experts?

Once there are - or next month when there will be - better models, agents, and agent harnesses for this, do you think that then we should concisely specify what is required instead of doing evals for particular models?

So meta-analysis and requisite language are too high-order for existing models and agents, and it's currently necessary to apply such procedural controls outside of the prompt?

[astrange → westurner]

> So research methods like the scientific method are still subjective in AI implementation according to the AI experts?

I am not up to date on philosophy of science, but the scientific method is certainly always subjective, or at least can't be successfully expressed in a formal system.

Here's a book you can read: https://metarationality.com

> Once there are - or next month when there will be - better models, agents, and agent harnesses for this, do you think that then we should concisely specify what is required instead of doing evals for particular models?

Hmm, not sure what you mean. "Evals" are another way of saying "regression tests", so they're useful when you want to change or compare any part of the system.

> and it's currently necessary to apply such procedural controls outside of the prompt?

In general I think you should try to move controls out of the prompt and into an external system, but the downside is that it costs more, so it's not always necessary.

[westurner → astrange]

I'm aware of what evals are.

Do you think it is wise to optimize prompts for specific models or agents when there is a new model every month?

So, to build something like Co-Scientist the controls should be in the agent? Or RLHF'd like other things when training the model?