On: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

[MattDaEskimo]

Just because you can generalize the topic doesn't mean you can ignore the specific conversation and choose your hill to argue.

Additionally, the conversation of this topic is about the model's ability to generalize and it's potential overfitting, which is arguably more important than parroting mathematics.

[whimsicalism → MattDaEskimo]

performance on a held-out set (like frontiermath) compared to putnam (which is not held out) is obviously relevant to a model's potential overfitting.

i'm not going to keep replying, others can judge whether they think what i'm saying is "relevant at all."

[MattDaEskimo → whimsicalism]

Again, you set your own goal posts and failed to add any insights.

The topic here isn't "o-series sucks", it's addressing a found concern.