Perhaps CoT and the like may be limited by this. If your model is cooked and does not adequately represent less immediately useful predictions, even if you slap a more global probability maximization mechanism, you can't extract knowledge that's been erased by RLHF/fine-tuning.
