> I don't get it, you have to save and restore all the state on context switches either way.
Actually, you don't. I'm not up to date as to what's done in this direction by current kernels, but a while back, a patch was merged that disabled floating point and MMX operations by default for a process, and enabled it for a while only when some FP or MMX instruction lead to an exception, which then allowed the kernel to avoid saving and restoring FP state on processes that weren't actually doing any FP/MMX operations.
Actually, you don't. I'm not up to date as to what's done in this direction by current kernels, but a while back, a patch was merged that disabled floating point and MMX operations by default for a process, and enabled it for a while only when some FP or MMX instruction lead to an exception, which then allowed the kernel to avoid saving and restoring FP state on processes that weren't actually doing any FP/MMX operations.