It would be possible, but Microsoft has already accumulated many decades of being extremely reluctant to make even the slightest changes in Windows in order to accommodate new CPU features.
Since 1990 and until now, the Intel-AMD ISA and the hardware architecture of the Intel-AMD CPUs have accumulated a large number of very undesirable features, which increase the cost of hardware and reduce the security of computers, for which the only justification was the fact that Microsoft refused to update Windows so they expected that the hardware vendors must do whatever it takes to keep their CPUs compatible with the legacy operating systems.
These workarounds have moved inside the CPUs many functions that properly belong to the operating system, like remote management and voltage-frequency control.
And do you know about Linux? After cursory search I couldn't find anything as well. In this case it seems very expensive, if not impossible without the huge penalty, to implement such a workaround in the hardware.
Heterogeneous compute aware kernel scheduler seems like the most natural approach to me.
The kernel can't do anything here, unless your answer is "threads can't ever migrate between types of cores, so cross your fingers on which core you spawn on". The kernel doesn't know what flavor of ISA userspace is expecting, because userspace doesn't know what flavor it wants until it actually gets to run: symbol multiversioning lets you compile a function for multiple versions of an instruction set and pick the best one at runtime.
You could imagine the kernel being clever and handling the first SIGILL in hardware and then marking the process as big-core-only, but you can't handle this generally since you can have new instructions get encoded as nop (I believe the AArch64 PAC instructions do that?). You could also have an ifunc resolver intentionally call the instruction and expect and handle the signal.
The solution is to make your hardware actually work, and not claim to support instructions that do not actually exist on all of your cores.
When an invalid instruction exception occurs, it is very easy to identify whether the invalid instruction was an AVX-512 instruction, because those start with a prefix different from any other instructions.
After that happens, the thread can be marked as requiring AVX-512 and then it can be scheduled only on P-cores.
Moreover, one can define a new flag for the ELF header of executable files, specifying the AVX-512 requirement, which will be set by the compiler, depending on the compilation options, and then the OS knows from the beginning how to schedule a process.
Of course, the best solution is still to implement a uniform ISA. If a lower performance is accepted, the execution of the AVX-512 instructions can be implemented at a negligible additional cost over the previous x86_64 ISA.
What has scared the E-cores designers into not implementing AVX-512 was not the execution part, but the instruction decoding part. The instruction decoder for x86_64 is already horrendously complicated when many instructions must be decoded concurrently in a single clock cycle and adding the need to handle the completely distinct encoding of the AVX-512 makes it even more complicated.
Nonetheless, the competition from AMD has eventually forced Intel to give up in their attempt of reducing production costs by saving die area with a simpler instruction decoder in the E-cores, so the E-cores of Nova Lake will have the same ISA as its P-cores.
And what happens when E-core decoder encounters the AVX-512 instruction set? Does this mean that E-cores will have to implement some sort of (downsized) vectorized execution engine too? Otherwise, I don't see how this would work. Migrating on the HW core level would be too complicated.
> When an invalid instruction exception occurs, it is very easy to identify whether the invalid instruction was an AVX-512 instruction, because those start with a prefix different from any other instructions.
You're missing the point: the processes executing AVX-512 do not have any particularly strong requirement for AVX-512 instructions, they're just optimistically using them to improve performance because CPUID claimed they were supported. It would be pathologically bad behavior to control scheduling behavior with this, because in practice it would result in things like glibc deciding to use AVX-512 for string functions resulting in every program that uses memcpy (i.e. every program) not being allowed to be scheduled on the little cores.
