"New processor tells old software it supports [whatever]"
Processors almost never tell software anything; their manuals tell the programmers how the CPU behaves. There may be instructions that can inform code about you optional features of the instruction set, but the software has to run on the CPU before it can query the CPU for its capabilities, so it has to make some assumptions about the CPU (you could bootstrap things by having a smaller CPU boot the hardware and having that CPU query the 'real' one for its capabilities, but that just moves the problem to the smaller CPU)
So, in the end, the only way that software can know how the CPU behaves is because the programmers have read the CPU's documentation.
As to this specific case: I haven't bothered reading up on it, so I do not know what happened. It could be lack of documentation by Intel, it could be that AMDs documentation was incomplete or ambiguous and that Intel followed it, but implemented something slightly different, or it could be that Intel's documentation explicitly warned about this incompatibility with AMD devices.
And most specs leave room for different implementations, either by design or because the people writing the spec forgot or did not bother to describe a case. For a famous example, various versions of the 6502 treated undocumented opcodes differently (http://visual6502.org/wiki/index.php?title=6502_Unsupported_...) And yes, there was code that made use of these.
I looked at the AMD and Intel specs to see that both use the CPUID instruction function 80000001H to set bit 11 in EDX if SYSRET is supported. Because both AMD and Intel 64-bit processors support the same general instruction set, software written to check only bit 11 could easily run into the scenario I described. Again, I do not know best practices in detecting CPU features, but I do know software developers often tend towards the least necessary to do the the job.
Processors almost never tell software anything; their manuals tell the programmers how the CPU behaves. There may be instructions that can inform code about you optional features of the instruction set, but the software has to run on the CPU before it can query the CPU for its capabilities, so it has to make some assumptions about the CPU (you could bootstrap things by having a smaller CPU boot the hardware and having that CPU query the 'real' one for its capabilities, but that just moves the problem to the smaller CPU)
So, in the end, the only way that software can know how the CPU behaves is because the programmers have read the CPU's documentation.
As to this specific case: I haven't bothered reading up on it, so I do not know what happened. It could be lack of documentation by Intel, it could be that AMDs documentation was incomplete or ambiguous and that Intel followed it, but implemented something slightly different, or it could be that Intel's documentation explicitly warned about this incompatibility with AMD devices.
And most specs leave room for different implementations, either by design or because the people writing the spec forgot or did not bother to describe a case. For a famous example, various versions of the 6502 treated undocumented opcodes differently (http://visual6502.org/wiki/index.php?title=6502_Unsupported_...) And yes, there was code that made use of these.