To add to this and to quote a friend who has more NDAs in regards to microarchitecture than I can count and thus shall remain nameless: "You can have a fast CPU or a secure CPU: Pick one". Pretty much everything a modern CPU does has side effects that are something that any sufficiently motivated attacker can find a way to use (most likely). While many are core specific (register rename, execution port usage for example), many are not (speculative execution, speculative loads). Side channels are a persnickety thing, and nearly impossible to fully account for.
Can you make a "Secure" CPU? In theory yes, but it won't be fast or as power efficient as it could in theory be. Because the things that allow those things are all possible side channels. This is why in theory the TPM in your machine is for those sorts of things (allegedly, they have their own side channels).
The harder question is "what is enough?" e.g. at what level does it not matter that much anymore? The answer based on the post above this is based on quite a lot of risk analysis and design considerations. These design decisions were the best balance of security and speed given the available information at the time.
Sure, can you build that theoretically perfect secure CPU? Yes. But, if you can't do anything that actually needs security on it because it's so slow; do you care?
This is also a fundamental property - if you can save time in some code/execution paths, but not in others (which is a very desirable attribute in most algorithms!), and that algorithm is doing something where knowing if it was able to go faster or slower has security implications (most any crypto algorithm, unless very carefully designed), then this is just the way it is - and has to be.
The way this has been trending is that in modern systems, we try to move as much of the ‘critical’ security information processing to known-slower-but-secure processing units.
But, for servers, in virtualized environments, or when someone hasn’t done the work to make that doable - we have these attacks.
Your friend is genuine in their interpretation, but there is definitely more to the discussion than the zero sum game they allude to. One can have both performance and security, but sometimes it boils down to clever and nuanced design, and careful analysis as you point out.
Can you make a "Secure" CPU? In theory yes, but it won't be fast or as power efficient as it could in theory be. Because the things that allow those things are all possible side channels. This is why in theory the TPM in your machine is for those sorts of things (allegedly, they have their own side channels).
The harder question is "what is enough?" e.g. at what level does it not matter that much anymore? The answer based on the post above this is based on quite a lot of risk analysis and design considerations. These design decisions were the best balance of security and speed given the available information at the time.
Sure, can you build that theoretically perfect secure CPU? Yes. But, if you can't do anything that actually needs security on it because it's so slow; do you care?