I got to meet him in his office and have dinner with him once. It was an unforgettable and hugely influential experience.
Two fun anecdotes that have never left me:
- He taught me that IPIs (inter-processor interrupts) are inherently and hugely expensive. Knowing this has helped me with architectural choices more times than I can count.
- He quoted (I think from someone else) a rebuttal to the idea that Physics is the reality and math is just theory. It goes something like: Math is the reality that physicists sometimes discover. Love it.
Let me see if I can remember what Arvind said. This was like 20 years ago so I won’t get it exactly right.
He was comparing to two CPUs sharing data with shared memory. In the shared memory case, each CPU stays on the happy path but sometimes has to do some cache coherence, which may stall the CPU, but that’s the worst that can happen, generally.
But if you try to send a message to another CPU, then you’re asking it to raise an interrupt. That’s not the happy path. CPU will have to stop everything to receive the interrupt and then divert execution to some interrupt gate. Arvind’s point was that every impl of this is going to be much much worse than the worst case of cache coherence.
Most processors are designed with very expensive task switching, but I don't think I'd call that "inherent". A processor could implement an interrupt as a forced branch 200 cycles in the future. Then it's the same speed as a function call with callee-save registers. That's still slower than letting cache hardware manage the whole thing in the background, but it should be as little as nanoseconds slower.
Sorry for the naive question - but what is an example of “send a message to another CPU”?
In high level programming languages, isn’t this usually over shared memory as well? Are there constructs to directly “message” a CPU? When might one use it?
There's a fairly recent study looking into the cost of raising an interrupt. TLDR: the cost on conventional systems is quite high; a significant part of the cost can be eliminated by using a custom kernel; even with a custom kernel, there remains a substantial cost.
To my knowledge, the remaining cost could be decreased to approximately the same cost as a branch mispredict, but getting there would require changes to the chip hardware and software stack.
> To my knowledge, the remaining cost could be decreased to approximately the same cost as a branch mispredict, but getting there would require changes to the chip hardware and software stack.
Do it even need to be a misprediction?
If you are completely focused on latency then flushing everything else makes sense. But I would think that if you continue execution for now and put a branch instruction into the queue you'd reduce the cost per interrupt even further.
Mathematical concepts are abstracted so to be beyond universal (ie, another universe will still have the same peano arithmetic). Our universe exhibits some subset of all mathematical properties
I got to meet him in his office and have dinner with him once. It was an unforgettable and hugely influential experience.
Two fun anecdotes that have never left me:
- He taught me that IPIs (inter-processor interrupts) are inherently and hugely expensive. Knowing this has helped me with architectural choices more times than I can count.
- He quoted (I think from someone else) a rebuttal to the idea that Physics is the reality and math is just theory. It goes something like: Math is the reality that physicists sometimes discover. Love it.