Distance from the CPU. And traditionally, RAM sits waaaaaay across the motherboard. If you put it on the CPU die, it’s much closer and much faster. Still slower than L1 and L2 cache, sure, but much faster than waiting on the fetch across the motherboard.
Look into Apple’s unified memory.