Local privilege escalation is always bad because it means you're one malware payload or RCE away from being rooted and conscripted into someone's botnet (or worse). This isn't just a physical access concern.
What about without local privilege escalation? Is there no way for a malware payload or RCE to turn your computer into a botnet without root privileges?
There certainly is, but rooting a box lets you ensure that you stuff says in place. Once a box is rooted, its owner can never really be sure they have it clean without wiping and rebuilding it.