That is correct. But it is possible to design a system with short lived auth tokens/keys and frequent key rotation. I designed such a system at $oldjob for remote access (see [1]). Obviously there is always a risk, and there are always syseng/ops people with access. That is correct.
