That is correct. But it is possible to design a system with short lived auth tokens/keys and frequent key rotation. I designed such a system at $oldjob for remote access (see [1]). Obviously there is always a risk, and there are always syseng/ops people with access. That is correct.
[1] https://blog.heckel.io/2019/11/19/providing-remote-access-to...