Not entirely true, there are many ways in which you can use a custom location and still take advantage of the binary cache. You can do it with chroot, file system namespaces, bind mounts and so on. There's also a nice user friendly tool that does exactly this [1].
[1]: https://github.com/DavHau/nix-portable