Yes but you can detect that. The enclave can sign a statement that you could verify with a public key for that enclave. So if you replace the hardware there is no way to do that anymore as you cannot extract the private key from the original enclave.
BTW, Google uses hardware-based security on all of their servers:
https://www.theregister.co.uk/2017/01/16/google_reveals_its_...