You can remember a small set of resolved hosts for this purpose, not unlike remembering DNS servers or CAs to trust. A quick search online also says it's possible to issue a cert to a public IP address, so you can also do HTTPS to a numbered IP instead of a host name.