On: Is your SSH password revealed when you attempt to connect to the wrong server?

[kleim]

If someone has remote access to your computer then either way you are screwed. Using password or key file won't change anything.

If your laptop get stolen, then just remove the corresponding line in your authorized_keys file.