1. The security implications. Without a passphrase, anyone who manages to snare a copy of your private key can use it to access any servers where you've authorized it with no further credentials needed. Not good. Using a passphrase turns SSH auth into a poor man's two factor system: what you have (the key file), and what you know (the passphrase).
2. It breaks agent-forwarding, which is the system that gives you the ability to SSH onwards from the remote machine to other machines without (again) being asked for your password. This has practical implications when using SSH to drive things like application deployment over git, when you have to ssh onward from the server to your git repository.
Doing things this way sidesteps the awkward use of deploy keys, and is actually easier to set up and use as well as being more secure. With agent forwarding, there's no need to keep your private key anywhere except your 'home' machine, and yet you get the same benefits your would if you had it everywhere.
The annoyance of having to type in your passphrase every time is greatly lessened by the use of a key agent on your 'home' machine. OS X has provided a key agent since 10.5 I believe, and stores the passphrase in your Keychain (you know you're doing things right if an OS X dialog box prompts you for your passphrase when running SSH in the terminal). Gnome has one built in as well, and ssh-agent is always around for people who want to roll their own.
If there's interest, I'll write up a blog post about how to get this stuff set up. It's not really easily discoverable stuff, but it _is_ really simple and useful once you get it up and running.
If you're like me, however, you'll use both keychain and ControlMaster (for the increased negotiation speed) :)
I've been using gpg-agent with a smartcard for ssh authentication for the past six months or so. It's the way, the truth, and the light.
Real two-factor authentication. The key can't be compromised without trying to physically take apart the chip on the card and somehow access the internals. Three wrong password attempts and the thing locks. Three wrong attempts with the admin unlock code and the card self-destructs. So even if I lose the card or someone jacks it, it can't be brute forced.
(Yes I don't need that level of security, and I'm not even being paranoid; it's just neat. Makes me feel like James Bond or Batman or some shit like that every time I ssh somewhere. Yes I'm that lame... )
And since it's still using a standard RSA key for ssh authentication, I don't need to install anything special like experimental pam modules, on the host machines. Just copy the public key into authorized_keys. That's nice since I don't have full admin rights on a lot of my host machines.
Basically you either get a card and reader:
Or get an all-in-one cryptostick:
And setup your gpg keys on there, either by generating them directly on the card or transferring existing keys. These are simple commands documented elsewhere. In addition to the normal signing and encryption keys, you also generate an authentication key.
Then 'ssh-add -L' will spit out your public key in ssh format to copy on the host machines as usual.
After that you just make sure that you'll use gpg-agent instead of ssh-agent. The man page for gpg-agent shows you what you'll want to add to .bashrc.
Then when you ssh into a machine, gpg-agent will take over, pop up a little dialog called pinentry, you enter your code, and you're good. When you go to lunch, remove card, and ssh authentication with that key no longer works.
How does it break agent forwarding?
I haven't tried but don't see why that should require a password on the private key.
ssh (the command) can access the material in your private keys in one of two ways, either by direct file access itself (in which case it will always ask you for a passphrase if there is one protecting the key in question), or else via an intermediating ssh-agent process. Roughly speaking, ssh does something like the following when looking for key material:
if private key file exists and has no passphrase on it
use it directly (and implicitly do not forward the agent)
else if ssh-agent is running
ask the agent for the key (and forward the agent)
ask the user for a passphrase ourselves (and implicitly do not forward the agent)
I haven't taken the time to figure out key agent -- for shame! -- although I would be interested in a write-up.