I believe s/he meant if the box you are using to log in is compromised, then all boxes your box has access to (by way of the private key it has in its file system) are compromised, whereas it seems you interpreted the comment to be implying that the host to which you are logging in (which only has the public key in its file system) has been compromised.
The point is that the computer on which you type becomes a lynch-pin.
Yes, if you have a computer with a private key that is available to the internet, and it gets knocked over, you're dead. Just as you would be if he knocked it over and installed a keylogger and you were using passwords. Cracked is cracked.
Assuming you are not running services on your main workstation, which is not that unreasonable (Ubuntu workstations are installed with 0 services available to the local network; my Mac has 0 services available by default), then you can get knocked over but it will probably happen because of an browser bug or something like that, i.e. not an active attack.
The whole point of keys is that knocking over one of the intermediate nodes in the network no longer gives you control over everything. Just because you rooted my server doesn't mean you can automatically log in to all my other machines, even if I'm using keys. Keys are better than passwords.
The point is that the computer on which you type becomes a lynch-pin.