The big problem with ssh keys is not being able to enforce ssh key passphrases on users. From the server perspective, you have no idea if the user has set up a passphrase. There are security standards which mandate certain kinds of passwords (complexity) and are silent on asymmetric keys, so you couldn't use keys in those environments.
The old solution was to do some post-login hack to require a password as well (e.g. to su), or do a VPN (which could have multiple forms of auth) and then ssh with keys after that, but the newest ssh (and I believe commercial ssh for a long time) now supports requiring multiple authentications per login, so you can do ssh key plus passphrase.
There are also DLP/etc. reasons why ssh can be problematic in some environments (i.e. where you're required to log/analyze actions taken by users, particularly admin users). The solution there is to use a bastion host and ssh in and then ssh out, with the user account locked down to log. SSH Communications (the commercial ssh people) have an interesting ssh MITM box which essentially does what all the SSL x509 MITM CA things do.
SSH keys do not satisfy a "fail closed" security model: they're there unless you explicitly remove them (and keep them removed). Certificates, tickets, and other expiring credentials eventually go away and lock users out unless they're explicitly renewed.
People often consider ssh keys to be a panacea and to think of passwords as outdated
I've also been using Duo Security as my new 2FA solution and I like it a lot (it also has support for using Yubikeys to provide your OTP.)
Every single auditor I've worked with has given a pass when they saw we were using keys instead of passwords. Do you also fail audits because your RSA token is only 10-12 digits?
You could, I suppose, demand users upload their private key (shudder) to the organization so you can check if it's encrypted; this has its own flaws (primarily that you've just taught your users to be giving with their private key), and you still can't enforce passphrase strength well.
I don't have a particularly high opinion of most security audit standards, though.