If you're already using key authentication, that doesn't add much security. A port is just a very short password shared by all users--- instead of logging in with "I'd like to login, please initiate public-key authentication", you now have to login with "I'd like to login, the magic number is 2222, and please initiate public-key authentication", which isn't really any more secure.
Call me an engineer, but I don't understand why anyone would dismiss security through obscurity just because it's theoretically useless. If you can reduce the lifetime expected value of successful attacks on your system, it should be considered a win regardless of how you do it. (Of course, relying solely on security through obscurity is a sure-fire way to greatly increase said EV.)