That alone allows you to avoid most attackers trying to guess valid server ips.
Now to put a honey pot on port 22.
I can't emphasize this one enough. Unless you need to login from a lot of different machines, there really isn't any excuse not to do this. It also has the bonus of making logins really easy since you don't have to type a password.
Also, yes, among with changing the port the only way that should be possible to get in is through 'keys.