Whether this is worth the hassle is left to the reader: if you have passwords disabled and only use keys it really shouldn’t matter.
I've run ssh on non-standard ports for over 20 years, and my auth.log is gets a hundred knocks an hour - and mind you, they all return "no key".
It's just life, and it will continue to get worse. Secure your server and ignore it.
Whether this is worth the hassle is left to the reader: if you have passwords disabled and only use keys it really shouldn’t matter.