No, I rate limit everything by default (per IP address, via a few nftables rules), until the user logs in, at which point I add the IP address to a whitelist. I also run SIP on non-default port and use SRV records to point the client to the right port. Helps with blind IP scans.
I don't really like the fail2ban approach.