Depending on the service, you may not want it to be indexed anyway (IE login pages) or, if you do, most commercial crawlers have methods to allow site owners to verify that the traffic is a legitimate crawler and not an impersonator, see:
That wouldn't work if a sysadmin implemented the rDNS lookup mentioned in TFA. The PTR specifically must end in `.googlebot.com`, not `.googleusercontent.com`.
https://developers.google.com/crawling/docs/crawlers-fetcher...
(Though restricting the Google UA to only legitimate Google IPs seemingly would break Brave's crawler, though that's their own fault: https://safe.search.brave.com/help/brave-search-crawler/)
https://duckduckgo.com/duckduckgo-help-pages/results/duckduc...
https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...
https://kagi.com/bot