It's worse than that, they will block all agents and specifically make exceptions for maybe 2 search engines per country. Thereby strengthening existing players and weakening all incumbants.
Cloudflare for instance does allow indexing by Google and Bing I think by default, but does challenge other bots. Double check me I am speaking from memory.
Cloudflare for instance does allow indexing by Google and Bing I think by default, but does challenge other bots. Double check me I am speaking from memory.