On: Cloudflare blocking my IP (2023)

[trekz]

Also note, Cloudflare blocks IPs of legitimate automated "good bots" by default, like RSS feed fetchers (1), but allows traffic to some of the most aggressive bots on the web, like Google bot and all sorts of AI bots (2).

(1) https://openrss.org/issue/144

(2) https://radar.cloudflare.com/traffic/verified-bots - GoogleBot, GPTBot, and AmazonBot are some of the first ones listed.

[theanonymousone → trekz]

Blocking RSS fetchers which are "bots" by design, intent, and definition, is really bizarre!

Seems the poor indy bots scraping price pages once a week are totally out of lock then.