Hacker News new | past | comments | ask | show | jobs | submit login
From 0 to 20B: How we built crawler hints at Cloudflare (cloudflare.com)
3 points by rajesh_bh on Dec 16, 2021 | hide | past | favorite | 1 comment



Which sounds fantastic except they actively block every crawler out there apart from the big ones like Googlebot. Cloudflare is cancer.

> Just a moment...

> This process is automatic. Your browser will redirect to your requested content shortly.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: