Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
qwertox
on Aug 7, 2023
|
parent
|
context
|
favorite
| on:
GPTBot – OpenAI’s Web Crawler
I wish every company which crawls had such a page. Maybe a centralized directory of crawlers would be a good for the internet.
Gasp0de
on Aug 7, 2023
[–]
I mean, technically you can just disallow all robots in your robots.txt and only allow those of which you are aware and you like?
Join us for
AI Startup School
this June 16-17 in San Francisco!
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
