Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This is a great resource to at least figure out all the LLMs out there and block them. I already updated my robots.txt file. Of course, that is not sufficient, but at least it's a start and hopefully the blocking can get more sophisticated as time goes on.


It looks like the opposite. It is a way to make your site easier to parse for LLMs.


It is, but you can use it as a list of targets for blocking.


it's not "productive", of course, but i don't see any issue with expressing this opinion whatsoever. and i say this being about as starry-eyed a techno-llm-utopian-esque dreamer as they come... sure, the "google" version of LLMs paving over industry has already crossed the rubicon, but everyone should have to reckon the value that they are truly providing not just for consumers but for producers as well... and no one should be offended by showing up in someone's robots.txt... just as i'm sure this commenter is realistic enough to know and understand that putting entries in one's robots.txt is nothing more than a principled, aspirational statement about how the world should be, rather than any sort of real technological impediment.

(but we'll just ignore the obvious irony in that end bit about detection of bots getting smarter... wonder where all this "intelligence" will come from? probably not some natural source, but possibly some sort of... Antinatural Intelligence?)




Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: