This actually blocks a lot more than just AI crawlers. You shouldn’t use this without reviewing it in detail so that you understand what you are actually blocking.
For instance, it includes ChatGPT-User. This is not a crawler. This is used when a ChatGPT user pastes a link in and asks ChatGPT about the contents of the page.
One of the entries is facebookexternalhit. When you share a link on Facebook, Threads, WhatsApp, etc., this is the user-agent Meta uses to fetch the OpenGraph metadata to display things like the title and thumbnail.
Skimming through the list, I see a bunch of things like this. Not every non-browser fetch is an AI crawler!
> the ability to read the whole content in the reader instead of having to go to the site. But that depends on how RSS/Atom exposes the content;
It rather depends on the amount of content the RSS _author_ includes in the RSS feed. There's nothing in the RSS/Atom protocol that prevents you from reading the entire article, but some website creators decide to truncate the feed content.
My RSS reader of choice, InoReader, has the option to download the original website which solves the problem. However, I have over 200 feeds and it's rare to find one without the entire content being included.
I've analyzed the palette with my own tool, a11y-contrast[1], and indeed the luminance is not uniform. I wrote [2] about why this might be a desired property of a color palette.
Some RSS readers are quite aggressive with their polling time. At the same time, a single request may serve thousands of users. However, you can check your logs and get the actual number of subscribers, at least for popular RSS readers: https://darekkay.com/blog/rss-subscriber-count/
I've been using XYplorer for 10 years now and it's fantastic. There are just so many features and quality of life improvements over the Windows explorer.
It's written by a single person, and they're very responsive. I've recently reported a bug with the search (the first that I've noticed in all those years), and it's been fixed within a day. I'm only worried about the bus factor a little.
https://github.com/ai-robots-txt/ai.robots.tx
reply