
Analyzing One Million Robots.txt Files - foob
https://intoli.com/blog/analyzing-one-million-robots-txt-files/
======
PaulHoule
Where is the insight? You have a bunch of clusters, but what do you actually
do with the clusters once you have them?

