Recursive wget isn't going to pass Anubis. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Dwedit 3 days ago \| parent \| context \| favorite \| on: If you love a website, back it up Recursive wget isn't going to pass Anubis.
		help

vaylian 3 days ago [–]

This. Aggressive AI crawlers have ruined many parts of the web. I guess you have to ask the owner of the site nicely, if they would provide a complete copy of the site for your personal archive.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact