On: Nepenthes is a tarpit to catch AI web crawlers

[hartator]

Every not found pages that don’t return a 404 http header is basically an infinite trap.

It’s useless to do this though as all crawlers have a way to handle this. It’s very crawler 101.