I started to think differently about web search for agents, instead of traditional index, I would prefer to have an agentic WALKER that literally walks through pages and finds out whats needed, even if seeded with some initial web search.
I needed to find a word on a wayback archived website. Could not. So I scraped the waybackmachine pages to .md… and found it that way… sorry Archive.org