Had the exact same thought some time ago now, even proposed it internally at my company. What makes me doubt this will work eventually is that scraping has been going on forever now and yet no standard has been accepted (as you noted robots.txt serves a different purpose, should have been called indexation.txt)
I was discussing this some time ago with a friend, I get the feeling that error messages these days try too much to apologize instead of providing the user with hints and clear explanations to solve the issue. It's all about emoji faces crying "boohoo there was an error" with a childish tone.
Can't wait to see AI avatars crying in front of you for failing to summarize an article.
I want to know who had the stupid idea to start the UI trend of making error messages tell you absolutely nothing (e.g. "There was an error. Enjoy being helpless.").
I have started working on that idea for a little more than a month now and it's been constantly amazing to feel how fast I can build and deploy my ideas on the cloudflare stack (using Pages, Workers and D1) for now.
For now it does not have millions of websites available to search through but still 250K and growing fast (crawling constantly through a cloudflare worker)
Good to see you're transparent in your profile about working at Cloudflare. Look up https://twitter.com/RozenMD/ . He also works at Cloudflare and on his side-hussle he's using Cloudflare workers to crawl websites.
Your content really helps me move forward with my project, thank you for that Max. Being back on twitter after six months, I get the same feeling. Cool and exciting to get retweeted and to see the followers count ramp up. But it does not necessarily translate into dollars at the end of the day. ROI is hard to see, at my level. Reddit feels more like an investment due to the SEO and hyper targeted niche audience on some subs.
Interesting! At the same time, HN current design already looks kind of brutalist to me, it's crisp and sharp without fluff. That's what makes it intemporal and easily readable on every devices.
I have learned something today, thank you for that! The pitch is clear and the fact that you put so much work in open sourcing it is really impressive.
