Making a network request from a script is not abuse, consuming excess bandwidth is. Scrapers already spoof browser reputation and cycle IPs while abusing bandwidth. Recently I wanted to convert the results of an Autotrader filter into a table so I could supplement it with data they don’t track (towing capacity). It was two pages of results, but requesting it from a script was aggressively blocked by browser fingerprinting. I had to port it to JS and run it in my browser console manually to get the data out, wasting my time.
CDN security SWE here. I have functionally zero interest in what individual people do in terms of automation. I automated my gym's class signup forum during COVID so I get the validity of the use case. You want to work around the mitigations to do the same? Go with my blessings.
The killer is that everything that works for individuals trying to get through the day and make the web a bit smoother is immediately used by industrial crawlers strip mining the Internet, and you can't block one without blocking the other. A future where the web is only accessible via device attestation is extremely dystopian but so is an Internet where every drop of content goes into an LLM training set.
Things like this is why all of the worthwhile content is going to drain into balkanized spaces, and we're all the poorer for it.
It’s not that dystopian if you are given the choice to trade your attestation for more valuable service. We have been paying for the web by letting advertisers correlate our behaviors together for a couple decades now.
Abusive crawlers use residential proxies to distribute crawl traffic between thousands of IPs, so that one agent that's making five requests blends in with the massive crawl. There is signal but there are ML inference engineers tuning their traffic to blend in, which requires that mitigations must get more sensitive.
At the end of the day, the defensive side has an obligation to protect customers and well behaved agents are the first thing that gets swept up along with the bad actors.
Why did you need it from a script? Right click -> Save Page As... is still a thing and works quite well in my experience, even with complex React SPAs.
It's not just excess bandwidth. It's cpu cycles, poor site coding, database issues. Like I said, not everything is backed by a huge company that has the money/hardware/engineering resources to absorb the load.
That’s the point. For every legitimate though ultimately unimportant use case like yours there are a thousand others who would scrape the results 86,400 times a day to either try to sell or mirror and plaster with their own ads.
Ideally, there would be a better way for web hosts to block bandwidth-hogging, but not block bots. I don't think anyone cares if some automated user-agent requests a 1KB file from their web site. They care that automated systems are sucking down TBs per day, all day, every day.
Making people actually read content doesn't strike me as cruel. Tedious, sure, but not cruel. What is cruel is seeing all of our collective individuality and artistic expression jammed into an LLM, sold back to us by the token, and forced into our lives by an economy increasingly skewed towards holders of capital.
