Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I'm wondering (a) if people are going to switch to screen scraping as a way to avoid the new API restrictions, and (b) how much Twitter could do to thwart such attempts.

My (uneducated) guess is that it might play out like other DRM wars. Twitter could come up with new ways to scramble their HTML/CSS/JS to make it harder to automatically scrape... until scrapers evolved to deciper the new patterns, ad infinitum.



If they're willing to flip off search engines and do all page rendering in JS, there's no limit to what they could do to make a scraper's life hell.

It'd be relatively simple to generate non-sensible fonts (where code points don't map as expected) where the browser still renders legible tweets but scrapers get unintelligible gibberish. The special font would essentially be a one-time pad, generate new font mappings every few requests and your HTML is essentially uncrackable to scrapers that don't have the horsepower to OCR every tweet (at which point you may as well pay for Gnip anyway).


True, but it would be much simpler to hack the Twitter native applications to do the dirty work for you, depending on your purposes.

That, or just use the oauth_secret from the official Twitter applications; those were cracked ages ago, and even if they changed them (thereby breaking all their old applications), they're super-easy to crack again.


Hardening the webpage would still decimate the amount of people with the skill/motivation to pry out tweets. HTML scraping is much more accessible than poking around native, source-less apps. (And if it was still a problem, the same tactics as above would work equally well in native apps.)


>If they're willing to flip off search engines and do all page rendering in JS

Isn't that a pretty big if?




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: