Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Get firebug, or use developer tools in chrome and look at the requests the browser is making when you load the page, and when you scroll down and it loads more data. Then you just implement that in whatever language you are scraping with. I usually use pythons requests module.. That's about as easy as it gets for something like this.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: