Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Not the same kind of scraping, but does anyone have thoughts/resources/best practices for doing link previews (like Twitter/iMessage/Facebook)?


You shouldn‘t really need to do any scraping tricks to get that, because it‘s data the websites (usually) want to give to bots. Or are people getting bot block screens from Cloudflare et all for that basic action these days?

It should be a matter of a simple GET request to fetch plain html and parse the OpenGraph meta tags out if that. There are many open source libraries to do that for you depending on your language.

If bot blocks really are a problem, a SaaS solution like Microlink could probably do it for you.


Bot blocks are definitely an issue for certain sites, I've implemented it that way currently.

Microlink is a good tip, thanks!




Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: