On: Show HN: Spider Pro – easy and cheap way to scrape...

[sillydinosaur]

https://www.crummy.com/software/BeautifulSoup/bs4/doc/

I mean...

soup.find(id="link3") # <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

Is this hard?

[Cyberdog → sillydinosaur]

For HN's audience, no, that's not hard. But this product is clearly not only for us, and you've also cherry-picked a very simple example.

There's also the aspect of downloading the page in the first place and dealing with things like authentication and bot detection which a product like this helps solve.

I personally don't have a use for this product right now, but I won't be so bold as to say I'll never find a case where using it wouldn't be easier or more cost-effective than hacking up my own solution.

[slig → sillydinosaur]

Good, now do that on a page that uses JS and authentication. Trivial on this extension, not so much using BS.

[penagwin → sillydinosaur]

If you show that to non-programmers they'll take one look at that and understand that about as well as egyptian hieroglyphics.

"Soup? Python? What's an 'id'? class? href? I just want to select the title!"