On: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

[eli]

> CSS/Xpath are very fragile. You most likely will be changing them in the future.

Genuinely curious what the alternative is

[brilliantcode → eli]

I've been doing research on this but it's not clear whether this problem is a pain for enough number of businesses to justify further investments.

I often feel like web scraping is a commodity without understanding any of the inherent technological complexities and challenges.

Very discouraging field to be in, especially when people claim to have pain but are unwilling to pay very much for it or show appreciation for the effort that goes into it.

edit: thanks for the downvotes. perfect illustration of how innovation is punished and unrewarded in this field.

[skinnymuch → brilliantcode]

How many downvotes did you get? A few may have just been because your response is vague and doesn't say much for the question. But that doesn't mean a downvote should occur, of course.

Otherwise complaining about downvotes is no good either. Some will downvote because of that.

[deadbunny → brilliantcode]

> especially when people claim to have pain but are unwilling to pay very much for it or show appreciation for the effort that goes into it.

Welcome to pretty much every profession in the world.

[jlgaddis → brilliantcode]

FYI, I only downvoted you after you complained about downvotes.

[brokenmachine → jlgaddis]

I only downvoted because there was no alternative offered, just complaining about how underappreciated scraper creators are.

> perfect illustration of how innovation is punished

I see no innovation in your post.

The complaining about downvotes was just the icing on the cake, cementing the downvote.

[nikcub → eli]

for unstructured data applying NLP

the other alternative is parsing schema.org schemas or other markup