Open Source LinikedIn Scraper

[devildani]

Hi All,

I am planning to build a LinkedIn scraper, which would scrape LinkedIn profiles, company pages, groups, searches (sales nav and normal searches), likes, comments, etc, for free. I already a set piece of codebase that I have built out. I wanted to know if anyone would be interested in using an open-source LinkedIn scraper? Will this be good option?

[mdbmdb]

I would be interested. I think LinkedIn aggressively tries to shut down things like you are trying to build though

[devildani → mdbmdb]

Good to know. Can you share your email id? I will ping you as soon as I have the first version of application ready.

[greenpinia]

As if there was not enough recruiter spam mail.

[whodev]

This is why I private my social media accounts. There is already enough spam and I don't feel it needs to be any easier.

[noashavit]

There are services that do that, might not be open source but with free tiers like common room.

LinkedIn might not like you scraping the site as well, esp if you use sale navigator to get access to deeper info. Keep that in mind

[devildani → noashavit]

Yes, I am aware about this and I won't be using my account for extraction. Users will put their session cookies for data extraction.

[paulcole → devildani]

Great idea. I love a tool that unexpectedly gets my account banned.

[xorl → paulcole]

Facts, they will ban your LinkedIn account for scraping. Good luck!

[paulcole → xorl]

I work with someone in Spain and asked her to log into my Recruiter Lite account to review candidates. LinkedIn gave a pretty stern warning about sharing the account with anyone and that I’d be banned if it happened again.

I’m guessing that the same thing would happen if you started letting a scraper act on your behalf.

[brianjking → paulcole]

lol

[enjoyyourlife]

From https://www.linkedin.com/robots.txt it says to contact whitelist-crawl@linkedin.com

[KomoD → enjoyyourlife]

That's just for search engine crawlers, not random people who want to scrape all their data

[altdataseller]

Ironically, sometimes I wish that LinkedIn vs HiQ verdict went the other way and made scraping LinkedIn illegal

[devildani → altdataseller]

https://www.linkedin.com/feed/update/urn:li:share:7006059900...

[satvikpendem → altdataseller]

The judge ruled towards LinkedIn in the latest ruling of that case, which I suppose most don't know about due to it being an appeal.

[altdataseller → satvikpendem]

I’m not a lawyer but does that mean scraping LinkedIn is indeed illegal? Or is it still yet to be determined?

[devildani → altdataseller]

TLDR: The short answer is: it is legal to scrape LinkedIn properly. However, scraping LinkedIn a certain way could be illegal.

Scraping data from public LinkedIn profiles? Ok. Scraping data from LinkedIn profiles with fake logged-in accounts? Not ok.

[devildani → satvikpendem]

I did read about this. Not most know about this.

[vrepo54]

I would be interested. How do you plan on dealing with IP address bans?

[ldjkfkdsjnv]

Use an LLM to do it

[devildani → ldjkfkdsjnv]

I am trying this out, I haven't really written any code to build the application. Generating codes via LLMs.