
Search all Craigslist cities for a remote job - chovy
https://remotejobs.today/
======
gerpsh
Dude, you're making ScraperAPI requests (with _your_ API key in plain text as
a parameter) directly from the client app...this thing is a nice idea but it'd
probably be a good idea to clean it up and repost.

~~~
criddell
I thought these multi-region searches were against Craigslist's terms of
service.

~~~
smt88
Craigslist is famously aggressive about scraping, too. While helping with some
university research, I found that their anti-scraping defenses were really
difficult to get around.

They've also been litigious about this in the past, so this project would
undoubtedly get a cease-and-desist if it ever became a blip on their radar.

~~~
Reedx
> I found that their anti-scraping defenses were really difficult to get
> around

What sort of defenses are they using?

~~~
smt88
They had fantastic detection of headless Chrome and curl requests, a thorough
IP blacklist, aggressive rate limiting, and possibly some JS stuff.

------
point78
Craigslist will send you a take down notice in a week, this is explicitly
against their terms. I'd advise taking it down before a lawsuit.

~~~
ravenstine
That's basically what happened to Padmapper, sadly.

With the amount of scams causing a decline of their platform, I'd think that
Craigslist would want other sources of traffic. I know that most sites have a
policy against scraping/indexing, but Craigslist has their head up their own
ass.

~~~
rdiddly
I feel like anybody who quibbles about scraping has their head up there. It's
like, are you on the internet or not? Yes? Then you are on the internet.

~~~
biztos
Maybe timeliness is the important thing to Craigslist.

If you scrape all the jobs data as soon as it shows up, and repost it to your
site, then maybe employers think you're the one they should be paying, based
on prospects coming in your site and not CL.

I would assume that matters more than if you scraped older data or got older
data by scraping it more slowly.

------
tomca32
Cool idea, but the implementation is kinda crazy. Good enough to just test the
idea at home on your own machine, but pretty risky to expose it to the world
and let a ton of other people go nuts making bazzilions of requests.

Also, exposed API key in the request.

------
femto113
I'm kinda curious how telecommuting to this job actually works.
[https://detroit.craigslist.org/okl/lab/d/asphalt-
laborers/68...](https://detroit.craigslist.org/okl/lab/d/asphalt-
laborers/6848551616.html)

~~~
rxhernandez
Robots probably.

------
whalesalad
Looks like we're performing a DDoS against
[https://api.scraperapi.com](https://api.scraperapi.com) (the underlying data
source) at this point.

------
rkho
You should really consider caching results and only fetching them again after
a certain amount of time has passed since the last time you scraped them. This
way you're not blowing through your ScraperAPI limit.

------
OrgNet
Play with dog remotely?
[https://i.imgur.com/OebiBWr.png](https://i.imgur.com/OebiBWr.png)

~~~
giancarlostoro
Can you stack up on those and make 5 grand a month?

------
yubiox
Might want to take this out:

console.log(items);

~~~
giancarlostoro
I usually set those to 'debug' instead, so I can filter them back into my JS
console:

console.debug(items);

------
dhritzkiv
The API it's calling seems to be overwhelmed at the time of this comment. The
console is full of 429 (too many request) errors

~~~
chovy
should be good now. i upgraded my plan.

------
worg
It'd be nice to have a loading indicator.

~~~
x2f10
Thank you for this comment. I considered the webpage dead.

------
noitsnot
Wow, I stopped it at 400 requests. I'm not sure how the site hasn't been shut
down at this point.

------
timsayshey
This is an amazing project! Probably one of the most useful ways to actually
find quality job listings. If only Craigslist would offer site-wide search
natively.

Btw, you should open-source this on GitHub. You should also include a
crypto/paypal donation option.

------
mothsonasloth
Its cool but the main problem I am facing with searching for remote jobs. Is
that most remote US jobs are not technically remote. They are for US citizens
only, so I can't work on it from Europe.

------
codesternews
Hmmm.

"You're sending requests a bit too fast! Please slow down your requests, or
contact support@scraperapi.com with any questions."

------
momofarm
what I see the freelance job is playing with dog..... should there be a
playing with cat job _lol_

~~~
chovy
thosse freelance dog walkers are huge spammers. I've already removed duplicate
titles across cities and you still see a ton of them.

------
skizm
Can you just search Google for “<term> remote site:craigslist.org” for similar
results?

------
chovy
Just enter your search phrase like "node.js" or "javascript"

------
ada1981
All you critics are missing the obvious blue ocean market in remote dog
entertainment services that clearly is begging to be served.

------
starpilot
nice it doesn't work.

~~~
anonytrary
I, too, entered a search query, pressed go, and witnessed absolutely nothing
happen. Turns out, you just have to wait a bit. It scrapes Craigslist in real
time from the client, so you have to wait a bit for results to pop up. It'd be
nice if they had a loading/progress indicator.

~~~
chovy
I'm working on an indicator next.

~~~
point78
Fill us in on your thoughts for not believing you will get sued by craigslist
after multiple comments in the thread.

~~~
sithlord
would this not fall into a similar category of happened to linkedin? The data
is publicly facing, and not behind a paywall or a password, therefore,
basically fair-use of internet?

ianal, nor do I know much about the subject, but seems like if its not at a
minimum password protected information, it is pretty fair-use.

------
fidla
hmm...doesn't work in chrome

------
diminoten
Spamming 429s in console, oops!

------
okay3637
i need a job

