
SpiderKeeper: Admin UI for scrapy/open source scrapinghub - r_singh
https://github.com/DormyMo/SpiderKeeper
======
oqulie
SpiderKeeper least received an update 2 years ago. Scrapyd last received an
update 9 months ago. Scrapy (the actual spider) is being actively maintained.

~~~
madc
Loads of not maintained open source projects make it to the front page
lately..

------
wp381640
This project seems like a more updated fork although it's also not maintained

[https://github.com/fliot/ScrapyKeeper](https://github.com/fliot/ScrapyKeeper)

------
dzink
Are these spiders able to fetch content that loads in segments after scroll
events?

~~~
jsmith99
Scrapy doesn't run javascript so it can't scrape client rendered content at
all. There are projects that combine scrapy with a headless browser or
javascript engine.

------
slightwinder
How does this compare with a general solution like Apache Airflow?

~~~
troymc
The following is a simplification but should suffice.

Airflow is for hooking together and managing a bunch of data sources, data-
processing nodes, and data sinks.

This project seems like a GUI for Scrapy, which is a web crawling framework,
i.e. one kind of data source.

------
r_singh
Anyone here using this in production (even for a side project)?

What are some of the challenges you've faced as opposed to just using
ScrapingHub's Scrapy Cloud service?

------
jatinshah
All such admin tools use scrapyd which is not maintained and not good enough
to be used in production.

~~~
gregsadetsky
What do you recommend using instead?

~~~
rozhok
One of the companies I've worked for developed in-house solution (with some
domain-specific stuff).

I've had to write some kind of admin tool for one of my clients.

And then I've used to start another project with scrapy and I feel like I need
to develop another one tool for scrapyd management :)

------
cs64org
I think it can faster than the normal then a basic function. ti increase the
case memory.

------
Pmop
Nice. I'm stealing this concept for my scraping engine.

