
Show HN: Sleeping Spider – Web page change monitor - linklet
https://sleepingspider.com/
======
TheDong
Does this site respect robots.txt?

Does it render javascript content (so e.g. <div ng-view> angular sites work)?

Does it exclude or include comments... if I watch a blogpost with comments
will every new comment cause an alert or only blogpost changes?

Will I get a notification if the page briefly 500 errors when you hit it, or
does it have to get a 200 before it reports a change?

Will I get a notification if a page disappears entirely?

Does it scrape only once per day or more often and alert only once a day?

What user-agent does it scrape from on the off-chance someone wants to block
it?

Does it diff binary data.. for example, if I ask it to watch an image url will
it hash the image and notify me when it gets updated?

Does it auto-detect pages with rss feeds or pubsub feeds and subscribe to
them?

Does it understand twitter so if I watch a user, it only alerts me of tweets,
not e.g. a change in the number of followers?

Does it have special code for any website whatsoever?

Is the webpage parsing/diffing library / algorithm open source somewhere?

Is there a monitization plan, and if not how can I trust it won't vanish?

Is there a limit on the number of sites I can watch?

I like the simplistic design, but it would be nice if there were more
information.

~~~
mattmanser
Go write your own then.

I find this a bizarre response to a simple and easy to understand tool.

~~~
TheDong
If it's easy to understand then I guess you can answer my questions?

I don't think "go write your own" is a legitimate response to my questions (if
that's ever a legitimate response).

~~~
baruch
I'm using websec as an existing tool to watch website changes. It's not
perfect but it's good enough.

For a while I had a webapp to do a better job but it never got any traction,
most likely because I had no idea how to get traction. It's an interesting
exercise to write such a tool though.

------
aw3c2
If you prefer privacy and control over your own tools, there are the FLOSS
[http://specto.sourceforge.net/](http://specto.sourceforge.net/) or the
free(as in no cost)ware
[http://www.cmcode.co.uk/webmon/](http://www.cmcode.co.uk/webmon/) for Windows

~~~
nousougi
I too used webmon until recently, but got tired of monitoring the monitoring
tool on my desktop. Now I use ChangeDetect -
[http://www.changedetect.com/](http://www.changedetect.com/) Despite what is
said on the homepage, it is NOT free. Good luck...

------
netman21
In the most amazing coincidence the first two pages I submitted had the same
digest: 90c0568! One was the front page of a news site, the other was the
management page of a company I follow. The mind boggles. Or, they have a bug.

edit: as did the 3rd. And 0 bytes returned so probably being blocked. On a
black list already?

~~~
nacs
Getting the same digest (90c0568) with no response code and a 0 byte response
size so looks like something is broken (the crawler maybe).

~~~
sleepingspider
Problem fixed. All failed probes retried. Thanks!

------
lucasjans
Adding watch results in a white screen of nothing, no new watches added. I'm
trying to monitor the Nexus 6 page :D

[https://play.google.com/store/devices/details/Nexus_6_32GB_C...](https://play.google.com/store/devices/details/Nexus_6_32GB_Cloud_White?id=nexus_6_white_32gb)

For my use case, anything more than an hour delay wouldn't be practical.
Ideally it would be up to the minute, but I'd imagine that would cause other
issues for a service like this. In the past I just used a Chrome Extension.
But living in Vietnam I have to make sure my VPN is up in order to see the
North American content.

~~~
sleepingspider
The probe received a 403 forbidden error from google. I realized that google
store forbid crawls. So a simple http request won't work.

I was not aware of this situation before. I will see what I can do.

------
sleepingspider
> Does it understand twitter so if I watch a user, it only alerts me of
> tweets, not e.g. a change in the number of followers?

What do you think if I provide a numeric filter option to watches, which
filters out numbers on the page before comparing if turned on.

------
sleepingspider
> Does it exclude or include comments... if I watch a blogpost with comments
> will every new comment cause an alert or only blogpost changes?

I don't have a good solution for this problem yet. Let the user specify a css
selector or xpath?

------
watson
Is it only for HTML, or will it alert on changes to JavaScript files for
instance?

~~~
sleepingspider
It's only for HTML. But if you have a need for JavaScript files, just let me
know, it's technically possible, I just don't understand the use case yet.

------
girishso
I just built "Page Poller" for the Rails Rumble 2014, same concept!
[http://pagepoller.r14.railsrumble.com/](http://pagepoller.r14.railsrumble.com/)

------
ideaoverload
Looks very similiar to
[https://followthatpage.com](https://followthatpage.com) . Worked for me
pretty well for non JS pages for last 3 years.

------
sleepingspider
> Does it diff binary data.. for example, if I ask it to watch an image url
> will it hash the image and notify me when it gets updated?

Can you elaborate in what case you might want to do this?

------
akx
Typo in the footer: "Sleeping Sider"

~~~
sleepingspider
Thank you. Fixed.

~~~
etw
What's your twitter id. You're trending on PH.

~~~
sleepingspider
The product doesn't have a twitter as the name was occupied by others. :)

Personally I don't use twitter neither. Email me if you need to contact.

~~~
etw
If you had a personal twitter account, I could have granted you commenting
access on Product Hunt. That's why I contacted you.

------
callesgg
What is regarded as a change?

~~~
sleepingspider
Text content changes. For example the site corrected a typo or published a new
post.

Change <h1> to <h2> will not trigger an alert.

------
seivui
Which timezone does "8 o'clock in the morning" consider?

~~~
sleepingspider
It will try to detect your time zone with JavaScript. If that doesn't work, it
falls back to Los Angeles and the user may contact the support to update this
setting.

------
sleepingspider
Just added HTTP authentication support.

------
zapt02
Nice branding!

------
_asciiker_
simple, clean and useful. I'm trying it out. Good job!

