Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This will break in a lot of pages as the html will be different on each request in most pages although the content will be the same. Perhaps it would be better to compare visible text rather than the raw html.


Neat. Do you have a recommendation for a headless browser library that would simplify this? The difference between checking the raw HTML and visible HTML can be pretty big.


In client-side Javascript, document.documentElement.innerText yields only the rendered text, without markup, CSS or JavaScript source.


But the theory was that the source would be different, and only the rendered content would be the same at the sample place, right? How would you find the 'same place' without fully rendering the page?


One way is for the source to use cache control headers so you can just do a quick HEAD instead of a GET on the whole contents.


lxml provides a module to perform html diffs mainly looking at the content rather than markup. It's not perfect but most of the times works. http://lxml.de/api/lxml.html.diff-module.html

For Go I don't know any.

When I was working on a project like this another issue that I found regularly is that there are regions of the page(counters, advertising, etc.) that change all the time too but they are part of the content. For this I would make several requests to the page and find the regions that would change in every request and disregard them when doing the diff.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: