

The Science of Crawl, Part 2: Content Freshness - jisaacso
http://blog.urx.com/urx-blog/2014/10/23/the-science-of-crawl-part-2-content-freshness

======
dennybritz
Nice post.

I only skimmed the post, but I believe you are assuming the utility of all
pages to be equal. By "utility" I mean the value of information contained on
the page relative to what your business is trying to achieve (not the
organization's utility as you define it in the blog post). However, in
practice, aren't information certain pages of much greater value to the
business than others? For example, finding a new front page article on the NYT
website could be more valuable than detecting 50 new Hacker News submissions.
However, the NYT page would exhibit less divergence than the HN page.

