Hi! I also build job scrapers (http://nolatechjobs.leesome.com) and have a question about the dependency on redis... why not just generate a time-stamp within the ruby scraper (which you already do) and store it within the resultant json itself? Then you can use string comparison to see if the next scrape has returned a more accurate time-stamp and just overwrite that part of the string.
Basically my question is why couldn't you build the whole scraper with just flat files and no database/key-value store at all?
That would definitely be an improvement. When I originally wrote this in 2011, I had just a few hours, so I went with the quickest solution. For little personal side projects like this I find that Redis makes a great persistent hash map for storing bits of data between runs of a script.
Basically my question is why couldn't you build the whole scraper with just flat files and no database/key-value store at all?