The point worth note is that it's not an archive of downloaded pages, it's data for the Blekko equivalent of PageRank (i.e. computed relationships, not just the pages). To generate this independently would not only require access to a large crawl, but also robust code and most probably a large cluster to compute it in reasonable time, not to mention legal advice to avoid stepping on Google (et al) patents.
If only certain other companies weren't so precious about their publicly derived data. Fabulous donation.
> not to mention legal advice to avoid stepping on Google (et al) patents.
You may be right, you may not be right, but if this is the equivalent of PageRanked data then you may not be in the clear to use this as is. After all if 'PageRank' went into producing it buying it and using it makes you the beneficiary of patent infringement.
Personally I'd say so-su-mi, but I still think that it should be noted that the fact that someone else did the infringing does not put you automatically in the clear when using the end product.
blekko doesn't compute PageRank, and we don't compute anything similar to it, either. It's highly gamed and less useful than you might think. (The academic equivalent of PageRank for research papers is highly gamed, too, by citation clubs...)
By the way, the original PageRank patent is owned and licensed by Stanford University, not by Google.
Can you tell us a little more about what the 'ranking metadata' is, as there's not much to go on from the announcement. It's also not clear whether the data is available only for Common Crawl's operational purposes, or whether it's intended to become an integral part of the public data set.
The ranking metadata consists of: domain ranks, url ranks, and booleans for whether blekko considers the domain or url to be webspam or porn. This list will expand in the future.
The data is currently available for Common Crawl's operational purposes, and is eventually going to be part of Common Crawl's public dataset. We're currently ironing out a useful format for making it efficiently accessible, compatible with some other metadata which Common Crawl is planning on making available.
This is great, but I'm confused by the part about avoiding porn:
> Common Crawl will use blekko’s metadata to improve its crawl quality, while avoiding webspam, porn, and the influence of excessive SEO (search engine optimization)
Why avoid porn? Millions of people deliberately search for porn on the internet every day. It's hardly less worthy of crawling than any other content. The next sentence goes on to suggest that porn is not "useful to humans", which is obviously false.
If Common Crawl is indeed filtering out content they determine to be pornographic, I hope they are taking care not to also remove information on sexual and relationship health and LGBT rights, which are often collateral damage of porn-blocking systems. And it would be nice to see an open acknowledgement that filtering is going on - I couldn't find any references to this at commoncrawl.org.
I work at Blekko and am the primary engineer working on our porn tagger. We include LGBT, reproductive/sexual health, breast cancer, bands like "Pussycat Riot," etc in our training set to make sure these sites do not get hidden from our search results.
We do not have anything against porn. However, when people are not searching for porn, showing them porn results makes for a bad search experience. So identifying porn, and only showing porn on relevant porn results is vitally important to search quality.
Jacques, the porn is there, its just identified as such. Whether or not it is included in results is a function on the query.
One of the funny things about language is that there is always a 'pun' or an innuendo which can trigger a hit on a porn site, however if most of what you're looking for isn't porn then the web site has to assume you are not looking for porn and avoid some NSFW link from surfacing into your search results. You could always explicitly ask for it with /porn but then that is a clear signal of what you are looking for.
Part of the crawl data includes an indication as to whether or not the ranker thought the document was 'porn' or 'not porn', so if you're selecting things to return you can ignore that bit, mixing porn with non-porn when someone searches for 'beavers' you get a wider variety of results than you would if you were assuming you meant the furry critters which chew on trees or sports teams and limiting results to those documents.
I am from Common Crawl. We don't filter for porn. A corpus of web data needs to include porn or it wouldn't be a representative sample of the web ;) We do want to enrich our sample of the web with high-value sites and that is where the blekko data will be so incredibly valuable.
I really appreciate your mention of LGBT and sexual health sites being collateral damage - we need to draw more attention to that problem. I would love to see someone work with Common Crawl to improve methods of distinguishing.
This is going to be invaluable for information retrieval researchers.
Google, MSR, and Yahoo! have an edge on research over universities because of the large amount of data they collect from the users; all the other institutions are left with either small-size benchmark datasets or synthetic data, which are usually not representative of the actual usage scenarios. I myself had to synthetize a query log from the Wikipedia request logs to test some of my data structures on large-scale data.
I expect to see a huge number of papers which will use these data in their experiments in the immediate future. Thanks, Blekko!
This is our first donation. We have a lot more we plan on giving, but for user queries, for example, the privacy issues are a lot more difficult to work through. We have no interest in being the next privacy scandal.
Common Crawl has a really neat mission, as there isn't a whole lot of free and open data out in the world right now and they're trying to change that. With this donation it looks like their commons will be augmented with some great stuff and that can only mean awesome things.
For me, blekko doesn't have to be "so much better" to make me use it over Google, it just has to be just as good or slightly better because I respect and support blekko's philosophy. If two products were of equal quality, wouldn't you rather use the one made by a company that shares your values?