Hacker Newsnew | comments | show | ask | jobs | submitlogin

They serve their robots.txt as a "application/octet-stream" not "text/plain". Might be confusing some search engines?

Also according to Internet Archive their robots.txt has changed in the last months.

This was a version that was archived:

http://web.archive.org/web/20080104044407/http://www.kogan.c...

At a glance, they are whitelisting some bots, including Googlebot, but not including Bingbot, and disallowing the rest.

  User-agent: Googlebot
  Disallow: 

  User-agent: *
  Disallow: /
Another marketing stunt?



Your link is pointing at 2008. That is four years ago. Hardly proof of anything.

You are right about the content type however.

-----


Correct. It doesn't proof much, but that they previously had a buggy robots.txt.

Usually these posts are accompanied by a declining traffic graph for Bing referrers. That would give us more proof and a timeframe to work with.

The post gives two options for their absence in Bing results: a fault at Microsoft or as a punishment. With their history I think a fault on their own part is a more likely option. I can't rule that out.

-----


You are arguing in bad faith. The robots.txt from 2008 is a whitelist of 190+ known crawlers. It doesn't include Bing, which was launched in 2009.

In 2008, kogan.com was a parked domain. It has nothing to do with the current owner(s).

-----


The choice for linking to a 2008 file was not the best. At least their current robots.txt is buggy too, so it isn't all that relevant for proving a track record.

I thought kogan.com was around in 2008 as the blog goes back to 2008.

The robots.txt is whitelisting bots with agents like "No" and "Due to a deficiency in Java it's not currently possible to set the User-agent." but then blocks all other known crawlers (like MSNbot and Yahoo Slurp).

I was not trying to be deceitful. IMO: Deceitful is a PR stunt arguing it is the fault or an evil plan of Microsoft, when you don't present anything to substantiate your claim, and it is a fact the webmaster is at fault in the vast majority of these cases.

-----




Applications are open for YC Summer 2015

Guidelines | FAQ | Support | Lists | Bookmarklet | DMCA | Y Combinator | Apply | Contact

Search: