
Kogan.com mysteriously disappears from Microsoft Search Engines - _Mark
http://www.kogan.com/au/blog/Kogan-dissapears-from-microsoft-engines-bing-yahoo/
======
blauwbilgorgel
They serve their robots.txt as a "application/octet-stream" not "text/plain".
Might be confusing some search engines?

Also according to Internet Archive their robots.txt has changed in the last
months.

This was a version that was archived:

[http://web.archive.org/web/20080104044407/http://www.kogan.c...](http://web.archive.org/web/20080104044407/http://www.kogan.com/robots.txt)

At a glance, they are whitelisting some bots, including Googlebot, _but not
including Bingbot_ , and disallowing the rest.

    
    
      User-agent: Googlebot
      Disallow: 
    
      User-agent: *
      Disallow: /
    

Another marketing stunt?

~~~
UnoriginalGuy
Your link is pointing at 2008. That is four years ago. Hardly proof of
anything.

You are right about the content type however.

~~~
blauwbilgorgel
Correct. It doesn't proof much, but that they previously had a buggy
robots.txt.

Usually these posts are accompanied by a declining traffic graph for Bing
referrers. That would give us more proof and a timeframe to work with.

The post gives two options for their absence in Bing results: a fault at
Microsoft or as a punishment. With their history I think a fault on their own
part is a more likely option. I can't rule that out.

~~~
Yver
You are arguing in bad faith. The robots.txt from 2008 is a whitelist of 190+
known crawlers. It doesn't include Bing, which was launched in 2009.

In 2008, kogan.com was a parked domain. It has nothing to do with the current
owner(s).

~~~
blauwbilgorgel
The choice for linking to a 2008 file was not the best. At least their current
robots.txt is buggy too, so it isn't all that relevant for proving a track
record.

I thought kogan.com was around in 2008 as the blog goes back to 2008.

The robots.txt is whitelisting bots with agents like "No" and "Due to a
deficiency in Java it's not currently possible to set the User-agent." but
then blocks all other known crawlers (like MSNbot and Yahoo Slurp).

I was not trying to be deceitful. IMO: Deceitful is a PR stunt arguing it is
the fault or an evil plan of Microsoft, when you don't present anything to
substantiate your claim, and it is a fact the webmaster is at fault in the
vast majority of these cases.

------
citricsquid
Microsoft themselves want their users to upgrade to latest browsers (see:
<http://www.ie6countdown.com/>) so the idea that they would take issue with
someone else doing this (so much so that they'd _remove_ them from bing) seems
a bit.. unlikely.

~~~
Maxious
For those who are not familiar with Mr. Kogan:
[http://en.wikipedia.org/wiki/Kogan_Technologies#Controversie...](http://en.wikipedia.org/wiki/Kogan_Technologies#Controversies)

As another commenter suggested, I would not put it past him that he blocked
bing somehow just for a publicity stunt. The wikipedia article hasn't even
caught up with last week's stunt (or there are just so many that it cannot
list them all): [http://www.gizmodo.com.au/2012/07/kogan-launches-campaign-
to...](http://www.gizmodo.com.au/2012/07/kogan-launches-campaign-to-counter-
carbon-tax/)

~~~
jawr
I like these guys, particularly for the 'Website glitch', I can't imagine many
retailers would take a similar stance.

Seems to me that these guys are great marketers.

------
chrislomax
Anyone else thinking they added a robots.txt entry for bing to disallow,
waited to get de-listed then removed the offending robots.txt entry, posted
this news article knowing bing would pick them back up in a couple of days?

~~~
michaelt
The HTTP last modified header for www.kogan.com/robots.txt has Last-Modified:
Wed, 06 Jun 2012 06:14:35 GMT

It's possible they did this and they were cunning enough to turn back the
last-modified clock.

~~~
reinhardt
How did you get that? Curl says May 4 instead:

    
    
      $ curl -I http://www.kogan.com/robots.txt
      HTTP/1.1 200 OK
      Server: nginx/1.0.14
      Date: Mon, 09 Jul 2012 18:48:34 GMT
      Content-Type: application/octet-stream
      Content-Length: 79
      Last-Modified: Fri, 04 May 2012 18:58:02 GMT
      Connection: keep-alive
      Accept-Ranges: bytes

~~~
michaelt
I get Last-Modified: Fri, 04 May 2012 18:58:02 GMT from my home machine (and
you obviously do too) where I connect to 74.118.68.224 - but I got Wed, 06 Jun
2012 06:14:37 GMT from work (and via <http://web-sniffer.net/> and
<http://www.rexswain.com/cgi-bin/httpview.cgi>) which was looking at
108.166.39.66. Perhaps we get DNS load balanced or something similar?

------
AshleysBrain
I think this is just some random technical glitch that has de-ranked them for
some reason at an awkward time, along the lines of Google accidentally marking
a legit website as spammy, sending it way down the results. Can it really be
proved that this was done directly in response to the IE7 tax? What would the
point be anyway? Maybe the storm of incoming links about the IE7 news
accidentally got identified as linkfarming somehow?

------
ot
"we noticed that the www.kogan.com website stopped appearing in organic search
results on Microsoft backed search engines"

They don't say explicitly, let alone prove, that the result _ever_ appeared on
Bing, while "stopped" would imply that.

If the comments about the wrong robots.txt got it right, it is well possible
that the website was never indexed by Bing, and they just noticed it (or
decided to talk about it) now.

------
nhebb
Funny, the ad I see displayed with the search results is:

    
    
      Upgrade from IE6 Today
      Microsoft.com/IE9 Say Goodbye to Outdated Browsers.
      Download the Latest IE Browser Now!

------
glesica
Interesting that it comes up in DDG, which I thought was partially backed by
Bing. They augment the results quite a bit, though, so not terribly
surprising.

Also, I made myself sad when I typed "kogan.com" into the address bar to test
DDG and was shocked when the actual web site came up. Didn't think that one
through...

------
magoon
I use Bing, so this is disappointing - but we can only assume it is a mistake,
and the most interesting part of this story will be how quickly Microsoft
reverts & responds.

------
UnoriginalGuy
Their robots.txt file is being delivered as the wrong content type for me. It
is being served as a "application/octet-stream" instead of a "text/html"

I have no idea what consequences this might have since I've never seen another
site which does it. Some quick Googling turned up nothing.

Maybe Bing choked on the wrong content type and decided to blackball their
entire domain in order to avoid indexing the unindexable?

~~~
X-Istence
Why would you want to serve up a robot.txt as "text/html" when in reality it
is "text/plain"?

------
Jacobi
When I perform a search on bing the first result is the kogan website ...

EDIT: It was the kogan.co.uk and not kogan.com since my country/region = UK

~~~
Jabbles
Neither "kogan" nor "kogan.com" return the website for me.

~~~
Jacobi
This is a screen shot of my search results <http://i.imgur.com/If1E2.jpg>

~~~
vladd
Notice the .co.uk ending.

------
damian2000
Ruslan Kogan must be rubbing his hands with glee at this development - more
pricele$$ free publicity for his site.

------
DeepDuh
Maybe their crawler identifies as IE7?

~~~
nekgrim
Kogan's website isn't blocked for IE7, so I could not see how it can be a
problem if the crawler identifies itself as IE7.

~~~
raverbashing
It is a problem if the site presents different content (that is, for example,
the light box announcing the tax)

This may throw the search engines away, because you're "de-SEOing" your
content

~~~
nekgrim
Never heard about crawlers changing their user agents. Never thought they
_can_ do that. Interessing idea.

------
brudgers
Just a thought, but I seem to recall that Google was giving higher rankings to
sites that advocated upgrading to Chrome, and created a bit of embarrassment.

Given that Kogan.com advocates upgrading and all the linking to their website
which occurred recently, perhaps it seems possible that a Microsoft algorithm
designed to avoid similar SEO manipulation was triggered.

In other words, the way in which the Kogan.com story unfolded was sufficiently
similar to the PR move by Google, that it triggered a Microsoft safeguard.

[http://www.pcworld.com/article/247257/google_disciplines_its...](http://www.pcworld.com/article/247257/google_disciplines_itself_in_chrome_browser_pagerank_controversy.html)

~~~
chrislomax
They did this to themselves as it's against their guidelines to pay for page
ranked incoming links. It was simply because in the end they were paying for
the link to their site

------
Killobyte
"We never waged war against Microsoft over IE7, we simply wanted people to
upgrade their web browsers..."

Maybe so, but the splash says "Use a better browser" and lists Chrome,
Firefox, Safari, and Opera. It would be ridiculous for Microsoft to remove the
page from their search results because of this, but at just a glance, and
maybe to the untrained user, it could look like the are waging war on IE as a
whole, and not just trying to get users to "simply upgrade their browsers."

~~~
pyre
IIRC, XP will only ever go as high as IE7 or IE8. Vista will only go as high
as IE9. Anyone hanging around on those operating systems, will need to look
outside of Microsoft for a more up-to-date browser.

------
mtgx
I think Microsoft themselves admitted that they do a great deal of manual
editing for their search engine rankings.

~~~
rplnt
Where? By quick search I did not find anything.

~~~
sreyaNotfilc
I think mtgx is referring to where Google claimed that Microsoft stole their
search results...

[http://googleblog.blogspot.com/2011/02/microsofts-bing-
uses-...](http://googleblog.blogspot.com/2011/02/microsofts-bing-uses-google-
search.html)

------
gouranga
I reckon this is because the ranking algorithm deranked them when their click
through from bing rate declined after the sudden publicity storm. Their
#visitors delta probably went very negative after everyone stopped clicking
links from various news sites.

They probably shot themselves.

~~~
jonursenbach
Doubtful. Even searching "site:kogan.com" pulls up zero results while
"site:kogan.co.uk" pulls up 1.9k results.

Kogan.com is completely gone from their resultset.

------
Isofarro
Looks like Kogan only serves the UK and Australian market, there's no support
from the US. Perhaps this is a ranking correction to delegating Kogan pages to
UK or Oz specific searches rather than global or US.

------
gary4gar
If this is real, then Microsoft would is going to get into a long legal
battle. I hope this is some technical glitch & not some random angry guy at
Microsoft

~~~
freehunter
There's only been one case I'm aware of where a site was penalized in search
rankings and filed a lawsuit. That site lost. The only complaint that the
judge seemed to find valid (although ultimately dismissed) was anti-trust
allegations. Microsoft wouldn't be in that position in search.

In short, there's no legal precedence for filing damages if you're dropped by
a search engine. It would be hard to imagine what, if anything, Kogan would
argue successfully.

[http://blog.ericgoldman.org/archives/2007/03/kinderstart_v_g...](http://blog.ericgoldman.org/archives/2007/03/kinderstart_v_g_2.htm)

------
sreyaNotfilc
This "tax" is elitist internet bullshit.

I'm all for using any browser that you'd want, but why go through elaborate
lengths to put Microsoft down? Especially on a version of IE that's 3
iterations in the past.

I understand that IE7 may not be as great as the other browsers. I'm fine with
being annoyed by that. But what about the other versions of IE? Why isn't
there a link for IE9 or IE10?

~~~
whatusername
This "tax" is internet marketing linkbait. Pure and Simple.

