

Google's new Ajax-powered search results breaks search keyword tracking for everyone - imp
http://getclicky.com/blog/150/googles-new-ajax-powered-search-results-breaks-search-keyword-tracking-for-everyone

======
lacker
Disclaimer: I work at Google.

This is not about analytics. This is just an experiment to test if using ajax
instead of the typical form submit can make the result page load faster.

Here's an official statement -

    
    
      We’re continually testing new interfaces and features to enhance the user experience.
      We are currently experimenting with a javascript enhanced result page because we believe
      that it may ultimately provide a faster experience for our users. At this time only a
      small percentage of users will see this experiment. It is not our intention to disrupt
      referrer tracking, and we are continuing to iterate on this project.
    

[http://searchengineland.com/google-ajax-search-results-
death...](http://searchengineland.com/google-ajax-search-results-death-to-
search-term-tracking-16431)

------
three14
Many of the comments seem to assume that websites are entitled to know what I
searched for. I think that it would be a great step to give millions of people
a little more privacy, even if it's just an accidental side effect of a change
to Ajax.

~~~
dejb
I feel that the benefit in letting webmasters know what their users are
looking for outweighs any privacy gains to be had. If people are that
concerned about their privacy they could always set their browser to not send
the referrer. I doubt that this move has anything to do with concern over user
privacy.

------
misterbwong
This is a potentially huge problem. Google is effectively cutting off all
other search analytic packages from their information.

~~~
jorgeortiz85
No, they're not. It's not like they're deliberately obfuscating the
information in order to shut everybody else out. They've made a technology
decision which will probably significantly speed up searches and result in
increased revenues for them. Google shouldn't be beholden to other people's
business models.

That said, given the amount of referrals that Google generates, and how
important referral information is for a lot of things online, as a courtesy
Google should probably communicate this to the web-at-large before rolling it
out on a massive scale. However, they're not "cutting off" other search
analytic packages. At worst they're forcing a rushed bugfix.

~~~
cakeface
Yes they are cutting off other search analytic packages. If there is no
tracking of the search term in the referring url then the analytics package
has no way to determine what keywords were used to bring a user to the page.

Since google provides one of the most commonly used analytics packages, Google
Analytics, and owns another, Urchin, I would think that they would be the last
ones to try this.

~~~
durin42
I thought Urchin became GA? I remember early on in my analytics use the script
mentioned urchin, and the newer one they provide doesn't anymore...

~~~
cakeface
I think that Urchin was the basis for GA. Urchin is still available from
google here though <http://www.google.com/urchin/>. The main difference that I
know of between them is that Urchin has the ability to parse server logs as
well as track users through javascript.

------
peregrine
Google is a business who wants you to use their analytical service over
others. They have no obligation to you send you the information their users
are using to find your site.

~~~
gojomo
Google is also arguably close to being a effective monopoly in web search.
Antitrust law prohibits using monopoly power in one market (eg search) to
acquire a monopoly in another market.

So there's a bit of a bind for Google: strategies that were perfectly legal
when they were up-and-coming become illegal at some point of dominance. But
there's no 'bright line' for when the rules change -- market definition,
thresholds, and acceptable practices can all be argued after the fact.

A naturally competitive company is almost certain to keep the 'pedal to the
metal' until it's too late.

~~~
madh
Is there ever really a "too late"? Shouldn't Google push for dominance as far
as it possibly can? Isn't it some other entity's role to stop them?

~~~
gojomo
Depends on how the risk and costs of antitrust punishment compare to the
benefits of 'full speed ahead'.

~~~
jonursenbach
If Google ever gets split up in an antitrust case, they'll most likely pull a
T-1000 and reemerge years down the line the same way that AT&T did.

------
raquo
I can't really see how google can easily pass the search query to Google
Analytics. GA requires atomic data to perform all this analysis and custom
segmentation magic.

The only way I see is using a cookie (with a user id) on the google.com
domain, which ga.js must interpret in backend on every request. But this means
that ga.js will have to be dynamically generated, rather than being a simple
static file (and yes, it won't be cached by users visiting your site, making
tracking even slower). I don't think google wants this additional load every
time anyone loads some GA-enabled page on the web. Besides, ga.js is currently
hosted on google-analytics.com, not google.com, so all users will need to
upgrade to a new tracking script. And it will not be possible to self-host the
script anymore.

Anyway, this does not sound like anything reasonable to me. I hope it won't be
implemented - it would destroy an entire industry, and not a redundant one.

~~~
schammy
When you click a link from a google SERP, it goes through a redirect script.
They could easily be using that to feed Google Analytics.

~~~
raquo
Yes, but in that case other web analytics packages will also have access to
referrer that contains the search query and there is no problem.

I wasn't clear enough, sorry - I was concerned with how Google could be able
to share the search query _only_ with GA but not other analytics tools,
because it is clear that Google will not sacrifice GA usefulness to some ajax
SERPs.

~~~
dshah
I might be missing something, but I think Google can easily modify their
redirect script to store the data they want (for use by Google Analytics) and
then strip that data out before it hits the target URL.

------
dchest
_See how there's a hash mark # in there now, and the "q=test" is after it? The
problem is that web browsers don't send anything after the # in the referrer
string._

I wonder why they don't send it in referrer string. Are there any technical or
security reasons?

~~~
lacker
Originally the content after the # was only used for automatic scrolling to
particular tags within a page, so there was no need to ever send it to a
server. Once Ajax came around, people realized you could use javascript to
read the url after the # and do arbitrary stuff based on it. So you can show
different pages at

    
    
      foo.com/
      foo.com/#bar
    

but since the server doesn't treat them differently, when the user transitions
from one to the other, there's no page reload. So you can move from page to
page, but cut n pasting links still works.

There are libraries like RSH that let you do this to speed up your site.

~~~
sqs
Seems to me that browsers ought to start including the fragment (the part
after the #) in the referrer, now that different fragments can refer to
entirely different pages. I can't see any security issues, as the previous
poster was wondering.

~~~
lacker
There would be security issues if any existing website has put confidential
user data after the #, expecting that no other website would see this in the
referers. This doesn't seem typical, but still I'm sure some websites would
have security issues if this was changed.

Also, practically speaking, even if all browser developers agree this was a
good thing, it takes quite a long time for people to upgrade their browsers.

------
sant0sk1
Just wow. I'm really starting to anti-trust Google.

~~~
axod
They don't owe you that information, sure it's nice I guess...

In any event, the only way I can think to do this with an ajax search is to
pass every outgoing link click through a redirection script, so that the
referrer is rewritten to something useful.

However, whilst improving things for webmasters, that would seriously negate
the point of using ajax, and be a pain for end users.

Google are serving their end users. I don't think trust comes into it TBH.

What would you rather google do? _not_ progress and use ajax search, or use a
redirection and make a worse experience for end users?

~~~
ryanwaggoner
Unless I'm way off, Google is already doing a redirection script on all
outbound clicks. The endpoint url shows up in the footer of the browser on
hover, but if you right-click and copy the link location, you get something
like:

[http://www.google.com/url?sa=t&source=web&ct=res&...](http://www.google.com/url?sa=t&source=web&ct=res&cd=4&url=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FY_Combinator&ei=k8KISfaEJpGksQOturCOBg&usg=AFQjCNGLIQ4vyID6JQEC5r9XGE36mIoZUg&sig2=CZQgmDgORRcK6IceMaiCsQ)

~~~
lacker
The author says he is seeing referers of just "google.com/" in his logs. If
they were redirected wouldn't he be seeing these /url pages instead?

------
ewiethoff
Hmmm. I'm not seeing this new "feature" when I search Google. I've turned on
Javascript, I'm accepting Google cookies, I've even logged in. And I still see
the usual <http://www.google.com/search?q=ajax+crap> results URL. I'm running
Firefox 3.0.5.

Anyone know what triggers the new search results?

~~~
raquo
Probably (and hopefully) it's just being bucket-tested...

------
sam_in_nyc
There are a variety of things which I don't think anybody has noted. First of
all, for those without Javascript, the old Google page is used. So, you'll
still see some sampling of keywords coming in... albeit only 2% of what it
originally was. So it's not _totally_ ruining things. You'll still get some
sample of referring keywords.

The benefits to Google are more than anybody is reading into. It is
_definitely_ an intentional move to cloak their redirects and keywords. And it
_definitely_ increases performance for them.

Performance gains:

The back/forward buttons do not cause anything to be reloaded. This should
save a nice percentage of page loads which were along the lines of "what was
my last search?" and going back and forth between pages.

If you look at Firebug, the Ajax request is actually going to the proper
google search page... google.com/search?q=... Google is taking the entire
search page and essentially replacing it without a page refresh. This doesn't
really mean anything significant, it's just a neat observation.

Strategy:

More than anything, in the long run this will improve search results. It's not
about locking down the "analytics" arena... as one poster noted, they probably
won't be able to show Google search terms either (unless they set a cookie on
the Google search page, and read it on GooAnal [yes, GooAnal] pages, which
might actually happen). The important thing is: _SEO gamers now do not have
access to which google keywords work and don't work_. Without giving SEO
gamers which keywords are working and which aren't, it's going to be a lot
tougher to game the system. These SEO firms that manage to get the spammiest
crap on the top pages won't have much to work off of other than the raw amount
of visitors coming from google. I imagine that Google has secretly rolled out
quite a few significant changes in their search algorithm as well, to work in
tandem with keeping analytics in the dark.

My prediction is, in a few months, you'll be thanking Google for the much
better search results. I welcome this change with open arms, because I'd much
rather see quality content rise to the top, rather than gamed content targeted
at keywords. The trade-off that people running websites won't be able to see
referring keywords... well I guess Google might lose .001% of their market
share.

~~~
schammy
"My prediction is, in a few months, you'll be thanking Google for the much
better search results."

I don't really think this is going to change anything. How is this going to
affect SEO? I doubt any of them use referrers logs - waste of time. They
scrape the SERPs. The old SERPs are still available. They can tell when a rank
goes up or down.

How is this going to change that?

~~~
sam_in_nyc
It's one thing to tell what rank you are on what keyword, but quite another to
track how much traffic you are getting from a specific keyword. Though I guess
there's ways to estimate that.

You bring up a good point, though. Guess it's not as grand a scheme as I
thought.

------
rgrieselhuber
Surprised to see this as the keyword reports are a big part of Google
Analytics as well.

------
redorb
I just looked at the Source for google.com's new search result page, You can't
read 1 word of english off it - totally Ajax. Looks amazingly complex.

------
th0ma5
the standard is the referrer being in the HTTP headers, not specifically that
if the referrer is google, that it must include the search terms used in their
search

we seem to be putting more reliance on these flimsy things... in the future
will i create an outage because i @somebody instead of #something on twitter?

~~~
ComputerGuru
In the case of web browsers, we don't even SEE the stuff that comes after the
#

So it's not like putting "@someone" instead of "#someone" but rather like
putting "" instead of "#someone"

See the difference+

~~~
th0ma5
sorry, maybe a bad example on my part, my point is that querystring parameters
being passed should include the search terms used on that search engine is
something of a rule of thumb more so than a standard... but i don't know that
there are any standards built on top of the http referrer other than this is
how google and many others operated, so these reporting companies wrote code
that relied on that, rather than coding to a standard.

~~~
ks
I can see your point, but if Google adds that capability to Google Analytics
they will have a significant advantage over their competitors. So the question
is not simply about following an unofficial standard.

~~~
eli
How would they even do that if there's no query data in the referer? Set a
cookie on google.com and then read it back from analytics.google.com is the
only thing I can think of.

------
randomwalker
I believe the problem is bigger than what the article points out. Under the
new scheme, all search results pages have the same URL. The part after the
hash not part of the URL, it is a fragment or anchor.

It breaks my firefox extension that nukes searchwiki. __It breaks my custom
CSS for the search results page (see userstyles.org). __I often bookmark
google search results in del.icio.us. Depending on how such services are
implemented, different search results pages might show up as the same page.

I'd guess there's just a lot of code out there that assume search results are
separate URLs, and that's going to break.

 __Those two breakages are simply due to the regex identifying the search
results pages not matching anymore, and are easily fixable.

~~~
sam_in_nyc
Search results to the old ?q= google will still work:

<http://www.google.com/search?q=test>

------
tlrobinson
Google could easily fix this by redirecting through some sort of shim script.
But will they?

------
andrewf
Google has ~65% of the search market.

The other 35% is probably still representative enough for you to figure out
what the population is searching for that leads to your website.

~~~
imp
The problem with that is that your search ranking for any given key word is
probably different in each search engine.

------
parenthesis
Is anyone _not_ getting this on google.com ? It looks like it might be
happening for everyone on google.com, but not on, e.g. google.co.uk or
google.it .

~~~
raquo
I don't get anything ajaxy on google.com

Even if I go directly to <http://www.google.com/#q=test> it redirects me to
[http://www.google.com/search?q=test&cad=h](http://www.google.com/search?q=test&cad=h)

~~~
semiquaver
<http://www.google.com/#q=test> sends me to a blank google search with no
results.

------
YoavShapira
Nice to get a comment from Google right away, though.

------
braindead_in
Nightmare scenario. What if analytics goes paid?

