

Ask HN: Are search engines indexing short-urls? - Seldaek

Now I do wonder. All those tweet containing links to a site. They all use bit.ly and whatnot. Fine. But what does Google (or others) do when it sees such a link? From what I can see in Google Webmaster Tools, it seems that all these links don't count as links pointing to the site. A search for "link:someurl" also doesn't seem to return any result from twitter. This means you get no pagerank from twitter traffic.<p>Any idea how to fix this? Specifying rev="canonical" maybe?
======
infinity
You could find out if the short URL service is using a robots.txt file to
block search engine crawlers.

Next you can look what kind of redirect the service is using: status code 301
or 302 or something unusual like 307. Rex Swain's HTTP Viewer is a nice tool
to find out the status codes: <http://www.rexswain.com/httpview.html>

A site: search on Google for the domain of the service may also be
interesting.

I have seen that Googlebot and other crawlers will also follow "nofollow"ed
links, I suspect that they use these links for page discovery like ordinary
links. The nofollow attribute just tells the search engines that this link
can't be trusted and shouldn't be considered for ranking calculations (like
PageRank etc.).

It is not clear how much value Google and other search engines give to a link
that goes through a redirect. Sometimes pages redirecting with the code 302
appear in the search results instead of the target page (302-hijacking).

The link: operator is broken, it shows you only a small selection of links
pointing to a site if anything at all. The Google webmaster tools show more
links, but not all of them. Yahoo Site Explorer shows more links:
<http://siteexplorer.search.yahoo.com/> I have seen nofollow links in both of
these tools. They do not tell anything about the value of a link.

I remember that I have seen links from tinyurl in the old version of the
Google Webmaster Tools.

------
Raphael
Twitter puts a nofollow attribute on its links to reduce incentive for
spammers to tweet. Nofollow means search indexers don't crawl from that link
nor count it toward any ranking. (Twitter would have low SEO values anyway
because it has too high a concentration of links compared to text.)

As for short URLs, they shouldn't be designed to steal page rank. If they do a
proper 301 redirect, then the original address should receive the SEO value.

~~~
Seldaek
Yes, bit.ly which is probably the most used service on twitter does proper
301s as far as I know, so this isn't the issue. Good point about nofollow
though, I completely overlooked that. Cheers.

------
pierrefar
Yes they index these links. The "problem" (if you can call it that) is whether
the crawlers discover the links or not, and if they do, how much of a priority
they are.

For Twitter specifically, remember that lots of websites syndicate tweets,
links and all. It's likely a crawler would discover the links on non-
twitter.com websites.

