Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This seems rather intuitive. Does the google bot even parse css? Is there any evidence that it ignores non-displayed text present in HTML?


Googlebot and some other search engine crawlers also crawl css stylesheet files, I have seen evidence in my server log files.

There are several patents describing how a search engine might analyse the visual appearance and structure of a website, with a special view to give some parts of the page content (the main content) a higher weight or priority then the rest (like footer text and navigation). I would expect that the algorithms used by Google will give a good impression about the page rendering. What conclusions are drawn from this and how it affects the ranking of individual pages for certain search queries, who knows, ... pretty much about this is kept secret and there is a lot of speculation.

Many search engine patents are explored and explained by Bill Slawski on his blog "SEO by the sea":

http://www.seobythesea.com/

That there are these patents does of course not tell us, that search engines are working exactly in this or another way, but I think it often gives a good impression about what might be going on behind the scenes.


Further reading:

  Techniques for approximating the visual layout of a web 
  page and determining the portion of the page containing 
  the significant content 
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Se...

  Automatic Visual Segmentation Of Webpages
http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sec...

  Techniques for approximating the visual layout of a web 
  page and determining the portion of the page containing 
  the significant content 
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Se....

  VIPS: a Vision-based Page Segmentation Algorithm
http://research.microsoft.com/apps/pubs/default.aspx?id=7002...

  Classifying functions of web blocks based on linguistic features
http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sec...

  How a Search Engine Might Identify the Functions of 
  Blocks in Web Pages to Improve Search Results
http://www.seobythesea.com/2011/02/how-a-search-engine-might...


Yes, it does take into account text that is hidden from view. This used to be (still is in some places) a deceptive tactic to rank a page higher in the search results. You could hide a massive block of keywords out of site of your visitor, or hide links to sites that you want ranking higher. Alternatively, more malicious people would (and still do) hack into a website in order to hide links in this same manner. The website owner isn't likely to notice them if they aren't visible on a page.

Google is pretty good at figuring out what is there for legitimate reasons versus spam.


Yes, Googlebot also parses Javascript. All part of the SEO arms race.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: