Hacker News new | comments | show | ask | jobs | submit login

Is all that HTML? Spiders generally only care about HTML -- they're not going to download the js and css and images and flash and whatever.

8 years ago, maybe. Today all search engine spiders pull down and execute javascript & css as well just so that they can index a real user's experience.

The better ones may, but perhaps not for all pages. Generally, when viewing logs I see search engine spiders make a single request, for only the html content and not js or css.

duh, I didn't think about that :) HTML is way smaller

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact