Ask HN: What to do when Google ignores you?

[Frozenlock]

Hello gentlemen,

I've spent months building my tools and services. I'm actively trying to promote my website, but Google stopped crawling my site for 3 months now. I've checked all my commits around the time Google stopped crawling, but there's nothing there that should have an influence!

I tested my website with proxies, web-sniffers, users from around the world: the website works for ALL of them. However if I try to 'fetch as Google' in the Google's webmaster tools, it always results in 'page unavailable'.

I emailed Google, but all I received (even after multiple tries) was pre-made answers of this kind "Have you checked our forums?" YES, I did! I posted my question, and didn't get any answer, which is why I'm emailing you now!

So yeah, I'm happy to see my website on DuckDuckGo, but I would really like to see it on Google; especially considering that my customers aren't necessary cutting edge...

- I tested with multiple proxies and web-sniffer tools. - I checked my DNS - I checked if my ISP was blocking Googlebot IPs - I even checked with Wireshark and can see the Googlebot connection initiated!!

I asked Google to get a more useful explanation than just 'page unavailable', but alas...

So now I'm asking help from you. Here is my website: http://goo.gl/aSX2u Is there anything obvious that I'm missing? Why is Google ignoring me?

Thank you very much in advance!

[neoterics]

Your site may be banned since it is not in the index, I would file a reconsideration request http://support.google.com/webmasters/bin/answer.py?hl=en&...

[Frozenlock → neoterics]

Thanks. Any idea if this would also affect 'fetch as Google'?

[AznHisoka]

Try changing from HTTPs to HTTP temporarily (a week or so) and see if that helps.

Another solution: Come up with a new domain, do a redirect.

[Frozenlock → AznHisoka]

I did try without HTTPS, and even without GZIP. Not for a week, but enough to test the 'fetch as Google' option in the Google webmaster tools.

I've also read that Googlebot shouldn't have any problems with HTTPS.

[AznHisoka → Frozenlock]

Google claims they don't have problems all the time. Doesn't mean you should trust them. Give it a shot, what will it hurt?

[Frozenlock → AznHisoka]

Tried again without HTTPS and this time it worked (What the f..).

I hope this will be fixed soon, I would prefer to force HTTPS everywhere.

Anyhow, thank you very much, at least now the bot will be able to crawl my site!

[rcavezza]

Do you use Google webmaster tools? They should show potential reasons on there and which pages are currently indexed.

[Frozenlock → rcavezza]

The reason shown in the crawl error is "Robots.txt fetch".

This isn't very surprising as 'fetch as Google' returns a page unavailable for everything.

(It's my understanding that a 404 on robots.txt is OK)

[27182818284 → Frozenlock]

In the last few days I've seen errors with our robots.txt on sites that have NEVER had a problem with it and no errors. I even looked around on Twitter thinking it was a Google Bot bug. Then it just went away. Out of curiosity, why would a 404 on a robots.txt be OK? Shouldn't you want "Fetch as Google" to return the proper HTTP/1.1 200 OK?

[Frozenlock → 27182818284]

I don't have a robots.txt, so -for this url alone- I'm expecting a 404. What I've read in the google documents is that when a request to get the robots.txt returns a 404, it will consider it as OK to crawl.

[propercoil]

three problems off the bat: 1. you have only 10 backlinks from different root domains. 2. one of them is bacnetwiki.com (yours). This can signal a backlink pyramid. 3. your server is dropping requests when pinging from holland. Something is definitely wrong here

[Frozenlock → propercoil]

Thanks! The pinging is deactivated on my router, no surprise there.

The problem is not the ranking, which backlinks influence, but the ability of the Googlebot to crawl the site.

Also, I did not add the entry to BACnetWiki. In fact it's probably one of the only mention of the site that I did not made. :-)

[orangethirty]

I'm going to point the Nuuton crawler towards your site. Shoot me an email to give you results.

[Frozenlock → orangethirty]

Thank you!

You can send me a mail at frozenlock@<the-domain-name>.com