

Ask HN: What to do when Google ignores you? - Frozenlock

Hello gentlemen,<p>I've spent months building my tools and services.
I'm actively trying to promote my website, but Google stopped crawling my site for 3 months now. I've checked all my commits around the time Google stopped crawling, but there's nothing there that should have an influence!<p>I tested my website with proxies, web-sniffers, users from around the world: the website works for ALL of them. However if I try to 'fetch as Google' in the Google's webmaster tools, it always results in 'page unavailable'.<p>I emailed Google, but all I received (even after multiple tries) was pre-made answers of this kind "Have you checked our forums?" YES, I did! I posted my question, and didn't get any answer, which is why I'm emailing you now!<p>So yeah, I'm happy to see my website on DuckDuckGo, but I would really like to see it on Google; especially considering that my customers aren't necessary cutting edge...<p>- I tested with multiple proxies and web-sniffer tools.
- I checked my DNS
- I checked if my ISP was blocking Googlebot IPs
- I even checked with Wireshark and can see the Googlebot connection initiated!!<p>I asked Google to get a more useful explanation than just 'page unavailable', but alas...<p>So now I'm asking help from you.
Here is my website: http://goo.gl/aSX2u
Is there anything obvious that I'm missing? Why is Google ignoring me?<p>Thank you very much in advance!
======
neoterics
Your site may be banned since it is not in the index, I would file a
reconsideration request
[http://support.google.com/webmasters/bin/answer.py?hl=en&...](http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35843)

~~~
Frozenlock
Thanks. Any idea if this would also affect 'fetch as Google'?

------
AznHisoka
Try changing from HTTPs to HTTP temporarily (a week or so) and see if that
helps.

Another solution: Come up with a new domain, do a redirect.

~~~
Frozenlock
I did try without HTTPS, and even without GZIP. Not for a week, but enough to
test the 'fetch as Google' option in the Google webmaster tools.

I've also read that Googlebot shouldn't have any problems with HTTPS.

~~~
AznHisoka
Google claims they don't have problems all the time. Doesn't mean you should
trust them. Give it a shot, what will it hurt?

~~~
Frozenlock
Tried again without HTTPS and this time it worked (What the f..).

I hope this will be fixed soon, I would prefer to force HTTPS everywhere.

Anyhow, thank you very much, at least now the bot will be able to crawl my
site!

------
rcavezza
Do you use Google webmaster tools? They should show potential reasons on there
and which pages are currently indexed.

~~~
Frozenlock
The reason shown in the crawl error is "Robots.txt fetch".

This isn't very surprising as 'fetch as Google' returns a page unavailable for
everything.

(It's my understanding that a 404 on robots.txt is OK)

~~~
27182818284
In the last few days I've seen errors with our robots.txt on sites that have
NEVER had a problem with it and no errors. I even looked around on Twitter
thinking it was a Google Bot bug. Then it just went away. Out of curiosity,
why would a 404 on a robots.txt be OK? Shouldn't you want "Fetch as Google" to
return the proper HTTP/1.1 200 OK?

~~~
Frozenlock
I don't have a robots.txt, so -for this url alone- I'm expecting a 404. What
I've read in the google documents is that when a request to get the robots.txt
returns a 404, it will consider it as OK to crawl.

------
propercoil
three problems off the bat: 1\. you have only 10 backlinks from different root
domains. 2\. one of them is bacnetwiki.com (yours). This can signal a backlink
pyramid. 3\. your server is dropping requests when pinging from holland.
Something is definitely wrong here

~~~
Frozenlock
Thanks! The pinging is deactivated on my router, no surprise there.

The problem is not the ranking, which backlinks influence, but the ability of
the Googlebot to crawl the site.

Also, I did not add the entry to BACnetWiki. In fact it's probably one of the
only mention of the site that I did not made. :-)

------
orangethirty
I'm going to point the Nuuton crawler towards your site. Shoot me an email to
give you results.

~~~
Frozenlock
Thank you!

You can send me a mail at frozenlock@<the-domain-name>.com

