Hacker News new | past | comments | ask | show | jobs | submit login

I got a chance to try it out just to see if a simple URL fetch was blocked:


Locally, this code reports 200 OK for all of those except the last reports a 302 redirect (to the https, I presume). As you can see, on Google, they all fail with an internal exception.

Paypal does have a universal Disallow in robots.txt, so I thought I'd setup why.gd to do the same. But it works fine.

Anyone else find any other URLs that are blocked?

Looks fixed now. Very fast turnaround. Nice work, Google.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact