
Google Deactivates Web Search API - woodruffw
https://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=small&safe=active&q=foobar&max-results=1&v=2&alt=json
======
woodruffw
Since I can't post both a title and text:

What's interesting about this is that the API has been officially deprecated
since 2010 (and "offline" since September 2014), but today is the first day
that it's actually become unavailable.

Edit: Additionally, there are probably going to be _big_ repercussions to the
web/average person's browsing experience as a result of this. A _massive_
number of sites (and programs, bots, etc) were using this API because of its
simplicity (and absence of registration/authentication).

~~~
tokenizerrr
> Additionally, there are probably going to be big repercussions to the
> web/average person's browsing experience as a result of this. A massive
> number of sites (and programs, bots, etc) were using this API because of its
> simplicity (and absence of registration/authentication).

Yep. My harmless little IRC bot can no longer Google search anymore :(

~~~
fixermark
Why not? There's a custom search API.

[https://developers.google.com/custom-search/json-
api/v1/over...](https://developers.google.com/custom-search/json-
api/v1/overview)

~~~
throwanem
Yes, and it's scoped by either domain or (rarely used) types from the
schema.org ontology. You can't use it to just search the web.

~~~
fixermark
You can use it to search the web; you don't get the exact same experience as
the old API.

[https://support.google.com/customsearch/answer/2631040?hl=en](https://support.google.com/customsearch/answer/2631040?hl=en)

~~~
throwanem
Neat. Now it's just a matter of dealing with the absurdly low free usage limit
and absurdly expensive paid tier.

~~~
Ntrails
Then it's just a market forces question. Use a provider of search services
which matches your budgetary/usage requirements.

------
visarga
We need a search engine that allows for deep search. It should be an open and
cooperative project, and users could possibly run an instance of the
spider/indexer as payment for executing searches, so it could be like a cross
between bittorrent and Tor.

A free search engine would enable API calls and also boost privacy and freedom
from the likes of Google. We have built a lot of experience about search
engines since 2000, we have access to scientific papers, cheap cloud servers
and a huge interest in freeing search, so I think the open source community do
it.

~~~
sdrinf
Writing an open, and distributed web crawler / indexer is a nice programming
exercise.

Writing an "objective" ranking function (for any values of "objective") in an
open, and distributed manner is structurally not favoured by humanity's
current incentive structure. As in:

* a dev team have to agree on signals, and weights: "SERP quality" has dedicated _teams of people_ assigned for specific verticals @ Google; replicating this in a distributed manner will be played politically

* Assuming any significant usage, the second you submit ranking code to public github repo, the algo will be played by thousand SEO scammers to their advantage

* Executing custom ranking function on other people's computer not only introduces security risks, but will have scammers setting up honeypots for collecting other people's ranking signals, and playing accordingly.

~~~
sourcd
> submit ranking code to public github repo, the algo will be played by
> thousand SEO scammers to their advantage

Just a thought : Ranking code could itself learn & adapt to each individual
user (the learned "weights" could be sync'd online across your devices).
Weighted signals from users can be fed back to the mother ranking algorithm
(un-customized one). Basically millions of distributed deep minds[1], instead
of a single one.

I can imagine there are a lot of holes in my theory, but we can't simply
accept that open sourcing the algorithm implies that it can't be done.

[1] : [https://deepmind.com/](https://deepmind.com/)

~~~
sangnoir
> Ranking code could itself learn & adapt to each individual user

Cue blackhat SEOs creating millions of subverted "users" on AWS spot
instances/Lambda

------
miduil
We had some trouble when the Youtube API was disabled for non-registered
users. We were using it for some shared & local web-playlist, which is based
on youtube-dl/mpv.

Sadly we couldn't use DDG because their API won't let you filter for video
results only (Source: Asked on IRC, got a reply from DDG staff + haven't found
any documentation mention this either).

We ended up using searx [0] which takes little more time for search results
(about 1-3 seconds) but we gain more video-sources (such as vimeo or
dailymotion).

I'm aware that youtube-dl got a search functionality build in as well, but the
requirement that the client needs to get results without being prepared by our
embedded computer wouldn't fit.

[0]: [https://github.com/asciimoo/searx](https://github.com/asciimoo/searx)

------
SwellJoe
What are the alternatives for a full web search API? Yahoo BOSS closed down a
few weeks ago. Bing seems the last man standing. And, unfortunately, none of
the ones I've tried are very good; one project I'm working on needs to
prioritize newer results over older ones, and only Google had a sort by date
option (which I think was also removed in the past).

~~~
pajop
the first thing that came to mind was DuckDuckGo :)
[https://duckduckgo.com/api](https://duckduckgo.com/api)

~~~
boyter
Its not a web index. They are unable to offer that as a lot of their results
come from external services such as Yandex and Bing.

The results are their zeroclick info snippets and it is very useful for
pulling an answer from wikipedia since they have already parsed it.

~~~
chris_wot
But do they consider spidering context problematic? What would stop someone
building an API on top of their simple websearch?

------
sourcd
This needs our love & support : commonsearch.org

Prev discussion (few weeks back)
[https://news.ycombinator.com/item?id=11281700](https://news.ycombinator.com/item?id=11281700)

Credit : @sylvinus

------
nfriedly
I wrote a PageRank library a while back and just got a report yesterday that
it also no longer works.

(PageRank never actually had an official API, but it was exposed for the
Google toolbar. They had announced that this would go away a while back but
just dropped the axe in the last day or two.)

~~~
woodruffw
I guess they're cleaning house.

------
masswerk
Phew! -- Just managed to reactivate some applications[1][2] by providing
cached search results.

The Google REST API was a great way to provide audiences with a known tasks in
order to connect them to a historic setup or interface. RIP.

[1] [http://www.masswerk.at/googleBBS/](http://www.masswerk.at/googleBBS/) [2]
[http://www.masswerk.at/google60/](http://www.masswerk.at/google60/)

------
bake
Does anyone know of an API that returns pages related to any given URL? I'm
looking for an alternative to Google's "related:..." search, which was only
reliably accessible through Google Web Search (Google Custom Search is very
expensive and has a limit of 10K queries per day).

For example:
[https://www.google.com/#q=related:http:%2F%2Fagiliq.com%2Fbl...](https://www.google.com/#q=related:http:%2F%2Fagiliq.com%2Fblog%2F2013%2F08%2Fwriting-
thread-safe-django-code%2F)

It returns pages related to any given page, and works at the specific page
level (not just at the domain level, like SimilarWeb's API).

I've migrated to Bing for all other web search needs (way cheaper and without
volume limits like Google Custom Search), but Bing does not offer a
"related:..." equivalent as far as I can tell.

~~~
partiallypro
While you're looking for this feature in the meantime, ping the Bing team and
ask for it as an added function of their API. Microsoft is a lot more open to
feedback these days.

~~~
bake
Any idea on how best to get in touch with them?

~~~
overbrimming
Hi! I work on the Microsoft Cognitive Services team, which includes the Bing
Web Search APIs in our family.

For feature requests, it would be awesome if you posted them to:
[https://cognitive.uservoice.com/](https://cognitive.uservoice.com/) (I'm also
forwarding your post to Bing).

If you want to get in touch with the team directly about a question/issue, the
"Contact Us" widget at the bottom of the Cognitive Services page is the way to
go: [https://www.microsoft.com/cognitive-
services](https://www.microsoft.com/cognitive-services)

[http://www.bing.com/partners](http://www.bing.com/partners) list all the
services/APIs/partnership opportunities Bing offers.

------
martinald
Slightly OT: Does anyone know how RankScanner works? My guess was this but it
still seems to work. Their site seems to suggest they have loads of servers,
but seems unlikely, I'd have thought Google would have captached them to hell
by now.

Very good product. I'm curious how it works though.

~~~
Jake232
Having built a ranktracker before, and still being good friends with people
who run one of the biggest rank trackers on the market, using the google API's
simply isn't an option.

Simply put, the API never returned the results in the exact same order as the
actual search results did.

The best / most reliable rank trackers did (and still do) simply use proxies
to get around the google captchas. I've scraped millions of pages from google
over the years, and with enough proxies, the correct proxy delays/timeouts,
and other little tricks you pick up along the way; you can actually scrape
google pretty easily.

This is especially true with the dropping cost of proxies. I'm obviously an
exception considering I run several scraping SaaS and have generally
specialized in it for years, but I can he hitting x.com with 30,000 different
IP's within the hour.

~~~
visarga
> with enough proxies, the correct proxy delays/timeouts, and other little
> tricks you pick up along the way; you can actually scrape google pretty
> easily

Try putting a site: in the search and it becomes suspicious fast.

~~~
Jake232
You're correct. However for Rank Tracking these types of queries aren't
needed.

------
dajohnson89
What is the rationale for Google doing this? How much money was this service
costing them?

~~~
skrowl
I believe the problem was that if you did /google or !google on your bots for
various chat platforms, you were able to find the top hit or two without ever
looking at Google's Ads.

Most of their revenue is ads.

~~~
dajohnson89
So the question becomes, how much _potential_ money were they losing through
the API? Does Google think that by removing the API, people are now going to
go to another window, perform a google search, and intently study the
advertising before reviewing the search results?

~~~
mwfunk
You just made a textbook strawman. It doesn't do anyone any good to theorize
about what the dumbest possible reason might be for Google to do this. It is
infinitely more useful to think about what rational, informed reasons they
might have for doing it (regardless of whether you agree with those reason).
People reflexively jumping to strawman arguments is probably #2 or #3 on the
top ten list of Why We Can't Have Nice Things, in tech, politics, and just
about anything else in the world where different groups of people have
imperfect knowledge about each other's activities and motivations.

------
mozumder
Why do API developers add a HTTP status code in their JSON response, when the
HTTP status code is already in the header?

~~~
X-Cubed
Depending on the technology you're using to access the API, the HTTP status
code may not be exposed from the underlying stack.

For example, Microsoft Silverlight only returned 200 or 404 status codes:
[https://msdn.microsoft.com/en-
us/library/system.net.httpstat...](https://msdn.microsoft.com/en-
us/library/system.net.httpstatuscode\(v=vs.95\).aspx#Anchor_1)

~~~
jsmthrowaway
That would be called a "bug," and a perplexing one, at that.

------
genop
The YouTube API used to be open too - no regstration or authentication. That
ended years ago. I don't really need an "API" to do searches and transform the
results into a more useful format than HTML. These API's make it easier and
are addictive, for sure. It's foolish to rely on them however.

------
allard
I believe Google will replace it — the Google Search Appliance is going away
too. Something will pick up those customers. An announcement at I/O, which is
pretty soon?

~~~
gizzlon
It has been deprecated for years, disallowing new customers.

No sign of any replacement

------
amelius
I don't want to sound negative, but this is why I don't use external APIs. You
never know when they will suddenly shut down.

~~~
mseebach
No, it's only been deprecated for six years. There was just no way to see it
coming.

------
fastest963
Afaik this has been broken for me for December 8, 2015 when I first received:
{"responseData": null, "responseDetails": "This API is no longer available.",
"responseStatus": 403}

------
TazeTSchnitzel
`"responseStatus": 403`, but the actual HTTP response is 200 OK?!

~~~
yolesaber
Because the request went through! /s

------
givan
With more people placing value on privacy, the api was used by disconnect
search along with tor and other privacy tools, google has just planted the
seed that will become their biggest competitor.

------
jayjay1010
This is terrible news and with Pagerank also disabled recently, now 2 features
of our domain health testing service has been crippled. Pagerank was easy to
replace with moz rank but this data was irreplaceable due to the fact that it
was specific and unique to google. Rip one of our main tools.

~~~
chrismorgan
PageRank had been completely useless for a couple of years, and practically
useless for several years before that.

~~~
jayjay1010
That depends on your reason of usage, as a guide to web ranks its been
obsolete as you say for years, but as a cheap and quick measure of a domains
potential (for passing link juice or to compare link juice between two
domains) in comparison against another, Pagerank had some value.

------
JustUhThought
#DuckDuckGoLooksUnprofessional

It seems pety, but I really feel the duck look is holding it back.

~~~
ocdtrekkie
You mean like the original Google home page? With it's clashing bright colored
logo and terrible font use?
[http://www.jaygeiger.com/index.php/2007/01/21/original-
googl...](http://www.jaygeiger.com/index.php/2007/01/21/original-google-
homepage/)

~~~
soared
No, Google had a nonsense name with a logo, like Bing. DDG is a silly name
with a cartoon duck.

