
Applebot, the web crawler for Apple - killwhitey
https://support.apple.com/en-us/HT204683
======
beltex
_“It looks like someone at Apple is running a web crawler written in Go.”_

Likely explains [1] from last year (see [2] for HN thread).

[1] [http://jan.moesen.nu/2014/11/06/apple-
crawler.txt](http://jan.moesen.nu/2014/11/06/apple-crawler.txt)

[2]
[https://news.ycombinator.com/item?id=8567205](https://news.ycombinator.com/item?id=8567205)

~~~
eridal
nice catch!

I remember some thread from the Safari's early ages where they hid the user-
agent, to avoid raising attention.

------
frankacter
>If robots instructions don't mention Applebot but do mention Googlebot, the
Apple robot will follow Googlebot instructions.

So if I set in my robots.txt to disallow all bots except Googlebot, Applebot
will index anyway? I don't think I like that precedent.

~~~
Bulk70
Serious question, because I can't imagine your use case - under what
circumstances would you wan't to block all bots except one?

~~~
mbel
I'm not an expert but my guess is: limiting bot traffic, but keeping the site
available for the most popular search engine.

~~~
mrweasel
My experience is that the worst bots don't respect robots.txt anyway.

Getting crawled by the major search engines typically isn't that bad, they
tend to know what they're doing. Getting hammered by some crappy local search
engine is what's annoying.

We don't limit any bots, except once where we completely blocked Eniro in our
firewall. Google, Bing and a ton of other could index at the same time, with
no issue. Eniro for some reason decided to just index way to much at once, no
reaction to robots.txt and no reply from the email they so kindly included in
the headers.

But I see your point, it's just a bit sad when Google has become "The
Internet".

~~~
karmakaze
I thought FB was the internet. Googlebot is just the Kleenex of indexers.

------
phn
Apple is making a smart move by not wanting to depend on anyone else.

Results may suck in the beginning, but well, competing in hard stuff is hard,
this is another apple maps. Hopefully they won't get bashed so hard since this
is not so user facing.

~~~
onion2k
_Apple is making a smart move by not wanting to depend on anyone else._

That's one way of looking at it. A different one would be that the wealthiest
company in the world could work with practically anyone and get a better
product than they could build themselves more quickly with more features. The
idea that they have to do everything in-house to get the best is paranoid and
stupid.

For example, rather than build AppleBot, why couldn't they pump a few billion
into DuckDuckGo to get use of DuckDuckBot? Or fund archive.org to access their
index? Or buy in to commoncrawl.org?

It would be possible for Apple to use its fortune to benefit both their
customers _and_ the world. Google and Bing are not the only options.

~~~
bad_user
Is DuckDuckBot for real? While I'm sure they are diversifying, DDG's results
are still based on Bing.

~~~
thomasfoster96
As far as I know DuckDuckGo has mostly been using Yandex's results lately.
DuckDuckBot is a real thing (and has been for a long time), but it doesn't do
much except power their anti-spam signals.

------
joelrunyon
This is interesting.

A while back, I think either Cook or Jobs mentioned that Apples makes PRODUCTS
and doesn't sell ADS.

If that's true (and stays true) AND this is the beginning of a search engine
for them, it's going to be VERY interesting to see what it looks like.

~~~
hahainternet
Apple already sells ads, and in fact restricts some technology to advertising
partners. WebGL, full screen ads etc. They also have a patent on unskippable
ads.

I don't know why people don't treat statements like that with enough cynicism.

~~~
liviu
WebGL is no more restricted technology in iOS 8+. Btw it was restricted only
in Safari.

~~~
hahainternet
Sure, but I find it quite unfair to say Apple doesn't sell Ads when they not
only own their own exclusive Ad network, they control their entire platform,
own many ad related patents and actively restricted access to new technologies
to their advertisers in order they could outcompete anything else on iOS.

~~~
happyscrappy
They don't outcompete anyone on iOS. Advertisers hate iAds because they don't
have as much access to user data as they do with Adwords etc.

------
JDDunn9
Please let this be the beginning of an Apple search engine. We really need
some better alternatives to Google.

~~~
BozeWolf
My question is: why do we need an alternative really? What is better when even
more companies index everything and perhaps screw with your privacy? My
intuition agrees with you though. More companies == more competition. At some
point privacy might be a selling point for them.

On the other hand: it is kind of crazy that when I read about "indexing the
web" it starts ringing all kind of privacy bells, while apple's incentive
might not even be to violate people's privacy.

~~~
mrweasel
When one company have more than 90% market share then: Yes, we do need an
alternative. (Google market in search has been reported to be as high as 94%
in countries like Denmark).

If a company manages to upset Google then that company ceases to exists
online, regardless of the validity of Google reason to blacklist that company.
Ideally no search engine would be above 20% market share (20% being a random
low number I just made up).

Right now websites and marketing material/money is directed at Google
exclusively, making it continually harder for new search companies to succeed.

~~~
PhilipA
I agree with you. Google has too much power over small webshops. If you
somehow come in bad standing at Google you can just shut down your operation,
because you won't make much money.

------
bambax
> _please contact us at: Apple-NOC “at” apple.com_

Can't Apple build itself some spam protection??!? Search is harder than this.

------
abritishguy
Based on logs it would seem that the crawler, or at least parts of it are
written in golang. When following some redirects the useragent would be
cleared and would identify itself as golang.

------
carlosrg
Introducing Apple Web Search. Only available on iOS and OS X devices.

------
mechazawa
>If robots instructions don't mention Applebot but do mention Googlebot, the
Apple robot will follow Googlebot instructions.

This is nice to see for a change. There are a lot of search engine bots out
there and forgetting a a lot of them is easy to do.

------
GameTheory8
Will this be more focused on improving search for digital products (App Store)
or more toward creating a comprehensive catalog for digital/physical products
to index + download/purchase?

------
danieleggert
Apple trying to get do what Google does faster, than Google doing what Apple
does.

In other words: Both Apple and Google in the mobile OS and device space. Both
Apple and Google in the (mobile) search space.

~~~
Retr0spectrum
Has anyone really been far even as decided to use even go want to do look more
like?

------
eridal
small-letter footprint:

    
    
        Risks are inherent in the use of the Internet

------
skrowl
I wonder if Apple Search will be as close to Google Search as Apple Maps is to
Google Maps.

That is to say, not very close at all.

------
crapplesrch
If you can't see the expected results, you are probably searching it the wrong
way! But, it's so shiny. Yes, but all the results seem Apple specific?

