
Applebot - jonbaer
https://support.apple.com/en-us/HT204683
======
jd20
Some fun facts:

\- Applebot was originally written in Go (and uncovered a user agent bug on
redirects, revealing it's Go origins to the world, which Russ Cox fixed the
next day).

\- Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in
an office. Those four Mac Pro's could crawl close to 1B web pages a day.

\- In it's first week of existence, it nearly took Apple's internal DNS
servers offline. It was then modified to do it's own DNS resolution and
caching, fond memories...

Source: I worked on the original version.

~~~
ospider
> It was then modified to do it's own DNS resolution and caching, fond
> memories...

Unlike other languages, Go bypasses system's DNS cache, and goes directly to
the DNS server, which is a root cause of many problems.

~~~
tylfin
Yeah, I've never had to implement my own DNS cache for a language before...

If you're on a system with cgo available, you can use `GODEBUG=netdns=cgo` to
avoid making direct DNS requests.

This is the default on MacOS, so if it was running on four Mac Pro's I
wouldn't expect it to be the root cause.

~~~
jd20
It's possible that wasn't the default setting on Macs back then. I don't know
that cgo would be a good choice either, if you're resolving a ton of domains
at once. Early versions of Go would create new threads if a goroutine made a
cgo call, and an existing thread was not available. I remember this required
us to throttle concurrent dial calls, otherwise we'd end up with thousands of
threads, and eventually bring the crawler to a halt.

To make DNS resolution really scale, we ended up moving all the DNS caching
and resolution directly into Go. Not sure that's how you'd do it today, I'm
sure Go has changed a lot. Building your own DNS resolver is actually not so
hard with Go, the following were really useful:

[https://idea.popcount.org/2013-11-28-how-to-resolve-a-
millio...](https://idea.popcount.org/2013-11-28-how-to-resolve-a-million-
domains/)

[https://github.com/miekg/dns](https://github.com/miekg/dns)

------
TheEnder8
The context for this is that Apple is rumored to be starting their own search
engine

[https://appleinsider.com/articles/20/08/27/apple-may-
launch-...](https://appleinsider.com/articles/20/08/27/apple-may-launch-its-
own-web-based-search-engine)

~~~
paxys
I can't believe this hasn't happened yet honestly. Apple Maps has been out for
7 years now. Also a bit troubling if it gains traction, considering Apple has
been a lot more heavy handed in keeping content they don't approve of out from
their ecosystem.

~~~
banachtarski
In all seriousness, Apple may do certain things well, but cloud-anything just
doesn't appear to be in its DNA. Maps, mail, iCloud, Timemachine, etc. Pretty
much every service I can think of is laden with bugs, quirks, actual data loss
risk, or is slow enough to be unusable. I'm not even remotely surprised that
there is no "Apple search" yet.

~~~
0xEFF
Maps is a search problem and Apple Maps has gotten quite good. It’s my go to
now after a decade of Google maps.

~~~
ako
It still lacks cycling, that is usually the reason fo me to open google maps
on my iphone.

~~~
SOLAR_FIELDS
As of the next iOS release that will no longer be the case

~~~
ColoradoDev
As long as you live in New York City, the San Francisco Bay Area, or Shanghai
and Beijing.

------
kentf
I remember when I rolled my eyes at Apple making a Maps product. I thought it
was a fruitless, dumb idea. I prefer Apple Maps to Google Maps now... don't
underestimate Apple's ability to plant a flag and move it inch by inch each
year.

~~~
actuator
Any specific reason why?

In my personal experience Google Maps has better navigation data, better place
information and overall much better maps. The user contributed content makes
the platform such a pleasure to use even if you are in remote parts of the
world.

~~~
baddox
On my iPhone I think Apple Maps has a vastly better navigation UI. Part of
that is its first-party ability to turn the phone screen on when issuing
instructions for upcoming maneuvers, which I much prefer to having the screen
on constantly for a multi-hour trip (not to mention working well with Siri).
Part of it is the overall aesthetic of the map features. I really like how it
highlights traffic lights along your route to let you easily count how many
lights are remaining until you need to turn.

That said, even here in the Bay Area I tend to do a sanity check of the route
on Google Maps first if I’m taking an unfamiliar route. Just a few months ago
a section of I-80 was closed just south of SF and Apple Maps had no clue! I
believe it was a fairly last-minute schedule change for some planned
construction due to vastly reduced traffic during the lockdowns.

~~~
iscrewyou
I’ve used Google Maps since it first came out on the nexus one. I recently
made the switch to Apple Maps. And it’s like stepping into a new world.

There’s the maps layout and where I want to go and my route. That’s it. Google
Maps has ads. Switching between Apple Maps and Google Maps really shows how
the Google product has friction and information overload. We are talking about
a Maps app. Less is better unless I ask for it. Apple has nailed the color
scheme, layout, and information density. If I need more information that Apple
Maps doesn’t provide, I double check google maps. On a long trip, I will check
routes from both apps. Apple Maps and google maps show the same route and same
time and same traffic congestions for me. I rely more and more on Apple Maps
now.

------
dvt
Is there any reason this is getting traction on HN? Applebot has been
confirmed for at least 5 years[1][2].

[1] [https://appleinsider.com/articles/15/05/06/apple-details-
new...](https://appleinsider.com/articles/15/05/06/apple-details-new-applebot-
web-crawler-used-by-siri-and-spotlight)

[2] [https://appleinsider.com/articles/15/05/06/apple-
challenges-...](https://appleinsider.com/articles/15/05/06/apple-challenges-
google-with-growing-web-search-program-fueled-by-topsy-acquisition)

~~~
cmckn
The page was updated in July with much more detail.

~~~
dmix
A random older archive from last year:

[https://web.archive.org/web/20190429220555/https://support.a...](https://web.archive.org/web/20190429220555/https://support.apple.com/en-
us/HT204683)

What seems to be new:

\- "About search rankings" section listing its purpose

\- Expanded bot identification and verification information

------
sollewitt
Google needs the competition. Search it's getting more and more laden with
ads, and I have found the kind of specific technical searches it used to excel
at are increasingly on page 2 after articles and videos and product listings.

~~~
karlshea
Not to mention the dozen poorly-formatted Stack Overflow and GitHub issue
aggregators that show up before the actual Stack Overflow and GitHub results.

~~~
jmnicolas
Ah yes, I noticed that recently. To add insult to injury mine are poorly
translated in French.

------
bcherny
I’m hearing a lot of theories in this thread about Apple building a search
product.

Yes, that’s possible, but Applebot is used in a lot of ways today that are
wholly unrelated to search. (If search consists of crawling, indexing,
ranking, retrieval, frontend, etc., Applebot only does the crawling part.)

Applebot is used for generating attachment previews in iMessage (eg. Send
someone a URL — the preview is from Applebot crawling it). From the docs, it
sounds like it’s also used for similar previews in Siri.

~~~
VWWHFSfQ
Apple is 100% building a search engine.

But do you remember when Apple launched their maps? A decade later and it's
still not nearly as good as Google maps. But it's "good enough". I imagine
that's what they're going for with this new search service

~~~
GeekyBear
The data that Apple previously paid TomTom and others to use was certainly
subpar compared to Google's data.

However, Apple started collecting their own data in 2015, and rolling out
their own maps in 2018.

>Apple is filling its map with so many [details] that Google now looks empty
in comparison and all of these details create the impression that Apple hasn’t
just closed the gap with Google—but has, in many ways, exceeded it.

[https://www.justinobeirne.com/new-apple-
maps](https://www.justinobeirne.com/new-apple-maps)

Apple's map data now covers the US and its territories and data for the UK and
Ireland are currently in testing.

[https://www.justinobeirne.com/new-apple-maps-
expansion-9](https://www.justinobeirne.com/new-apple-maps-expansion-9)

------
ricardo81
Think the summary from what I've read the past few days

\- It's established applebot is already used for Siri, but there is no general
web search

\- Much commentary diverts to how poor Apple maps (is?) was.

\- Google pay apple a handsome amount of money to be the default search engine
on Apple devices

\- If Apple were no longer to accept that payment, about 80% of people would
still use Google anyway, given a choice [0]

\- Googles high earnings per search means they can offer more than any other
search engine, like Bing, DDG et al

\- People would generally like more competition in the search space and Apple
has the means to do it

\- Anecdotally, there isn't a drop in replacement for a search default other
than Google that would meet the quality requirements for end users. In this
case, the only other alternative would be an Apple search engine.

\- As per recent articles, there does seem to be more activity from Apple wrt
applebot crawling and hiring

[0] [https://spreadprivacy.com/search-preference-menu-
research/](https://spreadprivacy.com/search-preference-menu-research/)

~~~
simonh
If they get a choice with no silent default yes, but if the default is set for
them and they have to go into preferences to change it the results are almost
completely reversed.

This is why Apple Maps is so heavily used on iOS. Most people don't know or
don't care. In the first year after the launch of Apple Maps it hit over 65%
of maps usage on iPhones in the UK, even though back then the UK maps were
awful.

------
bigdict
This reminded me of the fact that all of 17.0.0.0/8 was assigned to Apple.

~~~
xmprt
That's crazy. So Apple owns 1/256 of all IP Addresses?

~~~
bigdict
Yes, and a few other companies/agencies enjoy the same privilege:
[https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addre...](https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_address_blocks)

~~~
xmprt
AT&T and Apple make sense but why is Ford on that list. And how did Comcast
get theirs in 2005? I guess if you just ask very nicely, anything can happen.

------
30minAdayHN
> Aggregated user engagement with search results

My bad experience is generally rooted in the above criteria. It's particularly
problematic on news sites. For example, if I click a news article with certain
political association, I keep seeing only news from that political side. This
creates an echo chamber which makes uninformed believe that 'whole world' is
thinking that way. It definitely make sense to increase more engagement, but
might not be totally appropriate in all instances. Curious to see how it works
in live (if the apple search does become a reality)

~~~
drivebycomment
The only serious study I know of on the information bubble says rather the
opposite is true:
[https://www.pnas.org/content/117/6/2761](https://www.pnas.org/content/117/6/2761)

~~~
30minAdayHN
Thanks for the reference. It's good to know that there is research backing SNS
(social network sites) exposes us to more diverse news contrary to popular
belief. Size of the data set seems to be big enough to rely on the results.

I'm wondering if the same holds true when that experiment is conducted in
isolation. For example, if we take any particular news site, if a user is
repeatedly visiting the same news site, then my hypothesis is that the kind of
news one sees will converge onto a specific political view point. It might
vary based on each news site - google news vs apple news based on how they are
optimizing for users.

In reality, the isolated experiment might be moot as users do end up on news
site from SNS which ultimately leads to recommendation algos picking up
diverse signals to show diverse results. Given that, whether the algo itself
is diverse in isolation or not wouldn't matter as the environment that feeds
it is diverse. Does it make sense?

~~~
drivebycomment
Good questions. I suspect diverse news may not actually result in diverse
perspective anyway. I wouldn't be surprised significant fraction of people
simply don't change their opinion, even while consuming news from diverse
source. So it may be doubly moot in the end.

------
Razengan
On a related note, since there are video podcasts, I've always wondered why
they haven't been positioned as an alternative/competitor to YouTube by now.

A lot of YouTube channels and types of content could be replaced by video
podcasts, and be more convenient for viewers.

As for monetization, Apple wouldn't even have to build an ads platform;
content creators could embed ads themselves into the content, as they
currently do for audio podcasts.

------
sbahr001
This isn't anything new. Apple has had this bot as far back as 2015.
[https://searchengineland.com/apple-confirms-their-web-
crawle...](https://searchengineland.com/apple-confirms-their-web-crawler-
applebot-220423)

Everyone appears to be nervous because of the supposed search engine. I
believe this was part of their initiative to improve Siri results; when the
service was even worse.

------
protomyth
_If robots instructions don 't mention Applebot but do mention Googlebot, the
Apple robot will follow Googlebot instructions._

Is that pretty much normal?

~~~
kohtatsu
It's a courtesy, afaict.

~~~
moonchild
Rather the opposite. Many robots.txt will disallow * but allow googlebot. This
allows them to crawl those sites without getting flak for disregarding
robots.txt completely.

------
judge2020
I don't see this crawling my company's site much, but it's there (67 crawls
over the past month) [https://i.judge.sh/well-
off/Dash/chrome_yquafZlF2U.png](https://i.judge.sh/well-
off/Dash/chrome_yquafZlF2U.png)

------
jmull3n
I wish Apple would acquire Mozilla, use it to improve Safari and then create a
privacy first search engine.

~~~
Polylactic_acid
The most FOSS hostile company in the world acquiring the last properly open
source browser does not sound good.

~~~
jmull3n
Being open source will be irrelevant if nobody uses it anymore.

[https://gs.statcounter.com/browser-market-
share/desktop/worl...](https://gs.statcounter.com/browser-market-
share/desktop/worldwide/#monthly-201002-202008)

------
epaga
Given that the App Store search has one of the worst search algorithms in a
major search engine that I know of, I am not holding my breath for an Apple
search product...on their hand, when Apple does get something right, they hit
it out of the park.

~~~
mrtksn
I keep hearing that but when I dug a bit into it to do some ASO, the
impression I got is that it's simply not easy to game in name of Search Engine
Optimisation because the consensus is that the search engine only looks into
the App Title, subtitle, description and keywords and the ranking is affected
only by the App's performance. SEO people seem annoyed by Apple that they
cannot get their wallpapers app for the "stocks" keyword, so I have a very
little sympathy. When they complain that App Store Search is dumb, they
essentially mean that If I want to rank up my wallpaper app "stocks" and
figure a way to people use it every day. They have all kinds of tricks for
Google Play but the ASO tutorials are rather dull, recommending stuff like
picking an appropriate title and doing A/B tests on the keywords.

As a user, I tend to be quite happy with the search results in the App Store,
I don't remember any time where I could not find the thing I am looking for.

------
simonh
I suspect this is all about Siri. They went from being top of the game in
personal assistants to being a second stringer. Apple can't compete with
Google Assistant without the data set and analytics. Personal assistants and
web search are two sides of the same coin.

There are reports that right now Applebot is being used to scrape the content
used by Siri. If youre already scraping the web, indexing it and building
search and analytics services on top of it for Siri, user facing search is
just a short step to take.

------
mgh2
Here is a discussion from last week:
[https://news.ycombinator.com/item?id=24290613](https://news.ycombinator.com/item?id=24290613)

------
spectramax
This is one of the most exciting developments in the history of the internet.

Apple has always sided with privacy and while DDG toots it, they’re too small
to build an independent service.

I am thrilled, excited and perhaps surprised why it took so long for the most
used website for large majority of the people in the world to see some
competition.

Please don’t fuck this up, Apple.

------
amelius
Since this was posted on "support.apple.com", I assumed this to be about a
robot doing customer support.

------
achairapart
I think Applebot is nothing new (apart from the name). It's been years now
since I found in my logs traces of crawlers from Apple IPs with an iOS 8
Safari-like user-agent. Maybe, well, since iOS 8 at least.

Edit: see here @jd20 thread for insights and confirming this.

------
SethTro
Any idea what Apple is using this for?

The bottom of the page says "Apple Search may take the following into account
when ranking web search results:", is this another confirmation that Apple is
developing a search engine?

~~~
why_only_15
Apple already has a search engine, called Parsec, that's used when you search
in spotlight.

------
Jaruzel
I'm keeping an eye on siri.com - right now it redirects to a sort-of-holding
page on apple.com... but I can totally envision that URL becoming the home of
their new search engine.

------
mattfrommars
I am mega curious how does Apple circumnavigate problem of getting flagged for
crawling website. Doesn't CDN cloudfare eventually flag them?

If proxies, what is their source for proxies.

~~~
ricardo81
Likely whitelisted as a 'good bot' in Cloudflare. If a bot isn't well known,
doesn't respect robots.txt or crawls too fast then much more likely to get
flagged, but applebot are none of those.

------
mcemilg
I was scared while writing crawlers to various sites since I don't know how
far it's legal, but Apple says look, this is my crawler.

------
dvduval
Can we just start putting "firewall" after the name? We have the Google
firewall and the Apple firewall and the Facebook firewall.

~~~
aembleton
How is a crawler a firewall?

------
fireattack
>Customizing robot.txt rules

Shouldn't it be "robots.txt"? It even says so below.

------
throwaway4good
So when is this going to replace Google as default search engine in Safari?

------
m0zg
They're going to fail, yet pretend that they succeeded. Examples: Siri, Maps,
Home.

~~~
Polylactic_acid
Siri and maps work really well for me.

~~~
m0zg
Have you compared them to Google counterparts? There's really no comparison.

~~~
Polylactic_acid
I have used both and they both 100% cover my use cases. Gmaps could be 1000x
better but it doesn't matter because there is nothing more I need that isn't
in Apple maps.

------
segmondy
Apple can't built software for shit. They just want ad dollars. What does
Apple care about providing search? They don't.

------
w-ll
I'm going to slow all traffic with "Applebot" in the user agent by 30%.

Might do that same for all iOS traffic.

~~~
sethaurus
What would be your desired outcome from this?

~~~
tinus_hn
Of course to then complain that Apple and their ‘monopoly’ doesn’t get them
the exposure they imagine their users are demanding

