Hacker News new | past | comments | ask | show | jobs | submit login
Applebot (support.apple.com)
408 points by jonbaer 23 days ago | hide | past | favorite | 304 comments



Some fun facts:

- Applebot was originally written in Go (and uncovered a user agent bug on redirects, revealing it's Go origins to the world, which Russ Cox fixed the next day).

- Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day.

- In it's first week of existence, it nearly took Apple's internal DNS servers offline. It was then modified to do it's own DNS resolution and caching, fond memories...

Source: I worked on the original version.


> It was then modified to do it's own DNS resolution and caching, fond memories...

Unlike other languages, Go bypasses system's DNS cache, and goes directly to the DNS server, which is a root cause of many problems.


This is true but a little misleading. On Windows Go uses GetAddrInfo and DNSQuery which does the right thing. But on Linux there are two options: netgo and netcgo -- a pure Go implementation that doesn't know about NSS, and a C wrapper that uses NSS.

Since netgo is faster, by default Go will try its best to determine if it must use netcgo by parsing /etc/nsswitch.conf, looking at the tld, reading env variables, etc..

If you're building the code you can force it to use netcgo by adding the netcgo build tag.

If you're an administrator the least intrusive method I think would be setting LOCALDOMAIN to something or '' if you can't think of anything which will force it to use NSS.


Yeah, I've never had to implement my own DNS cache for a language before...

If you're on a system with cgo available, you can use `GODEBUG=netdns=cgo` to avoid making direct DNS requests.

This is the default on MacOS, so if it was running on four Mac Pro's I wouldn't expect it to be the root cause.


It's possible that wasn't the default setting on Macs back then. I don't know that cgo would be a good choice either, if you're resolving a ton of domains at once. Early versions of Go would create new threads if a goroutine made a cgo call, and an existing thread was not available. I remember this required us to throttle concurrent dial calls, otherwise we'd end up with thousands of threads, and eventually bring the crawler to a halt.

To make DNS resolution really scale, we ended up moving all the DNS caching and resolution directly into Go. Not sure that's how you'd do it today, I'm sure Go has changed a lot. Building your own DNS resolver is actually not so hard with Go, the following were really useful:

https://idea.popcount.org/2013-11-28-how-to-resolve-a-millio...

https://github.com/miekg/dns


And Java.

As I understand it, Go and Java are both trying to avoid FFI and calling out to system libs for name resolution.

I tend to always offer a local caching resolver available over a socket.


>- Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day.

Considering the timeline, are those Trash Can Mac Pro? Or was it the old Cheese Grater ?


Trash cans :)


>Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day.

The scale of web stuff sometimes surprises me. 1B web pages sounds like just about the daily web output of humanity? How can you handle this with 4 (fast) computers?


Computers are very fast. We just tend to not notice because today's software is obese.


Yes, let's all run separate web browsers as the application and run our own JavaScript inside our browser. Who cares if there's 5 other "apps" doing exactly the same!

Insanity.


Multiple tabs/browser windows is similar and generally not an issue.


I think they were referring more to apps like Slack and other similar JS browser/ JS based apps which run separate from the browser. Maybe I'm being generous? Slack is certainly itself a beast.


Yes this is precisely what I meant. eg. Electron apps, Slack, VS Code, Skype etc. etc. ad nauseum


Doesn't it depend on a lot of things? For example you can only do head requests to see if a page changed since a given timestamp. If not then there is no need to process it.


For anybody wondering how:

The HTTP HEAD method requests the headers that would be returned if the HEAD request's URL was instead requested with the HTTP GET method. For example, if a URL might produce a large download, a HEAD request could read its Content-Length header to check the filesize without actually downloading the file.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HE...


Typically you wouldn’t bother with a HEAD request, you’d do a conditional GET request.

When you request a page, the response includes metadata, usually including a Last-Modified timestamp and often including an ETag (entity tag). Then when you make subsequent requests, you can include these in If-Modified-Since and If-None-Match request headers.

If the resource hasn’t changed, then the server responds with 304 Not Modified instead of sending the resource all over again. If the resource has changed, then the server sends it straight away.

Doing it this way means that in the case where the resource has changed, you make one request instead of two, and it also avoids a race condition where the resource changes between the HEAD and the GET requests.


Do a lot of random pages return etags? I've only ever seen them in the AWS docs for boto3


nginx sends it by default for static files (example: hacker news [0]), I assume other web servers do too.

[0] https://news.ycombinator.com/y18.gif


I am particular curious about data storage.

Does it use a traditional relational database or another existing database-like product? Or is built from scratch just sitting on top of a file system.


Nope, you don't really need a database. What you need for fast, scalable web crawling is more like key-value storage: a really fast layer (something like RocksDB on SSD) for metadata about URL's, and another layer that can be very slow for storing crawled pages (like Hadoop or Cassandra). In reality, writing directly to Hadoop/Cassandra was too slow (because it was in a remote data center) so it was easier to just write to RAID arrays over Thunderbolt, and sync the data periodically as a separate step.


Interesting stuff. I've used libcurl to crawl at that kind of pace, is the parsing/indexing separate from that count per day? Also interested in how you dealt with DNS and/or rate limiting


I've done similar at smaller scale. Instead of messing with underlying DNS or other caching in our code we just dropped a tuned dnsmasq as the resolver in front. The crawler had a separate worker to fill hosts so it was mostly hot when the crawler was asking.


In my case I was just fetching the home page of all known domain names. First issue I'd noticed was ensuring DNS requests were asynchronous. I wanted to rate limit fetching per /28 IPv4 to respect the hosts getting crawled, but couldn't really do that without knowing the IP beforehand (and keeping the crawler busy) so ended up creating a queue based on IP. I used libunbound. Found that some subnets have hundreds of thousands of sites and although the crawl starts quickly, you end up rate limited on those.

Also interested at the higher end of the scale about how 'hard'/polite you should be with authoritative nameservers as some of them can rate limit also.


Roughly estimating, each Mac Pro could crawl around 3k pages per second.


Which is not possible


Say the average web page is 100kb, and assuming gigabit connection in the office, then that's about a thousand pages per second. If the office switch is on 10gbit that would work out to 4000p/s naively counting. But we're in the same order of magnitude for the speed even on gbit, and we're not accounting for gzip, and the actual average page size might be a bit lower too.


Everything was on 10gigE. The average page size was around 17KB gzipped. Everything's a careful balance between CPU, memory, storage, and message throughput between machines.

Apple's corporate network also had incredible bandwidth to the Internet at large. Not sure why, but I assumed it was because their earliest data centers actually ran in office buildings in the vicinity of 1 Infinite Loop.


The average is a lot closer to 25KB IIRC, gzipped


Why not?


Can you share some more details about the current state? Is it still written in Go?


No idea, it's been years since I last worked on it. It was also not the only Go service written at Apple (90% of cloud services at Apple were written in Java), though it may have been the first one used in production.


And I sit here kind of shocked that Apple would use Java for anything, backend or not. I thought Apple had a strong preference for owning its own tech stacks, whether that be ObjC/WebObjects or later Swift...


I think WebObjects was supporting Java even before it came to Apple from Next. In the early days, many of Apple's services built with WebObjects even ran on Sun server hardware, and XServe's. But nowadays it's all commodity Linux hardware, like you would find in any data center.


WebObjects has been fully Java since version 5 was released in 2001: https://en.wikipedia.org/wiki/WebObjects#WOWODC

Apple's server stack has been primarily Java for about 20 years.


Not sure why you'd be shocked, it's a solid language for enterprise services like Apple offers, and their other languages - C/C++, Objective-C, Swift - aren't very kind for web services.

Great use case for Go though, especially its concurrency features for web crawlers. I reckon Scala could work too, although it's a lot more complicated / clever.


Out of curiosity, why would C or C++ not be good for web services?


I would guess because the input sanitizing requirement is harder for the web; having a stackoverflow when running locally requires the attacker to execute locally -- having a use-after-free from port 80 would be a much wider audience


Some Apple services were written in C/C++. One downside is it's very hard to source engineers across the company who can then work on that code, or for those engineers to go work on other teams.


Apple employs the founder of the Netty project, who has given plenty of open talks about Apple’s use of Netty (which implies Java services). Same is true for Cassandra.


Apple had a very odd obsession with Java right after the NeXT purchase. WebObjects got converted and they tried to do a Java Cocoa. Both were worse than the original.


At the cocoa heads user group I heard that ruby is very popular for their services more recently.


Can you talk more about the specific? What kind of parsers did you guys use? How about storage? How often did you update pages?


You should check out Manning's "Introduction to Information Retrieval", it has far more detail about web crawler architecture than I can write in a post, and served as a blueprint for much of Applebot's early design decisions.


Nice, thanks for the recommendation!

The book is freely available online at https://nlp.stanford.edu/IR-book/information-retrieval-book....


With 1b pages per day I guess you needed 1gbit/s connections on each of those machines? Especially if they also wrote back to centralized storage.

I guess there are not many places where you can easily get 4GB/s sustained throughput from a single office (especially with proxy servers and firewalls in front of it). Is that standard at Apple or did the infrastructure team get involved to provide that kind of bandwidth?


Do you have a timeline of how AppleBot has evolved?


Was that including the ability to render js driven asynchronously loaded pages, including subsequent XHR requests? If so, it's beyond impressive.


Thanks for sharing mate. That is amazing insights!


Why did you leave Apple?


Sorry to be pedantic, but your misuse of apostrophes in an otherwise perfect text annoys me.

All three uses of "it's" should be "its".

And I would just write "Mac Pros" instead of Mac Pro's".


The context for this is that Apple is rumored to be starting their own search engine

https://appleinsider.com/articles/20/08/27/apple-may-launch-...


I know Apple's walled garden approach isn't for everybody, but I'd personally love to see them make a search engine that isn't built around Ads. The strategy also is consistent with their pursuit of vertical integration.

I use DuckDuckGo for search, and do trust them with privacy. But if Apple did a half-decent job at nailing the search experience, I'd switch.


I think it's too soon to assume that Apple's search engine would be accessible directly from the web, rather than just being a backend for search functionality from within iOS. After all, they don't host any Apple Maps equivalent to Google Maps on the web.


Actually they do offer maps for web, but the way you get to it is via DuckDuckGo. Example:

https://duckduckgo.com/?q=San+Francisco&t=h_&ia=web&iaxm=map...


Is this an exclusive DuckDuckGo and Apple partnership, or are there other websites that also use Apple Maps?


If you have an Apple Developer subscription, you should be able to use Apple Maps using Mapkit on any website - https://developer.apple.com/maps/web/



When I do that, I can't get directions, though. It refers me to Bing, GMaps, HERE (the Nokia one), and OpenStreetMap.


> I'd personally love to see them make a search engine that isn't built around Ads

The ads that Apple sells on the app store contribute $2 billion in revenue -- I don't think they'd build a search engine that isn't built around ads.


Which is pretty staggering, given that the ads only appear on search results within the App Store. The scale is amazing.


I find it unlikely Apple is going to compromise their their privacy first reputation to make an advertising platform which is competitive with Google's. I wouldn't be surprised to see advertising if Apple releases search, but it will likely be context (what you searched for) driven reliance rather than based on deep tracking. That would make it far less appealing to the skeezier advertisers. I'd expect something similar to what you see in News.

I can't see how Apple would even do the kind of search result pollution Google does now. In fact, I suspect the only reason Apple is launching advertising is because they see an opportunity to provide a better experience—namely one where you see organic results immediately.


Yes, and they've also been caught selling iTunes user data.

People thinking Apple aren't going to copy Google's revenue model to some degree are delusional.


If you’re referring to this lawsuit [0], then the plaintiffs provided no evidence and the court dismissed the case with prejudice.

[0] https://9to5mac.com/2019/11/26/accusing-apple-of-selling-cus...


A search engine that optimizes for good content instead of SEO-spam and ads would be a very welcome addition. Google seems to have given up on fighting malicious/spammy SEO (I'd expect them to identify and downrank spammy tricks like what the recipe websites do).


I'm in favour of competition for Google, but I can't be the only one who genuinely has no issue using Google? I see people complaining on HN quite a lot but genuinely the only thing I have - in years - struggled to find was the correct eBPF documentation but even then it only took a bit of nudging.


I think there are types of searches it does very poorly on, but also different people have different expectations. In my experience, if something is a product for sale, the first several pages are always places to buy it, which is never what I’m looking for, I already have online shops I like and saw the item there, or am searching from inside a store. I want reviews, forum experience, or even just pictures. for example, actual pictures of a bike rack installed on a bike like mine, not just the manufacturer’s steel floating in a white void pics.

In addition, google has become very aggressive with disregarding words. So a search like (a fake example, not tried the search, not a real place) Jim’s tasty pizza Albuquerque 90s where I’m searching for where a restaurant I remember as a child was located, and has maybe 5 real results from a local history Wordpress blog 15 years ago and Facebook posts, Google will just drop everything specific and give you results for Albuquerque pizza, and then mainly show ads for national chains up top. So I need quotes in the search pretty much always.


If you go into the advanced search options you can tell Google to explicitly search on what you type. I use it so often for the reasons you point out (because nowadays in their infinite wisdom they will helpfully ignore quotes too) it occurs to me I should either find an extension or script to just have it enabled by default.


I use Google and Google Maps plenty, though I'm one of the ones who might complain about their [lack of] privacy practices.

I use it for certain types of searches, and not others. I'm conscious of the fact that what I'm typing and searching, and subsequently clicking through will be piled on to the stack Google thinks is me. So it's usually tech searches, real estate/apartments, street view (property or exploring).

Health matters, personal matters, etc I tend to keep out of their search box.

Though more and more I tend to do a lot of cross-searching if privacy is not a concern.


Google is fine for me (I use it as fallback when DDG fails) for technical results but for anything else it's pretty bad (recipes are a typical example). It's not just the fault of Google to be honest - the internet itself has turned into a cesspool - however Google would be in the best position to curate that and discourage that behavior by penalizing the offenders' ranking.


Google ads don’t affect what search ranks. Those are two different entities. And google realllllly wants to get rid of search spam. But it’s insanely profitable to rank, so people try really hard and often beat google.


It's debatable whether Google Ads affect Google search ranking (it's impossible to prove either way considering how many factors affect ranking), but regardless of whether they currently influence rankings, there might be reluctance to change that due to Google's business model.

Google's job as a search engine is to provide relevant and quality content. Ads are spam and noise, so if Google does want to provide quality content they should be penalizing ads just like they would penalize keyword spam and similar unpleasant SEO tactics. Given their business model however it is not in their best interests to do so, so it's very unlikely that Google will do it.

Another company that does not make their money on advertising has a better chance of doing this.


Ads do get penalized for being a bad fit: they get a lower quality score and cost more.

Meanwhile, ads are distinct from search. The #1 result refers to the first result past the ads.


> Ads do get penalized for being a bad fit: they get a lower quality score and cost more.

My point is that as a user, ads, regardless of how "good" they are, are not what I am looking for when I click on a search result (I am talking about website ads and not ads on the results page itself) and thus a search engine that downranks pages with ads in favor of those without them would be beneficial for me.

It would be against Google's interest to implement that (as they'd also be fighting against their own ad network) but a different company not involved in the advertising business (and making high enough margins on other products) has a higher chance of delivering this successfully.


Millionshort is a good search engine for avoiding spam sites.


It would seem silly not to include ads based on the text string the user searches. Even without tracking/personalization those are insanely valuable.


I would love to see a decent search engine.

Google search has gone down the toilet. Most Google search results are ads or SEO optimized junk. It was funny once when I was searching for a medical term, all articles on first page were clearly SEO optimized and all articles were very similar from word usage, sentence construction, verbiage, like written by same person. I am surprised that Google doesn't filter out the results using some sort of similarity analysis to offer variety in results. I don't want to see all links giving exactly the same info.


Google is vulnerable. Never thought I'd be able to say that.


And the same rumours has been going on for years, and I seriously doubt that.

Reason is that Apple already has a Search Engine. It is called Siri. It is what you get when you ask Siri Questions. What I think Apple hopes to achieve is that get 80% of what you want, From may be Recipes, Simple Answers like Exchange Rate, Nutrients, Sport Scores... etc away from Google.

Letting you to Google the absurd, hard, questions that has less Data value.

All while collecting roughly $10 per user for Default Search Engine Placement in Safari.

And remember the Privacy stand Apple has giving less Data to Google.

Basically Apple is squeezing Google in every single direction. And pitching them against Azure and AWS for Cloud Services for discount.


I can't believe this hasn't happened yet honestly. Apple Maps has been out for 7 years now. Also a bit troubling if it gains traction, considering Apple has been a lot more heavy handed in keeping content they don't approve of out from their ecosystem.


In all seriousness, Apple may do certain things well, but cloud-anything just doesn't appear to be in its DNA. Maps, mail, iCloud, Timemachine, etc. Pretty much every service I can think of is laden with bugs, quirks, actual data loss risk, or is slow enough to be unusable. I'm not even remotely surprised that there is no "Apple search" yet.


They also run the App Store, Push Notifications, iAd, Apple News, Apple Music, Apple TV, Siri, Apple Pay and some of the largest systems for photo sharing, file storage, subscriptions, retail billing and more in the world. Saying Apple is bad at cloud services in general is incorrect.


The CDNs are fine, push and subscriptions are mercurial at best and you have no choice but to use, and Siri is an example of Apple's cloud weakness.


Mercurial is certainly a step above the antics of Loki, so count your blessings!


Ssssssssmokin'!


They do have plenty of high-volume services e.g. Siri, Siri Suggestions, Maps, Online Store, Apple Music that haven't had any major issues. And even iCloud is substantially better now than in the early days.

Most of your issues seemed like the old, rehashed ones from nearly a decade ago.


Their cloud-based Notes App is simply killer. Sure, it’s a simple app but I have had a flawless experience on it.

Photos is pretty great too, but I’m not sure how it compares to other photo services.


And yet they forced you to upgrade all your operating systems an all devices you own with the last Notes update (was it iOS 13?) or you'd lose data.


You wouldn't lose data. You wouldn't be able to access the new notes on the older devices.


Because old notes synced through CalDev which required an iCloud email address


Maps is a search problem and Apple Maps has gotten quite good. It’s my go to now after a decade of Google maps.


Apple Maps is my goto because Google Maps just hates showing street names for some reason. Zoom in so a street takes up 100% of your screen and it still won't display the name many times.

Obviously that's not the only reason, haha, it's gotten surprisingly good a few years ago. Public transit and cycling is better in Google (though nothing beats CityMapper for transit if it supports your city). But for walking and driving Apple Maps is my winner.


It still lacks cycling, that is usually the reason fo me to open google maps on my iphone.


As of the next iOS release that will no longer be the case


As long as you live in New York City, the San Francisco Bay Area, or Shanghai and Beijing.


That's coming in iOS 14, evidently.


So I'm using iOS 14 and just tried to pull it up. It has a message stating "Cycling directions are not yet available in [my city]"

A quick google search to see if it was simply because of beta or there is a limited roll out, produced this:

>New York City, the San Francisco Bay Area, and Shanghai and Beijing in China.


Look for the arrival of cycling in Apple maps upon the release of iOS 14 in the upcoming weeks


Just got an iphone last month and apple maps has worked perfectly every time for me. I'm not exactly a maps power user but I haven't seen an issue.


I find maps much more pleasant looking than Google Maps, and that's a big deal for me.


Lack of maps.apple.com is still problem for use it for primary map service.


define "good", the POI database is still woefully lacking outside the US.


You’re 2 for 4. Time Machine and Mail (separate from iCloud) are not cloud services.


I thought the same thing about Time Machine. It works really well with a local laptop that I plug in from time to time for this to work. However, is it considered Time Machine when an iDevice backs itself up to the cloud?

Mail is definitely a local app, even if they did screw the pooch pretty badly with that gmail bug they had a few years ago.


“However, is it considered Time Machine when an iDevice backs itself up to the cloud?”

No. Strictly speaking, iCloud doesn’t even do backups; it syncs data. Difference is that, with backups, you can get a copy back if you accidentally throw away a file. You typically also have multiple ‘old’ copies.

iWork apps (1) kind-of have backups in the form of versions that get synced to the cloud (https://support.apple.com/en-us/HT205411), but I think that’s implemented independently from iCloud’s syncing (You can use it to have multiple versions of a document locally, too)

(1) Third-party apps can use this, too (https://developer.apple.com/design/human-interface-guideline...), but I wouldn’t know how commonly this is used


> No. Strictly speaking, iCloud doesn’t even do backups; it syncs data

There’s literally a switch called “iCloud Backup” in Settings though. This isn’t a per-app data sync, it’s the full device restore that you can use if your phone gets stolen, dropped in the ocean, etc.

Although you should consider not using it for privacy reasons and running backups to a local machine instead, since last we heard Apple has the encryption keys and will happily hand the entire contents your phone to law enforcement given a warrant.


They have components that run in the cloud, and unfortunately, when I wrote "cloud services" I probably should have written "services that leverage the cloud."


I don't think Time Machine does, everything I've seen is local to the machine or across your local network.


As far as I am aware, neither Time Machine nor Mail have any components that "run in the cloud".


I have a vague sense that you're right, that the services team just doesn't perform as well as other teams.

However, Apple operates many services at scales few companies ever reach, and doesn't seem to have outages much more often than those other few companies. So they're clearly doing a lot of things incredibly well, in the if-things-work-you-don't-notice sense.

They don't seem to emphasize their cloud services like Google does, and Apple charges for theirs at a much lower level than Google does, but I'm starting to think my vague intuition is wrong.


My impression from Twitter “new job” announcements in the last 6 months is that Apple has seriously ramped up its kubernetes staff.


iCloud Mail and Photos have been nigh flawless for me for over 8 years now.


Also of note, Apple Maps in the iOS 14 beta just started accepting ratings and photos at map locations, rather than dumping you out to Yelp reviews. I'm happy with this change.

https://9to5mac.com/2020/08/26/apple-maps-reviews-inhouse/


I imagine the release of Apple Maps may have convinced them to go slow with this kind of thing.


Exactly this


There's nothing slow about this, it's already launched. I don't understand why people don't get this -- it even says so at the top of this page: "Products like Siri and Spotlight Suggestions use Applebot"

I don't know why anyone would think that Apple would make a web page search engine like Google. Apple has already launched the Apple version in your phones integrated into the software you use.


Maybe because they are desperate for a different search provider than Google?

I don't use Siri or any assistant, so I don't seem to use what I'm being averaged into using. Majority opinion here is that Siri is seriously lacking behind the other offerings. Apple won't be oblivious to this. If it is solely due to their back-end search engine, then making that a public search engine website won't do Apple any favors if the results are truly that bad.


Voice assisted search probably won't give you much feedback. With a web based search engine, users provide feedback by ignoring some results and clicking others. Therefore releasing a search engine could help them improving it and deliver results with Siri that people would've clicked on otherwise.


Why would they be desperate for using another search engine when Google pays them $8 billion+ a year?


Because Google makes enough money on ads from Google search to spend $8 billion on making it the default search on iOS devices


Apple is mostly a focused company. There is no money in search except advertising and user tracking. Apple has already completely failed at advertising once.

Yes I realize that the App Store has ads and it is a stain that should not have ever happened.


Well they've gone to the effort to build a search index and they have sufficient market power to push millions of user onto whichever search engine they'd like. It's essentially leaving money on the table not to take the final step of popping a web frontend on it and making it the Safari default.


It already is the Safari default. As you type your search, it gives you a Siri Suggestion. When you type a query into Spotlight, you use it.

Apple isn't an ad-supported business. It's not going to look like Google. It looks the way it's already released.


They are going slow, though.

That they use their own search for this is not a big headlining marketing feature so they aren’t really evaluated on it.

I do think that’s a direct consequence of their experience with Maps, though I guess their hand was more forced on Maps and they had to tell the world about it.

However, using Siri and search suggestion they can, bit by bit, replace search results from other sources with their own without telling anyone about it. It’s a more careful approach.


Google's results are already pretty clean. Search results are heavily filtered (e.g. copyright, adult content) so I don't think Apple would filter more. If anything, filtering more than Google could deteriorate results.

But the SEO market would certainly change if webmasters suddenly have to appeal to more than just Google to be successful.


I’m confused as to why it’s even happening now. A good index costs a fortune to build and maintain.

How does this benefit Apple? They’re a hardware company.


No, Apple is a 'full-stack' tech company. People buy their products both for their hardware and software. If they were just a hardware company, why would they bother making macOS, iOS, iMessage, Safari etc?


To sell more hardware, obviously. You wouldn’t buy a mac without macOS.

How does a search engine sell more hardware?


Haven’t you heard people whine about how Apple is so much about privacy but they take money from Google to have it as the default search?

They don’t have Google as the default search because of the money, they have it because they think it is the best. Even despite the privacy problems and the control this grants Google.

How does a maps service sell more hardware? Because using their own maps service Apple can completely control the experience.

Now maps was hard and search is much harder still so we may very well never see this released. But even then they can use this as leverage while negotiating with Google, both for money and for control.


> How does a search engine sell more hardware?

Because your claim about them being a hardware company is incorrect.

They market themselves as a privacy focused company. Privacy is all about software and services, not hardware. Search is something they could add to their portfolio to strengthen their privacy stance.


I would think that they can leverage a custom search engine in more flexible ways than Google to improve UX.

Also, Apples main revenue isn’t hardware but services IIRC. They’re not a hardware company.


You do not recall correctly.

Services are ca. 18% of Apple’s revenue, after years of huge spending and tons of dedicated effort to grow this segment. The fact that it is as high as a fifth is itself a monumental feat.

Hardware sales of the iPhone alone, excluding all computers, AirPods, and iPads, is over 40%.


I suspect the profit split is biased more towards services than the revenue split.


This is less meaningful when the vast majority of the services are only sold to people using their hardware.


The same reason why having a maps product benefits them. They want to have complete end-to-end control over a user's experience on their device, well beyond just the hardware. Having their biggest competitor dictate what happens whenever someone opens the web browser on an iPhone is a massive hole.


An index of enough of the web to keep 90% of users happy can be built for ~$10k of storage.

The last 10% quickly gets very expensive tho.


Hard drives are cheap enough that storing the index itself is the least expensive part of building it. Engineer time to make that index usable for a search engine is that much money every week.


One of the selling points of apple devices is getting away from google spyware. Every bit of spyware they can prevent the user from being exposed to is better.


Apple doesn’t care at all about the most common types of spyware. Nearly every app in the iOS App Store contains tons of spyware.

Apple’s view is that you consented to this sort of unaccountable invisible tracking by dozens of third parties (against whom you have zero recourse), when you agreed to the TOS of the App Store.

The data is aggregated, sold, and resold again.

https://techcrunch.com/2020/07/09/data-brokers-tracking/


They have a heading titled "About Search Rankings". Seems like a good indicator for a search engine to me.


Apple built their own search engine over 5 years ago, under the Siri / Spotlight umbrella. When people talk about Apple building their own search engine, they generally seem to expect a website dedicated primarily to web page results, but under the covers what powers Apple's Spotlight results is basically a search engine.

The big question would be what Apple would gain from a dedicated website for search results. Would people really switch to it from Google? Why would it be a better delivery mechanism for search results than Spotlight? Not sure the answers to these questions has changed much, from 5 years ago to today.


They have a lock on mobile devices for the western world. They can by default pry away search revenue from Google at a large scale. They have the cash reserves to see this done properly. They'd be crazy not to come onto Google turf. Their hardware lines are sagging revenue growth wise. Service revenue is their biggest growth area. I have no idea why they didn't do this before.


> They have a lock on mobile devices for the western world.

They don't though. There are only a couple countries where they're more popular than Android, and only barely there. There's no country where they have 60% market share.

https://deviceatlas.com/blog/android-v-ios-market-share


Or just like Iran's nuclear program, they can extract more concessions if they are permanently 1-2 years away from deploying a search engine. Google's spot as the default search in Safari already earns Apple billions per year.


It might look like that for people inside the walled garden but I suggest applying for an exit visa to have a look around the great wide world out there where you'll quickly find out that it is a whole lot more diverse than the picture painted inside the walls. Not only is it more diverse, those rumours spread about the constant onslaught of viruses on those poor creatures on the other side of that big, safe wall turn out to be untrue, most people seem to never have even seen one. Even stranger things will start to become clear, like those supposedly poor and oppressed outsiders needing far less money to get around the world than those inside the walls while at the same time having are larger choice of methods to navigate it. It wouldn't surprise me if you decided to stay, just don't forget to plan this carefully so your identity does not get stuck on the inside after which you won't be able to message your friends any more. It is a bit tricky but it is doable, many others have gone that way before.


Google pays Apple a reported $8 billion a year to be the primary search engine on iOS.

Apple’s hardware sells were also up slightly in every category year over year.

https://9to5mac.com/2020/07/30/apple-q3-2020-earnings/


Presumably Google are making quite a bit more on that traffic. Apple might just be eyeing those additional profits.


Through advertising? How would that be any better than what we have now.


It might not, I'm just trying to think of their motives.


Alternatively: Would people really switch back to Google if Apple changed the default search engine on iOS?


If Apple Maps has taught us anything, probably not. But Apple would first need to pour an equally large amount of resources into web search, they way it did for Maps.


The pay-off for doing so (ad revenue) is pretty huge. I'm surprised they have held off for so long.


Spotlight does result rankings.


Let's see if Apple is the first one to nail it. I love using DDG but their results are not close to the same quality as Google for me, especially with localized results and unspecific requests. And DDG is backed by Microsoft's Bing which invested billions into this. Though having run a few websites, Google's crawler easily crawls 10x the amount of all other crawlers combined if they don't detect page slowdowns. I never understood why Bing was so conservative when it comes to crawling.

I believe if anyone can build a search engine that can truly deliver the same quality as Google, the whole market could change. Google is dependent on search ad revenue like nothing else, if they lose that monopoly it could really change the company.


God that would be great, especially if it also ensures user privacy.


Bet it eventually becomes like Google but with less transparency. Apple will do the same stuff, but call it something different.


my first reaction , was oh god, please dont! considering how worst siri is, i dont want that.. but after reading other comments like native ad tracking etc etc it could be just fine.

But google pays Apple in billions to keep its search as default, what could be the key motivation for Apple to build its own engine? its a highly saturated market and we have Duck Duck Go for privacy based , bing for alternative and heck, even wolfram alpha for computational! what Apple can improve?


Based on usage of DDG, bing and wolfram alpha, I would think Apple can improve a great deal. I'm not saying it can necessarily achieve that, but I do think there is a lot of space to improve (indexing, ranking, UX)


That would be nice.


I remember when I rolled my eyes at Apple making a Maps product. I thought it was a fruitless, dumb idea. I prefer Apple Maps to Google Maps now... don't underestimate Apple's ability to plant a flag and move it inch by inch each year.


Any specific reason why?

In my personal experience Google Maps has better navigation data, better place information and overall much better maps. The user contributed content makes the platform such a pleasure to use even if you are in remote parts of the world.


On my iPhone I think Apple Maps has a vastly better navigation UI. Part of that is its first-party ability to turn the phone screen on when issuing instructions for upcoming maneuvers, which I much prefer to having the screen on constantly for a multi-hour trip (not to mention working well with Siri). Part of it is the overall aesthetic of the map features. I really like how it highlights traffic lights along your route to let you easily count how many lights are remaining until you need to turn.

That said, even here in the Bay Area I tend to do a sanity check of the route on Google Maps first if I’m taking an unfamiliar route. Just a few months ago a section of I-80 was closed just south of SF and Apple Maps had no clue! I believe it was a fairly last-minute schedule change for some planned construction due to vastly reduced traffic during the lockdowns.


I’ve used Google Maps since it first came out on the nexus one. I recently made the switch to Apple Maps. And it’s like stepping into a new world.

There’s the maps layout and where I want to go and my route. That’s it. Google Maps has ads. Switching between Apple Maps and Google Maps really shows how the Google product has friction and information overload. We are talking about a Maps app. Less is better unless I ask for it. Apple has nailed the color scheme, layout, and information density. If I need more information that Apple Maps doesn’t provide, I double check google maps. On a long trip, I will check routes from both apps. Apple Maps and google maps show the same route and same time and same traffic congestions for me. I rely more and more on Apple Maps now.


> which I much prefer to having the screen on constantly for a multi-hour trip

I’ve found myself swinging in the other direction. I tend to glance at Waze for my speed instead of my own speedometer, but I’ve been driving a mixed bag of cars lately.


For me having those stop signs and signals show up in navigation is big plus for Apple maps. I have a pixel phone but if I'm driving with my spouse I always use her apple maps


One great thing with the Apple Maps UI is that when using CarPlay it gives you a 3D view of the street. So you're really seeing buildings on the left and right, which has helped me not overshoot some destinations because I can 'see' the building in the map, and then irl.


Judging from the responses here, some of the reasons seem to be because Apple Maps is much more vertically integrated with iOS, and that doesn't seem like a good thing to me. Sure, you may get a better experience, but that's more likely because Apple is keeping Google Maps from getting as good than because it's a better solution, period. I have no love for Google but that's not a good model, and I feel like this kind of thing is making Spotify's case for Apple being a monopoly much stronger.


You don't need particularly tight integration with the operating system to make a good maps app; you'd want access to CarPlay, which Google Maps has, and you'd probably want a good sharing extension, which I don't think Google Maps has but could if Google cared about making better iOS apps than they seem to. Google is rather notorious for being very, very slow to adopt new iOS features in their apps, and it's definitely not because Apple is making it difficult for them to do so.

The history here is also at least worth a passing mention -- the iOS Maps app originally used Google mapping services. The reason Apple built their own back end is because Google refused to give them access to vector-based maps and turn-by-turn navigation unless Apple shared more user data with Google than they were willing to provide.


>but that's more likely because Apple is keeping Google Maps from getting as good than because it's a better solution,

I would say Waze is a solid counter-argument.


I also find that Apple Maps respects local mapping conventions. For example in New Zealand Google treats all state highways as equivalent and uses the same weight and colour line for all of them, but Apple Maps follows the NZ convention and assigns appropriate weights to the minor roads.

For those not familiar "state highway" simply means the central government agency pays for upkeep, rather than indicating any particular road quality or status.


Not OP, but I'm in the same boat. Agreed Google Maps' data is better. Their routing is sometimes better. But I still prefer Apple Maps in most cases. For me, it comes down to a few UX things:

* iPhone + Apple Watch combo is amazing to use w/ Apple Maps. Audible & haptic alerts on your wrist change how one interacts with the turn-by-turn directions, and it's an improvement.

* The fact that iPhone nav is available on the lock screen is a huge help. Doesn't require full unlock to get to the map.

* Apple Maps just seems to handle the accelerometer in the iPhone better. I found that Google Maps never knows what direction I'm facing/walking, and it jumps all over the place all the time. Apple Maps is just smoother and better to use.

The only thing that's missing in Apple Maps is bicycle directions, but apparently that's coming in iOS 14, which is great.


The main time I find myself using Apple Maps is when visiting China. Google Maps is blocked so the place data isn't as great, and the local apps are in Chinese, which I'm illiterate in.

I similarly find Apple Pay great for traveling, and easier to use than a credit card.


Same. I have no idea how I would navigate in China without an iPhone. All the other English language map applications are straight up unusable.


Not OP, but the Apple Maps + iPhone + Apple Watch combo is now my favorite way to drive with turn-by-turn directions. That alone won me over.


The little nudge on the watch is a nice addition


Yup, Google Maps has better data.

But, on my no-longer-bleeding-edge iPhone, the turn-by-turn directions suck. It's constantly telling me to turn onto a street I just passed, or rerouting to find a way to get me back to the street that I'm already on. The worst part is that I can't trust it even when it's right. If it says "in 300 feet turn left," I have to desperately look at the street names because it's quite possible that I actually have to turn in 30 feet.

It was such a relief when I realized that it's not my GPS hardware, Apple Maps gives directions perfectly. And the map data is way better than it used to be. Not as good as Google Maps, but good enough.


I hate Google maps when it shuts itself off with a 'journey complete' status while I'm still looking for the corner or the building that I was supposed to arrive at.


> better place information

Probably 75% of my Google Maps usage is to check business hours or grab a phone number; they've really got that down. Apple is catching up, though!


Yeah, I really liked when Pixel integrated the phone number bit into dialer itself. I can start typing a business name and it just pulls in from Google maps.

The iPhone I have now is a work phone, so don't use maps a lot, should see if it has improved.


I mostly use siri to call a business and it works every time for me.


Totally. In apple's own back yard, SF, business hours are too often totally out of whack, that you have to confirm in Google Maps each time.

And I am biased towards Apple Maps for privacy and usability reasons


I'm pretty sure Apple Maps is using Yelp's data for business hours -- those seemed to be mostly reliable until the pandemic. Now, not so much. I don't know if Google is doing better on that point.


Much much better voice directions; with Google, you have to check your screen often; with Apple, you just follow the voice


Google maps likes to wait until you 5m away from a turn before telling you you must cross 4 lanes to turn right


Its really small, but for me its traffic lights. It's SOOOO much easier to be navigating and say, turn in two lights, instead of "turn in the next three streets". Especially on busy roads when you need to be in a certain lane to make that left or right hand turn.

Google maps added this to their desktop app [1] and maybe their iOS app, but apple has had this for years.

Also Apple maps offers saved places in an offline, non-synced way. Google requires access to all my location data to save places (like web bookmarks) and that's just too privacy invasive for me.

[1]: https://arstechnica.com/gadgets/2020/08/google-maps-starts-w...


Even a few years ago I preferred the way a few-block area was represented on Apple Maps vs Google Maps. Apple Maps seemed to be much better at showing relevant landmarks and business names, where Google Maps would randomly select some businesses that happened to have an address there, but often no signage, etc.

Both applications would select the same number of businesses to show (obviously not showing ALL names due to density restrictions on the screen and clutter), but the Google ones were just so painfully arbitrary, and Apple's building skeletons seemed a lot more relevant/recognizable.


In my experience, Apple Maps has been more up to date with recent changes to roads here in Norway. There was a new tunnel here recently, and it was updated in Apple Maps when it opened, while with Google Maps it took a while.

I'm guessing that it depends a lot on the area. Google Maps is still probably better in most places, but Apple Maps is catching up and may be better in some populated places. It's impressive considering the head-start Google has.

Apple should add an option of OpenStreetMaps overlay or something. OSM has been better than Google Maps in mapping paths in the forest and such here.


I live in a bilingual country, GMaps decides location and street names at random, there is no way to pick which language to use.

I speak one of the languages, not the other. Guess which language Google decides is the correct one?


Not the parent commenter, but while I still use Google Maps over Apple Maps, I constantly have issues with the former.

I've put in business addresses or searched the business themselves, and then had it route me to a residential block behind the business on a completely different street, sometimes multiple blocks away from it. This is even after confirming the exact address is being shown and everything.

I only really use Google Maps because it's what I've been using, and I kind of like that its UI is a bit simpler with less animations and things than Apple Maps.


I don't have unlimited data, and I have found Apple maps is much lighter on data use than Google. I have also found that even if I don't have a solid connection Apple maps seems to still be capable of giving me directions quickly, whereas on the other hand with Google I may never get directions.


My main reasons for using Apple Maps are:

1 - It actually runs smoothly on my iPhone 11. Google Maps stutters and yanks. Yes, Google cannot build a performing maps app for the fastest phone in the world.

2 - It is a map. Not a yellow pages of coffee places, not an Instagram with amateur photos of locations.


Apple Maps has the unique and critical advantage of being operable while the phone is locked or sleeping.


Have you tried it lately? It gets better with each iteration, and the street photos are way better where they exist. The satellite and aerial photos are also better in most places that I travel too.

Also, the Google Maps app has gotten much more cluttered and complex than in past versions. They are still the gold standard map, but they regress over time.


My house has not been in Apple Map's index for approaching three years now. It's caused _all kinds_ of problems. Apple seems to drop the ball on the details that are needed to really get a product to be quality.

EDIT: I looked back in my emails and Twitter DMs and it's been 25 months at this point. So "approaching three years" is not accurate.


Have you tried reporting it? My parents recently moved into a new house built one year prior and I reported it a few days after they moved in. It took about 9 months, but they finally added the road in and populated ALL the addresses on the street, not just the one I reported. Google maps still does not have the roads or street numbers listed. Just wanted to throw that out there.

Edit: Also want to add that it has been two years since my parents moved in. Went in to Waze Map Editor and added the road myself, so their turnaround time was instant. Apple Maps took 9 months, like I said, but Google Maps still has no info on the street despite having all the houses in clear satellite view.


I replied with my interactions with Apple here: https://news.ycombinator.com/item?id=24338389

It did take Google a few months to add my complex. I'm not sure what an appropriate turn around time would be on something like this. But I do think it's less than "years".


Just read your reply. That's very frustrating to read -- hopefully the Apple employees come to their senses and add the building soon. It's also weird that they called you through FaceTime and still did nothing. Either way, sorry to hear about all the frustration you've been through.


What's the point of reporting anything to the Apple Maps team? I've reported stuff for a few years, and it hasn't changed nor have they had the basic courtesy to acknowledge the report.

It's not like it's some street address thing, they are missing a whole town.


I've gotten changes made by reporting stuff to the Apple Maps team. I remember years ago, my company's business location didn't show up correctly. I reported it, and within a week it showed up incorrectly in an entirely different way!


I’ve reported issues with POIs in the UK via the in-app features (“Edit Location” and “Report an issue”) and they were fixed within a few days.

In general, Google still has better POI data than Apple Maps in the UK, but the gap has closed a lot in recent years.


IIRC Apple Maps sources data from OpenStreetMaps, so contributing there should make the updates end up in Apple Maps, no?


Only in some places I believe!


try adding it in openstreetmap if it's not there already.


Can you add it?


Nope. You can report it's missing if you own an Apple device. I have done that (many times). I have also talked with Apple support on Twitter many times, and they even assigned a dedicated support person who called me several times. I facetimed with him and showed him how my complex, plus several of the nearby complexes, are not in Apple Maps but are in Google Maps. That was ... over a year ago. They still haven't fixed it. I finally gave up.

Whenever someone needs to come to my house I need to ask them to use Google Maps. The challenge from that comes from less techy people who don't know the difference. Many people just know their phone has a "maps app". I will admit, experiencing that more closely (how less technical people interact with their phones that is) was interesting fwiw.


Google Maps is making it easier on them by seemingly becoming worse and worse, in my opinion. The newest act of telling you it's changing your route if you don't click something is so idiotic it borders on malice. So going 70mph down the expressway, I need to take my eyes off the road to read the message and click a button asking it not to take action.

Further, it repeatedly picks a hard route. There's one park I go to that's easier to drive to using one few mile stretch of highway. Google insists it's faster to take these back, poorly lit, windy 2 lane roads. Maybe it is. But it turns a safe easy 3 turn trip into a nerve-wracking 10 turn trip. And it will, as noted above, keep trying to reroute me if I just ignore it and go the easy way.


Another thing google maps always does to me is telling me what the next turn is when its too late and I can't make it anymore. Using google maps with audio only is impossible.


Noticed that recently too - it insisted on taking stupid backroads and what not, instead of straight highway and no traffic. It seems to be getting dumber, or something.


I tried AppleMaps for the first time like 4-5 years ago. Feature-wise it wasn't where it is now but what annoyed me the most is that the GPS on map lagged FAR (1-2 seconds) behind my real position. More than once I missed a turn because of that, especially when you are driving in an urban area with a lot of smaller streets.

Fast-forward to 2020 - in the meantime I used Google Maps with no issues at all. A friend of mine suggested that I should try the latest version and maybe I would change my mind. So I used it for a day - same urban area - and it's still lagging behind - mind you: at this time I had a new iPhone and a new carrier. For me this makes the App unusable. I don't want to plan ahead 500 - 1000m (Germany) while using navigation.

I'm really curious, does someone have the same problem?


Absolutely this. Not sure if it's 1-2 seconds but the delay is enough to make it unusable. I've missed turns and exits because of it. Google Maps seems to have some predictive ability that allows it to appear to be "real-time".


I used to say that but lately apple maps seems to be wrong a lot - back to how it was when it first started, probably a temporary regression (I hope).

Yesterday for example it told me to do a U turn in a (sort of) one way street in the middle of the city, and not the first time its been dumb in the last few weeks.


I think Apple Maps has finally reached "competent maps app" status, but still has miles to go to catch up to Google Maps.


Apple Maps is way better than Google for the wandering pedestrian. Google hides so many restaurants and bars that don’t pay them for prominent display. Apple Maps shows every place near my location. Obviously hasn’t mattered recently, but I always go to Apple naps for nearby browsing.


Google Maps's heuristics for displaying POIs are nothing short of bizarre. I've seen it decide that weed dispensaries are the most important thing to display in an entire city. It's also really aggressive about displaying home businesses on the map, even if they aren't open to the public.


Does anyone know for sure if there is a way to pay to get your business to show up? It seems so unnatural it must be an advert. While zoomed out on my city it just shows icons for hospitals and pizza hut.


> Google hides so many restaurants and bars that don’t pay them for prominent display

Is there public info on the pricing for sponsored POIs, e.g. is it auction-based, for a period of time? Does the listing get dropped from the map after the business stops paying? How does Google get around the FTC requirement to label sponsored content?


I too prefer Apple Maps, but just for driving. Google Maps has way better coverage of public transportation in cities.

For instance Apple Maps doesn't have U-Bahn support in Munich, which makes it unusable for most people.


Only if the Map's Data works for you. From a Worldwide perspective, Google Map's Data is still thousand times better than Apple Map.


Correct

Meanwhile Bing maps manages to be the most frustrating experience of any current online maps.


Is there any reason this is getting traction on HN? Applebot has been confirmed for at least 5 years[1][2].

[1] https://appleinsider.com/articles/15/05/06/apple-details-new...

[2] https://appleinsider.com/articles/15/05/06/apple-challenges-...


I think people are assuming that this confirms a rumor that Apple is launching a search engine. Note that the linked page mentions “Apple Search”


The page was updated in July with much more detail.


A random older archive from last year:

https://web.archive.org/web/20190429220555/https://support.a...

What seems to be new:

- "About search rankings" section listing its purpose

- Expanded bot identification and verification information


The pívot podcast made a prediction that Apple will launch a search engine. Might be overlap in audience of hacker news and podcast


For me Applebot activity has increased by at least 500% YOY.


Google needs the competition. Search it's getting more and more laden with ads, and I have found the kind of specific technical searches it used to excel at are increasingly on page 2 after articles and videos and product listings.


Not to mention the dozen poorly-formatted Stack Overflow and GitHub issue aggregators that show up before the actual Stack Overflow and GitHub results.


Ah yes, I noticed that recently. To add insult to injury mine are poorly translated in French.


hell, it ain't just ads, it's outright malware at this point. i had to look up some shit about my printer, and hoo boy, searching for that churns up even more of a cesspool than i get when trying to search specialized windows topics. those don't turn up anything useful most of the time, but it's at least benign--a wall of posts on official MS forums where the top answer is some "top community contributor" yeeting out a "my friend, have you tried rebooting and reinstalling drivers?" copy-pasta response to any conceivable question

the printer stuff is something else. i put some fault on canon here for pushing me to unofficial sites because their official shit requires you to download a fucking exe to deliver a PDF manual (cmon, wtf, how can your docs delivery pipeline be THAT bad), but the unofficial stuff is some ninth circle of hell

the first was some bog-standard infini-redirect ad fraud crap, but the second was uh... well, i was impressed to the degree that the indian tech chop shop "YOU ARE INFECT DIAL 1-800-LEGIT-MICROSOFT TO CLEAN COMPUTER" site was able to shut down a fully up-to-date copy of firefox, somehow eating all my other tabs, grabbing hold of my mouse pointer in strange ways when the window was in focus AND preventing the windows "close window" button from killing it.

task manager was still able to execute it but damn, that was some ARTISTRY put into making the fraud site as clingy as possible--puts 90s-era malware sites to shame, and this shit kicks around on right on the first page of google search results for a printer manual query, not some shady-ass porn site you'd drunkenly click through to via 3 levels of ever-sketchier ads

people ranted and raved about goods of questionable quality gaming amazon reviews, but eh--honestly, most of those i've bought were fine, if not artisanal high-end shit--they just engaged in a lot of dubious "get free shit for a good review" growth hacking tactics that are A-OKAY THATS JUST HOW YA DO BUSINESS when it's some SF startup pulling the same crap in a different form. it's weird how seemingly nobody gives a fuck that a lot of google searches will boost things that are far worse--offering purestrain fraud on a platter is fine, selling mediocre flash drives and kitchen gadgets with inflated reviews is THE HIGHEST OF CRIMES


I love your writing style and attitude. Craving more.


it's rather divisive, seemingly--50/50 whether it gets upvoted or nigh-instantly -1 and flagged. lol hn, WE MUST WRITE VERY FORMAL TRADITIONAL ENGLISH HERE

Diversity would definitely be positive, the web is too important to have it indexed and sorted from one perspective only. Would prefer half a dozen engines and perhaps all enabled to provide results to metas who could further offer more diversity.


You can use/host searx (https://searx.space/) or yacy (https://yacy.net/) to go one step further.


Yacy is about as useful as searching the internet with grep. You can search "Facebook" and the first 20 pages are foreign spam blogs that have the word facebook on them.


way back in 00s, Google felt exactly like grepping the whole internet, including niche blogs, minus the spam (for the most part).

Somewhere in between, Google stopped caring, and we have what we have today.


They're definitely not great but they're a good place to start.


And Searx is only useful in English or so it seems.


I made the switch to a Searx instance from Google about 3 weeks ago and it's been great for me (now I just get Google's results in Searx with added privacy).

Dark mode to save my eyes, setting my own personal preferences for which search engines to use for results, Cached results link and proxied links, a files search engine...

The only difference is that I wasn't already heavily invested in other Google products. e.g. Gmail, Lots of saved Maps POI, etc.

For someone who is heavily invested, I'm not sure if the switch to Searx is worth the lack of convenience.


There's these alternatives also, all privacy focused but mostly reliant on other engines for data: https://www.eu-startups.com/2020/09/5-european-google-compet...


Part of that, I think, is just that the SEO industry has matured so much any random person can quickly climb the rankings by short-circuiting Google with basic SEO, taking over space which should've gone to more quality content instead.


Yes. Apple hardware is a business model but the search engine business is a hefty one.


Why would it be any better than Bing?


I’m hearing a lot of theories in this thread about Apple building a search product.

Yes, that’s possible, but Applebot is used in a lot of ways today that are wholly unrelated to search. (If search consists of crawling, indexing, ranking, retrieval, frontend, etc., Applebot only does the crawling part.)

Applebot is used for generating attachment previews in iMessage (eg. Send someone a URL — the preview is from Applebot crawling it). From the docs, it sounds like it’s also used for similar previews in Siri.


Applebot was built for crawling web pages, to be used for search results in Spotlight and Siri. That user agent might also be used for attachment previews, but the original intent of Applebot was for search indexing.


Pretty neat, likely the Applebot preview functionality that let us build Rapbits on iMessage: https://rapbits.com/video/ad.mp4


> Applebot is used for generating attachment previews in iMessage (eg. Send someone a URL — the preview is from Applebot crawling it)

Is this true? I would have assumed that would be done locally on device. The preview image, for example, is what's designated by the Open Graph[0] og:image tag.

[0]: https://ogp.me


Crawling and caching is pretty common for any service that offers unfurled previews to avoid sending extra traffic to sites that may not be able to handle it, and in many cases I'd imagine retrieving the pre-extracted cached version to be faster.


Apple is 100% building a search engine.

But do you remember when Apple launched their maps? A decade later and it's still not nearly as good as Google maps. But it's "good enough". I imagine that's what they're going for with this new search service


The data that Apple previously paid TomTom and others to use was certainly subpar compared to Google's data.

However, Apple started collecting their own data in 2015, and rolling out their own maps in 2018.

>Apple is filling its map with so many [details] that Google now looks empty in comparison and all of these details create the impression that Apple hasn’t just closed the gap with Google—but has, in many ways, exceeded it.

https://www.justinobeirne.com/new-apple-maps

Apple's map data now covers the US and its territories and data for the UK and Ireland are currently in testing.

https://www.justinobeirne.com/new-apple-maps-expansion-9


I'm totally fine with good enough if it keeps Google in check


The quality of the product is irrelvant. It will have usage as long as it is the default on Apple devices.


Applications are open for YC Winter 2021

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: