- Applebot was originally written in Go (and uncovered a user agent bug on redirects, revealing it's Go origins to the world, which Russ Cox fixed the next day).
- Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day.
- In it's first week of existence, it nearly took Apple's internal DNS servers offline. It was then modified to do it's own DNS resolution and caching, fond memories...
This is true but a little misleading. On Windows Go uses GetAddrInfo and DNSQuery which does the right thing. But on Linux there are two options: netgo and netcgo -- a pure Go implementation that doesn't know about NSS, and a C wrapper that uses NSS.
Since netgo is faster, by default Go will try its best to determine if it must use netcgo by parsing /etc/nsswitch.conf, looking at the tld, reading env variables, etc..
If you're building the code you can force it to use netcgo by adding the netcgo build tag.
If you're an administrator the least intrusive method I think would be setting LOCALDOMAIN to something or '' if you can't think of anything which will force it to use NSS.
It's possible that wasn't the default setting on Macs back then. I don't know that cgo would be a good choice either, if you're resolving a ton of domains at once. Early versions of Go would create new threads if a goroutine made a cgo call, and an existing thread was not available. I remember this required us to throttle concurrent dial calls, otherwise we'd end up with thousands of threads, and eventually bring the crawler to a halt.
To make DNS resolution really scale, we ended up moving all the DNS caching and resolution directly into Go. Not sure that's how you'd do it today, I'm sure Go has changed a lot. Building your own DNS resolver is actually not so hard with Go, the following were really useful:
>Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day.
The scale of web stuff sometimes surprises me. 1B web pages sounds like just about the daily web output of humanity? How can you handle this with 4 (fast) computers?
Yes, let's all run separate web browsers as the application and run our own JavaScript inside our browser. Who cares if there's 5 other "apps" doing exactly the same!
I think they were referring more to apps like Slack and other similar JS browser/ JS based apps which run separate from the browser. Maybe I'm being generous? Slack is certainly itself a beast.
Doesn't it depend on a lot of things? For example you can only do head requests to see if a page changed since a given timestamp. If not then there is no need to process it.
The HTTP HEAD method requests the headers that would be returned if the HEAD request's URL was instead requested with the HTTP GET method. For example, if a URL might produce a large download, a HEAD request could read its Content-Length header to check the filesize without actually downloading the file.
Typically you wouldn’t bother with a HEAD request, you’d do a conditional GET request.
When you request a page, the response includes metadata, usually including a Last-Modified timestamp and often including an ETag (entity tag). Then when you make subsequent requests, you can include these in If-Modified-Since and If-None-Match request headers.
If the resource hasn’t changed, then the server responds with 304 Not Modified instead of sending the resource all over again. If the resource has changed, then the server sends it straight away.
Doing it this way means that in the case where the resource has changed, you make one request instead of two, and it also avoids a race condition where the resource changes between the HEAD and the GET requests.
Does it use a traditional relational database or another existing database-like product? Or is built from scratch just sitting on top of a file system.
Nope, you don't really need a database. What you need for fast, scalable web crawling is more like key-value storage: a really fast layer (something like RocksDB on SSD) for metadata about URL's, and another layer that can be very slow for storing crawled pages (like Hadoop or Cassandra). In reality, writing directly to Hadoop/Cassandra was too slow (because it was in a remote data center) so it was easier to just write to RAID arrays over Thunderbolt, and sync the data periodically as a separate step.
Interesting stuff. I've used libcurl to crawl at that kind of pace, is the parsing/indexing separate from that count per day? Also interested in how you dealt with DNS and/or rate limiting
I've done similar at smaller scale. Instead of messing with underlying DNS or other caching in our code we just dropped a tuned dnsmasq as the resolver in front. The crawler had a separate worker to fill hosts so it was mostly hot when the crawler was asking.
In my case I was just fetching the home page of all known domain names. First issue I'd noticed was ensuring DNS requests were asynchronous. I wanted to rate limit fetching per /28 IPv4 to respect the hosts getting crawled, but couldn't really do that without knowing the IP beforehand (and keeping the crawler busy) so ended up creating a queue based on IP. I used libunbound. Found that some subnets have hundreds of thousands of sites and although the crawl starts quickly, you end up rate limited on those.
Also interested at the higher end of the scale about how 'hard'/polite you should be with authoritative nameservers as some of them can rate limit also.
Say the average web page is 100kb, and assuming gigabit connection in the office, then that's about a thousand pages per second. If the office switch is on 10gbit that would work out to 4000p/s naively counting. But we're in the same order of magnitude for the speed even on gbit, and we're not accounting for gzip, and the actual average page size might be a bit lower too.
Everything was on 10gigE. The average page size was around 17KB gzipped. Everything's a careful balance between CPU, memory, storage, and message throughput between machines.
Apple's corporate network also had incredible bandwidth to the Internet at large. Not sure why, but I assumed it was because their earliest data centers actually ran in office buildings in the vicinity of 1 Infinite Loop.
No idea, it's been years since I last worked on it. It was also not the only Go service written at Apple (90% of cloud services at Apple were written in Java), though it may have been the first one used in production.
And I sit here kind of shocked that Apple would use Java for anything, backend or not. I thought Apple had a strong preference for owning its own tech stacks, whether that be ObjC/WebObjects or later Swift...
I think WebObjects was supporting Java even before it came to Apple from Next. In the early days, many of Apple's services built with WebObjects even ran on Sun server hardware, and XServe's. But nowadays it's all commodity Linux hardware, like you would find in any data center.
Not sure why you'd be shocked, it's a solid language for enterprise services like Apple offers, and their other languages - C/C++, Objective-C, Swift - aren't very kind for web services.
Great use case for Go though, especially its concurrency features for web crawlers. I reckon Scala could work too, although it's a lot more complicated / clever.
I would guess because the input sanitizing requirement is harder for the web; having a stackoverflow when running locally requires the attacker to execute locally -- having a use-after-free from port 80 would be a much wider audience
Some Apple services were written in C/C++. One downside is it's very hard to source engineers across the company who can then work on that code, or for those engineers to go work on other teams.
Apple employs the founder of the Netty project, who has given plenty of open talks about Apple’s use of Netty (which implies Java services). Same is true for Cassandra.
Apple had a very odd obsession with Java right after the NeXT purchase. WebObjects got converted and they tried to do a Java Cocoa. Both were worse than the original.
You should check out Manning's "Introduction to Information Retrieval", it has far more detail about web crawler architecture than I can write in a post, and served as a blueprint for much of Applebot's early design decisions.
With 1b pages per day I guess you needed 1gbit/s connections on each of those machines? Especially if they also wrote back to centralized storage.
I guess there are not many places where you can easily get 4GB/s sustained throughput from a single office (especially with proxy servers and firewalls in front of it). Is that standard at Apple or did the infrastructure team get involved to provide that kind of bandwidth?
I know Apple's walled garden approach isn't for everybody, but I'd personally love to see them make a search engine that isn't built around Ads. The strategy also is consistent with their pursuit of vertical integration.
I use DuckDuckGo for search, and do trust them with privacy. But if Apple did a half-decent job at nailing the search experience, I'd switch.
I think it's too soon to assume that Apple's search engine would be accessible directly from the web, rather than just being a backend for search functionality from within iOS. After all, they don't host any Apple Maps equivalent to Google Maps on the web.
I find it unlikely Apple is going to compromise their their privacy first reputation to make an advertising platform which is competitive with Google's. I wouldn't be surprised to see advertising if Apple releases search, but it will likely be context (what you searched for) driven reliance rather than based on deep tracking. That would make it far less appealing to the skeezier advertisers. I'd expect something similar to what you see in News.
I can't see how Apple would even do the kind of search result pollution Google does now. In fact, I suspect the only reason Apple is launching advertising is because they see an opportunity to provide a better experience—namely one where you see organic results immediately.
A search engine that optimizes for good content instead of SEO-spam and ads would be a very welcome addition. Google seems to have given up on fighting malicious/spammy SEO (I'd expect them to identify and downrank spammy tricks like what the recipe websites do).
I'm in favour of competition for Google, but I can't be the only one who genuinely has no issue using Google? I see people complaining on HN quite a lot but genuinely the only thing I have - in years - struggled to find was the correct eBPF documentation but even then it only took a bit of nudging.
I think there are types of searches it does very poorly on, but also different people have different expectations. In my experience, if something is a product for sale, the first several pages are always places to buy it, which is never what I’m looking for, I already have online shops I like and saw the item there, or am searching from inside a store. I want reviews, forum experience, or even just pictures. for example, actual pictures of a bike rack installed on a bike like mine, not just the manufacturer’s steel floating in a white void pics.
In addition, google has become very aggressive with disregarding words. So a search like (a fake example, not tried the search, not a real place) Jim’s tasty pizza Albuquerque 90s where I’m searching for where a restaurant I remember as a child was located, and has maybe 5 real results from a local history Wordpress blog 15 years ago and Facebook posts, Google will just drop everything specific and give you results for Albuquerque pizza, and then mainly show ads for national chains up top. So I need quotes in the search pretty much always.
If you go into the advanced search options you can tell Google to explicitly search on what you type. I use it so often for the reasons you point out (because nowadays in their infinite wisdom they will helpfully ignore quotes too) it occurs to me I should either find an extension or script to just have it enabled by default.
I use Google and Google Maps plenty, though I'm one of the ones who might complain about their [lack of] privacy practices.
I use it for certain types of searches, and not others. I'm conscious of the fact that what I'm typing and searching, and subsequently clicking through will be piled on to the stack Google thinks is me. So it's usually tech searches, real estate/apartments, street view (property or exploring).
Health matters, personal matters, etc I tend to keep out of their search box.
Though more and more I tend to do a lot of cross-searching if privacy is not a concern.
Google is fine for me (I use it as fallback when DDG fails) for technical results but for anything else it's pretty bad (recipes are a typical example). It's not just the fault of Google to be honest - the internet itself has turned into a cesspool - however Google would be in the best position to curate that and discourage that behavior by penalizing the offenders' ranking.
Google ads don’t affect what search ranks. Those are two different entities. And google realllllly wants to get rid of search spam. But it’s insanely profitable to rank, so people try really hard and often beat google.
It's debatable whether Google Ads affect Google search ranking (it's impossible to prove either way considering how many factors affect ranking), but regardless of whether they currently influence rankings, there might be reluctance to change that due to Google's business model.
Google's job as a search engine is to provide relevant and quality content. Ads are spam and noise, so if Google does want to provide quality content they should be penalizing ads just like they would penalize keyword spam and similar unpleasant SEO tactics. Given their business model however it is not in their best interests to do so, so it's very unlikely that Google will do it.
Another company that does not make their money on advertising has a better chance of doing this.
> Ads do get penalized for being a bad fit: they get a lower quality score and cost more.
My point is that as a user, ads, regardless of how "good" they are, are not what I am looking for when I click on a search result (I am talking about website ads and not ads on the results page itself) and thus a search engine that downranks pages with ads in favor of those without them would be beneficial for me.
It would be against Google's interest to implement that (as they'd also be fighting against their own ad network) but a different company not involved in the advertising business (and making high enough margins on other products) has a higher chance of delivering this successfully.
Google search has gone down the toilet. Most Google search results are ads or SEO optimized junk. It was funny once when I was searching for a medical term, all articles on first page were clearly SEO optimized and all articles were very similar from word usage, sentence construction, verbiage, like written by same person. I am surprised that Google doesn't filter out the results using some sort of similarity analysis to offer variety in results. I don't want to see all links giving exactly the same info.
And the same rumours has been going on for years, and I seriously doubt that.
Reason is that Apple already has a Search Engine. It is called Siri. It is what you get when you ask Siri Questions. What I think Apple hopes to achieve is that get 80% of what you want, From may be Recipes, Simple Answers like Exchange Rate, Nutrients, Sport Scores... etc away from Google.
Letting you to Google the absurd, hard, questions that has less Data value.
All while collecting roughly $10 per user for Default Search Engine Placement in Safari.
And remember the Privacy stand Apple has giving less Data to Google.
Basically Apple is squeezing Google in every single direction. And pitching them against Azure and AWS for Cloud Services for discount.
I can't believe this hasn't happened yet honestly. Apple Maps has been out for 7 years now. Also a bit troubling if it gains traction, considering Apple has been a lot more heavy handed in keeping content they don't approve of out from their ecosystem.
In all seriousness, Apple may do certain things well, but cloud-anything just doesn't appear to be in its DNA. Maps, mail, iCloud, Timemachine, etc. Pretty much every service I can think of is laden with bugs, quirks, actual data loss risk, or is slow enough to be unusable. I'm not even remotely surprised that there is no "Apple search" yet.
They also run the App Store, Push Notifications, iAd, Apple News, Apple Music, Apple TV, Siri, Apple Pay and some of the largest systems for photo sharing, file storage, subscriptions, retail billing and more in the world. Saying Apple is bad at cloud services in general is incorrect.
They do have plenty of high-volume services e.g. Siri, Siri Suggestions, Maps, Online Store, Apple Music that haven't had any major issues. And even iCloud is substantially better now than in the early days.
Most of your issues seemed like the old, rehashed ones from nearly a decade ago.
Apple Maps is my goto because Google Maps just hates showing street names for some reason. Zoom in so a street takes up 100% of your screen and it still won't display the name many times.
Obviously that's not the only reason, haha, it's gotten surprisingly good a few years ago. Public transit and cycling is better in Google (though nothing beats CityMapper for transit if it supports your city). But for walking and driving Apple Maps is my winner.
I thought the same thing about Time Machine. It works really well with a local laptop that I plug in from time to time for this to work. However, is it considered Time Machine when an iDevice backs itself up to the cloud?
Mail is definitely a local app, even if they did screw the pooch pretty badly with that gmail bug they had a few years ago.
“However, is it considered Time Machine when an iDevice backs itself up to the cloud?”
No. Strictly speaking, iCloud doesn’t even do backups; it syncs data. Difference is that, with backups, you can get a copy back if you accidentally throw away a file. You typically also have multiple ‘old’ copies.
iWork apps (1) kind-of have backups in the form of versions that get synced to the cloud (https://support.apple.com/en-us/HT205411), but I think that’s implemented independently from iCloud’s syncing (You can use it to have multiple versions of a document locally, too)
> No. Strictly speaking, iCloud doesn’t even do backups; it syncs data
There’s literally a switch called “iCloud Backup” in Settings though. This isn’t a per-app data sync, it’s the full device restore that you can use if your phone gets stolen, dropped in the ocean, etc.
Although you should consider not using it for privacy reasons and running backups to a local machine instead, since last we heard Apple has the encryption keys and will happily hand the entire contents your phone to law enforcement given a warrant.
They have components that run in the cloud, and unfortunately, when I wrote "cloud services" I probably should have written "services that leverage the cloud."
I have a vague sense that you're right, that the services team just doesn't perform as well as other teams.
However, Apple operates many services at scales few companies ever reach, and doesn't seem to have outages much more often than those other few companies. So they're clearly doing a lot of things incredibly well, in the if-things-work-you-don't-notice sense.
They don't seem to emphasize their cloud services like Google does, and Apple charges for theirs at a much lower level than Google does, but I'm starting to think my vague intuition is wrong.
Also of note, Apple Maps in the iOS 14 beta just started accepting ratings and photos at map locations, rather than dumping you out to Yelp reviews. I'm happy with this change.
There's nothing slow about this, it's already launched. I don't understand why people don't get this -- it even says so at the top of this page: "Products like Siri and Spotlight Suggestions use Applebot"
I don't know why anyone would think that Apple would make a web page search engine like Google. Apple has already launched the Apple version in your phones integrated into the software you use.
Maybe because they are desperate for a different search provider than Google?
I don't use Siri or any assistant, so I don't seem to use what I'm being averaged into using. Majority opinion here is that Siri is seriously lacking behind the other offerings. Apple won't be oblivious to this. If it is solely due to their back-end search engine, then making that a public search engine website won't do Apple any favors if the results are truly that bad.
Voice assisted search probably won't give you much feedback. With a web based search engine, users provide feedback by ignoring some results and clicking others. Therefore releasing a search engine could help them improving it and deliver results with Siri that people would've clicked on otherwise.
Apple is mostly a focused company. There is no money in search except advertising and user tracking. Apple has already completely failed at advertising once.
Yes I realize that the App Store has ads and it is a stain that should not have ever happened.
Well they've gone to the effort to build a search index and they have sufficient market power to push millions of user onto whichever search engine they'd like. It's essentially leaving money on the table not to take the final step of popping a web frontend on it and making it the Safari default.
That they use their own search for this is not a big headlining marketing feature so they aren’t really evaluated on it.
I do think that’s a direct consequence of their experience with Maps, though I guess their hand was more forced on Maps and they had to tell the world about it.
However, using Siri and search suggestion they can, bit by bit, replace search results from other sources with their own without telling anyone about it. It’s a more careful approach.
Google's results are already pretty clean. Search results are heavily filtered (e.g. copyright, adult content) so I don't think Apple would filter more. If anything, filtering more than Google could deteriorate results.
But the SEO market would certainly change if webmasters suddenly have to appeal to more than just Google to be successful.
No, Apple is a 'full-stack' tech company. People buy their products both for their hardware and software. If they were just a hardware company, why would they bother making macOS, iOS, iMessage, Safari etc?
Haven’t you heard people whine about how Apple is so much about privacy but they take money from Google to have it as the default search?
They don’t have Google as the default search because of the money, they have it because they think it is the best. Even despite the privacy problems and the control this grants Google.
How does a maps service sell more hardware? Because using their own maps service Apple can completely control the experience.
Now maps was hard and search is much harder still so we may very well never see this released. But even then they can use this as leverage while negotiating with Google, both for money and for control.
Because your claim about them being a hardware company is incorrect.
They market themselves as a privacy focused company. Privacy is all about software and services, not hardware. Search is something they could add to their portfolio to strengthen their privacy stance.
Services are ca. 18% of Apple’s revenue, after years of huge spending and tons of dedicated effort to grow this segment. The fact that it is as high as a fifth is itself a monumental feat.
Hardware sales of the iPhone alone, excluding all computers, AirPods, and iPads, is over 40%.
The same reason why having a maps product benefits them. They want to have complete end-to-end control over a user's experience on their device, well beyond just the hardware. Having their biggest competitor dictate what happens whenever someone opens the web browser on an iPhone is a massive hole.
Hard drives are cheap enough that storing the index itself is the least expensive part of building it. Engineer time to make that index usable for a search engine is that much money every week.
One of the selling points of apple devices is getting away from google spyware. Every bit of spyware they can prevent the user from being exposed to is better.
Apple doesn’t care at all about the most common types of spyware. Nearly every app in the iOS App Store contains tons of spyware.
Apple’s view is that you consented to this sort of unaccountable invisible tracking by dozens of third parties (against whom you have zero recourse), when you agreed to the TOS of the App Store.
Apple built their own search engine over 5 years ago, under the Siri / Spotlight umbrella. When people talk about Apple building their own search engine, they generally seem to expect a website dedicated primarily to web page results, but under the covers what powers Apple's Spotlight results is basically a search engine.
The big question would be what Apple would gain from a dedicated website for search results. Would people really switch to it from Google? Why would it be a better delivery mechanism for search results than Spotlight? Not sure the answers to these questions has changed much, from 5 years ago to today.
They have a lock on mobile devices for the western world. They can by default pry away search revenue from Google at a large scale. They have the cash reserves to see this done properly. They'd be crazy not to come onto Google turf. Their hardware lines are sagging revenue growth wise. Service revenue is their biggest growth area. I have no idea why they didn't do this before.
> They have a lock on mobile devices for the western world.
They don't though. There are only a couple countries where they're more popular than Android, and only barely there. There's no country where they have 60% market share.
Or just like Iran's nuclear program, they can extract more concessions if they are permanently 1-2 years away from deploying a search engine. Google's spot as the default search in Safari already earns Apple billions per year.
It might look like that for people inside the walled garden but I suggest applying for an exit visa to have a look around the great wide world out there where you'll quickly find out that it is a whole lot more diverse than the picture painted inside the walls. Not only is it more diverse, those rumours spread about the constant onslaught of viruses on those poor creatures on the other side of that big, safe wall turn out to be untrue, most people seem to never have even seen one. Even stranger things will start to become clear, like those supposedly poor and oppressed outsiders needing far less money to get around the world than those inside the walls while at the same time having are larger choice of methods to navigate it. It wouldn't surprise me if you decided to stay, just don't forget to plan this carefully so your identity does not get stuck on the inside after which you won't be able to message your friends any more. It is a bit tricky but it is doable, many others have gone that way before.
If Apple Maps has taught us anything, probably not. But Apple would first need to pour an equally large amount of resources into web search, they way it did for Maps.
Let's see if Apple is the first one to nail it. I love using DDG but their results are not close to the same quality as Google for me, especially with localized results and unspecific requests. And DDG is backed by Microsoft's Bing which invested billions into this. Though having run a few websites, Google's crawler easily crawls 10x the amount of all other crawlers combined if they don't detect page slowdowns. I never understood why Bing was so conservative when it comes to crawling.
I believe if anyone can build a search engine that can truly deliver the same quality as Google, the whole market could change. Google is dependent on search ad revenue like nothing else, if they lose that monopoly it could really change the company.
my first reaction , was oh god, please dont! considering how worst siri is, i dont want that.. but after reading other comments like native ad tracking etc etc it could be just fine.
But google pays Apple in billions to keep its search as default, what could be the key motivation for Apple to build its own engine? its a highly saturated market and we have Duck Duck Go for privacy based , bing for alternative and heck, even wolfram alpha for computational! what Apple can improve?
Based on usage of DDG, bing and wolfram alpha, I would think Apple can improve a great deal. I'm not saying it can necessarily achieve that, but I do think there is a lot of space to improve (indexing, ranking, UX)
I remember when I rolled my eyes at Apple making a Maps product. I thought it was a fruitless, dumb idea.
I prefer Apple Maps to Google Maps now... don't underestimate Apple's ability to plant a flag and move it inch by inch each year.
In my personal experience Google Maps has better navigation data, better place information and overall much better maps. The user contributed content makes the platform such a pleasure to use even if you are in remote parts of the world.
On my iPhone I think Apple Maps has a vastly better navigation UI. Part of that is its first-party ability to turn the phone screen on when issuing instructions for upcoming maneuvers, which I much prefer to having the screen on constantly for a multi-hour trip (not to mention working well with Siri). Part of it is the overall aesthetic of the map features. I really like how it highlights traffic lights along your route to let you easily count how many lights are remaining until you need to turn.
That said, even here in the Bay Area I tend to do a sanity check of the route on Google Maps first if I’m taking an unfamiliar route. Just a few months ago a section of I-80 was closed just south of SF and Apple Maps had no clue! I believe it was a fairly last-minute schedule change for some planned construction due to vastly reduced traffic during the lockdowns.
I’ve used Google Maps since it first came out on the nexus one. I recently made the switch to Apple Maps. And it’s like stepping into a new world.
There’s the maps layout and where I want to go and my route. That’s it. Google Maps has ads. Switching between Apple Maps and Google Maps really shows how the Google product has friction and information overload. We are talking about a Maps app. Less is better unless I ask for it. Apple has nailed the color scheme, layout, and information density. If I need more information that Apple Maps doesn’t provide, I double check google maps. On a long trip, I will check routes from both apps. Apple Maps and google maps show the same route and same time and same traffic congestions for me. I rely more and more on Apple Maps now.
> which I much prefer to having the screen on constantly for a multi-hour trip
I’ve found myself swinging in the other direction. I tend to glance at Waze for my speed instead of my own speedometer, but I’ve been driving a mixed bag of cars lately.
For me having those stop signs and signals show up in navigation is big plus for Apple maps. I have a pixel phone but if I'm driving with my spouse I always use her apple maps
One great thing with the Apple Maps UI is that when using CarPlay it gives you a 3D view of the street. So you're really seeing buildings on the left and right, which has helped me not overshoot some destinations because I can 'see' the building in the map, and then irl.
Judging from the responses here, some of the reasons seem to be because Apple Maps is much more vertically integrated with iOS, and that doesn't seem like a good thing to me. Sure, you may get a better experience, but that's more likely because Apple is keeping Google Maps from getting as good than because it's a better solution, period. I have no love for Google but that's not a good model, and I feel like this kind of thing is making Spotify's case for Apple being a monopoly much stronger.
You don't need particularly tight integration with the operating system to make a good maps app; you'd want access to CarPlay, which Google Maps has, and you'd probably want a good sharing extension, which I don't think Google Maps has but could if Google cared about making better iOS apps than they seem to. Google is rather notorious for being very, very slow to adopt new iOS features in their apps, and it's definitely not because Apple is making it difficult for them to do so.
The history here is also at least worth a passing mention -- the iOS Maps app originally used Google mapping services. The reason Apple built their own back end is because Google refused to give them access to vector-based maps and turn-by-turn navigation unless Apple shared more user data with Google than they were willing to provide.
I also find that Apple Maps respects local mapping conventions. For example in New Zealand Google treats all state highways as equivalent and uses the same weight and colour line for all of them, but Apple Maps follows the NZ convention and assigns appropriate weights to the minor roads.
For those not familiar "state highway" simply means the central government agency pays for upkeep, rather than indicating any particular road quality or status.
Not OP, but I'm in the same boat. Agreed Google Maps' data is better. Their routing is sometimes better. But I still prefer Apple Maps in most cases. For me, it comes down to a few UX things:
* iPhone + Apple Watch combo is amazing to use w/ Apple Maps. Audible & haptic alerts on your wrist change how one interacts with the turn-by-turn directions, and it's an improvement.
* The fact that iPhone nav is available on the lock screen is a huge help. Doesn't require full unlock to get to the map.
* Apple Maps just seems to handle the accelerometer in the iPhone better. I found that Google Maps never knows what direction I'm facing/walking, and it jumps all over the place all the time. Apple Maps is just smoother and better to use.
The only thing that's missing in Apple Maps is bicycle directions, but apparently that's coming in iOS 14, which is great.
The main time I find myself using Apple Maps is when visiting China. Google Maps is blocked so the place data isn't as great, and the local apps are in Chinese, which I'm illiterate in.
I similarly find Apple Pay great for traveling, and easier to use than a credit card.
But, on my no-longer-bleeding-edge iPhone, the turn-by-turn directions suck. It's constantly telling me to turn onto a street I just passed, or rerouting to find a way to get me back to the street that I'm already on. The worst part is that I can't trust it even when it's right. If it says "in 300 feet turn left," I have to desperately look at the street names because it's quite possible that I actually have to turn in 30 feet.
It was such a relief when I realized that it's not my GPS hardware, Apple Maps gives directions perfectly. And the map data is way better than it used to be. Not as good as Google Maps, but good enough.
I hate Google maps when it shuts itself off with a 'journey complete' status while I'm still looking for the corner or the building that I was supposed to arrive at.
Yeah, I really liked when Pixel integrated the phone number bit into dialer itself. I can start typing a business name and it just pulls in from Google maps.
The iPhone I have now is a work phone, so don't use maps a lot, should see if it has improved.
I'm pretty sure Apple Maps is using Yelp's data for business hours -- those seemed to be mostly reliable until the pandemic. Now, not so much. I don't know if Google is doing better on that point.
Its really small, but for me its traffic lights. It's SOOOO much easier to be navigating and say, turn in two lights, instead of "turn in the next three streets". Especially on busy roads when you need to be in a certain lane to make that left or right hand turn.
Google maps added this to their desktop app [1] and maybe their iOS app, but apple has had this for years.
Also Apple maps offers saved places in an offline, non-synced way. Google requires access to all my location data to save places (like web bookmarks) and that's just too privacy invasive for me.
Even a few years ago I preferred the way a few-block area was represented on Apple Maps vs Google Maps. Apple Maps seemed to be much better at showing relevant landmarks and business names, where Google Maps would randomly select some businesses that happened to have an address there, but often no signage, etc.
Both applications would select the same number of businesses to show (obviously not showing ALL names due to density restrictions on the screen and clutter), but the Google ones were just so painfully arbitrary, and Apple's building skeletons seemed a lot more relevant/recognizable.
In my experience, Apple Maps has been more up to date with recent changes to roads here in Norway. There was a new tunnel here recently, and it was updated in Apple Maps when it opened, while with Google Maps it took a while.
I'm guessing that it depends a lot on the area. Google Maps is still probably better in most places, but Apple Maps is catching up and may be better in some populated places. It's impressive considering the head-start Google has.
Apple should add an option of OpenStreetMaps overlay or something. OSM has been better than Google Maps in mapping paths in the forest and such here.
Not the parent commenter, but while I still use Google Maps over Apple Maps, I constantly have issues with the former.
I've put in business addresses or searched the business themselves, and then had it route me to a residential block behind the business on a completely different street, sometimes multiple blocks away from it. This is even after confirming the exact address is being shown and everything.
I only really use Google Maps because it's what I've been using, and I kind of like that its UI is a bit simpler with less animations and things than Apple Maps.
I don't have unlimited data, and I have found Apple maps is much lighter on data use than Google. I have also found that even if I don't have a solid connection Apple maps seems to still be capable of giving me directions quickly, whereas on the other hand with Google I may never get directions.
1 - It actually runs smoothly on my iPhone 11. Google Maps stutters and yanks. Yes, Google cannot build a performing maps app for the fastest phone in the world.
2 - It is a map. Not a yellow pages of coffee places, not an Instagram with amateur photos of locations.
Have you tried it lately? It gets better with each iteration, and the street photos are way better where they exist. The satellite and aerial photos are also better in most places that I travel too.
Also, the Google Maps app has gotten much more cluttered and complex than in past versions. They are still the gold standard map, but they regress over time.
My house has not been in Apple Map's index for approaching three years now. It's caused _all kinds_ of problems. Apple seems to drop the ball on the details that are needed to really get a product to be quality.
EDIT: I looked back in my emails and Twitter DMs and it's been 25 months at this point. So "approaching three years" is not accurate.
Have you tried reporting it? My parents recently moved into a new house built one year prior and I reported it a few days after they moved in. It took about 9 months, but they finally added the road in and populated ALL the addresses on the street, not just the one I reported.
Google maps still does not have the roads or street numbers listed. Just wanted to throw that out there.
Edit: Also want to add that it has been two years since my parents moved in. Went in to Waze Map Editor and added the road myself, so their turnaround time was instant. Apple Maps took 9 months, like I said, but Google Maps still has no info on the street despite having all the houses in clear satellite view.
It did take Google a few months to add my complex. I'm not sure what an appropriate turn around time would be on something like this. But I do think it's less than "years".
Just read your reply. That's very frustrating to read -- hopefully the Apple employees come to their senses and add the building soon. It's also weird that they called you through FaceTime and still did nothing. Either way, sorry to hear about all the frustration you've been through.
What's the point of reporting anything to the Apple Maps team? I've reported stuff for a few years, and it hasn't changed nor have they had the basic courtesy to acknowledge the report.
It's not like it's some street address thing, they are missing a whole town.
I've gotten changes made by reporting stuff to the Apple Maps team. I remember years ago, my company's business location didn't show up correctly. I reported it, and within a week it showed up incorrectly in an entirely different way!
Nope. You can report it's missing if you own an Apple device. I have done that (many times). I have also talked with Apple support on Twitter many times, and they even assigned a dedicated support person who called me several times. I facetimed with him and showed him how my complex, plus several of the nearby complexes, are not in Apple Maps but are in Google Maps. That was ... over a year ago. They still haven't fixed it. I finally gave up.
Whenever someone needs to come to my house I need to ask them to use Google Maps. The challenge from that comes from less techy people who don't know the difference. Many people just know their phone has a "maps app". I will admit, experiencing that more closely (how less technical people interact with their phones that is) was interesting fwiw.
Google Maps is making it easier on them by seemingly becoming worse and worse, in my opinion. The newest act of telling you it's changing your route if you don't click something is so idiotic it borders on malice. So going 70mph down the expressway, I need to take my eyes off the road to read the message and click a button asking it not to take action.
Further, it repeatedly picks a hard route. There's one park I go to that's easier to drive to using one few mile stretch of highway. Google insists it's faster to take these back, poorly lit, windy 2 lane roads. Maybe it is. But it turns a safe easy 3 turn trip into a nerve-wracking 10 turn trip. And it will, as noted above, keep trying to reroute me if I just ignore it and go the easy way.
Another thing google maps always does to me is telling me what the next turn is when its too late and I can't make it anymore. Using google maps with audio only is impossible.
Noticed that recently too - it insisted on taking stupid backroads and what not, instead of straight highway and no traffic. It seems to be getting dumber, or something.
I tried AppleMaps for the first time like 4-5 years ago. Feature-wise it wasn't where it is now but what annoyed me the most is that the GPS on map lagged FAR (1-2 seconds) behind my real position. More than once I missed a turn because of that, especially when you are driving in an urban area with a lot of smaller streets.
Fast-forward to 2020 - in the meantime I used Google Maps with no issues at all. A friend of mine suggested that I should try the latest version and maybe I would change my mind. So I used it for a day - same urban area - and it's still lagging behind - mind you: at this time I had a new iPhone and a new carrier. For me this makes the App unusable. I don't want to plan ahead 500 - 1000m (Germany) while using navigation.
I'm really curious, does someone have the same problem?
Absolutely this. Not sure if it's 1-2 seconds but the delay is enough to make it unusable. I've missed turns and exits because of it. Google Maps seems to have some predictive ability that allows it to appear to be "real-time".
I used to say that but lately apple maps seems to be wrong a lot - back to how it was when it first started, probably a temporary regression (I hope).
Yesterday for example it told me to do a U turn in a (sort of) one way street in the middle of the city, and not the first time its been dumb in the last few weeks.
Apple Maps is way better than Google for the wandering pedestrian. Google hides so many restaurants and bars that don’t pay them for prominent display. Apple Maps shows every place near my location. Obviously hasn’t mattered recently, but I always go to Apple naps for nearby browsing.
Google Maps's heuristics for displaying POIs are nothing short of bizarre. I've seen it decide that weed dispensaries are the most important thing to display in an entire city. It's also really aggressive about displaying home businesses on the map, even if they aren't open to the public.
Does anyone know for sure if there is a way to pay to get your business to show up? It seems so unnatural it must be an advert. While zoomed out on my city it just shows icons for hospitals and pizza hut.
> Google hides so many restaurants and bars that don’t pay them for prominent display
Is there public info on the pricing for sponsored POIs, e.g. is it auction-based, for a period of time? Does the listing get dropped from the map after the business stops paying? How does Google get around the FTC requirement to label sponsored content?
Google needs the competition. Search it's getting more and more laden with ads, and I have found the kind of specific technical searches it used to excel at are increasingly on page 2 after articles and videos and product listings.
Not to mention the dozen poorly-formatted Stack Overflow and GitHub issue aggregators that show up before the actual Stack Overflow and GitHub results.
hell, it ain't just ads, it's outright malware at this point. i had to look up some shit about my printer, and hoo boy, searching for that churns up even more of a cesspool than i get when trying to search specialized windows topics. those don't turn up anything useful most of the time, but it's at least benign--a wall of posts on official MS forums where the top answer is some "top community contributor" yeeting out a "my friend, have you tried rebooting and reinstalling drivers?" copy-pasta response to any conceivable question
the printer stuff is something else. i put some fault on canon here for pushing me to unofficial sites because their official shit requires you to download a fucking exe to deliver a PDF manual (cmon, wtf, how can your docs delivery pipeline be THAT bad), but the unofficial stuff is some ninth circle of hell
the first was some bog-standard infini-redirect ad fraud crap, but the second was uh... well, i was impressed to the degree that the indian tech chop shop "YOU ARE INFECT DIAL 1-800-LEGIT-MICROSOFT TO CLEAN COMPUTER" site was able to shut down a fully up-to-date copy of firefox, somehow eating all my other tabs, grabbing hold of my mouse pointer in strange ways when the window was in focus AND preventing the windows "close window" button from killing it.
task manager was still able to execute it but damn, that was some ARTISTRY put into making the fraud site as clingy as possible--puts 90s-era malware sites to shame, and this shit kicks around on right on the first page of google search results for a printer manual query, not some shady-ass porn site you'd drunkenly click through to via 3 levels of ever-sketchier ads
people ranted and raved about goods of questionable quality gaming amazon reviews, but eh--honestly, most of those i've bought were fine, if not artisanal high-end shit--they just engaged in a lot of dubious "get free shit for a good review" growth hacking tactics that are A-OKAY THATS JUST HOW YA DO BUSINESS when it's some SF startup pulling the same crap in a different form. it's weird how seemingly nobody gives a fuck that a lot of google searches will boost things that are far worse--offering purestrain fraud on a platter is fine, selling mediocre flash drives and kitchen gadgets with inflated reviews is THE HIGHEST OF CRIMES
it's rather divisive, seemingly--50/50 whether it gets upvoted or nigh-instantly -1 and flagged. lol hn, WE MUST WRITE VERY FORMAL TRADITIONAL ENGLISH HERE
Diversity would definitely be positive, the web is too important to have it indexed and sorted from one perspective only. Would prefer half a dozen engines and perhaps all enabled to provide results to metas who could further offer more diversity.
Yacy is about as useful as searching the internet with grep. You can search "Facebook" and the first 20 pages are foreign spam blogs that have the word facebook on them.
I made the switch to a Searx instance from Google about 3 weeks ago and it's been great for me (now I just get Google's results in Searx with added privacy).
Dark mode to save my eyes,
setting my own personal preferences for which search engines to use for results,
Cached results link and proxied links,
a files search engine...
The only difference is that I wasn't already heavily invested in other Google products. e.g. Gmail, Lots of saved Maps POI, etc.
For someone who is heavily invested, I'm not sure if the switch to Searx is worth the lack of convenience.
Part of that, I think, is just that the SEO industry has matured so much any random person can quickly climb the rankings by short-circuiting Google with basic SEO, taking over space which should've gone to more quality content instead.
I’m hearing a lot of theories in this thread about Apple building a search product.
Yes, that’s possible, but Applebot is used in a lot of ways today that are wholly unrelated to search. (If search consists of crawling, indexing, ranking, retrieval, frontend, etc., Applebot only does the crawling part.)
Applebot is used for generating attachment previews in iMessage (eg. Send someone a URL — the preview is from Applebot crawling it). From the docs, it sounds like it’s also used for similar previews in Siri.
Applebot was built for crawling web pages, to be used for search results in Spotlight and Siri. That user agent might also be used for attachment previews, but the original intent of Applebot was for search indexing.
> Applebot is used for generating attachment previews in iMessage (eg. Send someone a URL — the preview is from Applebot crawling it)
Is this true? I would have assumed that would be done locally on device. The preview image, for example, is what's designated by the Open Graph[0] og:image tag.
Crawling and caching is pretty common for any service that offers unfurled previews to avoid sending extra traffic to sites that may not be able to handle it, and in many cases I'd imagine retrieving the pre-extracted cached version to be faster.
But do you remember when Apple launched their maps? A decade later and it's still not nearly as good as Google maps. But it's "good enough". I imagine that's what they're going for with this new search service
The data that Apple previously paid TomTom and others to use was certainly subpar compared to Google's data.
However, Apple started collecting their own data in 2015, and rolling out their own maps in 2018.
>Apple is filling its map with so many [details] that Google now looks empty in comparison and all of these details create the impression that Apple hasn’t just closed the gap with Google—but has, in many ways, exceeded it.
Think the summary from what I've read the past few days
- It's established applebot is already used for Siri, but there is no general web search
- Much commentary diverts to how poor Apple maps (is?) was.
- Google pay apple a handsome amount of money to be the default search engine on Apple devices
- If Apple were no longer to accept that payment, about 80% of people would still use Google anyway, given a choice [0]
- Googles high earnings per search means they can offer more than any other search engine, like Bing, DDG et al
- People would generally like more competition in the search space and Apple has the means to do it
- Anecdotally, there isn't a drop in replacement for a search default other than Google that would meet the quality requirements for end users. In this case, the only other alternative would be an Apple search engine.
- As per recent articles, there does seem to be more activity from Apple wrt applebot crawling and hiring
If they get a choice with no silent default yes, but if the default is set for them and they have to go into preferences to change it the results are almost completely reversed.
This is why Apple Maps is so heavily used on iOS. Most people don't know or don't care. In the first year after the launch of Apple Maps it hit over 65% of maps usage on iPhones in the UK, even though back then the UK maps were awful.
AT&T and Apple make sense but why is Ford on that list. And how did Comcast get theirs in 2005? I guess if you just ask very nicely, anything can happen.
Remember: ARPANet was originally just a DOD research project to connect some universities and military sites. Back when it was opened to the public you could get a .com domain name for free just by asking. Same deal with IP addresses: in the very early days you just had to ask and you could have your own /8 (assuming you were a university, medium-to-large company, or government entity).
There has never been any point in clawing back the large class A assignments. If that were done it wouldn't even give us an extra year of IPv4 address assignments before we exhaust the pool again... not that there is any legal mechanism to take them back anyway.
The solution is IPv6. Despite the slow rollout it is finally happening now that even the regional RIRs have started to run out of addresses.
My bad experience is generally rooted in the above criteria. It's particularly problematic on news sites. For example, if I click a news article with certain political association, I keep seeing only news from that political side. This creates an echo chamber which makes uninformed believe that 'whole world' is thinking that way. It definitely make sense to increase more engagement, but might not be totally appropriate in all instances. Curious to see how it works in live (if the apple search does become a reality)
Thanks for the reference. It's good to know that there is research backing SNS (social network sites) exposes us to more diverse news contrary to popular belief. Size of the data set seems to be big enough to rely on the results.
I'm wondering if the same holds true when that experiment is conducted in isolation. For example, if we take any particular news site, if a user is repeatedly visiting the same news site, then my hypothesis is that the kind of news one sees will converge onto a specific political view point. It might vary based on each news site - google news vs apple news based on how they are optimizing for users.
In reality, the isolated experiment might be moot as users do end up on news site from SNS which ultimately leads to recommendation algos picking up diverse signals to show diverse results. Given that, whether the algo itself is diverse in isolation or not wouldn't matter as the environment that feeds it is diverse. Does it make sense?
Good questions. I suspect diverse news may not actually result in diverse perspective anyway. I wouldn't be surprised significant fraction of people simply don't change their opinion, even while consuming news from diverse source. So it may be doubly moot in the end.
On a related note, since there are video podcasts, I've always wondered why they haven't been positioned as an alternative/competitor to YouTube by now.
A lot of YouTube channels and types of content could be replaced by video podcasts, and be more convenient for viewers.
As for monetization, Apple wouldn't even have to build an ads platform; content creators could embed ads themselves into the content, as they currently do for audio podcasts.
Everyone appears to be nervous because of the supposed search engine.
I believe this was part of their initiative to improve Siri results; when the service was even worse.
Rather the opposite. Many robots.txt will disallow * but allow googlebot. This allows them to crawl those sites without getting flak for disregarding robots.txt completely.
I had this same wish yesterday. I know it isn't the ideal scenario, but there are definitely some net positives that could come out of it in the form of keeping Mozilla's lights on for a long time to come and massive improvements to areas of Safari that (to me) are worse than competitors such as dev tools and extensions.
Given that the App Store search has one of the worst search algorithms in a major search engine that I know of, I am not holding my breath for an Apple search product...on their hand, when Apple does get something right, they hit it out of the park.
I keep hearing that but when I dug a bit into it to do some ASO, the impression I got is that it's simply not easy to game in name of Search Engine Optimisation because the consensus is that the search engine only looks into the App Title, subtitle, description and keywords and the ranking is affected only by the App's performance. SEO people seem annoyed by Apple that they cannot get their wallpapers app for the "stocks" keyword, so I have a very little sympathy. When they complain that App Store Search is dumb, they essentially mean that If I want to rank up my wallpaper app "stocks" and figure a way to people use it every day. They have all kinds of tricks for Google Play but the ASO tutorials are rather dull, recommending stuff like picking an appropriate title and doing A/B tests on the keywords.
As a user, I tend to be quite happy with the search results in the App Store, I don't remember any time where I could not find the thing I am looking for.
I suspect this is all about Siri. They went from being top of the game in personal assistants to being a second stringer. Apple can't compete with Google Assistant without the data set and analytics. Personal assistants and web search are two sides of the same coin.
There are reports that right now Applebot is being used to scrape the content used by Siri. If youre already scraping the web, indexing it and building search and analytics services on top of it for Siri, user facing search is just a short step to take.
This is one of the most exciting developments in the history of the internet.
Apple has always sided with privacy and while DDG toots it, they’re too small to build an independent service.
I am thrilled, excited and perhaps surprised why it took so long for the most used website for large majority of the people in the world to see some competition.
I think Applebot is nothing new (apart from the name). It's been years now since I found in my logs traces of crawlers from Apple IPs with an iOS 8 Safari-like user-agent. Maybe, well, since iOS 8 at least.
Edit: see here @jd20 thread for insights and confirming this.
The bottom of the page says "Apple Search may take the following into account when ranking web search results:", is this another confirmation that Apple is developing a search engine?
I'm keeping an eye on siri.com - right now it redirects to a sort-of-holding page on apple.com... but I can totally envision that URL becoming the home of their new search engine.
Likely whitelisted as a 'good bot' in Cloudflare. If a bot isn't well known, doesn't respect robots.txt or crawls too fast then much more likely to get flagged, but applebot are none of those.
I have used both and they both 100% cover my use cases. Gmaps could be 1000x better but it doesn't matter because there is nothing more I need that isn't in Apple maps.
- Applebot was originally written in Go (and uncovered a user agent bug on redirects, revealing it's Go origins to the world, which Russ Cox fixed the next day).
- Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day.
- In it's first week of existence, it nearly took Apple's internal DNS servers offline. It was then modified to do it's own DNS resolution and caching, fond memories...
Source: I worked on the original version.