
Apple might be running a web crawler written in Go - beltex
http://jan.moesen.nu/2014/11/06/apple-crawler.txt
======
unwind
Not at all sure why it's relevant which programming languages they use at
Apple.

Is this considered interesting since it somehow proves the mainstream adoption
of Go? Or since Apple, a competitor of Google's in cell phones, are using one
of Google's technologies? Or what?

The takeaway for me was that Apple has a whole /8 subnet to themselves. That's
just ... immense, for a single company. Gaah.

EDIT: Mis-typed the netmask, I meant /8 but typed /24\. Fixed.

~~~
mchanson
Back around 1990 I had a class /16 and I was just a 14 year old. Before the
web explosion it was a different time. Domains used to be free back then too!

~~~
fuckballs
Eeek, the "C" in CIDR stands for classless, so it is either a Class B or a /16
not a "class /16"

Also, it is highly unlikely that you had a /16 as a PI allocation even in
1990.

~~~
mturmon
Your nick is deliberately obscene. I'm not a prude, but it doesn't fit the
tone of the site.

~~~
sneak
There are no words that are obscene.

------
christop
I noticed this as well a couple of weeks ago.

They're still actively visiting every page on the websites that are associated
with our iOS apps.

Today alone (starting shortly before 8am CET) they've crawled over 8000 pages
on [https://trails.io](https://trails.io) and
[http://offmaps.com](http://offmaps.com) — without sending conditional caching
headers (our pages don't change _too_ often).

    
    
        $ grep ^17\\. /var/log/nginx/*.log | grep 'Go|Fetcher' | wc -l
        8254
    

Note that they don't appear to be scraping any URLs available from within the
apps themselves, but rather the the company/support websites linked in our App
Store listings.

I guess they're automatically scanning for objectionable content, since these
websites are linked from the App Store and the iTunes website?

~~~
0x0
Ooh, that's interesting. They already increased the requirements for online
privacy policies etc for ios8 apps, so perhaps we'll see some crackdown on
appstore apps who are missing those.

------
brycekahle
The Go http client definitely has a bug that doesn't maintain the User-Agent
across redirected requests.

[https://code.google.com/p/go/source/browse/src/net/http/clie...](https://code.google.com/p/go/source/browse/src/net/http/client.go#338)

~~~
bgentry
Looks like issue #4800 is the place where it's tracked:

[https://code.google.com/p/go/issues/detail?id=4800](https://code.google.com/p/go/issues/detail?id=4800)

------
pkulak
Go is a fantastic language for writing a crawler. I wrote the Showyou crawler
in Go and it's both one of our highest load processes (and it's very
efficient, trust me) and most stable.

------
bkeroack
I've seen golang mentioned in Apple job listings. I didn't think this was a
secret.

------
13
I've got several thousand requests for this in my logs with an IP address in
the same /24.

Two weeks ago it was using a different user agent:

    
    
        Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Fetcher/0.1)
    

This string is mostly from a beta build of iOS 6.0 beta 4, but with the same
suffix the author discovered.

------
pvsnp
It looks more likely that someone's learning Go and going through the crawler
example in the tutorials. Of course I may be wrong here. The trigger for me
was the term "Fetcher" here..
[http://tour.golang.org/#73](http://tour.golang.org/#73)

Fun exercise though.

~~~
kngspook
As a former employee, that was my first thought when I read the post too.
Every once in a while, I would stay late there and work on a little personal
project for fun there. Sometimes one of my friends would be working late, and
I'd just keep him/her company. After all, if I'm just going to do the same
thing at home, I might as well take advantage of coding on my big monitor and
comfy chair...

~~~
pyman
IMHO, a big couch is better than a comfy chair.

------
lifeforms
If Apple coders are reading, I noticed exactly the same issue of User-Agent
header not being used on redirects.

You can solve this by (ab?)using the CheckRedirect functionality of
http.Client to set the User-Agent again.

Here's an example:
[https://github.com/lifeforms/httpcheck/commit/07440d952d1660...](https://github.com/lifeforms/httpcheck/commit/07440d952d166002ab759873e2885e78e8fa5c61)
(The program is nothing special, just a little thing I use internally for
monitoring)

------
0x0
So they are "apple-maps'ing" search as well, hilariously using one of Google's
own tools in the process.

Would be interesting to know if/when any deals with bing/google/duckduckgo for
search is expiring, like they were with google maps. That's probably when we
will see an apple search engine.

Makes sense for them to fully control another part of the backend for
spotlight and siri.

------
yourad_io
_> So far, I have seen requests from two IPs: 17.147.18.33 (7 on 2014-10-15)
and 17.147.18.33 (7 on 2014-10-15)_

One of these should be .35, judging by the logs that follow ;)

------
struct
Perhaps a proxy designed for iOS testing?

~~~
13
I don't think so, it's only requesting HTML.

------
sean9999
This just in: Someone at Apple was doing stuff

------
jdalgetty
I've been seeing this as well.

------
jezfromfuture
Way to get someone sacked at apple.

~~~
aikah
lol,i'm not sure why anybody would be sacked because "Google product" ;)
however,I wonder why they didnt use Swift,and it's a question.

~~~
wyager
Swift was designed for coding user-facing applications, not internet
utilities.

~~~
pjmlp
Swift is a general purpose language. I fail to see what it lacks.

~~~
boomlinde
So are Python and Ruby, but most people will agree that those are unsuitable
for certain types of highly concurrent and performance sensitive network
applications.

Using the right language for a particular domain can save you a lot of
headache. Go is open source and a relatively proven choice, compared to Swift.

~~~
pjmlp
Python and Ruby aren't a suitable comparison, because they lack AOT compilers
to native code as part of their canonical implementation, which is not the
case for Swift.

As for access to the language implementation, Swift is Apple's creation.

~~~
boomlinde
The argument you made was that "Swift is a general purpose language.". Since
both Python and Ruby are generally considered general purpose languages, they
_are_ suitable for comparison to Swift in the exact sense that you chose to
describe Swift.

I'm not trying to tell you that you are wrong. I'm just saying that your
reasoning is flawed, and your arguments don't really support any particular
conclusion. "Swift is a general purpose language." \- OK? "I fail to see what
it lacks." \- Noted.

Let's only compare with native, canonically AOT compiled, general purpose
languages, though. All of a sudden there are a lot of questions. Why didn't
they write it in C? Why didn't they write it in Delphi? Fortran? Forth? The
answer is of course that beyond being a native, canonically AOT compiled,
general purpose language, different languages have different constructs and
models of particular patterns that make them more or less useful for different
kinds of tasks and development models.

Apple seemingly recognized that Swift was less useful than Go for the
particular task at hand, and while we can't really know for sure if this was a
matter of performance, development speed, personal preference of the
developers etc, we can probably assume that the choice was based on taking
several such quality dimensions into consideration.

------
hartator
I am doubtfull Apple wants to compete with Google in search.

Or maybe it's a way for Steve Jobs to get a revenge post-mortem on Google for
Android!

