

This is probably why everyone still relies on Google - robotmay
http://robotmay.com/post/23096164275/these-are-the-crawl-stats-for-the-last-month-for

======
stefanve
Yahoo is using Bing so it is not strange that is has zero indexed pages. Also
Duckduck go is using Bing and some others like wolfram alpha. The only reason
I still use Google it that they are still miles ahead of the competition. I
have switched for a while to Bing, DuckDuckGo and others but always wend back
to Google.

I do value my privacy, for instance, I switched from gmail to fastmail even if
fastmail has less functionality. But for search there is not a real
alternative for me.

~~~
WildUtah
Wait, DDG is using Bing? I thought it was an independent and written from
scratch enterprise. Does it depend on Bing for everything?

~~~
koeselitz
"DuckDuckGo gets its results from over 50 sources, including DuckDuckBot (our
own crawler), crowd-sourced sites (in our own index), Yahoo! BOSS, embed.ly,
WolframAlpha, EntireWeb, Bing & Blekko. For any given search, there is usually
a vertical search engine out there that does a better job at answering it than
a general search engine. Our long-term goal is to get you information from
that best source, ideally in instant answer form."

source:
[http://help.duckduckgo.com/customer/portal/articles/216399-s...](http://help.duckduckgo.com/customer/portal/articles/216399-sources)

------
jgrahamc
Those stats look a little off, and I wonder if it's because the site in
question is pretty new. For comparison here's a screenshot of the crawl
information from my CloudFlare analytics panel (the same information as
gathered by the OP): <http://i.imgur.com/vrKlz.png>

jgc.org has been around since 1997, daysoutnearme.com since April, 2012. I'm
guessing that the other engines don't yet know about the site.

~~~
robotmay
Ah ha, that's really interesting. My site hasn't been around very long, but it
has been submitted to both Google & Bing for the same amount of time. Nice to
know that it balances out over time, but it looks like Google pays more
attention to new sites than Bing does. Or Google has more resources with which
to crawl sites, of which I have no doubt.

~~~
dwc
My impression, based on eyeballing logs for my own low traffic, unimportant
sites: Google collects and uses stats on how often pages change, how often
they come up in searches, etc., and for static and/or low importance pages
they scale crawling back. This frees up crawling resources to spend elsewhere,
like on important, frequently updated pages, new sites, et al. I haven't
looked closely in a couple of years, but the pattern used to be that new
content would be crawled fairly agressively and after a while it would back
off to an "appropriate" level.

Back then, other search engines took a good while to begin crawling my new
stuff in earnest, and once they did their crawling pattern did not seem to
change much over time between a (relatively) popular changing page vs. a
lonely static page.

------
yaix
I tried DDG, but Goog's results are just better. And when I do tail -f
log/access_log on my box, GBot is always there, hitting my sites every few
seconds 24/7. Sometimes Bing is around, but less frequent and much slower.

~~~
robotmay
I always knew Google hit more (especially if you're watching the access logs)
but I didn't realise quite how much more until I saw the graphs on Cloudflare.

------
asadotzler
Google search is overwhelmed by new "hot" topics rather than solid valuable
enduring sites. It's almost as if their news crawl has invaded their web
crawl. I think the rush to have "fresh" content in the first 10 links is
actually making Google significantly less useful.

~~~
gcb
Ever tried searching a business phone number in Google?

No matter if said business has a page for 10yrs. The first 2 pages well be
phone directory seo spam sites that cropped up last couple years.

------
binarymax
OT, does anybody actually know how to get Bing to index your site? I looked
for maybe an hour with no luck.

~~~
dangrossman
> I looked for maybe an hour with no luck.

1\. On the Bing homepage, click "Help"

2\. Click "Find Help for site owners"

Arrive at the answer to your question.

~~~
binarymax
That's one of the paths I went down before, got to a place where I had to sign
into windows live, signed in, and arrived at a dead end. Robotmay's link above
looks good though!

------
bmj1
Reasons people still uses Google:

 _(Perception of) better results:_ until proven otherwise - why would you use
an alternative/inferior SE - unless you value other things like privacy more
than the content of the results - this doesn't apply to most people

 _Fast/it just works:_ give me a compelling reason to switch. "Privacy" really
isn't enough for most people.

 _It's default:_ the default FF SE? The default homepage for a bunch of
people? Default search in Chrome? Check. Check. Check

 _Network effect:_ Remember the expression "Nobody ever got fired for choosing
IBM" (that might be too old for a lot of the readers here) - but the same
thing applies - "nobody ever got in trouble for choosing Google"

~~~
vibrunazo
> Default search in Chrome?

It's not default in Chrome, it asks when you first use it what you wanna use
as the default search. You have to opt in to google and bing is just one click
away. Most still choose google anyway.

In contrast, to change bing from being default search, on the default browser,
on the most used OS in the planet. You still have to go inside preferences and
opt out, which takes several clicks. And then, Microsoft still tries to sneak
bing to you if you install MSN (windows live or w/e it's called now), which
will change your default search engine on FF and Chrome to Bing. Unless you
opt out during instalation.

And still people choose google.

~~~
balac
In Europe IE will also asks you to choose a default search engine on first
start.

~~~
replax
It will actually ask you to chose a browser before anything...

~~~
wgx
...and the order the browsers appear in the dialog is randomised.

------
imperialWicket
Mildly off-topic, but even if I preferred google's results to duckduckgo's, I
would still use duckduckgo's "g! my search phrase" over google - just for the
better interface/styling that ddg offers.

~~~
robotmay
I must admit that a significant portion of my love for DDG is because of the
duck. It just looks so enthusiastic.

------
jast
It's funny... just runned the query "days out near me" on google and bing.
Bing shows the site at number one while google only shows it at number 3, 4
and 5 and never the top domain.

~~~
robotmay
Hah, that's quite entertaining. Doing a non-bubbled search on Google has it at
2/3/4/5 for me. The site at #1 is, I'm assuming, much older.

------
PaulHoule
I dunno... I have a site that would get creamed by Baidu's web crawler if I
didn't block it, but never got noticeable traffic from it.

------
loverobots
From what I can tell, Bing is very picky in indexing pages, whereas Google
will index and ignore them if not up to their standards.

I use Bing and DDG and maybe once or twice a week I need to use Google to
double check. Bing is more than fine for "most users," but Google has a
powerful brand, they have a lot of goodwill _and_ they open their checkbook to
buy traffic (Mozilla, Opera, iOS, portals etc.)

Edit: All the above (even extra Googlebot servers to index new sites in
seconds) are because Google can "monetize better," or has a lot of traffic
which in turn attracts advertisers that bid up prices. Google is also
shameless in adding dozens of ads in all places so they have lots of money to
maintain their status quo. Put Bing or DDG results on Google.com, under the
Google logo and very few "regular" people will complain.

------
alan_cx
A random guess:

I reckon that people still use google because they have learned to "trust"
google's results and they are afraid that if they use another search engine
they will miss out on a useful result.

So, I wonder if new sites should show results comparisons to give new users
confidence.

