

A Duck & a Wiki Team Up Against the Content Farms - JackHerrick
http://www.readwriteweb.com/archives/a_duck_a_wiki_team_up_against_the_content_farms.php?sms_ss=hackernews&at_xt=4d4dea696c2807b0%2C0

======
estel
Sometimes, whilst DDG's partnerships can be great for getting quality content
to the top of search results; it's definitely not the sort of thing I'd want
to see a dominant search engine doing. DDG obviously needs to do things
differently in a way that can drive them traffic, but if Google were to do
this, they'd quite rightly be blasted because such partnerships could
massively impact the ability of other site's to grow organically without the
engine's blessing.

~~~
melvinram
I agree. Google's prominent ranking of Wikipedia articles usually drives me
nuts unless it's the only good article around.

~~~
redthrowaway
Google's ranking of Wikipedia articles is due in large part to Wikipedia's
great PageRank ranking. I don't think Google is giving Wikipedia any sort of
singled-out treatment, Wikipedia just ranks highly on Google's relevance
metrics.

~~~
zone411
The problem with Wikipedia is that many of their articles are poorly
regurgitated versions of other source articles from the Web and that these
source articles get no Google credit at all, even if they are linked from the
Wikipedia articles, since Wikipedia uses no-follow links. Wikipedia ranks near
the top in Google for almost any term for which it has an article, regardless
of its quality.

~~~
mahmud
Not always. Just yesterday I was knee-deep in wikipedia for linear algebra
stuff and was surprised to find it more thorough than MathWorld.

~~~
ErrantX
That rough area of maths is one area well covered, after military and
astrophyiscs I would say.

------
petervandijck
Google has mentioned they're working on the content farm problem. I would
expect them to make some moves faster that are smarter than banning content
farm X while partnering with content farm Y. Likely algorithmically, coz
that's how they roll.

~~~
timr
Google would do well for itself to stop horsing around with attempts to find
'algorithmic solutions' (if that's actually what they're doing), and in the
short-term doing what's necessary to make a perceivable improvement in the
quality of their product. Even if the content farms were changing domains
regularly, Google could probably kill them faster than they'll find a way to
algorithmically drop the ranking of their results, in aggregate.

For all of the recent noise that Google has been making about fixing this
problem, the number of gamed search results in their index is still insanely
high. Take plant diseases for example...every time I've made a search for some
specific problem related to my houseplants, the results are so loaded with
content-farm crap that I have to go at least a page or two in to get to
reliable sources of information.

We're not even talking about _relevant_ webspam from plant stores or online
merchants...just eHow garbage, which is poorly regurgitated from the other
sources that come lower in the results.

~~~
petervandijck
Yea, I think you're right there.

Although, perhaps we underestimate the scale they deal with. Plus, manually
removing sites that are not clearly evil but on the border may become a PR
nightmare.

------
zone411
I'm sorry but I have big problems with calling DDG a real search engine. Their
page states "DuckDuckGo is a search engine like Google." BS. It's a meta
search engine that relies on other real search engines, such as Bing, to get
the results. If ever becomes a threat to Bing, they'll cut them off in a
second. DDG doesn't do the hard and resource-intensive work of crawling and
ranking the pages, they just tweak the results of others in ways that gets
them publicity on tech blogs.

~~~
epi0Bauqu
For the record, I've been crawling the Web since the beginning of DDG (and
before the BOSS or the Bing APIs existed). Yes, a one FTE, self-funded search
engine, does not have the capacity to spend millions of dollars on crawling
(does Google spend billions?).

I now focus my crawling efforts where I see they can add significant value,
namely on spam removal and zero-click info. Our index actually shows the top
result about half the time, which is not from any external API. Given that
most people click the top result most of the time, this is not an
insignificant addition.

We of course use external sources, but I think your comment is too dismissive
of the intelligence we've woven on top and through them. I'm not interested in
sharing all of DDG's trade secrets just to prove a point, but needless to say,
you can simply compare the external APIs to our results across a swath of
different types of queries and see the differences.

Finally, I find the premise of your comment a bit short-sighted, but I'm
actually OK with being dismissed as a toy. In the eyes of the average user,
they don't care where results came from at all. They just want the right
information faster. So quite frankly, I think it is a reasonable move to use
external APIs and focus on things like the top result, UI, etc. More recently,
Yahoo and Ask seem to agree. Most people are dismissive of them too, but
they've been doing a lot of innovative UI things, which matter a lot to end
users.

~~~
petervandijck
Being dismissed as a "toy" is probably the best thing that could happen to DDG
right now, and for the foreseeable future. They're searching for an angle
against Google, a way in. They may well find one (although it's a tough
battle).

------
bryanh
DDG is supplanting labor intensive algorithmic improvements with manually
manicured search results. Compared to Google's vast network of resources, it
is a logical step for DDG to take. It is, essentially, a fast fix to a real
problem.

Google has to do things the hard (and dare I say... proper?) way.

~~~
InfinityX0
_DDG is supplanting labor intensive algorithmic improvements with manually
manicured search results._

This sentence is a paradox.

~~~
mkramlich
To illustrate what he meant, I'll give an example. Assume someone entered a
search, "What are the best dog breeds?". Would it be easier to:

1\. write an automated system which could dynamically deduce what the answer
to this question is and/or what pages to return in the results, based on
dynamically crawling the web, indexing, analying link structure, ranking,
filtering, etc.

-or-

2\. have a human write an answer to this fricking question and enter it into
the database as the result for this search

hint: pick 2

it doesn't scale up to handle bazillions of different searches, but on a
smaller scale and in tactical cases it's far simpler and easier to implement
and can produce better results.

------
CurrentB
DDG has a laughably small market share. This doesn't necessarily make them a
non-threat, but it's going to take more than this to change users' search
behavior (ie convert them from Google to DDG) on even a minute level. I mean
Google has the whole page preview thing now anyway, and the content farm thing
should be fixed soon, as mentioned. Honestly, Google Instant is the best thing
in search for a long time and it's so ingrained in my expectations now that I
couldn't see switching to a search engine without it.

So asking what Google will do isn't really an important question.

~~~
babeKnuth
their market share may be laughable, but that still doesn't eliminate the fact
that many knowledgeable people prefer either blekko or duckduckgo over google.

it can definitely be viewed as a "calm before the storm". or if you're
familiar with gentrification, the "cool cats" are living on this side of town.

~~~
chc
Yes, many knowledgeable people prefer Blekko or DuckDuckGo. But that's "many"
in the absolute sense, where "many" people are part of any arbitrary group on
a planet of 7 billion.

Relative to the size of the group we're talking about, even among geeks, using
DuckDuckGo as your main search engine is rare. It certainly doesn't look like
DuckDuckGo has a glut of tech industry leaders using it exclusively — mostly
just the odd geek here and there who wants to try something new or who has
just decided he doesn't like Google.

------
Vivtek
How odd that even a year ago it seemed that the search engine space was closed
to new entries, even solved - and now it feels wide open again.

~~~
citricsquid
I don't think so, the people using DDG are still a tiny tiny minority. If they
did what google did with the browser market when they released Chrome then you
would be warranted in saying they blew it wide open, but for what DDG has done
with search? I don't think so, not yet anyway. Maybe when they've penetrated
beyond the tech "elite" they might get somewhere, but right now if I ask any
of my friends (who are relatively in the know about technology) they haven't
heard of DDG.

Personally I don't think DDG will show wide success because the majority of
people don't care about the added extras it provides, it's just other search
engines (I think their major source is the bing api? Not 100% sure [1]) data
with pretty labels and positioning, which very few care for. I search a
relatively large amount and what DDG adds doesn't make it worth switching for
me because I've become accustomed to how Google displays search results and
adapted my behaviour to that.

[1] [http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go-
arc...](http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go-
architecture.html#IDComment79599091) (potentially no longer accurate which is
why I'm not sure)

~~~
benologist
I think the difference is the _attitude_ is different now - when Cuil launched
the world laughed and said nothing could be as good as Google. Almost 3 years
later a small, insignificant yet maybe not, number of people are starting to
use alternatives.

A good parallel could be Firefox - Internet Explorer was _the_ way to browse
the internet, now they're just _a_ way.

If Google wants to be _the_ search engine a decade from now they may have to
earn that right all over again.

~~~
chc
People laughed at Cuil because it was laughably bad, not because nobody is
allowed to make a search engine.

~~~
benologist
Sure it was bad - but was every decision* they ever made so fundamentally
wrong that they couldn't possibly have improved? The world wrote them off
completely 8 minutes after they launched, and they were a meme like the next
_day_. (* in search, not their ridiculous perks etc)

When I search for online games on DDG I get stuff like this in the first
'page' of results:

\- <http://onlinegames.innovative-solutions-group.com/>

\- <http://onlinegames.webfreehosting.net/>

\- <http://online--games.com/>

Did DDG and Blekko both independently, really stumble upon formulas that beat
Google, or are we actually open to the possibility of not using Google for
search now?

~~~
epi0Bauqu
Thx for the specific example, always very helpful! But in this case, I'm
having trouble reproducing. Do you have a region set or something? I'm looking
at <http://duckduckgo.com/?q=online+games> and don't see those domains. I'm
not saying the other domains are good :), but I just want to understand what
you're seeing. We definitely have a problem right now of over-weighted the
domain name in the url.

~~~
benologist
I'm in latin america (Nicaragua) which might hurt a lot if it's trying to pull
localized results from this little corner of the world heh.

<http://playtomic.com/ddg.png> is what I'm seeing.

Edit:

I'm doing a search from one of my US-based servers and seeing those same
sites?

<http://playtomic.com/ddg2.png>

~~~
epi0Bauqu
Ahh, thx. Turns out I was the one with the settings set :). Duly noted!

------
rmundo
So we have the website equivalent of spam. A simple solution would be to have
a secondary page labeled "We think this is spam" where all the banned links
would reside. Anyone who needs to see what's there could always dig in.

------
tectonic
Is DDG one person?

~~~
citricsquid
Yeah, Gabriel Weinberg: <http://www.gabrielweinberg.com/>

~~~
tectonic
While I'm kind of frustrated by the anti-Google marketing, I'm impressed by
how productive Gabriel clearly is.

~~~
tectonic
Why was this down voted?

------
ddemchuk
google's not going to do anything because they're in the business of making
money off of search and duckduckgo is not

~~~
JackHerrick
I actually have a lot of faith in Google to do the right thing for their
users. After all they mostly walked away, or at least severely hampered their
business in China. They will be more than willing to walk away from sites
their users think are low quality.

I'm not saying that this means they will do the exact same thing DDG did. I'm
just saying any decisions they make will be primarily based on looking out for
their users best interest rather than short term adsense revenue.

~~~
ddemchuk
what about your statement is supported by their actual actions in the recent
years? who are their actual customers, searchers or adwords bidders? They
simply need to provide a decent enough search experience to keep users coming
back so they can keep adwords bidders participating. It's their entire
business model.

