
Duck Duck Go Architecture - epi0Bauqu
http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go-architecture.html
======
almost
I've seen Duck Duck Go when you posted before and didn't think much of the
idea (hey, I can use Wikipedia!)

Just had a look again and I now think it's pretty awesome. Just for a really
speedy "what the hell is this?" type of query it looks absolutely amazing. I
love the AJAX extra search links, makes it all feel really snappy.

Works great on the iPhone as well, I've added an icon to my home screen. You
might want to consider making an app for the app store (you could pretty much
just make an app that loads the site as it is).

Have you considered adding an API? I currently have my Emacs set up to do a
dictionary lookup (using the dictd protocol) on a word when I hit C-cl. I'd
love to add Duck Duck Go results to the buffer this displays!

~~~
epi0Bauqu
Thanks! Yeah, we've gotten a lot better since then. On the zero-click info
(what is x?) we use lots of sources now beyond Wikipedia (e.g. Crunchbase) as
well as have a bunch of new features (<http://duckduckgo.com/about.html>). We
also have both an iPhone app and an API :)

iPhone App:
[http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftwa...](http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=301375111&mt=8)

API info: <http://duckduckgo.com/api.html>

~~~
almost
Whoops, should have spotted both of those :p

Both look good though!

------
njharman
I'm not exactly a search connoisseur or even pay much attention to startups
but wow. This is one of the first search innovations I've found that was any
good, in fact it's great.

I've just replaced my google FF keyword, 'g', with duckduckgo link. I'll try
it out for a week or so and see if I still think it's as awesome as I do now.

~~~
epi0Bauqu
Thanks! Please email me your thoughts after the week--always looking for real
feedback from real users. Also note we have a FF extension:
<https://addons.mozilla.org/en-US/firefox/addon/10536>

------
jwilliams
Cool article. Couple of queries :-)

It mentions that static content is served from nginx, but then mentions S3
later on as well. Why not just Amazon for all the static content?

Mentions nginx being more stable than Apache - you mean no-crash stable, or
more consistent under load?

~~~
epi0Bauqu
Thanks! I use S3 to serve images that are loaded after the initial page load
(incidentally, using YUI's ImageLoader:
<http://developer.yahoo.com/yui/imageloader/>). I use nginx to serve static
files, e.g. HTML, as well as images that get loaded on page load, e.g. logo.
Why not S3 for the latter? Too slow. Amazon's (relatively) new content network
may be faster, though I haven't tried it yet.

Wrt nginx being more stable, both. I've crashed Apache as well as pseudo-
crashed it (process still running, but not serving right). nginx hasn't
crashed yet. It's also more consistent (and faster) under load. However, in
Apache's defense, it does a lot more, e.g. mod_perl vs nginx proxy to FastCGI.

~~~
jwilliams
Cool - well I'm on Apache now, on a very undersized server - so I guess I'll
have to give nginx a shot.

Not sure how you'd extract more speed from your current setup. The natural
next step would be doing work in parallel - this something you do/have
considered?

If you do try the Amazon CDN, let us know how you go :)

~~~
epi0Bauqu
I use multiple worker processes in nginx (2 x # of cores), which work in
parallel. Then FCGI::Engine is configured to run multiple processes as well.
Finally, I use the external JS and image files (in part) to speed up the
client-side via parallel processing of requests. (Note that IE7 and below
limit concurrent connections per host by default often to 2:
<http://msdn.microsoft.com/en-us/library/cc304129(VS.85).aspx>).

------
brandnewlow
This passed my "does this not suck?" test. Good work. A search engine that
does not suck and is not named Google.

~~~
kingkongrevenge
gigablast finds stuff for me that others miss.

------
strlen
Question: why are you going with FastCGI over mod_perl? Issues with mod_perl
debugging/memory leaks? Desire to use nginx over Apache for higher throughput?
The big advantage -- to me -- of mod_perl is the power of having your code
live within Apache's process space.

I wonder if mod_perl works with event_mpm, yet.

~~~
epi0Bauqu
d) All of the above.

Initially I didn't think anything could beat mod_perl speed, but after
benchmarking, this setup is just as fast. It's also more stable, and uses less
memory. And yes, also it is easier to debug and I was getting memory leaks on
mod_perl.

Don't get me wrong, I love mod_perl. I used it at my last company and on many
other projects before this one. And I still use it for other stuff.

Last I looked I think you can run mod_perl with event_mpm but I didn't like
this: "The current Apache::DBI should be usable under threaded mpm, though it
doesn't share connections across threads. Each Perl interpreter has its own
cache, just like in the prefork mpm."
(<http://perl.apache.org/docs/2.0/user/performance/mpm.html>)

------
tdonia
complete side note - but your favicon has a lot of aliasing when it's on a
dark background (i.e. as it is when you have ddg in an inactive chrome tab).
the circle looks nice on white, but you may want to adjust it so it reads a
little better against other mattes.

aside from that tiny-minor issue, ddg seems cool & the overview of the
architecture is very clear. i'm going to give it a try as my default chrome
search engine & figure out just how behaviorally indoctrinated google has me
and how your architecture holds up as more people use the service.

~~~
epi0Bauqu
Thanks! I will look into the favicon issue. Please send me an email after you
try it for a while with your thoughts.

------
rozim
I must have missed something - what technologies are behind the actual search?
Lucene/Nutch/Xapian?? thx, Dave

~~~
epi0Bauqu
All proprietary Perl.

------
globalrev
do you make money?

~~~
epi0Bauqu
No ads on the site yet, but in general, search ads make money.

