
Findjango, A Week of Progress - twampss
http://lethain.com/entry/2009/mar/16/findjango-a-week-of-progress/
======
andrewljohnson
I see one major problem with your project, and that is that Google returns
good results for the types of queries you are struggling with. For example,
<http://www.google.com/search?q=django+tutorial>

Are you sure that we need a vertical search engine? It seems to me if I put
Django + other words into Google, I have myself a vertical Django search
engine.

Do you have some examples where Google fails?

~~~
lethain
There are a number of areas where Findjango--or vertical search engines in
general--can outperform a mainstream search engine. It's certainly a work in
progress.

1\. "Perfect answers" that relate to the topic at hand. For example, Findjango
has an index of Django's documentation and lists related documentation at the
top (above search results). For a large percentage of searches, those links
will answer the search immediately (also, the links deeplink to the correct
location on the documentation page, as opposed to simply dropping you at the
page's beginning).

2\. Because of a narrow scope, it is possible for a vertical search to be more
current.

3\. Because of the narrow scope, it is possible to apply custom relevancy
algorithms suited to the specific niche which may outperform generic relevancy
algorithms. Findjango has begun traveling this route, but is clearly not there
yet.

4\. The ability to collect and display targeted metadata. Although I'm not
taking advantage of it yet, Findjango knows the number of followers and last
commit date for each GitHub repository, and can display them in-line in the
SERP. For a targeted audience this can be worthwhile to add, whereas for a
main-stream search it would represented a negligible usecase.

5\. I believe that vertical search engines can be monetized much more
effectively--and while doing so create more added value for the user--than a
generic search engine. Certainly I have no numbers to back this up as of yet.

In the end, my argument is that a vertical search can outperform a general
search engine within its vertical. It is certainly not my argument that it
will inherently outperform, and certainly Findjango lacks the breadth of
content to compete thus far.

This is a long-term project, certainly not an overnight road to success.

~~~
andrewljohnson
So, I see where you are coming from, but one advantage Google has in any
vertical is the ability to do PageRank. Google crawls billions of pages, and
the results they give you in any given "vertical" benefit from being ranked by
the directed graph of all pages, not just the "vertical" pages. You seem to
think that limiting the number of pages you look at is an advantage, but I'd
suggest it's a big disadvantage. Google is very good at not returning
irrelevant results, even while using them for context.

Looking at your points above, in my experience, Google already provides
"perfect answers," which often link the docs. I have no problem with how
current the results are - they often include recent bug reports. And I think
your custom relevancy algorithms are nice, but I think you'll find that
Google's results are so good, because they crawl so many pages. Their
algorithms aren't generic - they are highly advanced indexing and linguistic
analysis, mixed with machine learning to tune the thing.

I think that if you are trying to return the most relevant web pages for a
query, it's going to be a long uphill battle to beat Google. I have never
thought to myself - "Google isn't doing a good job, I wish there were a Django
search engine." I can always answer my Django questions using Google.

When you are looking to make money off of something too, it's important to be
a painkiller, and not a vitamin. And right now, I don't have any Django search
pain.

Good luck with the thing though. I'll check back later to see how you are
doing. I encourage you not to try and compete with Google though, that is to
get in the game of "returning lists of webpages." If you want to make a good
Django search, you'll need to find a different niche. Your point about
metadata sounds interesting.

~~~
lethain
For what it's worth, this is a side project which has given me a lot of
perspective on my professional work as well, but is certainly not a "start up"
or otherwise intended to directly create revenue.

