Hacker News new | past | comments | ask | show | jobs | submit login

When I worked on a video search engine for UK football (soccer) clubs, we made the whole thing static.

We figured that inputs to the search was from a static range, i.e. these players, those games, that league, this type of incident (foul, goal, celebration, etc).

Then we pre-calculated all possible combinations and fired them through what we called a "cache cannon".

It was highly parallelizeable, simple to store on disk (we stored JavaScript files whose names were the form inputs), and worked extremely well.

Even for something like a search engine, unless you're doing full text search over a very wide corpus, you can look at pre-populating a cache and that cache actually being stored on your web servers and being directly addressable.

The design above, allowed that search engine to work over the weekend peak of 2 million users. That's where it shone... we just did not have to worry about the thundering herd with a pre-populated cache.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact