

Show HN: A distributed Recommendation Engine based on Redis + Ruby (and C)  - paulasmuth
https://github.com/paulasmuth/recommendify

======
jamii
I've been working on something similar - <https://github.com/jamii/springer-
recommendations>

All the open-source engines I've tried scale poorly. Since our input data
already contains >200m interactions I suspect recommendify would struggle
(from my quick reading it looks like the co-concurrence matrix is stored in
redis ie in-memory).

The approach I'm leaning towards at the moment is collating the article->IPs
and IP->articles tables in leveldb and then distributing read-only copies to
each worker. Everything else can easily be partitioned by article id.

~~~
jaylevitt
I can't tell from the README - is your data fairly wide? I'm playing with
using Postgres's new K-Nearest-Neighbor support to calculate similarity on 20D
cubes, but I suspect my approach won't work well for an arbitrary number of
columns (i.e. users x products) unless you first do some sort of PCA or SVD to
narrow it down, and it isn't optimized for binary ratings at all. I started
writing it up here: [http://parapoetica.wordpress.com/2012/02/15/feature-
space-si...](http://parapoetica.wordpress.com/2012/02/15/feature-space-
similarity-search-in-postgresql/)

~~~
jamii
> ... is your data fairly wide?

Around 200m download logs, 2m articles, some million IP addresses. I suspect
that interest in research papers is inherently high dimensional and
dimensional reduction would probably damage the results.

I don't have much hardware to throw at it either. I just started looking at
randomized algorithms - trying to produce a random walk on the download graph
that links articles with probability proportional to some measure of
similarity (probably cosine distance or Jaccard index).

------
fraserharris
Terrific readme

~~~
joeblau
+1 more for that awesome ReadMe. The project looks clean and simple.

------
steve19
How does this compare with other recommendation engines, such as the
opensource EasyRec[0] ?

[0] <http://easyrec.org/>

------
ambertch
Paul, how does C fits into the picture? I don't see anything in the source for
compiling native extensions, and the implementations src/recommendify.c calls
look incomplete?

I am guessing you still working on the gem?

------
sycr
Oh man, this is just awesome. Thanks so much for making this open source -
it's great to be able to pick apart the internals of something like this.

------
chrismealy
Does any part of this involve taking the square root of something? Amazon's
original similarities did that and I can't remember exactly how.

------
moe
Oof. This is beautiful on so many levels.

