The Netflix challenge forums are a treasure trove when it comes to this particular ML problem.
In the end the algorithms grew into huge monsters blending results from tens of algorithms, but the most interesting discoveries were made in the first year IMO.
Wow, I never thought of looking there. It's amazing reading tricks like that which people have come up with after years of thinking and experimenting. And they seem usable for quite a wide range of rating related problems.
KNN is not nearly as good as deeper probabilistic models. For example, if movie description mentions different subjects like "comedy" and "family", the model would not be able to differentiate them. It would only have a cluster "family comedies", but it won't be able to tell that a person likes "comedies".
Most of the challenge is in getting a way of assessing the value of innovations in the algorithms - how do you know how well it works ? Difficult unless you are running a large scale recommender that users can't opt out of (because pop goes your stats if the do!)
I've been thinking about something like this for a while. My idea was to use data from IMDB to create a graph where the distance between movies is some ranking of similarity based on the people involved in the movie, the genre, the setting and any other information you could get from the data set.
You could say, "I want to watch a movie like The Wolf of Wall Street" and it would find the closest 10 movies in the graph.
It's still something I'd like to play with if I find the time.
I started on something similar as a side project[1]. I decided to build a dataset from the AFI Top 100 Films list and persist it in neo4j. The goal was to find interesting questions to answer with this dataset that couldn't easily be googled.
Most of my time thus far has been spent gathering the dataset, but I do have a few example cypher queries answering the following simple questions [2]:
1) What actors have appeared in the most AFI Top 100 films?
2) What are the genres of the top ten films?
3) Have any actors appeared in 2 or more of the top 25 films?
I'm working on building a much larger data set using a combination of freebase and imdb so that I can have enough data to start exploring much more interesting interesting questions (e.g. graph the frequencies of genres over the past 60 years; for a given film, find movies with the greatest overlap in genres, actors, and directors; generalize the n-degrees-to-bacon problem to work on any two actors; etc).
Very cool. Thanks for posting this. The n-degrees-to-bacon problem is actual what made me think of this in the first place. It would be great to be able to plug in two actors and have it spit out an answer with the shortest path.
I've contemplated the idea of taking soundtracks from movies and comparing them to a user's favorite tracks.
It wouldn't provide accurate prediction of the best pick movie to watch, but it might come up with an indirect, quality pick that might otherwise never been seen.
That's awesome! What is your algorithm like for determining the similarity? There are some good answers, but The Wolf of Wall Street is apparently pretty close to Frozen. ;)
"Films similar to X" is definitly a good use case. But sometimes you can just go to the Amazon page for a DVD, and look at "people who bought this also bought that" ;)
This looks good, i will give it a try.Also GraphGists looks good way, quite similar to ipython.
I have one question: Why do i use neo4j when i can do k-NN in python with so many macine learning toolkits available? What benefit do i get when using neo4j for movie recommendation?
Neo4j as a data persistence layer has many, many advantages over other data persistence layers for graph-like problems (social, recommendation, etc.), principally from the point of view of performance.
You can do it in Python but you're going to have to persist and query your data somehow, and I think at that point you'll "get" why a graph DB might be beneficial. It's not an ML-related thing though, it's a data query performance thing.
This post was an entry in the Winter 2013 Neo4j GraphGist Contest. The author of this submission also has a blog post about doing k-NN in R [1]. In other words, this GraphGist is an example of how you could do basic ML in neo4j, but it's not meant to imply that neo4j should be chosen over python.
http://www.netflixprize.com/community/