

TweetRex launched: Twitter friend recommender built on the Directed Edge engine - wheels
http://blog.directededge.com/2009/03/24/announcing-tweetrex-twitter-friend-recommender-built-on-the-directed-edge-recommendations-engine/

======
apgwoz
There's an OAuth beta now. I'll wait until they make use of it.

~~~
wheels
Originally I wanted to do all of the password wrangling in javascript so that
it never hit our servers. We only need it to make the follow buttons and the
little viral message work (we ask for confirmation before posting and say
exactly what it'll post to minimize the evil), but using HTTP basic
authentication with Javascript seems to be a no-go, so we just bounce the
request off our server. All of the data that we pull is just public data
that's grabbed with our white-listed account.

I'll look into OAuth in the future. At the time I just saw in the Twitter API
docs, "For the time being, HTTP Basic Authentication is the only supported
authentication scheme."

~~~
FictiveEvan
What problem did you run into doing HTTP Basic Auth with javascript?

~~~
wheels
Security error. Which seemed to kind of make sense. Didn't seem to be allowed
via Javascript. Googling now it seems like it might be possible if you write
the header to the AJAX HTTP request explicitly. I might play more with that
later. Granted, there, for most users, the difference would be minimal since
most presumably wouldn't read the page's Javascript to verify what we'd said.

------
hajrice
This really encourages my team and me to continue developing a major engine
that'll parse content. It's so awesome that this product was built, but I'm
not sure how popular it'll be. From my perspective, it has extreme pottential.

------
epi0Bauqu
A few suggestions:

1) I would have sent the tweet about your site IF I could have revised it,
which wasn't working for me (not sure if it should have).

2) It would be great if you could expand people's stream to see more than one
tweet (if wanted).

3) It would be great if you could show the score, and at the same time give
many more recommendations (perhaps with a more button). That way, you can
still claim it wasn't a great match (by showing the score) but still giving
more to those who want them (like me).

~~~
wheels
All good ideas and things I had to fight off in one-more-thing-ism mode before
we put up the beta. Those should all appear in a soon-ish update.

------
champion
Cool, looks nice. There are several other friend recommenders out there:
whoshouldifollow.com (which I helped build), Twubble, MrTweet.

I'd be curious to know more about the recommendation engine -- my
recommendations skewed toward popular entities like "Whole Foods", "Zappos"
and "DiningFever". Interestingly, although I follow a lot of tech/web/rails
people, I didn't see any obvious folks in my recommendations.

~~~
wheels
We spider using random graph walks, and it makes sure that some _x_ users in
_y_ directions and _z_ steps away from you are spidered. However, this was the
first time we'd tried to run the engine on an "incomplete" (meaning that we
don't have the whole graph) data set and there are some quirks in the
recommendations. It was hard to tweak the algorithm, because until there was a
bunch of data, we weren't sure what tweaks would be necessary.

The skewing happens because we first check those _x_ and _y_ to see if we've
already got enough data before running the spider. If we happen to have
already spidered heavily in one subset of your friends, then you'll see the
results skewed towards those until we end up spidering the others.

The _other_ skewing, which other works better on complete graphs is that it
tries to do some weighting based on how many incoming and outgoing edges users
have (friends, followers), which if your tech friends have a whole lot of
friends and followers and you don't, might be part of why they'd be less
highly rated.

For the moment we're not really trying to compete with Mr. Tweet (though, when
I tried their recommendations they were pretty bad) and whoshouldifollow
(which does pretty well) -- it was just a neat way of showing off some of our
engine and applying it to a different area.

~~~
champion
Thanks for the reply. Like the idea of making is simple for sites to
incorporate a recommendation engine without reinventing the wheel.

------
wheels
Up on a second server now:

<http://tweetrextoo.directededge.com/>

------
dhimes
When I clicked someone to follow, then came back to your page (on _too), I had
to log in again. I was away for only moments.

I selected _don't touch my stream* because I had already tweeted.

------
sandGorgon
how does directed-edge compare to LinkedIn's OSS Project Voldemort
(<<http://project-voldemort.com/> >) ?

------
uuilly
Must be flooded. It hung for me for a while...

~~~
wheels
It takes a while to pull the data from Twitter. It actually runs around 500
worker threads spidering Twitter, and once the count gets up high some of the
requests start timing out, so yeah, the whole process takes a bit.

Unfortunately, since it's a limitation in how fast we can pull data from
Twitter, just throwing more hardware at the problem wouldn't help much.
Performance may get better as we have a large portion of the Twitter graph
cached.

We expected that it'd be a little rough today when it first goes up, but on
usual days it should be able to handle a more casual load.

We debated if we should post it here since we were pretty sure it'd swamp the
connection, but probably things will level out a bit by the end of the day.

Edit: Now it's hitting the VM pretty hard since once all of those threads are
actually pulling, well, data, things slow down. I may set up another VPS with
more RAM here in a bit and do a DNS swaperoo. We'll see how things go.

