

Please Solve the Information Redundancy Problem - codeismightier
http://codeismightier.com/2009/02/solving-the-information-redundancy-problem/

======
pj
<http://mathworld.wolfram.com/BayesianAnalysis.html>

It wouldn't be that difficult to implement this. Run your RSS feeds through a
program that will look in the articles for words you are already familiar
with.

If you read an article, you can rate it good or bad and it can look at which
terms are /in/ the article. If you rated it good, then the new terms are added
to a list of terms to look for, terms that the algorithm will think are part
of articles that will interest you.

If you rate it bad, it'll look at terms in common with other articles you've
rated bad and see know what to look for in articles it should not bring to
your attention.

It would be nice if such an algorithm would allow you to rate the articles on
different scales for entertainment, technology, philosophy, etc, so when you
are in one of those moods, you just click on the category and it'll hunt for
things that may interest you.

I think an algorithm like this would be much more suited to an individual than
a social group type filtering system like most groups use. At some point, the
individual wants to break off from the group and explore independently and to
do that effectively, they need a teacher.

In a way, this algorithm would be a teacher constantly showing you new things
in the world and keeping your interest up.

The problem with social sites now is that there are many large communities who
all at once look for a new venue for their point seeking. It promotes a king
of the hill mentality where individuals within the group compete with others
for points rather than personal excellence, education, and betterment.

~~~
codeismightier
I have implemented such an algorithm in the past. The problem is that it
actually makes things _even more redundant_ since it gives you stuff you have
already read and liked. The results returned are usually articles on the exact
same topic from a different source.

------
TrevorJ
I don't see redundancy as much of a problem for a couple reasons:

#1 If you teach yourself to scan text well, your brain becomes very good at
parsing the data that is new. I can get the gist of a one-page article in
10-15 seconds if it contains only a few paragraphs of truly "New" information.

#2 Redundancy cements important knowledge in your mind. I am willing to bet
that if you examined any 'designed' course of study such as you would find in
college, you will see a great deal of redundancy. Studies have shown that
spaced repetition and review can greatly impact how well you retain a certain
body of knowledge. If you are reading articles on the same subject in an
attempt to learn a body of knowledge there would be some benefit to the
overlap, especially considering that any new nuggets of info would be learned
in context and cemented in place next to already known facts.

