
NewsBlur: A feed reader with intelligence - conesus
http://www.newsblur.com
======
conesus
Hey folks, I'm a HNer. I created NewsBlur with Hacker News in mind the entire
time. In fact, I thought up the idea during a summer NYC Hackers and Founders
last year. For the past 16 months, I kept trying to pick a date when NewsBlur
would be ready to submit. Finally, embarrassed as I am by the bugs galore,
today is that day and I am glad to say that NewsBlur is far enough along to
show off.

NewsBlur is an RSS feed reader that tries to do two things very well:

1) Shows you the original site instead of a context-less feed. Read the
original and NewsBlur marks the stories you've read.

2) Filter stories you either like or dislike. A three-stop slider goes between
dislike, neutral, and like (red, yellow, and green). Training is super-easy
and all click-based (as opposed to you having to writing out what you like in
a site, NewsBlur asks you, semi-Hunch-style, your opinions on facets of the
site).

I started working on NewsBlur to see if I could do it. Put the AI together
with the back-end feed processing and fetching, along with the nifty front-end
of the original site. This is one of those projects where I just kept pushing
in all directions until I felt I had something good, not knowing if I could do
it at all, but believing the entire time that I was able to complete the
project.

If you have a project kicking around in the back of your mind, just dedicate
enough weeks and months to get it done and done well. Provided you do that,
you're almost guaranteed to succeed in some way. Those first few months of
writing NewsBlur, I felt like a fraud, wishing I could just get threaded feed
fetching to not break every few minutes. Even what I considered to be simple
python gave me headaches. (Turns out threading is evil, use multiprocessing
instead.)

NewsBlur is entirely open-source: <http://github.com/samuelclay/NewsBlur/>.
The iPhone app, the front-end JavaScript, and the back-end Python. Pick it
apart, but know that it is more a learning experience than a business. Turning
it into something profitable may not be easy, but I only need 80 premium users
to make that happen.

~~~
willydaemon
Whether or not its ultimately a success, you really developed a kick-ass
product! It looks good, seems to work pretty well. If you did this all on your
own, thats a tremendous accomplishment.

~~~
conesus
While I may have written and designed NewsBlur, I am using a ton of other
open-source technologies:

* Django: Web framework written in Python, used to serve all pages.

* Celery & RabbitMQ: Asynchronous queueing server, used to fetch and parse RSS feeds.

* MongoDB, Pymongo, & Mongoengine: Non-relational database, used to store stories, read stories, feed/page fetch histories, and proxied sites.

* PostgreSQL: Relational database, used to store feeds, subscriptions, and user accounts.

* jQuery: Cross-browser compliant JavaScript code. IE works without effort.

* Underscore.js: Functional programming for JavaScript. Indispensible.

* And many miscellaneous jQuery Plugins: Everything from resizable layouts, to progress bars, sortables, date handling, colors, corners, JSON, animations. See the complete list on NewsBlur's GitHub Javascript folder: <http://github.com/samuelclay/NewsBlur/tree/master/media/js/>

~~~
techbio
Very classy acknowledgement.

------
forgotusername
4 very rare things just happened:

1) An application announcement on HN that genuinely appealed to me.

2) A single guy beat Google at UX in one of their best web applications
(exposing the original site design for the feed is a fantastic idea!)

3) I paid for an application before it's ready.

4) I felt this was all so important I logged in to post this comment. :)

Kudos, you've done a fantastic job here. I really hope your environs permit
you to keep this project going - it's a great idea, and I'm in love with it
already!

One of the major problems I have with Reader is that certain low output feeds
(making up the majority of my subscriptions) just seem to blend together - I
forget why I subscribed to them, and have to go back to the original web page
to try and figure out who the person is. Exposing the original page by default
solves this beautifully. :)

Great work!

Edit: something I forgot. Visual display of the site allows me to instantly
demonstrate RSS to my mother. Google Reader confused the hell out of her. :)

------
petervandijck
Very impressive, the UI stuff has clearly been thought about. At the same
time, I found it a bit confusing. Also, landing straight in the app is cool,
but it doesn't tell me how it's different, so I don't know why I would spend
the time to figure it out.

It seems to have a lot of mysterious features like "instafetch" and
"intelligence trainer" and stuff.

I guess I'm saying that my first impression is that it looks nice but
complicated, like there's the promise of great stuff there but it seems like
it would be hard to figure out. Compare with Google in the early days: great
technology, crazy fast, but all that tech is hidden in the background and you
don't see that, you just see the search box and the results.

Perhaps you've been too enamored with the cool/powerful features and haven't
focused enough on keeping it simple?

But definitely awesome!

------
btipling
As a front-end engineer who used to work on Bloglines and helped create the
Bloglines Beta, I say good luck! Your UI is very much like the Bloglines Beta
and I realize this probably took a lot of work. Your challenges will probably
be maintaining read state, security issues with trusting content in feeds, and
keeping crawlers running so you don't miss items, searching content, handling
archives, and more!

And of course, there's no money in this and the use of feeds are declining (so
integrate Twitter and Facebook API).

Anyways, very nice. :)

------
llimllib
Thoughts:

* FiveThirtyEight breaks your frames.

* I want a listing of all my unread articles to browse... I have >600 feeds and the list of blogs is long at just 64

* All NYTimes blogs seem broken? Freakonomics is super slow. An important feature of Google Reader is its aggressive pre-loading. When I click a headline, the feedback is _instant_. This keeps me reading (probably more than I should. Which is good for you, the feed reader seller.)

* On APOD, I just get a blinking cursor in the bottom window, and "original" mode works but "feed" and "story" don't seem to?

At this point, I'm just going to chalk it up to your site having issues, and
check back again?
[http://img.skitch.com/20101026-tte8pqeksjx94syhuh7cbkt418.jp...](http://img.skitch.com/20101026-tte8pqeksjx94syhuh7cbkt418.jpg)

I _really_ like your idea and will probably purchase it if it starts working.
I'd also be happy to be a beta tester/constructive criticism provider/example
of a heavy feed reader user.

~~~
conesus
Wow, thanks for the great bugs. So, the site isn't having any issues that it
wasn't already having (bugs, the usual). What you're seeing is oddities in how
the iframe is working. It's not a perfect process and I'm trying to figure out
better ways to encapsulate the original site.

As for speed, I watch my logs like a hawk and have graphs for how each feed
performs. No feed should take more than 500ms to load (and only a very very
tiny percentage have taken more, based on my graphs from today. And nothing
took more than 1.5s). Speed is a super high priority for me. Moreso than many
features. Preloading is hard to do. I have no idea what you want to click on.

The iframe buster buster code works about 95% of the time. It should catch
most NYTimes sites and switch you automatically to the Feed view. It doesn't
always work.

Fixes for these bugs will be coming in the next few weeks as I sit down and
identify the culprits.

You should still go premium, even if there are a few bugs still lurking. Check
the features board. I have updates regularly and have for months. It's a long
road ahead and things only get better from here.

~~~
llimllib
Totally unrelated:

On the front page, there's no big "go premium!" button. When I went to the
site to subscribe, I said "oh, um, where do I do that again?" and the answer
was that I click on "choose your 64", which then gives me the option to
subscribe.

It should be a big honking button!

------
dmix
Good call on making preset feeds loaded as soon as you open the page, lets you
test it out right away. Instant engagement.

Also, it uses Google Reader shortcuts + imports feeds. Making the transition
easy.

Excellent way to present a product versus a strong competitor.

~~~
mattmcknight
I agree, but I feel like there should be some fill in the middle of the page.
When I first clicked on it, there was a big white space in the middle. Maybe a
carousel of prefetched articles would work?

------
trickjarrett
Google Reader import ftw. I would have just looked and kept moving if I hadn't
spotted that on the right.

Well played sir, well played.

~~~
samstokes
Very nice app, and as with the parent, Google Reader import got me to try it.

However, I was put off by having to choose 64 feeds to "turn on" immediately
after the import procedure. It meant my first experience of NewsBlur was the
difficult and uninteresting bookkeeping task of figuring out which of my few
hundred feeds I really, really cared about, and I pretty much disengaged from
the app at that point. If I didn't have to make that choice right then, my
first experience would instead have been simply my familiar feeds in a (rather
nice) new UI - and at your current price point that would probably have been
enough for me to buy a premium account.

(To give more context - I subscribe to lots of feeds, and rarely unsubscribe
except from really noisy feeds, so I probably have a lot of dormant feeds
sitting in Google Reader. I certainly don't recognise by name all of my feeds,
or which ones I'd choose to keep if I had to choose 64. The autoselect by
popularity is a good start, but for me isn't a good match with the feeds I
actually get most value out of.)

I fully understand why you'd limit free accounts - particularly with your
servers on fire :) - but I would suggest imposing the 64 feed limit after an
initial trial period, rather than immediately, so people get that initial
great experience.

~~~
conesus
Well, NewsBlur has auto-selected your 64 most popular feeds for you. All you
have to do is hit the big green button that says "Turn Them On". This makes it
obvious that the premium version is the way to go for your use case. It's very
low-barrier and gives the user a conceptual model of what eventually has to
happen for them to get more feeds.

~~~
llimllib
I think you should "shoot first, ask forgiveness later". Select just the 64
most popular feeds for the user, and then put a big button on top of their
feed list saying "Limited to 64 feeds... see everything for only $12", or some
better copy along those lines.

That is, I should go "wow!" before "awww, free is only 64", instead of the
other way around.

~~~
samstokes
> That is, I should go "wow!" before "awww, free is only 64", instead of the
> other way around.

Exactly what I was trying to say; you put it better than I did. (And your
suggested fix would have worked for me too.)

------
lazyant
This is great. One comment: I'm not sure why the landing page has a blank
central column taking over 1/3 of the space with nothing. Apparently it never
fills, what am I missing?

------
yan
This looks great and releasing the source is certainly very generous of you.
Definitely inspiring.

edit: I'm totally cloning this repo in case you change your mind :P

~~~
conesus
I mean, why not make it open-source? It's half-resume, and half-premium
accounts for a nice service.

------
bwooceli
Okay sir, you've raked in 166 points on HN (33.2/hour), we expect a full
emblogification on what happens next with this/you.

I'm going to be curious mainly about what kind of spikes you get in your
traffic, paid accounts, and leads to other publicity.

~~~
conesus
Well, I have a plethora of munin graphs, about 26 new premium accounts (I need
80 to reach profitability), and my app server barely broke a sweat. (My task
server, on the other hand, is only juuuust finishing fetching all of those new
feed subscriptions).

Next week, after the traffic and love die down a bit, I'll write up a post-
mortem with any advice I can scrounge from the effects.

------
mwexler
Some great ideas, but still missing smart deduping. This is a real pain when
having master feeds and aggregators that might also include those feeds: you
may see the same story on the same site multiple times as each aggregation
displays it. It would be nice if a tool recognized that it's been read and
marks it as read in all the aggregation feeds someone might also have in the
same folder.

~~~
conesus
You're right. But this feature is also completely out-of-scope for the type of
reader I'm making. Maybe one day, if I'm sitting on a pile of money and have
more time than I know what to do with, I may enter into the aggregation
aggregation business, but it doesn't sound fun. I think a very small
percentage of users want aggregation aggregation, whereas most folks are
looking to keep up to date with an ever increasing number of writers.

------
bwooceli
I can tell you made some very conscious choices about how to position the
premium version of this tool, but from my perspective I think you've erred too
far on the side of caution. I would recommend 3 changes that would keep the
spirit of "not in your face" premium pushing, and yet increase the take-rate
on paid customers.

* Be up-front (on the landing page) about what you get for free and simply allude to the fact that a power-user option is available

* The dashboard needs to know if I'm using this enough to warrant a premium account. On a dashboard note: once I've clicked a feed, I need an easy path back to the dashboard.

* I need to be able to broadcast somehow that I'm using this. If I "like" a story, why can't I have a goo.gl shortened url to the story wrapped by NewsBlur post to my twitter/facebook??

Great work overall!!

~~~
conesus
That dashboard idea is priority #1 (next to finishing up the iphone app). But
I want it to be a really swell dashboard. Graphs, relevant data, not just the
server puking aggregation on you.

I'm hesitant to go overboard on branding and pushing premium. Notice the logos
nowhere to be found when you use the site. It's like a desktop app for the
web.

And I'm also hesitant to wrap content in URLs like that. I don't know,
shouldn't the URL go directly to the original writer?

------
shrikant
Wow, awesome. This looks very very slick, and the Google Reader direct import
is just...nice!

Bug report, though: A couple of my blogs that I chose to "turn on" only show
up as their comments feed. If I go ahead and turn them on, and click on the
feed itself in the left pane, I get the site (fair enough), but the list of
items in the bottom pane then only shows up the comments feed items. And
clicking on them takes me to the comments. weird.

Haven't been able to figure out why. E.g. the XKCD blag
(<http://blag.xkcd.com>) and Cheap Talk (<http://cheeptalk.wordpress.com>)
feeds.

p.s: when this bug is fixed, and archiving support added, I'm totally premium-
ing - hold me to it! Great work once again!

------
pak
You've done a really good job here. A news reader that lets me both rate news,
theoretically learns my preferences over time, and lets me read the entire
article without clicking more than once, is a serious package. (Why can't
other readers get those things right?) My only problem is I'd prefer a native
reader app to a web-based one, just because the major upside of using a reader
is that it's constantly fetching and storing my feeds so there's no wait when
I want to go read some news. I may check out your iPhone app though.

~~~
conesus
If you want constant fetching, you want a premium account. Premium accounts
get updated 10x as often, and allow an unlimited number of subscriptions.

The iPhone app is not submitted yet. Feel free to play with it:
[http://github.com/samuelclay/NewsBlur/tree/master/media/ipho...](http://github.com/samuelclay/NewsBlur/tree/master/media/iphone/).
I still have a few screens to make, and a ton of bugs to sort through. Maybe
January?

------
nopal
How has Django been working out for you?

~~~
conesus
Well, the site is very front-end JavaScript heavy, so most of my time is spent
on JavaScript. For the most part, Django has been great. I grew out of many
relational SQL-based models and moved most of the key-value type data to
mongo. (Stories, classifiers, feed load histories are all in mongo. Feeds,
subscriptions, user accounts are still in postgresql.) And the feed fetcher is
all custom code to run distributed and consistent.

Django has stayed out of my way and that's been nice. I've used Django on
other projects (specifically Storybird: <http://storybird.com>, an
collaborative storytelling startup) and it's been the moon for us. Absolutely
incredible. And I write RoR at DocumentCloud and it too has been phenomenal.
They're both great. Choose the language, not the framework.

------
natep
Congrats on going from 'half-baked' to launch in 3 days!

Now that you've launched, can you take a look at the Hacker News 100 feed
(<http://feeds.feedburner.com/newsyc100>)?

Also, I don't know if this is intentional, but the feed view for a subreddit
shows the subreddit page, and each item in the feed just scrolls through that
page (instead of taking me to the comments of each link)

------
kevinp
Maybe I've allowed Dave Winer undue influence over me in this regard, but I
can't see giving up Google Reader's All Items "river of news" view. Are there
plans to offer this presentation style in NewsBlur for those of us who'd
rather not visit the original sites? It doesn't seem incompatible with the AI
management feature, especially if you can do better than Reader's "sort by
magic" setting.

------
boundlessdreamz
How are you handling sites with frameuster scripts ?

~~~
conesus
Handling iFrame busters? By using iFrame buster busters, of course!

When I detect an iframe buster, I prevent it from doing its job, but instead
of continuing to show the original site, I switch over to the feed view. It
complies with the wishes of the parent site, and I don't look like a bozo with
such a glaring and awful bug. The feed view is still a great view. (It's the
Google Reader view.)

~~~
boundlessdreamz
That's awesome. That's perfect tradeoff between the wishes of the parent
website and convenience for your users.

I'm assuming you fetch feeds using a queue.

* How do you add jobs to the queue ? Do you run a cron every n minutes to check which feeds need fetching and add them to queue ?

* When you add jobs do you check to see if a previous job for fetching the queue is still in the queue or do you use flags to mark feeds for which the jobs are in queue and eliminate duplicate jobs that way ?

~~~
conesus
Yes, I use celery/rabbitmq as a queue. But I don't bother checking for
duplicates. If the feed is still in the queue by the time it comes back around
for processing again, I let it run. That means that the queue is behind enough
that it will even out in the wash.

------
manveru
So, I just signed up, been planning to write exactly this application for a
few months now, but realized that I have to learn a lot more about neural
networks to pull this off. You saved me a lot of time and effort.

Faced with the limit of 64 feeds, I even signed up to the paid version, but
now it still only allows me to import 64 feeds.

Also, is an HTTPS version coming?

~~~
conesus
You can actually import any number of feeds you like, evne with the free
version. The "Choose Your 64" dialog allows you to mix and match which ones
you want turned on. Once you're a premium user, all of your feeds turn on
automatically. The home screen should say you're a Premium user (and the house
should be in color.)

And SSL is a bit of a ways off. I'm paying for the server out of pocket. I
still need 62 more premium subscribers to be able to pay for the hosting fees
(app server, db server, and task/fetch server).

------
jfb
Very nice so far. Seems to be stalled on the GR import. I like the UI so far.
I don't know how much of the learning I'll make use of -- I usually only
follow feeds where I want to read most everything, so further discrimination
isn't really helpful to me, but having a good UX on a feed reader that isn't
Google's makes me happy.

~~~
conesus
Yeah, there's over 10,000 feeds in the queue in front of all new users right
now. I fixed a bug which was causing all new feeds to queue, instead of just
new feeds that the user is activating. (New feed = feed never before fetched.
So most feeds have been seen more). That 10,000 would've been a 2,000, but it
still would've been too much in a single hour to handle. (That's 10,000 OVER
the 14,000 feeds beings fetched already, so it's catching up quickly.)

Give it a few hours and it'll catch up and all will be well.

~~~
jfb
Seems to have mostly caught up. Very good. The only features I would add (you
might not -- it's your app after all) are the ability to default to Feed view
for all feeds, and an 'all unread' bucket sorted by inverse posting time.
Otherwise, it's well ready to take over from GR for me.

~~~
conesus
Yup, soon. Those are going to be preferences and enough folks have requested
it that it will make it in within the next couple of weeks.

------
balakk
This looks very cool, but can't replace GR(for me) for two reasons:

\- The whole point of GR is I can read the original site even when it's not
available - works nicely with restrictive filters etc.

\- It's slower than GR - for obvious reasons.

However, it's great to see somebody taking a stab at an alternative news UI.
Good luck on your project.

------
davidedicillo
Interesting the fact that is definitely going again the trend of extracting
content from websites to present it in a more readable format (instapaper,
safari reader). I like the general interface, but I'm not a big fan of having
to ready my rss crowded with ads and banners.

~~~
drats
I am on a bit of an anti-Apple binge over the last week on HN, but I will
point out that Safari Reader, like most Apple "innovations", occurred
somewhere else first, in the Readability extension for Firefox and Chrome
released almost a year before SR that thousands of people were using, which
was being recommended non-stop on Reddit and HN. Can we please not let another
mythology grow up around Safari Reader as with all the other technology which
Apple appropriates.[1]

[1]<http://news.ycombinator.com/item?id=1812967>

------
Indyan
I don't use Google Reader, so I would love to have an option to import OPML
files. Edit: Scratch that. Found the option I was looking for. But, don't like
that uploading an OPML resets existing feed list. It should ideally check for
duplicates and add the new entries.

~~~
conesus
You can import OPML! When you login, just click on the green + button in the
bottom-left. (There's a callout that says, "First things first..." right above
it). There you can easily import OPML.

~~~
Indyan
Yeah. Spotted it just after posting the comment (see above)

~~~
conesus
Yes, I know it should look for duplicates, with an option to erase.
Unfortunately, this was too low on the priority list. But it's sitting there,
waiting to be implemented sometime in the next few months.

------
dstone
Have you thought about using Google Reader as a backend instead of keeping
your own database of feeds? It seems like that would cut down on your
processing of the feeds and enable you to just work on the stuff you care
about (the filtering and UI aspects of it).

~~~
conesus
Yes, I have thought about that. (Ohh, how many times I wanted to switch to
this while developing.) But that means I would lose a whole lot of control, as
well as the ability to perform real aggregation and intelligence on those
feeds.

That, and I get to build a full-on distributed feed fetching and processing
service, which has been a brutal, expensive, and wholly worthwhile experience.

------
sp4rki
Looks amazing to me, don't know yet if I'll be using it since I kinda prefer
desktops apps, but I'm definitely going to try it for a while, and that's
something that I almost never do regarding rss apps. Overall great great job.
Best thing I've seen all month.

------
sudhirc
This UI looks very similar to MS outlook and I really it.However unlike
outlook you cannot have all 3 panes vertically.On large screen ability display
data into 3 vertical panes makes it more usable. are you planning to add this
feature sometime in future?

------
FR6
-Why is there a "optionnal" label besides the password field?

-I really liked the little arrow, that we can lock on a page and allow us to create a bookmark like. It's innovative.

-It's really great that you released the source to the community!

Good luck!

------
pclark
I suspect the learning is not needed, I like the Google Reader learning
because it's so straight forward. (what I read, I like)

I love the reading experience you've designed, though.

------
achompas
Awesome site! Upvoted.

Signed up and imported my Google Reader account, but some feeds are taking
forever to load. Some other feeds don't show up at all after a while.

------
thereticent
As an inveterate Google Reader user, I say well done! One problem: the
"Preferences" dialog has no text, just two blank radio buttons (using Chrome).

~~~
conesus
Future features. They say ship while you're still embarrassed, right?
Preferences are a high priority and will find their way in within the next
couple weeks.

------
pclark
I get this error when I try to Google Reader import: <http://grab.by/73TI>

~~~
conesus
:-( I've seen that before and haven't been able to figure out why. I'm
definitely passing the scope parameter. (scope=reader, or something like
that).

After the hailstorm of traffic dies down, I'll look into this error. Thanks
for the report.

------
kmfrk
If this ever takes off, you might be able to feature other blogs' feed on the
demo page for a fee.

------
Indyan
Says that an iPhone app is in the pipeline. Once that is done, please make one
for Android also.

------
madhur
very cool! love the interface. just signed up and trying to import from google
reader. fetching feed is happening but very slowly. hopefully its just because
the server is overloaded temporarily.

------
sogjis
Is it possible to: \- change password \- delete account

~~~
conesus
Heh, soon. Those are lesser used features that just haven't seen the light of
day due to higher priority features. They will happen soon, though.

------
_pius
Very nice concept and UI ... great job!

------
d4ft
One gripe: When you click an article it should open in a "_blank" window/tab.
I hate having to click back.

~~~
conesus
This will be a preference within the next couple weeks. Lots of other
priorities, but this one is definitely high up there.

------
weegy
Great stuff

