

Open Sourced: Newscombinator Front End - tomw1808
https://github.com/tomw1808/newscombinator

======
tomw1808
What happens when you are on the HN top site (beside you open a beer and enjoy
that little tiny moment after months of hard work):

a live snapshot
[http://i.imgur.com/0KbPjcx.png](http://i.imgur.com/0KbPjcx.png)

~~~
tomw1808
Thank you everyone. Hard to describe the feelings, somewhere between moonwalk
and break-dance.

another snapshot:
[http://i.imgur.com/CBYRvV3.png](http://i.imgur.com/CBYRvV3.png)

server load peaking @2.00 seems to be stable

~~~
tomw1808
After a short break, still on the HN/top. Seems like america is sleeping now
and europe is not yet quite awake :)

another snapshot:
[http://i.imgur.com/LcgXdFy.png](http://i.imgur.com/LcgXdFy.png)

still everything running stable.

------
PeterWhittaker
Cool, I like it. One request: Start all stories collapsed. Right now, the
Ellen Pao story is at the top, with about two dozen related headlines beneath
it - that takes up a lot of screen space (every time I refresh I have to
collapse that space).

It would also be good to be able to collapse/expand by clicking on the
headline count.

I'm not sure how I feel about the continuous scroll. Often I like that, but
here I am unsure, but it may be because of my particular use case: I use
LinkVisitor to mark all links as visited, so that next time I know which I've
already read or decided to ignore. Having a fixed number of headlines makes it
easier for me to recognize that it is time to hit cmd-fn-f2. If others get a
lot of value from the continuous scroll, don't worry about it, mine is likely
quite the edge case.

~~~
tomw1808
Hey there, thanks for the feedback.

Its kind of a funny thing here with that Ellen Pao case. I didn't see similar
stories for quite a while on the very top and exactly now there is one.
Shouldn't be an excuse, rather something I literally just learned about my
app. And if everything is collapsed at the very beginning, it also looks kind
of odd (for me at least).

Thanks for the great feedback. Considering its an angular app, it shouldn't be
that hard to have a switch to choose between endless scroll / normal
pagination. But its a long way to go. I'll put it on the list :).

------
kriro
Seems really cool, it gets the great honour of being a pinned tab in my
browser (ńext to HN).

I think a little icon that indicates the source next to the title (on the
left) would be really useful to quickly scan what media ranks where and maybe
only select the top stuff from your most trusted source etc.

~~~
tomw1808
That is, indeed, a great honor. I didn't expect my Saturday ending at home,
monitoring the servers - in a good way.

The icon is a great idea, now the problem: What happens if a link was on 8
sites (2 x reddit, 4 x ycombinator, datatau, makernews). What would you expect
next to the title?

------
karbon
I also have been exploring what it may mean to automagically categorize news
content.. I took an approach on [http://www.rivyr.com](http://www.rivyr.com)
of following RSS feeds.. then reverse searching the titles against my product
database to attempt to autoclassify news by product. Wish I could see your
backend as that's where the secret sauce is but definitely understand not
doing that ;)

~~~
tomw1808
That is super interesting. I'm actually working more closely with Apache
OpenNLP for feature extraction and LibLINEAR to classify the news according to
their category. Its not too far from coming out of alpha to an open beta
version. Why not subscribe to the mailinglist and I'll shoot you more infos
over in the upcoming weeks?

------
serkanh
This was something i was planning to build soon for myself. Great job on the
UI.One thing kinda bugged me was when you select the source from the sidepanel
it collapses for each selection not allowing to select another source without
collapsing. I am on OSX Yosemite/Chrome-Version 41.0.2272.104. Also is the
backend Elasticsearch? Is the backend elasticsearch?

~~~
tomw1808
Hi. First: thank you for the kind words. Second: I'm going to fix that, I
know, it already annoyed me today and I just implemented the collapsed sources
last week for better readability.

The backend is Solr in combination with a custom scraper in java and a lamp
stack. The REST interface is wrapped around a php for better auth handling -
and mainly because I'm a big fan of the Zend Framework and Pimcore [1] - one
of the best CMS I've come across in my life (I'm not working for them).

Although I cannot give you access to the full source of the scraper/backend
(at the moment), why not get in touch and/or follow on twitter/mailinglist.
I'm going to send out updates what problems I was struggling on and how I
solved it from time to time.

[1] [https://www.pimcore.org/](https://www.pimcore.org/)

------
charlieegan3
The related stories feature's cool. I made my own tool for reading HN (and
some other sites) it's kind of different as it pulls the stories into a linear
feed. I'll need to have a go at adding related stories. My site's
[http://www.serializer.io](http://www.serializer.io) if you're interested.

~~~
tomw1808
I saw it today in the HN Notifications Thread (I think) and was already
impressed, its great work, imho.

~~~
charlieegan3
Cheers! Nice to know I'm not the only one that thinks it's cool.

------
zo1
Btw, your "Btw, JSON anyone?" link is broken:

    
    
        {
          "error": {
            "msg": "undefined field: \"log\"",
            "code": 400
          }
        }
    

Otherwise, great work! Looks very nice!

~~~
tomw1808
Interesting, as it works normally for me. Which browser you're on, if I may
ask?

It should just open a new tab with the same request as sent to the REST
interface anyways in the background...

Like that one [http://goo.gl/eLiUT1](http://goo.gl/eLiUT1)

------
sandGorgon
this is interesting - could you talk a bit about your solr schema ? In a very
naive way I would be simply storing it all as a indexable text blob. Did you
do anything more, just so that you can avoid a database ?

~~~
tomw1808
Sure. Well, basically Solr is just a flat index, although you can store now
mutliple-values in one field and even make joins. Here is the clou: Everything
is preprocessed and stored in a mysql-db and solr just "caches" everything.
Its some kind of the flattened mysql schema. So I have a combination of a
relational db and all the benefits with the huge bonus of the speed and
reliability of solr in between. I started of with just a MySQL db, but real-
time scoring is really a pain in the "back" with mysql and I couldn't figure a
nice way to do it. On the other hand, if you expand a story, you see that the
comments/upvotes from the sources are fetched, because joins like that are no
possible in solr (read: I couldn't find a way to do it).

So, all the information is preprocessed at index time and solr is basically a
text-blob spread across multiple fields, yes.

------
gabrielgiordano
Cool, thanks a lot :)

------
_RPM
why is it written in angular?

~~~
tomw1808
why not?

~~~
tomw1808
Let me comment more on my short "why not" \- here is why yes:

First off: I don't choose a language because its 2014, 1999 or 1985. Or
because its hip, cool or just "in". I choose a language for me, because it can
help me achieve what I want to do in a reasonable amount of time. If the
outcome is great, even the language is shit, its still great outcome. If the
outcome is aweful, no matter how great the language is, the outcome is still
not good. And I think the outcome is okay in this case and the framework is in
my opinion pretty cool (considering I started JS first on IE5.5), so I cannot
be blamed for being happy with a framework like this :).

And, based on my opinion, I choose angular, because I have learned it first
and to love it last year. Great community and good documentation imho. I think
its the wrong move to change the unterlying API with the new verison. If that
really happens, then I maybe switch, learn something new, better - maybe.
React is awesome, plain Javascript too imho.

Thoughts?

