
Show HN: GitHub News, using a peak detection algorithm to detect trending repos - zthomas
http://www.gitlogs.com
======
zthomas
Developer here, I subscribe to Github's official weekly newsletter and various
other ones but always found that they did a poor job at highlighting what's
trending. The "top trending" repos always seem to be most established projects
like Bootstrap or ReactJs.

I'm using a peak detection algorithm to find which repos are spiking in
traffic and rank the uptrend based on a normalized growth curve. This filters
out all the well established repos and actually uncovers new trends and
launches. Hope you guys find it useful!

Some notable features, there's also a better interface to search topics on
github as well as a curated topics section: [http://www.gitlogs.com/resource-
guides](http://www.gitlogs.com/resource-guides)
[http://www.gitlogs.com/search](http://www.gitlogs.com/search)

Appreciate any feedback,

~~~
kovacs
This is awesome. I had the same desire and even told folks at GitHub that
improving this report is low hanging fruit. It's been on my list of ideas to
hack on for awhile :-)

Ideally you (well, really I :-P) also want star/follow activity from people
you follow bundled in any update that's also reporting the projects discovered
by your peak detection algorithm over that same period of time. There'd likely
be some overlap in the report but you could indicate that in your report. This
also gives you more incentive to login with GitHub which is a little lacking
right now.

Anyway, thanks for doing this and great job :-)

~~~
zthomas
Thanks for the feedback, the first thing on my list is definitely more
personalization. I just wanted to get it out there and see if people liked it
first.

------
dc2
Rate Limit Reached:

You have reached the Github anonymous user limit, please login with your
Github account to continue.

Oof. I was just scrolling down the front page.

\---

Edit: After I signed in, I got a bit lower, then got the same message again
(telling me to sign in)!

Make / break suggestion: File the stats on a local server. Don't expect the
user to query Github or you're going to drop fans fast.

~~~
zthomas
Hi DC2, actually I do cache some of the results, first 10 days and top 50
repos per day. I think I didn't expect people to scroll as far as they did.
I'll up that cache later

~~~
dc2
Thanks! I was trying to find my repo, which trended about 15 days ago :)

Otherwise, it looks really good, and the algorithm results are refreshing.

------
Gys
Good name. Personally I would like to receive a weekly digest for one or two
specific languages.

~~~
zthomas
Thanks man, yup. Definitely on my todo list.

------
ecesena
I can recommend you to "split" your stream of repos in multiple streams [1],
and then create a function to merge multiple streams into a single one.

This way you can gain: 1\. detect spikes in less popular streams, and decide
via the merging function how to compare them to spikes in popular streams. 2\.
personalization, where each user can potentially get its own stream combining
the categories she's interested in.

[1] e.g., by language as also suggested elsewhere, by "category" if you can
come up with some definition, potentially by source (e.g. big company) as
well.

------
rmason
If this allowed you to filter by language it would be infinitely more
interesting.

~~~
zthomas
You can do that in the search function right now, in the future I'll add it to
the news section as well.
[http://www.gitlogs.com/most_popular?from=2016-03-03&language...](http://www.gitlogs.com/most_popular?from=2016-03-03&language=Ruby)

------
rubyfan
Can one fork this on GitHub?

I'm interested in the peak detection algorithm.

------
markmarrk
Why are you asking for "read and write all public repository data" & "read and
write all user data."? I understand the read, but not clear on the write

~~~
zthomas
I think someone else mentioned this as well, but unfortunately Github ties the
"staring a repo" to the write permission. I think I can try to separate it out
so it can ask for that scope when a user wants to use that feature.

~~~
markmarrk
Thanks for the response! Github is definitely a little unclear on the
permissions side of things. Might help clear things up if you mention what you
are doing with the permissions somewhere.

------
coder006
You get the "Success -- Thanks for subscribing" message even if you do not put
an email id in the input field! Please correct!

------
renaudpawlak
Pff. That's a vicious circle. Then trending repos will get even trendier.
Social networks are largely enough to expose that kind of "noise"/"buzz". I
think that a good metric to find interesting projects that are starting or are
not already under the spotlight would be to use the ratio: new visitors /
stargazers. Quite hard to achieve but I really think it would be more accurate
(and fair).

~~~
zthomas
That's definitely something on my mind originally. But since I can't get
traffic data from Github, the best thing I can do is to filter out the highly
established repos and weigh new repos higher in the rankings. You can see that
even with just 150+ stars gained for a brand new repo this system will show it
at the top top of the rank despite other popular repos that might have gotten
700+ gains on that day.

However, that said I do plan to improve discoverability even more with traffic
data since now I can atleast measure views on gitlogs.

------
splitbrain
I'd love to have this as an RSS feed.

------
toisanji
Is the source available? I'd like to do this on a small subset of repos I
watch, about 2k in total.

------
Etheryte
Cool concept, would love to see graphs over weekly/monthly timespans or
something like that.

~~~
zthomas
Thanks, I actually did build something like that. My original inspiration was
Google Trends (thus the UI) but the data wasn't very useful, traffic on github
is very spiky, even on a log graph it didn't look good.

------
fogleman
I don't think you should be title-casing the repository names.

------
guessmyname
Why does not this works with localStorage disabled?

~~~
zthomas
Hey thanks for the tip, it took me a while to find it but apparently react-
router 1.x had an issue [https://github.com/reactjs/react-
router/issues/1661](https://github.com/reactjs/react-router/issues/1661) and I
had to upgrade to react-router 2.x I'll push out a fix soon

------
skeletonjelly
gitlogs

Where git = github

Looks great though! Already found something sweet.

Another feature request, can you add an up/down trend indicator?

Can I ask which peak detection algorithm you used?

~~~
zthomas
Right now if a repo is trending down (from the moving average) it will have a
negative score and won't be showed at all.

The algorithm basically looks at the moving average, the new watchers gained
and the number of total watchers. It'll calculate the the rate of change in
new watchers and compare that against what it expects given the lifecycle of
where the repo is at. So a brand new repo adding 150 stars per day will beat
out an established repo that adds 250 stars per day.

