Hacker News new | comments | show | ask | jobs | submit login
Show HN: A cryptocurrency news scraper inspired by Hacker News (cryptonewsagency.com)
183 points by howiem 10 months ago | hide | past | web | favorite | 102 comments

If you model your site after Hacker News, people are going to try to use it like Hacker News, and the expected behavior of your site is incredibly not-hacker-news-like.

1. I click on a story expecting for it to take me to the site, but nothing happens. After feeling confused, I realize it actually added some text content after the story. What? It was only after looking in the comments here that I realized I had to click the link in parentheses (which on Hacker News shows you submissions from that website, not links you to the content)

2. I see a lot of "points", so I try to understand how things are upvoted - is it accounts? No, no accounts for your site. Is it based on the links popularity (eg trending tweets, hot news, hot reddit stories, etc)? No, there are some stories with hundreds of points, but linking to a reddit post with 2 upvotes. It's only then do I realize by clicking on the title, or the link, I'm "up-voting" it, even though it might not be good content (and in fact, each time I click it adds a point).

3. I search for the ability to comment, but realize that's not there, but again since it looks like Hacker News I think there's comments and go to click on something that doesn't exist.

In general, I think your site could be really useful, and there's a lot of cool stuff you did, but I'd either change your design so people don't treat your site (interaction wise) like the site your referencing, or try to at least link up expected behavior in a more consistent way.

Thanks for your comments. This is very much an early prototype and I appreciate your comments. I will be looking to add the ability to comment next. Regarding adding text at the bottom I thought it might be easier for people to read a glimpse of text before deciding to open the link, I’ve found this useful in my own browsing. The actual link has a 5 second redirect because I am using the free tier currently for webhose.io api (a great news scraping provider). if I get enough traction I may look at getting a paid account, but this is very much a side project

I can't middleclick links. Unusable for me.

Also, please don't attach /?sort=new (or remove it) to reddit threads. It overrides the user's sorting.

I almost liked it, but then I wanted to open 5 links in new tabs and it didn't work, also left-click also doesnt work, it just tries to load some information from external source, only reddit link worked "OK" (but really not OK, I wanted to see comments not content of a post) for me.

HN works great without js. This site I can't even click the links?

Inspired by HN without all the thinks that makes HN great.

Not just middle click, I can’t long press on mobile in order to open a new tab either. I suspect it’s just a <div> element with an onclick handler.

Use the link to the right of the title (shown as the name of the site it's on).

Clicking on titles expands/collapses.

That's pretty counter-intuitive, compared to reddit/hn/similar...

Also consider using np.reddit.com for reddit links (https://www.reddit.com/r/NoParticipation/wiki/intro).

I'd rather use ns.reddit.com (old style reddit)... but yeah I can't left click or middle click those links...

what does a middleclick do?

In most browsers, middle clicking a link will open in a new tab.

Furthermore, middle click a tab to close it. Faster and easier than hitting the small cross in the corner of the tab.

In Xorg -- the graphical environment used on a lot of Unix-like operating systems like FreeBSD and various Linux distros -- middle clicking anywhere you can type text, whether that be in a terminal, in a text editor running in a terminal, a standalone text editor, an e-mail client, an input field in a web browser, etc etc etc, will paste the text you last selected. Once you get used to that you will wonder why every OS isn't doing so by default. It's pure genius plain and simple.

'middle click a tab to close it' makes things much smoother for me!

I don't have a middle button so it's news to me. I have a modifier key for that or a right click and select "open in new tab"

Might depend on your browser but CTRL+left click will do the same as the middle mouse button--that is, open the link in a new background tab.

CTRL+Shift+Click opens the link in a new tab and switches to it.

One of the reasons I love Hacker News is because of the high quality of content and discussion. Looking like HN is nice but the fact that your top post includes a nice little "(((bankers)))" remark from /u/BigNuts881 was a rough reminder that this isn't to be a much better than just looking through the results of a broad Twitter search.

That is a good part of Hacker News, but I'd suspect if it's like any other community site maybe 10% of people bother reading comments and 1% bother to make them. There's plenty of room catering for the 90% who just want the links Drudge-style, and kicking off a community that actually has discussions is super hard work.

that's funny because I read the comments on far more posts that I actually follow the links to.

Me too. I thought most people on HN came here for the discussion -- even if that means primarily reading comments and not necessarily speaking up.

I spent about an hour yesterday learning all of these amazing insights about the solar energy industry, thanks to Hacker News comments.

This is not crypto news, but cryptocurrency news.

In a world where 'literally' can be defined as 'figuratively', I'm not sure why we bother splitting hairs on terminology... what is accepted is what is accepted.

This isn't the world. This is Hacker News, which is full of people who know what crypto is and know what cryptocurrency is and can expect each other to do better than ambiguity.

The title of this article on hacker news is "a cryptocurrency news scraper"

> what is accepted is what is accepted

Is it accepted? I would agree only when it appears in the NYT or The Economist with its 'new' meaning.

It's a subset of lazy people in tech who are abusing 'crypto' at this point in time, not Valley Girls. There's an inflection point here to interdict the abuse, before it gains mass-usage.

I see it as hopelessly yelling at kids to stop walking on your lawn

What is actually happening is that someone tells other people on the lawn that they wish kids wouldn't walk (and poop) on it, and then some of those other people play kiddo advocate and yell at that person. The kids are aren't providing any active resistance at all compared to that.

I guess I don't see the "pooping" part, I just see them shortening a long word they happen to be using.

Also, the headline clearly says cryptocurrency, it seems to be the domain name he is complaining about.

Sorry, I don't agree with adding 8 letters to a url to make a very small segment of people (who are apparently not the target audience) happy.

I don't accept crypto to mean cryptocurrency.

It's like saying butter when you mean butterscotch. Doesn't mean the same thing and shouldn't.

99% of the population would have never uttered the word "cryptography" before "cryptocurrencies" started to catch on.

In either case, crypto is an abbreviation.

This is being shown on HN, not Facebook.

@criddell: ok, carry on trying to keep the global slang out of your little utopic garden.

Unfortunately, the world could care less about what you personally accept.

And vice versa. Though actually, I can care less than I care about "what the world accepts", for example about some person speaking for the world. If you accept it, make your case; if you don't, leave "the world" out of it.

> could care less


May be referencing https://xkcd.com/1576/

Anything is "accepted" (by someone, somewhere). That gives us no Information. FWIW, what is rejected is what is rejected, and I reject this abuse of language.

But it is not accepted. You have people in every cryptocurrency thread reminding people that www.cryptoisnotcryptocurrency.com

I'd like to see other terms as well, incl. distributed ledger stuff.

Cool. Ill add that to the list. Please provide suggestions of other topics and I will add it in :D


Sorry if I confused anyone, didn’t mean to start a flame war. It was just the first domain I found that was available, I didn’t think too much about it

this is called linguistic prescriptivism and its wrong

Calling out linguistic prescriptivism is almost as bad a social faux pas as the linguistic prescriptivism itself.

IMO it's far more misguided. We can push for precise and efficient language, or we can shrug our shoulders and let the lowest common denominator strip out all nuance.

I was cured of that particular delusion when I realized that language has literally always been attacked since the day it was created.

Side note, I find the font very hard to read. The letters are too thin and grey. Removing the "font-weight: 300;" style from the body makes it infinitely more readable.

Why does a 0-points, downvoted Reddit thread with an (((echo))) have 640 points on this scraper?

I think the vote count is obviously manipulated. Stay away from this site.

I built this using the webhose api to scrape and filter for cryptocurrency related topics from reputable news sources and trending forum posts. I'm a terrible designer so I eyeballed the hacker news layout and finger painted some CSS over it. Hope people find it interesting :P

I’ve now added the ability to comment and am fine tuning replies :) thanks again for all the great feedback.

Gah. Crypto news? More like cryptocurrency. Wait. No. More like news for cryptocurrency traders.

Congrats on shipping!

What's the tech stack you used? I see from your comments buried at the bottom that you are not a designer, so will give you a pass on the usability issues that others are whining about.

Would suggest you buy a HTML template off of wrapbootstrap or themeforest and customize it. If you can't do that, hire someone from upwork to do the customization for you.

PM if you wanna chat more. Email in profile.

What is all the metamask code for?

3 suggestions.

- when I click on the link to go to the side, I'd much rather prefer it to open as a new page because it allows for quicker loading when you hit the back button, which would then close that page and bring you back to your site without needing it to reload.

- when visiting a link and then going back, I need to tap it twice. Just tapping once brings me to the redirect and then it just goes to the link again. This doesn't need time be fixed if the link is opened in a New tab though

- the tickers up top should give priority to a watchlist we could build and store in local storage so sign up isn't needed. Should be pretty trivial to implement

I would visit this daily if the content quality were better. Maybe start with scraping, then let people post themselves, and gradually move towards no scraping.

I like the ticker. Saves me from needing to check elsewhere.

Yes, the ticker is great! I would love a chrome extension that could attach a ticker like this to HN.

Thanks. I've simply used the coinmarketcap API to get the currency feeds, super straight forward. It fetches from the client side. Anyone can use it here https://coinmarketcap.com/api/

Good stuff, I highly recommend applying as a researcher to the top leading forecast reporting firm called cryptonaire.com. They have many clients that are large investment groups that will be using their research when the full app release is out. It would be nice to have more researchers that love crypto rather then investment. Give it a try, they are the first research firm in the industry.

Once again I’d appreciate any great 101 resources people have found explaining blockchain, cryptocurrencies and other related topics. I will look at adding more content to the 101 page. I know very little about the topic and just chose this as a random topic for news scraping. I am very much learning :)

Thank you everyone for the great feedback. Please feel free to share any helpful links that explain blockchain, cryptocurrencies, ICOs etc. and I will post on the 101 resource page. I am very new to this and only a curious observer of the cryptocurrency wave

Not sure how this one got in there at number 33: "Iranian woman reported missing after waving headscarf in public without wearing a hijab". Possibly a spam comment about cryptos got parsed.

I'd like to see a sentiment for each article as well. If you add that, then add a green/red indicator on the price chart as well for when the news came out.

Great idea, I actually had this before. I was feeding the articles into google's natural language processing API to get a sentiment index from -1 to 1 (-1 negative, 0 neutral +1 positive). The issue I was having that it wasn't really that accurate. Google NPL has trouble analysing contextual news articles. e.g. if a headline says "Bubble in cryptocurrency markets", it doesn't recognise that bubble is negative in news contexts. I have been looking for some different NPL libraries that might be better including AWS Rekognition. I'm not a data scientist so don't think I'll do very well building my own machine learning algorithm. Any suggestions of good libraries people have used for news sentiment would be greatly appreciated

Overall the site looks nice and functions well.

The ticker a the top is hard to use as I could not see that you could scroll.

I like the concept of the 101 section, could use some more info.

Awesome thanks for the feedback. I’ll feed some more content for the 101s :) id appreciate any interesting posts or links anyone recommends explaining crypto currency and block chain in plan English

Any plans to add a comment system? Edit: I'm excited by the idea of a crypto focused HN-ish site, I'd love if there was a comment system :)

I thought that was Reddit is for. The /r/Cryptocurrency subreddit exactly serves this purpose.

/r/Cryptocurrency is literally all price-talk and shilling -- very few posts of self-reflection or technicals.

I'm looking for comments and discussion with quality on par with HN. Reddit, generally, does not provide that.

Not everybody can post in this subreddit though. And it's always good to have different places to share about crypto :)

It would be really great to be able to login 1) save stuff and 2) discuss articles. It doesn't even need to be threaded or have voting at first.

I've just added the ability to save interesting posts. You will see a button that says "favorite" under each post. You will be prompted to sign up and then it will save it your account to access at a later stage. Next up comments and discussions :)

Thanks. I'm working on this now. I've built the authentication system and user models and will be working on comments next

I like the ticker, and I like it overall. Good job!

Unless I'm reading the cryptocurrency market table incorrectly, the market cap row appears to be a horizontal scroll bar.

Would you mind adding sentiment analysis to this?


bitcoin: negative; ethereum: neutral; bitcoin cash: very positive

I would love you forever.

Lose the tickers. Add discussions.

I can't open the links for the original posts, is this happening with you all as well?

You have to click the domain (reddit.com for example) for it to open, which is of course unintuitive

Thanks. It is indeed not very intuitive.

Can you say more about how you built this? How you did the scraping, etc. ?

For scraping I've used the webhose api (I am still on the free account hence when u click on the link it has a 5 second redirect). I've found it is an excellent resource and easy to use API, I highly recommend it http://webhose.io/

For the tech stack its just node.js and express on the backend and vanilla JS on the front end using webpack for bundling. I built this project really as a means to teach myself to write unit tests better and mock integration tests. In the first iterations I fed the news articles after they were scraped into Google's Natural Language Processing library to get the sentiment of the article. I scrapped this after it became too inaccurate.

Horizontal scroll bar on the ticker covers the Mktcap row.

Plenty of cryptocurrency chatter on HN these days anyway.

would be nice to add some kind of json feed.

maybe add some kind of account login to get the feed.

some kind of database to collect all this would be neat for some kind of price predictor

I am not able to click any story :(

Will you open the API like HackerNews ?

I may, in the meantime you can access it via my backend service. It's a mongodb store https://cryptoserver.now.sh/threads

Any plans on making the inverse?

That is not what “crypto” means.

Language isn't prescriptive, get over it.

It doesn't mean cryptography either, it just means hidden.

Hah, fair enough!

might add twitter.

Great idea, I'll put that on the list

This is A cryptocurrency news scraper inspired by Hacker News.

Reading the HN title, and looking at the site name listed on HN, I had expected there to be information relating to cryptography, which there is not.

Sorry for the confusion I've gotten used to referring to cryptocurrency as crypto as shorthand

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact