Tell HN: Hacker News BigQuery dataset is updated daily again

[wietsevenema]

As you might know, there's a Hacker News BigQuery dataset [1].

I recently noticed that the export was broken (since Nov 2022) and I worked with the team to help fix it. It'll now again run daily.

SELECT timestamp FROM `bigquery-public-data.hacker_news.full` ORDER BY id DESC LIMIT 1 # 2024-06-09 09:12:55 UTC

[1]: https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news

[zX41ZdbW]

Daily is too infrequently.

I have this dataset being updated in ClickHouse in real time: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...

I also provide a way to export it or attach it to clickhouse-local and analyze it locally: https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...

[tarasglek → zX41ZdbW]

I love that this dataset is accessible via cors from web frontends. Got me into experiments with clickhouse

Same with github, much better than than throttled apis

[dang]

Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.

[izyda → dang]

Any interest in getting this dataset on Snowflake too?

(I run a Snowflake funded data provider on Snowflake marketplace)

[dang → izyda]

The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API), so anyone is free to do this.

[wietsevenema → dang]

Yes, that’s how it works. Same data as in the API.

[Raed667 → dang]

a bit off topic: is there a way to display a HN page using the firebase API/sdk without doing N+1 requests ?

(sorry for noob question)

[dang → Raed667]

No, unfortunately. We're going to make a new API that will be much easier, but there's at least one other major project that needs to get done first.

[Raed667 → dang]

Thanks for the response!

[throwaway211]

How big is it?

[wietsevenema → throwaway211]

Closing in on 41 million items