How Cloudflare analyzes 1M DNS queries per second

[r4um]

https://blog.cloudflare.com/how-cloudflare-analyzes-1m-dns-queries-per-second/

[ndr]

If you are reading this comment before the article: they use Yandex's OLAP solution, ClickHouse, and Kafka.

The whole article explains how and why they got there, it is actually very well written!

[otterley]

The XFS "lock up" issue is concerning -- it's been a very long time since I've seen any showstopping issues with that filesystem. Is anyone aware of what's going on there?

[foobarbazetc → otterley]

Yeah would be nice to get more information about that.

We run everything on xfs so that is concerning.

[lima]

Currently using Kafka + ClickHouse for something very similar.

The Grafana integration comes in handy! I've been stalking https://github.com/vavrusa on GitHub for a while ;-)

Thanks for the writeup!

[tejasmanohar]

How does ClickHouse compare to Redshift? I'm sure CloudFlare doesn't want to use AWS products, but I am curious if anyone's used both.

[sidcool]

The technical aspects are explained very well here.