Hacker News new | past | comments | ask | show | jobs | submit login
We compress Pub/Sub messages and more, saving a load of money (lawrencejones.dev)
9 points by lawrjone on Dec 29, 2020 | hide | past | favorite | 4 comments

Since you’re on btrfs, a copy-on-write file system, you should also investigate disabling full page writes for Postgres WAL. I disabled it for our cluster on ZFS and got a nice reduction in write volume.

Author here, happy to answer any questions people may have!

Great blog! I saw you are now using fluentd to do gzip in real-time, any notes or feedback on that portion? (I’m the PM of the fluentd project)


I didn't implement the fluentd piece, but I can point you at the PR that did just that for our fork of the gcloud-pubsub plugin:


One comment I can make (from my own perspective, of course) is that we would normally upstream these changes, but fluentd tends to have a problem with a variety of different packages for the same thing, and no clear official choice.

That means it's difficult to know where to put your efforts, and can lead to individuals forking plugins like we have as we don't know where to focus the upstream efforts.

We didn't try too hard, truth be told, but thought it was worth mentioning.

Thanks for all your efforts on fluentd!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact