Hey OP (assuming you're the author), you might be interested in this similar experiment I did about four years ago, same dataset, same target, similar goal!
Similar sequence of investigations, but using regular Postgres rather than Timescale. With my setup I got another ~3x speedup over COPY by copying binary data directly (assuming your data is already in memory).
Wish I saw this before I started haha! I left a footnote about why I didn't try binary copy (basically someone else found its performance disappointing) but it sounds like I should give it a try.
Yeah I imagine it depends where the data is coming from and what exactly it looks like (num fields, dtypes...?). What I did was source data -> Numpy Structured Array [0] -> Postgres binary [1]. Bit of a pain getting it into the required shape, but if you follow the links the code should get you going (sorry no type hints!).
Had a read through parts 1 and 2, thank you for the engaging reads! Love how you've formatted your posts with the margin notes too. Thank you for providing the function to write numpy structured arrays to Postgres binary, I couldn't figure this out before.
Same exact experience, a year on Fly but moved to GCP (GKE in our case for reasons) a month or two ago. Super slick when it worked, but that wasn’t often enough…
The backend is straight Rust. Bun is used as a package management tool & as a script runner, to invoke Tauri commands that start up a Vite webserver to provide the UI, and start something that rebuilds the binary when Rust files change.
Typescript is used for the Svelte UI. Svelte talks to the Rust backend through Tauri (by default). Both support a headless/web mode where the Svelte frontend connects to the backend over HTTP.
Awesome, thanks for responding. Very keen to give Tauri a spin, seems like a happy middle way between native toolkits (which I'm just never going to spend the time to learn properly) and Electron (which I'm just never going to love).
I make sure to go on airplane mode and hard-reload the app, and it generally works great. If I have week signal, or don’t reload it, or anything like that, it gets confused and constantly freezes.
Surprised that no one has mentioned Placemark [0] yet. It's the other brand new collaborative online mapping thingy, but made by a single dev and more focused on data than cartography/print.
https://rdrn.me/optimising-sql/
Similar sequence of investigations, but using regular Postgres rather than Timescale. With my setup I got another ~3x speedup over COPY by copying binary data directly (assuming your data is already in memory).