
Writing a Time Series Database from Scratch (2017) - potomak
https://fabxc.org/tsdb/
======
rdtsc
After trying InfluxDB and it not working well for my use case I've written a
TSDB as well. I suspect that's the joke around these parts, everyone and their
cousin has written as TSDB.

Used time based blocks just like the article says. But they also switched if
they got larger than 2GB, because it allowed the index offset to be 4 bytes.
That allowed pretty fast searching as well as discarding old data.

The data and index and were both written to separate files. Some of the values
were not just simple integers/floats but large blobs that's why separate data
files.

Decided not to go explicitly with mmap-ing on read. Started that way but
abandoned it. A simple pread (read + seek in one syscall) and relying on page
caching worked just as well.

Should have used an inverted index for labels, that's a good idea. Maybe just
sqlite or something like that... Though my queries were not as free form and
some were more common than others. So built custom indices for the common
queries.

Index files were also partitioned by time but were synchronized to the same
partitioning schedule as the main data file. So if main data file opened a new
block file, indices did the same.

Didn't explicitly handle in memory vs not-in memory blocks when writing, but
relied on periodic fsync-ing.

Instead of a wal decided that some data that wasn't fsync-ed yet might be lost
if power was cut. To deal with corruption on start, data in main file and
extra indices were truncated to lasted sane fsync of the main time based
index.

------
ahmedalsudani
Previous discussion:
[https://news.ycombinator.com/item?id=14177411](https://news.ycombinator.com/item?id=14177411)

------
latchkey
When I read this blog post, it convinced me to use prometheus / grafana. I'm
now monitoring over 1000 Pi class boxes using a small custom golang agent
running on each one. Things have been rock solid. Couldn't be happier with
this solution. Thanks for all the hard effort.

------
marknadal
This is awesome :) having written a database myself, I very much encourage
others to try it out for fun, learning, and maybe seriously too!

