
Time Structured Merge Tree: From LSM Tree to B+Tree and Back Again - pauldix
https://influxdb.com/docs/v0.9/concepts/storage_engine.html
======
gtrubetskoy
IMHO if your volume is not in 300K data points per second, there is still
plain old PostgreSQL, which is perfectly suitable for storing time-series and
is time-tested and very reliable, allowing you to keep your TS data alongside
your other (business) data so that you can analyze it all together without
having to extract out of unfamiliar storage. I've hacked a project which
demonstrates how it can be done, see
[https://github.com/grisha/timeriver](https://github.com/grisha/timeriver) and
my blog explaining the technique [http://grisha.org/blog/2015/09/23/storing-
time-series-in-pos...](http://grisha.org/blog/2015/09/23/storing-time-series-
in-postgresql-efficiently/)

~~~
acconsta
Yeah, but...

Need replication? Gotta write your own sharding logic or set up pg_shard.

Need aggregations? Gotta write your own logic. Will you do them on the fly?
Use triggers? On demand?

Need to remove old data? Gotta set up a cron job. But wait, what if I want to
age different series at different rates? Now you need a policy system. Sigh.

Not saying Postgres can't be the storage engine, but there's a lot of work to
do on top of that.

~~~
gtrubetskoy
I believe that what you're referring to as "a lot work to do on top of that"
_is_ the correct solution, the ultimate OSS project.

The fallacy of the many time series db's out there is that they discarded the
relational database as a viable storage option prematurely and are now trapped
solving the very hard problem of horizontally scalable distibuted storage that
takes many years to solve instead of focusing on the time-series aspect of it.

Sooner or later something along the lines of pg_shard will become standard in
PostgreSQL and other databases, thus you don't really need to write your own
sharding logic, you just have to wait. OR you can write it if you want. You
have options.

Aggregations is what GROUP BY is for. Removing old data is a non-issue if
you're using a round-robin approach (see my blog link), it also makes aging
different series at different rates easy.

~~~
acconsta
It's good to have competition in this area. Influx is giving me whiplash (it's
on its third database engine in six months!)

The circular buffer approach is fine, but it does have the drawback of being
unable to represent variable-length data (like key-value pairs). It's also
harder to compress the data.

Can a GROUP-BY do windowed aggregations? Like, take an average over ten-minute
windows? My SQL knowledge is not great.

------
kbody
So, we were using InfluxDB on v0.8.x but had two data corruption incidents.
Both times I tried the recovery utilities with no luck. Even with some data
loss, it was time to move to another db.

We were only getting ~1k tps with some spikes of 10k tps. I hope for the rest
of the users, there was some work done of that front. Maybe in the future we
will revisit out of curiosity, since we have no issues at the moment.

~~~
gtrubetskoy
Curious - what db did you move to?

~~~
kbody
kdb+ pricey for sure, so it's not a fair fight but you gain the power of q and
amazing performance.

------
skybrian
I'm wondering if a new low-level db will come out of this as an alternative to
LevelDB, BoltDB, and so on? Or do you have to use InfluxDB?

~~~
btrask
We definitely need more and better low level key-value stores, especially
write-optimized ones. It sounds like InfluxDB has correctly identified the
pain points with LevelDB and LMDB, but it's yet to be seen whether they've
succeeded in addressing them. The article kind of petered out without a proper
conclusion.

The people who are talking about Postgres in this thread are thoroughly
confused. This isn't a competitor to Postgres, this is a platform for building
a competitor to Postgres (or, hypothetically, something Postgres itself could
use). I guess maybe they mean InfluxDB rather than the subject of the article.

~~~
robochat42
seriously? I'm always confused by the large number of different low level key-
value stores on offer. All claiming to be the fastest, with benchmarks to back
it up.

Reading the article, it seems that they could have contributed some
development to Rocksdb and saved themselves a lot of time rather than
implementing their own system.

~~~
btrask
RocksDB has the same problems as LevelDB (uses large numbers of file
descriptors, considerable read overhead compared to LMDB), plus it's even more
complex and harder to configure. I had high hopes for the LSM tree from
SQLite4[1], but last I checked development was stalled.

[1]
[https://www.sqlite.org/src4/doc/trunk/www/lsmusr.wiki](https://www.sqlite.org/src4/doc/trunk/www/lsmusr.wiki)

------
shubhra51
Woot! Nice throughput improvement over traditional LSM types with both read
and write accelerated.

------
philsnow
typo in the article's headline: Strucutred -> Structured

