
Prometheus 2.0 - bbrazil
https://prometheus.io/blog/2017/11/08/announcing-prometheus-2-0/
======
Dowwie
The TimescaleDB crew built a Prometheus-PG adapter [1]. TimescaleDB, and
consequently Postgres, is an option I am exploring. Does a developer realize
any benefits of 2.0 if the Prometheus database isn't used as a backend?

[1] [https://github.com/timescale/prometheus-postgresql-
adapter](https://github.com/timescale/prometheus-postgresql-adapter)

~~~
jrv
I am not too familiar with the PG-TimescaleDB adapter, but the storage engine
rewrite in Prometheus was definitely the most important improvement in
Prometheus 2.0.

There are other changes than the storage though that are outlined in the
migration guide:
[https://prometheus.io/docs/prometheus/latest/migration/](https://prometheus.io/docs/prometheus/latest/migration/)

* Minor PromQL changes (mainly removals of deprecated features)

* Staleness handling (not sure if the TimescaleDB storage handles this correctly, as it relies on storing a special NaN value bit-wise correctly for marking series stale)

* New rule file format is now YAML and supports per-group evaluation intervals and serialization of rule executions within one group (in case of data dependencies)

* Other minor bits

------
nopzor
Really psyched to see this drop, the new engines performance looks amazing.
And performance was already pretty stellar to start with!

Looks like Prometheus is also now the most popular tsdb on github!

The grafana team is working quite heavily on our Prometheus integrations, with
some major recent improvements to the query editor in 4.6. We have many more
plans to work even more tightly with the project.

Kudos to the Prometheus team!

------
jordan801
I was curious what this is, so I clicked on it. I spent around 30 seconds
scanning the homepage and this is my observation: Uses data to make insights.

What kind of data? I have no clue. How? Not the faintest.

Anyway, you might want to add a portion to describe what it is for us scrubs
that haven't a clue. Possibly a demo page?

~~~
kayoone
Prometheus is pretty popular in this community as a monitoring and metrics
server, which is why this announcement got upvoted so much even if it lacks a
general introduction.

------
sz4kerto
No good migration story for existing data, that'll hurt us quite a lot. :(

~~~
jrv
Yeah, as Prometheus's local storage is meant more as a transient / non-durable
metrics store, the only current way to migrate while simultaneously accessing
old and new data is to run both the old and new servers and have the new one
read old data from the other one via the remote-read integration.

Someone could write a tool to do a full migration of the old storage format to
the new one, but the formats are completely different and at least in the
naive version of such tooling, that would have to happen offline and take a
very long time to run for large storages.

EDIT: If you would like to fund development of such a tool, let us know :)

~~~
sz4kerto
> EDIT: If you would like to fund development of such a tool, let us know :)

We're too small for that, unfortunately. Someone said at DockerCon that
migrating large (multi-TB) stores would take a long time; this doesn't apply
to us, we have only ~0.1 TB perf data as of now.

~~~
jrv
Out of curiosity, do you care about migrating the data online, or would a
brief Prometheus downtime (and thus gap in data) be ok?

~~~
sz4kerto
Downtime would be acceptable I think, especially because we could just launch
a separate instance of Prom while the main one is being migrated.

------
SkyRocknRoll
Does this storage engine onpar with influxdb ?

~~~
will-not-smith
Nope, not on par.

While there is no direct comparison to InfluxDB in the article, look at the
disk usage and (more importantly) disk I/O utilization, which is
indistinguishable from zero on the graph, whereas before it was at 20-30% with
the same load.

So not on par. Much, much better.

(Of course, Prometheus and InfluxDB are not perfect substitutes for one
another, so there's much more to look at than just storage engine
performance.)

~~~
bbrazil
The third version of the InfluxDB database is basically the same core design
as Prometheus's third database version which 2.0 uses. I'd expect performance
to be broadly similar, maybe a little better with Prometheus as we can take
advantage of the characteristics of our domain.

We do have a comparison to Influx on our website:
[https://prometheus.io/docs/introduction/comparison/#promethe...](https://prometheus.io/docs/introduction/comparison/#prometheus-
vs.-influxdb) Which is the right choice really depends on the use case. If
you're doing metrics-based operational monitoring, Prometheus is generally
best.

Disclaimer: Prometheus developer.

------
pfranz
Literally yesterday I did an install of 1.8.2 after seeing 2.x still in beta.
Time to upgrade.

Congrats to the Prometheus team.

------
wanghq
Does anyone need to view metrics older than 2 weeks? What's your solution? I
feel it's odd if you rely on a different software, e.g. for the most recent
data, use Prometheus, and for older data, use something else. What if I want
to compare the same metric in the last 4 weeks?

~~~
koffiezet
Multi-year projections? Client of ours has busy periods every year, and they
want year-by-year comparisons. They expect/pay us to keep metrics (although
heavily compressed/condensed) of the last 5 years.

------
the_arun
How Prometheus compares to Splunk?

~~~
bbrazil
Splunk is a event logging system, compared to Prometheus which is metrics
based. You need both types of systems to be able to properly observe your
systems, they're complementary.

~~~
chimeracoder
> Splunk is a event logging system, compared to Prometheus which is metrics
> based. You need both types of systems to be able to properly observe your
> systems, they're complementary.

While this is the traditional way of looking at them, I strongly disagree that
metrics and logs are different toolsets, or that you would need both of them
in order to properly observe your systems.

I've written and spoken about this approach before:
[https://medium.com/@chimeracoder/dont-read-your-
logs-13586c7...](https://medium.com/@chimeracoder/dont-read-your-
logs-13586c790202) and
[https://vimeo.com/221049715](https://vimeo.com/221049715)

~~~
bbrazil
> I strongly disagree that metrics and logs are different toolsets, or that
> you would need both of them in order to properly observe your systems.

And from the link:

> Logging can be useful for some purposes. However, it’s rare that they’re the
> only tool for monitoring your code. And it’s even rarer that they’re the
> best tool.

Metrics are a tool that take a different approach to logs, once you get beyond
small systems you need both. I talked about this earlier in the year:
[https://www.youtube.com/watch?v=hCBGyLRJ1qo](https://www.youtube.com/watch?v=hCBGyLRJ1qo)

~~~
chimeracoder
> Metrics are a tool that take a different approach to logs, once you get
> beyond small systems you need both. I talked about this earlier in the year:

Quite the opposite - the "some purposes" I'm talking about are precisely the
small scale. As scale grows, the use case of logs and metrics converges, and
metrics become a strictly better tool.

~~~
scotch_drinker
Not for high cardinality events like what happened to a particular user during
a single session. Metrics will never help with that type of problem.

~~~
chimeracoder
> Not for high cardinality events like what happened to a particular user
> during a single session. Metrics will never help with that type of problem.

No, and as I explain in both that article and the video, logs aren't the best
solution for that use case either.

------
frik
Does Prometheus 2 still use the unmaintained charting library "flot charts" ?

[http://www.flotcharts.org/](http://www.flotcharts.org/)

No commits since 2014. Unfortunately, because it's a great chart library
(actually probably the best free one), only the reliance on JQuery isn't that
fashionable.

Edit: oh, it was Grafana and Kibana that used Flot charts (at least when I
tried it two years ago, maybe things have changed) quick Google (2014):
[https://github.com/grafana/grafana/issues/222](https://github.com/grafana/grafana/issues/222)

~~~
jrv
No, we never used flot charts. We're using Rickshaw since the beginning:
[http://code.shutterstock.com/rickshaw/](http://code.shutterstock.com/rickshaw/)

It was unmaintained for a while, but now it seems there've been commits in the
recent past again. However, for real dashboarding (not just simple ad-hoc
queries), we recommend using Grafana anyway.

