
Are Prometheus and Thanos developers against fair competition? - valyala
Hi all.<p>I&#x27;m the core developer of VictoriaMetrics - open source time series database, which can be used as remote storage for Prometheus monitoring system. It implements opinionated query language, which is backwards compatible with PromQL from Prometheus. Is it called Extended PromQL [1]. It slightly derives from the original PromQL in controversial cases [2]. Additionally it implements user-friendly functionality on top of PromQL:<p>* `WITH templates` (aka `CTE expressions`) [3].<p>* User-friendly simplifications such as `rate(q)` as a shorthand to `rate(q[$__interval])`.<p>* Additional functions and operators aimed towards solving practical issues, which cannot be solved with the original PromQL.<p>VictoriaMetrics constantly evolves and sometimes implements Prometheus feature requests. For instance, recently it gained the ability to show the lower and upper bounds of the estimation returned by `histogram_quantile()` [4]. I mentioned about this in the original feature request, so users could find the solution. This led to the post from Prometheus devs [5], where they:<p>* Blame VictoriaMetrics for pointing Prometheus and Thanos users to solutions for their issues in public channels.<p>* Say that VictoriaMetrics cannot use `PromQL` word in the documentation, since it is registered trademark. Why anybody didn&#x27;t patent SQL yet?<p>To me this looks like a case for unfair competition. What do you think about this situation?<p>[1] https:&#x2F;&#x2F;github.com&#x2F;VictoriaMetrics&#x2F;VictoriaMetrics&#x2F;wiki&#x2F;ExtendedPromQL<p>[2] https:&#x2F;&#x2F;github.com&#x2F;prometheus&#x2F;prometheus&#x2F;issues&#x2F;3746<p>[3] https:&#x2F;&#x2F;victoriametrics.com&#x2F;promql&#x2F;expand-with-exprs<p>[4] https:&#x2F;&#x2F;github.com&#x2F;prometheus&#x2F;prometheus&#x2F;issues&#x2F;5706<p>[5] https:&#x2F;&#x2F;github.com&#x2F;prometheus&#x2F;prometheus&#x2F;issues&#x2F;5706#issuecomment-564532929
======
valyala
Yet another case of "fair" competition from Prometheus and Thanos devs [1]:

* Prometheus users proposed extending `/api/v1/label/.../values` API [2], which has been already implemented in VictoriaMetrics.

* I mentioned about this and described high-level architecture for this feature, so it can be implemented by Prometheus devs.

* They say "you cannot point Prometheus users to the solution in VictoriaMetrics, since this is self-promotion" [1].

The same issue already contains Thanos promotion in the first comments, but
nobody complains about this.

[1]
[https://github.com/prometheus/prometheus/issues/6178#issueco...](https://github.com/prometheus/prometheus/issues/6178#issuecomment-565694157)

[2]
[https://github.com/prometheus/prometheus/issues/6178](https://github.com/prometheus/prometheus/issues/6178)

------
shitloadofbooks
I took offence with @valyala posting about Victoria Metrics in the Cortex
Slack channel in the past, but I'm so glad he did because we had _endless_
issues with Cortex and we found Victoria Metrics to be _incredibly_ fast and
reliable after trying it when he suggested that it could do something Cortex
couldn't.

This github response wasn't cross promotion though -- this was @vayala
explaining how it could be implemented, as someone who has done it. The fact
that no one complained when Thanos was mentioned, but complained when VM was
mentioned shows how hypocritical some Prometheus contributors are.

Honestly, @valyala should just create a metrics collector and then I wouldn't
have to deal with the "we know better" attitude from some Prometheus
contributors.

The Victoria Metrics contributors are absolutely amazing. Every issue or
suggestion has been implemented within days (sometimes hours) and every
question answered in incredible detail, with warmth and friendliness.

~~~
valyala
Thanks for the warm response!

> Honestly, @valyala should just create a metrics collector and then I
> wouldn't have to deal with the "we know better" attitude from some
> Prometheus contributors

We didn't want going this route, since this could hurt Prometheus community in
the long run. But we constantly keep hearing this feature request from
VictoriaMetrics users, who deal with high ingestion rates. Probably such a
collector will be implemented in the future unless Prometheus devs address the
following issues in Prometheus:

* Increased RAM and CPU usage when enabling remote_write.

* Increased disk space usage on high ingestion rate.

* High network bandwidth usage for pushing data to remote storage.

------
FrankiFrank
Sometimes when I read your comments at Prometheus issues or at twitter, it
sounds like you want to put prometheus into bad light. It often sounds like
you're subliminally insulting prometheus.

~~~
valyala
I'm trying hard to write polite comments, which could help users finding the
solution for their issues.

My comments could be harsh only in response to misleading posts about
VictoriaMetrics [1].

[1] [https://www.robustperception.io/evaluating-performance-
and-c...](https://www.robustperception.io/evaluating-performance-and-
correctness)

------
jrv
You continue to claim that VictoriaMetrics is backwards-compatible with
Prometheus and only include a side-note that it actually is not in all cases.
Such knowingly misleading statements are dangerous for users and the
ecosystem.

As for the more general point, my comment at
[https://news.ycombinator.com/item?id=21772791](https://news.ycombinator.com/item?id=21772791)
stands. It's not about competition (we welcome and love that from all sides),
it's about overly marketing a product on Prometheus OSS community channels,
while at the same time not being clear about or ignoring compatibility issues
that will for sure cause problems for the ecosystem.

EDIT: One more point: The people seeing a problem around VM behavior here are
not just Prometheus team members. Multiple external community members / users
have pointed this out as well to us.

~~~
valyala
> You continue to claim that VictoriaMetrics is backwards-compatible with
> Prometheus and only include a side-note that it actually is not in all cases

* There are a few controversial cases where VictoriaMetrics' implementation for PromQL intentionally diverges from the original implementation in Prometheus. This discrepancy is needed because VictoriaMetrics tries hard to make PromQL more user-friendly at edge cases, which aren't solved for a long time on Prometheus side [1].

* Of course, there may be non-intentional discrepancies in PromQL implementation for VictoriaMetrics, because it is implemented from scratch without using Prometheus data model and source code. Such discrepancies are usually fixed as soon as users report about them.

> It's not about competition (we welcome and love that from all sides)

I see. Facts mentioned in this topic shows the real love :)

[1]
[https://github.com/prometheus/prometheus/issues/3746](https://github.com/prometheus/prometheus/issues/3746)

------
hagen1778
This is my response to ongoing discussion at
[https://github.com/prometheus/prometheus/issues/6178](https://github.com/prometheus/prometheus/issues/6178)
because github issues is truly the wrong place for doing this.

>> @hagen1778 My comment at #5706 (comment) goes into this a bit more. It can
indeed be helpful to cross-link to other products/projects at times, and I
don't see this individual comment as a big problem - but it fits into an
overall pattern that is less ok (see details in my linked comment).

@juliusv I am familiar with the comment you linked. I am also sharing your
opinion about being `a bit more sensitive about using Prometheus channels`.
However, I find it a bit unfair comparing to other OSS mentions in Prometheus
issues. Anyone can verify this easily by searching for `thanos`, `cortex` or
`victoriametrics` on
[https://github.com/prometheus/prometheus/issues](https://github.com/prometheus/prometheus/issues)

I disagree with second point about `own implementation of PromQL`. The reason
why it was re-implemented(AFAIK) with so much efforts instead of just reusing
the existing Prometheus code is the unwillingness to extend existing
implementation. I'm pretty sure you are familiar with that, because this
discussion appeared in your own proposal ticket from Jun 25. And I understand
this, authors should protect their projects from being overwhelmed by other
people proposals and needs to keep it solid. But I don't understand why it is
not allowed to mention the alternative way for users on public platforms like
github.

I am very grateful for Prometheus and the ecosystem around it and strongly
believe it becomes a standard for the industry, so we all should work on it.
That's why such confrontations make me sad. I hope that such situations will
never occur in future and opensource will remain the place for creating cool
software.

~~~
roidelapluie
To be fair, members of Thanos and Cortex projects do contribute code to
Prometheus, which might explain why they are named more often here.

~~~
valyala
To be fair, VictoriaMetrics also contributes to Prometheus ecosystem by
providing open source cost-effective remote storage for Prometheus, which is
easy to setup and operate.

Additional contributions to Prometheus ecosystem include:

* Filing issues and feature requests for Prometheus [1], [2], [3].

* Publishing introduction materials for Prometheus users [4], [5].

* Helping Prometheus users at prometheus-users Google group [6].

[1]
[https://github.com/prometheus/prometheus/issues/4456](https://github.com/prometheus/prometheus/issues/4456)

[2]
[https://github.com/prometheus/prometheus/issues/4769](https://github.com/prometheus/prometheus/issues/4769)

[3]
[https://github.com/prometheus/prometheus/issues/5166](https://github.com/prometheus/prometheus/issues/5166)

[4] [https://medium.com/@valyala/promql-tutorial-for-
beginners-9a...](https://medium.com/@valyala/promql-tutorial-for-
beginners-9ab455142085)

[5] [https://medium.com/@valyala/prometheus-storage-technical-
ter...](https://medium.com/@valyala/prometheus-storage-technical-terms-for-
humans-4ab4de6c3d48)

[6] [https://groups.google.com/forum/#!searchin/prometheus-
users/...](https://groups.google.com/forum/#!searchin/prometheus-
users/valyala%7Csort:date)

~~~
roidelapluie
I explicitely said "code".

Thanos and Cortex _use_ some code from prometheus. That can explain that. Try
looking timescale or opentsdb in the issues/pr.

------
valyala
An additional case of unfair competition was at PromCon 2019. There was a talk
named `Remote storage wars` from Adidas at the conference [1]. Various remote
storage solutions for Prometheus were compared in the talk for use cases
specific to Adidas - Cortex, M3DB, VictoriaMetrics and Thanos receiver.
VictoriaMetrics showed the best performance, the lowest RAM usage and the
lowest disk space usage.

The was a video recording, but unfortunately slides were broken during the
talk [2]. I was hoping that slides will be published in the next months after
the conference. Unfortunately slides for this talk weren't published yet,
while slides from other talks are already published - see `slides` link at the
bottom of every talk page from [3].

This doesn't look very fair :)

[1] [https://promcon.io/2019-munich/talks/remote-write-storage-
wa...](https://promcon.io/2019-munich/talks/remote-write-storage-wars/)

[2]
[https://youtu.be/4vZ1PqZ_Foc?t=16025](https://youtu.be/4vZ1PqZ_Foc?t=16025)

[3]
[https://promcon.io/2019-munich/schedule/](https://promcon.io/2019-munich/schedule/)

~~~
SuperQue
The videos will be uploaded soon. Don't worry, none of the slides or other
content was lost.

We're just waiting for our video company to finish the final renders with all
the correct color and lighting quality adjustments made. Each talk will be
individually published on YouTube just like all previous PromCon events.

~~~
valyala
Great news! I hope this isn't a sarcasm :)

------
valyala
Another case of unfair competition is the article [1] published by Prometheus
author, where he claims VictoriaMetrics doesn't compress random data well. I
hope Prometheus author knows information theory [2] and was aware that random
data cannot be compressed. Then it is unclear why he decided to publish the
comparison :) I published response to this article [3], but unfortunately it
isn't referenced from the original article spreading the FUD.

[1] [https://www.robustperception.io/evaluating-performance-
and-c...](https://www.robustperception.io/evaluating-performance-and-
correctness)

[2]
[https://en.wikipedia.org/wiki/Information_theory#Entropy_of_...](https://en.wikipedia.org/wiki/Information_theory#Entropy_of_an_information_source)

[3] [https://medium.com/@valyala/evaluating-performance-and-
corre...](https://medium.com/@valyala/evaluating-performance-and-correctness-
victoriametrics-response-e27315627e87)

