Ask HN: Python Metrics Library

[1gnition]

Coming from Java/Scala background, I am used to libraries that provide vendor agnostic APIs, such as micrometer.io, Dropwizard, etc., where you can plug-in different monitoring backends: AWS CloudWatch, Datadog, Prometheus, etc., without changing the code.

Right now I want to add metrics to my Python application but I don't find any metrics library that does that. OpenTelemetry is not an option because it's not what I'm looking for: you need to setup Agents, Gateways, Collectors, etc., and in addition, their CloudWatch exporter is only EMF which is not what I need.

I've found one library which is "pymetrics" [1], which looks promising but it's in the beginning of its way, it has not much adoption, and it has only a few monitoring backends out-of-the-box.

What's left for me is to couple my code to the monitoring system I use, which means code changes whenever we want to migrate to another monitoring system.

Am I missing something?

[1] https://pypi.org/project/pymetrics/

[gatesn]

I actually found that DataDog's ddtrace [1] and datadog [2] libraries are very good for instrumenting Python, even in the absence of any connection to DataDog (and no affiliation to them on my part).

ddtrace patches popular Python libraries to emit metrics which can be collected by any statsd-compatible endpoint. The trace spans may well be more specific to the Datadog API though.

[1] https://github.com/DataDog/dd-trace-py/

[2] https://datadogpy.readthedocs.io/en/latest/

[crabbone]

Are you collecting data from your own program or from other programs?

On a separate note: I don't think it's a good idea to "abstract" different metrics publishers. Just like it's not a good idea with "abstracting" SQL database by using a made-up SQL-like language.

In reality, publishers are very different. If you try to find an intersection of their features, you end up with a very small subset of what either one of them has to offer. On top of that, there are often subtle differences in semantics or in collection procedures that aren't immediately apparent, but may cause serious miscalculations downstream if you try to substitute one for the other.

If, in practice, if you want to bridge between different publishers, the best way to accomplish it is to have both publishers working together and have some import/export mechanism between them. Often such import/export mechanisms are developed by publishers themselves (eg. Prometheus "exporter" for collectd), so you have something that's more reliable than a third-party library trying to mediate between two different publishers (which is bound to always play catch with those publishers in terms of feature support).

[est]

I just dump everything to vector[1] using udp and mangle the output to destinations.

[1]: https://vector.dev/

[vb-8448]

Why not https://github.com/prometheus/client_python ?

[dengolius → vb-8448]

You also can try https://github.com/gistart/prometheus-push-client as well as a https://github.com/paksu/pytelegraf

[fer → vb-8448]

Because it isn't vendor agnostic, I suspect?

[jarofgreen → fer]

But then, Prometheus is open source and it's pretty easy to host your own server.

[phrotoma → fer]

The prometheus scrape format is widely supported by APM vendors.

[calmoo]

statsd is a lightweight solution that may interest you.

[josephmosby → calmoo]

+1 to statsd. We've moved from SignalFX to Chronosphere, all using statsd and without having to change any of our application metrics-gathering code.

[ddorian43]

> OpenTelemetry is not an option because it's not what I'm looking for: you need to setup Agents, Gateways, Collectors, etc., .

You don't need any of that. You can publish to an endpoint directly from your code (used like a library, no other tools needed).

[galbar]

I've had good results using https://github.com/Lightricks/pyformance

[ddorian43]

opentelemetry metrics?