It's been a breath of fresh air this summer to pull out New Relics agent libraries from several of our apps and have them replaced with OpenTelemetry dittos with auto-instrumentation turned off.
No more outages because NR's .NET agent generates code at runtime that causes segfaults. No more outages from the JS agent leaking memory if you instrument long-running functions. No more junior python devs spending days chasing the wrong call path because the Python agent produces bogus timing data on async python..
Just a dumb thing where we can ask for important flows to be traced and set up simple metrics for the things we want to keep track of.
Now we can keep using NRs dashboards, NRQL etc backend systems, which IMO are really good, without the downside of monthly outages each time we update an NR agent library.
That is super interesting. I'm curious if you'd be interested in sharing more about your .NET experience with OpenTelemetry? There's really not a lot of writing out there, good or bad, about experiences with OTel on .NET
OTEL on .NET has been "meh". I'm considering ripping out all of the metrics components in favor of Prometheus-net.
It's missing basic stuff it should have. For example, .NET auto uses a trace id when returning `Problem` or similar using Activity.Id. The trace id in OTEL is different for whatever reason, which means you have to add a customization in order to match the trace ids.
It doesn't keep track of http codes, routes, etc.
It's lacking basic functionality and outside of tracing is practically useless.
This is not surprising behaviour for datadog. They've used a lot of goodwill and pr, over the last few years, to attract tech teams to embed their APM into codebases. They are now applying the squeeze on everyone with outrageous renewal rates. It's pretty obvious that their aim now is purely profit, and their original love for logs, telemetry and monitoring are second class citizens or a distant memory.
What's surprising me here though is that New Relic are a bit better, when they too employed similar practices once. Has there been an attitude change of some kind?
I think Datadog have started doing too many things in one product, to the extent that it makes difficult to use and I imagine too difficult to maintain.
I found it a lot easier to work with distributed tracing in New Relic than I did with DD. I was surprised when I came to that conclusion because this time, about a decade ago, I was not in favour of using New Relic at all.
If I found myself in a position to choose the tech I use, I'd probably think twice before accepting Datadog's introduction deal, or even using New Relic for that matter. You could get the job done with Kibana and Grafana for some time.
cool! but does Datadog even allow anyone to do what Signoz just did?
8.2. Customer Restrictions
(g) access or use the Services or Documentation for the purpose of competing (or enabling others to compete) with Datadog, including: copying ideas, features, functions or graphics, developing competing products or services, or performing competitive analyses;
My company has a medium 5 figure monthly spend on DD, and it seems massively overpriced. I was discussing New Relic with someone the other day (just what I'm familiar with). But very curious what other services are solid that we should look into? We use APM, logs, infrastructure monitoring, distributed tracing, pretty much everything.
If you're considering switching, you can definitely spend less by embracing OpenTelemetry. The right solution will depend on how much technical lifting you want to do/how much control you need. SigNoz does metrics, logs, and distributed traces; and has both a self-hosted open source OpenTelemetry backend and dashboard, and a new cloud offering.
There's also Honeycomb if you're more focused on tracing, and Prometheus/Grafana if metrics are what you care about most.
Disclaimer: I'm one of the founders of HyperDX (we're on HN today too!)
I think it depends if you're just looking at price or other factors as well, I'd say the incumbents would be Grafana or Elastic. Though they both have a bit of their pitfalls - Grafana Loki doesn't handle high-cardinality search well (ex. search by ID) versus Elastic is super heavy to have self-hosted and involve some manual schema management. Though they're both very mature offerings and can likely flex to arbitrary workloads.
On the other side of things I'd say Signoz or HyperDX (what we're building) can work as well, though again each tool has its set of UX tradeoffs it's made so far and it really depends on what workflows you care for the most.
No more outages because NR's .NET agent generates code at runtime that causes segfaults. No more outages from the JS agent leaking memory if you instrument long-running functions. No more junior python devs spending days chasing the wrong call path because the Python agent produces bogus timing data on async python..
Just a dumb thing where we can ask for important flows to be traced and set up simple metrics for the things we want to keep track of.
Now we can keep using NRs dashboards, NRQL etc backend systems, which IMO are really good, without the downside of monthly outages each time we update an NR agent library.