I agree. I tried to get it to work recently with datadog, but there was so many hiccups. I ended up having to use datadogs solution mostly. The documentation across everything is also kind of confusing
It's a nice dream. At Google Cloud Next last year, the vendors kinda of came in two buckets. Datadog, and everyone trying to replace Datadog's outrageous bills.
FWIW the "datadog doesn't like otel" thing is kind of old hat, and the story was a little more complicated at the time too.
Nowadays they're contributing more to the project directly and have built some support to embed the collector into their DD agent. Other vendors (splunk, dynatrace, new relic, grafana, honeycomb, sumo logic, etc.) contribute to the project a bunch and typically recommend using OTel to start instead of some custom stuff from before.
They support ingesting via otel (ie competing with other vendors for their customers) but won't support ingesting via their SDKs (they still try very hard to lock you in to their tooling).
Yeah their agent will accept traces from the standard Otel SDK but there is no way to change their SDK to send the traces to anyone other than Datadog when I last checked a couple(?) of years ago.
I mean I understand why they did that but it really removes one of the most compelling parts about Otel. We ended doing the hard work of using the standard Otel libraries. I had to contribute a PR or two to get it all to work with our services but am glad that's the route we went because now we can switch vendors if needed (which is likely in the not too distant future in our case.
part of the reason for that experience is also because DataDog is not open telemetry native and all their docs and instructions encourage use of their own agents. Using DataDog with Otel is like trying to hold your nose round over your head
You should try Otel native observability platforms like SigNoz, Honeycomb, etc. your life will be much simpler
Disclaimer : i am one of the maintainers at SigNoz
The biggest barrier to setting up oTel for me is the development experience. Having a single open specification is fantastic, especially for portability, but the SDKs are almost overwhelmingly abstract and therefore difficult to intuit.
I used to really like Datadog for being a one-stop observability shop and even though the experience of integrating with it is still quite simple, I think product and pricing wise they've jumped the shark.
I'm much happier these days using a collection of small time services and self-hosting other things, and the only part of that which isn't joyful is the boilerplate and not really understanding when and why you should, say, use gRPC over HTTP, and stuff like that.
You are generally correct but I've used https://github.com/openobserve/openobserve for several projects for dev-only complete OTel stack (dashboards included) and I liked it. There are better dashboards out there for sure, but for what I needed locally it did the job fantastically well. Zero complaints.
It's extremely easy to self-host, either on a dev machine, a VPS, or in any Docker-based PaaS.
