> OpenTelemetry is a project that aims to standardize all telemetry collection, but by trying to please everyone and make something idiomatic for all languages the result is a bit of a hot mess.
Huh. I’m glad it isn’t just me. I’ve fallen over OT more than a few times. Sometimes seemingly trivial things would take embarrassingly large amounts of trial and error, and I’d often be left with the sense that I’m not totally clear on what I’m doing, why, and if my solutions are correct for the logging I want to do.
I’m perfectly willing to admit that I don’t know what I’m doing, but I guess in some sense I’m glad I’m not the only one.
One time I needed to deploy a rust-based api that serviced a React-based front end, and getting telemetry to work across both on GCP was a hellish nightmare. By the end I sincerely didn’t know why it started working. Dark times.
I've been heel dragging for over a year on OpenTelemetry for this reason. The org doesn't need it either. They saw something shiny and want it for no rational reason other than they think everyone else is doing it.
Realistically they need to focus that energy on more important things like maintenance and quality rather than shovelling more crap into the inferno that has to be maintained.
Good telemetry is an absolute game changer to be fair.
The annoying thing about OpenTelemetry is just how underbaked/rudimentary it is, and how opinionated the implementations are.
For C#, I consider Microsoft's Application Insights to be the gold standard. You get per request tracing, stack traces, hot paths, dependency operations etc. with absolutely no setup on your part.
I was looking for an alternative since my current place is on AWS and OpenTelemetry seemed like a flexible, Terraform-like alternative to the platform specific SDKs.
It's not that simple though... AWS doesn't provide any direct sinks to their services, instead you must send it to a collector running in a sidecar container, or if it's a lambda: create a custom layer.
X-Ray's surfacing of the traces sucks as well. It'll only show traces in 6 hour ranges. Hopping between 5 different products trying to find the right metric is not an enjoyable experience.
Reasonable telemetry i.e. logging which can be correlated across services, is probably good enough. I think we're hitting 15,000 logs a second so even sampling OT becomes financially infeasible.
Huh. I’m glad it isn’t just me. I’ve fallen over OT more than a few times. Sometimes seemingly trivial things would take embarrassingly large amounts of trial and error, and I’d often be left with the sense that I’m not totally clear on what I’m doing, why, and if my solutions are correct for the logging I want to do.
I’m perfectly willing to admit that I don’t know what I’m doing, but I guess in some sense I’m glad I’m not the only one.
One time I needed to deploy a rust-based api that serviced a React-based front end, and getting telemetry to work across both on GCP was a hellish nightmare. By the end I sincerely didn’t know why it started working. Dark times.