I disagree. I'm an observability geek, and OTel is... fine.
It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:
1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.
2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).
I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects
Every time I share your blog (and I share it a lot) I tell people:
"This guy started a blog in 2024. Wrote three posts and all three of them would still make my top ten list of 'greatest posts on observability' today".
'A practitioner's guide to wide events' especially is still my number 1.
I'd make a wager that things would go better smoother faster if folks tried more stuff, ventures forth more on their own. It's obviously not great that there's no semantic convention that's perfect and just works for everything, and yeah it takes a while. I feel like the real data I'd want is who else, how many people show up to say they've tried something. Is that happening? Whether specs are really good enough advance or not, to me, is often whether enough people have tried it to find out.
The net of this is, otel is a very flexible system you can use and adapt in all kinds of ways and while the spec is important, using the toolkit to FAFO yourself, ahead of any beaten path, should really be encouraged. That's the message I'd want to see being radiated out about otel.
OTel has a nice property of working, which sometimes can't be said about vendor stuff.
I think the actual APIs kinda smell at the language level, and when Honeycomb decided to lean into otel and deprecate its Python libs I was super sad, cuz HC's libs were _way_ more usable IMO. Docs are also... painful. Real painful.
I wish that I could get a Python lib which is like "here this is Otel but the config phase isn't weird, and the API just looks a bit better". One of these days.
The biggest trouble I have with Otel recently is getting fixes patched upstream in contrib. Using contribs is super dangeerous, and I would basically recommend people write their own instrumentation and treat the contrib packages as just examples of how to do it
Agreed. Otel itself is fine. The documentation is bad though and full of inconsistent best practices and examples that are flat out wrong and other things.
My life of working with it got easier when I started just looking at the actual code, using network level tools like nc/tcpdump, making extensive use of the debug exporter, and almost ignoring the docs entirely except as a basic summary of what a thing does.
We did the same for 1, we have a service that picks up messages from a queue and drops 95% of them. In Python, at least, it was really really ugly. We had the alternative of using refinery, since we're on honeycomb, but didn't want to get stuck with something that only works with one vendor.
I don't understand 2, though, that should be easy? You have access to the span and you can set its status code to 0. With the Python SDK that was trivial for us to do.
The one thing I really hate about the Python SDK is that there's no simple way to say "attach this attribute to the top level span in this app". That way, when you get the user id further down in your app, you can just annotate the top level span with it. Thankfully, honeycomb has any, any2, root, parent (that allows you to filter on any span in the trace, or on the root or parent) but that's slow and again, vendor specific.
It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:
1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.
2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).