Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: OpenLIT – Open-Source LLM Observability with OpenTelemetry (github.com/openlit)
62 points by aman_041 7 months ago | hide | past | favorite | 22 comments
Hey HN, we're super excited to share something we've been working on: OpenLIT. After an engaging preview that some of you might recall, we are now proudly announcing our first stable release!

*What's OpenLIT?* Simply put, OpenLIT is an open-source tool designed to make monitoring your Large Language Model (LLM) applications straightforward. It’s built on OpenTelemetry, aiming to reduce the complexities that come with observing the behavior and usage of your LLM stack.

*Beyond Basic Text Generation:* OpenLIT isn’t restricted to just text and chatbot outputs. It now includes automatic monitoring capabilities for GPT-4 Vision, DALL·E, and OpenAI Audio. Essentially, we're prepared to assist you with your multi-modal LLM projects all through a single platform and we're not stopping here; more updates and model support are on their way!

*Key Features:*

- *Instant Alerts:* Offers immediate insights on cost & token usage, in-depth usage analysis, and latency metrics. - *Comprehensive Coverage:* Supports a range of LLM Providers, Vector DBs, and Frameworks - everything from OpenAI and AnthropicAI to ChromaDB, Pinecone, and LangChain. - *Aligned with Standards:* OpenLIT follows the OpenTelemetry Semantic Conventions for GenAI, ensuring your monitoring efforts meet the community's best practices.

*Wide Integration Compatibility:* For those already utilizing observability tools, OpenLIT integrates with various telemetry destinations, including OpenTelemetry Collector, Jaeger, Grafana Cloud, and more, expanding your data’s reach and utility.

*Getting Started:* Check our quickstart guide and explore how OpenLIT can enhance your LLM project monitoring: https://docs.openlit.io/latest/quickstart

We genuinely believe OpenLIT can change the game in how LLM projects are monitored and managed. Feedback from this community could be invaluable as we continue to improve and expand. So, if you have thoughts, suggestions, or questions, we’re all ears.

Let’s push the boundaries of LLM observability together.

Check out OpenLIT here: https://github.com/openlit/openlit

Thanks for checking it out!




It would be great for this to actually explain what sorts of metrics are being computed here beyond what you get for free by instrumenting the requests library.

From looking at the screenshots, it looks like it can monitor number of tokens, which seems useful, but I'm not clear why that needed a whole big project.

I feel like the stuff you actually want to monitor in prod for ML that you don't get from infra monitoring are things that are not trivial to drop in because you want a sense for how well the ML components are working, which is generally pretty application specific. Having a general framework for that seems useful, but not really what we have here, at least for the moment.

Also, it just seems a bit weird for this to have it's own UI. Part of the point of OTEL is so that you can send all your metrics to one place. Not totally possible all the time and turning metrics into dashboards takes time, but the point of OTEL seems to be to separate these concerns.


Hey! On the metrics front, we do basic metrics like requests, tokens and cost for now as all of the usefull information about llm is included directly in the Span Attributes of the Trace.

So what a lot of users have told me is the want to track their usage(cost/tokens) and want to keep an eye on the user interactions like prompts and responses, We are looking to add more accuracy based metrics too which is very highly asked to us.

Also re on UI: UI is optional, You can use the sdk to directly send otel traces and meyrics to your preferred destination. https://docs.openlit.io/latest/connections/intro


On the span is an interesting attribute to view, but had you considered using otel meters/metrics for that instead?

I think cardinality in span attributes can be a problem, and meters are better for aggregating and graphing


We do Otel metrics for the main things needed in dashboarding 1. Requests (Counter) 2. Tokens (Counter) (Seperate metric for Prompt Tokens, Completion and Total Tokens) 3. Cost (Histogram)

I did attach a Grafana dashboard too (Works for Grafana Cloud, Ill get something for OSS this week) https://docs.openlit.io/latest/connections/grafanacloud

Since Otel doesn't yet support Synchronous gauge + Users wanted a trace of the LLM RAG application, We opted to Traces which is quite standard now in LLM Observability tool.

Lemme know if you had something more in mind, Love getting feedback from amazing folks like you!


Very cool. QQ: How does this differ from Langtrace (https://www.langtrace.ai) / (https://github.com/Scale3-Labs/langtrace)?


Some others have mentioned before, but there are actually several projects working on exactly this:

https://github.com/Arize-ai/openinference

https://github.com/traceloop/openllmetry

https://github.com/Scale3-Labs/langtrace


Yup thats correct, Love as more tools come up! We though tried to follow the actual OpenTelemetry's Semantic Convention for GenAI, the others dont.

Additionally none of these libraries "then"(Can check again) seemed to send the cost attribute which was what most of our user's mentioned, So we added that.


Langtrace core maintainer here. We capture input/output/total tokens for both streaming and non streaming across various LLM providers. For streaming we calculate using tiktoken library as OpenAI does not show that with their outputs. Cost is calculated on the client side as it may keep changing. But all you need is the cost table for the llm vendor and you can get the cost for token usage


Does the dashboard/UI support traces? I would love a tool in which to view opentelemetry traces, that can neatly display full prompt and response for the spans that represent LLM queries. I'm planning to add opentelemetry instrumentation to magentic [1] and looking for a UI that is easy to run locally that makes it easy to see what an agent is doing (via OTEL traces). I have more of my thoughts on the github issue: https://github.com/jackmpcollins/magentic/issues/136

[1] https://github.com/jackmpcollins/magentic


Shameless plug. We are building exactly this. Fully open source and open telemetry standard tracing with a visualization client that’s optimized for LLM application observability. Check it out here

- https://github.com/Scale3-Labs/langtrace - https://langtrace.ai/


I appreciate you sharing your project with me. It's great to see others working on solutions in this space. While our offerings may have some similarities, I'm sure there are unique aspects to what you've built that could be valuable to users. I encourage you to continue innovating and pushing the boundaries of what's possible. Healthy competition ultimately benefits the entire community, as it drives us all to create better products. I wish you the best of luck with your project.


Yup, The Dashboard is entirely powered by OTEL traces and yeah, you can see full prompts and responses including cost and request metadata.

The UI is pretty easy to run, Its one single image.

Lemme know if you need any help on instrumentation, If you have a PR, Lemme know, Ill try to assist on this


You can add whatever span attributes you like to otel traces, and then show those attributes in whatever UI you have (I use Grafana).


How does this differ from Traceloop’s Openllmetry? https://github.com/traceloop/openllmetry


From what I can see, Openllmetry asks you to manually call tracer for non-default OpenAI libraries (i.e. not Python/Node.js) [1]

OpenLIT might be easier to integrate with any language that supports OTEL for HTTP clients — you just trace your HTTP calls to OpenAI.

[1] https://www.traceloop.com/docs/openllmetry/getting-started-r...


How does it do that for ruby for example? (which is in the link you provided). OTEL instrumentation for HTTP doesn’t instrument the body so you won’t be able to see token usage, prompts and completions. Or am I missing something?


So, do I understand this right. But is this supposed to be a centralized (in a way) application for collecting all OpenTelemtry data related to LLM's? And it's supposed to support various of related services and/or services today?

I see how this can be really useful. But at the same time, how do you guys see your self differentiate your self from cloud hosted equivalents? (e.g dashboard-like services in Azure and similar).

Anyhow, interesting project. I'll keep an eye on it for future use


Yup, You are on point! We do see the UI/dashnoarding layer to find other usage a bit more as we add more tools(Compare/Evaluate) to help the overall LLM Developer Experience.

Also, I totally get the existing tools like Grafana are great, which is why we built our SDK in a very vendor neutral way using OTEL, So that yiu can send these traces and meyrics to our choice of OTLP backend.

Thanks!


I browsed through the readme, and don’t quite understand how LLM or “GenAI” is supposed to help me with observability.

Pro tip: if you truly want to build an open source community around this. Don’t build your community using proprietary chat platforms (ie, slack). Slack in particular only keeps a small amount of history.


I don’t think it was ambiguous at all, perhaps you just don’t have the problem they describe?

They have an observability platform for LLMs and other generative AI services. Just like you can use observability tools on services like databases or APIs, they’ve made an observability tool for the aforementioned AI services and the applications built with them. They use OpenTelemetry, a pretty standard observability protocol, to interoperate with other tools that ingest observability-oriented data.


Yeah, 100%, I couldn't agree more on this!

Though if you'd like to see something more, Lemme know!

Thanks!


@xyst

https://discord.gg/RbNPvG54

Thanks for the suggestion on community (Not Slack). I Created a Discord server now, Feel free to spam questions, suggestions! Thanks for the feedback!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: