Introduction

Debugging generative AI applications is often a black box experience, where developers struggle to understand why a specific prompt led to a hallucination or where latency spikes occur in a complex RAG pipeline. OpenLLMetry, an open-source observability framework built on OpenTelemetry, solves this by providing deep, non-intrusive visibility into the execution of LLM-powered applications. With its wide support for providers like AWS Bedrock, OpenAI, and LangChain, OpenLLMetry allows teams to trace every interaction, monitor token usage, and integrate with existing APM tools like Datadog or Honeycomb without vendor lock-in.

What Is OpenLLMetry?

OpenLLMetry is an open-source observability project that provides a set of extensions and instrumentations built on top of OpenTelemetry to give developers complete visibility over their GenAI applications. It is maintained by Traceloop and licensed under the Apache 2.0 license, ensuring that the telemetry data generated is vendor-neutral and portable.

Unlike traditional monitoring tools, OpenLLMetry is specifically designed for the probabilistic nature of LLMs. It captures AI-aware semantic conventions, such as prompt details, completion text, and token counts, which are then exported via the OpenTelemetry Protocol (OTLP) to any compatible backend. This allows developers to move their observability stack from one provider to another without changing a single line of instrumentation code.

Why OpenLLMetry Matters

The shift from deterministic software to probabilistic AI introduces a new class of failure modes. A subtle change in a model version or a poorly constructed prompt can lead to catastrophic failures in production that are nearly impossible to debug using standard logs. OpenLLMetry fills this gap by transforming the “black box” of LLM calls into a structured trace tree, allowing developers to pinpoint exactly where a chain of thought failed.

Furthermore, the industry is moving away from proprietary, vendor-locked observability platforms. By leveraging the OpenTelemetry standard, OpenLLMetry ensures that organizations can maintain full control over their data and avoid the “observability tax” associated with closed-source AI monitoring tools. This makes it a critical choice for enterprise teams requiring strict data governance and compliance with regulations like the EU AI Act.

Key Features

• Automatic Instrumentation: OpenLLMetry provides out-of-the-box support for 20+ LLM providers and frameworks, including OpenAI, AWS Bedrock, LangChain, and LlamaIndex, eliminating the need for manual logging.
• Vendor-Neutral Telemetry: Because it is based on OpenTelemetry, traces can be exported to any OTLP-compatible backend such as Datadog, New Relic, Honeycomb, Dynatrace, or Grafana Tempo.
• AI-Aware Semantic Conventions: It captures specific GenAI metrics including input/output tokens, model IDs, temperature, and finish reasons, providing a standardized way to analyze AI performance.
• Non-Intrusive Integration: The library can be integrated into existing Python or TypeScript applications with minimal code changes, often requiring just a few lines of initialization.
• Privacy and Data Masking: It includes controls to disable logging of sensitive data or mask prompts and completions to ensure compliance with privacy laws.
• Multi-Language Support: While primarily focused on Python, OpenLLMetry has sister projects for TypeScript/JavaScript, Ruby, and Go, expanding its reach across different tech stacks.

How OpenLLMetry Compares

When choosing an LLM observability tool, the primary tradeoff is usually between “all-in-one” proprietary platforms and vendor-neutral instrumentation. OpenLLMetry sits in the middle, providing the instrumentation layer that can plug into any backend.

OpenLLMetry is the best choice for teams that already have an established observability stack (e.g., Datadog or New Relic) and want to add LLM tracing without introducing a new, separate tool. In contrast, LangSmith is highly optimized for the LangChain ecosystem but creates significant vendor lock-in. Arize Phoenix is excellent for evaluation and notebook-based experimentation, but OpenLLMetry’s focus on portability and standard OTLP makes it more suitable for production-grade infrastructure.

Getting Started: Installation

OpenLLMetry can be installed as a comprehensive SDK or as individual instrumentation libraries for specific providers. For most users, the Traceloop SDK is the recommended starting point.

Using the Traceloop SDK

The SDK simplifies the setup process by bundling the necessary OpenTelemetry configurations.

pip install traceloop-sdk

Using Individual Instrumentations

If you prefer to manage your own OpenTelemetry pipeline, you can install specific instrumentations. For example, to instrument AWS Bedrock:

pip install opentelemetry-instrumentation-bedrock

Prerequisites

Ensure you have Python 3.9+ installed. If you are using the SDK, you will need an API key from Traceloop or a configured OTLP endpoint for your chosen observability backend.

How to Use OpenLLMetry

The core workflow of OpenLLMetry involves initializing the SDK at the entry point of your application and then running your LLM code as usual. The library intercepts the calls to the LLM provider via monkey-patching, meaning you don’t have to rewrite your existing API calls.

Once initialized, every call to a supported provider (like OpenAI or Bedrock) is automatically wrapped in a span. This span captures the prompt, the response, and the metadata (like token usage). These spans are then batched and sent to your configured OTLP exporter.

If you are using the standalone instrumentation for AWS Bedrock, you simply call the .instrument() method on the instrumentor class to enable tracing for all Boto3 calls to Bedrock.

Code Examples

Basic SDK Implementation

This is the simplest way to get started. The SDK handles the tracer provider and exporter configuration automatically.

from traceloop.sdk import Traceloop

# Initialize Traceloop with your app name
Traceloop.init(app_name="my-llm-app", disable_batch=True)

# Your existing LLM code (e.g., using OpenAI)
import openai
client = openai.OpenAI()
client.chat.completions.create(
  model="gpt-4",
  messages=[{"role": "user", "content": "Explain OpenTelemetry in one sentence."}]
)

AWS Bedrock Instrumentation

For those not using the full SDK, the Bedrock instrumentor allows you to add observability to Boto3 calls without changing your business logic.

from opentelemetry.instrumentation.bedrock import BedrockInstrumentor

# Enable tracing for all AWS Bedrock calls
BedrockInstrumentor().instrument()

# Your Boto3 Bedrock client code
import boto3
bedrock = boto3.client("bedrock-runtime")
response = bedrock.invoke_model(
  modelId="anthropic.claude-v2",
  body={"prompt": "Hello, Claude!", "max_tokens": 200}
)

Real-World Use Cases

OpenLLMetry shines in complex AI workflows where a single user request triggers a sequence of multiple LLM calls and tool executions.

• Debugging RAG Pipelines: A developer can use OpenLLMetry to see if a failure in the final answer was caused by a poor retrieval step (the context provided to the LLM was irrelevant) or a failure in the LLM’s reasoning process.
• Cost and Token Monitoring: An operations engineer can track token usage across different models (e.g., comparing GPT-4o vs Claude 3.5 Sonnet) to optimize costs without needing to manually log every request.
• Compliance and Auditing: A legal team in a regulated industry can use the exported OTLP traces to create a full audit trail of every prompt and response, ensuring that the AI is not leaking sensitive data or violating safety guidelines.
• Latency Analysis: A performance engineer can identify which specific call in a multi-step agentic workflow is causing the most delay, allowing them to target their optimization efforts on the specific model or tool.

Contributing to OpenLLMetry

OpenLLMetry is an open-source project and welcomes contributions from the community. Since it is built on OpenTelemetry, contributors can help by adding new instrumentations for LLM providers or improving the semantic conventions for AI telemetry.

To contribute, developers should start by reporting bugs via GitHub Issues or submitting a Pull Request. The project follows standard GitHub flow and encourages the use of the Contributing Guide found in the repository to ensure consistent code quality and documentation.

Community and Support

OpenLLMetry is maintained by Traceloop and has a strong community of developers building GenAI applications. Support is primarily handled through GitHub Discussions, where users can ask questions and collaborate on new features.

The project also provides comprehensive documentation on their official site, which is powered by Mintlify, and offers integration guides for connecting to various observability backends like Dynatrace, Honeycomb, and New Relic.

Conclusion

OpenLLMetry is the ideal choice for developers who want professional-grade observability for their LLM applications without sacrificing portability. By adhering to the OpenTelemetry standard, it removes the risk of vendor lock-in and allows teams to integrate AI tracing into their existing monitoring infrastructure.

Whether you are building a simple chatbot or a complex autonomous agent, the ability to see exactly what is happening inside your model calls is essential for moving from a prototype to a production-ready system. Star the repo, try the quickstart, and join the community to start bringing transparency to your AI workflows.

Resources

• OpenLLMetry GitHub Repository
• Official OpenLLMetry Documentation
• OpenTelemetry Official Site