Introduction

Large Language Models (LLMs) are increasingly deployed in high-stakes environments like healthcare and finance, but their tendency to generate plausible-sounding yet false information—known as hallucinations—remains a critical barrier to production reliability. UQLM (Uncertainty Quantification for Language Models) is an open-source Python library that provides a standardized framework for detecting these hallucinations at generation time without requiring ground truth data. By implementing state-of-the-art uncertainty quantification (UQ) techniques, UQLM allows developers to assign a confidence score to every model response, effectively flagging potentially unreliable outputs before they reach the end user.

What Is UQLM?

UQLM is a Python library designed for LLM hallucination detection using uncertainty quantification techniques. It provides a suite of response-level scorers that compute confidence scores ranging from 0 to 1, where higher scores indicate a lower likelihood of hallucination. The library is maintained by CVS Health and is released under the Apache License 2.0, making it suitable for both commercial and personal use.

Unlike traditional evaluation toolkits that require a gold-standard dataset for comparison, UQLM is “zero-resource,” meaning it analyzes the model’s own signals—such as token probabilities or consistency across multiple samples—to determine if the model is “uncertain” about its answer. This makes it a practical tool for real-time production pipelines where ground truth is unavailable.

Why UQLM Matters

The primary challenge with LLM hallucinations is that they are often indistinguishable from correct answers to the human eye. For developers building RAG (Retrieval-Augmented Generation) systems or autonomous agents, the risk of a model confidently stating a falsehood is a significant liability. UQLM fills this gap by providing a programmatic way to quantify the “confidence” of a response at the moment of generation.

By integrating UQLM, teams can implement safety thresholds. For example, if a response scores below 0.7, the system can automatically trigger a human-in-the-loop review or ask the model to refine its answer. This shifts the paradigm from blind trust in LLM outputs to a verifiable, uncertainty-aware generation process. The project’s academic backing, including a publication in the Journal of Machine Learning Research (JMLR), underscores its methodology’s reliability.

Key Features

• Black-Box Scorers: These measure semantic consistency across multiple LLM generations for the same prompt. They are universal and work with any model API, as they require no access to internal model states.
• White-Box Scorers: These leverage token probabilities (logprobs) to assess confidence. They offer minimal latency and zero additional cost because they use data already returned by the model during a single generation.
• LLM-as-a-Judge Scorers: These utilize one or more separate LLMs to evaluate the reliability of a response. This approach is highly customizable via prompt engineering and can be used to create a “panel of judges” for higher accuracy.
• Ensemble Scorers: These combine multiple UQ signals (e.g., combining a white-box and a black-box score) via weighted averaging to create a more robust confidence estimate than any single method could provide.
• Long-Text Scorers: These provide claim-level uncertainty scoring for long-form responses, allowing developers to identify exactly which part of a long answer is likely a hallucination.
• LangChain Integration: UQLM is designed to work seamlessly with any LLM called via LangChain interfaces, making it easy to plug into existing AI orchestration frameworks.

How UQLM Compares

Most hallucination detection tools fall into two categories: RAG-centric metrics (which compare output to a source document) or Evals-based tools (which compare output to a ground-truth label). UQLM differs by focusing on uncertainty quantification, which requires neither source documents nor labels.

UQLM is the right choice when you have a closed-book QA system where no external context is provided, or when you want a second layer of verification that doesn’t depend on the quality of your retrieval step. While RAG metrics are excellent for measuring faithfulness to a document, UQLM measures the model’s internal confidence in its own knowledge.

Getting Started: Installation

UQLM can be installed directly from PyPI. It is recommended to use a virtual environment to manage dependencies.

Using pip

pip install uqlm

Prerequisites: UQLM requires Python 3.9+ and integrates with LangChain. You will need API keys for the LLMs you intend to use (e.g., OpenAI, Google Vertex AI, or Azure OpenAI).

How to Use UQLM

The basic workflow in UQLM involves wrapping a LangChain LLM instance with a UQ scorer. You then call generate_and_score, which handles the generation of the response and the calculation of the confidence score simultaneously.

Depending on your needs, you choose a scorer type based on the tradeoff between latency, cost, and compatibility. For instance, if you need the fastest possible detection, you would use a White-Box scorer. If you are using a model API that does not expose logprobs, you would use a Black-Box scorer.

Code Examples

Black-Box Uncertainty Quantification

This example demonstrates how to use BlackBoxUQ to measure consistency across multiple samples. This method is universal and works with any LLM.

from langchain_google_vertexai import ChatVertexAI
from uqlm import BlackBoxUQ

llm = ChatVertexAI(model='gemini-pro')
# Initialize scorer with semantic negentropy as the metric
bbuq = BlackBoxUQ(llm=llm, scorers=["semantic_negentropy"], use_best=True)

# Generate and score a prompt
results = await bbuq.generate_and_score(prompts=["What is the capital of France?"], num_responses=5)
print(results.to_df())

White-Box Uncertainty Quantification

This example uses WhiteBoxUQ, which is the most efficient method as it leverages token probabilities. Note that this requires the LLM API to support logprobs.

from langchain_google_vertexai import ChatVertexAI
from uqlm import WhiteBoxUQ

llm = ChatVertexAI(model='gemini-pro')
# Use min_probability as the scoring metric
wbuq = WhiteBoxUQ(llm=llm, scorers=["min_probability"])

results = await wbuq.generate_and_score(prompts=["Explain quantum computing in one sentence."])
print(results.to_df())

Ensemble Scoring

For maximum reliability, you can combine multiple scorers into a UQEnsemble. This allows you to balance the speed of white-box methods with the robustness of black-box methods.

from langchain_google_vertexai import ChatVertexAI
from uqlm import UQEnsemble

llm = ChatVertexAI(model='gemini-pro')
# Combine multiple scoring methods
scorers = ["exact_match", "noncontradiction", "min_probability", llm]

uqe = UQEnsemble(llm=llm, scorers=scorers)
results = await uqe.generate_and_score(prompts=["Who won the Oscars in 2024?"])
print(results.to_df())

Real-World Use Cases

UQLM is particularly useful in domains where the cost of a hallucination is high. Here are a few concrete scenarios:

• Medical Information Retrieval: A healthcare provider uses UQLM to flag responses about drug interactions. If the confidence score is low, the system prevents the output from being shown to the clinician, instead prompting the user to consult a verified medical manual.
• Financial Compliance: An analyst uses UQLM to automate the extraction of data from annual reports. When the model extracts a figure that it is uncertain about, UQLM flags it for human review, ensuring that financial statements are not based on fabricated numbers.
• Customer Support Automation: A company implements UQLM in their support bot. When the bot is uncertain about a technical answer, it gracefully degrades to a human agent hand-off rather than providing a potentially incorrect technical instruction.
• Closed-Book QA Systems: For systems that don’t use RAG, UQLM is the only way to detect hallucinations at generation time without having a reference document to check against.

Contributing to UQLM

UQLM is an open-source project and welcomes contributions from the community. You can contribute by reporting bugs via GitHub Issues, suggesting new uncertainty quantification metrics, or submitting pull requests to improve the library’s performance.

The project follows standard GitHub flow for contributions. If you are interested in contributing, please review the contributor guide in the repository to ensure your code meets the project’s quality standards.

Community and Support

The primary hub for UQLM support is the GitHub repository. Developers can use GitHub Discussions for architectural questions and collaboration. The project also provides a comprehensive documentation site built with Sphinx, which includes an API reference and example notebooks for various UQ approaches.

Conclusion

UQLM is a critical tool for any developer moving LLMs from a prototype to a production-ready system. By quantifying uncertainty, it provides a safety layer that allows for the more responsible deployment of generative AI. While no single UQ method is perfect, the library’s ability to ensemble multiple signals makes it one of the most robust frameworks available for hallucination detection.

If you are building high-stakes AI applications, we recommend starting with the White-Box scorers for efficiency and then moving to the Ensemble approach for maximum reliability. Star the repo, try the quickstart, and join the community to help build more trustworthy AI.

Resources

• UQLM GitHub Repository
• UQLM Official Documentation
• UQLM Research Paper (arXiv)
• GitHub Discussions