Documentation  ·  Examples  ·  Citation  ·  Contributing

📄 Publications: JMLR (Software) · TMLR (Ensemble UQ) · TMLR (Long-Text UQ) · EMNLP (Code Generation UQ)

UQLM is a Python library for detecting hallucinations in Large Language Model (LLM) outputs using state-of-the-art uncertainty quantification techniques.

The latest version can be installed from PyPI:

pip install uqlm

UQLM provides a suite of response-level scorers, each returning a confidence score between 0 and 1, where higher scores indicate a lower likelihood of errors or hallucinations.

We categorize these scorers into different types:

^{*Does not apply to multi-generation white-box scorers, which have higher cost and latency.}

Below we provide illustrative code snippets and details about available scorers for each type.

These scorers assess uncertainty by measuring the consistency of multiple responses generated from the same prompt. They are compatible with any LLM, intuitive to use, and don't require access to internal model states or token probabilities.

Example Usage: Below is a sample of code illustrating how to use the BlackBoxUQ class to conduct hallucination detection.

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o-mini")

from uqlm import BlackBoxUQ
bbuq = BlackBoxUQ(llm=llm, scorers=["semantic_negentropy"], use_best=True)

results = await bbuq.generate_and_score(prompts=prompts, num_responses=5)
results.to_df()

Above, use_best=True implements mitigation so that the uncertainty-minimized responses is selected. Note that although we use ChatOpenAI in this example, any LangChain Chat Model may be used. For a more detailed demo, refer to our Black-Box UQ Demo.

Available Scorers:

• Discrete Semantic Entropy (Farquhar et al., 2024; Bouchard & Chauhan, 2025)
• Number of Semantic Sets (Lin et al., 2024; Vashurin et al., 2025; Kuhn et al., 2023)
• Non-Contradiction Probability (Chen & Mueller, 2023; Lin et al., 2024; Manakul et al., 2023)
• Entailment Probability (Chen & Mueller, 2023; Lin et al., 2024; Manakul et al., 2023)
• Exact Match (Cole et al., 2023; Chen & Mueller, 2023)
• BERTScore (Manakul et al., 2023; Zheng et al., 2020)
• Cosine Similarity (Shorinwa et al., 2024)
• Code-adapted scorers via CodeGenUQ.

These scorers leverage token probabilities to estimate uncertainty. They offer single-generation scoring, which is significantly faster and cheaper than black-box methods, but require access to the LLM's internal probabilities, meaning they are not necessarily compatible with all LLMs/APIs.

Example Usage: Below is a sample of code illustrating how to use the WhiteBoxUQ class to conduct hallucination detection.

from langchain_google_vertexai import ChatVertexAI
llm = ChatVertexAI(model='gemini-2.5-pro')

from uqlm import WhiteBoxUQ
wbuq = WhiteBoxUQ(llm=llm, scorers=["min_probability"])

results = await wbuq.generate_and_score(prompts=prompts)
results.to_df()

Again, any LangChain Chat Model may be used in place of ChatVertexAI. For more detailed examples, refer to our demo notebooks on Single-Generation White-Box UQ and/or Multi-Generation White-Box UQ.

Single-Generation Scorers (minimal latency, zero extra cost):

• Minimum token probability (Manakul et al., 2023)
• Length-Normalized Sequence Probability (Malinin & Gales, 2021)
• Sequence Probability (Vashurin et al., 2024)
• Mean Top-K Token Negentropy (Scalena et al., 2025; Manakul et al., 2023)
• Min Top-K Token Negentropy (Scalena et al., 2025; Manakul et al., 2023)
• Probability Margin (Farr et al., 2024)

Self-Reflection Scorers (one additional generation per response):

• P(True) (Kadavath et al., 2022)

Multi-Generation Scorers (several additional generations per response):

• Monte carlo sequence probability (Kuhn et al., 2023)
• Consistency and Confidence (CoCoA) (Vashurin et al., 2025)
• Semantic Entropy (Farquhar et al., 2024)
• Semantic Density (Qiu et al., 2024)

These scorers use one or more LLMs to evaluate the reliability of the original LLM's response. They offer high customizability through prompt engineering and the choice of judge LLM(s).

Example Usage: Below is a sample of code illustrating how to use the LLMPanel class to conduct hallucination detection using a panel of LLM judges.

from langchain_ollama import ChatOllama
llama = ChatOllama(model="llama3")
mistral = ChatOllama(model="mistral")
qwen = ChatOllama(model="qwen3")

from uqlm import LLMPanel
panel = LLMPanel(llm=llama, judges=[llama, mistral, qwen])

results = await panel.generate_and_score(prompts=prompts)
results.to_df()

Note that although we use ChatOllama in this example, we can use any LangChain Chat Model as judges. For a more detailed demo illustrating how to customize a panel of LLM judges, refer to our LLM-as-a-Judge Demo.

Available Scorers:

• Categorical LLM-as-a-Judge (Manakul et al., 2023; Chen & Mueller, 2023; Luo et al., 2023)
• Continuous LLM-as-a-Judge (Xiong et al., 2024)
• Likert Scale LLM-as-a-Judge (Bai et al., 2023)
• Panel of LLM Judges (Verga et al., 2024)

These combine multiple individual scorers via weighted averaging to produce more robust uncertainty estimates. They are highly customizable for specific use cases and can be used off-the-shelf with fixed weights (unsupervised) or trained for optimal performance (supervised). The following workflow demonstrates the supervised training process.

Example Usage: Below is a sample of code illustrating how to use the UQEnsemble class to conduct hallucination detection.

from langchain_openai import AzureChatOpenAI
llm = AzureChatOpenAI(deployment_name="gpt-4o", openai_api_type="azure", openai_api_version="2024-12-01-preview")

from uqlm import UQEnsemble
## ---Option 1: Off-the-Shelf Ensemble---
# uqe = UQEnsemble(llm=llm)
# results = await uqe.generate_and_score(prompts=prompts, num_responses=5)

## ---Option 2: Tuned Ensemble---
scorers = [ # specify which scorers to include
    "exact_match", "noncontradiction", # black-box scorers
    "min_probability", # white-box scorer
    llm # use same LLM as a judge
]
uqe = UQEnsemble(llm=llm, scorers=scorers)

# Tune on tuning prompts with provided ground truth answers
tune_results = await uqe.tune(
    prompts=tuning_prompts, ground_truth_answers=ground_truth_answers
)
# ensemble is now tuned - generate responses on new prompts
results = await uqe.generate_and_score(prompts=prompts)
results.to_df()

As with the other examples, any LangChain Chat Model may be used in place of AzureChatOpenAI. For more detailed demos, refer to our