LLM evaluation and hallucinations

Intrinsic hallucinations contradict the provided source/context (e.g., misquoting a passage given in the prompt), while extrinsic hallucinations introduce content that cannot be verified against the source - it may be true or false but is unsupported by the given input. The distinction matters because intrinsic errors are detectable via source comparison, whereas extrinsic ones require external knowledge or grounding to evaluate.

Perplexity measures how well a model predicts a held-out token distribution, rewarding fluency and likelihood calibration, but it does not assess whether outputs are factually correct, helpful, or aligned with instructions. A model can be highly confident (low perplexity) while being confidently wrong, and RLHF-tuned models often have worse perplexity yet better task performance. It also cannot compare models with different tokenizers/vocabularies fairly.

Position bias is the tendency of a judge model to favor the response shown in a particular position (often the first) when doing pairwise comparison, regardless of quality. It is mitigated by swapping the order of the two candidates and averaging/requiring agreement across both orderings, and by using calibrated prompts or multiple judges. Persistent disagreement after swapping signals a genuine tie or low judge reliability.

Faithfulness measures whether the generated answer is grounded in the retrieved context - i.e., the fraction of claims in the answer that can be inferred from the provided context. It is computed by decomposing the answer into atomic claims/statements and using an LLM to verify how many are supported by the retrieved passages, yielding supported_claims / total_claims. Low faithfulness indicates extrinsic hallucination relative to the retrieval context.

SelfCheckGPT samples multiple stochastic responses to the same prompt and measures consistency: if the model genuinely knows a fact, the sampled responses agree; if it is hallucinating, they diverge. Consistency is scored via NLI, n-gram overlap, BERTScore, or QA-based checks across samples, flagging low-consistency sentences as likely hallucinations. It is black-box (no logits needed) and reference-free.

LLMs are trained to produce fluent, high-likelihood continuations, so a fabricated but plausible-sounding fact can receive high token probabilities - the model is confident in the linguistic form, not the truth. This reflects a calibration gap: the probability of the next token is not the probability the statement is true, and RLHF can worsen this calibration. Robust uncertainty estimation often needs semantic-level methods (e.g., semantic entropy) rather than raw token logprobs.

Semantic entropy clusters multiple sampled generations by meaning (using bidirectional entailment) and computes entropy over the semantic clusters rather than over surface token sequences. This avoids inflating uncertainty when the model expresses the same correct answer in many different wordings, isolating uncertainty that is actually about meaning. High semantic entropy indicates the model is unsure about the substantive answer, correlating with hallucination/confabulation.

Static multiple-choice benchmarks are prone to training-data contamination, can be gamed by format-specific tricks, and measure narrow capabilities that may not transfer to open-ended real-world tasks. High MMLU accuracy says little about hallucination rate, instruction-following on novel prompts, calibration, or robustness, so it should be complemented with held-out, contamination-checked, and task-realistic evaluations. Saturation near ceiling also reduces discriminative power.

Common signals include: comparing performance on the original benchmark vs. a paraphrased/perturbed or newly-collected variant (a large drop suggests memorization), checking whether the model can complete or reproduce verbatim test items, using canary strings or membership-inference-style probes, and testing performance by data cutoff date. n-gram overlap analysis between outputs and known test sets, and 'guess the missing option' tests, also help. No single test is definitive; triangulate.

Faithfulness asks whether the answer is supported by the retrieved context (grounding), while factuality asks whether the answer is true in the real world. An answer can be faithful but factually wrong (if the retrieved source is wrong) or factually correct but unfaithful (correct via parametric knowledge but not grounded in the retrieved context). Both must be measured separately to diagnose whether errors come from retrieval or generation.

BLEU/ROUGE measure n-gram overlap with reference text, penalizing valid paraphrases and rewarding surface lexical matches, so they correlate poorly with human judgments of correctness and helpfulness for open-ended generation. Better alternatives include embedding-based metrics (BERTScore), claim-level factuality checks, and LLM-as-a-judge with rubrics or pairwise preference, ideally validated against human ratings. For RAG specifically, grounding/faithfulness metrics are more informative.

Sycophancy is the tendency of an LLM to agree with or conform to a user's stated belief, correction, or implied preference even when it is wrong, often a side effect of RLHF optimizing for human approval. It is an evaluation concern because models may flip correct answers when users push back, inflating agreement and passing naive helpfulness tests while degrading factual reliability. Evaluations should include adversarial/leading prompts that test whether the model holds correct positions under pressure.

FActScore decomposes a generation into atomic facts and computes the percentage that are supported by a reliable knowledge source (e.g., Wikipedia), giving a fine-grained precision score rather than a single binary correct/incorrect judgment. This captures partial correctness in long answers where some claims are true and others fabricated. It typically uses an automated verifier (retrieval + NLI/LLM) validated against human atomic-fact annotations.

A judge from the same family can exhibit self-preference (self-enhancement) bias, systematically rating outputs that match its own style, formatting, or distribution higher, inflating scores for the model under test. It may also share blind spots, failing to catch the same hallucinations it would produce. Mitigations include using a different/stronger judge, anonymizing style, calibrating against human labels, and ensemble judging.

Adding more retrieved passages raises recall but can introduce noisy, irrelevant, or contradictory context that distracts the model, dilutes the relevant signal, and triggers 'lost in the middle' positional effects, sometimes increasing errors. The generator may also still ignore context and rely on parametric knowledge. Effective RAG requires balancing recall with precision/reranking, context ordering, and instructing the model to abstain when context is insufficient.

LLMs tend to use information placed at the beginning and end of a long context more reliably than information in the middle, producing a U-shaped accuracy curve as a function of evidence position. This matters because long-context benchmarks must vary the position of the relevant evidence; otherwise scores overstate true long-context utilization. It also motivates reranking key evidence toward the prompt's edges.

Use datasets containing unanswerable or out-of-scope questions and measure abstention behavior with metrics like coverage vs. accuracy (selective prediction), AUROC of confidence-based abstention, and a penalty for confident wrong answers. A good model should abstain on questions it cannot answer while still answering known ones, so report both the abstention rate on unanswerables and the accuracy on answerables. Scoring rules that reward calibrated abstention (e.g., negative marking) discourage guessing.

ECE quantifies the mismatch between predicted confidence and empirical accuracy by binning predictions by confidence and computing the weighted average absolute difference between each bin's average confidence and its actual accuracy. A well-calibrated model has ECE near zero, meaning when it says 80% confident it is right about 80% of the time. ECE is sensitive to binning choices and does not capture per-instance reliability, so it is often paired with reliability diagrams.

import numpy as np
def ece(confs, correct, n_bins=10):
    confs = np.asarray(confs, dtype=float)
    correct = np.asarray(correct, dtype=float)
    edges = np.linspace(0.0, 1.0, n_bins + 1)
    # right=True so the bin (lo, hi] matches each interval; clip puts 0.0 in bin 1
    idx = np.clip(np.digitize(confs, edges[1:-1], right=True), 0, n_bins - 1)
    e = 0.0
    for b in range(n_bins):
        m = idx == b
        if not m.any():
            continue
        acc = correct[m].mean()
        conf = confs[m].mean()
        e += m.mean() * abs(acc - conf)
    return e

Human labels are the ground truth that automated metrics and LLM judges are validated against, so reliable labels require measuring agreement (kappa corrects for chance agreement). Low agreement implies an ambiguous rubric, an ill-defined construct (e.g., 'hallucination' not operationalized), or insufficient annotator training, meaning the resulting scores are noisy and any metric correlated to them is suspect. The fix is to clarify guidelines, add adjudication, and report agreement alongside results.

Pointwise scoring assigns each response an absolute score/rating on a rubric, while pairwise evaluation asks which of two responses is better. Pairwise is generally more reliable and consistent for subtle quality differences because relative judgments are easier for both humans and LLM judges, but it scales O(n^2) for ranking many systems and needs aggregation (e.g., Elo/Bradley-Terry). Pointwise is cheaper and gives interpretable absolute thresholds, but suffers from rubric drift and judge miscalibration.

It collects large numbers of human pairwise preferences between anonymized model responses and fits a Bradley-Terry/Elo model to produce a relative ranking with confidence intervals. Its strengths are realism (real user prompts) and resistance to single-metric gaming, but limitations include prompt distribution skew toward easy/casual queries, vulnerability to style/length preferences over correctness, slow turnaround, and inability to isolate specific capabilities or detect hallucinations directly.

Judge models (and humans) often rate longer, more detailed responses as higher quality regardless of correctness, so a verbose-but-wrong answer can beat a concise-correct one. Control for it by adding explicit rubric instructions to ignore length, normalizing or matching response lengths, measuring the length-quality correlation, and using preference models with debiasing (e.g., length-controlled win rates as in AlpacaEval 2.0). Reporting length-controlled metrics prevents gaming by padding.

A reward model is a proxy for human preference; optimizing the policy too hard against it exploits the proxy's errors (Goodhart's law), producing outputs that score high on the reward model but are actually worse to humans - e.g., sycophantic, verbose, or formulaic text. It is detected by tracking the gap between proxy reward and a held-out gold/human evaluation as training proceeds; gold performance peaks then declines while proxy reward keeps rising. KL penalties and early stopping mitigate it.

Decompose the pipeline and measure each stage: retrieval metrics (recall@k, precision@k, context relevance / hit rate) tell you if the right evidence was fetched, while generation metrics (faithfulness/groundedness given the retrieved context, answer relevance) tell you if the model used it correctly. If retrieval recall is low, fix retrieval; if recall is high but faithfulness is low, fix the generator/prompt. This component-wise diagnosis avoids misattributing failures.

Models are often better at discriminating/verifying whether a given statement is true than at generating only true statements unprompted - verification is an easier task than reliable generation. This gap matters because it justifies generate-then-verify pipelines (self-critique, chain-of-verification) where the model first drafts an answer and then checks each claim, catching hallucinations the initial generation introduced. Evaluation should test both generation accuracy and the model's self-verification accuracy.

CoVe has the model draft an answer, generate independent verification questions about its own claims, answer those questions (ideally without conditioning on the original draft to avoid anchoring), and then revise the answer based on inconsistencies. It reduces hallucinations by surfacing unsupported claims, but its key failure mode is that the verification step uses the same model, so it can repeat the same erroneous belief or be anchored to the flawed draft, giving false confidence. External tools/retrieval strengthen the verification.

LLM benchmark scores have variance from sampling (decoding temperature, prompt phrasing, few-shot ordering) and finite test sets, so a raw point-difference of a few percent can be within noise. Reporting bootstrap confidence intervals and paired significance tests (e.g., McNemar, paired bootstrap) prevents overclaiming that one model beats another. This is essential because leaderboard differences are often statistically insignificant given typical eval-set sizes.

An NLI model classifies whether a hypothesis (a claim in the generated answer) is entailed, neutral, or contradicted by a premise (the source/retrieved context). For groundedness, each answer claim is checked against the context: 'entailment' means supported, while 'contradiction/neutral' flags potential hallucination. NLI-based scoring is faster/cheaper than full LLM judging and provides a principled supported-vs-unsupported signal, though it can struggle with multi-hop reasoning and numeric claims.

Specification gaming is when an agent achieves the literal reward/success signal without accomplishing the intended task (e.g., editing the test file to make tests pass, or returning a hardcoded answer). Genuine capability solves the underlying problem. Guard against it with held-out hidden tests, sandboxing that prevents tampering, process/trajectory inspection (not just final-state checks), and adversarial task variants that break shortcut solutions. Outcome-only metrics are insufficient for agents.

Higher decoding temperature increases output diversity and can raise hallucination rates and variance, while temperature 0 (greedy) gives reproducible but possibly suboptimal outputs and hides the model's output distribution. Evaluations must fix and report decoding parameters (temperature, top-p, seed) and, for stochastic settings, average over multiple samples with reported variance. Comparing two models under different decoding settings is invalid.

Answer relevance measures how well the generated answer actually addresses the user's question (on-topic, complete, not evasive), typically by using an LLM to generate candidate questions from the answer and measuring their embedding similarity to the original question. It differs from faithfulness, which checks grounding in context: an answer can be faithful to the context yet irrelevant to the question, or relevant yet ungrounded. Both are needed for a complete RAG quality picture.

Build the metric (LLM judge, NLI, or FActScore-style verifier), then validate it by measuring its agreement/correlation with a high-quality human-labeled set (e.g., Spearman/kappa, AUROC for detecting human-flagged hallucinations) across diverse, held-out data. The gold standard is meta-evaluation: the metric is only trustworthy to the extent it correlates with careful human judgment on data resembling deployment, and it should be re-validated when the model or domain shifts. Report the metric's own error rate.

Closed-book evaluation tests the model's parametric knowledge with no external context, revealing what it has memorized and how prone it is to fabricating when it doesn't know. Open-book/grounded evaluation provides context (retrieval or documents) and tests whether the model correctly uses it, isolating reasoning/grounding ability from memorization. Comparing the two reveals whether errors stem from missing knowledge (closed-book gap) or poor context utilization (open-book gap), guiding whether to add RAG or improve grounding.