Large language models (LLMs) are often introduced as tools for summarization, question answering, content generation, or conversational interfaces. However, one of their most valuable applications in business and technical workflows is more structured: evaluating a limited set of candidate documents and returning a typed relevance decision that humans and systems can review, monitor, and continuously improve.

This approach is fundamentally different from asking an LLM, “Which document is best?” A reliable evaluation system should define what relevance means, constrain the model to a known set of documents, require structured outputs, preserve evidence for every decision, and measure performance over time.

The result is not simply an AI-powered search feature. It is a decision-making layer that can support document triage, compliance review, customer support, knowledge management, incident investigation, legal discovery, research workflows, and many other applications.

Understanding The Core Problem

Consider a system that receives a user question and five to twenty candidate documents. The task is to determine how each document relates to the question.

A simplistic implementation might look like this:

prompt = f"""
User question:
{question}

Documents:
{documents}

Which documents are relevant?
"""

response = llm(prompt)

Although simple, this design creates several problems.

The model may use an undefined interpretation of the word “relevant.” It may return inconsistent explanations. It may omit documents without explaining why. It may produce an answer that is difficult to parse automatically. Most importantly, it becomes difficult to audit whether the model’s decisions are correct.

A better system asks the model to evaluate each candidate according to an explicit schema.

For example:

Document A:
- relevance: DIRECT
- confidence: 0.94
- evidence: "The policy states that refunds are available within 30 days."
- rationale: "The document directly answers the user's question about refund eligibility."

Document B:
- relevance: RELATED
- confidence: 0.71
- evidence: "The document discusses exchanges rather than refunds."
- rationale: "It provides related policy information but does not directly answer the question."

Document C:
- relevance: NOT_RELEVANT
- confidence: 0.98
- rationale: "The document concerns shipping restrictions."

This is the foundation of typed relevance.

Why Typed Relevance Is Better Than A Binary Decision

A binary relevant/not relevant classification is often too limited for real-world review workflows.

A document can directly answer a question, provide useful background, contain partial evidence, or be completely unrelated. Treating all these possibilities as either “relevant” or “irrelevant” removes useful information.

A typed taxonomy might include:

DIRECT
PARTIAL
CONTEXTUAL
CONTRADICTORY
NOT_RELEVANT
INSUFFICIENT_INFORMATION

Each category has a specific meaning.

DIRECT means that the document contains information that directly answers or resolves the question.

PARTIAL means that the document addresses part of the question but does not provide a complete answer.

CONTEXTUAL means that the document provides useful supporting information without directly answering the question.

CONTRADICTORY means that the document contains information that conflicts with another relevant source, an expected answer, or a stated assumption.

NOT_RELEVANT means that the document does not materially contribute to answering the question.

INSUFFICIENT_INFORMATION means that the document may concern the subject but does not contain enough information to determine its relevance confidently.

This richer classification enables better downstream behavior. A system can automatically use DIRECT documents, send PARTIAL documents for additional retrieval, flag CONTRADICTORY documents for human review, and ignore high-confidence NOT_RELEVANT documents.

Designing A Strong Relevance Schema

The schema should contain more than a relevance label. It should capture the information necessary for review and monitoring.

In Python, a structured result can be represented with Pydantic:

from enum import Enum
from typing import Optional
from pydantic import BaseModel, Field


class RelevanceType(str, Enum):
    DIRECT = "direct"
    PARTIAL = "partial"
    CONTEXTUAL = "contextual"
    CONTRADICTORY = "contradictory"
    NOT_RELEVANT = "not_relevant"
    INSUFFICIENT_INFORMATION = "insufficient_information"


class DocumentEvaluation(BaseModel):
    document_id: str

    relevance: RelevanceType

    confidence: float = Field(
        ge=0.0,
        le=1.0,
        description="Model confidence in the relevance classification."
    )

    evidence: Optional[str] = Field(
        default=None,
        description="A short excerpt or reference supporting the decision."
    )

    rationale: str = Field(
        description="Concise explanation for the classification."
    )

    requires_human_review: bool = False

The purpose of this schema is not merely to make parsing easier. It creates a contract between the model and the rest of the system.

Every evaluation has:

1. An identifiable document.
2. A typed decision.
3. A confidence estimate.
4. Evidence.
5. A rationale.
6. An escalation signal.

This makes the output operational.

Constraining The LLM To A Limited Document Set

The phrase “limited set of documents” is important. The system is not asking the LLM to search the entire internet or an uncontrolled knowledge base. Instead, another process has already produced a manageable candidate set.

For example:

candidate_documents = [
    {
        "id": "doc_001",
        "title": "Refund Policy",
        "content": "Customers may request a refund within 30 days..."
    },
    {
        "id": "doc_002",
        "title": "Exchange Policy",
        "content": "Items may be exchanged within 60 days..."
    },
    {
        "id": "doc_003",
        "title": "International Shipping",
        "content": "Delivery times vary by destination..."
    }
]

The LLM should be instructed that its task is to evaluate only these documents.

For example:

You are evaluating a fixed set of candidate documents.

Do not use external knowledge.

Do not infer facts that are not supported by the supplied documents.

Evaluate every document independently.

For each document, assign exactly one relevance type from the allowed taxonomy.

Support relevant or contradictory classifications with evidence from the document whenever possible.

This constraint improves auditability. If a reviewer disagrees with a decision, they can inspect the exact document and prompt that produced it.

Evaluating Documents Independently

One important design decision is whether to ask the LLM to compare all documents simultaneously or evaluate them independently.

A single batch prompt can be efficient:

prompt = {
    "question": question,
    "documents": candidate_documents,
    "task": "Evaluate every document."
}

However, batch evaluation can introduce interaction effects. The model may rank one document differently because another document appears more persuasive. This can be useful for comparative ranking, but it can make individual relevance judgments less stable.

An alternative is to evaluate documents one at a time.

def build_evaluation_input(question, document):
    return {
        "question": question,
        "document": {
            "id": document["id"],
            "title": document["title"],
            "content": document["content"]
        }
    }

Then:

evaluations = []

for document in candidate_documents:
    result = evaluate_document(
        question=question,
        document=document
    )
    evaluations.append(result)

Independent evaluation provides cleaner monitoring because each classification corresponds to one input pair:

(question, document) -> typed relevance decision

This makes it easier to build datasets, calculate metrics, and identify systematic errors.

Returning Structured Output Instead Of Free-Form Text

Free-form model responses are difficult to process consistently.

For example:

This document seems fairly relevant because it talks about refunds.

What does “fairly relevant” mean? Is the confidence 60 percent or 90 percent? Should the document be automatically used?

A structured output approach removes ambiguity.

A conceptual JSON result could look like this:

{
  "document_id": "doc_001",
  "relevance": "direct",
  "confidence": 0.94,
  "evidence": "Customers may request a refund within 30 days.",
  "rationale": "The document directly states the eligibility period for refunds.",
  "requires_human_review": false
}

The application can then enforce validation:

evaluation = DocumentEvaluation.model_validate(response)

print(evaluation.relevance)
print(evaluation.confidence)

If the model returns an invalid category, missing document identifier, or confidence outside the expected range, the application can reject the result and retry.

Building A Practical Evaluation Prompt

A strong prompt should define the task, the allowed labels, and the expected evidence requirements.

For example:

SYSTEM_PROMPT = """
You are a document relevance evaluator.

Your job is to determine how a supplied document relates to a user question.

Use only the information contained in the document.

Allowed relevance labels:

- direct: The document directly answers the question.
- partial: The document answers part of the question.
- contextual: The document provides useful supporting context.
- contradictory: The document conflicts with relevant information or an explicit assumption.
- not_relevant: The document does not materially help answer the question.
- insufficient_information: The document is related to the subject but does not contain enough information for a confident conclusion.

Rules:

1. Evaluate the supplied document only.
2. Do not use external knowledge.
3. Do not invent evidence.
4. Return concise evidence where available.
5. Set requires_human_review to true when the document is contradictory, ambiguous, or difficult to classify.
"""

The user-level content can then contain the specific evaluation:

def build_prompt(question, document):
    return {
        "question": question,
        "document_id": document["id"],
        "title": document["title"],
        "content": document["content"]
    }

This separation makes the taxonomy reusable across different questions.

Adding Human Review Thresholds

The LLM does not need to make every final decision. One of the most effective architectures is to use the model for triage.

For example:

def should_review(evaluation: DocumentEvaluation) -> bool:
    if evaluation.requires_human_review:
        return True

    if evaluation.confidence < 0.75:
        return True

    if evaluation.relevance == RelevanceType.CONTRADICTORY:
        return True

    return False

The system can divide decisions into categories:

High confidence + clear relevance
        |
        v
Automated processing

Low confidence
        |
        v
Human review

Contradictory evidence
        |
        v
Priority escalation

This approach is often more practical than attempting to maximize complete automation.

The objective should not be “remove humans from the process.” The objective should be “direct human attention toward decisions where it creates the most value.”

Preserving Evidence For Every Important Decision

Evidence is critical for review.

A relevance label without evidence forces a reviewer to repeat the entire evaluation process. A relevance label with a short supporting excerpt allows the reviewer to validate the decision quickly.

For example:

class Evidence(BaseModel):
    text: str
    location: Optional[str] = None


class DocumentEvaluation(BaseModel):
    document_id: str
    relevance: RelevanceType
    confidence: float
    evidence: list[Evidence] = []
    rationale: str
    requires_human_review: bool

A result might contain:

{
  "document_id": "doc_001",
  "relevance": "direct",
  "confidence": 0.91,
  "evidence": [
    {
      "text": "Refund requests must be submitted within 30 days.",
      "location": "Section 2"
    }
  ],
  "rationale": "The document directly specifies the refund eligibility period.",
  "requires_human_review": false
}

Evidence can also support automated quality checks. If the returned evidence does not actually occur in the source document, the result can be flagged.

Creating A Reviewable Decision Record

Every evaluation should ideally become a durable record.

For example:

from datetime import datetime, timezone
from uuid import uuid4


def create_decision_record(question, evaluation, model_name, prompt_version):
    return {
        "decision_id": str(uuid4()),
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "question": question,
        "document_id": evaluation.document_id,
        "relevance": evaluation.relevance.value,
        "confidence": evaluation.confidence,
        "evidence": [
            item.model_dump()
            for item in evaluation.evidence
        ],
        "rationale": evaluation.rationale,
        "requires_human_review": evaluation.requires_human_review,
        "model": model_name,
        "prompt_version": prompt_version
    }

The inclusion of model and prompt versions is particularly important.

Suppose accuracy declines after a prompt modification. Without version information, it may be difficult to determine whether the decline resulted from the new prompt, a model update, a new document source, or a change in user behavior.

Decision records turn model outputs into observable system events.

Monitoring The Quality Of Decisions

Once decisions are structured, the system can calculate useful metrics.

For example:

def relevance_distribution(records):
    counts = {}

    for record in records:
        label = record["relevance"]
        counts[label] = counts.get(label, 0) + 1

    total = len(records)

    return {
        label: count / total
        for label, count in counts.items()
    }

A sudden increase in the NOT_RELEVANT category may indicate that upstream retrieval is returning poor candidates.

A sudden increase in INSUFFICIENT_INFORMATION may indicate changes in document quality.

A sudden increase in CONTRADICTORY decisions may reveal a real inconsistency in the underlying knowledge base.

This is an important distinction: monitoring the model is not only about measuring whether the model is correct. Model outputs can also reveal problems elsewhere in the system.

Comparing LLM Decisions With Human Judgments

A sample of decisions should be reviewed by humans and stored as ground truth.

For example:

ground_truth = {
    "doc_001": "direct",
    "doc_002": "contextual",
    "doc_003": "not_relevant"
}

The model’s results can then be compared:

def calculate_accuracy(predictions, labels):
    correct = 0

    for document_id, expected in labels.items():
        if predictions.get(document_id) == expected:
            correct += 1

    return correct / len(labels)

Accuracy is useful, but it should not be the only metric.

A system that labels nearly everything NOT_RELEVANT may achieve high accuracy if most candidates are irrelevant.

More detailed measurements may include:

• Precision for DIRECT classifications.
• Recall for DIRECT classifications.
• Confusion between PARTIAL and CONTEXTUAL.
• Frequency of unnecessary human review.
• Frequency of missed CONTRADICTORY documents.
• Agreement between human reviewers.
• Confidence calibration.

The choice of metric should reflect the business risk.

If missing a relevant legal document is extremely costly, recall may matter more than precision. If automatically approving an irrelevant compliance document is dangerous, precision may be the primary concern.

Testing Confidence Calibration

LLM confidence should not automatically be interpreted as a mathematically calibrated probability.

Instead, calibration should be measured.

For example, if the model gives 100 decisions a confidence score between 0.90 and 1.00, approximately how many are actually correct according to human review?

def bucket_confidence(confidence):
    if confidence < 0.50:
        return "0.00-0.49"
    elif confidence < 0.75:
        return "0.50-0.74"
    elif confidence < 0.90:
        return "0.75-0.89"
    return "0.90-1.00"

If high-confidence predictions are frequently wrong, the confidence field is not useful as a review threshold without adjustment.

One possible improvement is to use confidence comparatively rather than literally:

0.90+  -> low review priority
0.75-0.89 -> sample for review
Below 0.75 -> mandatory review

These thresholds should be based on observed performance rather than intuition alone.

Handling Contradictory Documents

A particularly useful typed relevance category is CONTRADICTORY.

Imagine evaluating two documents:

Document A:
Refunds are available within 30 days.

Document B:
Refunds are available within 14 days.

A system that simply selects the “most relevant” document may ignore an important conflict.

Instead, the workflow can preserve both decisions:

if evaluation.relevance == RelevanceType.CONTRADICTORY:
    create_review_ticket(
        document_id=evaluation.document_id,
        priority="high",
        reason="Potential conflicting information"
    )

The model should not necessarily decide which document is authoritative. That may depend on metadata such as publication date, source authority, jurisdiction, or organizational ownership.

The LLM’s role is to surface the conflict.

Improving The System Through Error Analysis

Continuous improvement should be based on specific error categories.

For example:

Error Type 1: Direct documents labeled contextual.
Error Type 2: Contextual documents labeled direct.
Error Type 3: Evidence not supported by source text.
Error Type 4: Contradictions missed.
Error Type 5: Confidence too high on incorrect decisions.

Each category suggests a different improvement strategy.

If DIRECT and CONTEXTUAL are frequently confused, the taxonomy definitions may be unclear.

If unsupported evidence appears, the prompt may need stronger grounding requirements or automated evidence validation.

If contradictory documents are missed, the model may need explicit comparison against a known fact set or additional context.

A simple error logging structure might look like this:

def log_review(
    decision_id,
    human_label,
    human_notes
):
    return {
        "decision_id": decision_id,
        "human_label": human_label,
        "human_notes": human_notes,
        "reviewed": True
    }

Over time, these reviews become an evaluation dataset.

Using Evaluation Data To Improve Prompts And Models

The accumulated review dataset can support controlled experiments.

For example, compare two prompt versions:

Prompt Version A:
"Determine whether the document is relevant."

Prompt Version B:
"Classify the document using the defined relevance taxonomy and provide source-grounded evidence."

Both versions can be tested against the same benchmark set.

results = {
    "prompt_a": run_benchmark(prompt_a, test_cases),
    "prompt_b": run_benchmark(prompt_b, test_cases)
}

A more mature system can compare:

• Different prompts.
• Different models.
• Different temperatures.
• Different document chunking strategies.
• Different relevance taxonomies.
• Different review thresholds.

The key principle is controlled comparison. Improvement should be measured rather than assumed.

A Simple End-To-End Architecture

A practical architecture might contain six stages.

1. Candidate Selection
        |
        v
2. Document Normalization
        |
        v
3. LLM Relevance Evaluation
        |
        v
4. Schema Validation
        |
        v
5. Automated Routing
        |
        +--> Accepted
        |
        +--> Human Review
        |
        +--> Conflict Escalation
        |
        v
6. Monitoring And Feedback Storage

The evaluation function might look like this:

def process_document(question, document):
    evaluation = evaluate_document(
        question=question,
        document=document
    )

    validated = DocumentEvaluation.model_validate(
        evaluation
    )

    decision = create_decision_record(
        question=question,
        evaluation=validated,
        model_name="example-model",
        prompt_version="v3"
    )

    if should_review(validated):
        route_to_human_review(decision)
    else:
        store_automated_decision(decision)

    return decision

The architecture remains relatively simple while providing a strong foundation for governance.

Avoiding Common Implementation Mistakes

One common mistake is allowing the model to invent its own relevance categories.

Another is treating a natural-language explanation as if it were a reliable API response.

A third mistake is storing only the final label. Without the input, evidence, prompt version, and model version, later investigation becomes difficult.

A fourth mistake is evaluating system quality only through anecdotal examples.

A fifth mistake is using a single “confidence” number as the sole basis for automation. Confidence should be validated against real human judgments.

Finally, organizations should avoid overcomplicating the initial taxonomy. A taxonomy with twenty labels may sound precise but can produce inconsistent decisions and poor reviewer agreement.

It is usually better to begin with a small number of clearly defined categories and expand only when the data demonstrates a need.

Conclusion

Using LLMs to evaluate a limited set of documents becomes significantly more powerful when the objective is not simply to obtain an answer, but to produce a decision that can be inspected, challenged, measured, and improved.

The central idea is to replace vague relevance judgments with typed relevance. Instead of asking an LLM whether a document is “good” or “relevant,” the system should define exactly how documents can contribute to a decision. A document may directly answer a question, provide only partial information, supply useful context, contradict other evidence, be irrelevant, or lack sufficient information for a confident determination. These distinctions transform relevance from an informal opinion into structured operational data.

The next critical step is enforcing structured output. Every document evaluation should identify the document, assign a valid relevance type, include a confidence estimate, preserve supporting evidence when available, provide a concise rationale, and indicate whether human review is required. This creates a stable interface between the LLM and downstream systems while making decisions understandable to human reviewers.

Constraining the model to a fixed set of documents is equally important. It reduces ambiguity, limits unsupported external assumptions, and creates a clear audit boundary. The system can always answer the question, “What information was the model allowed to consider when making this decision?” This is essential in workflows where traceability and accountability matter.

Human review should also be treated as an integral part of the architecture rather than a sign of failure. High-confidence, low-risk decisions may proceed automatically, while ambiguous, contradictory, or low-confidence cases are routed to reviewers. This allows organizations to focus human expertise on the decisions that deserve the most attention.

Monitoring completes the feedback loop. Once relevance decisions are stored as structured records, teams can observe label distributions, error rates, reviewer disagreement, confidence calibration, contradiction frequency, and changes associated with prompt or model versions. These measurements make it possible to distinguish between model problems, retrieval problems, document quality problems, and taxonomy problems.

Perhaps most importantly, reviewed decisions create the raw material for continuous improvement. Human corrections can be analyzed to discover systematic errors, refine category definitions, improve prompts, adjust thresholds, compare models, and construct benchmark datasets. The system gradually evolves from a collection of individual model calls into a measurable decision-making process.

The most successful implementations therefore do not treat the LLM as an infallible judge. They treat it as a structured evaluator operating within clearly defined boundaries. Its decisions are typed, its evidence is preserved, its uncertainty is visible, its difficult cases are escalated, and its performance is continuously measured against human judgment.

When designed this way, an LLM-based document evaluation workflow can provide both automation and accountability. It can accelerate review without making the decision process opaque. It can identify useful documents without silently discarding uncertainty. It can surface contradictions rather than hiding them behind a single generated answer. And, because every decision becomes observable data, the entire system can be monitored and improved over time.

Ultimately, the goal is not merely to build a smarter document filter. The goal is to create a reliable decision pipeline in which LLM judgments are structured enough to drive automation, transparent enough to support human review, and measurable enough to support continuous improvement. Typed relevance provides the foundation for that pipeline, transforming document evaluation from an informal AI interaction into a robust, reviewable, and increasingly dependable component of modern decision-making systems.