---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 6015a9f8a907ba021d369b11ced81273ffdccfb3ec7cd4cb9e06c5447abfbb6c
  pageDirectory: concepts
  sources:
    - built-in-llm-judges-databricks-on-aws.md
  confidence: 0.85
  provenanceState: inferred
  freshnessStatus: unverified
  aliases:
    - llm-as-judge-evaluation-paradigm
    - LEP
  citations:
    - file: built-in-llm-judges-databricks-on-aws.md
title: LLM-as-judge evaluation paradigm
description: The methodology of using one LLM to evaluate the outputs of another LLM-powered application, scoring along quality dimensions such as relevance, safety, groundedness, and correctness.
tags:
  - llm-evaluation
  - methodology
  - genai
timestamp: "2026-06-18T10:55:35.890Z"
---

# LLM-as-judge evaluation paradigm

**LLM-as-judge evaluation paradigm** is an approach to assessing generative AI applications where a large language model (LLM) acts as the evaluator instead of traditional human annotation or heuristic-based metrics. In this paradigm, an LLM is given a scoring rubric or set of criteria and asked to rate the quality of another model's outputs across dimensions such as relevance, safety, groundedness, and correctness. ^[built-in-llm-judges-databricks-on-aws.md]

The paradigm is particularly valuable for evaluating open-ended generative tasks — such as [Retrieval Augmented Generation (RAG)](/concepts/retrieval-augmented-generation-rag.md), conversational AI, and creative text generation — where traditional metrics like BLEU or ROUGE correlate poorly with human judgment. An LLM judge can assess semantic qualities that are difficult to capture with exact-match or n-gram overlap methods. ^[built-in-llm-judges-databricks-on-aws.md]

## How it works

In the LLM-as-judge paradigm, an evaluator LLM is configured with:

1. **A scoring rubric** — a natural-language description of what constitutes good or bad output for the dimension being measured.
2. **Input data** — the original user query or prompt sent to the system being evaluated.
3. **The system's output** — the response generated by the application under test.
4. **Optional ground truth** — a reference answer or expected output against which correctness can be judged.

The judge LLM then produces a rating, typically as a numeric score, a categorical label (e.g., "pass" / "fail"), or a natural-language justification. Some frameworks also extract structured assessments like binary flags for safety violations. ^[built-in-llm-judges-databricks-on-aws.md]

## Built-in LLM judges

Platforms like Databricks and [MLflow](/concepts/mlflow.md) provide predefined LLM judges — sometimes called built-in LLM judges — that use Databricks-hosted LLMs to evaluate common quality dimensions of GenAI applications. These are designed for teams that want to start evaluating quality quickly without designing custom rubrics. ^[built-in-llm-judges-databricks-on-aws.md]

Built-in judges are a subclass of [[scorers]] in the MLflow evaluation framework. They cover dimensions such as: ^[built-in-llm-judges-databricks-on-aws.md]

| Judge | Required ground truth | What it evaluates |
|-------|----------------------|-------------------|
| RelevanceToQuery | No | Is the response directly relevant to the user's request? |
| RetrievalRelevance | No | Is the retrieved context directly relevant to the user's request? |
| Safety | No | Is the content free from harmful, offensive, or toxic material? |
| RetrievalGroundedness | No | Is the response grounded in the provided context? (detects hallucination) |
| Correctness | Yes | Is the response correct compared to ground truth? |
| RetrievalSufficiency | Yes | Does the context provide all necessary information? |
| Guidelines | No | Does the response meet specified natural-language criteria? |
| ToolCallCorrectness | Yes | Are the tool calls and arguments correct? |
| ToolCallEfficiency | No | Are the tool calls efficient without redundancy? |

^[built-in-llm-judges-databricks-on-aws.md]

### Multi-turn judges

For conversational AI systems, MLflow provides multi-turn judges that evaluate entire conversations rather than individual turns. These judges analyze the complete conversation history to assess quality patterns that emerge over multiple interactions. Multi-turn judges can be used both for evaluation during development and for production monitoring. ^[built-in-llm-judges-databricks-on-aws.md]

## Custom LLM judges

When built-in judges don't fit a specific use case, practitioners can create [Custom LLM Judges](/concepts/custom-llm-judges.md). Custom judges allow teams to define their own scoring criteria, prompt templates, and output parsing logic. This is useful for domain-specific evaluations where the general-purpose rubrics of built-in judges are insufficient. ^[built-in-llm-judges-databricks-on-aws.md]

## Aligning judges with human feedback

A key challenge in the LLM-as-judge paradigm is ensuring that the judge's assessments align with human preferences. Approaches to alignment include:

- Prompt engineering: refining the judge's scoring rubric with examples.
- Fine-tuning the judge LLM on human-annotated data.
- Using few-shot prompting with curated reference examples.

Platforms such as MLflow provide tooling to [align judges with human feedback](/concepts/aligning-judges-with-human-experts.md) to improve accuracy on specific domains. ^[built-in-llm-judges-databricks-on-aws.md]

## Selecting the LLM that powers the judge

Not all LLMs are equally effective as judges. The choice of the judge model affects evaluation quality. Smaller or less capable models may produce unreliable scores, while more powerful models tend to correlate better with human judgment. The selection typically involves trade-offs between cost, latency, and evaluation accuracy. See the documentation on how to [select the LLM that powers a judge](/concepts/llm-as-a-judge.md) for guidance. ^[built-in-llm-judges-databricks-on-aws.md]

## Comparison with alternative evaluation paradigms

The LLM-as-judge paradigm differs from other evaluation approaches:

- **Heuristic metrics** (e.g., BLEU, ROUGE, METEOR): Fast and reproducible but poor at capturing semantic quality for generative tasks.
- **Human evaluation**: The gold standard for accuracy, but expensive, slow, and difficult to scale.
- **Code-based scorers**: Python functions that compute scores programmatically. These are deterministic and fast but limited to well-defined quantitative criteria.
- **LLM-as-judge**: Best suited for evaluating semantic and qualitative dimensions where human-like judgment is needed, but at lower cost and higher throughput than human annotation.

MLflow supports both LLM judges and [Code-based Scorers](/concepts/code-based-scorers.md), and practitioners can mix both approaches in a single evaluation pipeline. ^[built-in-llm-judges-databricks-on-aws.md]

## Limitations and considerations

- **Judge bias**: LLM judges can exhibit biases similar to their training data, including preferences for longer or more verbose responses.
- **Calibration**: Scores may not be well-calibrated across different dimensions or domains without careful prompt engineering.
- **Cost**: Using a powerful LLM for evaluation incurs inference costs, especially at production scale.
- **Hallucination in judges**: The judge LLM can itself produce errors or inconsistent ratings, particularly on ambiguous or poorly defined criteria.
- **Reproducibility**: LLM outputs are stochastic, so scores may vary across runs unless temperature is set to zero and a fixed seed is used.

## Related concepts

- [MLflow](/concepts/mlflow.md) — The open-source platform providing built-in LLM judge implementations
- GenAI evaluation and monitoring — The broader practice of evaluating generative AI applications
- [[Scorers]] — The abstraction used by MLflow to define evaluation functions
- [Custom LLM Judges](/concepts/custom-llm-judges.md) — User-defined judges for domain-specific criteria
- [Code-based Scorers](/concepts/code-based-scorers.md) — Deterministic Python-based scorers as an alternative to LLM judges
- [Model-as-judge](/concepts/llm-as-a-judge.md) — A synonym for the LLM-as-judge paradigm
- [RAGAS](/concepts/ragas-retrieval-augmented-generation-assessment.md) — A related framework for evaluating RAG systems
- Human evaluation — The gold standard that LLM-as-judge aims to approximate
- Prompt engineering — Techniques used to craft effective judge prompts

## Sources

- built-in-llm-judges-databricks-on-aws.md

# Citations

1. [built-in-llm-judges-databricks-on-aws.md](/references/built-in-llm-judges-databricks-on-aws-15825704.md)
