---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 1354d8df995ed2002c29827040958a8e1bbd674f91e783f8f6d301326ec4e27a
  pageDirectory: concepts
  sources:
    - custom-judges-databricks-on-aws.md
  confidence: 0.95
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - custom-llm-judges-make_judge
    - CLJ(
  citations:
    - file: custom-judges-databricks-on-aws.md
    - file: create-a-custom-judge-using-make_judge-databricks-on-aws.md
title: Custom LLM Judges (make_judge)
description: A function in MLflow that lets you define custom evaluation criteria for GenAI applications using natural language prompts, extending beyond built-in judge metrics.
tags:
  - llm-evaluation
  - mlflow
  - genai
timestamp: "2026-06-19T18:03:44.070Z"
---

```markdown
---
title: Custom LLM Judges (make_judge)
summary: Custom LLM judges allow defining nuanced scoring guidelines for GenAI applications using natural language, created via the make_judge() function. They return `mlflow.entities.Feedback` objects and can evaluate agent outputs or full execution traces.
sources:
  - custom-judges-databricks-on-aws.md
  - create-a-custom-judge-using-make_judge-databricks-on-aws.md
kind: concept
createdAt: "2026-06-18T11:25:49.797Z"
updatedAt: "2026-06-19T09:26:57.437Z"
tags:
  - mlflow
  - evaluation
  - genai
  - judges
aliases:
  - custom-llm-judges-make_judge
  - CLJ(
confidence: 0.95
provenanceState: merged
inferredParagraphs: 0
---

# Custom LLM Judges (make_judge)

## Overview

Custom LLM Judges are user‑defined evaluators created with the `make_judge()` function that allow you to define complex and nuanced scoring guidelines for GenAI applications using natural language. While [[MLflow]] provides [[built-in LLM judges]] as excellent starting points for common quality dimensions, custom judges give you full control over evaluation criteria. ^[custom-judges-databricks-on-aws.md]

Custom judges are LLM‑based scorers that evaluate agent outputs or execution traces against criteria you define in natural language instructions. Each judge returns structured `mlflow.entities.Feedback` objects, which can include categorical labels, boolean decisions, or numeric scores. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

## Prompt Template Variables

To create a judge, you provide a prompt with natural language instructions on how to assess the quality of your agent. `make_judge()` accepts template variables to access the agent’s inputs, outputs, expected outputs or behaviors, and even complete traces. ^[custom-judges-databricks-on-aws.md]

Your instructions must include at least one template variable, but you don’t need to use all of them. ^[custom-judges-databricks-on-aws.md]

| Variable | Description |
|----------|-------------|
| `{{ inputs }}` | Input data provided to the agent |
| `{{ outputs }}` | Output data generated by your agent |
| `{{ expectations }}` | Ground truths or expected outcomes |
| `{{ trace }}` | The complete execution trace of your agent |

These are the only variables allowed. Custom variables like `{{ question }}` will throw validation errors to ensure consistent behavior and prevent template injection issues. ^[custom-judges-databricks-on-aws.md]

## Standard (Non‑Trace) Judges

For standard judges that evaluate only inputs and outputs (not traces), you must provide a `scoring_model` argument in `make_judge()`. This designates the model used to score the agent’s outputs. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

The following example defines a standard judge that evaluates issue resolution quality: ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

```python
from mlflow.genai.judges import make_judge
from typing import Literal

issue_resolution_judge = make_judge(
    name="issue_resolution_judge",
    instructions=(
        "You are evaluating the quality of a customer support agent's response.\n"
        "Analyze whether the response resolves the user's issue.\n\n"
        "Return one of the following values: fully_resolved, partially_resolved, needs_follow_up."
    ),
    feedback_value_type=Literal["fully_resolved", "partially_resolved", "needs_follow_up"],
    scoring_model="databricks:/databricks-gpt-5-mini",
)
```

## Trace‑Based Judges

Trace‑based judges analyze execution traces to understand what happened during agent execution. They autonomously explore traces using Model Context Protocol (MCP) tools and can: ^[custom-judges-databricks-on-aws.md]

* Validate tool usage patterns
* Identify performance bottlenecks
* Investigate execution failures
* Verify multi‑step workflows

The following example defines a judge that assesses tool calling correctness by analyzing traces: ^[custom-judges-databricks-on-aws.md]

```python
from mlflow.genai.judges import make_judge
from typing import Literal

tool_usage_judge = make_judge(
    name="tool_usage_validator",
    instructions=(
        "Analyze the {{ trace }} to verify correct tool usage.\n\n"
        "Check that the agent selected appropriate tools for the user's request "
        "and called them with correct parameters."
    ),
    feedback_value_type=Literal["correct", "incorrect"],
    model="databricks:/databricks-gpt-5-mini"  # Required for trace-based judges
)
```

For trace‑based judges to analyze the full trace, the `model` argument must be specified in `make_judge()`. ^[custom-judges-databricks-on-aws.md]

### Model Requirements for Trace‑Based Judges

Trace‑based judges require a model capable of trace analysis. The model can be served by: ^[custom-judges-databricks-on-aws.md]

* [[Foundation Model APIs]] (recommended)
* External Model Serving Endpoints
* [[Custom Model Serving Endpoint Support|Custom Model Serving Endpoints]]

Recommended models include `databricks:/databricks-gpt-5-mini`, `databricks:/databricks-gpt-5`, `databricks:/databricks-gpt-oss-120b`, and `databricks:/databricks-claude-opus-4-5`. ^[custom-judges-databricks-on-aws.md]

## Best Practices for Writing Judge Instructions

Be specific about expected output format. Your instructions should clearly specify what format the judge should return: ^[custom-judges-databricks-on-aws.md]

* **Categorical responses**: List specific values (for example, `fully_resolved`, `partially_resolved`, `needs_follow_up`)
* **Boolean responses**: Explicitly state the judge should return `true` or `false`
* **Numeric scores**: Specify the scoring range and what each score means

For complex evaluation tasks, structure your instructions into clear sections: ^[custom-judges-databricks-on-aws.md]

* What to evaluate
* What information to examine
* How to make the judgment
* What format to return

## Using Judges in Evaluation

Custom judges are used as scorers in `mlflow.genai.evaluate()`. You can include multiple judges in a single evaluation run, each assessing different quality dimensions. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

For example, a set of judges might evaluate issue resolution, expected behaviors, and tool call correctness: ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

```python
result = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=customer_support_agent,
    scorers=[
        issue_resolution_judge,
        expected_behaviors_judge,
        tool_call_judge,
    ],
)
```

Each judge returns structured feedback values that can be compared across different agent configurations for A/B comparison. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

## Aligning Judges with Human Feedback

The base judge is a starting point. As you gather expert annotations on agent outputs, you can align the LLM judges to the feedback to further improve judge accuracy. See Align judges with human feedback. ^[custom-judges-databricks-on-aws.md]

## Related Concepts

* make_judge()|Make Judge API — The `make_judge()` function API reference
* [[Built-in LLM Judges]] — Pre‑defined judges for common quality dimensions
* [[MLflow Evaluation UI|MLflow Evaluation]] — The evaluation framework that uses judges
* [[MLflow Trace-based Evaluation|Trace-Based Evaluation]] — Using execution traces for deeper quality analysis
* [[A/B Comparison of Agent Configurations]] — Comparing agent variants with consistent judges
* [[Production Quality Monitoring (MLflow GenAI)|Production Monitoring for GenAI]] — Deploying judges for continuous quality monitoring
* Human Feedback Alignment — Improving judge accuracy with expert annotations

## Sources

* custom-judges-databricks-on-aws.md
* create-a-custom-judge-using-make_judge-databricks-on-aws.md
```

# Citations

1. [custom-judges-databricks-on-aws.md](/references/custom-judges-databricks-on-aws-7a56fe4f.md)
2. [create-a-custom-judge-using-make_judge-databricks-on-aws.md](/references/create-a-custom-judge-using-make_judge-databricks-on-aws-956237bf.md)
