---
title: Custom judges | Databricks on AWS
source: https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/custom-judge/
ingestedAt: "2026-06-18T08:15:11.353Z"
---

Custom LLM judges let you define complex and nuanced scoring guidelines for GenAI applications using natural language.

While MLflow [built-in LLM judges](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/concepts/scorers#built-in-judges) offer excellent starting points for common quality dimensions, custom judges created using `make_judge()` give you full control over evaluation criteria.

## Prompts and template variables[​](#prompts-and-template-variables "Direct link to Prompts and template variables")

To create a judge, you provide a prompt with natural language instructions on how to assess the quality of your agent. [`make_judge()`](https://mlflow.org/docs/3.4.0/api_reference/python_api/mlflow.genai.html#mlflow.genai.judges.make_judge) accepts template variables to access the agent's inputs, outputs, expected outputs or behaviors, and even complete traces.

Your instructions must include at least one template variable, but you don't need to use all of them.

*   `{{ inputs }}` - Input data provided to the agent
*   `{{ outputs }}` - Output data generated by your agent
*   `{{ expectations }}` - Ground truths or expected outcomes
*   `{{ trace }}` - The complete execution trace of your agent

These are the only variables allowed. Custom variables like `{{ question }}` will throw validation errors in order to ensure consistent behavior and prevent template injection issues.

## Trace-based judges[​](#trace-based-judges "Direct link to Trace-based judges")

Trace-based judges analyze execution traces to understand what happened during agent execution. They autonomously explore traces using Model Context Protocol (MCP) tools and can:

*   Validate tool usage patterns
*   Identify performance bottlenecks
*   Investigate execution failures
*   Verify multi-step workflows

The following example defines a judge that assesses tool calling correctness by analyzing traces:

Python

    from mlflow.genai.judges import make_judgefrom typing import Literal# Agent judge for tool calling correctnesstool_usage_judge = make_judge(    name="tool_usage_validator",    instructions=(        "Analyze the {{ trace }} to verify correct tool usage.\n\n"        "Check that the agent selected appropriate tools for the user's request "        "and called them with correct parameters."    ),    feedback_value_type=Literal["correct", "incorrect"],    model="databricks:/databricks-gpt-5-mini"  # Required for trace-based judges)

For trace-based judges to analyze the full trace, the `model` argument must be specified in `make_judge()`.

For a complete tutorial, see [Create a custom judge using `make_judge()`](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/custom-judge/create-custom-judge).

### Model requirements for trace-based judges[​](#model-requirements-for-trace-based-judges "Direct link to Model requirements for trace-based judges")

Trace-based judges require a model capable of trace analysis. The model can be served by:

*   [Foundation Model APIs](https://docs.databricks.com/en/machine-learning/foundation-models/index.html) (recommended)
*   [External Model Serving Endpoints](https://docs.databricks.com/en/generative-ai/external-models/index.html)
*   [Custom Model Serving Endpoints](https://docs.databricks.com/en/machine-learning/model-serving/index.html)

Recommended models:

*   `databricks:/databricks-gpt-5-mini`
*   `databricks:/databricks-gpt-5`
*   `databricks:/databricks-gpt-oss-120b`
*   `databricks:/databricks-claude-opus-4-5`

## Best practices for writing judge instructions[​](#best-practices-for-writing-judge-instructions "Direct link to Best practices for writing judge instructions")

Be specific about expected output format. Your instructions should clearly specify what format the judge should return:

*   **Categorical responses**: List specific values (for example, 'fully\_resolved', 'partially\_resolved', 'needs\_follow\_up')
*   **Boolean responses**: Explicitly state the judge should return `true` or `false`
*   **Numeric scores**: Specify the scoring range and what each score means

Break down complex evaluations. For complex evaluation tasks, structure your instructions into clear sections:

*   What to evaluate
*   What information to examine
*   How to make the judgment
*   What format to return

## Align judges with human experts[​](#align-judges-with-human-experts "Direct link to Align judges with human experts")

The base judge is a starting point. As you gather expert feedback on your application's outputs, you can align the LLM judges to the feedback to further improve judge accuracy. See [Align judges with humans](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/align-judges).

## Next steps[​](#next-steps "Direct link to Next steps")

See [Create a custom judge](https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/custom-judge/create-custom-judge) for a hands-on tutorial that demonstrates both standard and trace-based judges.
