---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 0a8aa25957e2f7a74e863fc43b1cbebff730b2c1200150c2fb05e5b80827745c
  pageDirectory: concepts
  sources:
    - custom-judges-databricks-on-aws.md
  confidence: 0.98
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - judge-prompt-template-variables
    - JPTV
  citations:
    - file: custom-judges-databricks-on-aws.md
title: Judge Prompt Template Variables
description: A restricted set of four template variables ({{ inputs }}, {{ outputs }}, {{ expectations }}, {{ trace }}) that can be used in custom judge prompts, with validation to prevent injection of arbitrary variables.
tags:
  - mlflow
  - prompt-engineering
  - security
timestamp: "2026-06-19T14:39:30.000Z"
---

# Judge Prompt Template Variables

**Judge Prompt Template Variables** are placeholders used within custom LLM judge prompts to reference data from a GenAI agent's execution. When creating a judge with the [`make_judge()`](https://mlflow.org/docs/3.4.0/api_reference/python_api/mlflow.genai.html#mlflow.genai.judges.make_judge) function, these variables allow the judge to access the agent's inputs, outputs, expected outputs, and execution traces, enabling flexible and precise evaluation criteria. ^[custom-judges-databricks-on-aws.md]

## Available Template Variables

`make_judge()` accepts exactly four built-in template variables. At least one must appear in the instructions for the judge to function. Custom variables such as `{{ question }}` are not allowed and will trigger validation errors. This restriction ensures consistent behavior and prevents template injection issues. ^[custom-judges-databricks-on-aws.md]

| Variable | Description |
|----------|-------------|
| `{{ inputs }}` | Input data provided to the agent |
| `{{ outputs }}` | Output data generated by the agent |
| `{{ expectations }}` | Ground truths or expected outcomes |
| `{{ trace }}` | The complete execution trace of the agent |

### Usage Example

```python
from mlflow.genai.judges import make_judge
from typing import Literal

judge = make_judge(
    name="tool_usage_validator",
    instructions=(
        "Analyze the {{ trace }} to verify correct tool usage.\n\n"
        "Check that the agent selected appropriate tools for the user's request "
        "and called them with correct parameters."
    ),
    feedback_value_type=Literal["correct", "incorrect"],
    model="databricks:/databricks-gpt-5-mini"  # Required for trace-based judges
)
```

^[custom-judges-databricks-on-aws.md]

## Template Variables in Trace-Based Judges

When the `{{ trace }}` variable is included, the judge becomes a **trace-based judge**. Trace-based judges analyze the full execution trace of an agent call, including tool calls, intermediate reasoning, and results. They can validate tool usage patterns, identify performance bottlenecks, investigate execution failures, and verify multi-step workflows. ^[custom-judges-databricks-on-aws.md]

For trace-based judges, the `model` argument must be specified in `make_judge()` because the judge requires a model capable of performing trace analysis. The model can be served by Foundation Model APIs, External Model Serving Endpoints, or Custom Model Serving Endpoints. ^[custom-judges-databricks-on-aws.md]

## Best Practices

When writing judge instructions that use template variables:

- Be specific about the expected output format (categorical, boolean, or numeric).
- Break down complex evaluations into clear sections: what to evaluate, what information to examine, how to make the judgment, and what format to return.
- Include the minimum necessary template variables to keep prompts focused and efficient.

## Related Concepts

- [Custom Judges](/concepts/custom-judges.md) — Overview of custom LLM judges created with `make_judge()`.
- make_judge() — API for creating custom judges with template variables.
- [MLflow GenAI Evaluation](/concepts/mlflow-genai-evaluation.md) — Framework for evaluating generative AI applications.
- [Trace-based evaluation](/concepts/mlflow-trace-based-evaluation.md) — Evaluation that uses execution traces for deeper analysis.
- Align judges with human feedback — Process of refining judge prompts using expert annotations.

## Sources

- custom-judges-databricks-on-aws.md

# Citations

1. [custom-judges-databricks-on-aws.md](/references/custom-judges-databricks-on-aws-7a56fe4f.md)
