---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: cd367687a58391c8ee2d665f79096d65bda1cf404c1675ae8bdccb760f4823a1
  pageDirectory: concepts
  sources:
    - create-a-custom-judge-using-make_judge-databricks-on-aws.md
  confidence: 1
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - judge-prompt-templates-with-variables
    - JPTWV
  citations:
    - file: create-a-custom-judge-using-make_judge-databricks-on-aws.md
title: Judge Prompt Templates with Variables
description: Custom judge instructions support templated variables such as `{{ inputs }}`, `{{ outputs }}`, `{{ expectations }}`, and `{{ trace }}` that dynamically inject evaluation context from the agent invocation.
tags:
  - mlflow
  - prompt-engineering
  - evaluation
timestamp: "2026-06-18T11:12:29.105Z"
---

# Judge Prompt Templates with Variables

**Judge prompt templates with variables** allow you to create Custom Judges (make_judge)|custom judges using make_judge() that dynamically incorporate evaluation context — such as the conversation history, agent outputs, expected behaviors, or execution traces — directly into the LLM judge’s instructions. These templates are defined as strings passed to the `instructions` parameter of `make_judge()` and use Jinja-like placeholders enclosed in double curly braces. The placeholders are replaced at evaluation time with the actual data for each record. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

## Available Template Variables

The following variables are commonly available in judge prompt templates:

- **`{{ inputs }}`** — The user’s messages or input data passed to the agent. For a conversation, this is typically the list of messages from the user side of the exchange. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]
- **`{{ outputs }}`** — The agent’s responses generated by the model or the function being evaluated. In the evaluation dataset, `outputs` correspond to the agent’s completions. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]
- **`{{ expectations }}`** — Optional expected behaviors supplied in the evaluation dataset under the `expectations` key. The judge can compare the agent’s actual response against these predefined expectations. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]
- **`{{ trace }}`** — When included in the instructions, the judge becomes a *trace-based judge* that gains the ability to autonomously explore the execution trace (for example, tool calls and internal steps taken by the agent). This is useful for validating that the agent performed the correct actions, not just that it produced the right output. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

## Using Variables in Judge Instructions

You embed variables in the `instructions` string using the `{{ variable_name }}` syntax. The following examples, drawn from a customer support agent evaluation, illustrate the pattern:

### Example 1: Issue Resolution Judge (uses `inputs` and `outputs`)

```python
issue_resolution_judge = make_judge(
    name="issue_resolution",
    instructions=(
        "Evaluate if the customer's issue was resolved in the conversation.\n\n"
        "User's messages: {{ inputs }}\n"
        "Agent's responses: {{ outputs }}"
    ),
    feedback_value_type=Literal["fully_resolved", "partially_resolved", "needs_follow_up"],
)
```

The judge receives the user’s messages as `{{ inputs }}` and the agent’s replies as `{{ outputs }}`, and decides a resolution status. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

### Example 2: Expected Behaviors Judge (uses `outputs`, `expectations`, and `inputs`)

```python
expected_behaviors_judge = make_judge(
    name="expected_behaviors",
    instructions=(
        "Compare the agent's response in {{ outputs }} against the expected behaviors in {{ expectations }}.\n\n"
        "User's question: {{ inputs }}"
    ),
    feedback_value_type=Literal["meets_expectations", "partially_meets", "does_not_meet"],
)
```

This judge compares the agent’s output to the `expectations` dictionary defined per evaluation record. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

### Example 3: Trace-Based Judge (uses `trace`)

```python
tool_call_judge = make_judge(
    name="tool_call_correctness",
    instructions=(
        "Analyze the execution {{ trace }} to determine if the agent called appropriate tools for the user's request.\n\n"
        "Examine the trace to:\n"
        "1. Identify what tools were available and their purposes\n"
        "2. Determine which tools were actually called\n"
        "3. Assess whether the tool calls were reasonable for addressing the user's question"
    ),
    feedback_value_type=bool,
    model="databricks:/databricks-gpt-5-mini",
)
```

Because the instructions contain `{{ trace }}`, the judge becomes trace-based and can explore the execution trace (e.g., tool calls) to evaluate correctness. A trace-based judge also requires the `model` parameter to be explicitly set. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

## How Variables Are Evaluated

At evaluation time, the placeholders are replaced with the corresponding fields from the evaluation dataset or from the output of the agent being evaluated. For example:

- `{{ inputs }}` is populated from the `inputs` field of each record in the `data` passed to `mlflow.genai.evaluate()`.
- `{{ outputs }}` is populated from the return value of the `predict_fn` (or from a recorded trace).
- `{{ expectations }}` comes from the optional `expectations` field in the dataset.
- `{{ trace }}` is sourced from the full execution trace recorded by [MLflow Tracing](/concepts/mlflow-tracing.md) during evaluation.

The judge receives the filled-in prompt and uses the specified LLM to produce a [Feedback (MLflow)|Feedback](/concepts/mlflow-feedback-object.md) object with the defined `value` type. ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

## Best Practices

- **Be explicit about what the judge should inspect.** Clearly reference the relevant variables so the LLM judge has the right context.
- **Use `{{ trace }}` only when you need to validate internal agent actions** (e.g., tool calls, API calls). For simple output quality checks, `{{ inputs }}` and `{{ outputs }}` may suffice.
- **Supply `expectations` in your dataset** when you want to compare agent behaviour against a known standard. The `expectations` field should be a dictionary with boolean flags or short descriptions that the judge can reason about.

## Related Concepts

- make_judge() — The API for creating custom judges
- Custom Judges (make_judge)|Custom judges using make_judge() — Walkthrough tutorial with full examples
- [Trace-based Judges](/concepts/trace-based-judges.md) — Judges that use the `{{ trace }}` variable for deeper evaluation
- Feedback (MLflow) — The structure returned by judges
- [mlflow.genai.evaluate()](/concepts/mlflowgenaievaluate.md) — The evaluation function that runs judges
- [Production monitoring for GenAI](/concepts/production-quality-monitoring-mlflow-genai.md) — Deploying judges for continuous evaluation
- Align judges with human feedback — Improving judge accuracy with expert ratings

## Sources

- create-a-custom-judge-using-make_judge-databricks-on-aws.md

# Citations

1. [create-a-custom-judge-using-make_judge-databricks-on-aws.md](/references/create-a-custom-judge-using-make_judge-databricks-on-aws-956237bf.md)
