---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 4ab8e675aacc5f0a97d687750db3fa3b1d47838b19cc701275d11dedc2e85ff8
  pageDirectory: concepts
  sources:
    - create-a-custom-judge-using-make_judge-databricks-on-aws.md
    - custom-judges-databricks-on-aws.md
  confidence: 0.95
  provenanceState: merged
  freshnessStatus: unverified
  aliases:
    - trace-based-judges
    - Trace-Based Judge
    - Trace-based judge
  citations:
    - file: custom-judges-databricks-on-aws.md
    - file: create-a-custom-judge-using-make_judge-databricks-on-aws.md
title: Trace-based Judges
description: A type of custom judge that analyzes execution traces to validate tool calls, enabled by including {{ trace }} in judge instructions.
tags:
  - MLflow
  - GenAI
  - evaluation
  - tracing
timestamp: "2026-06-19T17:54:35.833Z"
---

# Trace-based Judges

**Trace-based judges** are a type of custom [LLM judge](/concepts/llm-judges.md) that evaluate GenAI agents by analyzing the complete execution trace of an agent’s run. Unlike standard judges that only examine inputs and outputs, trace-based judges autonomously explore execution traces using Model Context Protocol (MCP) tools to understand what happened during agent execution. ^[custom-judges-databricks-on-aws.md, create-a-custom-judge-using-make_judge-databricks-on-aws.md]

## Creating a Trace-based Judge

To create a trace-based judge, use `make_judge()` from `mlflow.genai.judges` and include `{{ trace }}` in the instructions. The `model` argument is required for trace-based judges to analyze the full trace. ^[custom-judges-databricks-on-aws.md, create-a-custom-judge-using-make_judge-databricks-on-aws.md]

```python
from mlflow.genai.judges import make_judge
from typing import Literal

tool_usage_judge = make_judge(
    name="tool_usage_validator",
    instructions=(
        "Analyze the {{ trace }} to verify correct tool usage.\n\n"
        "Check that the agent selected appropriate tools for the user's request "
        "and called them with correct parameters."
    ),
    feedback_value_type=Literal["correct", "incorrect"],
    model="databricks:/databricks-gpt-5-mini"  # Required for trace-based judges
)
```

## Template Variables

Your instructions must include at least one template variable. The available variables are: ^[custom-judges-databricks-on-aws.md]

- `{{ inputs }}` — Input data provided to the agent
- `{{ outputs }}` — Output data generated by your agent
- `{{ expectations }}` — Ground truths or expected outcomes
- `{{ trace }}` — The complete execution trace of your agent

These are the only allowed variables. Custom variables like `{{ question }}` will throw validation errors to ensure consistent behavior and prevent template injection issues. ^[custom-judges-databricks-on-aws.md]

## Capabilities

Trace-based judges can autonomously explore traces to: ^[custom-judges-databricks-on-aws.md]

- **Validate tool usage patterns** — Determine if the agent selected appropriate tools for the user’s request.
- **Identify performance bottlenecks** — Detect slow or inefficient execution paths.
- **Investigate execution failures** — Understand why certain steps failed.
- **Verify multi-step workflows** — Confirm that complex sequences of operations were executed correctly.

## Model Requirements

Trace-based judges require a model capable of trace analysis. The model can be served by: ^[custom-judges-databricks-on-aws.md]

- [Foundation Model APIs](/concepts/foundation-model-apis.md) (recommended)
- External Model Serving Endpoints
- [Custom Model Serving Endpoints](/concepts/custom-model-serving-endpoint-support.md)

Recommended models include: ^[custom-judges-databricks-on-aws.md]

- `databricks:/databricks-gpt-5-mini`
- `databricks:/databricks-gpt-5`
- `databricks:/databricks-gpt-oss-120b`
- `databricks:/databricks-claude-opus-4-5`

## Example: Validating Tool Calls

The following example shows a trace-based judge that validates whether appropriate tools were called during agent execution: ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

```python
from mlflow.genai.judges import make_judge

tool_call_judge = make_judge(
    name="tool_call_correctness",
    instructions=(
        "Analyze the execution {{ trace }} to determine if the agent called "
        "appropriate tools for the user's request.\n\n"
        "Examine the trace to:\n"
        "1. Identify what tools were available and their purposes\n"
        "2. Determine which tools were actually called\n"
        "3. Assess whether the tool calls were reasonable for addressing "
        "the user's question"
    ),
    feedback_value_type=bool,
    model="databricks:/databricks-gpt-5-mini",
)
```

## Using Trace-based Judges in Evaluation

Trace-based judges can be used alongside other judges in `mlflow.genai.evaluate()` to evaluate different aspects of an agent: ^[create-a-custom-judge-using-make_judge-databricks-on-aws.md]

```python
result = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=customer_support_agent,
    scorers=[
        issue_resolution_judge,      # Standard judge (inputs/outputs)
        expected_behaviors_judge,    # Standard judge (expectations)
        tool_call_judge,             # Trace-based judge (analyzes traces)
    ],
)
```

## Best Practices for Writing Instructions

When creating trace-based judges, follow these guidelines: ^[custom-judges-databricks-on-aws.md]

- **Be specific about expected output format** — Clearly specify whether the judge should return categorical responses (e.g., `'fully_resolved'`, `'partially_resolved'`, `'needs_follow_up'`), boolean responses (`true`/`false`), or numeric scores with defined ranges.
- **Break down complex evaluations** — Structure instructions into clear sections: what to evaluate, what information to examine, how to make the judgment, and what format to return.

## Aligning Trace-based Judges

The base judge is a starting point. As you gather expert feedback on your application’s outputs, you can align the LLM judges to the feedback to further improve judge accuracy. See [Align judges with human experts](/concepts/aligning-judges-with-human-experts.md). ^[custom-judges-databricks-on-aws.md]

## Related Concepts

- [Custom Judges](/concepts/custom-judges.md) — Overview of creating custom LLM judges.
- [Built-in LLM Judges](/concepts/built-in-llm-judges.md) — Pre-defined judges for common quality dimensions.
- [MLflow evaluation](/concepts/mlflow-evaluation-ui.md) — The `mlflow.genai.evaluate()` API for offline assessment.
- [Production monitoring for GenAI](/concepts/production-quality-monitoring-mlflow-genai.md) — Deploying judges for continuous quality monitoring.
- Feedback (MLflow) — The structured output object returned by judges.
- [Code-based Scorers](/concepts/code-based-scorers.md) — Alternative approach for custom evaluation logic.

## Sources

- custom-judges-databricks-on-aws.md
- create-a-custom-judge-using-make_judge-databricks-on-aws.md

# Citations

1. [custom-judges-databricks-on-aws.md](/references/custom-judges-databricks-on-aws-7a56fe4f.md)
2. [create-a-custom-judge-using-make_judge-databricks-on-aws.md](/references/create-a-custom-judge-using-make_judge-databricks-on-aws-956237bf.md)
