---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: cb3991379c1d42e667af90be91c288d013fcf88ebd86619c7900b983bc4b8d55
  pageDirectory: concepts
  sources:
    - custom-judges-databricks-on-aws.md
  confidence: 0.9
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - judge-instruction-best-practices
    - JIBP
  citations:
    - file: custom-judges-databricks-on-aws.md
title: Judge Instruction Best Practices
description: Guidelines for writing effective judge instructions including specificity about output formats (categorical, boolean, numeric) and breaking down complex evaluations.
tags:
  - llm-evaluation
  - best-practices
  - prompt-engineering
timestamp: "2026-06-18T11:25:55.548Z"
---

# Judge Instruction Best Practices

**Judge Instruction Best Practices** are guidelines for writing effective natural-language instructions that [Custom Judges](/concepts/custom-judges.md) use to evaluate GenAI agent outputs. Well-crafted instructions improve the reliability, consistency, and interpretability of automated quality assessments by [MLflow GenAI](/concepts/mlflow-3-for-genai.md).

## Overview

When you create a judge with [`make_judge()`](https://mlflow.org/docs/3.4.0/api_reference/python_api/mlflow.genai.html#mlflow.genai.judges.make_judge), you provide a prompt that defines how the judge should score an agent’s outputs. The instructions should be precise, structured, and aligned with the evaluation goals. The following practices apply to both standard and [Trace-based Judges](/concepts/trace-based-judges.md). ^[custom-judges-databricks-on-aws.md]

## Template Variables

The judge prompt must include at least one of these template variables to access runtime data. Using the correct variable avoids validation errors and ensures the judge sees the information it needs: ^[custom-judges-databricks-on-aws.md]

| Variable | Content |
|----------|---------|
| `{{ inputs }}` | Input data provided to the agent |
| `{{ outputs }}` | Output data generated by the agent |
| `{{ expectations }}` | Ground truths or expected outcomes |
| `{{ trace }}` | Complete execution trace of the agent |

Only these four variables are allowed. Custom variables such as `{{ question }}` will cause validation errors. ^[custom-judges-databricks-on-aws.md]

## Specify the Output Format

Instructions must clearly define the format the judge should return. Ambiguous instructions lead to inconsistent scoring. Choose one of the following patterns: ^[custom-judges-databricks-on-aws.md]

- **Categorical responses**: List explicit values (e.g., `'fully_resolved'`, `'partially_resolved'`, `'needs_follow_up'`).
- **Boolean responses**: State that the judge must return `true` or `false`.
- **Numeric scores**: Define a scoring range and explain what each score means.

Example:

> Return one of: `'fully_resolved'`, `'partially_resolved'`, or `'needs_follow_up'`.

## Break Down Complex Evaluations

For multi‑faceted evaluation tasks, structure the instructions into clear sections. A well‑organized prompt helps the judge reason about each dimension separately: ^[custom-judges-databricks-on-aws.md]

1. **What to evaluate** – Describe the quality dimension (e.g., issue resolution, tone, tool usage).
2. **What information to examine** – Name the template variables the judge should inspect.
3. **How to make the judgment** – Provide decision criteria (e.g., “If the agent solved the problem in the first response, return `'fully_resolved'`”).
4. **What format to return** – Restate the expected output format from the previous section.

## Trace-Based Judge Considerations

Trace-based judges analyze the full execution trace of an agent call, including tool invocations and intermediate reasoning steps. When using the `{{ trace }}` variable: ^[custom-judges-databricks-on-aws.md]

- **Specify a model** – The `model` argument is required for trace-based judges (e.g., `model="databricks:/databricks-gpt-5-mini"`).
- **Focus instructions on trace analysis** – Instruct the judge to examine tool calls, parameter correctness, workflow steps, and failures.
- **Use recommended models** – For best results, use models served by Foundation Model APIs, such as `databricks:/databricks-gpt-5-mini` or `databricks:/databricks-claude-opus-4-5`.

Example instruction for a tool‑usage validator:

> Analyze the {{ trace }} to verify correct tool usage. Check that the agent selected appropriate tools for the user's request and called them with correct parameters.

## Align with Human Feedback

The initial judge instructions are a starting point. As you collect expert annotations on agent outputs, you can [align judges with humans|align the judge instructions](/concepts/aligning-judges-with-human-experts.md) to better match human quality assessments. This iterative tuning improves judge accuracy over time. ^[custom-judges-databricks-on-aws.md]

## Summary

- Use only the four built‑in template variables: `{{ inputs }}`, `{{ outputs }}`, `{{ expectations }}`, `{{ trace }}`.
- Define explicit output formats (categorical, boolean, or numeric).
- Structure complex evaluations into logical sections.
- For trace‑based judges, set a capable model and focus instructions on trace analysis.
- Continuously refine instructions based on human expert feedback.

## Related Concepts

- [Custom Judges](/concepts/custom-judges.md)
- make_judge()|Make Judge API
- [Built-in LLM Judges](/concepts/built-in-llm-judges.md)
- [Trace-Based Evaluation](/concepts/mlflow-trace-based-evaluation.md)
- [Align Judges with Humans](/concepts/aligning-judges-with-human-experts.md)
- GenAI Agent Evaluation
- [MLflow GenAI](/concepts/mlflow-3-for-genai.md)

## Sources

- custom-judges-databricks-on-aws.md

# Citations

1. [custom-judges-databricks-on-aws.md](/references/custom-judges-databricks-on-aws-7a56fe4f.md)
