---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: d7d34e594e3abfeee7872e3861237029c611d1c3d638e853bf26faba35663cf0
  pageDirectory: concepts
  sources:
    - code-based-scorer-examples-databricks-on-aws.md
  confidence: 0.95
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - llm-as-a-judge-in-mlflow-evaluation
    - LIME
  citations:
    - file: code-based-scorer-examples-databricks-on-aws.md
title: LLM-as-a-judge in MLflow evaluation
description: Pattern where a scorer integrates a custom or externally hosted LLM to evaluate response quality, using structured prompts and parsing the LLM's JSON output into Feedback objects with AssessmentSource metadata.
tags:
  - mlflow
  - llm-judge
  - evaluation
timestamp: "2026-06-19T17:43:54.464Z"
---

# LLM-as-a-judge in MLflow Evaluation

**LLM-as-a-judge in MLflow evaluation** is a semantic evaluation approach that uses a large language model (LLM) to assess the quality of outputs generated by another AI model or application. This method enables the evaluation of subjective qualities — such as relevance, safety, and adherence to guidelines — that are difficult to capture with code-based metrics.

## Overview

In MLflow Evaluation for GenAI, LLM-as-a-judge evaluation works by configuring a judge LLM to score responses according to predefined criteria. The judge LLM receives the original user query, the AI application's response, and evaluation instructions, and produces a structured assessment — typically including a score and a rationale. ^[code-based-scorer-examples-databricks-on-aws.md]

This approach can be simpler to define than code-based scorers for many semantic evaluation tasks. ^[code-based-scorer-examples-databricks-on-aws.md]

## Built-in LLM Judges

MLflow provides several predefined LLM judges that can be used directly or wrapped in custom scorers. These include:

- **is_context_relevant** – Evaluates whether the assistant's response is relevant to the user's query. ^[code-based-scorer-examples-databricks-on-aws.md]
- **Safety** – Assesses safety concerns in the model's responses. ^[code-based-scorer-examples-databricks-on-aws.md]
- **Guidelines** – Checks whether responses follow specified content policies or guidelines. ^[code-based-scorer-examples-databricks-on-aws.md]

## Custom LLM Judge Scorers

You can integrate your own LLM as a judge within a custom [code-based scorer](/concepts/code-based-scorers.md). This gives full control over the judging process, including prompt design, model selection, and output parsing. ^[code-based-scorer-examples-databricks-on-aws.md]

### Key Components

When building a custom LLM judge scorer, you typically:

1. **Define evaluation prompts** – A system prompt sets the judge's role and output format (e.g., a JSON object with `score` and `rationale`). A user prompt includes the query and response to evaluate. ^[code-based-scorer-examples-databricks-on-aws.md]
2. **Call the judge LLM** – Use an API client (e.g., OpenAI or Databricks-hosted models) to send the prompts to the judge model. ^[code-based-scorer-examples-databricks-on-aws.md]
3. **Parse the response** – Extract structured feedback from the LLM's output. ^[code-based-scorer-examples-databricks-on-aws.md]
4. **Return a [Feedback](/concepts/feedback-object.md) object** – Optionally set the `source` field to `AssessmentSourceType.LLM_JUDGE` to indicate the assessment came from an LLM judge. ^[code-based-scorer-examples-databricks-on-aws.md]

### Example

A custom LLM judge can evaluate answer quality by sending the user query and the AI's response to a judge LLM, asking it to produce a score from 1 to 5 with a rationale. The judge's output is parsed as JSON and returned as a `Feedback` object with the appropriate source type. ^[code-based-scorer-examples-databricks-on-aws.md]

## Guidelines Judge

The `Guidelines` judge is a predefined judge that evaluates whether responses adhere to specified content policies. It can be used directly or wrapped in custom scorers with conditional logic — for example, applying different guidelines based on user tier (premium vs. standard). ^[code-based-scorer-examples-databricks-on-aws.md]

## Wrapping Predefined Judges

Custom scorers can wrap predefined LLM judges to preprocess trace data or post-process feedback. For instance, a scorer can extract the last user message from the `inputs` dictionary before passing it to the `is_context_relevant` judge. ^[code-based-scorer-examples-databricks-on-aws.md]

## Best Practices

- **Use deterministic settings** – Set `temperature=0.0` for more consistent judge outputs. ^[code-based-scorer-examples-databricks-on-aws.md]
- **Provide clear output formats** – Instruct the judge LLM to output structured data (e.g., JSON) for reliable parsing. ^[code-based-scorer-examples-databricks-on-aws.md]
- **Set appropriate token limits** – Configure `max_tokens` to allow sufficient space for the judge's rationale. ^[code-based-scorer-examples-databricks-on-aws.md]
- **Handle errors gracefully** – Design scorers to handle scenarios where the judge fails or returns unexpected output. ^[code-based-scorer-examples-databricks-on-aws.md]

## Workflows

LLM-as-a-judge evaluation supports both offline evaluation (using `mlflow.genai.evaluate()`) and production monitoring. For production monitoring, scorers must use the `@scorer` decorator rather than class-based definitions. ^[code-based-scorer-examples-databricks-on-aws.md]

Results can be chained — for example, identify problematic traces with a `Safety` judge, then analyze the failed subset with a more specific `Guidelines` judge. ^[code-based-scorer-examples-databricks-on-aws.md]

## Related Concepts

- MLflow Evaluation for GenAI – The overall evaluation framework that LLM-as-a-judge is part of.
- [Code-based Scorers](/concepts/code-based-scorers.md) – Alternative evaluation approach using Python code rather than LLMs.
- [Custom LLM scorers](/concepts/custom-judge-scorers.md) – Documentation for defining semantic evaluation metrics.
- [Production Monitoring](/concepts/production-monitoring.md) – Deploying scorers for continuous monitoring of live traffic.
- [[MLflow Trace|MLflow Traces]] – Trace data that can be passed to LLM judges for fine-grained assessment.
- [Feedback objects](/concepts/feedback-objects.md) – The data structure returned by LLM judges to capture scores and rationales.

## Sources

- code-based-scorer-examples-databricks-on-aws.md

# Citations

1. [code-based-scorer-examples-databricks-on-aws.md](/references/code-based-scorer-examples-databricks-on-aws-de913078.md)
