---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: e49ae6f193f8be39f8332d02f14d162551d6afad2d14b3838bb35a3a438fa138
  pageDirectory: concepts
  sources:
    - answer-and-context-relevance-judges-databricks-on-aws.md
  confidence: 0.95
  provenanceState: merged
  freshnessStatus: unverified
  aliases:
    - llm-as-a-judge-for-relevance
    - LFR
  citations:
    - file: answer-and-context-relevance-judges-databricks-on-aws.md
title: LLM-as-a-Judge for Relevance
description: The pattern of using a configurable LLM (defaulting to a Databricks-hosted model) to automatically score the relevance of responses and retrieved documents in GenAI applications.
tags:
  - llm-evaluation
  - genai
  - evaluation-patterns
timestamp: "2026-06-19T17:34:11.591Z"
---

# LLM-as-a-Judge for Relevance

**LLM-as-a-Judge for Relevance** is a quality assessment technique that uses a large language model (LLM) as an automated evaluator to determine whether content generated by a GenAI application is contextually relevant to the user's original query. This approach is a key component of the [LLM-as-a-Judge](/concepts/llm-as-a-judge.md) evaluation paradigm and is particularly important for diagnosing and improving [Retrieval Augmented Generation (RAG)](/concepts/retrieval-augmented-generation-rag.md) systems.

## Overview

MLflow provides two built-in LLM judges specifically designed to assess relevance in GenAI applications. These judges help diagnose quality issues by verifying that the application's output or retrieved context directly addresses the user's input. If the context is not relevant, the generation step cannot produce a helpful response. ^[answer-and-context-relevance-judges-databricks-on-aws.md]

The two built-in relevance judges are:

- **`RelevanceToQuery`**: Evaluates whether your application's response directly addresses the user's input without deviating into unrelated topics. ^[answer-and-context-relevance-judges-databricks-on-aws.md]
- **`RetrievalRelevance`**: Evaluates whether each document returned by your application's retriever(s) is relevant to the input request. ^[answer-and-context-relevance-judges-databricks-on-aws.md]

## Prerequisites

To use the built-in relevance judges, you must:

1. Install MLflow and required packages (`mlflow[databricks]>=3.4.0`, `openai`, `databricks-connect>=16.1`). ^[answer-and-context-relevance-judges-databricks-on-aws.md]
2. Create an MLflow experiment by following the environment setup quickstart. ^[answer-and-context-relevance-judges-databricks-on-aws.md]

## `RelevanceToQuery` Judge

This scorer evaluates if your application's response directly addresses the user's input and stays on topic. You can invoke the scorer directly for testing a single input, or pass it to `mlflow.genai.evaluate` for running full evaluation on a dataset. ^[answer-and-context-relevance-judges-databricks-on-aws.md]

**Requirements**: The Trace's root span must contain `inputs` and `outputs`. ^[answer-and-context-relevance-judges-databricks-on-aws.md]

```python
import mlflow
from mlflow.genai.scorers import RelevanceToQuery

assessment = RelevanceToQuery(name="my_relevance_to_query")(
    inputs={"question": "What is the capital of France?"},
    outputs="The capital of France is Paris.",
)
print(assessment)
```

## `RetrievalRelevance` Judge

This scorer evaluates whether each document returned by your application's retriever(s) is relevant to the input request. It is essential for diagnosing [RAG quality evaluation](/concepts/evaluation-run.md) issues — if retrieved context is irrelevant, the generation step cannot produce a helpful response. ^[answer-and-context-relevance-judges-databricks-on-aws.md]

**Requirements**: The MLflow Trace must contain at least one span with `span_type` set to `RETRIEVER`. ^[answer-and-context-relevance-judges-databricks-on-aws.md]

```python
from mlflow.genai.scorers import retrieval_relevance
import mlflow

trace = mlflow.get_trace("<your-trace-id>")
feedbacks = retrieval_relevance(trace=trace)
print(feedbacks)
```

### RAG Example

A complete RAG evaluation example shows creating a retriever function with a `RETRIEVER` span type, defining an application that uses the retriever, and running evaluation with the `RetrievalRelevance` scorer: ^[answer-and-context-relevance-judges-databricks-on-aws.md]

```python
import mlflow
from mlflow.genai.scorers import RetrievalRelevance
from mlflow.entities import Document
from typing import List

@mlflow.trace(span_type="RETRIEVER")
def retrieve_docs(query: str) -> List[Document]:
    if "capital" in query.lower() and "france" in query.lower():
        return [
            Document(id="doc_1", page_content="Paris is the capital of France.", metadata={"source": "geography.txt"}),
            Document(id="doc_2", page_content="The Eiffel Tower is located in Paris.", metadata={"source": "landmarks.txt"})
        ]
    else:
        return [Document(id="doc_3", page_content="Python is a programming language.", metadata={"source": "tech.txt"})]

@mlflow.trace
def rag_app(query: str):
    docs = retrieve_docs(query)
    return {"response": f"Found {len(docs)} relevant documents."}

eval_dataset = [
    {"inputs": {"query": "What is the capital of France?"}},
    {"inputs": {"query": "How do I use Python?"}}
]

eval_results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=rag_app,
    scorers=[
        RetrievalRelevance(
            model="databricks:/databricks-gpt-oss-120b",
        )
    ]
)
```

## Selecting the Judge LLM

By default, built-in judges use a Databricks-hosted LLM designed specifically for GenAI quality assessments. You can change the judge model by using the `model` argument when creating the judge. The model must be specified in the format `<provider>:/<model-name>`, where `<provider>` is a LiteLLM-compatible model provider. When using `databricks` as the provider, the model name is the same as the serving endpoint name. ^[answer-and-context-relevance-judges-databricks-on-aws.md]

```python
from mlflow.genai.scorers import RelevanceToQuery, RetrievalRelevance

relevance_judge = RelevanceToQuery(model="databricks:/databricks-gpt-5-mini")
retrieval_judge = RetrievalRelevance(model="databricks:/databricks-claude-opus-4-5")

eval_results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=rag_app,
    scorers=[relevance_judge, retrieval_judge]
)
```

## Interpreting Results

The judge returns a `Feedback` object with two key fields:

- **`value`**: "yes" if the context is relevant, "no" if it is not. ^[answer-and-context-relevance-judges-databricks-on-aws.md]
- **`rationale`**: An explanation of why the judge found the context relevant or irrelevant. ^[answer-and-context-relevance-judges-databricks-on-aws.md]

## Related Concepts

- [Groundedness LLM Judge](/concepts/retrievalgroundedness-judge.md) — Another built-in judge for evaluating factual accuracy
- [Safety LLM Judge](/concepts/safety-judge-scorer.md) — Built-in judge for evaluating content safety
- [Correctness LLM Judge](/concepts/correctness-judge.md) — Built-in judge for evaluating answer correctness
- [Custom LLM Judges](/concepts/custom-llm-judges.md) — How to build specialized judges for custom use cases
- [MLflow Evaluation](/concepts/mlflow-evaluation-ui.md) — The evaluation framework that uses LLM judges
- RAG Quality Evaluation — Comprehensive evaluation of RAG applications

## Sources

- answer-and-context-relevance-judges-databricks-on-aws.md

# Citations

1. [answer-and-context-relevance-judges-databricks-on-aws.md](/references/answer-and-context-relevance-judges-databricks-on-aws-5f3620ab.md)
