---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 8dca313f6d17727f852dd48bf92f5d956db79d1c2157cbcf865e890a27bd65f9
  pageDirectory: concepts
  sources:
    - concepts-data-model-databricks-on-aws.md
  confidence: 0.95
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - evaluation-harness-mlflow-genai
    - EH(G
    - Evaluation (MLflow GenAI)
    - Evaluation in Development (MLflow GenAI)
  citations:
    - file: concepts-data-model-databricks-on-aws.md
title: Evaluation Harness (MLflow GenAI)
description: SDK (mlflow.genai.evaluate()) for systematically testing an app version against an evaluation dataset using scorers, producing traces with feedback assessments.
tags:
  - mlflow
  - evaluation
  - testing
timestamp: "2026-06-18T14:41:38.856Z"
---

# Evaluation Harness (MLflow GenAI)

The **Evaluation Harness** in [MLflow GenAI](/concepts/mlflow-3-for-genai.md) is the SDK and framework for systematically evaluating the quality of a GenAI application. It is accessed via `mlflow.genai.evaluate()` and is used to test application versions against curated [Evaluation Datasets](/concepts/evaluation-datasets.md) using a set of [[scorers]].

## Overview

The evaluation harness takes an evaluation dataset, a set of scorers, and your application's prediction function as input. It creates an evaluation run that contains traces with feedback assessments by:

1. Running your app for every record in the evaluation dataset, producing traces.
2. Running each scorer on the resulting traces to assess quality, producing feedback.
3. Attaching each feedback to the appropriate trace.

^[concepts-data-model-databricks-on-aws.md]

## Purpose

The evaluation harness is used to iteratively evaluate potential improvements to your application. It helps developers:

- **Validate** if an improvement improved (or regressed) quality.
- **Identify** additional improvements to further enhance quality.
- **Compare** different versions of your application side-by-side.
- **Track** quality evaluations over time.

^[concepts-data-model-databricks-on-aws.md]

## Key Components

### Evaluation Datasets

[Evaluation Datasets](/concepts/evaluation-datasets.md) are curated collections of test cases for systematically testing your application. They include inputs and optionally expectations (ground truth labels). Datasets are typically created by selecting representative traces from production or development, and they are versioned over time to track how your test suite evolves.

^[concepts-data-model-databricks-on-aws.md]

### Scorers

[[Scorers]] (accessed via `mlflow.genai.scorers.*`) are functions that evaluate a trace's quality. They parse a trace for relevant data fields and use that data to evaluate quality using either deterministic code or [LLM judge](/concepts/llm-judges.md)-based evaluation criteria. Scorers return feedback entities with the results of that evaluation. The same scorer can be used for evaluation in both development and production.

^[concepts-data-model-databricks-on-aws.md]

### Evaluation Runs

An **evaluation run** is the result of testing an application version against an evaluation dataset using a set of scorers. Evaluation runs contain:

- The traces (and their assessments) generated by evaluation.
- Aggregated metrics based on the assessments.

^[concepts-data-model-databricks-on-aws.md]

### Assessments

[Assessments](/concepts/assessments.md) are quality measurements and ground truth labels attached to a trace. There are two types:

- **Feedback**: Judgments about the quality of your app's outputs (e.g., LLM judge assessments, thumbs up/down ratings).
- **Expectations**: Ground truth labels that define the correct output for a given input, added by domain experts.

^[concepts-data-model-databricks-on-aws.md]

## Relationship to Other Evaluation Mechanisms

### Scorers vs. Judges

In MLflow, a **judge** is a callable SDK (like `mlflow.genai.judges.is_correct`) that evaluates text based on specific criteria. However, judges cannot directly process traces — they only understand text inputs. **Scorers** extract the relevant data from a trace (such as the request, response, and retrieved context) and pass it to the judge for evaluation. Scorers act as the "adapter" that connects traces to evaluation logic, whether that is an LLM judge or custom code.

^[concepts-data-model-databricks-on-aws.md]

### Development vs. Production Evaluation

The evaluation harness is designed for **development** evaluation. The equivalent mechanism for **production** is [production monitoring for GenAI](/concepts/production-quality-monitoring-mlflow-genai.md), accessed via `mlflow.genai.Scorer.start()`, which schedules scorers to automatically evaluate traces from your deployed application.

^[concepts-data-model-databricks-on-aws.md]

## Usage Context

The evaluation harness is a core part of the MLflow GenAI development workflow. It enables systematic quality evaluation, helping teams:

- Validate changes before promoting a configuration to production.
- Prevent regressions in quality.
- Compare agent configurations (see [A/B Comparison of Agent Configurations](/concepts/ab-comparison-of-agent-configurations.md)).
- Track quality metrics over the lifecycle of an application.

^[concepts-data-model-databricks-on-aws.md]

## Related Concepts

- [MLflow Experiment](/concepts/mlflow-experiment.md) — The container that organizes all GenAI application data.
- [[Scorers]] — Functions that evaluate trace quality.
- [Evaluation Datasets](/concepts/evaluation-datasets.md) — Curated collections of test cases.
- [Production Monitoring for GenAI](/concepts/production-quality-monitoring-mlflow-genai.md) — The production equivalent for quality evaluation.
- [Custom Judges](/concepts/custom-judges.md) — LLM-based scorers for custom evaluation criteria.
- [Traces](/concepts/traces.md) — Execution logs used as input to evaluation.
- [Assessments](/concepts/assessments.md) — Quality measurements attached to traces.

## Sources

- concepts-data-model-databricks-on-aws.md

# Citations

1. [concepts-data-model-databricks-on-aws.md](/references/concepts-data-model-databricks-on-aws-1534caf0.md)
