---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 13a281381086b058bfd24a92bcdf0491cb7911bf2e1e3cc3fd2bf52dab3efd24
  pageDirectory: concepts
  sources:
    - concepts-data-model-databricks-on-aws.md
  confidence: 0.95
  provenanceState: merged
  freshnessStatus: unverified
  aliases:
    - evaluation-datasets-and-runs
    - Runs and Evaluation Datasets
    - EDAR
  citations:
    - file: concepts-data-model-databricks-on-aws.md
title: Evaluation Datasets and Runs
description: Evaluation datasets are curated test cases (with optional ground-truth expectations) and evaluation runs are the results of testing an app version against them using scorers, producing traces and aggregated metrics.
tags:
  - mlflow
  - evaluation
  - datasets
  - runs
timestamp: "2026-06-19T17:49:41.323Z"
---

---
title: Evaluation Datasets and Runs
summary: Curated collections of test cases (datasets) and the results of testing an application version against them using scorers (runs), used to iteratively measure and compare quality across app versions.
sources:
  - concepts-data-model-databricks-on-aws.md
kind: concept
createdAt: "2026-06-19T09:21:18.043Z"
updatedAt: "2026-06-19T09:21:18.043Z"
tags:
  - mlflow
  - evaluation
  - testing
aliases:
  - evaluation-datasets-and-runs
  - Runs and Evaluation Datasets
  - EDAR
confidence: 0.95
provenanceState: merged
inferredParagraphs: 0
---

# Evaluation Datasets and Runs

**Evaluation datasets** and **evaluation runs** are core components of the MLflow GenAI data model used for systematically testing and measuring the quality of generative AI applications. Together, they form the backbone of iterative quality improvement during development.^[concepts-data-model-databricks-on-aws.md]

## Evaluation Datasets

Evaluation datasets are curated collections of test cases for systematically testing an application. Each entry in an evaluation dataset contains `inputs` (such as conversation history or user queries) and optionally `expectations` (ground truth labels that define the correct output).^[concepts-data-model-databricks-on-aws.md]

Evaluation datasets are typically created by selecting representative traces from production or development logs. They are versioned over time to track how the test suite evolves as the application changes.^[concepts-data-model-databricks-on-aws.md]

The primary use cases for evaluation datasets are:
- Iteratively evaluating and improving application quality.
- Validating changes to prevent regressions in quality.^[concepts-data-model-databricks-on-aws.md]

## Evaluation Runs

Evaluation runs are the results of testing an application version against an evaluation dataset using a set of [[scorers]]. Each evaluation run contains:
- The traces (and their assessments) generated by running the application on the dataset.
- Aggregated metrics based on the assessments from those traces.^[concepts-data-model-databricks-on-aws.md]

Evaluation runs serve three main purposes:
- Determining whether application changes improved or regressed quality.
- Comparing versions of an application side-by-side.
- Tracking quality evaluations over time.^[concepts-data-model-databricks-on-aws.md]

## Evaluation in Development

The primary SDK for running evaluations is `mlflow.genai.evaluate()`, which takes three inputs:
1. An **evaluation dataset** containing test cases.
2. A set of **scorers** that analyze quality.
3. The application's **prediction function** to be tested.

The evaluation harness processes each record in the dataset by:
1. Running the application on every record to produce [Traces](/concepts/traces.md).
2. Running each scorer on the resulting traces to assess quality, producing feedback [Assessments](/concepts/assessments.md).
3. Attaching each feedback assessment to the appropriate trace.

The result is an evaluation run containing traces with quality assessments and aggregated metrics.^[concepts-data-model-databricks-on-aws.md]

## Evaluation in Production

The same scorers used in development evaluation can be scheduled for production monitoring using `mlflow.genai.Scorer.start()`. Production monitoring runs scorers automatically on traces from deployed applications, attaching feedback to source traces. This enables continuous quality detection and identification of problematic queries or use cases to improve in development.^[concepts-data-model-databricks-on-aws.md]

## Relationship to Other Components

Evaluation datasets and runs are closely related to:
- [[Scorers]] — Functions that evaluate trace quality, producing feedback assessments.
- [Traces](/concepts/traces.md) — Execution logs generated during evaluation.
- [Assessments](/concepts/assessments.md) — Quality measurements attached to traces, including feedback and expectations.
- [Labeling Sessions](/concepts/labeling-sessions.md) — Human review queues that can create ground truth data for evaluation datasets.^[concepts-data-model-databricks-on-aws.md]

## Sources

- concepts-data-model-databricks-on-aws.md

# Citations

1. [concepts-data-model-databricks-on-aws.md](/references/concepts-data-model-databricks-on-aws-1534caf0.md)
