---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 92310bcc740c3c3c6866245a58e56d4fedd817e7b4895a6f99dba07905160d7b
  pageDirectory: concepts
  sources:
    - trace-agents-deployed-on-databricks-databricks-on-aws.md
  confidence: 1
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - mlflow-experiment-as-trace-storage-backend
    - MEATSB
    - MLflow experiment trace storage
  citations:
    - file: trace-agents-deployed-on-databricks-databricks-on-aws.md
title: MLflow Experiment as Trace Storage Backend
description: MLflow experiments serve as the primary storage location for production traces, with optional custom artifact locations (e.g., Unity Catalog volumes) enabling Unity Catalog volume privilege governance for trace data access.
tags:
  - mlflow
  - storage
  - experiments
  - databricks
timestamp: "2026-06-19T23:07:36.055Z"
---

# [MLflow Experiment](/concepts/mlflow-experiment.md) as Trace Storage Backend

**MLflow Experiment as Trace Storage Backend** refers to the use of an [MLflow Experiment](/concepts/mlflow-experiment.md) as the primary storage location for production [Traces](/concepts/traces.md) generated by GenAI applications deployed on Databricks. When a GenAI application is deployed using either Custom Agents or custom CPU [Model Serving](/concepts/model-serving.md), Databricks automatically logs [Traces](/concepts/traces.md) to the [MLflow Experiment](/concepts/mlflow-experiment.md) specified during deployment. [Traces](/concepts/traces.md) are stored as artifacts within the experiment and are available for real-time viewing in the [MLflow](/concepts/mlflow.md) UI. ^[trace-agents-deployed-on-databricks-databricks-on-aws.md]

## Storage Mechanism

During deployment, the application connects to a specific [MLflow Experiment](/concepts/mlflow-experiment.md) via [`mlflow.set_experiment(...)`](https://[MLflow](/concepts/mlflow.md).org/docs/latest/api_reference/python_api/[MLflow](/concepts/mlflow.md).html#mlflow.set_experiment). All [Traces](/concepts/traces.md) produced by the deployed agent are then logged to that experiment. The experiment serves as the **short-term storage backend**; [Traces](/concepts/traces.md) remain accessible in the UI for real-time monitoring without additional configuration. ^[trace-agents-deployed-on-databricks-databricks-on-aws.md]

[Traces](/concepts/traces.md) are stored as artifacts, and you can specify a custom artifact location when creating the workspace experiment. For example, setting `artifact_location` to a [Unity Catalog](/concepts/unity-catalog.md) volume makes trace data access governed by [Unity Catalog](/concepts/unity-catalog.md) volume privileges](https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/privileges-reference#read-volume). ^[trace-agents-deployed-on-databricks-databricks-on-aws.md]

## Long‑Term Storage with [Production Monitoring](/concepts/production-monitoring.md)

For durable, long‑term trace retention, you can optionally enable [Production Monitoring](/concepts/production-monitoring.md) (in beta) on the same [MLflow Experiment](/concepts/mlflow-experiment.md). [Production Monitoring](/concepts/production-monitoring.md) syncs [Traces](/concepts/traces.md) to Delta tables approximately every 15 minutes, providing:

- **Durable storage** beyond the [MLflow Experiment](/concepts/mlflow-experiment.md) artifact lifecycle.
- **No trace size limits** – unlike alternative storage methods.
- **Automated quality assessment** via [MLflow Scorers](/concepts/mlflow-scorers.md) on production [Traces](/concepts/traces.md).

^[trace-agents-deployed-on-databricks-databricks-on-aws.md]

## Setup Requirements

- A workspace [MLflow Experiment](/concepts/mlflow-experiment.md) must be created (or an existing one used) prior to deployment.
- For Custom Agents deployments, tracing works automatically; you only need to instrument the agent with [MLflow Tracing](/concepts/mlflow-tracing.md) and set the experiment using `mlflow.set_experiment()`.
- For custom CPU [Model Serving](/concepts/model-serving.md) deployments, you must additionally set environment variables (`ENABLE_MLFLOW_TRACING`, `MLFLOW_EXPERIMENT_ID`, and authentication credentials) on the serving endpoint.
- If the agent is deployed from a notebook stored in a [Databricks Git folder](/concepts/databricks-git-folders-for-cicd.md), real‑time tracing does not work by default – you must explicitly set the experiment to a non‑Git‑associated experiment before deploying. ^[trace-agents-deployed-on-databricks-databricks-on-aws.md]

## Related Concepts

- [MLflow Experiments](/concepts/mlflow-experiment.md) – The organizational unit for runs, [Traces](/concepts/traces.md), and artifacts.
- [Production Monitoring](/concepts/production-monitoring.md) – Long‑term trace storage and evaluation using Delta tables.
- Custom Agents – Recommended deployment method with automatic trace logging.
- [MLflow Tracing](/concepts/mlflow-tracing.md) – Instrumentation framework for capturing GenAI application [Traces](/concepts/traces.md).
- [Unity Catalog](/concepts/unity-catalog.md) – Governed storage layer for custom artifact locations.

## Sources

- trace-agents-deployed-on-databricks-databricks-on-aws.md

# Citations

1. [trace-agents-deployed-on-databricks-databricks-on-aws.md](/references/trace-agents-deployed-on-databricks-databricks-on-aws-962e29f6.md)
