---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 52ffe6263476033e7bfb60ba68d7d7fa502154d16b2dcc4062b7b61142624d3a
  pageDirectory: concepts
  sources:
    - deepeval-scorers-databricks-on-aws.md
  confidence: 1
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - deepeval-framework
  citations:
    - file: deepeval-scorers-databricks-on-aws.md
title: DeepEval Framework
description: A comprehensive evaluation framework for LLM applications that provides metrics for RAG systems, agents, conversational AI, and safety evaluation.
tags:
  - llm-evaluation
  - framework
  - deepeval
timestamp: "2026-06-19T09:58:26.689Z"
---

# DeepEval Framework

**DeepEval** is a comprehensive evaluation framework for LLM applications developed by Confident AI. It provides a suite of metrics designed to assess the performance and quality of retrieval-augmented generation (RAG) systems, AI agents, conversational AI, and model safety. ^[deepeval-scorers-databricks-on-aws.md]

## Overview

DeepEval offers structured, automated evaluation capabilities for large language model (LLM) applications, covering multiple evaluation dimensions including retrieval quality, answer relevance, faithfulness, agent behavior, conversational coherence, and output safety. ^[deepeval-scorers-databricks-on-aws.md]

## Integration with MLflow

[MLflow](/concepts/mlflow.md) integrates with DeepEval so that developers can use DeepEval metrics as scorers within MLflow's evaluation workflows. This integration allows DeepEval's evaluation capabilities to be leveraged through both direct Python calls and the `mlflow.genai.evaluate()` API. ^[deepeval-scorers-databricks-on-aws.md]

### Requirements

To use DeepEval scorers, the `deepeval` package must be installed. LLM-based metrics require a `model` parameter specifying the underlying language model to use for evaluation. ^[deepeval-scorers-databricks-on-aws.md]

## Available Metrics

### RAG Metrics

These scorers evaluate retrieval quality and answer generation in retrieval-augmented generation (RAG) applications. They assess how well the system retrieves relevant context and generates responses that are faithful to that context. ^[deepeval-scorers-databricks-on-aws.md]

### Agentic Metrics

These scorers evaluate AI agent behavior, including task completion and tool usage. They measure how effectively an agent accomplishes assigned goals and whether it uses available tools appropriately. ^[deepeval-scorers-databricks-on-aws.md]

### Conversational Metrics

These scorers evaluate multi-turn conversational AI quality. They assess aspects of dialogue such as turn relevance, coherence across conversation history, and overall conversational effectiveness. ^[deepeval-scorers-databricks-on-aws.md]

### Safety Metrics

These scorers evaluate the safety and responsibility of model outputs. They help detect harmful, biased, or otherwise inappropriate content generated by LLMs. ^[deepeval-scorers-databricks-on-aws.md]

### Other and Non-LLM Metrics

DeepEval also provides additional metrics, including non-LLM-based metrics that do not require a language model for evaluation. ^[deepeval-scorers-databricks-on-aws.md]

## Usage

### Direct Scorer Invocation

DeepEval scorers can be called directly to evaluate a single input-output pair: ^[deepeval-scorers-databricks-on-aws.md]

```python
from mlflow.genai.scorers.deepeval import AnswerRelevancy

scorer = AnswerRelevancy(threshold=0.7, model="databricks:/databricks-gpt-5-mini")
feedback = scorer(
    inputs="What is MLflow?",
    outputs="MLflow is an open-source AI engineering platform for agents and LLMs.",
)
print(feedback.value)      # "yes" or "no"
print(feedback.metadata["score"])  # 0.85
```

### Using with mlflow.genai.evaluate()

DeepEval scorers can be used within MLflow's batch evaluation framework for assessing multiple examples at once: ^[deepeval-scorers-databricks-on-aws.md]

```python
import mlflow
from mlflow.genai.scorers.deepeval import AnswerRelevancy, Faithfulness

eval_dataset = [
    {
        "inputs": {"query": "What is MLflow?"},
        "outputs": "MLflow is an open-source AI engineering platform for agents and LLMs.",
    },
]

results = mlflow.genai.evaluate(
    data=eval_dataset,
    scorers=[
        AnswerRelevancy(threshold=0.7, model="databricks:/databricks-gpt-5-mini"),
        Faithfulness(threshold=0.8, model="databricks:/databricks-gpt-5-mini"),
    ],
)
```

### Dynamic Scorer Creation

Scorers can be created dynamically by passing the metric name as a string: ^[deepeval-scorers-databricks-on-aws.md]

```python
from mlflow.genai.scorers.deepeval import get_scorer

scorer = get_scorer(
    metric_name="AnswerRelevancy",
    threshold=0.7,
    model="databricks:/databricks-gpt-5-mini",
)
```

## Configuration

DeepEval scorers accept metric-specific parameters as keyword arguments to the constructor. LLM-based metrics require a `model` parameter specifying the underlying model to use for evaluation. Additional parameters such as `include_reason`, `window_size`, and `strict_mode` are available depending on the specific metric. ^[deepeval-scorers-databricks-on-aws.md]

```python
from mlflow.genai.scorers.deepeval import AnswerRelevancy, TurnRelevancy

# LLM-based metric with common parameters
scorer = AnswerRelevancy(
    model="databricks:/databricks-gpt-5-mini",
    threshold=0.7,
    include_reason=True,
)

# Metric-specific parameters
conversational_scorer = TurnRelevancy(
    model="openai:/gpt-4o",
    threshold=0.8,
    window_size=3,
    strict_mode=True,
)
```

For metric-specific parameters and advanced usage options, refer to the [DeepEval documentation](https://docs.confident-ai.com/). ^[deepeval-scorers-databricks-on-aws.md]

## Related Concepts

- [MLflow Evaluation](/concepts/mlflow-evaluation-ui.md) — The MLflow framework for evaluating LLM applications
- [GenAI Scoring](/concepts/mlflow-genai-scorer.md) — The scoring infrastructure used by MLflow for third-party evaluators
- [LLM Evaluation Metrics](/concepts/llm-as-a-judge-evaluation-metrics.md) — General concepts for measuring LLM output quality
- RAG Evaluation — Evaluating retrieval-augmented generation systems
- [AI Agent Evaluation](/concepts/mlflow-agent-evaluation.md) — Assessing agent behavior and task completion

## Sources

- deepeval-scorers-databricks-on-aws.md

# Citations

1. [deepeval-scorers-databricks-on-aws.md](/references/deepeval-scorers-databricks-on-aws-95eb5ac0.md)
