---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 4d8582a8432c6f0b8a9470a38f63b98067460408b00c923f2074ac079b9b6834
  pageDirectory: concepts
  sources:
    - deepeval-scorers-databricks-on-aws.md
  confidence: 0.9
  provenanceState: inferred
  freshnessStatus: unverified
  aliases:
    - llm-as-a-judge-scoring
title: LLM-as-a-Judge Scoring
description: DeepEval scorers use an LLM-as-a-judge approach where a language model (e.g., databricks-gpt-5-mini or gpt-4o) evaluates outputs against configurable thresholds.
tags:
  - llm-evaluation
  - judge-llm
  - scoring
timestamp: "2026-06-19T14:58:44.710Z"
---

# LLM-as-a-Judge Scoring

**LLM-as-a-Judge Scoring** is an evaluation paradigm where a large language model (LLM) is used as the scoring function to assess the quality of other LLM outputs. Rather than relying on traditional static metrics or human evaluation, an LLM "judge" evaluates generated text against predefined criteria or ground truth.

## Overview

This approach treats the LLM as an automated evaluator that can assess aspects such as relevance, faithfulness, safety, and overall quality of responses generated by other models. The scoring LLM can be configured with specific parameters including a threshold for binary pass/fail decisions and a model endpoint that serves as the judge.

## Configuration

When implementing LLM-as-a-Judge scoring, the scoring function accepts metric-specific parameters as keyword arguments. Key configuration options include:

- **model** - Specifies the LLM endpoint to use as the judge (e.g., `"databricks:/databricks-gpt-5-mini"`)
- **threshold** - A numerical value between 0 and 1 that determines the minimum score for passing evaluation. For binary scorers, outputs above the threshold receive a "yes" while those below receive a "no"
- **include_reason** - When set to `True`, the scorer provides an explanation for its evaluation decision

Additional metric-specific parameters may be available depending on the evaluation framework used.

## Scoring Output

An LLM-as-a-Judge scorer returns a feedback object containing:

- **value** - A binary string indicating the overall assessment ("yes" or "no")
- **metadata** - A dictionary containing the numerical score (typically a float between 0 and 1)

## Integration with Evaluation Frameworks

LLM-as-a-Judge scoring is commonly integrated with specialized evaluation frameworks. For example, MLflow supports using LLM-based metrics from DeepEval as scorers, allowing developers to evaluate RAG systems, agent behavior, conversational quality, and safety aspects of model outputs.

## Use Cases

This scoring paradigm is particularly useful for evaluating:

- Retrieval-augmented generation (RAG) applications
- AI agent behavior including task completion and tool usage
- Multi-turn conversational AI quality
- Model output safety and responsibility

## Related Concepts

- [DeepEval Scorers](/concepts/deepeval-scorer-api.md) - Available LLM-as-a-Judge metrics for RAG, agentic, and safety evaluation
- [MLflow GenAI Evaluation](/concepts/mlflow-genai-evaluation.md) - Framework for evaluating generative AI models
- RAG Metrics - Evaluation metrics for retrieval quality in RAG systems
- Agentic Metrics - Scorers for AI agent behavior assessment
- Conversational Metrics - Metrics for multi-turn dialogue evaluation
- Safety Metrics - Evaluators for model output safety
- Threshold-based Scoring - Using numerical thresholds for binary evaluation decisions

## Sources

- deepeval-scorers-databricks-on-aws.md
