---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 8ea88e012787d197e76a165d79d987830bfbd741cb207bb97e99185fa50d32cc
  pageDirectory: concepts
  sources:
    - scorer-lifecycle-management-api-reference-databricks-on-aws.md
  confidence: 0.9
  provenanceState: inferred
  freshnessStatus: unverified
  aliases:
    - production-quality-monitoring-on-databricks
    - PQMOD
    - Production monitoring on Databricks
  citations:
    - file: scorer-lifecycle-management-api-reference-databricks-on-aws.md
title: Production Quality Monitoring on Databricks
description: Framework for continuous quality assessment of GenAI application traces using configurable scorers with sampling and filtering
tags:
  - databricks
  - monitoring
  - genai
timestamp: "2026-06-19T20:19:30.007Z"
---

# Production Quality Monitoring on Databricks

**Production Quality Monitoring on Databricks** refers to the practice of using [MLflow Scorers](/concepts/mlflow-scorers.md) to perform continuous, online quality assessment on traces generated by GenAI applications running in production. By registering, starting, and updating scorers, teams can evaluate a configurable fraction of live traces against custom or built-in scoring functions without interrupting the application.^[scorer-lifecycle-management-api-reference-databricks-on-aws.md]

## Key Concepts

### Scorer

A scorer is a function—defined with the `@scorer` decorator—that takes a trace's outputs and returns a metric (e.g., response length, safety score). Scorers are registered with an experiment and can be started, stopped, or updated independently. Each scorer maintains a sampling rate and an optional filter string that controls which traces are evaluated.^[scorer-lifecycle-management-api-reference-databricks-on-aws.md]

### Scorer Lifecycle

The lifecycle of a scorer consists of four operations, each of which returns a new immutable `Scorer` instance:

1. **Register** (`Scorer.register()`) – Makes a custom scorer function known to the MLflow server under a unique name within the experiment.
2. **Start** (`Scorer.start()`) – Begins online evaluation with a specified Sampling Configuration. The scorer will process a fraction of live traces (0.0 to 1.0) that optionally match a filter string.
3. **Update** (`Scorer.update()`) – Modifies the sampling configuration of an active scorer. The original scorer remains unchanged; a new instance is returned.
4. **Stop** (`Scorer.stop()`) – Halts online evaluation by setting the sample rate to 0. The scorer stays registered and can be restarted later.^[scorer-lifecycle-management-api-reference-databricks-on-aws.md]

### Scorer Registry

The MLflow Python SDK provides three top‑level functions for managing scorers:

- `mlflow.genai.scorers.get_scorer(name)` – Retrieves a registered scorer by name.
- `mlflow.genai.scorers.list_scorers()` – Returns all registered scorers for the current experiment.
- `mlflow.genai.scorers.delete_scorer(name)` – Permanently removes a scorer from the experiment’s registry.^[scorer-lifecycle-management-api-reference-databricks-on-aws.md]

### Scorer Properties

Each `Scorer` instance exposes:

- `sample_rate` – The fraction of traces currently being evaluated (0.0–1.0). A stopped scorer returns 0.
- `filter_string` – An MLflow‑compatible filter that restricts which traces are scored (e.g., `"trace.status = 'OK'"`).^[scorer-lifecycle-management-api-reference-databricks-on-aws.md]

### Configuration Class

`ScorerSamplingConfig` is a data class used when starting or updating a scorer:

```python
from mlflow.genai import ScorerSamplingConfig

config = ScorerSamplingConfig(
    sample_rate=0.5,
    filter_string="trace.status = 'OK'"
)
```

- `sample_rate` (float, optional): Sampling rate between 0.0 and 1.0.
- `filter_string` (str, optional): MLflow trace filter.^[scorer-lifecycle-management-api-reference-databricks-on-aws.md]

## Metric Backfill

When historical traces need to be evaluated (e.g., after adding a new scorer or changing a scoring function), the `backfill_scorers()` function from `databricks.agents.scorers` can be used. It runs a batch job over a specified time range, applying one or more scorers with optional custom sample rates. The function returns a job ID for tracking progress.^[scorer-lifecycle-management-api-reference-databricks-on-aws.md]

Parameters (all keyword‑only):

| Parameter | Type | Description |
|-----------|------|-------------|
| `experiment_id` | str, optional | ID of the experiment to backfill; defaults to current experiment context |
| `scorers` | `Union[List[BackfillScorerConfig], List[str]]`, required | Either `BackfillScorerConfig` objects (with optional custom sample rates) or scorer name strings (uses current sample rates) |
| `start_time` | datetime, optional | Start of the backfill time range |
| `end_time` | datetime, optional | End of the backfill time range |

Returns: `str` – Job ID of the created backfill job.^[scorer-lifecycle-management-api-reference-databricks-on-aws.md]

## Related Concepts

- [MLflow Scorers](/concepts/mlflow-scorers.md)
- GenAI Monitoring
- [Trace Evaluation](/concepts/mlflow-trace-based-evaluation.md)
- Sampling Configuration
- Backfill Metrics

## Sources

- scorer-lifecycle-management-api-reference-databricks-on-aws.md

# Citations

1. [scorer-lifecycle-management-api-reference-databricks-on-aws.md](/references/scorer-lifecycle-management-api-reference-databricks-on-aws-55d28735.md)
