---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 3a6e080ca6fd6d92edab9cdf387e3cb6c59d61594f5fa0f0206bba7c21d25582
  pageDirectory: concepts
  sources:
    - 10-minute-demo-collect-human-feedback-databricks-on-aws.md
  confidence: 0.85
  provenanceState: merged
  freshnessStatus: unverified
  aliases:
    - multi-level-human-feedback-in-genai-pipelines
    - MHFIGP
    - Human Feedback in GenAI
  citations:
    - file: 10-minute-demo-collect-human-feedback-databricks-on-aws.md
title: Multi-Level Human Feedback in GenAI Pipelines
description: "A pattern for collecting feedback at three distinct stages: end-user (in-app thumbs up/down), developer (UI annotations), and domain expert (structured labeling sessions)"
tags:
  - genai
  - feedback
  - workflow
  - evaluation
timestamp: "2026-06-18T14:15:13.881Z"
---

# Multi-Level Human Feedback in GenAI Pipelines

**Multi-Level Human Feedback in GenAI Pipelines** is a structured approach to collecting and incorporating human evaluations at different stages of the generative AI application lifecycle. By gathering feedback from end users, developers, and domain experts, teams can systematically assess and improve the quality of their GenAI applications. ^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

## Overview

Human feedback in GenAI pipelines operates at multiple levels, each serving a distinct purpose in the quality assurance workflow. End users provide real-time satisfaction signals during application usage, developers add detailed annotations during analysis and debugging, and domain experts offer authoritative assessments that serve as ground truth for evaluation. This multi-layered approach ensures that quality improvements are grounded in both user experience and expert judgment. ^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

## Levels of Feedback

### End-User Feedback

End-user feedback captures how actual users perceive the quality of responses generated by the GenAI application. This is typically collected through simple UI elements such as thumbs up/down buttons, rating scales, or direct text input. End-user feedback is collected in real time during application usage and is associated with specific traces via a `trace_id`. ^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

Using the MLflow SDK, end-user feedback can be recorded programmatically:

```python
from mlflow.entities.assessment import AssessmentSource, AssessmentSourceType

mlflow.log_feedback(
    trace_id=trace_id,
    name="user_feedback",
    value=False,  # False for thumbs down
    rationale="Missing details about key features",
    source=AssessmentSource(
        source_type=AssessmentSourceType.HUMAN,
        source_id="enduser_123",
    ),
)
```

^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

### Developer Annotations

Developers can add their own feedback and notes interactively through the MLflow UI. Developer annotations provide a way for engineers building the application to evaluate responses during development and debugging. These annotations can include numeric scores, categorical assessments, or free-text rationale. ^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

To add a developer annotation in the UI:

1. Open a trace in the **Logs** tab.
2. Click on any span (the root span for trace-level feedback).
3. In the **Assessments** tab, click **Add new assessment**.
4. Provide the **Type**, **Name**, **Value**, and **Rationale**.

After refreshing the page, columns for new assessments appear in the Logs table. ^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

### Expert Feedback

Expert feedback involves structured review by domain experts who can provide authoritative assessments of response quality. This is the most rigorous level of human feedback and serves as ground truth for evaluating and improving the application. Expert feedback is collected through [Labeling Sessions](/concepts/labeling-sessions.md). ^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

Expert reviewers can assess multiple dimensions of response quality, including factual accuracy, completeness, and adherence to expected behaviors. They can also provide ideal or corrected responses that serve as reference standards for automated evaluation. ^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

## Creating Labeling Sessions for Expert Review

To collect expert feedback, create a labeling session that defines what feedback to collect and then add traces for review:

```python
from mlflow.genai.label_schemas import create_label_schema, InputCategorical, InputText
from mlflow.genai.labeling import create_labeling_session

# Define what feedback to collect
accuracy_schema = create_label_schema(
    name="response_accuracy",
    type="feedback",
    title="Is the response factually accurate?",
    input=InputCategorical(options=["Accurate", "Partially Accurate", "Inaccurate"]),
    overwrite=True
)

ideal_response_schema = create_label_schema(
    name="expected_response",
    type="expectation",
    title="What would be the ideal response?",
    input=InputText(),
    overwrite=True
)

# Create a labeling session
labeling_session = create_labeling_session(
    name="quickstart_review",
    label_schemas=[accuracy_schema.name, ideal_response_schema.name],
)

# Add traces to the session
labeling_session.add_traces(traces)

# Share with reviewers
print(f"Share this link with reviewers: {labeling_session.url}")
```

^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

Labeling sessions can also be created through the MLflow UI:
1. On the Experiment page, click the **Labeling** tab.
2. Use the **Sessions** and **Schemas** tabs to add new label schemas and create a new session.

^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

## Using Feedback for Evaluation

After experts provide feedback, their labels can be used to evaluate the application using automated scorers. For example, the `Correctness` scorer compares the application's outputs against expert-provided `expected_response` labels:

```python
from mlflow.genai.scorers import Correctness

# Get traces from the labeling session
labeled_traces = mlflow.search_traces(
    run_id=labeling_session.mlflow_run_id,
)

# Evaluate app against expert expectations
eval_results = mlflow.genai.evaluate(
    data=labeled_traces,
    predict_fn=my_chatbot,
    scorers=[Correctness()]
)
```

This provides quantitative feedback on alignment with expert expectations. ^[10-minute-demo-collect-human-feedback-databricks-on-aws.md]

## Best Practices

- **Collect feedback at all three levels** for a comprehensive understanding of application quality.
- **Use end-user negative feedback as a signal** to trigger expert review, confirming whether issues are genuine.
- **Return the `trace_id` with responses** to the frontend so that user feedback can be associated with specific traces.
- **Add labeled traces to an MLflow Evaluation Dataset** for version tracking and lineage.
- **Define structured label schemas** for expert review to ensure consistent, actionable feedback.

## Related Concepts

- [MLflow Tracing](/concepts/mlflow-tracing.md) — The foundation for associating feedback with specific application executions
- [Labeling Sessions](/concepts/labeling-sessions.md) — Structured expert review workflows
- [MLflow Evaluation](/concepts/mlflow-evaluation-ui.md) — Automated scoring using expert feedback as ground truth
- [Custom Judges](/concepts/custom-judges.md) — LLM-based evaluators that can be aligned with human feedback
- [Production Monitoring for GenAI](/concepts/production-quality-monitoring-mlflow-genai.md) — Deploying judges for continuous quality monitoring
- Align judges with human feedback — Improving automated evaluation with expert annotations

## Sources

- 10-minute-demo-collect-human-feedback-databricks-on-aws.md

# Citations

1. [10-minute-demo-collect-human-feedback-databricks-on-aws.md](/references/10-minute-demo-collect-human-feedback-databricks-on-aws-3f033604.md)
