---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 6e68a81ca6659a102360dc29fe27ff54a3c3873f51d1a94e24c92c5ced33bc78
  pageDirectory: concepts
  sources:
    - collect-user-feedback-databricks-on-aws.md
  confidence: 0.95
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - feedback-collection-patterns-in-genai-applications
    - FCPIGA
  citations:
    - file: collect-user-feedback-databricks-on-aws.md
title: Feedback Collection Patterns in GenAI Applications
description: "Two approaches for linking user feedback to MLflow traces: using MLflow-generated trace IDs or custom client request IDs"
tags:
  - mlflow
  - feedback
  - pattern
timestamp: "2026-06-19T17:46:11.902Z"
---

# Feedback Collection Patterns in GenAI Applications

**Feedback Collection Patterns in GenAI Applications** refers to the structured approaches and architectural patterns for capturing, storing, and analyzing user feedback on outputs generated by generative AI applications. These patterns enable organizations to understand real-world application quality, drive continuous improvement, and build high-quality evaluation datasets from production data. ^[collect-user-feedback-databricks-on-aws.md]

## Overview

Collecting and logging user feedback is essential for understanding the real-world quality of a GenAI application. MLflow provides a structured way to capture feedback as assessments on traces, enabling teams to track quality over time, identify areas for improvement, and build evaluation datasets from production data. ^[collect-user-feedback-databricks-on-aws.md]

User feedback provides ground truth about an application's performance through several mechanisms: real-world quality signals that reveal how actual users perceive outputs, patterns in negative feedback that guide development priorities, training data creation from high-quality feedback examples, quality monitoring across different user segments, and data for model fine-tuning. ^[collect-user-feedback-databricks-on-aws.md]

## Prerequisites

MLflow 3 is required for collecting user feedback. MLflow 2.x is not supported due to performance limitations and missing features essential for production use. ^[collect-user-feedback-databricks-on-aws.md]

For production deployments, install the `mlflow-tracing` package, which is optimized for production use with minimal dependencies and better performance characteristics. The `log_feedback` API is available in both the standard MLflow package and the `mlflow-tracing` package. ^[collect-user-feedback-databricks-on-aws.md]

## Feedback Data Model

In MLflow, user feedback is captured using the **Feedback** entity, which is a type of [Assessment](/concepts/assessments.md) that can be attached to [Traces](/concepts/traces.md) or specific Spans. The Feedback entity provides a structured way to store: ^[collect-user-feedback-databricks-on-aws.md]

- **Value**: The actual feedback, which can be boolean, numeric, text, or structured data
- **Source**: Information about who or what provided the feedback (human user, LLM judge, or code)
- **Rationale**: Optional explanation for the feedback
- **Metadata**: Additional context like timestamps or custom attributes

The `AssessmentSource` object identifies who or what provided the feedback. The `source_type` field can be `"HUMAN"` for user feedback or `"LLM_JUDGE"` for automated evaluation, and the `source_id` field identifies the specific user or system providing feedback. ^[collect-user-feedback-databricks-on-aws.md]

## Feedback Collection Approaches

When implementing feedback collection in production, there are two primary approaches for linking user feedback to specific traces: ^[collect-user-feedback-databricks-on-aws.md]

### Using MLflow Trace IDs

The simplest approach uses the trace ID that MLflow automatically generates for each trace. During request processing, the application retrieves the current trace ID using `mlflow.get_current_active_span().trace_id` and returns it to the client in the response. The client then submits feedback by referencing this trace ID. ^[collect-user-feedback-databricks-on-aws.md]

### Using Client Request IDs

Alternatively, applications can generate their own unique client request IDs when processing requests and reference them later for feedback. This approach provides more control over the ID format and can integrate with existing request tracking systems. ^[collect-user-feedback-databricks-on-aws.md]

## Feedback Collection Flow

Both approaches follow a similar pattern: ^[collect-user-feedback-databricks-on-aws.md]

1. **During the initial request**: The application either generates a unique client request ID or retrieves the MLflow-generated trace ID
2. **After receiving the response**: The user provides feedback by referencing either ID
3. **Feedback is logged**: MLflow's `log_feedback` API creates an assessment attached to the original trace
4. **Analysis and monitoring**: Teams can query and analyze feedback across all traces

## Handling Different Feedback Types

Feedback collection can be extended to support more complex, multi-dimensional feedback. For example, an application might collect separate ratings for accuracy, helpfulness, and relevance, with each dimension logged as a separate assessment for granular analysis. Values can be normalized to a consistent scale (such as 0-1) for easier comparison and aggregation. ^[collect-user-feedback-databricks-on-aws.md]

## Streaming Response Considerations

When using streaming responses (Server-Sent Events or WebSockets), the trace ID is not available until the stream completes, presenting a unique challenge for feedback collection. ^[collect-user-feedback-databricks-on-aws.md]

### Why Streaming Is Different

In traditional request-response patterns, the complete response and trace ID are received together. With streaming: ^[collect-user-feedback-databricks-on-aws.md]

1. **Tokens arrive incrementally**: The response is built up over time as tokens stream from the LLM
2. **Trace completion is deferred**: The trace ID is only generated after the entire stream finishes
3. **Feedback UI must wait**: Users cannot provide feedback until they have both the complete response and the trace ID

### Implementation Strategy for Streaming

For streaming implementations, the backend sends the trace ID as a final event in the stream after all tokens have been delivered. The frontend handles streaming events and enables feedback controls only after receiving the trace ID. ^[collect-user-feedback-databricks-on-aws.md]

Key considerations for streaming feedback collection include: ^[collect-user-feedback-databricks-on-aws.md]

- **Trace ID timing**: Design the UI to disable feedback controls until the trace ID is received
- **Event structure**: Use a consistent event format with a `type` field to distinguish between content tokens, completion events, and errors
- **State management**: Track streaming content and trace ID separately, resetting all state at the start of each new interaction
- **Error handling**: Include error events in the stream and log errors to the trace when possible
- **Buffer management**: Use headers like `X-Accel-Buffering: no` to disable proxy buffering, and implement proper line buffering for partial SSE messages
- **Performance optimization**: Add small delays between tokens to prevent overwhelming clients, and consider backpressure mechanisms for slow clients

## Analyzing Feedback Data

Once feedback is collected, it can be analyzed to gain insights about application quality and user satisfaction. Feedback is stored as assessments on the trace, meaning it is permanently associated with the specific interaction, can be queried alongside the trace data, and is visible in the MLflow UI when viewing the trace. ^[collect-user-feedback-databricks-on-aws.md]

### Retrieving Traces with Feedback

Traces can be retrieved from a specific time window using the MLflow client's `search_traces` method, filtering by experiment name and timestamp. Full trace details, including assessments, can then be obtained using `client.get_trace()`. ^[collect-user-feedback-databricks-on-aws.md]

### Analyzing Feedback Patterns

Feedback analysis can calculate metrics such as: ^[collect-user-feedback-databricks-on-aws.md]

- **Feedback rate**: The percentage of traces that received feedback
- **Positive rate**: The percentage of feedback that was positive
- **Dimension-specific ratings**: Average scores for multi-dimensional feedback

For multi-dimensional feedback, ratings can be analyzed by dimension to identify specific areas for improvement. ^[collect-user-feedback-databricks-on-aws.md]

## Related Concepts

- [Traces](/concepts/traces.md) — The core unit of observability in MLflow GenAI applications
- Spans — Individual operations within a trace
- [Assessments](/concepts/assessments.md) — The entity type used to store feedback on traces
- [Production Monitoring](/concepts/production-monitoring.md) — Monitoring quality metrics based on feedback
- [Evaluation Datasets](/concepts/evaluation-datasets.md) — Building test datasets from collected feedback
- [LLM Judges](/concepts/llm-judges.md) — Automated evaluation using LLM-based assessment
- [Human Feedback](/concepts/mlflow-human-feedback-collection.md) — Manual annotation and feedback workflows

## Sources

- collect-user-feedback-databricks-on-aws.md

# Citations

1. [collect-user-feedback-databricks-on-aws.md](/references/collect-user-feedback-databricks-on-aws-0b0ba83c.md)
