---
okf_version: "0.1"
---

# Knowledge Bundle

## Concepts

* [@distributed Decorator](/concepts/distributed-decorator.md) - A decorator from the serverless_gpu library that annotates functions to run across multiple GPUs, accepting parameters for GPU count and type.
* [@distributed Decorator Data Loading](/concepts/distributed-decorator-data-loading.md) - Best practice to load data inside the @distributed decorator to avoid pickle serialization issues when datasets are too large.
* [@distributed decorator for distributed training](/concepts/distributed-decorator-for-distributed-training.md) - A Python decorator from the serverless_gpu library that handles data parallelism across multiple GPUs, automatically managing rank, local rank, and world size environment variables.
* [@distributed Decorator Pattern](/concepts/distributed-decorator-pattern.md) - A Python decorator from Databricks serverless_gpu library that provisions GPU resources and handles distributed training setup automatically
* [@mlflow.trace decorator](/concepts/mlflowtrace-decorator.md) - A Python decorator that wraps any function to automatically capture its inputs, outputs, and execution metadata as a span in an MLflow trace.
* [@mlflow.trace Function Decorator](/concepts/mlflowtrace-function-decorator.md) - A Python decorator that creates a span for any function, automatically logging function name, inputs, outputs, and execution time with minimal code changes.
* [@scorer decorator](/concepts/scorer-decorator.md) - The primary way to define custom code-based scorers for MLflow GenAI evaluation, using a decorator-based function signature with keyword-only arguments.
* [%run command in Databricks](/concepts/run-command-in-databricks.md) - A magic command that imports and executes another notebook inline, making its functions and variables available in the calling notebook.
* [$CODE_SOURCE_PATH](/concepts/code_source_path.md) - A Databricks-resolved environment variable that points to the uploaded code location on the remote compute node, used to reference training scripts without hardcoding paths.
* [20B to 120B+ Parameter Model Training](/concepts/20b-to-120b-parameter-model-training.md) - The parameter range where FSDP is particularly useful, enabling training of models between 20 billion and 120+ billion parameters
* [30-Day Lookback Window](/concepts/30-day-lookback-window.md) - A profiling constraint where for time series or inference profiles, the analysis looks back 30 days from profile creation, potentially causing partial windows in the first analysis.
* [30-Day Lookback Window Limitation](/concepts/30-day-lookback-window.md) - For time series or inference profiles, the profile looks back 30 days from when it is created, which may cause the first analysis to include a partial window (e.g., cutting mid-week or mid-month).
* [403 PERMISSION_DENIED Serverless Budget Policy Error](/concepts/403-permission_denied-serverless-budget-policy-error.md) - An error that occurs when a workspace disables the default serverless budget policy and no fallback policy is available for MLflow serverless workloads.
* [8xH100 Single-Node Configuration](/concepts/8xh100-single-node-configuration.md) - A Databricks Serverless GPU configuration providing 8 H100 GPUs on a single node for distributed workloads
* [A/B Comparison of Agent Configurations](/concepts/ab-comparison-of-agent-configurations.md) - Using custom judges to compare different agent configurations (e.g., toggling a behavior flag) by running `mlflow.genai.evaluate()` separately on each variant and comparing judge ratings across the same evaluation dataset.
* [A100 GPU Support on Databricks](/concepts/a100-gpu-support-on-databricks.md) - Databricks supports A100 GPUs across all clouds for deep learning tasks like LLM training, NLP, object detection, and recommendation engines, though availability may be limited
* [ABAC (Attribute-Based Access Control)](/concepts/abac-attribute-based-access-control.md) - An access-control model in Unity Catalog that uses governed tags and policies to grant permissions based on object attributes rather than per-object grants.
* [ABAC Built-in Tag Functions](/concepts/abac-built-in-tag-functions.md) - The has_tag() and has_tag_value() functions used in policy conditions to match tables and columns based on governed tags, supporting logical operators and snake_case naming.
* [ABAC Column Mask Policies](/concepts/abac-column-mask-policy.md) - A Unity Catalog policy type that masks column values in query results using a UDF, applicable to columns matched by governed tags or custom expressions.
* [ABAC Column Mask Policy](/concepts/abac-column-mask-policy.md) - A Unity Catalog ABAC policy that masks column values by applying a masking function to columns matched by governed tags or custom expressions.
* [ABAC Column-Level Mask (Unity Catalog)](/concepts/abac-column-level-mask-unity-catalog.md) - Attribute-based access control policy that can be automatically applied to columns tagged by custom classifiers, enabling dynamic data masking based on user attributes.
* [ABAC Compute Requirements](/concepts/abac-compute-requirements.md) - Compute types and runtime versions required to use attribute-based access control row filter and column mask policies in Unity Catalog
* [ABAC Conditions and Built-in Functions](/concepts/abac-conditions-and-built-in-functions.md) - Policy conditions use has_tag() and has_tag_value() built-in functions in WHEN (table) and MATCH COLUMNS (column) clauses to match objects based on governed tags, with support for logical operators and aliases.
* [ABAC Dynamic Policy Evaluation and Transparency](/concepts/abac-dynamic-policy-evaluation-and-transparency.md) - The query-time nature of ABAC policy evaluation and practices to improve visibility for data consumers and table owners
* [ABAC Governance based on Data Classification](/concepts/abac-governance-from-classification.md) - Using attribute-based access control (ABAC) policies in Unity Catalog to mask sensitive data based on classification tags, with the ability to create policies directly from the Data Classification UI.
* [ABAC Governance Controls from Classification](/concepts/abac-governance-from-classification.md) - Using Attribute-based Access Control (ABAC) policies in Unity Catalog to mask sensitive columns based on detected classification tags, optionally combining multiple tags in a single policy.
* [ABAC Governance from Classification](/concepts/abac-governance-from-classification.md) - Using attribute-based access control policies in Unity Catalog to mask or restrict sensitive columns based on data classification tags detected by the classification engine.
* [ABAC GRANT Policies](/concepts/abac-grant-policy.md) - Attribute-based access control policies that dynamically grant Unity Catalog privileges to securable objects whose governed tags match a condition, rather than granting on explicitly named objects.
* [ABAC GRANT Policies and Direct Grants Auditing](/concepts/abac-grant-policies-and-direct-grants-auditing.md) - Auditing both direct grants and ABAC GRANT policies together to uncover unintended permissions
* [ABAC GRANT Policy](/concepts/abac-grant-policy.md) - An attribute-based access control policy that dynamically grants Unity Catalog privileges to securable objects whose governed tags match a condition, evaluated at access time.
* [ABAC GRANT Policy and Direct Grant Auditing](/concepts/abac-grant-policy-and-direct-grant-auditing.md) - Effective privileges on a data object are the union of both direct grants and ABAC GRANT policies, requiring auditors to check both surfaces to avoid hidden unintended permissions.
* [ABAC on Materialized Views and Streaming Tables](/concepts/abac-on-materialized-views-and-streaming-tables.md) - Limitations and requirements for applying ABAC policies to materialized views and streaming tables, including pipeline owner exemption
* [ABAC Policies and AI Search Indexes](/concepts/abac-policies-and-ai-search-indexes.md) - AI Search indexes created from ABAC-secured tables do not enforce row filter or column mask policies when serving queries
* [ABAC Policies from Data Classification](/concepts/abac-policies-from-data-classification.md) - The practice of creating attribute-based access control policies that mask or restrict sensitive columns based on the classification tags detected by the Data Classification engine.
* [ABAC Policies in Unity Catalog](/concepts/abac-policies-in-unity-catalog.md) - Policies attached at a level in the Unity Catalog hierarchy (catalog, schema, table) that are evaluated dynamically — when a securable object has targeted attributes, the policy takes effect automatically.
* [ABAC Policies on SAP Governance Tags](/concepts/abac-policies-on-sap-governance-tags.md) - Using attribute-based access control (ABAC) policies in Unity Catalog that reference SAP system-governed tags to control access to sensitive SAP BDC data.
* [ABAC Policies on Shared Data](/concepts/abac-policies-on-shared-data.md) - Attribute-based access control (ABAC) policies such as row filters and column masks can be applied to shared tables, schemas, and catalogs but not to shared streaming tables or materialized views.
* [ABAC policies vs table-level row filters and column masks](/concepts/abac-policies-vs-table-level-row-filters-and-column-masks.md) - Comparison of attribute-based access control (schema/catalog-level, tag-driven) versus per-table manual row filters and column masks in Unity Catalog.
* [ABAC Policy Audit Logging](/concepts/abac-policy-audit-logging.md) - Databricks logs all governed tag and ABAC policy operations (create, delete, list, get) in the system.access.audit table for compliance and monitoring.
* [ABAC Policy Conditions and Built-in Functions](/concepts/abac-policy-conditions-and-built-in-functions.md) - Tag-based expressions using has_tag() and has_tag_value() functions to match tables (WHEN clause) and columns (MATCH COLUMNS clause) within a policy's scope, supporting up to 3 column expressions.
* [ABAC Policy CRUD Operations](/concepts/abac-policy-crud-operations.md) - Operations to create, edit, view (SHOW/DESCRIBE), and delete ABAC row filter and column mask policies in Unity Catalog using Catalog Explorer UI, SQL, Python SDK, REST APIs, SDKs, or Terraform.
* [ABAC Policy Dependency Lifecycle Management](/concepts/abac-policy-dependency-lifecycle-management.md) - Handling of removed policy dependencies (governed tags, UDFs, tagged columns) including error codes, fail-closed behavior, and resolution steps required when dependencies are deleted while policies still reference them.
* [ABAC Policy Evaluation and Enforcement](/concepts/abac-policy-evaluation-and-enforcement.md) - Two-stage process where Unity Catalog evaluates which policies apply based on user identity and tags, then Databricks Runtime enforces row filters and column masks during query execution.
* [ABAC Policy Integration with Classification](/concepts/abac-policy-integration-with-classification.md) - Integration between Data Classification and Attribute-Based Access Control (ABAC) allowing users to create masking policies directly from classification results, protecting sensitive data by masking columns based on detected classification tags.
* [ABAC Policy Quotas and Service Limits](/concepts/abac-policy-quotas-and-service-limits.md) - Quotas governing the number and scope of ABAC row filter and column mask policies in Unity Catalog
* [ABAC Policy Requirements](/concepts/abac-policy-requirements.md) - Prerequisites for creating and managing ABAC policies including MANAGE permission on the securable object, EXECUTE on the UDF, Databricks Runtime 16.4+ or serverless compute, and governed tags on target objects.
* [ABAC Policy Requirements and Permissions](/concepts/abac-policy-requirements.md) - Prerequisite permissions (MANAGE, object ownership, EXECUTE on UDFs) and compute requirements (Databricks Runtime 16.4+ or serverless) for creating and managing ABAC policies.
* [ABAC Policy Requirements and Prerequisites](/concepts/abac-policy-requirements.md) - Requirements for creating ABAC policies including MANAGE privilege or ownership on the securable object, Databricks Runtime 16.4+ or serverless compute, EXECUTE on a UDF, and governed tags applied to target objects.
* [ABAC Policy Scoping (Catalog vs Schema vs Table)](/concepts/policy-scoping-at-catalog-or-schema-level.md) - Attaching ABAC policies at the highest applicable scope (catalog or schema) rather than table level to maximize reuse and simplify management.
* [ABAC Policy Scoping and Sprawl Prevention](/concepts/abac-policy-scoping-and-sprawl-prevention.md) - Attaching policies at catalog or schema level rather than table level, starting with a small number of broad policies, and periodically consolidating overlapping ones to avoid complex authorization checks.
* [ABAC Policy Scoping at Catalog and Schema Level](/concepts/policy-scoping-at-catalog-or-schema-level.md) - Attaching ABAC policies at the highest applicable scope (catalog or schema) rather than table level for simpler management.
* [ABAC Policy Scoping Hierarchy](/concepts/abac-policy-scoping-hierarchy.md) - Principle of attaching ABAC policies at the highest applicable scope (catalog or schema level) rather than table level to maximize reusability
* [ABAC Policy Sprawl Prevention](/concepts/abac-policy-sprawl-prevention.md) - Avoiding excessive tags and policies by starting with a small set of broad policies, consolidating overlapping ones, and periodic review.
* [ABAC Policy Structure](/concepts/abac-policy-structure.md) - Every ABAC policy specifies a scope (catalog/schema/table), principals (TO/EXCEPT clauses), actions (row filter, column mask, or grant), and tag-based conditions (WHEN and MATCH COLUMNS clauses).
* [ABAC Policy Types](/concepts/abac-policy-types.md) - Three policy types in Unity Catalog ABAC: row filter policies, column mask policies, and GRANT policies (Beta), each enforcing different access controls based on tag conditions.
* [ABAC policy UDF complexity and performance](/concepts/abac-policy-udf-complexity-and-performance.md) - How the complexity of UDFs in row filter and column mask policies directly affects query execution time, with guidance on keeping UDFs simple, avoiding external calls, and favoring built-in SQL functions.
* [ABAC Row Filter Policies](/concepts/abac-row-filter-policy.md) - A Unity Catalog policy type that filters rows from query results based on a user-defined function (UDF), where rows returning FALSE are excluded.
* [ABAC Row Filter Policy](/concepts/abac-row-filter-policy.md) - A Unity Catalog ABAC policy that filters rows from query results using a boolean user-defined function (UDF), excluding rows where the function returns FALSE.
* [ABAC Separation of Duties](/concepts/abac-separation-of-duties.md) - The role-based permission model for ABAC that distributes responsibilities across specialized teams (tag taxonomy creators, data stewards, governance admins, data creators, data consumers) with distinct permission requirements per role.
* [ABAC Tag Governance and Standardization](/concepts/abac-tag-governance-and-standardization.md) - Establishing a consistent, controlled tagging taxonomy for ABAC policies with restricted tag permissions
* [ABAC Tagging as a Security Boundary](/concepts/abac-tagging-as-a-security-boundary.md) - The principle that tag mutation permissions are a security boundary since tags determine which ABAC policies apply to a data asset
* [ABAC Tagging Taxonomy and Governance](/concepts/abac-tagging-taxonomy-and-governance.md) - Establishing consistent, controlled tag taxonomies with standardized key names and allowed values, and restricting tag modification to authorized stewards.
* [ABAC User-Defined Functions (UDFs)](/concepts/abac-user-defined-functions-udfs.md) - Row filter and column mask policies rely on UDFs (SQL or Python) registered in Unity Catalog to implement filtering or masking logic, with SQL UDFs recommended for better performance.
* [ABAC View Sharing Ownership Change (April 2026)](/concepts/abac-view-sharing-ownership-change-april-2026.md) - A policy change effective April 23, 2026, shifting the exemption requirement from the view owner to the share owner for sharing views protected by ABAC policies, with a grace period until July 22, 2026.
* [Acceptable Use Models Policy](/concepts/acceptable-use-models-policy.md) - Databricks policy governing which AI/ML models are allowed to be served on the Databricks Model Serving platform
* [Acceptable Use Policy for ML Models](/concepts/acceptable-use-policy-for-ml-models.md) - Policies and terms governing which types of machine learning models are permissible for deployment on Databricks Model Serving.
* [Access Control Differences Between Unity Catalog and Hive Metastore](/concepts/access-control-differences-between-unity-catalog-and-hive-metastore.md) - Unity Catalog and the legacy Hive metastore have differing access control models, including metastore scope (account-level vs workspace-level), permission requirements (USE CATALOG/USE SCHEMA), view ownership rules, and support for DENY, ANY FILE, ANONYMOUS FUNCTION, and READ_METADATA.
* [Access Control for Data Quality Monitoring Results](/concepts/access-control-for-data-quality-monitoring-results.md) - Only account admins can access the system.data_quality_monitoring.table_results table and must explicitly grant access to other users.
* [Access Control for MLflow System Tables](/concepts/access-control-for-mlflow-system-tables.md) - By default only account admins can access system tables; they can grant USE and SELECT permissions to other users, who then see metadata across all experiments in the account.
* [Access Control Separation for PII-Containing Tables](/concepts/access-control-separation-for-pii-containing-tables.md) - A security pattern in Unity Catalog where raw tables containing unredacted PII have tightly restricted access (only pipeline service principals and administrators), while redacted tables are broadly available for analytics, dashboards, and observability.
* [Access Expiration via Mapping Tables](/concepts/access-expiration-via-mapping-tables.md) - Using an expires_on date column in a mapping table to automatically revoke access without manual intervention, useful for contractors and time-limited data sharing.
* [Access Request Approval Methods](/concepts/access-request-approval-methods.md) - Two methods for approving requests: adding the requester to an existing group with privileges, or directly granting privileges to the principal.
* [Access Request Destination Inheritance](/concepts/access-request-destination-inheritance.md) - Destinations configured at higher levels of the Unity Catalog hierarchy automatically apply to child objects that lack explicit destinations.
* [Access Request Destinations](/concepts/access-request-destinations.md) - Configured endpoints (email, Slack, Teams, webhook, redirect URL) that receive privilege requests for Unity Catalog securable objects.
* [Access Requests in Unity Catalog](/concepts/access-requests-in-unity-catalog.md) - Configurable destinations (email, Slack, Teams, webhooks) for access requests on Unity Catalog securable objects.
* [ACCESS_DENIED (Delta Uniform error)](/concepts/access_denied-delta-uniform-error.md) - A sub-error of DELTA_UNIFORM_COMPATIBILITY_MISSING_OR_INVALID_LOCATION indicating the location cannot be accessed.
* [ACCESS_DENIED sub-error (Delta Uniform Compatibility)](/concepts/access_denied-sub-error-delta-uniform-compatibility.md) - A specific reason code under the DELTA_UNIFORM_COMPATIBILITY_MISSING_OR_INVALID_LOCATION error class indicating the configured location cannot be accessed due to permissions
* [Accessing Data Profiling Dashboards in Databricks](/concepts/accessing-data-profiling-dashboards-in-databricks.md) - Two navigation paths to locate the auto-generated data profiling dashboard: via Catalog Explorer (Quality tab on a table) or via the Dashboards sidebar listing.
* [Accessing Databricks secrets in scorers](/concepts/accessing-secrets-in-scorers.md) - Custom scorers can securely use Databricks secrets by importing dbutils from databricks.sdk.runtime inside the scorer function.
* [Accessing Databricks Utilities via Databricks Connect](/concepts/accessing-databricks-utilities-via-databricks-connect.md) - How to use dbutils.fs and dbutils.secrets utilities through Databricks Connect, including file copy operations between local and remote filesystems and accessing secrets.
* [Accessing secrets in scorers](/concepts/accessing-secrets-in-scorers.md) - How to securely use Databricks secrets within custom scorers by importing dbutils from databricks.sdk.runtime.
* [Accessing the Data Profiling Dashboard](/concepts/accessing-the-data-profiling-dashboard.md) - Two methods to view the dashboard: via Catalog Explorer (Quality tab on the primary table) or via the Dashboards sidebar.
* [Account Admin (Unity Catalog)](/concepts/account-admin-unity-catalog.md) - A highly privileged role operating at the Databricks account level, responsible for creating and linking metastores and workspaces, and assigning admin roles.
* [Account Admin Role (Unity Catalog)](/concepts/account-admin-unity-catalog.md) - A highly privileged role operating at the Databricks account level, responsible for creating and linking metastores and workspaces, and assigning admin roles.
* [Account Admin Role in Unity Catalog](/concepts/account-admin-unity-catalog.md) - A highly privileged role operating at the Databricks account level, responsible for creating and linking metastores and workspaces, and assigning admin roles.
* [Account admins (Unity Catalog)](/concepts/account-admin-unity-catalog.md) - Highly privileged role operating at the Databricks account level, responsible for creating and linking metastores and workspaces, and assigning admin roles.
* [Account Admins in Unity Catalog](/concepts/account-admin-unity-catalog.md) - Account-level admin role in Databricks that creates and links metastores and workspaces, assigns admin roles, and operates across the entire Databricks account.
* [Account-Level Identity Federation for Unity Catalog](/concepts/account-level-identity-management-for-unity-catalog.md) - Enabling Unity Catalog shifts identity management from workspace-level to account-level interfaces, requiring SCIM provisioning connectors and automation to target account endpoints.
* [Account-Level Identity Management for Unity Catalog](/concepts/account-level-identity-management-for-unity-catalog.md) - The shift from workspace-level to account-level management of users, groups, and service principals when Unity Catalog is enabled, requiring updates to SCIM and automation.
* [Account-Level Identity Provisioning for Unity Catalog](/concepts/account-level-identity-provisioning-for-unity-catalog.md) - The requirement to provision users, groups, and service principals at the Databricks account level rather than workspace level for Unity Catalog compatibility.
* [Account-Level Legacy Feature Settings](/concepts/account-level-legacy-feature-settings.md) - An account-level setting to disable legacy features, including the Hive metastore, for new workspaces before they are created
* [ACID Transactions in Data Lakes](/concepts/acid-transactions-in-data-lakes.md) - The capability of Delta Lake to provide ACID (Atomicity, Consistency, Isolation, Durability) transaction guarantees on data stored in data lakes.
* [ACID Transactions on Data Lakes](/concepts/acid-transactions-in-data-lakes.md) - Delta Lake provides ACID (Atomicity, Consistency, Isolation, Durability) transaction guarantees on top of existing data lakes, bringing database-like reliability to data lake storage.
* [ACL_ENABLED Error on Delta Writes](/concepts/acl_enabled-error-on-delta-writes.md) - A DELTA_METADATA_MISMATCH sub-error that occurs when Table ACLs are enabled on a cluster, preventing automatic schema migration and requiring ALTER TABLE commands instead.
* [ACL_ENABLED sub-error](/concepts/acl_enabled-sub-error.md) - A DELTA_METADATA_MISMATCH sub-error that occurs when table ACLs are enabled, preventing automatic schema migration
* [Activation Checkpointing](/concepts/activation-checkpointing.md) - A memory optimization technique that trades compute for memory by not storing intermediate activations during forward pass and recomputing them during backward pass.
* [Activation Link (Delta Sharing)](/concepts/activation-link-delta-sharing.md) - A secure URL generated when creating an Open Sharing recipient, used by non-Databricks recipients to download their credential file once for authentication.
* [Activation Link and Credential File Distribution](/concepts/activation-link-and-credential-file-distribution.md) - The mechanism by which a Delta Sharing data provider shares connection information with a recipient via a secure activation link that allows one-time download of a credential file containing the bearer token.
* [Activation Link for Delta Sharing Recipients](/concepts/activation-link-delta-sharing.md) - A one-time-use URL generated by Databricks when creating or rotating a recipient token, allowing the recipient to download their credential file and access shared data.
* [Active experiment configuration in MLflow](/concepts/active-experiment-in-mlflow.md) - Methods to control where MLflow runs are logged by setting the active experiment using set_experiment(), start_run(), or environment variables.
* [Active Experiment in MLflow](/concepts/active-experiment-in-mlflow.md) - Mechanism to control where MLflow runs are logged by setting an active experiment via set_experiment(), start_run(), or environment variables.
* [Active Maintenance Definition for Libraries](/concepts/active-maintenance-definition-for-libraries.md) - Criteria Databricks uses to determine if a library is actively maintained: commits within 3 months and releases within 9 months, no archived repository, and no announced stop of maintenance.
* [Active MLflow run management](/concepts/active-mlflow-run-management.md) - The practice of wrapping fit() calls inside a 'with mlflow.start_run():' statement to ensure tuning information is logged under its own MLflow main run for better organization.
* [Active MLflow Run Management in Tuning](/concepts/active-mlflow-run-management-in-tuning.md) - Best practice of wrapping CrossValidator or TrainValidationSplit fit() calls inside a 'with mlflow.start_run():' context to ensure clean parent runs and avoid UUID name conflicts
* [Active MLflow Run Management with fit()](/concepts/active-mlflow-run-management-with-fit.md) - The recommended practice of wrapping CrossValidator/TrainValidationSplit fit() calls inside a 'with mlflow.start_run():' statement to ensure proper logging under a dedicated MLflow main run and enable additional custom logging.
* [Active MLflow run management with Hyperopt](/concepts/active-mlflow-run-management-with-hyperopt.md) - Databricks recommends wrapping fmin() calls in an active MLflow run to ensure each call is logged to a separate main run and avoid name conflicts.
* [Adapting Single-Node PyTorch to Distributed Training](/concepts/adapting-single-node-pytorch-to-distributed-training.md) - The process of modifying PyTorch code originally written for single-node training to run in a distributed manner across multiple nodes
* [Add Traces to Labeling Session](/concepts/add-traces-to-labeling-session.md) - The workflow for adding MLflow Traces (via UI selection, search results, or individual trace objects) into a labeling session for expert review.
* [addCompiledArtifacts API](/concepts/addcompiledartifacts-api.md) - API in Databricks Connect for Scala that uploads compiled classes and JARs from local development to a remote Databricks cluster so that UDFs and typed Dataset operations can execute on the server side.
* [Adding Traces to Labeling Sessions](/concepts/adding-traces-to-labeling-sessions.md) - The process of selecting and associating existing MLflow Traces (via UI or API) with a labeling session for expert review, supporting bulk selection from search results and individual trace objects.
* [Admin Consent for OIDC Delta Sharing Apps](/concepts/admin-consent-for-oidc-delta-sharing-apps.md) - One-time admin approval required in Entra ID tenant to authorize the Databricks multi-tenant app for OIDC federation
* [Admin-Side Data Discovery](/concepts/admin-side-data-discovery.md) - Administrator tools and configurations in Unity Catalog that shape data discoverability through metadata, trust signals, and organization, directly determining the quality of the user discovery experience.
* [Admin-side Metadata Governance](/concepts/admin-side-metadata-governance.md) - Administrators shape data discoverability by applying metadata, trust signals, and organizational structures to catalog objects.
* [Administrator's Role in Data Discoverability](/concepts/administrators-role-in-data-discoverability.md) - Administrators shape discoverability by applying metadata, trust signals, and organization to make it easier for users to locate and assess data quality.
* [Advanced AutoML Configuration Options](/concepts/advanced-automl-configuration-options.md) - Advanced parameters for AutoML experiments including evaluation metrics, training framework exclusion, stopping conditions (time-based or early stopping), and time-based data splitting.
* [Advanced Workflow Tracing](/concepts/advanced-workflow-tracing.md) - Manual tracing support for complex production scenarios including multi-threaded operations, async workflows, streaming responses with custom aggregation, and complex nested operations.
* [AG2 Multi-Agent Termination with Message Content](/concepts/ag2-multi-agent-termination-with-message-content.md) - A pattern where a conversational agent workflow terminates when a specific string (e.g., 'TERMINATE') appears in a message's content, controlled via the is_termination_msg callback.
* [Agent Bricks Knowledge Assistant Incompatibility](/concepts/agent-bricks-knowledge-assistant-incompatibility.md) - A known limitation where Knowledge Assistant does not support foundation model Unity Catalog permissions, requiring users to contact their account team before enabling the feature.
* [Agent Evaluation Dataset Design](/concepts/agent-evaluation-dataset-design.md) - Evaluation datasets for GenAI agents consist of input message histories and optional `expectations` dictionaries that define expected agent behaviors, enabling both input/output evaluation and correctness checking.
* [Agent Function Interface](/concepts/agent-function-interface.md) - The predict_fn contract for conversational agents being evaluated, accepting conversation history as a list of message dicts and returning a string response.
* [Agent Function Interface for MLflow Evaluation](/concepts/agent-function-interface.md) - The predict_fn contract that wraps a conversational agent for use with MLflow evaluation, accepting conversation history and returning responses.
* [Agent Function Interface for Simulated Conversations](/concepts/agent-function-interface-for-simulated-conversations.md) - The predict_fn interface that receives conversation history and context (including mlflow_session_id and custom context fields) and returns assistant responses.
* [Agent Function Interface for Simulation](/concepts/agent-function-interface.md) - The predict_fn interface that conversational AI agents must implement to participate in simulation, accepting conversation history as a list of message dicts (supports 'input' or 'messages' parameter names) and returning a string response.
* [Agent Labeling and Review](/concepts/agent-labeling-and-review.md) - Workflows for viewing labeling sessions, assigning reviewers to traces, and inspecting labeling schemas for feedback criteria such as ratings and comments
* [Agent Mode](/concepts/agent-mode.md) - A toggle in the Genie Code pane that enables interactive, prompt-driven troubleshooting and analysis of model serving endpoints.
* [Agent Tool Registration Pattern](/concepts/agent-tool-registration-pattern.md) - A design pattern in AG2 where a function is registered twice — once for the LLM to call (register_for_llm) and once for execution (register_for_execution) — decoupling tool discovery from secure execution.
* [Agent-Based System Observability](/concepts/agent-based-system-observability.md) - Observability techniques tailored for complex agent-based AI systems, including tracking intermediate steps and decision chains.
* [Aggregate Metrics](/concepts/aggregate-metrics.md) - Summary statistics computed across all evaluated examples in an evaluation run, such as mean correctness or safety pass rate.
* [Aggregate Metrics (MLflow Evaluation)](/concepts/aggregate-metrics.md) - Summary statistics computed across all evaluated examples in an evaluation run, such as mean correctness or pass rates.
* [Aggregate Metrics in MLflow Evaluation](/concepts/aggregate-metrics.md) - Summary statistics computed across all evaluated examples in an evaluation run, such as mean correctness, mean relevance, and safety pass rate.
* [Aggregation Functions and Column Selection](/concepts/aggregation-functions-and-columnselection.md) - Features can be computed via aggregation functions (Sum, Avg, Count, ApproxCountDistinct, etc.) wrapped in AggregationFunction with a time window, or pass-through via ColumnSelection for direct value lookup.
* [Aggregation Functions and ColumnSelection](/concepts/aggregation-functions-and-columnselection.md) - Two function types for feature computation: AggregationFunction (wrapping operators like Sum, Avg, Count with time windows) for time-series aggregations, and ColumnSelection for pass-through features without aggregation.
* [AggregationFunction and ColumnSelection](/concepts/aggregationfunction-and-columnselection.md) - Two types of function wrappers for feature computation: AggregationFunction for window-based aggregation (Sum, Avg, Count, ApproxCountDistinct) and ColumnSelection for pass-through features
* [AI Assets as Securable Objects](/concepts/ai-assets-as-securable-objects.md) - AI resources (models, functions, connections, hosted foundation models) managed as securable objects in Unity Catalog with standard privileges for grant and revoke operations.
* [AI Code Generation Human Review Requirement](/concepts/ai-code-generation-human-review-requirement.md) - A safety requirement that all AI-generated code must undergo human review before production use, with additional restrictions for external users.
* [AI Functions](/concepts/ai-functions.md) - SQL-based functions tightly integrated with Model Serving for batch inference, allowing users to call models directly from SQL analytics workflows without configuring endpoints for pre-provisioned models.
* [AI Functions (Databricks)](/concepts/ai-functions-databricks.md) - Built-in SQL functions on Databricks that enable applying AI models directly to data using task-specific or general-purpose functions like ai_query
* [AI Functions and Batch Inference](/concepts/ai-functions.md) - Databricks SQL functions (including ai-query) tightly integrated with Model Serving to perform batch inference using pre-provisioned or custom models directly within analytics workflows.
* [AI Functions for Batch Inference](/concepts/ai-functions-for-batch-inference.md) - Batch inference mode using AI Functions within Databricks, recommended for running inference on large datasets using generative AI or ML models.
* [AI Functions for PII Redaction](/concepts/ai-functions-for-pii-redaction.md) - Built-in Databricks SQL functions (ai_mask) that use foundation models to detect and mask personally identifiable information in free-text columns.
* [AI Functions on Databricks](/concepts/ai-functions.md) - SQL-callable functions tightly integrated with Model Serving for batch inference, allowing users to enrich data using pre-provisioned or custom models directly from analytics workflows.
* [AI Gateway](/concepts/ai-gateway.md) - A feature for centrally managing permissions, usage limits, and monitoring quality across all model endpoints including externally hosted ones
* [AI Gateway (Databricks)](/concepts/ai-gateway-databricks.md) - Centralized governance layer for managing, monitoring, and setting usage limits on all model endpoints (including externally hosted ones) from the Serving UI.
* [AI Gateway Inference Tables](/concepts/ai-gateway-inference-tables.md) - Automatically logs online prediction requests and responses into Unity Catalog Delta tables for monitoring, debugging, training data generation, and compliance audits.
* [AI Gateway on Databricks](/concepts/ai-gateway-on-databricks.md) - Databricks service to govern and monitor access to deployed models with usage tracking, payload logging, and security controls.
* [AI Governance](/concepts/ai-governance.md) - Extending data governance frameworks to AI resources, applying access control, lineage, and audit models to AI assets and AI traffic.
* [AI Governance (Unity Catalog)](/concepts/ai-governance-unity-catalog.md) - Extension of Unity Catalog's data governance capabilities to AI assets, applying access control, lineage, and audit models to AI resources.
* [AI Governance on Databricks](/concepts/ai-governance-on-databricks.md) - Extends data governance capabilities of Unity Catalog to AI resources, applying access control, lineage, and audit models to AI assets and AI traffic.
* [AI Governance with Unity AI Gateway](/concepts/ai-governance-with-unity-ai-gateway.md) - Extension of Unity Catalog governance to AI, providing enterprise governance for LLM endpoints, agents, and MCP servers
* [AI Governance with Unity Catalog](/concepts/ai-governance-unity-catalog.md) - Extending Databricks' data governance capabilities (access control, lineage, audit) to AI assets such as models, functions, connections, and hosted foundation models using Unity Catalog.
* [AI Playground](/concepts/ai-playground.md) - A Databricks interface for interactively experimenting with supported foundation models before deploying them into production workloads.
* [AI Playground (Databricks)](/concepts/ai-playground-databricks.md) - Chat-like environment within the Databricks workspace for testing, prompting, and comparing supported large language models before deployment.
* [AI Playground on Databricks](/concepts/ai-playground-databricks.md) - A chat-like environment within Databricks workspaces for testing, prompting, and comparing large language models interactively before deploying them via Model Serving.
* [AI Runtime](/concepts/ai-runtime.md) - Databricks' on-demand serverless GPU compute platform for distributed training workloads.
* [AI Runtime (AI v5) on Databricks](/concepts/ai-runtime-ai-v5-on-databricks.md) - Databricks' pre-configured environment for AI workloads, bundling Unsloth, TRL, PEFT, bitsandbytes, xformers, and other LLM finetuning dependencies.
* [AI Runtime (air) CLI Distributed Workflows](/concepts/ai-runtime-air-cli-distributed-workflows.md) - Databricks' CLI tool and YAML-based workload configuration system for defining and submitting multi-node distributed training jobs, with built-in secret injection, MLflow integration, Unity Catalog volume access, and automatic rendezvous variable injection.
* [AI Runtime (air) CLI Workload YAML](/concepts/ai-runtime-air-cli-workload-yaml.md) - A YAML configuration format for the Databricks AI Runtime CLI that defines distributed training jobs with compute requirements, environment dependencies, code sources, and inline bootstrap commands.
* [AI Runtime (Databricks)](/concepts/ai-runtime.md) - Databricks' on-demand serverless GPU compute platform designed for machine learning training workloads
* [AI Runtime and Databricks Runtime for Machine Learning](/concepts/ai-runtime-and-databricks-runtime-for-machine-learning.md) - Customizable training environments with GPU support that enable training and fine-tuning across classic ML, deep learning, and GenAI paradigms on Databricks.
* [AI Runtime Base Environments](/concepts/ai-runtime-base-environments.md) - The two base environment options for AI Runtime: None (default environment without AI libraries) and the AI environment (databricks_ai_v4/v5) which includes pre-installed AI and ML libraries.
* [AI Runtime CLI](/concepts/ai-runtime-cli.md) - A Databricks command-line interface for submitting, monitoring, and managing machine learning training workloads on serverless GPU infrastructure.
* [AI Runtime CLI (air)](/concepts/ai-runtime-cli-air.md) - Command-line interface for submitting and managing machine learning workloads on Databricks, currently in Beta.
* [AI Runtime CLI (Databricks)](/concepts/ai-runtime-cli.md) - A command-line interface for submitting and managing machine learning training workloads on Databricks serverless GPU infrastructure, currently in Beta.
* [AI Runtime CLI authentication profiles](/concepts/ai-runtime-cli-authentication-profiles.md) - The air CLI uses Databricks authentication profiles via the -p/--profile flag, allowing users to select a named credential profile instead of the default.
* [AI Runtime CLI Authentication via Databricks CLI Profiles](/concepts/ai-runtime-cli-authentication-via-databricks-cli-profiles.md) - The air CLI reuses Databricks CLI authentication profiles, which can be passed via the -p flag or the DATABRICKS_CONFIG_PROFILE environment variable.
* [AI Runtime CLI Commands](/concepts/ai-runtime-cli-commands.md) - The set of CLI commands (air run, air logs, air list runs, air cancel) used to submit, monitor, and manage training jobs.
* [AI Runtime CLI global flags](/concepts/ai-runtime-cli-global-flags.md) - Cross-cutting flags available to all air CLI commands, including --version, --profile (-p), and --help (-h).
* [AI Runtime CLI Installation via uv](/concepts/ai-runtime-cli-installation-via-uv.md) - The recommended method to install the air CLI is using the uv Python package manager, which isolates the tool in its own environment.
* [AI Runtime CLI Log Management](/concepts/ai-runtime-cli-log-management.md) - Commands and patterns for streaming, filtering by node, downloading to local directory, and inspecting logs from AI Runtime training runs.
* [AI Runtime CLI YAML configuration help system](/concepts/ai-runtime-cli-yaml-configuration-help-system.md) - Built-in help system that provides per-field documentation for YAML configuration sections via 'air -h config' and 'air -h config.compute'.
* [AI Runtime CLI YAML workload definition](/concepts/ai-runtime-cli-yaml-workload-definition.md) - Workloads for the AI Runtime CLI are defined via YAML configuration files, with the CLI offering field-level help (-h config) and per-section help (-h config.compute) for authoring these files.
* [AI Runtime connection modes](/concepts/ai-runtime-connection-modes.md) - Three primary ways to connect to AI Runtime: interactive notebooks, scheduled jobs, and Jobs API/Databricks Asset Bundles.
* [AI Runtime Connectivity Methods](/concepts/ai-runtime-connectivity-methods.md) - Ways to connect to AI Runtime including interactive notebooks, scheduled jobs, Jobs API, and Databricks Asset Bundles.
* [AI Runtime Data Access and Integration](/concepts/ai-runtime-data-access-and-integration.md) - Data loading patterns and native integration points for AI Runtime including notebooks, jobs, Unity Catalog, MLflow experiment tracking, and Genie Code AI assistant for deep learning workloads.
* [AI Runtime Data Loading](/concepts/ai-runtime-data-loading.md) - The mechanism for reading data into AI Runtime workloads, requiring an understanding of how data access works on the serverless platform for a smooth experience.
* [AI Runtime Distributed Training](/concepts/ai-runtime-distributed-training.md) - A Beta API for multi-GPU training on AI Runtime using the @distributed decorator from the serverless_gpu Python package, supporting PyTorch DDP, FSDP, and DeepSpeed with minimal configuration on a single multi-GPU node.
* [AI Runtime Distributed Training API](/concepts/ai-runtime-distributed-training.md) - A Beta API using the @distributed decorator for multi-GPU workloads on single-node AI Runtime with PyTorch DDP, FSDP, or DeepSpeed.
* [AI Runtime environment configuration](/concepts/ai-runtime-environment-configuration.md) - Configuration of AI Runtime involves selecting a serverless GPU compute, an accelerator type, and a base environment (None/default or AI v4/v5).
* [AI Runtime Environments](/concepts/ai-runtime-environments.md) - Two managed Python environments for AI Runtime: a minimal default base environment for maximum dependency flexibility, and a full-featured Databricks AI environment pre-loaded with popular ML frameworks like PyTorch and Transformers.
* [AI Runtime example notebooks](/concepts/ai-runtime-example-notebooks.md) - Curated collection of notebook examples demonstrating AI Runtime usage for LLMs, computer vision, recommender systems, classic ML, and distributed training.
* [AI Runtime Experiment Tracking and Observability](/concepts/ai-runtime-experiment-tracking-and-observability.md) - Integration with MLflow for experiment tracking, log viewing, and model checkpoint management on AI Runtime
* [AI Runtime for Computer Vision](/concepts/ai-runtime-for-computer-vision.md) - Databricks AI Runtime provides a managed environment with pre-configured libraries and serverless GPUs for training and fine-tuning computer vision models on AWS.
* [AI Runtime for Recommender Systems](/concepts/ai-runtime-for-recommender-systems.md) - Databricks AI Runtime provides infrastructure for building deep learning based recommendation systems, supporting both single-node (Public Preview) and multi-GPU distributed training (Beta).
* [AI Runtime for single-node tasks](/concepts/ai-runtime-for-single-node-tasks.md) - Public Preview feature of Databricks AI Runtime for running single-node machine learning workloads on GPU instances.
* [AI Runtime Genie Code Support](/concepts/ai-runtime-genie-code-support.md) - Integration with Genie Code to assist with deep learning code generation, library debugging, optimization suggestions on AI Runtime.
* [AI Runtime Hardware Accelerators](/concepts/ai-runtime-hardware-options.md) - The GPU hardware options available for AI Runtime on AWS, including A10 and H100 accelerators (single-node, multi-GPU), with region-specific availability and capacity considerations.
* [AI Runtime Hardware Options](/concepts/ai-runtime-hardware-options.md) - The supported GPU accelerators for AI Runtime, currently limited to A10 and H100 types, each provisioned on a single node with a specific number of GPUs depending on the accelerator type.
* [AI Runtime idle auto-termination](/concepts/ai-runtime-idle-auto-termination.md) - AI Runtime compute auto-terminates after 60 minutes of inactivity to preserve resources.
* [AI Runtime Interactive Notebook Connection](/concepts/ai-runtime-interactive-notebook-connection.md) - How to connect a Databricks notebook to an AI Runtime environment using serverless GPU compute via the Environment side panel.
* [AI Runtime Jobs API and Asset Bundles](/concepts/ai-runtime-jobs-api-and-asset-bundles.md) - Programmatic creation and management of AI Runtime jobs using the Databricks Jobs API or Databricks Asset Bundles with YAML configuration for serverless GPU compute.
* [AI Runtime Limitations and Constraints](/concepts/ai-runtime-limitations-and-constraints.md) - Key operational constraints for AI Runtime including no support for compliance security profiles (HIPAA/PCI), maximum 7-day runtime, limited accelerator types (A10/H100 only), potential capacity constraints, and cross-region egress costs.
* [AI Runtime Local Caching for Volumes](/concepts/ai-runtime-local-caching-for-volumes.md) - Mechanism where files from Unity Catalog volumes are copied to local storage on first access, avoiding repeated FUSE reads across epochs.
* [AI Runtime Logging](/concepts/ai-runtime-logging.md) - AI Runtime provides two logging channels: notebook cell output for stdout/stderr and the MLflow experiment UI for training metrics, parameters, and artifacts.
* [AI Runtime Managed Environments](/concepts/ai-runtime-environments.md) - Two Python environments offered by AI Runtime: a minimal default base environment and a full-featured Databricks AI environment pre-loaded with ML frameworks
* [AI Runtime MLflow Integration](/concepts/ai-runtime-mlflow-integration.md) - Native integration of MLflow with Databricks AI Runtime for experiment tracking, model logging, and metric visualization, including autologging, custom run naming, and experiment configuration.
* [AI Runtime Model Checkpointing](/concepts/ai-runtime-model-checkpointing.md) - Manual checkpointing mechanism using UCVolumeWriter and UCVolumeReader from the serverless_gpu.data module to enable resumption of workloads that may exceed the 7-day maximum runtime.
* [AI Runtime model checkpointing with Unity Catalog](/concepts/model-checkpointing-with-unity-catalog.md) - For long-running workloads exceeding 7 days, implement manual checkpointing using UCVolumeWriter and UCVolumeReader from serverless_gpu.data with Unity Catalog volumes.
* [AI Runtime on Databricks](/concepts/ai-runtime-on-databricks.md) - A Databricks runtime optimized for AI and machine learning workloads, supporting single-node tasks (Public Preview) and multi-GPU distributed training (Beta).
* [AI Runtime programmatic job management](/concepts/ai-runtime-programmatic-job-management.md) - AI Runtime jobs can be created and managed programmatically using the Databricks Jobs API or Databricks Asset Bundles with YAML configuration specifying serverless GPU compute and environment settings.
* [AI Runtime Python API](/concepts/ai-runtime-python-api.md) - The in-notebook Python API (`@distributed` and `@ray_launch`) for multi-GPU workload training, an alternative to the CLI.
* [AI Runtime Regional Availability and Cross-Region GPUs](/concepts/ai-runtime-regional-availability-and-cross-region-gpus.md) - AI Runtime availability limited to specific AWS regions (us-west, us-east, ca-central-1, sa-east-1) with potential cross-region GPU spillover during high demand, which may incur egress costs.
* [AI Runtime scheduled job limitations](/concepts/ai-runtime-scheduled-job-limitations.md) - Scheduled jobs using serverless GPU have key limitations: no Environments panel for dependencies (must use %pip install), no auto-recovery on failure, and a 7-day maximum runtime requiring manual checkpointing.
* [AI Runtime Scheduled Jobs](/concepts/ai-runtime-scheduled-jobs.md) - Scheduling notebooks that use serverless GPU as recurring jobs with limitations on dependency management and auto-recovery.
* [AI Runtime Tabular Data Loading with Spark Connect](/concepts/ai-runtime-tabular-data-loading-with-spark-connect.md) - Loading tabular ML data from Delta tables using Spark Connect, with conversion to pandas/NumPy via toPandas() and to_numpy().
* [AI Runtime Usage Constraints](/concepts/ai-runtime-usage-constraints.md) - Key operational constraints for AI Runtime: 60-minute auto-termination on inactivity, no Environments panel for scheduled jobs, no auto-recovery on failure, 7-day maximum job runtime, and required manual checkpointing for long-running workloads.
* [AI Runtime Use Cases](/concepts/ai-runtime-use-cases.md) - Recommended workloads for AI Runtime including LLM fine-tuning, computer vision, recommender systems, reinforcement learning, and time series forecasting
* [AI Runtime Workload YAML Configuration](/concepts/ai-runtime-workload-yaml-configuration.md) - A YAML-based configuration format that defines training workloads including experiment name, compute specs, code source, environment dependencies, and execution command.
* [AI Traffic Governance](/concepts/ai-traffic-governance.md) - The practice of controlling access, enforcing rate limits, and tracking usage and costs for LLM endpoints and MCP servers through a central gateway.
* [ai_mask Function](/concepts/ai_mask-function.md) - An LLM-backed SQL function in Databricks that detects and masks standard PII types (email, phone, name, address, SSN, credit card, IP address, date of birth) across varied formats without requiring separate regex patterns.
* [ai_query function](/concepts/ai_query-function.md) - A general-purpose AI function in Databricks SQL for running batch inference against supported foundation models
* [AI-Assisted Development (Genie Code)](/concepts/ai-assisted-development-genie-code.md) - An integrated AI assistant for development, debugging, and operations across the ML workflow on Databricks
* [AI-Assisted Development with Genie Code](/concepts/ai-assisted-development-genie-code.md) - An AI assistant integrated across Databricks notebooks and workspace that helps with development, debugging, and operations using enterprise-context knowledge at every step of the ML workflow.
* [air cancel command](/concepts/air-cancel-command.md) - Cancels a running workload, enabling lifecycle management of submitted jobs
* [air CLI](/concepts/air-cli.md) - Command-line interface for submitting and managing distributed training workloads on Databricks AI Runtime, using `air run -f train.yaml` as the primary submission command.
* [air CLI (Databricks AI Runtime CLI)](/concepts/air-cli-databricks-ai-runtime-cli.md) - Command-line tool for submitting and managing distributed AI/ML training workloads on Databricks infrastructure
* [air get run / air list runs commands](/concepts/air-get-run-air-list-runs-commands.md) - Subcommands for querying run metadata, status, and configuration; list supports --limit and --active flags to filter results.
* [air get run and air list runs commands](/concepts/air-get-run-and-air-list-runs-commands.md) - Commands to query run metadata, status, and configuration, with filtering options like --limit and --active
* [air get run command](/concepts/air-get-run-command.md) - Displays metadata, status, and configuration details for a specific AI Runtime run.
* [air list runs command](/concepts/air-list-runs-command.md) - Lists recent AI Runtime runs with optional filters for result count and active-only status.
* [AIR Logs and Run Inspection](/concepts/air-logs-and-run-inspection.md) - Commands and patterns for inspecting training runs: checking status, streaming/downloading logs (with per-node support), listing active/recent runs, and canceling runs.
* [air logs command](/concepts/air-logs-command.md) - Streams or downloads logs for a run, with targeting of specific nodes via --node and directory download via --download-to
* [air run - Run management commands](/concepts/air-run-run-management-commands.md) - Commands for inspecting, listing, canceling, and retrieving logs from AI Runtime runs, including 'air get run', 'air list runs', 'air logs', and 'air cancel'.
* [air run - Workload submission](/concepts/air-run-workload-submission.md) - The primary CLI command for submitting AI workloads defined by YAML configuration files to Databricks AI Runtime.
* [air run -f train.yaml](/concepts/air-run-f-trainyaml.md) - Primary command pattern for launching distributed training workloads via the Databricks AI Runtime CLI using a YAML workload definition file
* [air run command](/concepts/air-run-command.md) - The primary CLI command used to submit AI training workloads, with support for YAML config files, field overrides, dry-run validation, idempotency keys, and log watching.
* [AIR Run Submission and Lifecycle](/concepts/air-run-submission-and-lifecycle.md) - The workflow for submitting training runs via `air run`, including idempotency keys, dry-run validation, field overrides, and run lifecycle tracking (status, logs, cancellation).
* [air run workflow](/concepts/air-run-workflow.md) - Mechanism to submit distributed training jobs by specifying a YAML configuration file via the `air run -f train.yaml` command
* [AIR Workload YAML Configuration](/concepts/air-workload-yaml-configuration.md) - A YAML-based configuration format (train.yaml) that defines experiment name, compute spec, command, environment dependencies, and code source for AI Runtime CLI training jobs.
* [Alert Notification Channels](/concepts/alert-notification-channels.md) - Supported destinations for profile alert notifications including email, webhook, Slack, and PagerDuty.
* [Alert Query Parameter Defaults](/concepts/alert-query-parameter-defaults.md) - The behavior where profile alerts use default parameter values when the underlying query includes parameters, requiring confirmation that defaults reflect alert intent.
* [Alert Rule Permissions Model](/concepts/alert-rule-permissions-model.md) - Privilege requirements for creating and managing anomaly detection alert rules: MANAGE privilege on the schema for schema-level alerts, MANAGE privilege on the catalog for catalog-level alerts.
* [Alert Rule Scope](/concepts/alert-rule-scope.md) - Alert rules in Databricks are scoped to either a specific schema or an entire catalog; when a monitored table within the scope becomes unhealthy, notifications are sent to designated recipients.
* [Alert Rule Scoping (Catalog vs Schema)](/concepts/alert-rule-scoping-catalog-vs-schema.md) - Alert rules in Unity Catalog Data Quality Monitoring can be scoped to an entire catalog (all schemas) or a specific schema, controlling which monitored tables are covered.
* [Alert Rule Scoping and Permissions](/concepts/alert-rule-scoping-and-permissions.md) - Alert rules in Databricks Data Quality Monitoring are scoped to a catalog or schema, with MANAGE privilege required on the scoped object to create or manage rules.
* [Alerting on Fairness Metric Thresholds](/concepts/alerting-on-fairness-metric-thresholds.md) - The practice of setting up automated alerts when fairness metrics exceed defined thresholds, enabling teams to investigate and remediate bias
* [Alias-based model lifecycle in Unity Catalog](/concepts/alias-based-model-lifecycle-in-unity-catalog.md) - Replacement for fixed stages (Staging, Production, etc.) in the Workspace Model Registry — up to 10 custom, reassignable aliases can be used to reference specific model versions.
* [Aligning Judges with Human Experts](/concepts/aligning-judges-with-human-experts.md) - The process of improving LLM judge accuracy by iteratively aligning judge behavior with human expert feedback on application outputs.
* [Aligning LLM Judges with Human Experts](/concepts/aligning-judges-with-human-experts.md) - A process to improve judge accuracy by aligning LLM judge behavior with expert human feedback on application outputs.
* [Alignment Optimizer Framework](/concepts/alignment-optimizer-framework.md) - An extensible framework for judge alignment including the default MemAlign optimizer and the AlignmentOptimizer base class for custom optimization strategies.
* [Alignment Optimizers](/concepts/alignment-optimizers.md) - Optimization algorithms (e.g., MemAlign) and the AlignmentOptimizer base class used to align judges, supporting both default and custom optimizer implementations.
* [Alignment Validation](/concepts/alignment-validation.md) - The practice of comparing original and aligned judge performance against human ground truth to measure improvement in accuracy after alignment.
* [AlignmentOptimizer Base Class](/concepts/alignmentoptimizer-base-class.md) - An extensible base class in mlflow.genai.judges.base for implementing custom alignment strategies, providing a template with an align() method for specialized optimization logic.
* [Allotted Time Exceeded in Delta Conflict Resolution](/concepts/allotted-time-for-conflict-resolution.md) - A timeout condition where Delta Lake's row-level conflict resolution exceeds the allowed time, requiring a retry of the operation.
* [Allotted Time Exceeded in Delta Lake](/concepts/allotted-time-exceeded-in-delta-lake.md) - A sub-error of DELTA_CONCURRENT_APPEND indicating row-level conflict resolution exceeded the allowed time limit, requiring a retry.
* [Allotted Time for Conflict Resolution](/concepts/allotted-time-for-conflict-resolution.md) - A timeout condition (ALLOTTED_TIME_EXCEEDED) indicating row-level conflict resolution exceeded the maximum allowed time, requiring a retry
* [ALLOTTED_TIME_EXCEEDED in Delta Lake](/concepts/allotted_time_exceeded-sub-error.md) - A specific sub-error of DELTA_CONCURRENT_DELETE_READ indicating that row-level conflict resolution exceeded its allotted time and the operation should be retried.
* [ALLOTTED_TIME_EXCEEDED Sub-error](/concepts/allotted_time_exceeded-sub-error.md) - A sub-error of DELTA_CONCURRENT_DELETE_DELETE indicating that row-level conflict resolution exceeded the allotted time and the operation should be retried.
* [Allow-list Model Access Pattern](/concepts/allow-list-model-access-pattern.md) - A security pattern for foundation model governance where default EXECUTE permission is revoked from the system.ai schema and then selectively granted on approved individual models.
* [Allowlist (Unity Catalog)](/concepts/allowlist-unity-catalog.md) - A Databricks Unity Catalog feature that controls which libraries (JARs, Maven artifacts) and init scripts can run on standard access mode compute, enhancing security and isolation.
* [Allowlist Path Permission Enforcement](/concepts/allowlist-path-permission-enforcement.md) - The mechanism by which allowlist entries use prefix matching on paths in Unity Catalog volumes or object storage, supporting permissions at base path, nested directory, or single file levels, with trailing slashes controlling prefix matching scope.
* [Allowlist Path Permissions and Prefix Matching](/concepts/allowlist-path-permissions-and-prefix-matching.md) - Rules governing how file paths and directories are matched in the allowlist, including prefix matching, trailing slash semantics, and the requirement for READ VOLUME permissions on volumes.
* [Allowlist Path Prefix Matching](/concepts/allowlist-path-prefix-matching.md) - The mechanism by which allowlist permissions propagate to contained files and directories, with trailing slashes controlling whether prefix matching is performed at a given directory level.
* [Allowlist Security Best Practices](/concepts/allowlist-security-best-practices.md) - Recommended practices for managing the Unity Catalog allowlist, including granting MANAGE ALLOWLIST sparingly, using specific paths, read-only storage, and formal approval processes.
* [Allowlisting Maven Coordinates](/concepts/allowlisting-maven-coordinates.md) - The process of adding Maven coordinates (groupId:artifactId:version) to the Unity Catalog allowlist, including wildcard patterns for versions and group IDs.
* [ALTER TABLE SET MANAGED](/concepts/alter-table-set-managed.md) - A Databricks SQL DDL command to convert an external table into a managed (Metastore-owned) table, which can fail with metadata cleanup errors.
* [ALTER TABLE SET MANAGED (Databricks)](/concepts/alter-table-set-managed.md) - Databricks SQL command to convert an external (unmanaged) Delta table to a managed table, with associated error conditions
* [ALTER TABLE SET MANAGED (Delta Lake)](/concepts/alter-table-set-managed.md) - A Delta Lake SQL command that attempts to convert an unmanaged (external) table to a managed table in Databricks.
* [ALTER TABLE SET MANAGED Command](/concepts/alter-table-set-managed.md) - Databricks SQL command to convert an external Delta table into a managed (internal) table
* [ALTER TABLE SET MANAGED operation](/concepts/alter-table-set-managed.md) - A Delta Lake DDL operation that migrates an external (unmanaged) table to a managed table, which can fail due to concurrent operations or file validation issues.
* [ALTER TABLE UNSET MANAGED command](/concepts/alter-table-unset-managed-command.md) - A Databricks SQL command used to roll back a Unity Catalog managed table to its original external table state.
* [ALTER VIEW as Alternative to DROP](/concepts/alter-view-as-alternative-to-drop.md) - When a view cannot be deleted because it is shared, Databricks recommends using ALTER VIEW to update it instead of dropping and recreating it.
* [Alternatives to DDP for large models](/concepts/alternatives-to-ddp-for-large-models.md) - FSDP and DeepSpeed are alternatives to DDP when models do not fit in single GPU memory.
* [Alternatives to DDP: FSDP and DeepSpeed](/concepts/alternatives-to-ddp-fsdp-and-deepspeed.md) - Fully Sharded Data Parallel (FSDP) and DeepSpeed as alternatives to DDP for models that don't fit in a single GPU's memory
* [Amazon DynamoDB Online Store Integration](/concepts/amazon-dynamodb-online-store-integration.md) - Details of using Amazon DynamoDB as a third-party online store with Feature Store, including combined primary keys, automatic table creation via publish_table(), and instance profile authentication.
* [Anaconda License Impact on MLflow](/concepts/anaconda-license-impact-on-mlflow.md) - Changes to Anaconda's terms of service that led Databricks to stop using the defaults channel as default for MLflow models, switching to conda-forge, and implications for models logged before MLflow v1.18.
* [Anaconda Licensing for Legacy MLflow Models](/concepts/anaconda-licensing-for-legacy-mlflow-models.md) - Notice that MLflow models logged before v1.18 (Databricks Runtime 8.3 ML or earlier) may have a dependency on the Anaconda 'defaults' channel, which now requires a commercial license; newer models default to the community-managed 'conda-forge' channel.
* [Anaconda Licensing Impact on Legacy MLflow Models](/concepts/anaconda-licensing-for-legacy-mlflow-models.md) - MLflow models logged before v1.18 may have an unintended dependency on Anaconda's 'defaults' channel, requiring a commercial license per Anaconda's updated terms of service; affected customers can re-register models with a new conda.yaml.
* [Anaconda licensing implications for legacy MLflow models](/concepts/anaconda-licensing-for-legacy-mlflow-models.md) - MLflow models logged before v1.18 defaulted to Anaconda's 'defaults' conda channel; Anaconda's updated terms of service may require a commercial license. Affected models can be re-registered with a new conda.yaml specifying 'conda-forge' instead.
* [Analyzing GenAI Feedback with MLflow SDK](/concepts/analyzing-genai-feedback-with-mlflow-sdk.md) - Using MlflowClient to search traces, retrieve assessments, and compute feedback metrics (feedback rate, positive rate, per-dimension averages) from production data.
* [Anomaly Detection](/concepts/anomaly-detection.md) - One-click scalable data quality monitoring that uses intelligent scanning to prioritize important tables, analyzing historical patterns to detect freshness and completeness issues.
* [Anomaly Detection (Databricks)](/concepts/anomaly-detection-databricks.md) - A Databricks feature that automatically monitors data quality across schema tables by analyzing historical patterns of freshness and completeness.
* [Anomaly Detection Alert](/concepts/anomaly-detection-alert.md) - A Databricks feature that notifies users when anomaly detection identifies data quality issues such as stale tables or unexpected drops in row count.
* [Anomaly Detection Alerts](/concepts/anomaly-detection-alert.md) - Notifications triggered when Databricks anomaly detection identifies data quality issues such as stale tables or unexpected drops in row count.
* [Anomaly Detection Alerts in Databricks Unity Catalog](/concepts/anomaly-detection-alerts-in-databricks-unity-catalog.md) - Overview of alerting mechanisms that notify users when anomaly detection identifies data quality issues in Unity Catalog, supporting both UI-based and SQL-based configuration.
* [Anomaly detection expense tracking](/concepts/anomaly-detection-expense-tracking.md) - Technique to isolate anomaly detection costs by filtering on a non-null schema_id in the billing usage table.
* [Anomaly Detection for Table Completeness](/concepts/anomaly-detection-for-table-completeness.md) - A monitoring feature that analyzes historical row counts to predict an expected range of rows over 24 hours and marks a table as incomplete if the actual count falls below the lower bound.
* [Anomaly Detection for Table Freshness](/concepts/anomaly-detection-for-table-freshness.md) - An intelligent scanning feature that predicts the expected time of the next table commit using historical patterns and marks a table as stale if a commit is unusually late.
* [Anomaly Detection Output System Table](/concepts/anomaly-detection-output-system-table.md) - The system table system.data_quality_monitoring.table_results that stores anomaly detection output results and can be queried for alert configuration, with legacy beta jobs using an alternative table path.
* [Anomaly Detection Result Schema](/concepts/anomaly-detection-result-schema.md) - Schema of the anomaly detection results table, with rows representing scanned tables and columns including commit_freshness, total_row_count, daily_row_count, upstream_jobs, and downstream_impact.
* [Answer Sheet Evaluation](/concepts/answer-sheet-evaluation.md) - An evaluation mode for scoring pre-computed outputs or existing traces without running the GenAI app directly during evaluation.
* [Answer Sheet Evaluation Mode](/concepts/answer-sheet-evaluation.md) - Evaluation mode for scoring pre-computed outputs or existing traces without running the app directly, useful for external systems or historical data.
* [AnswerRelevancy Scorer](/concepts/answerrelevancy-scorer.md) - A DeepEval scorer that evaluates how relevant a model's answer is to the given input query, with configurable thresholds and scoring models.
* [Anthropic Messages API (on Databricks)](/concepts/anthropic-messages-api-on-databricks.md) - Databricks-hosted access to Anthropic's Messages API for Claude models, supporting text and image inputs.
* [Anthropic Tool Calling Trace with MLflow](/concepts/anthropic-tool-calling-trace-with-mlflow.md) - Capturing and annotating Anthropic function/tool calling responses in MLflow Traces, including decorating tool functions with @mlflow.trace to create spans
* [ANY FILE Securable](/concepts/any-file-securable.md) - A special Hive metastore securable object type that controls access to files at arbitrary paths, going beyond standard table-level permissions.
* [Apache Airflow Orchestration for Databricks Pipelines](/concepts/apache-airflow-orchestration-for-databricks-pipelines.md) - Triggering pipeline updates from Apache Airflow DAGs using the DatabricksSubmitRunOperator with a pipeline_task configuration.
* [Apache Iceberg Metadata Version Parsing](/concepts/apache-iceberg-metadata-version-parsing.md) - The requirement that Apache Iceberg metadata file names follow a specific version convention; failures in parsing the version from metadata locations can cause ingress violations.
* [Apache Iceberg to Delta Lake Conversion](/concepts/iceberg-to-delta-conversion.md) - Converting Iceberg tables (with Parquet file format) to Delta Lake by using the Iceberg manifest for metadata, schema, and partitioning information.
* [Apache Software Foundation Trademark Attribution](/concepts/apache-software-foundation-trademark-attribution.md) - Standard attribution notice for Apache, Apache Spark, Spark and the Spark logo as trademarks of the Apache Software Foundation.
* [Apache Spark API compatibility with Delta Lake](/concepts/apache-spark-api-compatibility-with-delta-lake.md) - Full compatibility of Delta Lake with Apache Spark APIs, allowing users to leverage existing Spark knowledge and codebases when working with Delta Lake.
* [Apache Spark Compatibility](/concepts/apache-spark-compatibility.md) - Delta Lake is fully compatible with Apache Spark APIs, allowing users to leverage existing Spark code and workflows when working with Delta Lake tables.
* [Apache Spark DataFrames to TFRecord Conversion](/concepts/apache-spark-dataframes-to-tfrecord-conversion.md) - The recommended approach for saving Apache Spark DataFrames as TFRecord files for use in distributed deep learning, as documented by Databricks.
* [Apache Spark MLlib](/concepts/apache-spark-mllib.md) - The scalable machine learning library in Apache Spark, providing common algorithms for classification, regression, clustering, collaborative filtering, dimensionality reduction, and optimization primitives.
* [Apache Spark Server Incompatibility with Databricks Connect](/concepts/apache-spark-server-incompatibility-with-databricks-connect.md) - Databricks Connect is built against Databricks Runtime and is incompatible with open-source Apache Spark servers due to differing dependencies such as the json4s library version, causing class-not-found, API mismatch, or serialization errors.
* [Apache Spark Trademark Attribution](/concepts/apache-spark-trademark-attribution.md) - Legal attribution notice that Apache, Apache Spark, Spark, and the Spark logo are trademarks of the Apache Software Foundation, as required by Databricks documentation.
* [Apache Spark Trademark Attribution in Databricks Documentation](/concepts/apache-spark-trademark-attribution.md) - Databricks documentation includes standard attribution for Apache Spark, Spark, and the Spark logo as trademarks of the Apache Software Foundation.
* [Apache Spark Trademark Usage](/concepts/apache-spark-trademark-usage.md) - Guidelines and legal terms for using Apache Spark trademarks, as referenced in Databricks documentation
* [API Response Structure for Reasoning Models](/concepts/api-response-structure-for-reasoning-models.md) - The chat completions API response for reasoning models includes both a reasoning block (with summary text and signature) and a text block (the final answer), allowing clients to access the model's step-by-step thinking.
* [API-Based Endpoint Configuration with Instance Profile](/concepts/api-based-endpoint-configuration-with-instance-profile.md) - Using the Databricks Serving Endpoints API (POST/PUT /api/2.0/serving-endpoints) with the instance_profile_arn field to assign an instance profile during creation or update.
* [App Resource Environment Variable Injection](/concepts/app-resource-environment-variable-injection.md) - Pattern in Databricks Apps where resource identifiers (e.g., MLflow experiment IDs) are automatically exposed as environment variables at deployment, referenced via the valueFrom field in app.yaml configuration.
* [App Resource Permission Levels](/concepts/app-resource-permission-levels.md) - Three-tiered permission model (Can read, Can edit, Can manage) for granting Databricks app service principals access to MLflow experiment resources.
* [APPLY (CHECKPOINT)](/concepts/apply-checkpoint.md) - A REORG TABLE subcommand that performs Delta checkpointing on the table's latest Delta version to improve metadata management.
* [APPLY (PURGE)](/concepts/apply-purge.md) - A REORG TABLE subcommand that rewrites files containing soft-deleted data (e.g., from ALTER TABLE DROP COLUMN) to physically remove it from files.
* [APPLY (UPGRADE UNIFORM)](/concepts/apply-upgrade-uniform.md) - A REORG TABLE subcommand that upgrades a Delta table to a specified Apache Iceberg version (1 or 2) for Iceberg compatibility.
* [apply_fsdp Sharding Pattern](/concepts/apply_fsdp-sharding-pattern.md) - A design pattern that applies FSDP sharding to individual transformer layers using fully_shard() before sharding the entire model, enabling distributed training of modular architectures.
* [AppState Pattern for Distributed Checkpointing](/concepts/appstate-pattern-for-distributed-checkpointing.md) - A Stateful-protocol-compliant wrapper class that encapsulates model and optimizer state dict logic for use with PyTorch's distributed checkpoint save/load APIs in FSDP contexts.
* [AppState Stateful Wrapper for Checkpointing](/concepts/appstate-stateful-wrapper-pattern.md) - A PyTorch Stateful protocol-compliant wrapper class that encapsulates model and optimizer state, automatically managing FSDP fully qualified names and sharded state dict types during dcp.save/load operations.
* [AppState Stateful Wrapper Pattern](/concepts/appstate-stateful-wrapper-pattern.md) - A design pattern implementing PyTorch's Stateful protocol to wrap model and optimizer state for use with Distributed Checkpoint API, handling FSDP fully qualified names and SHARDED_STATE_DICT automatically.
* [Archival and the Delta Transaction Log](/concepts/archival-and-the-delta-transaction-log.md) - The critical constraint that the Delta transaction log directory (_delta_log/) must never be archived, as doing so makes the entire table inaccessible.
* [Archival support for Delta tables](/concepts/archival-support-in-databricks.md) - A Databricks feature enabling Delta tables to coexist with cloud object lifecycle policies by ignoring archived files during queries.
* [Archival support for Delta tables in Databricks](/concepts/archival-support-in-databricks.md) - A Databricks feature that allows Delta tables to use cloud lifecycle policies on object storage, enabling queries to ignore files older than a specified period and fail early when archived files must be scanned.
* [Archival Support in Databricks](/concepts/archival-support-in-databricks.md) - A Databricks feature that enables cloud-based lifecycle policies on Delta tables, allowing queries to safely coexist with archived cloud object storage data.
* [Archival support limitations](/concepts/archival-support-limitations.md) - The documented restrictions on Delta Lake archival support including inapplicability to transaction logs, exclusion of non-creation-time-based policies, and unsupported operations on archived data.
* [Archival support sampling behavior](/concepts/archival-support-sampling-behavior.md) - The mechanism by which Databricks samples files older than the retention period to determine if restored data is available before scanning a table.
* [Archival-aware query optimization in Delta Lake](/concepts/archival-aware-query-optimization-in-delta-lake.md) - Optimizations that allow queries to succeed without touching archived files by using metadata-only queries or filters that avoid archived data.
* [Archival-compatible S3 storage tiers](/concepts/archival-compatible-s3-storage-tiers.md) - Archival support requires S3 Glacier Deep Archive or Glacier Flexible Retrieval; Glacier Instant Retrieval works without configuration, and Amazon S3 Intelligent-Tiering async archive tiers are incompatible due to access-time-based archiving.
* [Archived Data Sampling Mechanism](/concepts/archived-data-sampling-mechanism.md) - Databricks' approach to detecting restored archived files by sampling files older than the retention period; if sampled files appear restored, all archived files for the query are assumed restored.
* [Arize Phoenix Scorers](/concepts/arize-phoenix-scorers.md) - Built-in scorers from Arize Phoenix for evaluating LLM outputs, integrated with MLflow GenAI evaluation
* [Arize Phoenix Scorers Integration](/concepts/arize-phoenix-scorers-integration.md) - MLflow integration with Arize Phoenix offering a lightweight set of scorers for hallucination detection, relevance, toxicity, QA correctness, and summarization quality
* [AS OF join](/concepts/as-of-join.md) - A temporal join that matches each row in the label DataFrame with the most recent feature value from a time series feature table whose timestamp is not later than the label's timestamp.
* [Assessment Data (Feedback vs Expectation)](/concepts/assessments-feedback-vs-expectation.md) - The two types of human-generated assessments collected during labeling sessions: Feedback (subjective quality ratings) and Expectations (ground-truth reference responses).
* [Assessment Objects](/concepts/assessment-objects.md) - MLflow entity objects that store collected labels as structured assessments attached to traces within labeling sessions, retrievable via the MLflow SDK.
* [Assessment Objects in MLflow Tracing](/concepts/assessment-objects-on-traces.md) - Labels collected from domain experts are stored as Assessment objects on each Trace within a labeling session, retrievable programmatically for analysis or evaluation dataset creation
* [Assessment Objects on Traces](/concepts/assessment-objects-on-traces.md) - Labels collected from domain experts are stored as Assessment objects on each Trace within a labeling session, retrievable via MLflow SDK for analysis and evaluation dataset creation.
* [Assessment Source Tracking in MLflow](/concepts/assessment-source-tracking-in-mlflow.md) - Tagging feedback with AssessmentSource and AssessmentSourceType to distinguish feedback provenance (end users, developers, or expert reviewers) for traceability
* [Assessment types: Feedback vs Expectation](/concepts/assessment-types-feedback-vs-expectation.md) - Two distinct categories of assessments in labeling schemas: subjective feedback (ratings, opinions) and objective ground-truth expectations (correct answers, expected behavior).
* [Assessments](/concepts/assessments.md) - Quality measurements attached to a trace, comprising feedback (judgments from scorers, users, or experts) and expectations (ground-truth labels from domain experts).
* [Assessments (Feedback & Expectations)](/concepts/assessments-feedback-expectations.md) - The human-generated labels collected through labeling sessions, consisting of either Feedback (opinion-based ratings) or Expectations (ground-truth data) that can drive evaluation and improvement workflows.
* [Assessments (MLflow GenAI)](/concepts/assessments.md) - Quality measurements and ground truth labels attached to traces, divided into feedback (judgments) and expectations (ground truth).
* [Assessments on Traces](/concepts/assessments-on-traces.md) - Traces support assessments (feedback and expectations) searchable by name, type, span_id, with details like value, source, rationale, and metadata.
* [Assessments: Feedback vs Expectation](/concepts/assessments-feedback-vs-expectation.md) - The two types of human-generated labels that can be captured in labeling sessions: Feedback (subjective evaluation) and Expectation (ground-truth reference data).
* [AssessmentSource Entity](/concepts/assessmentsource-entity.md) - An MLflow entity that identifies who or what provided feedback, supporting source_type (HUMAN or LLM_JUDGE) and source_id fields
* [AssessmentSource Entity for Feedback Attribution](/concepts/assessmentsource-entity.md) - The AssessmentSource object identifies who or what provided feedback, supporting HUMAN, LLM_JUDGE, or CODE source types with source_id for specific attribution.
* [AssessmentSource in MLflow](/concepts/assessmentsource-entity.md) - Entity that identifies the origin of feedback, supporting source types like HUMAN for user feedback and LLM_JUDGE for automated evaluation.
* [Assigned User Management for Labeling](/concepts/assigned-user-management-for-labeling.md) - The process of adding, replacing, and managing domain experts assigned to labeling sessions, including automatic WRITE permission grants on the MLflow experiment.
* [Assigned User Management for Labeling Sessions](/concepts/assigned-user-management-for-labeling.md) - The process of adding, replacing, and configuring permissions for domain expert reviewers in labeling sessions, including automatic WRITE permission grants on the underlying MLflow Experiment.
* [Assigning a Metastore Admin](/concepts/assigning-a-metastore-admin.md) - The process by which account admins assign the metastore admin role to a user, service principal, or group (groups recommended), with changes taking up to 30 seconds plus caching delays.
* [Assigning the Metastore Admin Role](/concepts/assigning-a-metastore-admin.md) - Process by which account admins assign the metastore admin role to a user, service principal, or group in the account console; Databricks recommends using a group. Changes may take up to 30 seconds to propagate due to caching.
* [Async Anthropic Tracing in MLflow](/concepts/async-anthropic-tracing-in-mlflow.md) - Support for tracing asynchronous Anthropic SDK calls (AsyncAnthropic) added in MLflow 2.21.0, enabling non-blocking concurrent invocation tracing
* [Async-to-Sync Wrapper Pattern for Agent Evaluation](/concepts/async-to-sync-wrapper-pattern-for-agent-evaluation.md) - A pattern wrapping asynchronous Claude agent execution inside a synchronous predict function to enable compatibility with MLflow's evaluation API.
* [Asynchronous Checkpoint I/O with DCP](/concepts/asynchronous-checkpoint-io-with-dcp.md) - Using dcp.async_save with UCVolumeWriter to upload checkpoints in the background while training continues, requiring a CPU backend on the process group with gloo and nccl.
* [Asynchronous Checkpoint Save](/concepts/asynchronous-checkpoint-save.md) - AI Runtime supports dcp.async_save with a UCVolumeWriter to upload checkpoints in the background while training continues, requiring a CPU backend on the process group.
* [Asynchronous Iceberg Metadata Generation](/concepts/asynchronous-iceberg-metadata-generation.md) - The process by which Databricks automatically generates Iceberg metadata (commit metadata JSON files) in the background after each Delta Lake write transaction, using the same compute resource without blocking writes.
* [Asynchronous Model and Code Lifecycles](/concepts/asynchronous-model-and-code-lifecycles.md) - The fundamental concept in MLOps that model artifacts and the code that produces them evolve on different cadences, enabling scenarios like weekly model retraining without code changes or code updates without retraining.
* [Asynchronous Pipeline Update Handling](/concepts/asynchronous-pipeline-update-handling.md) - Managing downstream task dependencies when pipeline update requests return asynchronously, requiring polling or until-loops to wait for completion.
* [Attribute-Based Access Control (ABAC)](/concepts/attribute-based-access-control-abac.md) - Dynamic, fine-grained access policies based on attributes of data and users, enabling row-level filtering and column-level masking without managing permissions table by table.
* [Attribute-Based Access Control (ABAC) in Unity Catalog](/concepts/attribute-based-access-control-abac-in-unity-catalog.md) - Tag-driven centralized policies that dynamically filter and mask data across catalogs, recommended over per-table filters for scaling access control.
* [Audio/Video Input Methods for Foundation Models](/concepts/audiovideo-input-methods-for-foundation-models.md) - Two supported methods for providing media to Gemini models on Databricks: publicly accessible URLs and base64-encoded inline data URIs.
* [Audit Log Access Requirements](/concepts/audit-log-access-requirements.md) - Prerequisites for accessing OpenSharing audit logs, including enabling system tables, account admin privileges, and access to system.access.audit.
* [Audit Log System Table Requirements](/concepts/audit-log-system-table-requirements.md) - Prerequisites for accessing OpenSharing audit logs, including enabling the audit log system table and granting access to system.access.audit for non-admin users.
* [Audit Logging for ABAC Policy Operations](/concepts/audit-logging-for-abac-policy-operations.md) - Databricks logs governed tag assignment/deletion events and ABAC policy CRUD operations (create, delete, get, list) in the system.access.audit table for monitoring and compliance.
* [Auditing ABAC GRANT Policies with Direct Grants](/concepts/auditing-abac-grant-policies-with-direct-grants.md) - Auditing both direct grants and ABAC GRANT policies together since effective privileges are the union of both.
* [Auditing Direct Grants and ABAC GRANT Policies](/concepts/auditing-direct-grants-and-abac-grant-policies.md) - The need to audit both direct grants and ABAC GRANT policies together since effective privileges are the union of both
* [Authentication for Databricks remote MLflow tracking](/concepts/mlflow-authentication-methods.md) - Methods for authenticating with a remote Databricks MLflow tracking server, including Personal Access Tokens (PAT) and OAuth via service principals.
* [Authentication methods for Databricks MLflow remote tracking](/concepts/mlflow-authentication-methods.md) - Two authentication options for remote MLflow tracking: Personal Access Tokens (PAT) for simple development setups and OAuth using service principals for production.
* [Authentication methods for remote MLflow tracking](/concepts/mlflow-authentication-methods.md) - Two authentication approaches for remote tracking server connections: Personal Access Tokens (PAT) for simple token-based auth and OAuth using service principals.
* [Authentication via DatabricksSession for Scala Utilities](/concepts/authentication-via-databrickssession-for-scala-utilities.md) - Authentication for the Databricks Utilities for Scala library is determined through initializing the DatabricksSession class in a Databricks Connect project
* [Auto Loader for Image Ingestion](/concepts/auto-loader-for-image-ingestion.md) - Using Databricks Auto Loader to incrementally ingest large volumes of images from cloud object storage into Delta tables, handling continuously arriving new files.
* [Auto Loader for Parquet Migration](/concepts/auto-loader-for-parquet-migration.md) - Using Databricks Auto Loader's incremental ingestion capabilities to migrate Parquet data to Delta Lake with exactly-once guarantees.
* [Auto Time-to-Live (Auto-TTL) for Unity Catalog Tables](/concepts/auto-time-to-live-auto-ttl-for-unity-catalog-tables.md) - A configurable automatic retention mechanism in Databricks Unity Catalog that deletes trace data older than a specified number of days, helping comply with GDPR and other data protection regulations.
* [Auto Time-to-Live for Compliance](/concepts/auto-time-to-live-for-compliance.md) - Automatic row deletion based on a time column in Unity Catalog managed Delta tables, used to enforce data retention policies without scheduled jobs.
* [Auto-ARIMA and Algorithm Selection in Serverless Forecasting](/concepts/auto-arima-and-algorithm-selection-in-serverless-forecasting.md) - The use of the Auto-ARIMA algorithm (via pmdarima) and the ability to choose training frameworks for AutoML to explore during model tuning.
* [Auto-ARIMA and Regular Frequency Requirement](/concepts/auto-arima-and-regular-frequency-requirement.md) - Requirement that time series data must have a regular, consistent frequency for Auto-ARIMA, with AutoML automatically filling in missing time steps using forward fill.
* [Auto-ARIMA for Forecasting](/concepts/auto-arima-for-forecasting.md) - An algorithm used by Databricks AutoML that requires time series data with a regular frequency; missing time steps are filled with the previous value.
* [Auto-ARIMA Frequency Requirements](/concepts/auto-arima-frequency-requirements.md) - Constraint in AutoML forecasting requiring time series to have a regular frequency with consistent intervals between data points, matching the frequency unit specified in the UI or API.
* [Auto-ARIMA in Databricks AutoML](/concepts/auto-arima-in-databricks-automl.md) - An algorithm option for forecasting that requires regularly-spaced time series data; Databricks AutoML handles missing time steps by forward-filling with the previous value.
* [Auto-assign metastores for workspace provisioning](/concepts/auto-assign-metastores-for-workspace-provisioning.md) - Admins must set a Unity Catalog metastore to auto-assign for every region where workspaces are deployed, ensuring new workspaces automatically receive a metastore.
* [Auto-tagging for Detected Classifications](/concepts/auto-tagging-for-detected-classifications.md) - Automatic tagging feature that applies classification tags to all existing and future detections of a classification type. Configurable at both metastore level and catalog level, with catalog-level settings taking precedence.
* [Auto-Tracing for Sequential LLM Calls](/concepts/auto-tracing-for-sequential-llm-calls.md) - How MLflow automatic tracing captures multiple sequential LLM calls within a single workflow, creating a parent span with child spans for each OpenAI invocation.
* [Auto-Tracing with Databricks Foundation Model APIs](/concepts/auto-tracing-with-databricks-foundation-model-apis.md) - Configuring OpenAI client to call Databricks Foundation Model APIs (e.g., Llama 4 Maverick) while mlflow.openai.autolog() automatically traces those calls using Databricks authentication.
* [AutoGen Auto-Tracing](/concepts/autogen-auto-tracing.md) - MLflow's automatic tracing capability for AutoGen agent workflows, enabled via mlflow.autogen.autolog()
* [Autologging Administration in Databricks](/concepts/autologging-administration-in-databricks.md) - Workspace-level control for administrators to enable or disable Databricks Autologging across all clusters from the admin settings page.
* [Autologging Configuration for Databricks Environments](/concepts/autologging-configuration-for-databricks-environments.md) - Prerequisites and configuration steps for enabling MLflow Tracing on Databricks Foundation Models, including installing mlflow[databricks]>=3.1 and the OpenAI SDK, setting environment variables (DATABRICKS_HOST, DATABRICKS_TOKEN), and noting that serverless compute clusters require explicit autolog() calls.
* [Autologging for GenAI Tracing](/concepts/autologging-for-genai-tracing.md) - One-line code pattern using mlflow.<library>.autolog() to automatically instrument and trace 20+ supported AI libraries and frameworks without manual instrumentation.
* [Autologging Limitations and Constraints](/concepts/autologging-limitations-and-constraints.md) - MLflow Gemini autolog has known constraints: not auto-enabled on serverless compute clusters, only supports synchronous text interactions (no async), and may not record full inputs for multi-modal inputs.
* [Autologging Limitations and Edge Cases](/concepts/autologging-limitations-and-edge-cases.md) - Known constraints of Databricks Autologging, including driver-node-only execution, incompatibility with explicit mlflow.start_run(), and unsupported XGBoost scikit-learn integration.
* [Autologging on Serverless Compute](/concepts/autologging-on-serverless-compute.md) - Databricks Autologging is not automatically enabled on serverless compute; users must explicitly call mlflow.autolog() to use it
* [Autologging Security and Administration](/concepts/autologging-security-and-administration.md) - Security via MLflow Experiment permissions and workspace-level admin controls to enable or disable Databricks Autologging for all interactive notebooks
* [Autologging Security and Data Management](/concepts/autologging-security-and-data-management.md) - The security model for autologged data, which is stored in MLflow Tracking and secured by MLflow Experiment permissions, allowing sharing, modification, or deletion via MLflow APIs or UI.
* [Automated Container Compliance Rebuilding](/concepts/automated-container-compliance-rebuilding.md) - Databricks automatically rebuilds outdated served containers within 30 days to maintain compliance, with event log alerts when the automated job fails.
* [Automated Data Classification with AI](/concepts/automated-data-classification-with-ai.md) - Databricks Data Classification uses a large language model (LLM) agent to automatically classify and tag tables in Unity Catalog, identifying sensitive data such as PII, GDPR, HIPAA, and DPDPA-related content.
* [Automated Evaluation and Monitoring](/concepts/automated-evaluation-and-monitoring.md) - Using the same LLM judges and scorers during development and production to replace manual testing and continuously improve GenAI app quality
* [Automatic Container Rebuilding for Compliance](/concepts/automatic-container-rebuilds-for-compliance.md) - Compliance standards require served containers to be rebuilt within the last 30 days; Databricks automatically rebuilds outdated containers, with failure notifications via event logs.
* [Automatic Container Rebuilds for Compliance](/concepts/automatic-container-rebuilds-for-compliance.md) - Compliant Model Serving requires containers built within the last 30 days; Databricks automatically rebuilds outdated containers, with event log notifications if the automated rebuild fails.
* [Automatic Dependency Declaration for Databricks Utilities Scala Library](/concepts/automatic-dependency-declaration-for-databricks-utilities-scala-library.md) - Databricks Connect for Scala automatically declares a dependency on the Databricks Utilities for Scala library, so no explicit build file configuration is needed
* [Automatic Exception Capture in Spans](/concepts/automatic-exception-capture-in-spans.md) - When an exception is raised during a traced operation, MLflow captures partial data and records exception details as span events, indicated in the UI as an unsuccessful invocation.
* [Automatic Exception Handling in MLflow Traces](/concepts/automatic-exception-handling-in-mlflow-traces.md) - Built-in capture of exceptions during trace-instrumented operations, showing failure indicators in the UI and recording exception details as span events.
* [Automatic exception handling in traces](/concepts/automatic-exception-capture-in-spans.md) - If an Exception is raised during a trace-instrumented operation, MLflow captures the partial trace data and records exception details as span events for debugging.
* [Automatic Feature Lineage](/concepts/automatic-feature-lineage.md) - Models trained using Databricks Feature Store automatically track lineage to the features used during training, enabling reproducibility and governance.
* [Automatic Feature Lineage Capture](/concepts/automatic-feature-lineage-capture.md) - Lineage information tracking feature tables and Python UDFs used in models is automatically captured when calling log_model
* [Automatic Feature Lineage Tracking](/concepts/automatic-feature-lineage.md) - Models trained using Databricks features automatically track lineage to the features they were trained on, enabling runtime feature lookup from online stores.
* [Automatic Feature Lookup for Model Serving](/concepts/automatic-feature-lookup-for-model-serving.md) - Databricks Model Serving can automatically look up feature values from online feature stores (Databricks or third-party) during model inference, eliminating the need for manual feature engineering in the serving pipeline.
* [Automatic Input/Output Parsing for Evaluation](/concepts/automatic-inputoutput-parsing-for-evaluation.md) - How the Guidelines judge automatically extracts request and response fields from traces, supporting OpenAI-format messages, ChatCompletion objects, and arbitrary dicts.
* [Automatic Instrumentation with mlflow.openai.autolog()](/concepts/automatic-instrumentation-with-mlflowopenaiautolog.md) - A feature that automatically captures trace data from OpenAI SDK calls (including Databricks-hosted LLMs) without requiring manual instrumentation of each API invocation.
* [Automatic Provider and Recipient Objects](/concepts/automatic-provider-and-recipient-objects.md) - Two objects automatically created when an SAP BDC connection is established, enabling bidirectional sharing
* [Automatic Provider Object Creation](/concepts/automatic-provider-object-creation.md) - For Databricks-to-Databricks sharing, provider objects are created automatically in the recipient's Unity Catalog metastore, so recipients typically do not need to manually create them.
* [Automatic Request/Response Parsing](/concepts/automatic-requestresponse-parsing.md) - How the Guidelines judge automatically extracts request and response data from traces, handling OpenAI format messages, ChatCompletions, and arbitrary dicts.
* [Automatic System Metrics Capture](/concepts/automatic-system-metrics-capture.md) - GPU, CPU, and memory system metrics are captured automatically for every run without configuration, viewable in the MLflow run's System metrics tab.
* [Automatic Tagging](/concepts/automatic-tagging.md) - A mechanism in Databricks Data Classification that automatically applies governed tags to detected columns, configurable at both metastore and catalog levels with inheritance semantics.
* [Automatic Tagging in Data Classification](/concepts/automatic-tagging.md) - The ability to automatically apply governed tags to all existing and future column detections of a given classification type, configurable at catalog and metastore levels.
* [Automatic Tagging with Governed Tags](/concepts/automatic-tagging.md) - A mechanism to automatically apply governed classification tags to detected columns; configurable at metastore and catalog levels with inherited, active, or inactive states.
* [Automatic Tracing](/concepts/automatic-tracing.md) - An MLflow tracing approach that uses a single line of code (mlflow.<library>.autolog()) to automatically capture traces for 20+ supported libraries without manual instrumentation.
* [Automatic Tracing (autolog)](/concepts/automatic-tracing.md) - One-line `mlflow.<library>.autolog()` approach to automatically capture traces for 20+ supported libraries
* [Automatic Tracing (MLflow)](/concepts/automatic-tracing.md) - One-line mlflow.<library>.autolog() approach to automatically capture traces for 20+ supported libraries without manual instrumentation.
* [Automatic Tracing for Databricks Foundation Model APIs](/concepts/auto-tracing-with-databricks-foundation-model-apis.md) - Using mlflow.openai.autolog() to trace calls to Databricks Foundation Model APIs via the OpenAI-compatible endpoint.
* [Automatic Tracing for OpenAI](/concepts/automatic-tracing-for-openai.md) - One-line integration that automatically traces OpenAI SDK calls by wrapping the client with the tracedOpenAI function from the mlflow-openai package.
* [Automatic Transformer Block Detection for FSDP Wrapping](/concepts/automatic-transformer-block-detection-for-fsdp-wrapping.md) - A technique to programmatically infer which module classes in a model correspond to transformer decoder layers, enabling correct FSDP auto-wrap configuration without hardcoding class names.
* [Automatic Type Casting for Column Mask Functions](/concepts/automatic-type-casting-for-column-mask-functions.md) - Databricks automatically casts input column values to match the mask function's parameter type and the function output to match the target column's data type, with special support for struct-to-VARIANT casting on Runtime 18.1+.
* [Automatic UDF Dependency Discovery with withAutoDependencies](/concepts/automatic-udf-dependency-discovery-with-withautodependencies.md) - Databricks Connect can automatically discover and upload local modules and public PyPI dependencies used in UDF imports, eliminating manual specification via the withAutoDependencies() API.
* [Automatic Unity Catalog Enablement](/concepts/automatic-unity-catalog-enablement.md) - For workspaces created after November 8, 2023, Unity Catalog is auto-enabled with a metastore, eliminating the need for a separate metastore admin by granting workspace admins sufficient metastore-level privileges.
* [Automatic Unity Catalog Enablement and Workspace Admin Default Privileges](/concepts/automatic-unity-catalog-enablement.md) - For workspaces created after November 8, 2023, Unity Catalog is auto-enabled and workspace admins receive default metastore-level privileges (CREATE CATALOG, CREATE EXTERNAL LOCATION, etc.) represented by the auto-generated _workspace_admins_databricks system group.
* [Automatic Unity Catalog Workspace Admin Privileges](/concepts/automatic-unity-catalog-workspace-admin-privileges.md) - Default metastore-level privileges (CREATE CATALOG, CREATE EXTERNAL LOCATION, etc.) automatically granted to workspace admins in workspaces created after November 8, 2023, making the metastore admin role optional.
* [Automatic vs Manual Tracing Instrumentation](/concepts/automatic-vs-manual-tracing-instrumentation.md) - Two approaches to instrumenting GenAI applications for tracing: automatic (framework-integrated) and manual (developer-defined).
* [Automatically Populated Trace Metadata](/concepts/automatically-populated-trace-metadata.md) - MLflow automatically setting certain standard metadata fields based on the execution environment, with the ability to override them via mlflow.update_current_trace().
* [Automatically Populated Trace Metadata in MLflow](/concepts/automatically-populated-trace-metadata.md) - MLflow's automatic population of certain standard metadata fields based on execution environment, with the ability to override via update_current_trace()
* [AutoML Advanced Configuration Options](/concepts/automl-advanced-configuration-options.md) - Optional parameters for AutoML classification experiments including evaluation metrics, framework exclusion, stopping conditions, and time column splitting
* [AutoML Advanced Configuration Parameters](/concepts/automl-advanced-configuration-options.md) - Configurable options for AutoML experiments including evaluation metrics, training framework exclusions, stopping conditions, time-based data splitting, and data directory security.
* [AutoML API Feature Store Configuration](/concepts/automl-api-feature-store-configuration.md) - The AutoML API accepts a `feature_store_lookups` parameter, a list of dictionaries specifying table names and lookup keys to join feature tables to the training dataset.
* [AutoML API feature_store_lookups Parameter](/concepts/automl-run-api-feature-store-parameters.md) - The feature_store_lookups parameter in the AutoML run specification allows programmatic configuration of feature table joins.
* [AutoML Classification (classify)](/concepts/automl-classification-classify.md) - Method to configure and train classification models via AutoML, supporting parameters like target_col, primary_metric, timeout_minutes, and pos_label
* [AutoML Classification Data Preparation](/concepts/automl-classification-data-preparation.md) - How Databricks AutoML prepares datasets for classification model training, including feature type detection, imputation, sampling, and splitting.
* [AutoML Classification Experiment Setup](/concepts/automl-classification-experiment-setup.md) - Step-by-step process for configuring and launching a classification experiment using the Databricks AutoML UI
* [AutoML Classification Experiment Setup on Databricks](/concepts/automl-classification-experiment-setup.md) - GUI-based workflow to configure and launch a classification experiment using AutoML on Databricks, including dataset selection, prediction target, and compute specification.
* [AutoML Classification on Databricks](/concepts/automl-classification-on-databricks.md) - Automated machine learning service on Databricks that automatically finds the best classification algorithm and hyperparameter configuration to predict a label or category from input data.
* [AutoML Classify API](/concepts/automl-classify-api.md) - Databricks AutoML method to configure and run classification model training
* [AutoML classify()](/concepts/automl-classify.md) - Python API method to start an AutoML classification run, supporting parameters such as target_col, primary_metric, imputers, pos_label, and more.
* [AutoML Column Selection](/concepts/automl-column-selection.md) - Ability to specify which columns AutoML uses for training, including excluding columns via UI checkbox or API parameter, with restrictions on dropping target or time columns.
* [AutoML Data Preparation](/concepts/automl-data-preparation.md) - Data preparation features for AutoML including column selection for training, null value imputation, and chronological data splitting
* [AutoML Data Preparation and Column Configuration](/concepts/automl-data-preparation.md) - Capabilities to specify which columns AutoML uses for training, configure null value imputation strategies, and control chronological data splitting using a time column.
* [AutoML Data Preparation for Classification](/concepts/automl-data-preparation-for-classification.md) - Overview of how Databricks AutoML prepares data for classification model training, including configurable settings via UI and API.
* [AutoML Data Preparation Settings](/concepts/automl-data-preparation.md) - Configuration options for data preparation in AutoML including column selection, null value imputation, and chronological data splitting
* [AutoML Data Splitting Strategies](/concepts/automl-data-splitting-strategies.md) - Methods for dividing datasets into training, validation, and test sets in Databricks AutoML, including random, chronological, and manual splits.
* [AutoML Dataset Sampling](/concepts/automl-dataset-sampling.md) - Automatic stratified (classification) or random (regression) sampling of large datasets in Databricks AutoML to fit within single-worker-node memory limits.
* [AutoML Dataset Warnings and Quality Checks](/concepts/automl-dataset-warnings-and-quality-checks.md) - Feature in Databricks Runtime 10.1 ML+ that displays warnings for potential dataset issues such as unsupported column types or high cardinality columns during AutoML experiments.
* [AutoML Experiment Advanced Configuration](/concepts/automl-experiment-advanced-configuration.md) - Additional AutoML configuration options including evaluation metrics selection, framework exclusion, stopping conditions, early stopping, chronological data splitting, and data directory security.
* [AutoML Experiment Configuration](/concepts/automl-experiment-configuration.md) - Setup process for configuring an AutoML experiment including compute, dataset, prediction target, and advanced options
* [AutoML Experiment Configuration and Stopping Conditions](/concepts/automl-experiment-configuration-and-stopping-conditions.md) - Standard configuration parameters for AutoML experiments including compute selection, dataset browsing, prediction target, time column, evaluation metrics, and stopping conditions (default 120 minutes for forecasting).
* [AutoML Experiment Configuration UI](/concepts/automl-experiment-configuration-ui.md) - User interface workflow for configuring AutoML classification experiments including dataset selection, prediction target specification, compute cluster choice, and experiment naming.
* [AutoML Experiment Configuration via UI](/concepts/automl-experiment-configuration-via-ui.md) - Step-by-step process for setting up a classification experiment using the AutoML user interface in Databricks
* [AutoML Experiment Monitoring and Notebooks](/concepts/automl-experiment-monitoring-and-notebooks.md) - Tools for monitoring AutoML experiment progress, viewing dataset warnings (e.g., unsupported column types, high cardinality), accessing data exploration notebooks, and reviewing trial run notebooks stored as MLflow artifacts.
* [AutoML Experiment Monitoring and Results](/concepts/automl-experiment-monitoring-and-results.md) - How to monitor AutoML experiment progress, view warnings, explore generated notebooks, and interpret trial run results
* [AutoML Experiment Requirements by Runtime](/concepts/automl-experiment-requirements-by-runtime.md) - Classification and regression AutoML experiments with feature store integration require Databricks Runtime 11.3 LTS ML+, while forecasting experiments require 12.2 LTS ML+.
* [AutoML Experiment Results and Monitoring](/concepts/automl-experiment-results-and-monitoring.md) - How to view, monitor, and interpret results of an AutoML experiment including warnings, runs table, and generated notebooks
* [AutoML Experiment with feature_store_lookups](/concepts/automl-experiment-with-feature_store_lookups.md) - The mechanism to pass Feature Store tables as covariates to an AutoML forecasting experiment via the feature_store_lookups parameter
* [AutoML Feature Store Integration](/concepts/automl-feature-store-integration.md) - How Databricks AutoML can augment input datasets with features from Unity Catalog or legacy Workspace Feature Store tables
* [AutoML Feature Store Lookups](/concepts/automl-feature-store-lookups.md) - A mechanism in Databricks AutoML to pass external feature tables as covariates to forecasting experiments using the feature_store_lookups parameter.
* [AutoML Feature Store Lookups Parameter](/concepts/automl-feature-store-lookups.md) - The feature_store_lookups parameter in Databricks AutoML that links one or more Feature Store tables (keyed by lookup_key) as covariates in a forecasting experiment.
* [AutoML feature_store_lookups Parameter](/concepts/automl-feature-store-lookups.md) - A configuration parameter in Databricks AutoML forecasting API that accepts a list of feature table lookups to include covariates in the experiment.
* [AutoML Forecast API](/concepts/automl-forecast-api.md) - Databricks AutoML method to configure and run time-series forecasting model training
* [AutoML forecast()](/concepts/automl-forecast.md) - Python API method to start an AutoML forecasting run with parameters for time column, frequency, horizon, identity column, and country code.
* [AutoML Forecasting (Classic Compute)](/concepts/automl-forecasting-classic-compute.md) - Databricks AutoML's classic compute mode for automatically finding the best forecasting algorithm and hyperparameters for time-series data, requiring Databricks Runtime 10.0 ML or above.
* [AutoML Forecasting (Databricks Classic Compute)](/concepts/automl-forecasting-classic-compute.md) - Automated machine learning workflow on Databricks for time-series forecasting using classic (non-serverless) compute, requiring Databricks Runtime 10.0 ML+
* [AutoML Forecasting (forecast)](/concepts/automl-forecasting-forecast.md) - Method for time series forecasting model training, requiring target_col, time_col, frequency, horizon, and supporting identity_col for multiple time series
* [AutoML Forecasting Data Preparation](/concepts/automl-forecasting-data-preparation.md) - How Databricks AutoML prepares data for forecasting training, including supported feature types, imputation, splitting, and aggregation.
* [AutoML Forecasting Data Splitting Strategy](/concepts/automl-forecasting-data-splitting-strategy.md) - How Databricks AutoML partitions forecasting data into train, validation, and test sets using time series cross-validation, where the number of folds depends on input table characteristics such as number of time series, covariates presence, and series length.
* [AutoML Forecasting Experiment (Classic Compute)](/concepts/automl-forecasting-experiment-classic-compute.md) - Setting up a time-series forecasting experiment using Databricks AutoML with user-managed compute clusters, including dataset configuration, target column selection, and launch workflow.
* [AutoML Forecasting Experiment Configuration](/concepts/automl-forecasting-experiment-configuration.md) - Configuring and launching an automated forecasting experiment in Databricks AutoML with parameters for target column, time column, frequency, horizon, and timeout.
* [AutoML Forecasting Experiment Lifecycle](/concepts/automl-forecasting-experiment-stages.md) - The three-stage pipeline of a Databricks forecasting experiment: Preprocessing (validation, imputation, feature engineering), Tuning (algorithm selection and hyperparameter search), and Training (final model training and registration).
* [AutoML Forecasting Experiment Stages](/concepts/automl-forecasting-experiment-stages.md) - The three-phase pipeline of a Databricks forecasting experiment: Preprocessing (validation, imputation, feature generation), Tuning (algorithm exploration), and Training (final model training and registration).
* [AutoML Import Notebook](/concepts/automl-import-notebook.md) - Method to import a notebook saved as an MLflow artifact
* [AutoML import_notebook()](/concepts/automl-import-notebook.md) - Method to import a notebook saved as an MLflow artifact from an AutoML run, returning an ImportNotebookResult.
* [AutoML Imputation Strategies](/concepts/automl-imputation-strategies.md) - How Databricks AutoML handles missing values in classification datasets, including default type-based imputation and custom imputation methods.
* [AutoML Missing Value Imputation](/concepts/automl-missing-value-imputation.md) - Configurable strategies for handling null values in AutoML regression training, including default per-column-type imputation and custom overrides via UI or API.
* [AutoML Model Registration and Deployment](/concepts/automl-model-registration-and-deployment.md) - Process for registering AutoML-generated models to Unity Catalog or Model Registry and deploying to model serving endpoints
* [AutoML on Databricks](/concepts/automl-on-databricks.md) - A no-code UI and Python API feature that automatically finds the best algorithm and hyperparameter configuration for machine learning datasets.
* [AutoML Pandas Compatibility Issue](/concepts/automl-pandas-compatibility-issue.md) - Known pandas version incompatibility error when serving AutoML models, with resolution via a dependency-fix script
* [AutoML Python API](/concepts/automl-python-api.md) - Programmatic interface for Databricks AutoML that provides methods to start classification, regression, and forecasting runs.
* [AutoML Regress API](/concepts/automl-regress-api.md) - Databricks AutoML method to configure and run regression model training
* [AutoML regress()](/concepts/automl-regress.md) - Python API method to start an AutoML regression run, using r2 as the default primary metric and supporting similar parameters to classification.
* [AutoML Regression (regress)](/concepts/automl-regression-regress.md) - Method to configure and train regression models via AutoML, with primary_metric defaulting to r2 and supporting similar parameters to classify
* [AutoML Result Monitoring and Model Deployment](/concepts/automl-result-monitoring-and-model-deployment.md) - Workflow for monitoring experiment progress, viewing warnings, reviewing best model notebooks, and registering/deploying models to Unity Catalog or Model Registry via the AutoML UI.
* [AutoML Run API Feature Store Parameters](/concepts/automl-run-api-feature-store-parameters.md) - The feature_store_lookups parameter in the AutoML run specification API that enables programmatic feature table joins
* [AutoML Run Duration Control](/concepts/automl-run-duration-control.md) - Controlling AutoML run duration via timeout_minutes, with max_trials deprecated
* [AutoML Sampling for Large Datasets](/concepts/automl-sampling-for-large-datasets.md) - Automatic memory-aware dataset sampling in AutoML, using stratified sampling for classification and random sampling for regression.
* [AutoML Stopping Conditions](/concepts/automl-stopping-conditions.md) - Default stopping conditions for AutoML experiments including time limits, trial limits, and early stopping based on validation metric improvement
* [AutoML Supported Feature Types](/concepts/automl-supported-feature-types.md) - The specific Spark data types supported as features in Databricks AutoML classification, including numeric, boolean, string, timestamps, arrays, and decimal types.
* [AutoML Trial Notebooks](/concepts/automl-trial-notebooks.md) - Generated notebooks per model trial during an AutoML run, importable via import_notebook from MLflow artifact URIs
* [AutoML UI Feature Selection](/concepts/automl-ui-feature-selection.md) - The graphical workflow in the Databricks AutoML UI for selecting, configuring, and joining multiple feature tables to an experiment's training dataset.
* [AutoML UI Feature Table Selection](/concepts/automl-ui-feature-selection.md) - The workflow in the Databricks AutoML UI for selecting and joining feature tables to an experiment through the 'Join features' option
* [AutoML Warnings and Diagnostics](/concepts/automl-warnings-and-diagnostics.md) - AutoML feature that displays warnings for potential dataset issues such as unsupported column types or high cardinality columns
* [AutoMLSummary](/concepts/automlsummary.md) - Summary object returned by AutoML runs that describes metrics, parameters, and trial details, and provides methods to load trained models.
* [AutoModel and AutoTokenizer for Fine-tuning](/concepts/automodel-and-autotokenizer-for-fine-tuning.md) - Using Hugging Face AutoModelForSequenceClassification and AutoTokenizer to load pre-trained models and tokenize data for fine-tuning tasks.
* [AutoModelForSequenceClassification](/concepts/automodelforsequenceclassification.md) - A Hugging Face Auto Model class that loads a pre-trained transformer model with a classification head for text classification fine-tuning.
* [AutoModelForSequenceClassification for Fine-tuning](/concepts/automodelforsequenceclassification-for-fine-tuning.md) - Loading and configuring pre-trained sequence classification models with label mappings for fine-tuning tasks
* [Autoscaling Limitation with ray.data.from_spark()](/concepts/autoscaling-limitation-with-raydatafrom_spark.md) - Autoscaling Spark clusters (including spot instances) must set use_spark_chunk_api=False when using ray.data.from_spark() to avoid cache misses caused by executor termination.
* [Autoscaling Restriction for Distributed XGBoost](/concepts/autoscaling-restriction-for-distributed-xgboost.md) - Distributed XGBoost training cannot be used on clusters with autoscaling enabled because new worker nodes remain idle
* [AutoTokenizer and Tokenization for Fine-tuning](/concepts/autotokenizer-and-tokenization-for-fine-tuning.md) - Using AutoTokenizer with Hugging Face datasets to tokenize text for model training
* [AutoTokenizer for Transformers](/concepts/autotokenizer-for-transformers.md) - A Hugging Face Auto class that loads the tokenizer corresponding to a pre-trained model, ensuring consistent tokenization across training and inference datasets.
* [Available DeepEval Metric Categories](/concepts/deepeval-metric-categories.md) - DeepEval provides five categories of metrics: RAG, Agentic, Conversational, Safety, and Non-LLM metrics
* [Avoid Spark Caching with Delta Lake](/concepts/avoid-spark-caching-with-delta-lake.md) - Databricks recommends against using Spark caching for Delta tables because it breaks data skipping optimizations and can serve stale data if the table is accessed via a different identifier.
* [Avoiding Data Silos in Lakehouse](/concepts/avoiding-data-silos-in-lakehouse.md) - Principles and best practices for preventing data silos by using Unity Catalog views, OpenSharing, single sources of truth, and lineage tracking instead of copying or duplicating datasets across departments.
* [AWS Graviton Support for Databricks Runtime ML](/concepts/aws-graviton-support-in-databricks-runtime-ml.md) - Support for Graviton instance types starting from Databricks Runtime 15.4 LTS ML, offering improved price-to-performance for Spark, Photon, and ML libraries
* [AWS Graviton Support in Databricks Runtime ML](/concepts/aws-graviton-support-in-databricks-runtime-ml.md) - Support for AWS Graviton (ARM-based) instance types in Databricks Runtime 15.4 LTS ML and above, offering improved performance for Spark, Photon, feature engineering, XGBoost, LightGBM, and Spark MLlib gradient boosting.
* [AWS S3 Bucket Naming Restrictions for Delta Sharing](/concepts/aws-s3-bucket-naming-restrictions-for-delta-sharing.md) - Dots (periods) in S3 bucket names cause SSL/certificate errors when Delta Sharing generates pre-signed URLs, due to an AWS limitation on bucket naming conventions.
* [Axolotl](/concepts/axolotl.md) - A high-performance framework for LLM fine-tuning that supports QLoRA and other efficient training techniques, with integration for MLflow, HuggingFace, and distributed training
* [Axolotl Framework for LLM Fine-Tuning](/concepts/axolotl.md) - A high-performance framework for post-training large language models, supporting QLoRA and multi-GPU distributed training
* [Azure Data Factory Orchestration for Databricks Pipelines](/concepts/azure-data-factory-orchestration-for-databricks-pipelines.md) - Running pipeline updates from Azure Data Factory using Web activities that call the Databricks Pipelines REST API endpoint.
* [Backfill Scorers](/concepts/backfill-scorers.md) - Technique to retrospectively apply evaluation scorers to historical MLflow experiment traces within a specified date range.
* [Backfill Scorers (MLflow GenAI)](/concepts/backfill-scorers-mlflow-genai.md) - Technique for retroactively applying scoring functions to historical trace data within a specified time range using the databricks.agents.scorers.backfill_scorers() API.
* [BackfillScorerConfig](/concepts/backfillscorerconfig.md) - Configuration object used to specify custom sample rates for individual scorers during a backfill operation.
* [BackfillScorers](/concepts/backfillscorers.md) - A Databricks function that applies scorers to historical trace data over a specified time range, returning a job ID.
* [Base Environment vs AI Environment in AI Runtime](/concepts/default-vs-ai-environments-in-ai-runtime.md) - Two environment options for AI Runtime: the default base environment (None) and the pre-configured AI environment (AI v4 / databricks_ai_v5)
* [BASE_INDEX in Delta Streaming Offsets](/concepts/base_index-in-delta-streaming-offsets.md) - A required offset index format for Delta streaming skip offsets, validated as part of the INVALID_INDEX subcondition of the DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE error.
* [BASE_INDEX Offset Index Requirement](/concepts/base_index-offset-index-requirement.md) - The requirement that when an invalid offset index occurs in a Delta streaming skip offset range, the index must be of the form BASE_INDEX (baseIndex)
* [BASE_INDEX Offset Reference](/concepts/base_index-offset-reference.md) - A requirement that Delta streaming skip offset indices must be expressed as BASE_INDEX, a specific reference format for identifying offsets in a Delta source stream.
* [Base64 Encoding for Media in ML Inference](/concepts/base64-encoding-for-media-in-ml-inference.md) - The technical process of encoding local audio or video files as base64 strings for inline transmission to model serving endpoints, using Python's base64 library.
* [Baseline Drift](/concepts/baseline-drift.md) - A drift comparison method that compares a data window to a baseline distribution defined by a separate baseline table, used when a static reference distribution exists.
* [Baseline Table](/concepts/baseline-table.md) - An optional reference table in Databricks data profiling used to compute drift metrics relative to expected data distributions and quality standards.
* [Baseline Table Comparison for Data Drift](/concepts/baseline-table-comparison-for-data-drift.md) - The practice of comparing current data distribution against a baseline table to detect statistical drift and trigger retraining of models.
* [Baseline Table for Data Profiling](/concepts/baseline-table-for-data-profiling.md) - An optional reference table used in data profiling to measure drift; it should reflect expected data quality, distributions, and match the primary table's schema.
* [Batch Inference on Databricks](/concepts/batch-inference-on-databricks.md) - Databricks' recommended approach for running batch predictions on large datasets using AI Functions and SQL queries
* [Batch Inference Pipelines](/concepts/batch-inference-pipelines.md) - A Databricks pattern for deploying and orchestrating batch inference workloads as pipelines, enabling scalable processing of large datasets.
* [Batch Inference with ai_query on Databricks](/concepts/batch-inference-with-ai_query-on-databricks.md) - Databricks recommends using the ai_query AI function as the modern replacement for legacy Hugging Face batch inference patterns
* [Batch Processing with Nested Span Hierarchies](/concepts/batch-processing-with-nested-span-hierarchies.md) - Pattern for tracking complex multi-level workflows by creating nested span hierarchies for each item in a batch, with structured error handling per item.
* [Bearer Token Authentication for Delta Sharing](/concepts/bearer-token-authentication-for-open-sharing.md) - A token-based authentication method for Open Sharing recipients where Databricks generates a bearer token, a credential file, and an activation link for the recipient to download credentials and access shared data.
* [Bearer Token Authentication for Delta Sharing Recipients](/concepts/bearer-token-authentication-for-open-sharing.md) - A Databricks-to-Open sharing authentication method where the provider creates a recipient object, Databricks generates a bearer token and credential file, and the recipient authenticates via an activation link.
* [Bearer Token Authentication for Open Sharing](/concepts/bearer-token-authentication-for-open-sharing.md) - A token-based authentication method for open sharing where Databricks generates a bearer token, a credential file, and an activation link for the recipient to download and use for read access to shared data.
* [Bearer Token Authentication for OpenSharing](/concepts/bearer-token-authentication-for-opensharing.md) - Alternative authentication method for non-Databricks users accessing Databricks-to-Open sharing, contrasted with OIDC federation in this guide.
* [Best Practices for ABAC GRANT Policies](/concepts/best-practices-for-abac-grant-policies.md) - Design recommendations for GRANT policies including using groups over individual users, attaching at smallest scope, using tag inheritance, and avoiding mixing with direct grants.
* [Best Practices for Adding Context to MLflow Traces](/concepts/best-practices-for-adding-context-to-mlflow-traces.md) - Guidelines including consistent ID formats, session boundaries, environment variables for metadata, combining context types, and regular analysis
* [Best Practices for Deep Learning on Databricks](/concepts/best-practices-for-deep-learning-on-databricks.md) - Optimization guidelines and recommended workflows for developing and fine-tuning deep learning models efficiently on the Databricks platform.
* [Best Practices for Judge Instructions](/concepts/best-practices-for-judge-instructions.md) - Guidelines for writing effective LLM judge prompts including specifying output format (categorical, boolean, or numeric) and breaking down complex evaluations into structured sections.
* [Best practices for labeling schema design](/concepts/best-practices-for-labeling-schema-design.md) - Guidelines for writing clear questions, providing reviewer context, setting reasonable input limits, ensuring mutually exclusive categorical options, and managing schema lifecycles.
* [Best practices for LLM judge instructions](/concepts/best-practices-for-judge-instructions.md) - Guidelines for writing effective judge prompts: specifying output format (categorical, boolean, or numeric), breaking complex evaluations into clear sections, and aligning with human expert feedback.
* [Best Practices for MLflow Experiment Organization](/concepts/best-practices-for-mlflow-experiment-organization.md) - Recommendations for structuring MLflow experiments in Databricks apps including logical grouping by project or model type, consistent naming conventions, and retention policy management.
* [Best Practices for MLflow Trace Context](/concepts/best-practices-for-mlflow-trace-context.md) - Guidelines for adding context to traces including consistent ID formats, defining session boundaries, using environment variables for metadata, combining context types, and setting up dashboards.
* [Best Practices for mlflow.search_traces()](/concepts/best-practices-for-mlflowsearch_traces.md) - Recommended practices including using keyword arguments, proper filter_string formatting (prefixes, single quotes, backticks, timestamps), and avoiding unsupported operators like OR.
* [Best Practices for Ray on Databricks](/concepts/best-practices-for-ray-on-databricks.md) - Collection of optimization and troubleshooting guidance including GPU allocation, pickling errors, OOM errors, transformers MLflow integration, and batch data processing
* [Best Practices for Trace Context Management](/concepts/best-practices-for-trace-context-management.md) - Guidelines for consistent ID formats, session boundary definitions, environment variable usage for metadata, combining context types, and regular trace analysis.
* [Best Practices for Writing Evaluation Guidelines](/concepts/best-practices-for-writing-evaluation-guidelines.md) - Principles for crafting effective pass/fail natural language criteria including specificity, clear conditions, context references, and structured multi-constraint guidelines.
* [Best Practices for Writing Judge Instructions](/concepts/best-practices-for-judge-instructions.md) - Guidelines for crafting effective judge prompts, including specifying expected output formats (categorical, boolean, numeric), and breaking complex evaluations into clear sections covering what to evaluate, what information to examine, how to judge, and what format to return.
* [Beta Feature Lifecycle at Databricks](/concepts/beta-feature-lifecycle-at-databricks.md) - The Beta release stage for Databricks features, indicating the AI Runtime CLI is not yet GA and may have limited availability.
* [Beta features in Databricks](/concepts/beta-features-in-databricks.md) - Databricks release process designates certain features as Beta, indicating they are not yet generally available and may have limitations.
* [Beta Scope and Limitations of GRANT Policies](/concepts/beta-scope-and-limitations-of-grant-policies.md) - In Beta, GRANT policies only support EXECUTE privilege on models (MLflow and foundation models), cannot grant USE CATALOG/SCHEMA, and policies can only be attached at catalog or schema level, not model level.
* [Billing portal LakehouseMonitoring tag](/concepts/billing-portal-lakehousemonitoring-tag.md) - A method to view data profiling expenses via the Databricks account console billing portal using the LakehouseMonitoring tag key set to true.
* [Bin-packing optimization (Delta Lake)](/concepts/bin-packing-optimization-delta-lake.md) - An idempotent compaction technique that balances data files by size on disk, producing evenly-sized files without effect on re-run.
* [Bloom Filter Index](/concepts/bloom-filter-index.md) - A legacy indexing technique for Delta tables in Databricks used to skip irrelevant data files during query execution.
* [Bloom Filter Index (Databricks)](/concepts/bloom-filter-index.md) - A deprecated indexing feature in Databricks that used Bloom filters to speed up data skipping during queries on Delta tables.
* [Bloom Filter Index (Deprecated)](/concepts/bloom-filter-index.md) - A deprecated indexing feature in Databricks that used Bloom filters to optimize data skipping during queries, now replaced by alternative optimizations.
* [Bloom Filter Index Cleanup Semantics](/concepts/bloom-filter-index-cleanup-semantics.md) - When a Bloom filter index is dropped, all related metadata is removed from specified columns and underlying index files are cleaned only after a vacuum
* [Bloom Filter Index Deprecation](/concepts/bloom-filter-index-deprecation.md) - Bloom filter indexes are deprecated in Databricks and should not be created anymore.
* [Bloom Filter Index Deprecation (Databricks)](/concepts/bloom-filter-index-deprecation.md) - Bloom filter indexes are deprecated in Databricks, with migration guidance to predictive I/O or liquid clustering.
* [Bloom filter index deprecation in Databricks](/concepts/bloom-filter-index-deprecation.md) - Bloom filter indexes are deprecated in Databricks; users are recommended to drop all existing Bloom filter indexes
* [Bloom Filter Index Deprecation on Databricks](/concepts/bloom-filter-index-deprecation.md) - Official recommendation from Databricks to drop all existing Bloom filter indexes and migrate to recommended alternatives.
* [Bloom Filter Index Lifecycle and Cleanup](/concepts/bloom-filter-index-lifecycle-and-cleanup.md) - Bloom filter index files on disk are only physically removed when the table is vacuumed after dropping the index.
* [Bloom filter index metadata cleanup on DROP](/concepts/bloom-filter-index-metadata-cleanup-on-drop.md) - Dropping a Bloom filter index removes all Bloom filter related metadata from specified columns or entire table
* [Bloom Filter Index Metadata Removal](/concepts/bloom-filter-index-metadata-removal.md) - All Bloom filter-related metadata is removed from specified columns when the DROP BLOOMFILTER INDEX command succeeds.
* [Bloom filter indexes](/concepts/bloom-filter-index.md) - A deprecated indexing feature in Delta Lake used to skip data files during reads; users are advised to migrate to Predictive I/O or Liquid Clustering.
* [Bloom Filter Indexes (Databricks)](/concepts/bloom-filter-index.md) - A deprecated indexing technique in Databricks that uses Bloom filters to skip non-matching files during query execution, improving performance on equality predicates.
* [Bloom Filter Metadata Cleanup](/concepts/bloom-filter-metadata-cleanup.md) - Behavior of the DROP command to remove all Bloom filter related metadata from specified columns, with index file cleanup occurring on VACUUM.
* [Boolean Slice Expressions for Protected Group Definition](/concepts/boolean-slice-expressions-for-protected-group-definition.md) - A technique for defining protected and unprotected groups in fairness analysis using Boolean expressions that partition data into slices
* [brickster package db_repl function](/concepts/brickster-package-db_repl-function.md) - Alternative approach using the brickster package's db_repl function to use all sparklyr's Spark MLlib functions when Databricks Connect's DataFrame-API-only limitation is a blocker.
* [Budget Policy Permission Requirements](/concepts/budget-policy-permission-requirements.md) - Users and service principals must have explicit entitlement to use a budget policy before they can assign it to an MLflow experiment.
* [Build Logs for Model Serving](/concepts/build-logs-for-model-serving.md) - Displays output from the process that creates a production-ready Python environment for model serving endpoints, retained for up to 30 days.
* [Building a Scala JAR with sbt for Databricks](/concepts/building-a-scala-jar-with-sbt-for-databricks.md) - Step-by-step process to create a Scala project with sbt, configure build.sbt with correct versions, add sbt-assembly plugin, and produce a fat JAR for Databricks serverless compute.
* [Building JARs with sbt for Databricks Serverless](/concepts/building-jars-with-sbt-for-databricks-serverless.md) - Step-by-step process to create a Scala JAR compatible with Databricks serverless using sbt, with sbt-assembly plugin, matching JDK 17, Scala 2.13, and Databricks Connect versions.
* [Building Python Wheels with setuptools](/concepts/building-python-wheels-with-setuptools.md) - The process of using setuptools and the wheel package to compile Python source code into a .whl distribution file via the 'python3 setup.py bdist_wheel' command.
* [Building Scala JARs with sbt-assembly for Serverless](/concepts/building-scala-jars-with-sbt-assembly-for-serverless.md) - Step-by-step process to build a serverless-compatible Scala JAR using sbt and sbt-assembly plugin
* [Built-in Conversation Scorers](/concepts/built-in-conversation-scorers.md) - Predefined multi-turn judges like ConversationCompleteness, UserFrustration, and Safety provided by MLflow for evaluating conversation quality without custom development.
* [Built-in Hive Metastore](/concepts/built-in-hive-metastore.md) - A per-workspace managed service in Databricks that deploys an instance to each cluster and provides centralized metadata storage for the workspace's Hive tables.
* [Built-in Judges](/concepts/built-in-judges.md) - Pre-defined evaluation judges in MLflow/Databricks such as RelevanceToQuery, Safety, and Correctness, which can be used directly or aligned to a domain.
* [Built-in Judges (MLflow)](/concepts/built-in-judges-mlflow.md) - Pre-configured LLM-based judges for assessing GenAI quality dimensions including safety, relevance, groundedness, and correctness.
* [Built-in Judges vs Custom Judges](/concepts/built-in-judges-vs-custom-judges.md) - The distinction between predefined MLflow judges (like RelevanceToQuery, Safety, Correctness) and user-defined judges created with make_judge(), both of which support the alignment workflow.
* [Built-in LLM Judge Labeling Schemas](/concepts/built-in-llm-judge-schemas.md) - Predefined schema names (e.g., EXPECTED_FACTS, GUIDELINES, EXPECTED_RESPONSE) provided by MLflow for compatibility with built-in LLM Judges that use the Expectation type.
* [Built-in LLM Judge schemas](/concepts/built-in-llm-judge-schemas.md) - Predefined labeling schema names (EXPECTED_FACTS, GUIDELINES, EXPECTED_RESPONSE) provided by MLflow for compatibility with built-in LLM Judges.
* [Built-in LLM Judges](/concepts/built-in-llm-judges.md) - Predefined scorers that use Databricks-hosted LLMs to evaluate common quality dimensions of GenAI applications such as relevance, safety, groundedness, and correctness.
* [Built-in LLM judges predefined schemas](/concepts/built-in-llm-judge-schemas.md) - Predefined labeling schema names (EXPECTED_FACTS, GUIDELINES, EXPECTED_RESPONSE) that ensure compatibility with MLflow's built-in LLM evaluation judges.
* [Built-in Tools (Responses API)](/concepts/built-in-tools-responses-api.md) - Platform-provided tool capabilities like apply_patch that return structured outputs and are fully managed by the platform, requiring no backend implementation from the user.
* [Built-in vs Custom Judge Alignment](/concepts/built-in-vs-custom-judge-alignment.md) - The distinction that alignment applies to both built-in judges (generic criteria adapted to domain) and custom judges (refined specialized evaluation logic), using the same align() method.
* [BY POSITION Column Mapping](/concepts/by-position-column-mapping.md) - Syntax for matching source columns to target Delta table columns by ordinal position, primarily for headerless CSV files.
* [BY POSITION Column Matching](/concepts/by-position-column-matching.md) - A syntax option for matching source columns to target table columns by ordinal position, supporting headerless CSV files
* [BY_CLEAN_ROOMS Sub-Error](/concepts/by_clean_rooms-sub-error.md) - A sub-condition of DELTA_SHARING_SECURABLE_DELETE_BLOCKED that appears when the securable is shared through Clean Rooms, listing the central clean room IDs.
* [CACHE SELECT](/concepts/cache-select.md) - A Databricks SQL construct that caches data accessed by a simple SELECT query in the disk cache, supporting column and row subsets.
* [CACHE SELECT (Databricks SQL)](/concepts/cache-select.md) - SQL command that caches data from a SELECT query into the Databricks disk cache, supporting column and row subset selection for Delta and Parquet tables.
* [CACHE SELECT with Views](/concepts/cache-select-with-views.md) - Support for caching views with CACHE SELECT, but with the restriction that the expanded view query must be a simple SELECT as defined by the syntax.
* [Cache Selectivity (Databricks)](/concepts/cache-selectivity-databricks.md) - The ability to cache only a subset of columns (via column list) and rows (via WHERE predicate) from a table, reducing cache footprint and improving performance.
* [CAN EDIT METADATA Permission](/concepts/can-edit-metadata-permission.md) - A permission level allowing users to view and edit a feature table's description and metadata
* [CAN MANAGE Permission](/concepts/can-manage-permission.md) - The highest permission level for feature tables, allowing management of other users' permissions and deletion of the table
* [CAN VIEW METADATA Permission](/concepts/can-view-metadata-permission.md) - A permission level allowing users to view a feature table in the UI and see its metadata
* [CANNOT_FINALIZE_REDIRECT](/concepts/cannot_finalize_redirect.md) - A sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED indicating the migration cannot finalize because the redirect configuration does not exist, possibly due to a concurrent rollback to external.
* [CANNOT_FINALIZE_REDIRECT error](/concepts/cannot_finalize_redirect.md) - A sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED raised when the redirect configuration for an external location does not exist, often during a rollback-to-external scenario.
* [CANNOT_FINALIZE_REDIRECT Sub-Error](/concepts/cannot_finalize_redirect.md) - Sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED triggered when redirect configuration is missing during table migration
* [Capybara Dataset](/concepts/capybara-dataset.md) - A conversational AI training dataset from the TRL library used for supervised fine-tuning of language models
* [Capybara Dataset for Conversational AI Training](/concepts/capybara-dataset.md) - A dataset from the TRL library used for conversational AI training, employed as the training dataset in the supervised fine-tuning example for Llama 3.2 1B.
* [Cast-compatible masking functions](/concepts/cast-compatible-masking-functions.md) - Patterns for designing row filter and column mask UDFs where the return type is automatically cast to match the target column's data type, including techniques to avoid numeric overflow and return compatible types in all branches.
* [Catalog and Schema](/concepts/catalog-and-schema.md) - Catalogs are the first-level and schemas the second-level container objects in Unity Catalog's three-level namespace, used to organize data and AI assets by organizational unit, project, or lifecycle scope.
* [Catalog Explorer](/concepts/catalog-explorer.md) - The Databricks UI tool for navigating, discovering, and managing Unity Catalog assets including tables and their quality configurations.
* [Catalog Explorer for OpenSharing](/concepts/catalog-explorer-for-opensharing.md) - The Databricks UI tool used to visually manage OpenSharing shares, recipients, and grants through the Catalog pane.
* [Catalog Explorer Lineage Graph](/concepts/catalog-explorer-lineage-tab.md) - Visual UI in Databricks Catalog Explorer showing lineage relationships between feature tables, models, and functions with an interactive graph view
* [Catalog Explorer Lineage Tab](/concepts/catalog-explorer-lineage-tab.md) - The UI surface in Databricks Catalog Explorer where users can view lineage for tables, model versions, and functions, including a 'See lineage graph' option for a visual graph view.
* [Catalog Explorer Lineage Visualization](/concepts/catalog-explorer-lineage-tab.md) - Interactive graph interface in Databricks Catalog Explorer showing table, job, dashboard, and column-level lineage with expandable nodes and a Lineage details panel.
* [Catalog Explorer Upgrade Wizard](/concepts/catalog-explorer-upgrade-wizard.md) - UI-based tool in Databricks Catalog Explorer for upgrading Hive schemas and tables to Unity Catalog external tables with destination mapping, ownership assignment, and notebook generation.
* [Centrality Measures (Degree and Betweenness)](/concepts/centrality-measures-degree-and-betweenness.md) - Metrics for measuring the importance of a node within a network, including degree centrality and betweenness centrality
* [Centralized Feature Store Architecture](/concepts/centralized-feature-store-architecture.md) - Architectural pattern designating a single Databricks workspace to store all feature store metadata, with multiple workspaces accessing it for creating, writing to, or reading from feature tables.
* [Certification Status System Tag](/concepts/certification-status-system-tag.md) - A system-governed tag (system.certification_status) in Unity Catalog that marks data assets as certified or deprecated to indicate data quality and lifecycle status.
* [Certified and Deprecated Data Flags](/concepts/certified-and-deprecated-data-flags.md) - System tags that mark data assets as certified (trusted, meeting quality standards) or deprecated (outdated, unsuitable for new use), influencing search and inline display.
* [Certified and Deprecated Data Signals](/concepts/certified-and-deprecated-data-flags.md) - System tags applied by admins to mark data assets as certified (trusted, meeting quality standards) or deprecated (outdated, unsuitable for new use), influencing search and inline display.
* [Certified Data (Unity Catalog)](/concepts/certified-data-unity-catalog.md) - System tags applied to data assets to mark them as trusted and meeting quality standards in Unity Catalog.
* [Chaining MLflow Evaluation Results](/concepts/chaining-mlflow-evaluation-results.md) - A pattern using evaluation results to filter problematic traces and re-evaluate them with different scorers or an updated application for targeted iteration.
* [Champion/Challenger Model Deployment Strategy](/concepts/championchallenger-model-deployment-strategy.md) - A pattern where a newly trained and validated model is given the 'Challenger' alias and compared against the current production model (the 'Champion') before being promoted to production status.
* [Change Data Feed (CDC) Column Conflicts](/concepts/change-data-feed-column-conflict.md) - A specific conflict scenario where a table contains a user-defined column named '_change_type' that interferes with Delta Lake's Change Data Feed metadata, preventing row-level conflict detection.
* [Change Data Feed (CDF) for Efficient Profiling](/concepts/change-data-feed-cdf-for-efficient-profiling.md) - Best practice to enable change data feed on tables to process only newly appended data during TimeSeries and Inference profile refreshes instead of re-processing the entire table.
* [Change Data Feed (CDF) for Recipient-Side Replication](/concepts/change-data-feed-cdf-for-recipient-side-replication.md) - Enabling CDF on shared tables allows recipients to access changes and merge them into a local copy, limiting egress to the incremental refresh rather than full query access across regions.
* [Change Data Feed (CDF) Metadata Column Conflict](/concepts/change-data-feed-column-conflict.md) - A specific conflict scenario where a table column named '_change_type' conflicts with CDF metadata columns, preventing row-level conflict detection.
* [Change Data Feed (CDF) Metadata Column Conflicts](/concepts/change-data-feed-column-conflict.md) - A conflict scenario where a table contains a column named '_change_type' that interferes with Delta Lake's Change Data Feed metadata columns, preventing row-level conflict detection.
* [Change Data Feed (CDF) metadata conflicts](/concepts/change-data-feed-column-conflict.md) - A conflict scenario where a user-defined column named '_change_type' collides with Change Data Feed metadata columns, preventing row-level conflict detection in Delta Lake.
* [Change Data Feed Column Conflict](/concepts/change-data-feed-column-conflict.md) - A conflict condition where a user-defined column named '_change_type' clashes with Delta Lake's Change Data Feed metadata, preventing row-level conflict detection.
* [Change Data Feed Column Conflicts](/concepts/change-data-feed-column-conflict.md) - An error condition where a user-created column named '_change_type' conflicts with Change Data Feed (CDF) metadata columns, preventing row-level conflict detection in Delta Lake.
* [Change Data Feed for Delta Streaming](/concepts/change-data-feed-for-streaming.md) - Approach to use Delta Lake change data feed to capture inserts, updates, and deletes from source tables in streaming queries for robust change data capture (CDC).
* [Change Data Feed for Incremental Profiling](/concepts/change-data-feed-for-incremental-profiling.md) - A best practice to enable change data feed (CDF) on tables so profiling processes only newly appended data instead of re-processing the entire table on every refresh.
* [Change Data Feed for Streaming](/concepts/change-data-feed-for-streaming.md) - Using Delta Lake change data feed to stream all change types (inserts, updates, deletes) and handle them explicitly in downstream tables
* [Change Data Feed for streaming all change types](/concepts/change-data-feed-for-streaming.md) - Using Delta Lake change data feed to stream inserts, updates, and deletes from a source table, providing row-level change tracking for robust downstream processing.
* [Change Data Feed Metadata Column Conflict](/concepts/change-data-feed-column-conflict.md) - A conflict condition where a table contains a column named '_change_type' that clashes with Change Data Feed (CDC) metadata columns, preventing row-level conflict detection.
* [Change Data Feed Metadata Column Conflicts](/concepts/change-data-feed-column-conflict.md) - A conflict where user-defined columns named '_change_type' interfere with Change Data Feed (CDF) metadata columns, preventing row-level conflict detection in Delta Lake.
* [Change Data Feed Metadata Conflict](/concepts/change-data-feed-column-conflict.md) - A conflict condition where a user-defined column named '_change_type' collides with Change Data Feed (CDC) metadata columns, preventing row-level conflict detection in Delta Lake; resolution requires renaming the column or disabling CDC.
* [Change Data Feed Metadata Conflicts](/concepts/change-data-feed-column-conflict.md) - A conflict that occurs when a user-defined column named '_change_type' collides with Delta Lake's Change Data Feed metadata columns, preventing row-level conflict detection
* [Change Data Feed with OpenSharing](/concepts/change-data-feed-with-opensharing.md) - Reading incremental change data from OpenSharing tables using the readChangeFeed option in Spark DataFrames and Structured Streaming.
* [Chart Customization in MLflow](/concepts/chart-customization-in-mlflow.md) - Ability to add, move, resize, edit, delete, and download charts in the MLflow Chart View, including full-screen mode and kebab menu controls.
* [Chat Completion Response Format](/concepts/chat-completion-response-format.md) - Standard structure of responses from Databricks chat model endpoints, including choices, usage tokens, and metadata.
* [Chat Completions API](/concepts/chat-completions-api.md) - An API on Databricks for multi-turn conversations using 'messages' array, supporting content types including text, image, reasoning, and document content.
* [Chat Completions API (Databricks)](/concepts/chat-completions-api-databricks.md) - Multi-turn conversation API compatible with OpenAI's Chat Completions format, supporting system/assistant/user messages, tool calls, and streaming.
* [Chat Completions API for Multimodal Media Input](/concepts/chat-completions-api-for-multimodal-media-input.md) - How to use the OpenAI-compatible Chat Completions API on Databricks to pass video_url and audio_url content types in the messages array.
* [Chat Completions API for Vision Models](/concepts/chat-completions-api-for-vision-models.md) - The OpenAI-compatible Chat Completions API used to send text and image content to vision models hosted on Databricks, supporting multiple image inputs.
* [Chat Template Formatting for Fine-Tuning](/concepts/chat-template-formatting-for-fine-tuning.md) - The practice of applying a chat template (e.g., ChatML format) to conversational datasets when fine-tuning causal language models, ensuring proper structured conversation formatting for assistant-style responses.
* [Chat Template Processing with Unsloth](/concepts/chat-template-processing-with-unsloth.md) - The pipeline for converting conversational datasets (ShareGPT format) into model‑specific chat template text using Unsloth's get_chat_template and standardize_sharegpt utilities.
* [Checkpoint CRC Corruption Detection](/concepts/crc-checksum-corruption-detection.md) - Detection of CRC checksum corruption in Delta Lake checkpoint files, identified by the fileCrcCorrupt column in FSCK REPAIR TABLE output.
* [Checkpoint Model Ranking via MLflow API](/concepts/checkpoint-model-ranking-via-mlflow-api.md) - Using mlflow.search_logged_models to rank checkpoint models by metrics like accuracy, enabling programmatic selection of the best-performing model version.
* [Checkpoint V2 requirement](/concepts/checkpoint-v2-requirement.md) - APPLY (CHECKPOINT) requires the table to have Checkpoint V2 enabled to prevent corruption caused by race conditions.
* [Choosing Parallelism Techniques for Multi-GPU Training](/concepts/choosing-parallelism-techniques-for-multi-gpu-training.md) - The process of selecting between DDP, FSDP, and DeepSpeed based on model size, GPU memory capacity, and performance requirements.
* [Chronological Data Splitting in AutoML](/concepts/chronological-data-splitting-in-automl.md) - Using a time column to split data chronologically into training, validation, and test sets for classification and regression experiments, ensuring temporal order is preserved.
* [CI/CD for Machine Learning](/concepts/cicd-for-machine-learning.md) - Automated process for developing, deploying, monitoring, and maintaining ML applications, combining MLOps, DataOps, ModelOps, and DevOps principles.
* [CI/CD for Machine Learning (Databricks)](/concepts/cicd-for-machine-learning.md) - Automated process for developing, deploying, monitoring, and maintaining ML applications, applied to code, data pipelines, and model artifacts.
* [CI/CD for ML Pipelines](/concepts/cicd-for-ml-pipelines.md) - Integration of GitHub Actions or Azure DevOps with MLOps Stacks to automatically test and deploy ML code and resources across workspaces, ensuring only tested code reaches production.
* [Class-based Scorer Definition](/concepts/class-based-scorer-definition.md) - A stateful, Pydantic-based approach to defining MLflow scorers by subclassing the Scorer base class, supporting additional fields and complex logic (offline evaluation only).
* [Class-based Scorer for stateful evaluation](/concepts/class-based-scorer-for-stateful-evaluation.md) - A Pydantic-based Scorer base class for defining evaluation scorers that require state (additional fields), supported only for offline evaluation with mlflow.genai.evaluate()
* [Class-based Scorer limitations in production monitoring](/concepts/class-based-scorer-limitations-in-production-monitoring.md) - Class-based Scorer subclasses are not supported for production monitoring in Databricks MLflow; only @scorer-decorated functions work.
* [Class-based Scorer subclass limitation](/concepts/class-based-scorer-subclass-limitation.md) - Class-based Scorer subclasses are not supported for production monitoring in Databricks; only @scorer-decorated functions work
* [Class-based Scorer subclasses limitation](/concepts/class-based-scorer-subclass-limitation.md) - Class-based Scorer subclasses are not supported for production monitoring in Databricks
* [Class-based vs decorator-based scorer definitions](/concepts/class-based-vs-decorator-based-scorer-definitions.md) - Two approaches to defining MLflow scorers: the @scorer decorator for stateless functions (supported in both offline and production monitoring) and the Scorer base class for stateful Pydantic objects (offline evaluation only).
* [Classic Compute Forecasting](/concepts/classic-compute-forecasting.md) - The non-serverless alternative to Databricks Serverless Forecasting that requires managing compute resources, presented as a comparison to the serverless approach.
* [Classic ML on AI Runtime](/concepts/classic-ml-on-ai-runtime.md) - Notebook examples for traditional ML tasks including XGBoost training and time series forecasting on AI Runtime
* [Classic ML on Databricks AI Runtime](/concepts/classic-ml-on-databricks-ai-runtime.md) - Support for traditional machine learning tasks like XGBoost training and time series forecasting within the AI Runtime environment.
* [Classic ML on Databricks GPU](/concepts/classic-ml-on-databricks-gpu.md) - Notebook examples for traditional machine learning tasks on Databricks AI Runtime, including XGBoost model training and time series forecasting.
* [Classic ML vs Deep Learning vs GenAI](/concepts/classic-ml-vs-deep-learning-vs-genai.md) - The distinctions and overlaps between classic machine learning, deep learning, and generative AI paradigms, and the platform features that support all three.
* [Classification Results UI](/concepts/classification-results-ui.md) - The Databricks user interface that displays per-classification detection counts, auto-tagging status, user access patterns, and provides review panels for detected columns and user access details.
* [Classification Tags and Governed Tags System](/concepts/classification-tags-and-governed-tags-system.md) - A system of governed tags (personal_name, email_address, phone_number, etc.) organized by global, regional, and compliance frameworks (PII, GDPR, HIPAA, DPDPA) that are applied to columns when sensitive data is detected.
* [Classify](/concepts/classify.md) - The databricks.automl.classify method configures an AutoML run to train a classification model.
* [Claude Agent SDK (ClaudeSDKClient)](/concepts/claude-agent-sdk-claudesdkclient.md) - Anthropic's SDK client for building agentic interactions with Claude, supporting async query/response streaming patterns.
* [Clean Rooms in Databricks](/concepts/clean-rooms-in-databricks.md) - A data collaboration environment where shared objects are protected from deletion; the error message distinguishes clean room sharing from standard share-based sharing.
* [Client Options for Querying Models](/concepts/client-options-for-querying-models.md) - Multiple available client interfaces for querying Databricks models including OpenAI client, REST API, SQL, MLflow Deployments SDK, Databricks Python SDK, and LangChain.
* [Client Request IDs vs MLflow Trace IDs for Feedback Correlation](/concepts/client-request-ids-vs-mlflow-trace-ids-for-feedback-correlation.md) - Two approaches for linking user feedback to specific traces: using MLflow's auto-generated trace IDs or using application-generated client request IDs.
* [Client vs Server Concurrency Ratio](/concepts/client-vs-server-concurrency-ratio.md) - The optimal performance point for an ML serving system is determined by the ratio of client-side concurrency (number of simultaneous requesters) to server-side concurrency (provisioned parallel capacity), which load testing aims to identify.
* [Client-Server Architecture for Spark](/concepts/client-server-architecture-for-spark.md) - A decoupled architecture for Apache Spark using Spark Connect that allows remote connectivity to Spark clusters via unresolved logical plans and Apache Arrow over gRPC, enabling thin embedded clients.
* [Client-Server Architecture for Spark Jobs](/concepts/client-server-architecture-for-spark-jobs.md) - The architectural pattern where Databricks Connect sends logical representations of Spark DataFrame commands to a remote Spark server running in Databricks for execution, rather than running locally.
* [Client-Server Architecture of Databricks Connect](/concepts/client-server-architecture-of-databricks-connect.md) - The architectural pattern where Databricks Connect separates code execution into local (general code), remote cluster (DataFrame APIs), and cluster-side (UDFs), enabling interactive debugging on the client while leveraging Spark on Databricks for data transformations.
* [Client-Server Decoupled Architecture](/concepts/client-server-decoupled-architecture.md) - The architectural pattern where Databricks Connect separates local client code execution from remote Spark DataFrame API execution on Databricks compute clusters.
* [Client-Server Execution Model of Databricks Connect](/concepts/client-server-execution-model-of-databricks-connect.md) - A decoupled architecture where general code runs locally for interactive debugging, DataFrame APIs execute remotely on Databricks compute, and UDFs are serialized and transmitted to the cluster.
* [Client-Side Serving Optimization](/concepts/client-side-serving-optimization.md) - Strategies for client applications to improve interaction with serving endpoints, including connection pooling, error handling with retry logic and exponential backoff, and payload size minimization.
* [Client-Side Timeout – MLflow Configuration](/concepts/client-side-timeout-mlflow-configuration.md) - Timeouts caused by MLflow environment variables (MLFLOW_HTTP_REQUEST_TIMEOUT, MLFLOW_HTTP_REQUEST_MAX_RETRIES) that can be tuned for long-running model inference.
* [Client-Side Timeout – Third Party Client APIs](/concepts/client-side-timeout-third-party-client-apis.md) - Timeouts from third-party API clients (e.g., OpenAI SDK) used in custom PyFunc model pipelines, and how to configure their timeout parameters.
* [CLONE Limitations for Streaming Tables and Materialized Views](/concepts/clone-limitations-for-streaming-tables-and-materialized-views.md) - Streaming tables and materialized views cannot be used as source or target tables for the CLONE operation.
* [CLONE Parquet](/concepts/clone-parquet.md) - A Databricks SQL command to incrementally copy data from Parquet to Delta Lake, supporting both shallow and deep clones.
* [CLONE Restrictions with Streaming Tables and Materialized Views](/concepts/clone-limitations-for-streaming-tables-and-materialized-views.md) - Streaming tables and materialized views are not supported as source or target tables for CLONE operations.
* [Clone Source Table Compatibility](/concepts/clone-source-table-compatibility.md) - Delta, Parquet, and foreign Iceberg tables support both deep and shallow cloning; managed Iceberg tables support only deep cloning.
* [CLONE Syntax and Parameters](/concepts/clone-syntax-and-parameters.md) - The SQL syntax for cloning tables, including CREATE TABLE, REPLACE TABLE, IF NOT EXISTS, SHALLOW/DEEP CLONE, TBLPROPERTIES, and LOCATION clauses.
* [CLONE Table Limitations](/concepts/clone-table-limitations.md) - Streaming tables and materialized views cannot be used as source or target tables for CLONE operations in Databricks.
* [CLONE with IF NOT EXISTS](/concepts/clone-with-if-not-exists.md) - A SQL syntax modifier for CLONE that causes the operation to be silently ignored if the target table already exists.
* [CLONE with Unity Catalog](/concepts/clone-with-unity-catalog.md) - Support for shallow cloning Unity Catalog managed tables in Databricks SQL and Runtime 13.3 LTS+, with no shallow clone support in Runtime 12.2 LTS and below.
* [Clone Workflow Pattern for UC Migration](/concepts/clone-workflow-pattern-for-uc-migration.md) - Pattern of cloning existing training, deployment, and inference jobs as a starting point for building Unity Catalog-compatible parallel pipelines
* [Cloud Storage and Service Access Governance](/concepts/cloud-storage-and-service-access-governance.md) - Unity Catalog provides storage credentials (for cloud storage authentication), external locations (storage path + credential pairs), service credentials (for external cloud services), and connections (for external databases and services), governing access to external resources.
* [Cloud Token Access (Directory-based Access Mode)](/concepts/cloud-token-access-directory-based-access.md) - A mechanism where Databricks uses temporary, path-scoped cloud credentials to give recipients direct read access to shared Delta table files, bypassing the need for long-lived bearer tokens.
* [Cloud Token Access (Directory-Based Access)](/concepts/cloud-token-access-directory-based-access.md) - A mechanism using temporary, path-scoped cloud credentials to give recipients direct read access to shared Delta table files. Applicable to Databricks-to-Databricks and Databricks-to-Open sharing under specific conditions (full history, no partition filter, etc.). Grants read access to data files and the Delta log.
* [Cloud Token Access for Delta Sharing](/concepts/cloud-token-access-for-delta-sharing.md) - Temporary, path-scoped cloud credentials that give recipients direct read access to shared Delta table files, improving performance by avoiding server-side materialization.
* [Cloud Token Eligibility (Directory-Based Access Mode)](/concepts/cloud-token-access-directory-based-access.md) - A mechanism that uses temporary, path-scoped cloud credentials to give recipients direct read access to shared Delta table files without long-lived bearer tokens.
* [Cloudflare R2 Integration for Zero-Egress Sharing](/concepts/cloudflare-r2-integration-for-zero-egress-sharing.md) - Using Cloudflare R2 object storage as the backing store for OpenSharing shares to eliminate cloud vendor egress fees, including setup steps for mounting R2 buckets as external locations in Unity Catalog.
* [Cluster Connectivity Troubleshooting for Databricks Connect](/concepts/cluster-connectivity-troubleshooting-for-databricks-connect.md) - Common connectivity failures when Databricks Connect cannot reach a cluster, indicated by errors like StatusCode.UNAVAILABLE, DNS resolution failure, or HTTP 500 status codes.
* [Cluster ID configuration in Databricks Connect](/concepts/cluster-id-configuration-in-databricks-connect.md) - Methods for specifying cluster_id when connecting to classic compute, including inline, in config profiles, or via DATABRICKS_CLUSTER_ID environment variable.
* [Cluster-level Hive Metastore Access Control](/concepts/cluster-level-hms-access-control.md) - Using Spark configurations (spark.databricks.unityCatalogOnlyMode and spark.databricks.sql.initial.catalog.namespace) to disable Hive metastore access on individual compute clusters
* [Cluster-Level Hive Metastore Disablement](/concepts/cluster-level-hive-metastore-disablement.md) - Gradually disabling direct Hive metastore access on individual compute clusters via Spark configurations during Unity Catalog migration.
* [Cluster-Level HMS Access Control](/concepts/cluster-level-hms-access-control.md) - The ability to disable Hive metastore access on a per-cluster basis using Spark configurations (spark.databricks.unityCatalogOnlyMode and spark.databricks.sql.initial.catalog.namespace).
* [Cluster-Scoped Data Access Permissions](/concepts/cluster-scoped-data-access-permissions.md) - When using the Hive metastore alongside Unity Catalog, cluster-scoped credentials are used for Hive metastore data access but not for Unity Catalog data, and are also used for paths outside Unity Catalog.
* [Clustering columns](/concepts/clustering-columns.md) - Designated columns in a Liquid table used for data organization; certain commands on Liquid tables may only reference clustering columns in their predicates.
* [Clustering Columns in Delta Lake](/concepts/clustering-columns-in-delta-lake.md) - Columns designated for data clustering in Delta Lake Liquid tables; commands on Liquid tables may only reference these columns in predicates, with specific error sub-conditions for violations.
* [CNN Architecture for MNIST Classification in PyTorch](/concepts/cnn-architecture-for-mnist-classification-in-pytorch.md) - A simple CNN with two convolutional layers (kernel size 5), max pooling, dropout, and two fully connected layers, ending with log_softmax for 10-class digit classification.
* [COCO128 dataset](/concepts/coco128-dataset.md) - COCO128 is a subset of the COCO dataset used for training object detection models such as YOLO11n on Databricks serverless GPUs.
* [Code Path Parameter for Shared Modules](/concepts/code-path-parameter-for-shared-modules.md) - MLflow pyfunc's code_path parameter allows logging shared utility code alongside a model, making it importable from within custom pyfunc classes.
* [Code Path Sharing with mlflow pyfunc](/concepts/code-path-sharing-with-mlflow-pyfunc.md) - Using the code_path parameter in mlflow.pyfunc.log_model to share and re-use custom Python modules across logged pyfunc models
* [Code Source and Environment Management in AIR](/concepts/code-source-and-environment-management-in-air.md) - Mechanisms for packaging local code (snapshot with root_path) and declaring Python dependencies (environment block with version and dependency list) for serverless GPU training runs.
* [Code Source Management](/concepts/code-source-management.md) - Uploading local code to remote training nodes via the `code_source` block, supporting plain tarball snapshots, git-versioned snapshots, and folder filtering with `include_paths`.
* [Code Source Management in AI Runtime](/concepts/code-source-management-in-ai-runtime.md) - Mechanism for uploading local code directories to remote execution nodes using the code_source block and $CODE_SOURCE_PATH variable in workload configurations.
* [Code Source Snapshot](/concepts/code-source-snapshot.md) - Mechanism to upload local code to the remote execution node using a snapshot block in the YAML config, with $CODE_SOURCE_PATH as the recommended reference path.
* [Code source snapshot pattern](/concepts/code-source-snapshot-pattern.md) - Pattern for uploading local training code to AI Runtime via code_source.snapshot configuration, where $CODE_SOURCE_PATH resolves to the remote upload directory.
* [code_path for Shared Module Dependencies](/concepts/code_path-for-shared-module-dependencies.md) - The code_path parameter in mlflow.pyfunc.log_model() allows logging full code references (shared modules) that load into the model's context and are usable from custom pyfunc models.
* [code_path parameter for shared modules](/concepts/code_path-parameter-for-shared-modules.md) - MLflow pyfunc log_model's code_path parameter allows packaging shared code modules alongside custom models for reuse across deployments
* [code_paths Parameter for Custom Libraries](/concepts/code_paths-parameter-for-custom-libraries.md) - MLflow log_model parameter that allows inclusion of custom libraries stored outside Unity Catalog volumes or DBFS by embedding them in the model artifact's code directory
* [Code-based Scorers](/concepts/code-based-scorers.md) - Custom programmatic scorers offering maximum flexibility for defining evaluation metrics through heuristics, advanced logic, or custom LLM integration beyond what custom judges provide.
* [Code-based Scorers (MLflow GenAI)](/concepts/code-based-scorers-in-mlflow.md) - Custom evaluation metrics for GenAI applications defined using the @scorer decorator in MLflow Evaluation.
* [Code-based Scorers for MLflow GenAI](/concepts/code-based-scorers-in-mlflow.md) - Custom evaluation metrics defined as Python functions using the @scorer decorator pattern for evaluating GenAI applications in MLflow.
* [Code-based Scorers for MLflow GenAI Evaluation](/concepts/code-based-scorers-in-mlflow.md) - Custom Python functions decorated with @scorer that define flexible evaluation metrics for AI agents and applications in MLflow Evaluation for GenAI.
* [Code-based scorers in MLflow](/concepts/code-based-scorers-in-mlflow.md) - Custom Python functions or classes that define flexible evaluation metrics for GenAI agent evaluation, supporting primitive returns, Feedback objects, trace data access, and error handling.
* [Code-based scorers in MLflow GenAI](/concepts/code-based-scorers-in-mlflow.md) - Custom Python functions or classes decorated with @scorer that define flexible evaluation metrics for AI agents and applications in MLflow Evaluation.
* [Collaborative Editing via Group Ownership](/concepts/collaborative-editing-via-group-ownership.md) - Transferring ownership of a view or metric view to a group enables collaborative editing, where all group members can edit the definition while data access is limited to what the group has permission to see.
* [Column mapping in Delta Lake](/concepts/column-mapping-in-delta-lake.md) - A Delta Lake feature enabling column aliasing and rename without breaking existing data files, required for alias-based reconciliation queries.
* [Column Mapping Requirement for Delta Reconciliation Queries](/concepts/column-mapping-requirement-for-delta-reconciliation-queries.md) - Column mapping must be enabled on a Delta table to use an alias in a reconciliation query, otherwise a COLUMN_RENAME_WITHOUT_COLUMN_MAPPING error occurs.
* [Column Mask (CM) policies in Delta Lake](/concepts/column-mask-cm-policies-in-delta-lake.md) - A Delta table security feature that masks column values at query time, unsupported by external metadata sources.
* [Column Mask for Nested STRUCT Fields](/concepts/column-mask-for-nested-struct-fields.md) - A technique to apply column masks to individual fields within a STRUCT column by reconstructing the STRUCT with named_struct(), conditionally redacting sensitive sub-fields.
* [Column Mask Policies](/concepts/column-mask-policies.md) - ABAC policy type that enforces column-level security on tables, materialized views, and streaming tables by masking column values based on attributes
* [Column mask policies in Unity Catalog](/concepts/column-mask-policies.md) - ABAC policy type that supports column-level security on tables, materialized views, and streaming tables.
* [Column masks (Unity Catalog)](/concepts/column-masks-unity-catalog.md) - SQL UDFs applied to specific columns that transform or redact column values at query time based on user identity or other attributes.
* [Column Masks in Delta Lake](/concepts/column-masks-in-delta-lake.md) - A Delta Lake table feature for dynamically masking column values at read time, which is incompatible with REFRESH SYNC UNIFORM.
* [Column Masks in Unity Catalog](/concepts/column-masks-unity-catalog.md) - A column-level security feature that uses SQL UDFs to mask or transform column values at query time based on user identity or group membership.
* [Column masks with Delta Uniform](/concepts/column-masks-with-delta-uniform.md) - Column mask table features are incompatible with REFRESH SYNC UNIFORM operations on Delta tables
* [Column Selection Control in AutoML](/concepts/column-selection-in-automl.md) - Ability to specify which columns to include or exclude from training in Databricks AutoML via UI checkboxes or the exclude_cols API parameter.
* [Column Selection for AutoML Training](/concepts/column-selection-for-automl-training.md) - Ability to specify which columns AutoML should use for training by excluding columns via the UI or the exclude_cols API parameter; the prediction target and time column cannot be dropped.
* [Column Selection in AutoML](/concepts/column-selection-in-automl.md) - The ability to include or exclude specific columns from AutoML training via UI checkboxes or the exclude_cols API parameter.
* [Column-Level External Lineage](/concepts/column-level-external-lineage.md) - The ability to map columns between external metadata objects and Unity Catalog objects when configuring lineage relationships, enabling fine-grained field-level lineage tracking.
* [Column-Level Lineage](/concepts/column-level-lineage.md) - Granular tracking of data at the column level within Unity Catalog, showing how specific columns in downstream tables derive from upstream columns, with limitations for path-based references and UDFs.
* [Column-only predicate row filtering](/concepts/column-only-predicate-row-filtering.md) - Pattern for row filter policies that use simple boolean logic referencing only table columns, enabling predicate pushdown for better query performance on protected tables.
* [Column-only predicates for row filtering with predicate pushdown](/concepts/column-only-predicates-for-row-filtering-with-predicate-pushdown.md) - Using simple boolean expressions that reference only table columns in row filter UDFs to enable predicate pushdown, allowing the engine to skip irrelevant data during scans.
* [ColumnSelection (Pass-through Features)](/concepts/columnselection-pass-through-features.md) - A non-aggregating function that selects a single column from a data source, returning the latest value per entity via point-in-time join or passing through inference-time request data.
* [ColumnSelection Materialization](/concepts/columnselection-materialization.md) - Declarative features that select the latest single-column value per entity key without aggregation; can only materialize to online stores (not offline).
* [Combined Automatic and Manual Tracing](/concepts/combined-automatic-and-manual-tracing.md) - Hybrid approach that mixes automatic autolog() tracing with manual tracing for complete coverage
* [Combined Manual and Automatic Tracing](/concepts/combined-manual-and-automatic-tracing.md) - Pattern of using @mlflow.trace decorator and @mlflow.trace(span_type=...), alongside mlflow.autolog() to create unified traces that capture both library-internal calls and custom business logic.
* [Combined Tracing](/concepts/combined-tracing.md) - A hybrid MLflow tracing approach that mixes automatic and manual instrumentation for complete coverage of generative AI application logic.
* [Combined Tracing (MLflow)](/concepts/combined-tracing-mlflow.md) - Strategy that mixes automatic and manual tracing approaches for complete coverage of application traces.
* [Combined Tracing Approach](/concepts/combined-tracing.md) - Mixing automatic and manual tracing to achieve complete coverage of generative AI application traces.
* [Combining Autologging with Manual MLflow Tracking](/concepts/combining-autologging-with-manual-mlflow-tracking.md) - A pattern using mlflow.autolog(exclusive=False) and mlflow.start_run() to augment automatically captured autologging data with additional custom metrics, parameters, and artifacts.
* [Combining Manual and Automatic MLflow Tracing](/concepts/combining-manual-and-automatic-tracing.md) - Using @mlflow.trace decorators alongside autolog() to create unified traces that capture custom business logic and auto-traced LLM calls together.
* [Combining Manual and Automatic Tracing](/concepts/combining-manual-and-automatic-tracing.md) - Using `@mlflow.trace` decorator alongside automatic autolog tracing to create unified traces covering both LLM calls and custom business logic.
* [Combining Manual and Automatic Tracing in MLflow](/concepts/combining-manual-and-automatic-tracing.md) - Using @mlflow.trace decorators alongside mlflow.autolog() to create unified traces that capture both automatic LLM calls and custom business logic as a single trace hierarchy.
* [commit_freshness Struct](/concepts/commit_freshness-struct.md) - A struct in the anomaly detection results table that captures data freshness metrics for each scanned table.
* [Common AutoML Parameters](/concepts/common-automl-parameters.md) - Shared configuration parameters across classify, regress, and forecast methods including dataset, target_col, primary_metric, timeout_minutes, exclude_cols, exclude_frameworks, and feature_store_lookups.
* [Common Pitfalls of Low-Level Tracing APIs](/concepts/common-pitfalls-of-low-level-tracing-apis.md) - List of frequent mistakes when using client APIs including forgetting to end spans, incorrect parent-child relationships, mixing API levels, hardcoding IDs, and thread safety issues.
* [Common Table Expressions (CTE) in DELETE](/concepts/common-table-expressions-ctes-in-dml.md) - Optional use of CTEs with DELETE FROM to reuse named queries and improve complex deletion logic
* [Common Table Expressions (CTE) in DML](/concepts/common-table-expressions-cte-in-dml.md) - Use of named CTE queries within DELETE statements on Delta Lake to improve readability and avoid repeated computation.
* [Common Table Expressions (CTE) in UPDATE](/concepts/common-table-expressions-cte-in-update.md) - Named queries that can be reused within the UPDATE statement to avoid repeated computations or improve readability.
* [Common Table Expressions (CTE) with DELETE](/concepts/common-table-expressions-cte-in-dml.md) - A CTE can optionally prefix a DELETE FROM statement to define named subqueries that improve readability and avoid repeated computation.
* [Common Table Expressions (CTEs) in DML](/concepts/common-table-expressions-ctes-in-dml.md) - CTEs can be used with DELETE FROM statements to reuse named subqueries for improved readability and performance.
* [Common Table Expressions with DELETE FROM](/concepts/common-table-expressions-with-delete-from.md) - Databricks SQL allows using common table expressions (CTEs) as named queries that can be reused within the DELETE FROM statement to improve readability or avoid repeated computations.
* [Comparative Agent Evaluation with Toggles](/concepts/comparative-agent-evaluation-with-toggles.md) - A technique using global configuration flags (e.g., RESOLVE_ISSUES) to switch between agent behaviors and compare evaluation results across different agent configurations.
* [Comparative Prompt Evaluation](/concepts/comparative-prompt-evaluation.md) - The systematic process of evaluating multiple prompt versions against a shared dataset and scoring them to identify the best-performing version.
* [Comparative Prompt Evaluation Methodology](/concepts/comparative-prompt-evaluation.md) - The practice of evaluating multiple prompt versions against identical datasets and using weighted composite scores to objectively identify the best-performing version.
* [Comparative Prompt Evaluation Workflow](/concepts/comparative-prompt-evaluation-workflow.md) - Systematic methodology for evaluating multiple prompt versions against the same dataset using consistent scorers to identify the best performer.
* [Comparing Runs from Multiple Experiments](/concepts/comparing-runs-from-multiple-experiments.md) - Ability to display and compare MLflow runs across different experiments for broader analysis.
* [Compatibility matrix for Databricks Feature Engineering](/concepts/compatibility-matrix-for-databricks-feature-engineering.md) - A table mapping Databricks Runtime ML versions to the appropriate databricks-feature-engineering or databricks-feature-store package versions
* [Compatible Cloud Storage Tiers for Delta Archival](/concepts/compatible-cloud-storage-tiers-for-delta-archival.md) - The specific AWS S3 storage tiers compatible with Databricks archival support: S3 Glacier Deep Archive and Glacier Flexible Retrieval; excluding Glacier Instant Retrieval (no config needed) and Intelligent-Tiering asynchronous archive tiers.
* [Completeness (data quality metric)](/concepts/completeness-data-quality.md) - A metric measuring expected row counts in the last 24 hours, with anomalous low counts flagging a table as incomplete, and optionally augmented by percent-null analysis per column.
* [Completeness (data quality)](/concepts/completeness-data-quality.md) - A data quality metric tracking expected row count over 24 hours; anomaly detection predicts a range and flags tables falling below the lower bound as incomplete.
* [Completeness Monitoring (Databricks Anomaly Detection)](/concepts/completeness-monitoring-databricks-anomaly-detection.md) - A quality dimension that monitors the expected number of rows written to a table in the last 24 hours by analyzing historical row counts and flagging tables as incomplete if row counts fall below predicted bounds.
* [Completions API](/concepts/completions-api.md) - Text completion API for generating responses to single prompts, supporting batched independent prompts in one request.
* [Completions API (Databricks)](/concepts/completions-api-databricks.md) - Text completion API for single prompts supporting batched inputs of multiple independent prompts in one request.
* [Complex RAG Pipeline Tracing](/concepts/complex-rag-pipeline-tracing.md) - Comprehensive pattern for instrumenting a full RAG pipeline with multiple span types (CHAIN, RETRIEVER, CHAT_MODEL, TOOL), custom tags, Document objects, chat tools, and token usage tracking.
* [Compliance Framework Reference (Databricks)](/concepts/compliance-framework-reference-databricks.md) - A mapping of classification tags to common regulatory compliance frameworks including PII, PCI DSS, GDPR, HIPAA, GLBA, DPDPA, and PIPEDA.
* [Compliance Security Profile (Databricks on AWS)](/concepts/compliance-security-profile-databricks-on-aws.md) - A compliance framework for Databricks Model Serving that enforces standards like HIPAA, PCI-DSS, FedRAMP, IRAP, CCCS, and UK Cyber Essentials Plus, requiring container rebuilds within the last 30 days.
* [Compliance Security Profile Standards for Model Serving](/concepts/compliance-security-profile-standards-for-model-serving.md) - Compliance standards (e.g., UK Cyber Essentials Plus) that apply to served model containers, requiring containers to be rebuilt within the most recent 30 days.
* [Composite Scoring for Prompt Selection](/concepts/composite-scoring-for-prompt-selection.md) - A technique for combining multiple evaluation metrics (e.g., correctness and compliance) with weighted scores to produce a single quantitative ranking for prompt version selection.
* [Comprehensive Trace Analysis Utility](/concepts/comprehensive-trace-analysis-utility.md) - Pattern for building a complete trace analyzer that extracts basic info, tags, token usage, span types, retrieval metrics, assessments, performance breakdown, and data flow from a trace.
* [Compute configuration for Databricks Connect](/concepts/compute-configuration-for-databricks-connect.md) - Configuration details for setting up Databricks compute clusters to work with Databricks Connect.
* [Compute Migration for Unity Catalog](/concepts/compute-migration-for-unity-catalog.md) - The requirement to upgrade all-purpose and job compute resources to Databricks Runtime 13.3 LTS or above and supported access modes for Unity Catalog compatibility.
* [Compute Options for dbt Tasks in Databricks](/concepts/compute-options-for-dbt-tasks-in-databricks.md) - dbt tasks can run against serverless SQL warehouses, pro SQL warehouses, or Databricks all-purpose compute, but not job compute; SQL warehouse is recommended for most use cases.
* [Compute Resource Specification](/concepts/compute-resource-specification.md) - Configuration of GPU resources for a training workload using `num_accelerators` and `accelerator_type` fields, supporting types like GPU_1xA10 and GPU_8xH100.
* [Compute Scale-Out and Scale-to-Zero](/concepts/compute-scale-out-and-scale-to-zero.md) - Model serving endpoints support Small/Medium/Large compute scale-out sizes (0-4, 8-16, 16-64 concurrent requests) and optional scale-to-zero, though scale-to-zero incurs cold-start latency and is not recommended for production.
* [Compute Scale-out Configuration](/concepts/compute-scale-out-configuration.md) - Configuring concurrency and scaling for serving endpoints using Small (0-4), Medium (8-16), or Large (16-64) scale-out sizes, with optional scale-to-zero for non-production use.
* [Compute type options for Model Serving endpoints](/concepts/compute-type-options-for-model-serving-endpoints.md) - Model Serving provides CPU and GPU compute types including CPU_MEDIUM and CPU_LARGE variants that trade concurrency for higher memory per worker; GPU requires explicit code setup for GPU-based predictions, though MLflow handles this automatically for PyTorch and Transformers flavors.
* [Compute Types for Databricks Model Serving](/concepts/compute-types-for-databricks-model-serving.md) - Variety of CPU and GPU workload types for model deployment, including CPU_MEDIUM and CPU_LARGE which trade concurrency for more memory per worker on the same CPU hardware.
* [Compute-Scoped vs Notebook-Scoped Library Installation](/concepts/compute-scoped-vs-notebook-scoped-library-installation.md) - Two methods for installing additional libraries on Databricks: compute-scoped libraries (available to all notebooks on a compute resource) and notebook-scoped Python libraries (available only to a specific notebook session).
* [Computer Vision Model Training with RetinaNet on Databricks](/concepts/computer-vision-model-training-with-retinanet-on-databricks.md) - A tutorial demonstrating how to train a RetinaNet object detection model on serverless GPUs using Databricks AI Runtime.
* [Computer Vision Model Training with YOLO on Databricks](/concepts/computer-vision-model-training-with-yolo-on-databricks.md) - A tutorial demonstrating how to train a YOLO11n object detection model on COCO128 using Databricks AI Runtime, with MLflow tracking and Model Serving deployment.
* [Computer vision on Databricks](/concepts/computer-vision-on-databricks.md) - Notebook examples for computer vision tasks on Databricks AI Runtime, including object detection and image classification.
* [Computer Vision on Databricks AI Runtime](/concepts/computer-vision-on-databricks-ai-runtime.md) - Capabilities for computer vision tasks such as object detection and image classification using GPU-accelerated AI Runtime notebooks.
* [Concurrency in LLM Serving Endpoints](/concepts/concurrency-in-llm-serving-endpoints.md) - The number of parallel requests sent to an LLM endpoint; controlling concurrency is central to balancing latency and throughput, with low concurrency for real-time uses and high concurrency for batch/offline uses.
* [Concurrent COPY INTO Invocations](/concepts/concurrent-copy-into-invocations.md) - COPY INTO supports concurrent runs on distinct file sets, with transaction conflicts on overlapping files. Single invocation with many files outperforms concurrent single-file runs.
* [Concurrent notebook execution in Databricks](/concepts/concurrent-notebook-execution-in-databricks.md) - Running multiple notebooks in parallel using Python or Scala threading/futures constructs alongside dbutils.notebook.run().
* [Concurrent Request Saturation](/concepts/concurrent-request-saturation.md) - The phenomenon where increasing parallel requests beyond an endpoint's provisioned capacity causes throughput to plateau while latency continues to rise due to queueing.
* [Condition Functions for ABAC](/concepts/condition-functions-for-abac.md) - Built-in functions such as has_tag and has_tag_value used in GRANT policy WHEN clauses to match governed tags on securable objects and determine policy applicability.
* [Conditional Column Masking](/concepts/conditional-column-masking.md) - A column masking technique where the mask applied to PII data depends on the value of another column on the same row (e.g., order_priority), allowing confidential rows to override user clearance levels.
* [Config Override and Dry-Run Patterns](/concepts/config-override-and-dry-run-patterns.md) - Common CLI patterns for overriding YAML configuration fields at submission time and validating configurations without actually submitting a job using --dry-run.
* [Configurable Forecasting Data Settings in AutoML](/concepts/configurable-forecasting-data-settings-in-automl.md) - The ability for users to adjust data preparation options (imputation, aggregation) during AutoML experiment setup via UI dropdowns or the Python API.
* [Configurable Judge Models](/concepts/configurable-judge-models.md) - MLflow built-in judges can be backed by different LLMs specified in LiteLLM-compatible format, defaulting to a Databricks-hosted model
* [Configuring a Lakeflow Job for Python Wheels](/concepts/configuring-a-lakeflow-job-for-python-wheels.md) - The step-by-step process of creating a Databricks job that uses a Python wheel task, including specifying package name, entry point, compute resources, and the wheel file itself.
* [Configuring Feature Table Permissions via UI](/concepts/configuring-feature-table-permissions-via-ui.md) - Users with CAN MANAGE permission can configure permissions on individual feature tables through a drop-down menu on the feature table page in the Databricks UI.
* [Configuring Judge LLMs in MLflow](/concepts/configuring-judge-llms-in-mlflow.md) - The ability to customize which LLM powers the built-in judges by specifying a model in '<provider>:/<model-name>' format, supporting LiteLLM-compatible providers including Databricks serving endpoints.
* [Configuring Permissions Across All Feature Tables](/concepts/configuring-permissions-across-all-feature-tables.md) - Workspace admins and Feature Store CAN MANAGE holders can set base permissions for all current and future feature tables from the Feature Store page
* [Configuring Permissions on an Individual Feature Table](/concepts/configuring-permissions-on-an-individual-feature-table.md) - How to set or override permissions for a specific feature table via its Permissions dropdown.
* [Configuring Permissions on Individual Feature Tables](/concepts/configuring-permissions-on-individual-feature-tables.md) - Procedure to edit permissions on a single feature table via the UI, requiring CAN MANAGE to modify
* [Conflict handling during metadata upgrade](/concepts/conflict-handling-during-metadata-upgrade.md) - Behavior when source and target table metadata differs: comments are skipped with a warning (or cause an error in older versions), other mismatches cause errors unless overwrite=True is passed.
* [Conflict resolution by data migration](/concepts/conflict-resolution-by-data-migration.md) - Resolution strategy for storage path conflicts where data must be physically moved because managed tables exist directly under the root bucket path that conflicts with the workspace default storage path.
* [Conflict resolution by path narrowing](/concepts/conflict-resolution-by-path-narrowing.md) - Resolution strategy for storage path conflicts where the external location path is updated to a more specific path (e.g., s3://bucket/region/workspace-id) without moving data, resolving the overlap while retaining data access.
* [Connecting to AI Runtime from Interactive Notebooks](/concepts/connecting-to-ai-runtime-from-interactive-notebooks.md) - Step-by-step process to attach a notebook to Serverless GPU compute with accelerator selection and environment configuration
* [Connecting to AI Runtime via Interactive Notebooks](/concepts/connecting-to-ai-runtime-via-interactive-notebooks.md) - Primary method to use AI Runtime by attaching a notebook to Serverless GPU compute and selecting an accelerator and base environment.
* [Connection Identifier Exchange](/concepts/connection-identifier-exchange.md) - The handshake process where a Databricks workspace generates a connection identifier, shares it with an SAP BDC admin who produces a connection link, which is then used to finalize the connection on Databricks.
* [Connection Identifier Exchange Protocol](/concepts/connection-identifier-exchange.md) - A two-phase handshake where a Databricks-generated connection identifier is provided to an SAP BDC admin, who returns a connection link to finalize the connection.
* [Connection Ownership and Deletion Prerequisites](/concepts/connection-ownership-and-deletion-prerequisites.md) - Connection ownership is restricted to individual users (not groups or service principals), and deleting a connection requires first unmounting all shared catalogs and being the connection owner.
* [Connection Ownership Constraints](/concepts/connection-ownership-constraints.md) - Rules governing SAP BDC connection ownership, including requirement for individual user owners and ownership transfer restrictions
* [Connection Timeout](/concepts/connection-timeout.md) - Network-level timeout where a client waits to establish a connection with the server, commonly seen with JDBC/SQL endpoints.
* [Connection Timeouts in Model Serving](/concepts/connection-timeouts-in-model-serving.md) - Timeouts related to establishing a network connection between a client and the model serving server, commonly seen with JDBC/SQL endpoints and socket connections.
* [Connection Timeouts in Model Serving Pipelines](/concepts/connection-timeouts-in-model-serving.md) - Timeouts related to establishing network connections (e.g., JDBC SocketTimeout) between model serving endpoints and downstream services like SQL endpoints or AI Search indices.
* [Consecutive and Baseline Drift](/concepts/consecutive-and-baseline-drift.md) - Two drift comparison methods: consecutive drift compares a window to the previous time window, while baseline drift compares against a user-provided baseline table.
* [Consecutive Drift](/concepts/consecutive-drift.md) - A drift comparison method that compares a data window to the immediately previous time window to detect changes over sequential time periods.
* [Consecutive vs Baseline Drift](/concepts/consecutive-vs-baseline-drift.md) - Two types of drift computed in Databricks profiling: consecutive drift compares a window to the previous time window, while baseline drift compares to a baseline distribution from a baseline table.
* [Consistent hashing (deterministic pseudonymization)](/concepts/consistent-hashing-deterministic-pseudonymization.md) - Replacing sensitive data with a deterministic hash (e.g., SHA2) via a DETERMINISTIC UDF so the same input always produces the same pseudonym across tables, with a version parameter to support key rotation.
* [Consistent hashing for deterministic pseudonymization](/concepts/consistent-hashing-deterministic-pseudonymization.md) - Replacing sensitive data with a deterministic hash that is consistent across tables using DETERMINISTIC UDFs, with a version parameter to support key rotation without breaking historical data.
* [Consistent Training and Serving Environments](/concepts/consistent-training-and-serving-environments.md) - A key benefit of express deployments where the model serving environment matches the training environment, avoiding divergence issues common in non-express deployments.
* [Contacting Databricks support to repair Delta tables](/concepts/contacting-databricks-support-to-repair-delta-tables.md) - Databricks recommends contacting their support team to repair tables affected by the DELTA_VERSIONS_NOT_CONTIGUOUS error.
* [Container Build Debugging for Model Serving](/concepts/container-build-debugging-for-model-serving.md) - Debugging container build failures and successes using event logs, including interpreting SERVED_ENTITY_CONTAINER_EVENT types and resolving build errors like OSError and GPU unavailability.
* [Container Build Event Logs](/concepts/container-build-event-logs.md) - Event logs available in the workspace UI's Events tab that track container build progress, with successful builds emitting a SERVED_ENTITY_CONTAINER_EVENT with a completion message.
* [Containerized Deployment of MLflow Traced Agents](/concepts/containerized-deployment-of-mlflow-traced-agents.md) - Patterns for passing MLflow tracing configuration environment variables to Docker and Kubernetes deployments so external GenAI agents can send traces to Databricks.
* [Content Block Types (Databricks APIs)](/concepts/content-block-types-databricks-apis.md) - The supported content types in messages across Databricks APIs: TextContent, ImageContent, ReasoningContent, DocumentContent, and FileContent.
* [Context Features](/concepts/context-features.md) - Features provided at inference time (not stored in feature tables) that serve as runtime inputs to feature functions for dynamic computation
* [Context Variable Extraction in LLM Judges](/concepts/context-variable-extraction-in-llm-judges.md) - The Guidelines judge automatically parses trace inputs and outputs into `request` and `response` context keys, handling OpenAI-format messages, ChatCompletion objects, and arbitrary dicts with specific extraction rules.
* [Context Variable Referencing in Guidelines](/concepts/context-variable-referencing-in-guidelines.md) - Guidelines can reference any key from the evaluation context dictionary (e.g., request, response, retrieved_documents, user_preferences) to create dynamic, data-aware pass/fail rules.
* [Context Variables in Guidelines](/concepts/context-variables-in-guidelines.md) - A mechanism allowing guidelines to reference any key from the evaluation context dictionary (e.g. request, response, retrieved_documents, user_preferences) directly in natural language criteria.
* [ContextPrecision Scorer](/concepts/contextprecision-scorer.md) - A RAGAS scorer that evaluates retrieval quality by measuring how much of the retrieved context is relevant to the query.
* [Contrast with Row Filter and Column Mask Policies](/concepts/contrast-with-row-filter-and-column-mask-policies.md) - GRANT policies differ from row filter/column mask policies: they determine whether a user can access an object at all (vs restricting content), and they use inline conditions (vs requiring a UDF).
* [Control Tag Pattern for Secure Defaults](/concepts/control-tag-pattern-for-secure-defaults.md) - A governance pattern using a schema-level control tag (e.g., review_status) to automatically lock down new tables by default until a data steward reviews and releases them.
* [ConversableAgent](/concepts/conversableagent.md) - The fundamental agent class in AG2 (AutoGen 0.2) used to define conversational agents with system messages, LLM configs, and termination logic.
* [Conversation Assessment Storage](/concepts/conversation-assessment-storage.md) - The design principle of storing multi-turn assessment results on the first trace of each session to ensure stability when new turns are added.
* [Conversation Assessment Storage on First Trace](/concepts/conversation-assessment-storage-on-first-trace.md) - Design pattern where multi-turn assessment results are stored on the first (chronologically earliest) trace in each session to ensure stability and efficient UI display.
* [Conversation Evaluation](/concepts/conversation-evaluation.md) - Assessing entire conversation sessions rather than individual turns to capture multi-interaction quality like user frustration, completeness, and coherence.
* [Conversation Evaluation (Multi-turn Evaluation)](/concepts/conversation-evaluation.md) - Assessment of entire conversation sessions rather than individual turns, capturing session-level quality like user frustration, conversation completeness, knowledge retention, and dialogue coherence.
* [Conversation Simulation](/concepts/conversationsimulator.md) - Generating synthetic multi-turn conversations programmatically for testing conversational AI agents using defined goals, personas, and context, without relying on manual test creation or production data.
* [Conversation Simulation (MLflow)](/concepts/conversationsimulator.md) - A technique in MLflow for programmatically generating synthetic multi-turn conversations to test conversational AI agents without manual creation or production data.
* [Conversation Simulation for Evaluation](/concepts/conversation-simulation-for-evaluation.md) - Approach that generates new conversations by simulating user interactions with defined test cases, goals, and personas to test agent versions systematically.
* [Conversation Stopping Conditions](/concepts/conversation-stopping-conditions.md) - Rules that determine when a simulated conversation ends: reaching the max_turns limit or detecting the user's goal has been accomplished.
* [Conversation Termination Conditions](/concepts/conversation-termination-conditions.md) - Rules that stop simulated conversations when max turns are reached or the simulator detects the user's goal has been achieved.
* [ConversationSimulator](/concepts/conversationsimulator.md) - MLflow API class that generates synthetic multi-turn conversations for testing conversational AI agents by defining test scenarios and simulating realistic user interactions.
* [ConversationSimulator API](/concepts/conversationsimulator.md) - The MLflow class (mlflow.genai.simulators.ConversationSimulator) that orchestrates multi-turn conversation simulation, accepting test cases and configuration parameters like max_turns, temperature, and user_model.
* [Conversion Caveats and File Visibility](/concepts/conversion-caveats-and-file-visibility.md) - Post-conversion, untracked Parquet files become invisible to Delta Lake and can be deleted by VACUUM; writes must go through Delta Lake; multiple external tables sharing the same directory face access issues.
* [CONVERT TO DELTA](/concepts/convert-to-delta.md) - A SQL command that transforms an existing directory of Parquet files into a Delta table in a single operation.
* [CONVERT TO DELTA Command](/concepts/convert-to-delta-command.md) - A SQL command that converts existing Apache Parquet or Iceberg tables to Delta tables in-place by creating a Delta Lake transaction log.
* [CONVERT TO DELTA SQL Command](/concepts/convert-to-delta-sql-command.md) - A one-time SQL command that converts Parquet and Apache Iceberg tables to Delta Lake tables on Databricks.
* [CONVERT TO DELTA Statistics and NO STATISTICS Option](/concepts/convert-to-delta-statistics-and-no-statistics-option.md) - The command automatically collects statistics on data files during conversion for query performance; the NO STATISTICS keyword bypasses this to finish faster.
* [CONVERT_TO_DELTA Duplicate Column Validation](/concepts/convert_to_delta-duplicate-column-validation.md) - The validation that occurs when converting an existing data source to Delta format, which checks for duplicate columns and raises DELTA_DUPLICATE_COLUMNS_FOUND if found.
* [Converting Iceberg tables to Delta Lake](/concepts/converting-iceberg-tables-to-delta-lake.md) - One-time conversion of Apache Iceberg tables to Delta Lake with specific version and feature limitations
* [Converting managed and external tables in Unity Catalog](/concepts/converting-managed-and-external-tables-in-unity-catalog.md) - Process for converting Hive metastore Parquet tables to Unity Catalog Delta Lake tables using CONVERT TO DELTA or CTAS
* [Converting Parquet directories to Delta Lake](/concepts/converting-parquet-directories-to-delta-lake.md) - Converting directories of Parquet data files stored in external locations into Delta Lake tables
* [COPY INTO (Databricks SQL)](/concepts/copy-into-delta-lake.md) - A SQL command for loading data from file locations into Delta tables, supporting retryable and idempotent ingestion.
* [COPY INTO (Databricks)](/concepts/copy-into-delta-lake.md) - A SQL command for idempotently loading data from file storage into Delta tables, skipping previously loaded files.
* [COPY INTO (Delta Lake)](/concepts/copy-into-delta-lake.md) - A SQL command in Databricks that loads data from file locations into Delta tables with idempotent, retryable semantics.
* [COPY INTO BY POSITION](/concepts/copy-into-by-position.md) - A syntax option for matching source columns to target Delta table columns by ordinal position rather than by name, only supported for headerless CSV files.
* [COPY INTO Column Mapping (BY POSITION)](/concepts/copy-into-column-mapping-by-position.md) - Syntax for matching source columns to target Delta table columns by ordinal position, supporting headerless CSV and explicit column name lists.
* [COPY INTO Command](/concepts/copy-into-command.md) - A SQL command in Databricks that loads data from file locations into Delta tables with retryable and idempotent semantics.
* [COPY INTO Concurrency Model](/concepts/copy-into-concurrency-model.md) - Concurrent COPY INTO invocations are supported against the same table as long as they operate on distinct input file sets
* [COPY INTO COPY_OPTIONS](/concepts/copy-into-copy_options.md) - Configuration options for COPY INTO including force (disable idempotency) and mergeSchema (evolve schema for incoming data).
* [COPY INTO File Selection (FILES and PATTERN)](/concepts/copy-into-file-selection-files-and-pattern.md) - Two mutually exclusive mechanisms for selecting which files to ingest: listing specific file names (FILES, up to 1000) or using a glob pattern (PATTERN).
* [COPY INTO FORCE and mergeSchema Options](/concepts/copy-into-force-and-mergeschema-options.md) - Copy options to disable idempotency (force) or enable automatic schema evolution (mergeSchema) during ingestion.
* [COPY INTO Source Access with Credentials](/concepts/copy-into-source-access-with-credentials.md) - Mechanisms for providing access to source file locations via named storage credentials, inline temporary credentials, or Unity Catalog external locations.
* [COPY INTO Source Credential Options](/concepts/copy-into-source-credential-options.md) - Mechanisms for providing access to source file locations including named credentials, inline temporary credentials, and external locations via Unity Catalog
* [COPY INTO Source Credentials](/concepts/copy-into-source-credentials.md) - Methods for providing access credentials to source storage locations in COPY INTO, including named credentials, inline temporary credentials, and external locations via Unity Catalog.
* [COPY INTO Source Credentials & Encryption](/concepts/copy-into-source-credentials-encryption.md) - Mechanisms for providing temporary or named credentials and encryption options to access source file locations during COPY INTO operations.
* [COPY INTO VALIDATE](/concepts/copy-into-validate.md) - A validation-only mode of COPY INTO that checks whether data can be parsed, schema matches, and constraints are met without writing to the table.
* [COPY INTO VALIDATE Mode](/concepts/copy-into-validate-mode.md) - A validation mode that checks parseability, schema compatibility, and constraints without writing data; supports row-limited previews.
* [COPY INTO with SELECT Expressions](/concepts/copy-into-with-select-expressions.md) - The ability to specify SELECT expressions including window operations and global aggregates when loading source data into a Delta table
* [copy_model_version API](/concepts/copy_model_version.md) - An MLflow API method used to copy a specific model version from a source registry (workspace) to a destination registry (Unity Catalog)
* [copy_model_version()](/concepts/copy_model_version.md) - An MLflow API function (≥3.4.0) that copies a model version from the Workspace Model Registry to a destination Unity Catalog model, creating the destination model if it doesn't exist.
* [Copying Experiments Between Databricks Workspaces](/concepts/copying-experiments-between-databricks-workspaces.md) - MLflow experiments can be migrated between workspaces using the community-driven MLflow Export-Import open source tool, enabling cross-workspace collaboration, cloning experiments, and backup of experiments and models.
* [Core Schema Notation (CSN)](/concepts/core-schema-notation-csn.md) - A schema description format used to describe Databricks Delta Shares to SAP BDC so that SAP BDC users can read and understand the shared data assets.
* [Core Schema Notation (CSN) for SAP BDC](/concepts/core-schema-notation-csn.md) - A schema notation used to describe data shares to SAP BDC so that SAP BDC users can read and interpret the shared data
* [Correctness Judge](/concepts/correctness-judge.md) - An MLflow built-in LLM judge that evaluates whether a GenAI application's response is factually correct by comparing it against ground truth information (expected_facts or expected_response).
* [Correctness LLM Judge](/concepts/correctness-judge.md) - A built-in LLM judge that evaluates whether a GenAI application's response is factually correct by comparing it against provided ground truth information (expected_facts or expected_response).
* [Correctness Scorer](/concepts/correctness-scorer.md) - An MLflow GenAI evaluation scorer that measures the correctness of model outputs against expected responses, used as a feedback signal during prompt optimization.
* [Correctness Scorer for Fact Coverage](/concepts/correctness-scorer-for-fact-coverage.md) - A built-in MLflow scorer that evaluates whether LLM outputs contain expected facts defined in the evaluation dataset, measuring factual completeness.
* [Correctness Scorer for Factual Accuracy](/concepts/correctness-scorer-for-factual-accuracy.md) - A built-in MLflow scorer that checks whether generated output covers expected facts defined in the evaluation dataset.
* [Correctness Scorer for GenAI Evaluation](/concepts/correctness-scorer-for-genai-evaluation.md) - MLflow scorer that compares GenAI app outputs against expert-provided expected responses to produce quantitative correctness metrics.
* [Correctness Scorer in MLflow GenAI](/concepts/correctness-scorer.md) - An evaluation scorer that assesses whether model outputs match expected responses based on provided ground-truth labels and facts
* [CORS Configuration for Managed Volumes](/concepts/cors-configuration-for-managed-volumes.md) - Cross-origin resource sharing (CORS) configuration required on the S3 bucket to enable Databricks to upload data to managed volumes in Unity Catalog.
* [CORS configuration for Unity Catalog managed volumes](/concepts/cors-configuration-for-managed-volumes.md) - Cross-origin resource sharing (CORS) settings that must be applied to the S3 bucket to enable Databricks to upload data to managed volumes in Unity Catalog.
* [Cost Optimization for Online Feature Stores](/concepts/cost-optimization-for-online-feature-stores.md) - Best practices for managing online store costs including reusing stores across projects, right-sizing capacity (starting at CU_2), and deleting unused stores that continuously incur costs.
* [Covariates (External Regressors) for Forecasting](/concepts/covariates-external-regressors-for-forecasting.md) - Additional variables outside the target time series that can improve forecasting model accuracy by providing external context.
* [Covariates (External Regressors) in Time Series Forecasting](/concepts/covariates-external-regressors-for-forecasting.md) - Additional variables outside the target time series that can improve forecasting model accuracy by providing exogenous information.
* [Covariates in Time Series Forecasting](/concepts/covariates-in-time-series-forecasting.md) - External regressors — additional variables outside the target time series that can improve forecasting model accuracy.
* [CPU Offloading](/concepts/cpu-offloading.md) - A memory optimization technique in DeepSpeed that offloads optimizer states or parameters from GPU memory to CPU memory to reduce GPU memory pressure
* [CPU Offloading in DeepSpeed](/concepts/cpu-offloading.md) - A feature of DeepSpeed ZeRO that allows offloading optimizer states, gradients, or parameters to CPU memory to reduce GPU memory usage during large model training.
* [CPU Resource Planning for Locust Tests](/concepts/cpu-resource-planning-for-locust-tests.md) - Locust performance scales at approximately 4000 requests per second per CPU core, and the framework auto-detects available cores via the --processes -1 flag to maximize throughput.
* [CPU-Constrained Load Test Orchestration](/concepts/cpu-constrained-load-test-orchestration.md) - Locust's throughput is limited by CPU resources at roughly 4000 requests per second per core; the test notebook uses the --processes -1 flag to auto-detect and utilize all available CPU cores on the driver, and recommends CPU-optimized instances with at least 32 cores for the test cluster.
* [CRC Checksum Corruption Detection](/concepts/crc-checksum-corruption-detection.md) - The ability of FSCK REPAIR TABLE to detect CRC (Cyclic Redundancy Check) checksum corruption in Delta Lake checkpoint files (.checkpoint.parquet).
* [CRC checksum corruption in Delta Lake](/concepts/crc-checksum-corruption-in-delta-lake.md) - Corruption of CRC checksum in Delta Lake checkpoint files detected by FSCK REPAIR TABLE
* [CREATE BLOOM FILTER INDEX](/concepts/create-bloom-filter-index.md) - The deprecated SQL command for creating Bloom filter indexes on Delta tables; no longer recommended for use.
* [CREATE BLOOM FILTER INDEX (Databricks SQL)](/concepts/create-bloom-filter-index.md) - The SQL command syntax and usage for creating Bloom filter indexes in Databricks, now deprecated and not recommended for new use.
* [CREATE BLOOM FILTER INDEX Statement](/concepts/create-bloom-filter-index.md) - The deprecated SQL statement used to create Bloom filter indexes on Delta tables in Databricks.
* [CREATE OR REPLACE CLONE](/concepts/create-or-replace-clone.md) - A variant of the CLONE statement that replaces an existing table if it exists or creates a new one if it does not (without CREATE OR, the target table must already exist).
* [CREATE OR REPLACE TABLE CLONE](/concepts/create-or-replace-table-clone.md) - A DDL syntax variant that creates a new cloned table or replaces an existing one at the target location.
* [CREATE OR REPLACE TABLE CLONE Syntax](/concepts/create-or-replace-table-clone-syntax.md) - A SQL syntax variant for cloning that either replaces an existing target table or creates a new one if it does not exist.
* [CREATE RECIPIENT Privilege](/concepts/create-recipient-privilege.md) - A Unity Catalog privilege required to create recipient objects in OpenSharing, granted at the metastore level.
* [CREATE RECIPIENT Privilege and Command](/concepts/create-recipient-privilege-and-command.md) - A privilege required to create recipients in Unity Catalog, and the SQL command used to create recipient objects for Delta Sharing
* [CREATE TABLE AS SELECT (CTAS) for Migration](/concepts/create-table-as-select-ctas-for-migration.md) - Alternative method to create Unity Catalog managed tables by querying Hive tables with SELECT, allowing column/row filtering during migration.
* [CREATE TABLE AS SELECT Workaround](/concepts/create-table-as-select-workaround.md) - The CREATE TABLE AS SELECT statement is not supported in Databricks Connect for Scala; the recommended alternative is spark.sql(...).write.saveAsTable(...).
* [CREATE TABLE CLONE for Migration](/concepts/create-table-clone-for-migration.md) - Method using DEEP CLONE to copy managed Delta tables from Hive metastore to Unity Catalog managed tables; table history and time travel are not preserved.
* [CREATE TABLE CLONE Syntax](/concepts/create-table-clone-syntax.md) - The SQL syntax for cloning tables in Databricks, including CREATE TABLE, CREATE OR REPLACE TABLE, IF NOT EXISTS, SHALLOW/DEEP CLONE, TBLPROPERTIES, and LOCATION clauses.
* [Create Training Set with Point-in-Time Features](/concepts/create-training-set-with-point-in-time-features.md) - Workflow for using declaratively defined features to compute point-in-time aggregated training datasets via create_training_set, ensuring temporal consistency between feature values and labels.
* [create_training_set for Point-in-Time Features](/concepts/create_training_set-for-point-in-time-features.md) - An API that computes point-in-time correct feature values for ML training by joining labeled data with feature definitions, ensuring no data leakage by aligning timestamps.
* [Creating and Managing Tables with Databricks Connect](/concepts/creating-and-managing-tables-with-databricks-connect.md) - The pattern of creating in-memory DataFrames locally, saving them as managed tables on a remote Databricks cluster, querying them, and cleaning up via Databricks Connect for Scala.
* [Creating and persisting DataFrames to Databricks tables](/concepts/creating-and-persisting-dataframes-to-databricks-tables.md) - The end-to-end workflow of creating in-memory Spark DataFrames, saving them as managed tables via saveAsTable(), querying them with Spark SQL, and cleaning up with DROP TABLE.
* [Creating and Running JAR Tasks in Databricks Jobs](/concepts/creating-and-running-jar-tasks-in-databricks-jobs.md) - How to configure a Databricks job to run a JAR as a serverless task, including main class, parameters, and environment setup
* [Creating and saving DataFrames as tables with Databricks Connect](/concepts/creating-and-saving-dataframes-as-tables-with-databricks-connect.md) - Pattern for creating in-memory DataFrames, saving them as persistent tables on a Databricks cluster, querying them, and cleaning up
* [Creating MLflow Experiments on Databricks](/concepts/creating-mlflow-experiments-on-databricks.md) - Experiments can be created via the Databricks workspace UI, the Experiments page (with presets for Foundation Model Fine-tuning, AutoML forecasting/classification/regression, or Custom), the MLflow API, or the Databricks Terraform provider.
* [Creator Identity for Model Serving](/concepts/creator-identity-for-model-serving.md) - The recorded identity (typically a service principal) that created an endpoint, used to access Unity Catalog resources; cannot be changed after creation and must remain a workspace member.
* [Creator Identity for Serving Endpoints](/concepts/creator-identity-for-serving-endpoints.md) - The recorded caller identity (typically a service principal) that owns a model serving endpoint, used for Unity Catalog access; it is immutable after creation and must remain a workspace member for the endpoint lifetime.
* [Credential File (Delta Sharing)](/concepts/credential-file-delta-sharing.md) - A file containing the bearer token that a recipient downloads once via an activation link, treated as a secret and used to authenticate and get read access to shared tables.
* [Credential file (OpenSharing)](/concepts/credential-file-opensharing.md) - A file downloaded via an activation link in the Databricks-to-Open sharing model, used for bearer token authentication; downloadable only once and should be stored securely.
* [Credential File Authentication](/concepts/credential-file-authentication.md) - A bearer-token-based authentication method for Databricks-to-Open sharing, where recipients download a credential file (one-time) from an activation URL to access shared data; tokens are valid for up to one year.
* [Credential File-Based Access Model](/concepts/credential-file-based-access-model.md) - A security model where data providers share a credential file containing endpoints and bearer tokens, which recipients use to gain read access to shared datasets.
* [Credential Injection for External Model Endpoints](/concepts/credential-injection-for-external-model-endpoints.md) - Using secrets-based environment variables in Databricks Model Serving to pass credentials to external services like OpenAI and LangChain models, enabling secure access from serving endpoints.
* [CrewAI Knowledge Source Integration](/concepts/crewai-knowledge-source-integration.md) - CrewAI agents can attach knowledge sources (e.g., StringKnowledgeSource) with metadata for contextual awareness
* [CrewAI Synchronous Execution Limitation](/concepts/crewai-synchronous-execution-limitation.md) - MLflow CrewAI integration only supports synchronous task execution; async tasks and kickoff are not traced
* [CrewAI Task and Agent Tracing](/concepts/crewai-task-and-agent-tracing.md) - MLflow captures nested traces for CrewAI tasks, agents, LLM calls, memory ops, and exceptions
* [Cross-Catalog Joins Between Hive Metastore and Unity Catalog](/concepts/cross-catalog-joins-between-hive-metastore-and-unity-catalog.md) - Using three-level namespace notation, users can join tables across the legacy Hive metastore and Unity Catalog, though such joins only work on the workspace where the Hive metastore data resides.
* [Cross-Framework Trace Consolidation](/concepts/cross-framework-trace-consolidation.md) - The practice of routing traces from multiple instrumentation frameworks (Langfuse, OpenTelemetry SDKs, etc.) to a single Databricks MLflow backend for unified querying, comparison, and governance.
* [Cross-Geo Data Processing](/concepts/cross-geo-data-processing.md) - Databricks may process Foundation Model API request data outside the originating region/cloud provider; workspaces in non-EU/US regions must enable cross-Geo data processing to access certain models.
* [Cross-Geo Data Processing (Databricks)](/concepts/cross-geo-data-processing.md) - A mechanism allowing Foundation Model API requests to be processed outside the workspace's home region and cloud provider, required for workspaces in non-US/non-EU Model Serving regions.
* [Cross-Geo Data Processing for Foundation Model APIs](/concepts/cross-geo-data-processing.md) - Databricks may process Foundation Model API request data outside the region and cloud provider where it originated; workspaces outside US/EU regions must enable cross-Geo data processing to access certain models.
* [Cross-Geography Routing for Global Endpoints](/concepts/cross-geography-routing-for-global-endpoints.md) - Some hosted models require enabling cross-geography routing to be accessible via global endpoints, as noted for OpenAI GPT-5.1 Codex models and Google Gemini 3 Pro Preview.
* [Cross-model hyperparameter comparison](/concepts/cross-model-hyperparameter-comparison.md) - Technique of tuning hyperparameters for multiple model types (e.g., different scikit-learn models) in parallel to select the best overall model
* [Cross-model hyperparameter optimization](/concepts/cross-model-hyperparameter-comparison.md) - The technique of tuning hyperparameters across multiple distinct model types (e.g., different scikit-learn classifiers) and comparing their best configurations to select an overall best model.
* [Cross-Model Hyperparameter Tuning](/concepts/cross-model-hyperparameter-tuning.md) - A workflow that tunes hyperparameters for multiple distinct model types simultaneously and selects the best overall model, rather than tuning a single model type.
* [Cross-Workspace Access for Feature Tables](/concepts/cross-workspace-access-for-feature-tables.md) - Feature tables, functions, and models are automatically available across any workspace that has access to the Unity Catalog.
* [Cross-workspace Feature Access](/concepts/cross-workspace-feature-access.md) - Feature tables, functions, and models are automatically available across any workspace that has access to the Unity Catalog catalog
* [Cross-Workspace Feature Sharing](/concepts/cross-workspace-feature-sharing.md) - The ability to discover and share feature tables across different Databricks workspaces through Unity Catalog integration.
* [Cross-workspace Feature Store Access with Tokens](/concepts/cross-workspace-feature-store-access-with-tokens.md) - Mechanism for accessing a centralized feature store from a local workspace using personal access tokens stored in the local workspace's secret manager, with each API request including the token.
* [Cross-Workspace Model Performance](/concepts/cross-workspace-model-performance.md) - Capability to view model performance metrics across all MLflow experiments and workspaces on a single page within the model registry UI.
* [CrossValidator](/concepts/crossvalidator.md) - A Spark MLlib model tuning tool that, when used with MLlib automated MLflow tracking, automatically logs hyperparameter settings and evaluation metrics to MLflow as nested runs.
* [CrossValidator and TrainValidationSplit](/concepts/crossvalidator.md) - Spark MLlib model tuning utilities whose hyperparameter search results are automatically logged as nested MLflow runs when automated tracking is enabled.
* [CrossValidator and TrainValidationSplit MLflow Integration](/concepts/crossvalidator.md) - The two Spark MLlib model tuning estimators that, when used with MLlib automated MLflow tracking, automatically log tuning results as nested MLflow runs without requiring explicit API calls.
* [CTAS for Migrating Hive Tables to Unity Catalog Managed Delta Tables](/concepts/ctas-for-migrating-hive-tables-to-unity-catalog-managed-delta-tables.md) - Using CREATE TABLE AS SELECT (CTAS) to convert legacy Hive metastore managed Parquet tables to managed Unity Catalog Delta Lake tables.
* [CUDA Error Troubleshooting for Training](/concepts/cuda-error-troubleshooting-for-training.md) - Common CUDA errors during model training (out-of-memory, kernel errors) and recommended solutions such as reducing batch size, lower precision, and GPU memory cleanup.
* [CUDA Out of Memory (OOM) Mitigation for Transformers Training](/concepts/cuda-out-of-memory-oom-mitigation-for-transformers-training.md) - A collection of strategies to resolve CUDA out-of-memory errors during single-GPU training, including reducing batch size, using FP16 precision, gradient accumulation steps, 8-bit Adam optimizer, and GPU memory cleanup.
* [CUDA Out of Memory Troubleshooting for Transformers Training](/concepts/cuda-out-of-memory-troubleshooting-for-transformers-training.md) - Common CUDA out-of-memory errors during Hugging Face training and resolution strategies
* [CUDA Peer Access Error on AWS G5 GPUs](/concepts/cuda-peer-access-error-on-aws-g5-gpus.md) - A specific distributed training error (peer access not supported) encountered on AWS G5 GPU instances, resolved by disabling NCCL P2P
* [CURRENT_METASTORE](/concepts/current_metastore.md) - A Databricks default SQL function that returns the sharing identifier of the attached Unity Catalog metastore
* [CURRENT_METASTORE SQL Function](/concepts/current_metastore.md) - A Databricks SQL function that returns the sharing identifier of the current Unity Catalog metastore, used by recipients to discover their metastore identity for sharing.
* [Custom Agents Deployment for GenAI on Databricks](/concepts/custom-agents-deployment-for-genai-on-databricks.md) - The recommended deployment method for GenAI applications on Databricks that provides automatic MLflow Tracing with minimal configuration by wrapping agent code with MLflow ResponsesAgent and deploying via agents.deploy().
* [Custom Alert Notification Templates](/concepts/custom-alert-notification-templates.md) - The ability to customize email notification templates for anomaly detection alerts in Databricks SQL using HTML, allowing detailed table-level information to be included in alert emails.
* [Custom Alignment Optimizer](/concepts/custom-alignment-optimizer.md) - An extensible framework allowing users to implement specialized alignment strategies by extending the AlignmentOptimizer base class with custom optimization logic.
* [Custom Classifier (Databricks Unity Catalog)](/concepts/custom-classifier-databricks.md) - A custom classifier extends Databricks' built-in data classification system to detect organization-specific sensitive data (e.g., employee IDs, product codes) by associating a governed tag with example columns.
* [Custom Classifier (Databricks)](/concepts/custom-classifier-databricks.md) - User-defined classifier that extends Databricks Data Classification to detect organization-specific sensitive data using governed tags and example columns.
* [Custom Classifier Lifecycle and Limits](/concepts/custom-classifier-lifecycle-and-limits.md) - Rules governing custom classifier creation, editing, deletion, and enforcement — including the 50-classifier metastore limit, 1-10 example column requirement, immutability of governed tag after creation, and metastore-wide scope.
* [Custom Classifier Lifecycle Management](/concepts/custom-classifier-lifecycle-management.md) - The process of creating, editing, deleting, and troubleshooting custom classifiers, including handling suspended classifiers and understanding detection timing.
* [Custom Classifier Limitations](/concepts/custom-classifier-limitations.md) - Technical constraints include a maximum of 50 classifiers per metastore, 1-10 example columns each, no per-catalog scoping, immutable tag association after creation, and application only to subsequent scans.
* [Custom Classifier Limitations (Databricks)](/concepts/custom-classifier-limitations-databricks.md) - Constraints on custom classifiers including a 50-classifier-per-metastore cap, 1-10 example column requirement, immutable tag assignment, and metastore-wide application scope.
* [Custom classifier limitations and scoping](/concepts/custom-classifier-limitations-and-scoping.md) - Operational constraints including 50-classifier metastore limit, 1-10 example column requirement, global metastore-wide application, and tag immutability after creation.
* [Custom Classifier Requirements and Permissions](/concepts/custom-classifier-requirements-and-permissions.md) - To use custom classifiers, users must be metastore admins, have ASSIGN privileges on the governed tag, SELECT on example columns, and have Data Classification enabled with serverless compute available.
* [Custom Classifier Suspension](/concepts/custom-classifier-suspension.md) - Automatic suspension mechanism that halts detection for a custom classifier when rule generation or validation fails due to inaccessible columns, unrepresentative data, or invalid governed tags.
* [Custom Classifier Suspension and Troubleshooting](/concepts/custom-classifier-suspension-and-troubleshooting.md) - When rule generation or validation fails, classifiers are suspended with no new detections; common causes include deleted tables, unrepresentative examples, or invalid tags.
* [Custom Classifiers](/concepts/custom-classifiers.md) - User-defined classifiers that extend Databricks' built-in data classification to detect organization-specific sensitive data types like internal employee IDs, proprietary product codes, and partner account numbers.
* [Custom Classifiers for Data Classification](/concepts/custom-classifiers.md) - User-defined classifiers that extend Databricks' built-in data classification system to detect organization-specific sensitive data types such as internal employee IDs, proprietary product codes, or partner account numbers.
* [Custom classifiers for Databricks Data Classification](/concepts/custom-classifiers.md) - User-defined classifiers that extend built-in data classification to detect organization-specific sensitive data by associating governed tags with example columns.
* [Custom Code-Based Scorers (MLflow GenAI)](/concepts/code-based-scorers-in-mlflow.md) - User-defined Python functions annotated with @scorer that compute flexible evaluation metrics for GenAI applications in MLflow.
* [Custom CPU Model Serving as Alternative Deployment](/concepts/custom-cpu-model-serving-as-alternative-deployment.md) - Alternative deployment method for GenAI agents on Databricks when Custom Agents cannot be used, requiring manual environment variable configuration (ENABLE_MLFLOW_TRACING, MLFLOW_EXPERIMENT_ID) and credential setup.
* [Custom dbt Profiles in Databricks Jobs](/concepts/custom-dbt-profiles-in-databricks-jobs.md) - Using a custom profiles.yml file to configure dbt task connections to SQL warehouses or all-purpose compute, with dynamic credential injection via environment variables.
* [Custom Dependency Management in Model Serving Containers](/concepts/custom-dependency-management-in-model-serving-containers.md) - Ensuring correct specification of model dependencies including custom .whl artifacts, GPU libraries (CUDA, cuDNN), and flash-attn wheel versions to avoid build and runtime errors like ModuleNotFoundError.
* [Custom Email Notification Templates for Anomaly Alerts](/concepts/custom-email-notification-templates-for-anomaly-alerts.md) - Feature allowing customization of email notification templates for Databricks SQL alerts, using Handlebars-style template syntax with query result row iteration.
* [Custom Email Templates for Databricks Alerts](/concepts/custom-email-templates-for-databricks-alerts.md) - The ability to customize email notification templates for anomaly detection alerts using HTML and query result placeholders.
* [Custom Framework Instrumentation](/concepts/custom-framework-instrumentation.md) - The capability to add tracing to proprietary or internal frameworks, custom LLM wrappers, and new libraries that lack official MLflow integration.
* [Custom HTTP Headers for Proxy in Databricks Connect](/concepts/custom-http-headers-in-databricks-connect.md) - Using the `header()` method on DatabricksSession.builder to add custom HTTP headers for proxy services that sit between the client and the Databricks cluster.
* [Custom HTTP Headers in Databricks Connect](/concepts/custom-http-headers-in-databricks-connect.md) - Using the `header()` method on DatabricksSession.builder to add custom HTTP headers to gRPC requests, useful when a proxy is placed between the client and the Databricks cluster.
* [Custom HTTP Headers in Databricks Connect Proxy Configuration](/concepts/custom-http-headers-in-databricks-connect.md) - Using the header() method to add custom HTTP headers for proxy servers when using Databricks Connect with gRPC over HTTP/2.
* [Custom Hyperparameters (HYPERPARAMETERS_PATH)](/concepts/custom-hyperparameters-hyperparameters_path.md) - Passing structured YAML configuration to training scripts via the `parameters` field in the workload YAML, accessible at runtime through the `HYPERPARAMETERS_PATH` environment variable.
* [Custom Judge Creation with make_judge](/concepts/custom-judge-creation-with-make_judge.md) - API for building custom multi-turn LLM judges using make_judge, with access to the full conversation history via the {{ conversation }} template variable.
* [Custom Judge Metrics with make_judge](/concepts/custom-judge-metrics-with-make_judge.md) - Creating domain-specific evaluation criteria using the make_judge API to assess LLM outputs against custom requirements like sentence count compliance.
* [Custom Judge Model Selection](/concepts/custom-judge-model-selection.md) - The ability to override MLflow's default judge LLM with any LiteLLM-compatible model, specified using the format '<provider>:/<model-name>', enabling tailored evaluation.
* [Custom Judge Scorers](/concepts/custom-judge-scorers.md) - Custom evaluation metrics created using MLflow's make_judge API with custom prompt instructions to assess specific criteria like sentence count compliance.
* [Custom Judge Scorers with make_judge](/concepts/custom-judge-scorers-with-make_judge.md) - Creating bespoke evaluation metrics by defining a judge LLM with custom instructions and feedback types via mlflow.genai.make_judge.
* [Custom Judges](/concepts/custom-judges.md) - Domain-specific evaluation judges created with make_judge() that define specialized evaluation logic and can be aligned to human preferences.
* [Custom Judges (LLM-based Scorers)](/concepts/custom-judges-llm-based-scorers.md) - LLM-based scorers that evaluate GenAI agent outputs for specific quality criteria like issue resolution, expected behaviors, and tool usage
* [Custom Judges (make_judge)](/concepts/custom-judges-make_judge.md) - LLM-based scorers created with `make_judge()` that evaluate GenAI agent outputs against user-defined quality criteria, returning `mlflow.entities.Feedback` objects.
* [Custom Labeling Schema Creation](/concepts/custom-labeling-schema-creation.md) - The process of defining custom labeling schemas via the MLflow UI or Python API, specifying name, type, title, input type, and optional instructions for tailored human feedback collection.
* [Custom Libraries for Databricks Model Serving](/concepts/custom-libraries-for-databricks-model-serving.md) - Pattern for including custom Python libraries or libraries from private mirrors when deploying ML models to Databricks Model Serving endpoints
* [Custom LLM Entrypoint and MLflow Logging](/concepts/custom-llm-entrypoint-and-mlflow-logging.md) - The technique of logging a custom LLM with a custom entrypoint in MLflow, specifying the task as llm/v1/chat and configuring the vLLM server to listen on port 8080 for Model Serving compatibility.
* [Custom LLM Judge](/concepts/custom-llm-judge.md) - User-defined evaluators created when built-in LLM judges do not fit a specific use case, offering more control over evaluation criteria and behavior.
* [Custom LLM Judges](/concepts/custom-llm-judges.md) - User-defined judges created with custom prompts and instructions for specialized evaluation tasks, with support for judge alignment to match human standards.
* [Custom LLM Judges (make_judge)](/concepts/custom-llm-judges-make_judge.md) - A function in MLflow that lets you define custom evaluation criteria for GenAI applications using natural language prompts, extending beyond built-in judge metrics.
* [Custom LLM Serving on Databricks](/concepts/custom-llm-serving-on-databricks.md) - A workflow for deploying custom large language models on Databricks Model Serving using a vLLM engine, supporting fine-tuned models, PEFT variants, multimodal models, and foundation models not available in FMAPI.
* [Custom LLM Serving Telemetry and Monitoring](/concepts/custom-llm-serving-telemetry-and-monitoring.md) - Observability infrastructure for custom LLM serving endpoints, including live logs, persisted logs and metrics to Unity Catalog Delta tables, and automatic scraping of vLLM Prometheus metrics.
* [Custom LLM-as-a-Judge in scorers](/concepts/custom-llm-as-a-judge-in-scorers.md) - Integrating a custom or externally hosted LLM within a scorer to evaluate response quality, with full control over prompts, model selection, and feedback generation.
* [Custom LLM-as-a-Judge Integration](/concepts/custom-llm-as-a-judge-integration.md) - Using an external or custom LLM within a scorer by making API calls and parsing structured JSON output for evaluation feedback.
* [Custom LLM-based Judges (make_judge)](/concepts/custom-judges-make_judge.md) - Factory function in MLflow for creating LLM-based scorers that evaluate GenAI agents against custom quality criteria.
* [Custom Metrics and Slicing Expressions](/concepts/custom-metrics-and-slicing-expressions.md) - Advanced profiling options allowing users to define custom SQL-based metrics and slice data into subsets for granular analysis alongside default metrics.
* [Custom Metrics for Data Profiling](/concepts/custom-metrics-for-data-profiling.md) - User-defined SQL-based metrics that augment built-in profiling metrics, with configurable types (Aggregate, Derived, Drift), input columns, output types, and SQL definitions.
* [Custom Metrics in Data Profiling](/concepts/custom-metrics-in-data-profiling.md) - A feature allowing users to define custom metrics beyond the built-in summary statistics when profiling tables in Databricks.
* [Custom Metrics in Databricks Data Profiling](/concepts/custom-metrics-in-databricks-data-profiling.md) - User-defined SQL-based metrics (Aggregate, Derived, or Drift) that appear alongside built-in metrics in profiling output tables for specialized data quality analysis.
* [Custom MLflow Logging via MLFLOW_RUN_ID](/concepts/custom-mlflow-logging-via-mlflow_run_id.md) - The platform exposes the MLflow run ID via the `MLFLOW_RUN_ID` environment variable, allowing users to log custom parameters, metrics, and artifacts using the MLflow tracking API.
* [Custom MLflow PyFunc Model](/concepts/custom-mlflow-pythonmodel.md) - A pattern for packaging arbitrary Python code as an MLflow model using mlflow.pyfunc.PythonModel, requiring implementations of load_context and predict methods.
* [Custom MLflow PythonModel](/concepts/custom-mlflow-pythonmodel.md) - Pattern for packaging arbitrary Python code as an MLflow model by implementing load_context and predict methods in a class that extends mlflow.pyfunc.PythonModel
* [Custom Model Instrumentation Patterns](/concepts/custom-model-instrumentation-patterns.md) - Best practices for adding logging and OpenTelemetry instrumentation to MLflow pyfunc models, including Python logging capture, logging level configuration, and model code organization.
* [Custom Model Serving Endpoint (Databricks)](/concepts/custom-model-serving-endpoint-databricks.md) - A serving endpoint that hosts custom ML models for real-time inference on Databricks, created via UI, REST API, MLflow Deployments SDK, or Workspace Client.
* [Custom Model Serving Endpoint Support](/concepts/custom-model-serving-endpoint-support.md) - Genie Code for observability currently only supports custom model serving endpoints, not all endpoint types.
* [Custom Models (Databricks Model Serving)](/concepts/custom-models-databricks.md) - Any Python ML model or custom code deployed as a production-grade API on Databricks using CPU or GPU compute resources, logged in MLflow format and registered in Unity Catalog or workspace registry.
* [Custom Models (Databricks)](/concepts/custom-models-databricks.md) - Traditional ML models or customized Python models packaged in MLflow format, including scikit-learn, XGBoost, PyTorch, and Hugging Face transformers, registered in Unity Catalog or workspace model registry
* [Custom Models on Databricks](/concepts/custom-models-databricks.md) - Python models packaged in MLflow format (scikit-learn, PyTorch, Hugging Face, etc.) registered in Unity Catalog or Workspace Model Registry for serving
* [Custom models on Databricks Model Serving](/concepts/custom-models-databricks.md) - Any Python ML model or custom code deployable as a production-grade API on Databricks using CPU or GPU compute, logged in MLflow format and registered in Unity Catalog or workspace registry.
* [Custom Models on Model Serving](/concepts/custom-models-on-model-serving.md) - Python MLflow-packaged models (scikit-learn, PyTorch, Hugging Face, etc.) registered in Unity Catalog or Workspace Model Registry that can be deployed on Databricks for real-time and batch inference, including agent serving.
* [Custom Multi-turn Judges with make_judge](/concepts/custom-judge-creation-with-make_judge.md) - The ability to create custom multi-turn evaluation judges using MLflow's make_judge API, leveraging the {{ conversation }} template variable to inject full conversation history.
* [Custom OTel Span Attribute Visibility Limitations in MLflow](/concepts/custom-otel-span-attribute-visibility-limitations-in-mlflow.md) - Custom OTel span attributes set via span.set_attribute() outside recognized mappings are not surfaced as MLflow trace tags, though preserved on the underlying span.
* [Custom Python Code Dependencies with code_paths](/concepts/custom-python-code-dependencies-with-code_paths.md) - Including custom .py files or Python wheel files as model artifacts using the code_paths parameter so they are added to the Python path when the model is loaded.
* [Custom request and response previews](/concepts/custom-request-and-response-previews.md) - Ability to customize the Request and Response column previews in the MLflow Tracing UI using mlflow.update_current_trace() with request_preview and response_preview parameters.
* [Custom Review App Template](/concepts/custom-review-app-template.md) - An open-source, customizable frontend template for the MLflow Review App that allows teams to build tailored trace visualization, labeling interfaces, and workflows while using the same MLflow backend APIs.
* [Custom Review App UI](/concepts/custom-review-app-ui.md) - An open-source deployable template for customizing the Review App frontend with specialized trace visualizations, tailored labeling interfaces, and domain-specific workflows
* [Custom Scorer Definition in MLflow](/concepts/custom-scorer-definition-in-mlflow.md) - Pattern for defining user-provided evaluation functions using the @scorer decorator with aggregation metrics (mean, min, max) for GenAI trace evaluation.
* [Custom Scorer Definition with @scorer Decorator](/concepts/custom-scorer-definition-with-scorer-decorator.md) - The pattern of defining custom evaluation functions in MLflow GenAI using the @scorer decorator, including aggregation configuration and manual registration/starting lifecycle.
* [Custom Scorer Serialization Constraints](/concepts/custom-scorer-serialization-constraints.md) - Constraints for custom @scorer functions in production monitoring: must be defined in notebooks, self-contained with inline imports, no class-based scorers, and no type hints requiring imports.
* [Custom Scorer with Aggregations](/concepts/custom-scorer-with-aggregations.md) - Pattern to define user-defined evaluation scorers using the @scorer decorator with aggregate statistics like mean, min, and max.
* [Custom Scorers (MLflow GenAI)](/concepts/custom-scorers-mlflow-genai.md) - User-defined scoring functions created with the @scorer decorator that compute metrics on model outputs (e.g., response length) and can be registered and started with sampling configs.
* [Custom scorers for LLM evaluation](/concepts/custom-scorers-for-llm-evaluation.md) - User-defined evaluation functions that go beyond built-in scorers to assess LLM outputs on criteria specific to a given use case, such as markdown formatting quality.
* [Custom Scorers for Prompt Evaluation](/concepts/custom-scorers-for-prompt-evaluation.md) - MLflow's support for custom evaluation scorers (e.g., Correctness, Safety) and aggregation functions to guide prompt optimization based on specific quality metrics.
* [Custom Spark Batch Migration](/concepts/custom-spark-batch-migration.md) - Using custom Apache Spark batch logic to migrate Parquet data to Delta Lake, offering flexibility but requiring manual tracking and handling of incremental updates.
* [Custom SSL/TLS Certificates for Databricks Connect](/concepts/ssltls-certificates-for-databricks-connect.md) - Configuring custom SSL/TLS certificates in Databricks Connect via the GRPC_DEFAULT_SSL_ROOTS_FILE_PATH environment variable
* [Custom Tools in Foundation Model APIs](/concepts/custom-tools-in-foundation-model-apis.md) - A tool type in Databricks Foundation Model APIs that allows models to return arbitrary string output (e.g., code, patches) instead of JSON-formatted function arguments, supporting grammar-based output formats for GPT-5 series models.
* [Custom Tools in Responses API](/concepts/custom-tools-in-responses-api.md) - GPT-5 series model feature allowing models to return arbitrary plain text output instead of JSON-formatted function arguments, with optional grammar-based format constraints.
* [Custom Trace ID Generation](/concepts/custom-trace-id-generation.md) - Technique for generating business-specific trace IDs by integrating MLflow tracing with existing observability systems using custom naming schemes.
* [Custom Trace UI Previews](/concepts/custom-trace-ui-previews.md) - The ability to set custom request_preview and response_preview strings via mlflow.update_current_trace() to control how traces are summarized in the MLflow UI's trace list view.
* [Custom Transformers in MLlib](/concepts/custom-transformers-in-mllib.md) - The ability to create custom pipeline transformers in Apache Spark MLlib for domain-specific feature transformations.
* [Custom vs Built-in Scorers](/concepts/custom-vs-built-in-scorers.md) - The distinction between predefined LLM judges provided by Databricks/MLflow and custom alternatives (custom LLM judges or Python code-based scorers) for more controlled evaluation.
* [Customer Service Chatbot Evaluation Patterns](/concepts/customer-service-chatbot-evaluation-patterns.md) - Real-world patterns for evaluating customer service chatbots using guidelines judges, covering tone, policy compliance, urgent situations, cancellations, and billing disputes.
* [Customer-Managed Key (CMK) Encryption for Online Feature Stores](/concepts/customer-managed-keys-for-online-feature-stores.md) - Encryption at rest for online feature stores using customer-managed keys, automatically applied when the workspace has CMK configured and the backing Lakebase project was created after CMK support became available.
* [Customer-Managed Key Encryption for Online Feature Stores](/concepts/customer-managed-keys-for-online-feature-stores.md) - Online feature stores support encryption at rest with customer-managed keys (CMK) via underlying Lakebase Autoscaling support, applying automatically when workspace CMK is configured and the store was created after CMK became available in the region.
* [Customer-Managed Key Encryption for Online Stores](/concepts/customer-managed-keys-for-online-feature-stores.md) - Encryption at rest for online feature stores using customer-managed keys, automatically applied when the workspace has CMK configured and the backing Lakebase project was created after CMK support.
* [Customer-Managed Keys (CMK) for Online Feature Stores](/concepts/customer-managed-keys-for-online-feature-stores.md) - Encryption at rest for online feature store data using a workspace-level customer-managed key, automatically applying when the workspace, online store, and region all support CMK.
* [Customer-Managed Keys for Online Feature Stores](/concepts/customer-managed-keys-for-online-feature-stores.md) - Encryption at rest support using customer-managed keys (CMK) via underlying Lakebase Autoscaling, requiring specific workspace and regional prerequisites.
* [Customizable Judge Models](/concepts/customizable-judge-models.md) - The ability to change which LLM powers a built-in judge by specifying a model in '<provider>:/<model-name>' format, compatible with LiteLLM providers including Databricks serving endpoints.
* [Customizable Judge Models in MLflow](/concepts/customizable-judge-models.md) - The ability to select which LLM powers a built-in MLflow judge by specifying a model in `<provider>:/<model-name>` format, with support for LiteLLM-compatible providers including Databricks serving endpoints.
* [Customizable Review App Template](/concepts/custom-review-app-template.md) - An open-source, deployable template that provides full frontend control over trace visualization, labeling interfaces, and workflows while using the same MLflow backend APIs and data model.
* [Customizing Autologging with Exclusive Mode](/concepts/customizing-autologging-with-exclusive-mode.md) - Technique to track additional metrics and parameters alongside autologged content by calling mlflow.autolog(exclusive=False) combined with explicit MLflow tracking calls
* [Customizing Databricks Data Profiling Dashboards](/concepts/customizing-databricks-data-profiling-dashboards.md) - The data profiling dashboard is fully customizable and shareable — users can edit date ranges, data slices, models, modify charts, add new charts, and adjust dashboard-level and chart-level parameters.
* [Customizing LLM Judge Models](/concepts/customizing-llm-judge-models.md) - The ability to swap the underlying LLM powering a built-in judge by specifying a different model via the 'model' argument using LiteLLM-compatible provider format, enabling use of custom or proprietary models for evaluation.
* [Customizing MLflow Judge Models](/concepts/customizing-mlflow-judge-models.md) - The ability to replace the default Databricks-hosted LLM powering MLflow's built-in judges with any LiteLLM-compatible model, specified in the format <provider>:/<model-name>.
* [Customizing the Judge LLM](/concepts/customizing-the-judge-llm.md) - The ability to override the default Databricks-hosted LLM that powers built-in MLflow judges by specifying a different model via the 'model' argument in a provider:/model-name format.
* [Customizing Trace Request/Response Previews](/concepts/customizing-trace-requestresponse-previews.md) - Using mlflow.update_current_trace() with request_preview and response_preview parameters to control how traces appear in the MLflow UI's trace list view.
* [Cut Cross Entropy](/concepts/cut-cross-entropy.md) - A memory-efficient loss computation technique for large language models that reduces GPU memory usage during training by avoiding materialization of the full logit matrix
* [Cut Cross-Entropy Loss](/concepts/cut-cross-entropy-loss.md) - A memory-efficient loss computation technique for large language models, reducing GPU memory usage during training
* [Dashboard access methods](/concepts/dashboard-access-methods.md) - Two ways to access the data profiling dashboard: via the Catalog Explorer UI from a table's Quality tab, or via the Dashboards sidebar listing.
* [Dashboard Customization and Parameters](/concepts/dashboard-customization-and-parameters.md) - User-editable parameters for both the entire dashboard and individual charts, allowing customization of date range, data slices, models, and chart modifications.
* [Dashboard customization and sharing](/concepts/dashboard-customization-and-sharing.md) - The ability to edit, add new charts, modify parameters, and share the data profiling dashboard like any other Databricks dashboard.
* [Dashboard Customization with Parameters (Databricks)](/concepts/dashboard-customization-and-parameters.md) - User-editable parameters at both the dashboard and individual chart level in Databricks profiling dashboards, allowing customization of date range, data slices, models, and chart selection.
* [Dashboard for Single Run Details](/concepts/dashboard-for-single-run-details.md) - A pre-built dashboard pattern that replicates the MLflow run details page, allowing filtering by experiment ID, run ID, and metric name to display tags, parameters, and metric graphs.
* [Dashboard Import and Export in Databricks](/concepts/dashboard-import-and-export-in-databricks.md) - Ability to export dashboards as JSON files and import them into a Databricks workspace for reuse and customization
* [Dashboard Refresh Mechanism](/concepts/profile-refresh-mechanism.md) - A two-step process where profile calculations must be refreshed to update metric tables, then the dashboard must be separately refreshed to reflect the new data.
* [Dashboard Refresh vs Profile Refresh](/concepts/dashboard-refresh-vs-profile-refresh.md) - A key architectural distinction in Databricks data profiling where the profile recalculation and the dashboard query are separate operations that must each be triggered independently.
* [Dashboard Widget Selectors for Data Filtering](/concepts/dashboard-widget-selectors-for-profile-data.md) - Widgets at the top of the profiling dashboard that control which data is displayed, with different selectors appearing depending on the profile type (Snapshot, Timeseries, or InferenceLog).
* [Dashboard Widget Selectors for Profile Data](/concepts/dashboard-widget-selectors-for-profile-data.md) - Interactive widgets at the top of the dashboard that control which data is displayed, with different selectors for Snapshot, Timeseries, and InferenceLog analysis types.
* [Data Certification](/concepts/data-certification.md) - The practice of labeling data assets (catalogs, schemas, tables, etc.) as meeting internal standards for accuracy, completeness, and trust, indicated by a checkmark icon in the Databricks workspace.
* [Data Certification and Deprecation](/concepts/data-certification-and-deprecation.md) - A system in Unity Catalog that allows administrators to flag data assets as certified (trusted, meeting quality standards) or deprecated (outdated, unsuitable for new use), with signals appearing in the workspace and influencing search results.
* [Data Certification and Deprecation Signals](/concepts/data-certification-and-deprecation.md) - System tags that mark data assets as certified (trusted, meeting quality standards) or deprecated (outdated, unsuitable for new use) to provide inline trust signals.
* [Data Certification in Unity Catalog](/concepts/data-certification-in-unity-catalog.md) - The practice of labeling Unity Catalog objects as 'certified' to indicate they meet internal standards for accuracy, completeness, and trust, displayed with a check mark icon.
* [Data Classification](/concepts/data-classification.md) - AI-agent-based automatic scanning of catalog data to tag sensitive information such as PII, financial data, and credentials, which can integrate with ABAC policies.
* [Data Classification (Databricks)](/concepts/data-classification-databricks.md) - A Databricks feature that detects sensitive data in tables and automatically assigns system tags to matching columns.
* [Data Classification (Unity Catalog)](/concepts/data-classification-unity-catalog.md) - Built-in system in Databricks Unity Catalog that scans tables and automatically detects sensitive data using built-in and custom classifiers.
* [Data Classification Billing and Cost Monitoring](/concepts/data-classification-billing-and-cost-tracking.md) - How to view and understand Databricks Data Classification expenses using the system.billing.usage table and usage dashboards, with initial scans costing more than incremental scans.
* [Data Classification Billing and Cost Tracking](/concepts/data-classification-billing-and-cost-tracking.md) - The ability to query Data Classification expenses via the system.billing.usage system table and usage dashboard, with initial scans costing more than subsequent incremental scans.
* [Data Classification Billing and Usage Tracking](/concepts/data-classification-billing-and-usage-tracking.md) - Usage and billing for Data Classification can be tracked via the system.billing.usage table or a usage dashboard, with initial scans being more costly than subsequent incremental scans.
* [Data Classification in Unity Catalog](/concepts/data-classification-unity-catalog.md) - The broader Databricks data classification system that automatically scans tables across catalogs to detect sensitive data using built-in and custom classifiers.
* [Data Classification Results Page](/concepts/data-classification-results-page.md) - The UI surface in Databricks Catalog Explorer where built-in and custom classifier detections are displayed and where custom classifiers are managed.
* [Data Classification Results System Table](/concepts/data-classification-system-table.md) - A system table (system.data_classification.results) storing all classification results across the metastore, including sample values, accessible only via serverless compute.
* [Data Classification Scans](/concepts/data-classification-scans.md) - The recurring scan process in Unity Catalog that detects built-in and custom classifications; new or updated classifiers only apply to subsequent scans, and results typically appear within a few hours.
* [Data Classification System Table](/concepts/data-classification-system-table.md) - The system.data_classification.results table that stores all classification results across the metastore, including sample values, accessible only via serverless compute.
* [Data Collation for Transformers](/concepts/data-collation-for-transformers.md) - Using DataCollatorWithPadding to batch input sequences of varying lengths for training and evaluation of transformer models.
* [Data Collation with DataCollatorWithPadding](/concepts/data-collation-with-datacollatorwithpadding.md) - Using Hugging Face data collators to batch input sequences with dynamic padding for training and evaluation
* [Data Completeness](/concepts/data-completeness.md) - A data quality metric measuring whether the expected number of rows were written to a table in the last 24 hours, based on historical row count analysis; tables are marked incomplete if row count falls below predicted lower bound.
* [Data Completeness (Anomaly Detection)](/concepts/data-completeness.md) - A quality metric that measures expected row counts over 24 hours using historical analysis, flagging tables with unusually low row additions
* [Data Completeness (Table Monitoring)](/concepts/data-completeness.md) - A monitoring metric that tracks the expected number of rows written to a table in a 24-hour window; tables are marked incomplete if row commits fall below the predicted lower bound.
* [Data Deprecation](/concepts/data-deprecation.md) - The practice of warning that a data asset is outdated, no longer reliable, or should not be used in new workflows, indicated by a restricted icon in the Databricks workspace.
* [Data Deprecation in Unity Catalog](/concepts/data-deprecation-in-unity-catalog.md) - The practice of labeling Unity Catalog objects as 'deprecated' to warn users that a data asset is outdated, unreliable, or should not be used in new workflows, displayed with a restricted icon.
* [Data Discovery in Unity Catalog](/concepts/data-discovery-in-unity-catalog.md) - The capability for users across an organization to find, understand, and trust data assets in Databricks Unity Catalog, shaped by administrator-defined metadata, trust signals, and organization.
* [Data Drift Detection](/concepts/data-drift-detection.md) - The practice of measuring changes in data distributions over time using Databricks profiling, with optional baseline tables and drift metrics for comparison.
* [Data Drift Measurement](/concepts/data-drift-measurement.md) - The process of measuring statistical changes in data distributions over time or relative to a baseline, including drift between successive time windows and between current data and expected values.
* [Data Drift Monitoring](/concepts/data-drift-monitoring.md) - The practice of tracking changes in data distributions over time against a baseline, detecting when data integrity, statistical distributions, or model performance shifts from expected norms.
* [Data Feature Types Supported by Databricks AutoML Forecasting](/concepts/data-feature-types-supported-by-databricks-automl-forecasting.md) - The specific column types (Numeric, Boolean, String, Timestamps, ArrayType[Numeric], DecimalType) that are supported for forecasting tasks in Databricks AutoML.
* [Data File Readability Verification](/concepts/data-file-readability-verification.md) - Direct read validation of data files to detect corruption or unreadable files in Delta Lake tables, performed by VERIFY ALL FILES mode.
* [Data Freshness](/concepts/data-freshness.md) - A data quality metric measuring how recently a table has been updated, determined by analyzing commit history to predict next commit timing; tables are marked stale if commits are unusually late.
* [Data Freshness (Anomaly Detection)](/concepts/data-freshness.md) - A quality metric that measures how recently a table has been updated, using historical commit patterns to predict expected next commit times
* [Data Freshness (Table Monitoring)](/concepts/data-freshness.md) - A monitoring metric that tracks how recently a table has been updated by analyzing commit history to predict the next commit time; tables are marked stale if a commit is unusually late.
* [Data Ingestion Strategy for Lakehouse](/concepts/data-ingestion-strategy-for-lakehouse.md) - Methods and patterns for ingesting data into the bronze layer, including Lakeflow Connect, partner tools, custom pipelines, batch/streaming ingestion, and change data capture (CDC).
* [Data leakage in feature engineering](/concepts/data-leakage-in-feature-engineering.md) - A modeling error that occurs when feature values unavailable at prediction time (e.g., future data) are included in the training set, degrading model performance.
* [Data Lineage](/concepts/data-lineage.md) - Automatic column-level tracking of how data flows across tables, notebooks, jobs, and pipelines, enabling impact analysis and origin tracing.
* [Data Lineage in Unity Catalog](/concepts/data-lineage-in-unity-catalog.md) - Automatic capture of data flow across tables, notebooks, jobs, and pipelines at the column level for tracing origins and impact analysis
* [Data Loading with Delta Lake for Deep Learning](/concepts/data-loading-with-delta-lake-for-deep-learning.md) - Using Delta Lake tables and streaming approaches (PyTorch IterableDataset, Hugging Face datasets, Ray Data) to optimize data throughput for large deep learning datasets.
* [Data Parallelism (Spark)](/concepts/data-parallelism-spark.md) - Spark excels at data parallelism — applying the same operation to each element of a large dataset — making it ideal for ETL, analytics, feature engineering, and large-scale data processing.
* [Data Pipeline Checkpointing](/concepts/data-pipeline-checkpointing.md) - Limitations of model checkpointing for data pipeline state and strategies for resuming training from the correct position in the dataset.
* [Data Pipeline Checkpointing Gap](/concepts/data-pipeline-checkpointing-gap.md) - Model checkpoints capture model and optimizer state but not the data pipeline position, so resumed runs cannot fast-forward to the exact sample where training stopped.
* [Data Pipeline State vs Model Checkpointing](/concepts/data-pipeline-state-vs-model-checkpointing.md) - Model checkpoints capture model and optimizer state but not data pipeline position, requiring explicit tracking of processed shards or epoch-boundary resumption strategies.
* [Data preparation for distributed training](/concepts/data-preparation-for-distributed-training.md) - The step of preparing data before running distributed training notebooks with HorovodRunner on Databricks.
* [Data Preparation for LLM Fine-Tuning with Hugging Face](/concepts/data-preparation-for-llm-fine-tuning-with-hugging-face.md) - Preparing data for fine-tuning open-source large language models using Hugging Face Transformers and Datasets libraries on Databricks.
* [Data Profile (Databricks)](/concepts/data-profile-databricks.md) - A monitoring construct on a Unity Catalog Delta table that tracks data quality metrics over time, supporting three profile types: TimeSeries, InferenceLog, and Snapshot.
* [Data Profile Refresh and Metric Tables](/concepts/data-profile-refresh-and-metric-tables.md) - The process of refreshing data profile metrics using create_refresh, viewing refresh history, and accessing metric tables stored as Unity Catalog tables for querying.
* [Data Profile Scheduling and Notifications](/concepts/data-profile-scheduling-and-notifications.md) - Options to run data profiling on a recurring schedule and configure email notifications for profile events, supporting up to 5 emails per event type.
* [Data Profiling](/concepts/data-profiling.md) - Summary statistics and historical metrics tracking for table data distributions, ML model performance, and inference endpoints; formerly known as Lakehouse Monitoring.
* [Data Profiling (Databricks)](/concepts/data-profiling-databricks.md) - A feature in Databricks that computes summary statistics for tables over time, enabling data quality monitoring, drift detection, and ML model performance tracking.
* [Data Profiling (Lakehouse Monitoring)](/concepts/data-profiling.md) - A feature that captures historical metrics of a table's data distribution or ML model performance, providing summary statistics, drift detection, and custom metrics to track data quality over time.
* [Data Profiling Analysis Types](/concepts/profiling-analysis-modes.md) - The three modes of analysis in Databricks data profiling: time series (temporal metrics over windows), inference (ML model performance tracking), and snapshot (single-point-in-time profiling).
* [Data profiling dashboard](/concepts/data-profiling-dashboard.md) - A dashboard automatically created by Databricks when a data profile is run, displaying key metrics computed by the profile in customizable visualizations.
* [Data Profiling Dashboard (Databricks)](/concepts/data-profiling-dashboard.md) - An auto-generated dashboard in Databricks that displays key metrics computed by a data profile run, organized into sections based on profile type.
* [Data Profiling Dashboard Filters](/concepts/data-profiling-dashboard-filters.md) - Widgets at the top of the dashboard that control which data is displayed, with different selectors for Snapshot, Timeseries, and InferenceLog analysis profiles.
* [Data profiling expense tracking](/concepts/data-profiling-expense-tracking.md) - Technique to isolate data profiling costs by filtering on a non-null table_id in the billing usage table.
* [Data Profiling in Databricks](/concepts/data-profiling-in-databricks.md) - A feature in Databricks Unity Catalog that analyzes table data to compute summary statistics, detect anomalies, and monitor data quality over time, configurable via the UI or API.
* [Data Profiling in Databricks Unity Catalog](/concepts/data-profiling-in-databricks.md) - The process of analyzing table statistics and data quality metrics using the Databricks UI within Unity Catalog, supporting profile types for general analysis, time series, and ML inference monitoring.
* [Data Profiling in Unity Catalog](/concepts/data-profiling-in-databricks.md) - The practice of analyzing and generating statistical summaries of table data within Databricks Unity Catalog to monitor data quality over time.
* [Data Profiling Limitations](/concepts/data-profiling-limitations.md) - Constraints for Databricks data profiling including supported table types (only Delta tables), regional availability, 30-day time window for time series/inference profiles, and 4TB maximum for snapshot profiles.
* [Data Profiling Limitations and Requirements](/concepts/data-profiling-limitations.md) - Constraints for Databricks data profiling including Unity Catalog enablement, supported table types (Delta only), regional availability, and size limits for snapshot profiles.
* [Data Profiling UI Workflow](/concepts/data-profiling-ui-workflow.md) - The step-by-step process in Databricks Catalog Explorer to enable, configure, schedule, refresh, view, and delete data profiles through the Quality tab interface.
* [Data provider and data recipient roles](/concepts/data-provider-and-data-recipient-roles.md) - In OpenSharing, a data provider (a Databricks user) shares data with a data recipient (a person or group outside their organization), with distinct responsibilities and workflows for each.
* [Data Quality in the Medallion Pipeline](/concepts/data-quality-in-the-medallion-pipeline.md) - A strategy for progressively enforcing data quality through constraints, primary/foreign keys, expectations, and monitoring as data moves from bronze (lenient) to silver and gold (strict).
* [Data Quality Monitoring](/concepts/data-quality-monitoring.md) - Anomaly detection across all tables in a schema and data profiling at the table level, automatically monitoring freshness and completeness using historical data patterns.
* [Data quality monitoring (Databricks)](/concepts/data-quality-monitoring-databricks.md) - A Unity Catalog feature that ensures the quality of data assets through anomaly detection and data profiling capabilities.
* [Data Quality Monitoring (Unity Catalog)](/concepts/data-quality-monitoring-databricks.md) - A Databricks capability in Unity Catalog that ensures data asset quality through anomaly detection and data profiling without modifying tables or adding job overhead.
* [Data Quality Monitoring Alert Rules](/concepts/data-quality-monitoring-alert-rules.md) - Rules that notify workspace users by email when anomaly detection identifies unhealthy tables within a catalog or schema scope.
* [Data Quality Monitoring Alerts](/concepts/data-quality-monitoring-alerts.md) - Notification system that alerts workspace users when anomaly detection identifies data quality issues like stale tables or unexpected drops in row count.
* [Data Quality Monitoring Incidents Management](/concepts/data-quality-monitoring-incidents-management.md) - Workflow for reviewing, assigning, and dismissing data quality anomalies from the UI, including auto-resolved incidents that self-heal without manual intervention.
* [Data Quality Monitoring UI](/concepts/data-quality-monitoring-ui.md) - A Databricks interface (Beta) for creating and managing anomaly detection alert rules directly within the Data Quality Monitoring page, including scope and recipient configuration.
* [Data Quality Monitoring UI (Beta)](/concepts/data-quality-monitoring-ui-beta.md) - A Beta UI feature in Databricks Catalog Explorer that provides a visual interface for viewing data quality results and managing alert rules without leaving the page.
* [Data Quality Monitoring UI (Databricks)](/concepts/data-quality-monitoring-ui-databricks.md) - A user interface for viewing anomaly detection results, incidents per schema or catalog, with tabs for unhealthy, healthy, or error tables and features for assigning and dismissing incidents.
* [Data Quality Monitoring UI Alert Management](/concepts/data-quality-monitoring-ui.md) - Beta feature for creating and managing anomaly detection alert rules directly from the Data Quality Monitoring UI in Catalog Explorer, scoped to catalogs or schemas.
* [Data Recency Constraint for MLflow System Tables](/concepts/data-recency-constraint-for-mlflow-system-tables.md) - The MLflow system tables began recording data from all regions on September 2, 2025; data before that date may not be available.
* [Data Recipient](/concepts/data-recipient.md) - A person or group outside the data provider's organization who receives and accesses shared data via OpenSharing, with read-only access.
* [Data Recipient (OpenSharing)](/concepts/data-recipient-opensharing.md) - A named object in OpenSharing that represents the identity of a user or group of users who consume shared data, identified by a sharing identifier for Databricks-to-Databricks mode.
* [Data Slicing Expressions](/concepts/data-slicing-expressions.md) - A mechanism to compute per-segment statistics by partitioning data based on column values or Boolean expressions, generating multiple slice keys and values.
* [Data Slicing in Databricks Profiling](/concepts/data-slicing-in-databricks-profiling.md) - A technique for computing profile metrics on subsets of data defined by boolean expressions or column values, enabling segmented analysis of data quality.
* [Data Sources for Features](/concepts/data-sources-for-features.md) - DeltaTableSource and RequestSource define where feature data originates: Delta tables (with optional filtering and transformations) or inference-time request payloads with a defined schema.
* [Data Splitting Strategies](/concepts/data-splitting-strategies.md) - Methods for splitting data into train, validation, and test sets: random stratified split, chronological split by time column, and manual split using a designated column.
* [Data Splitting Strategies for AutoML](/concepts/data-splitting-strategies.md) - Methods to divide data into training, validation, and test sets in AutoML: random split (default, 60/20/20), chronological split (by time column), and manual split (by designated split column).
* [Data Splitting Strategies in AutoML](/concepts/data-splitting-strategies.md) - Methods for splitting data into training, validation, and test sets, including random split, chronological split, and manual split options.
* [Data type mismatch in row filters and column masks](/concepts/data-type-mismatch-in-row-filters-and-column-masks.md) - Risks and unexpected behavior when UDF parameter types do not match table column types, including silent NULL conversion with ANSI mode disabled.
* [Databricks @distributed Decorator](/concepts/databricks-distributed-decorator.md) - A Python decorator in Databricks Serverless GPU Compute that orchestrates multi-GPU distributed training by launching a function across all GPUs with proper distributed setup.
* [Databricks @distributed Decorator for GPU Training](/concepts/databricks-distributed-decorator-for-gpu-training.md) - A decorator from the serverless_gpu library that enables execution of GPU workloads on Databricks AI Runtime, automatically provisioning GPUs (e.g., 8xH100) and handling distributed training setup.
* [Databricks @distributed Decorator for Multi-GPU Training](/concepts/databricks-distributed-decorator.md) - A decorator-based API in Databricks Serverless GPU Compute that automatically orchestrates distributed training across multiple GPUs, handling provisioning, data distribution, and synchronization.
* [Databricks Acceptable Use Policy for Models](/concepts/databricks-acceptable-use-policy-for-models.md) - A governing policy that defines acceptable and prohibited uses of models served through Databricks Model Serving.
* [Databricks Agent Evaluation Scorers](/concepts/databricks-agent-evaluation-scorers.md) - Framework from Databricks for evaluating AI agent outputs, including built-in scorers like Safety() and a lifecycle of register/start.
* [Databricks AI Runtime](/concepts/databricks-ai-runtime.md) - A serverless GPU compute offering on Databricks for deep learning workloads, providing fully managed GPU infrastructure with native integration across notebooks, jobs, Unity Catalog, and MLflow.
* [Databricks AI Runtime (AI v5)](/concepts/databricks-ai-runtime-ai-v5.md) - A Databricks environment image pre-packaged with AI/ML libraries including Unsloth, TRL, PEFT, bitsandbytes, xformers, and einops, eliminating manual installations.
* [Databricks AI Runtime (AI v5) for LLM Workloads](/concepts/databricks-ai-runtime-ai-v5-for-llm-workloads.md) - A curated Databricks runtime environment (AI v5) that bundles Unsloth, TRL, PEFT, bitsandbytes, xformers, and other LLM finetuning dependencies, accessible via serverless GPU clusters with accelerator selection.
* [Databricks AI Runtime CLI (air)](/concepts/databricks-ai-runtime-cli-air.md) - Command-line interface for submitting and managing distributed AI/ML workloads on Databricks clusters, using commands like `air run -f train.yaml`
* [Databricks AI Runtime CLI (air) workload submission](/concepts/databricks-ai-runtime-cli-air.md) - CLI tool for submitting distributed training workloads to Databricks using YAML configuration with inline dependencies, compute specs, and command scripts.
* [Databricks AI Runtime CLI YAML config](/concepts/databricks-ai-runtime-cli-yaml-config.md) - The YAML-based configuration system used by the air CLI to define workload specifications, with field-level help accessible via '-h config'.
* [Databricks AI Runtime Environment](/concepts/databricks-ai-runtime-environment.md) - A pre-configured runtime environment (AI v5) on Databricks that bundles all required libraries for running serverless GPU compute workloads with H100 accelerators.
* [Databricks AI Runtime Environments](/concepts/ai-runtime-environments.md) - Two managed Python environments offered by AI Runtime: a minimal default base environment and a full-featured Databricks AI environment pre-loaded with ML frameworks like PyTorch and Transformers.
* [Databricks AI Runtime for Distributed Training](/concepts/databricks-ai-runtime-for-distributed-training.md) - A Databricks platform runtime that supports distributed training frameworks including DeepSpeed, DDP, and FSDP on AWS.
* [Databricks AI Runtime for GPU Fine-Tuning](/concepts/databricks-ai-runtime-for-gpu-fine-tuning.md) - A pre-configured Databricks environment (AI v5) that bundles Unsloth, bitsandbytes, TRL, xformers, MLflow and other dependencies, optimized for running GPU-accelerated LLM fine-tuning workloads on serverless compute with A10 accelerators.
* [Databricks AI Runtime GPU training](/concepts/databricks-ai-runtime-gpu-training.md) - Databricks' managed environment supporting distributed GPU training with DDP, PyTorch Lightning, and serverless GPU resources.
* [Databricks AI Runtime Observability](/concepts/databricks-ai-runtime-observability.md) - Overview of observability features in Databricks AI Runtime including MLflow logging, notebook output, GPU monitoring, and checkpoint management.
* [Databricks AI Runtime on AWS](/concepts/databricks-ai-runtime-on-aws.md) - A managed AI runtime environment on AWS for running distributed machine learning workloads, including LLM fine-tuning and GPU-based training.
* [Databricks AI Runtime release stages](/concepts/databricks-ai-runtime-release-stages.md) - AI Runtime features are released in different stages: single-node tasks in Public Preview, multi-GPU distributed training in Beta.
* [Databricks AI Runtime Requirements](/concepts/databricks-ai-runtime-requirements.md) - Prerequisites for using AI Runtime, including a Databricks workspace with AI Runtime enabled and Python 3.10+.
* [Databricks AI v5 Environment](/concepts/databricks-ai-v5-environment.md) - A Databricks runtime environment pre-packaged with libraries for AI/ML GPU workloads on Serverless GPU compute
* [Databricks Anomaly Detection](/concepts/databricks-anomaly-detection.md) - Automated monitoring of data quality across tables in a schema using historical pattern analysis to detect freshness and completeness anomalies
* [Databricks App Resource Permission Levels](/concepts/app-resource-permission-levels.md) - Three permission levels (Can read, Can edit, Can manage) that control an app's service principal access to a resource like an MLflow experiment.
* [Databricks App Resource Permissions](/concepts/app-resource-permission-levels.md) - Permission levels (Can read, Can edit, Can manage) that govern an app's service principal access to bound resources like MLflow experiments.
* [Databricks App Service Principal Authorization](/concepts/databricks-app-service-principal-authorization.md) - The mechanism by which Databricks grants a Databricks App's service principal specific permissions on resources like MLflow experiments, scoping access to only the explicitly added resources.
* [Databricks Apps MLflow Experiment Resource](/concepts/databricks-apps-mlflow-experiment-resource.md) - An MLflow experiment added as a first-class resource in a Databricks App, enabling experiment tracking, agent debugging, prompt management, and model training logging directly from the app.
* [Databricks Apps Resource Permission Levels](/concepts/app-resource-permission-levels.md) - Three permission tiers (Can read, Can edit, Can manage) that control what a Databricks App can do with a bound resource like an MLflow experiment, from viewing metadata to full administrative access.
* [Databricks Apps Service Principal Authorization](/concepts/databricks-apps-service-principal-authorization.md) - The mechanism by which Databricks Apps use a service principal to obtain permissions on resources — when a resource is added, the app's service principal is granted the specified permissions on that resource.
* [DATABRICKS Authentication Type](/concepts/databricks-authentication-type.md) - An authentication type for recipient objects that restricts data access to a specific Unity Catalog metastore identified by a sharing identifier, used in Databricks-to-Databricks sharing.
* [Databricks Autologging](/concepts/databricks-autologging.md) - A Databricks feature that enables automatic MLflow tracking by default on supported runtimes, including MLflow PySpark ML autologging.
* [Databricks Autologging Administration](/concepts/databricks-autologging-administration.md) - Workspace-level controls allowing administrators to enable or disable Databricks Autologging for all interactive notebook sessions across their workspace via the admin settings page, requiring cluster restart to take effect.
* [Databricks Autologging Limitations](/concepts/databricks-autologging-limitations.md) - Key constraints including that autologging is only enabled on the driver node, must be explicitly enabled on worker nodes, and the XGBoost scikit-learn integration is not supported.
* [Databricks AutoML](/concepts/databricks-automl.md) - Automated machine learning system on Databricks that trains models and generates trial notebooks for classification, regression, and forecasting tasks.
* [Databricks AutoML Experiment Lifecycle for Forecasting](/concepts/databricks-automl-experiment-lifecycle-for-forecasting.md) - The three-stage pipeline (Preprocessing, Tuning, Training) that Databricks Serverless Forecasting follows when running an AutoML experiment.
* [Databricks AutoML Feature Store Lookups](/concepts/automl-feature-store-lookups.md) - Mechanism to pass external feature tables as covariates to AutoML forecasting experiments via the feature_store_lookups parameter
* [Databricks AutoML for Classification](/concepts/databricks-automl-for-classification.md) - How to use Databricks AutoML to automatically find the best classification algorithm and hyperparameter configuration
* [Databricks AutoML Forecasting API](/concepts/databricks-automl-forecasting-api.md) - The automl.forecast() Python function that automates time series model selection, training, and evaluation on Databricks.
* [Databricks AutoML Forecasting Data Preparation](/concepts/databricks-automl-forecasting-data-preparation.md) - Process and configurable settings for preparing time series data for automated forecasting model training on Databricks.
* [Databricks AutoML Forecasting Experiment](/concepts/databricks-automl-forecasting-experiment.md) - The automated machine learning workflow in Databricks for forecasting tasks, supporting parameters like target_col, time_col, frequency, horizon, and identity_col.
* [Databricks AutoML Python API](/concepts/databricks-automl-python-api.md) - Programmatic Python interface for Databricks AutoML, providing methods to start classification, regression, and forecasting runs
* [Databricks AutoML Regression](/concepts/databricks-automl-regression.md) - Automated machine learning process for predicting continuous numeric values using Databricks AutoML
* [Databricks AutoML UI Experiment Workflow](/concepts/databricks-automl-ui-experiment-workflow.md) - End-to-end process for setting up, running, and monitoring a forecasting AutoML experiment through the Databricks UI, including compute selection, dataset browsing, and progress tracking
* [Databricks Budget Policy Permission Model](/concepts/databricks-budget-policy-permission-model.md) - The requirement that users and service principals can only assign a budget policy they are entitled to use, governing who can set serverless budget policies on experiments.
* [Databricks Built-in SQL Functions and Operators](/concepts/databricks-built-in-sql-functions-and-operators.md) - The comprehensive reference of SQL functions and operators available in Databricks, from which a security-restricted subset is chosen for use in Databricks-to-Databricks view sharing.
* [Databricks Chat Completions API](/concepts/databricks-chat-completions-api.md) - A unified API on Databricks that works across all model providers, mentioned as an alternative to the Anthropic-specific Messages API.
* [Databricks Claude Model Naming Convention](/concepts/databricks-claude-model-naming-convention.md) - The naming pattern for Anthropic Claude models hosted on Databricks, using the prefix 'databricks-claude-' followed by the model tier and version (e.g., databricks-claude-sonnet-4-5).
* [Databricks Clean Rooms](/concepts/databricks-clean-rooms.md) - Secure collaboration environments within Databricks that allow multiple parties to share and jointly analyze data, which can cause deletion blocks on shared securables.
* [Databricks CLONE SQL Syntax](/concepts/databricks-clone-sql-syntax.md) - The SQL language syntax for the CREATE TABLE CLONE and REPLACE TABLE CLONE statements, including parameters such as IF NOT EXISTS, SHALLOW/DEEP, TBLPROPERTIES, and LOCATION.
* [Databricks Cluster Access Modes and Table ACLs](/concepts/databricks-cluster-access-modes-and-table-acls.md) - Table access control is enabled by default on clusters with Standard access mode, and is not supported on Machine Learning Runtime.
* [Databricks Cluster Access Modes for Table ACLs](/concepts/databricks-cluster-access-modes-for-table-acls.md) - How table access control is enabled by default in Standard access mode clusters and configured via Spark conf settings.
* [Databricks Commitment to Open-Source ML Ecosystem](/concepts/databricks-open-source-ml-ecosystem.md) - Full support for open-source ML frameworks (scikit-learn, PyTorch, TensorFlow, etc.) with open formats via Delta Lake, open-source MLflow, and open-source Unity Catalog APIs for portability.
* [Databricks Configuration Profiles](/concepts/databricks-configuration-profiles.md) - Named configuration files used to store connection properties (host, token, cluster_id, etc.) for Databricks authentication and compute targeting
* [Databricks configuration profiles for authentication](/concepts/databricks-configuration-profiles-for-authentication.md) - Using profiles (in .databrickscfg) to store authentication fields (host, token, client_id, client_secret) and cluster_id for Databricks Connect connections.
* [Databricks Configuration Profiles for Connect](/concepts/databricks-configuration-profiles-for-connect.md) - Named configuration profiles (including DEFAULT) store connection fields like host, token, cluster_id, client_id, and client_secret for Databricks Connect authentication.
* [Databricks Connect](/concepts/databricks-connect.md) - A client library for the Databricks Runtime that enables remote Spark execution from IDEs, notebooks, and custom applications using Spark Connect.
* [Databricks Connect (Legacy)](/concepts/databricks-connect-legacy.md) - A client library that allows users to connect local IDEs and applications to Databricks clusters, running Spark jobs remotely instead of locally.
* [Databricks Connect API Portability](/concepts/databrickssession-api-portability.md) - Databricks Connect APIs are available in Databricks notebooks as part of the corresponding Databricks Runtime, allowing code to run in notebooks without any changes after local development.
* [Databricks Connect Authentication](/concepts/databricks-connect-authentication.md) - Supported authentication methods for Databricks Connect including OAuth user-to-machine (U2M) and OAuth machine-to-machine (M2M), with specific configuration requirements.
* [Databricks Connect authentication defaults](/concepts/databricks-connect-authentication-defaults.md) - The default authentication mechanism used by Databricks Connect client setup, configurable via environment variables such as SPARK_REMOTE
* [Databricks Connect Authentication for R](/concepts/databricks-connect-authentication-for-r.md) - The authentication mechanism for Databricks Connect for R, which currently only supports Databricks personal access tokens (PAT) stored in environment variables.
* [Databricks Connect Authentication Methods](/concepts/databricks-connect-authentication.md) - Supported authentication mechanisms for Databricks Connect including OAuth user-to-machine (U2M) and machine-to-machine (M2M) authentication via the Databricks SDK for Python.
* [Databricks Connect Authentication Requirements](/concepts/databricks-connect-authentication-requirements.md) - Databricks Connect supports OAuth U2M and M2M authentication; U2M requires prior authentication via Databricks CLI, and both are supported on Databricks SDK for Python 0.19.0+.
* [Databricks Connect Authentication Type Requirements](/concepts/databricks-connect-authentication-type-requirements.md) - The specific configuration fields required for each authentication type (PAT, OAuth M2M, OAuth U2M) when connecting via Databricks Connect.
* [Databricks Connect Authentication Types](/concepts/databricks-connect-authentication.md) - Supported authentication methods for Databricks Connect including personal access tokens, OAuth M2M, and OAuth U2M
* [Databricks Connect Authentication via PAT](/concepts/databricks-connect-authentication-via-pat.md) - Databricks Connect for R currently only supports Databricks personal access token (PAT) authentication for workspace and cluster access.
* [Databricks Connect Authentication via Personal Access Tokens](/concepts/databricks-connect-authentication-via-pat.md) - The authentication mechanism for Databricks Connect for R, which currently only supports Databricks personal access tokens (PAT) for workspace authentication.
* [Databricks Connect authentication with personal access tokens](/concepts/databricks-connect-authentication-via-pat.md) - Databricks Connect for R currently only supports authentication via Databricks personal access tokens (PAT), configured through environment variables DATABRICKS_HOST, DATABRICKS_TOKEN, and DATABRICKS_CLUSTER_ID.
* [Databricks Connect build file configuration](/concepts/databricks-connect-build-file-configuration.md) - How to configure build.sbt, pom.xml, or build.gradle to add the Databricks Connect library dependency matching the target Databricks Runtime version.
* [Databricks Connect Client Setup](/concepts/databricks-connect-client-setup.md) - Step-by-step process to install the databricks-connect package, uninstall conflicting PySpark, configure connection properties (workspace URL, personal access token, cluster ID, port), and test connectivity.
* [Databricks Connect Client Setup and Configuration](/concepts/databricks-connect-client-setup.md) - The step-by-step process of installing the Databricks Connect client package, configuring connection properties (workspace URL, token, cluster ID, port), and testing connectivity to a Databricks cluster.
* [Databricks Connect Client-Server Architecture](/concepts/databricks-connect-client-server-architecture.md) - Architecture where the local client sends logical command representations (e.g., DataFrames) to a Spark server on the remote Databricks cluster for execution, decoupling the client from the cluster.
* [Databricks Connect Cluster Access Modes](/concepts/databricks-connect-cluster-access-modes.md) - Target clusters for Databricks Connect must use either Assigned or Shared cluster access mode.
* [Databricks Connect cluster configuration requirements](/concepts/databricks-connect-cluster-configuration-requirements.md) - The target Databricks workspace and cluster must meet specific compute configuration requirements for Databricks Connect to work, including appropriate runtime versions and cluster settings.
* [Databricks Connect cluster connectivity troubleshooting](/concepts/databricks-connect-cluster-connectivity-troubleshooting.md) - Common errors and fixes when Databricks Connect cannot reach a cluster, including checking workspace instance name, cluster ID, and cluster version compatibility.
* [Databricks Connect Cluster Disabling](/concepts/databricks-connect-service-disabling.md) - Databricks Connect and Spark Connect services can be disabled on a cluster by setting spark.databricks.service.server.enabled false in Spark configuration.
* [Databricks Connect Cluster Requirements](/concepts/databricks-connect-cluster-requirements.md) - Requirements for target compute resources including cluster access modes (Assigned or Shared) and serverless compute support (starting from Databricks Connect 15.1)
* [Databricks Connect code examples and tutorials](/concepts/databricks-connect-code-examples-and-tutorials.md) - Tutorials and example applications demonstrating Databricks Connect for Python, including ETL, Plotly, and PySpark AI examples.
* [Databricks Connect Code Portability](/concepts/databrickssession-api-portability.md) - The principle that all Databricks Connect APIs are available in Databricks notebooks as part of the corresponding Databricks Runtime, enabling seamless transitions from local development to deployment without code changes.
* [Databricks Connect compute configuration](/concepts/databricks-connect-compute-configuration.md) - Configuration options for classic compute and serverless compute when using Databricks Connect.
* [Databricks Connect Configuration](/concepts/databricks-connect-configuration.md) - The process of configuring Databricks Connect including workspace URL, personal access token, cluster ID, and port settings via CLI, SQL configs, or environment variables.
* [Databricks Connect Configuration and Setup](/concepts/databricks-connect-client-setup.md) - Step-by-step process for installing the Databricks Connect client, configuring connection properties (workspace URL, personal access token, cluster ID, port), and testing connectivity.
* [Databricks Connect Configuration Precedence](/concepts/databricks-connect-configuration-precedence.md) - Databricks Connect searches for configuration properties in a defined order (remote() method, config profile, env vars, DEFAULT profile) and uses the first match.
* [Databricks Connect Configuration Profiles](/concepts/databricks-connect-configuration-profiles.md) - Using .databrickscfg configuration profiles (including DEFAULT) to store connection properties like host, token, cluster_id, and serverless_compute_id.
* [Databricks Connect Configuration Resolution Order](/concepts/databricks-connect-configuration-resolution-order.md) - The ordered sequence (5 levels) in which Databricks Connect searches for connection configuration properties, using the first match found.
* [Databricks Connect Connection String Configuration](/concepts/databricks-connect-connection-string-configuration.md) - How to configure Databricks Connect using the Spark Connect connection string via the remote() function or the SPARK_REMOTE environment variable
* [Databricks Connect Connection Validation](/concepts/databricks-connect-connection-validation.md) - Methods to validate that the environment, credentials, and compute connection are correctly configured, including the databricks-connect test command and validateSession().
* [Databricks Connect Credential Resolution](/concepts/databricks-connect-credential-resolution.md) - The mechanism by which the PySpark shell picks up default credentials from environment variables (DATABRICKS_* vars) or a DEFAULT configuration profile when started without additional parameters.
* [Databricks Connect Custom Proxy Headers](/concepts/databricks-connect-custom-proxy-headers.md) - Using the header() method to add custom HTTP headers to gRPC requests when a proxy service sits between the client and the Databricks cluster
* [Databricks Connect DataFrame Logical Plan Size Limit](/concepts/databricks-connect-dataframe-logical-plan-size-limit.md) - Databricks Connect cannot create DataFrames with an unresolved logical plan larger than 128 MB (applies to plan size, not data) for Runtime 13.3 LTS and below.
* [Databricks Connect Dependency Management](/concepts/databricks-connect-dependency-management.md) - The approach to handling dependencies where application dependencies are installed locally on the client machine and UDF dependencies are installed on the Databricks cluster.
* [Databricks Connect End-of-Support Lifecycle](/concepts/databricks-connect-support-lifecycle.md) - Databricks Connect follows the Databricks Runtime support lifecycle policy; versions that have reached end-of-support should be upgraded to a currently supported version.
* [Databricks Connect Environment Variable Configuration](/concepts/databricks-connect-environment-variable-configuration.md) - Environment variables such as DATABRICKS_CONFIG_PROFILE, DATABRICKS_CLUSTER_ID, DATABRICKS_HOST, DATABRICKS_TOKEN can configure the connection without hardcoding values.
* [Databricks Connect Execution Model](/concepts/databricks-connect-execution-model.md) - The architecture where general Python/Scala code runs locally on the client, DataFrame APIs execute on the remote Databricks compute, and UDF code is serialized and transmitted to the cluster for execution.
* [Databricks Connect Feature Limitations by Runtime Version](/concepts/databricks-connect-feature-limitations-by-runtime-version.md) - Databricks Connect functionality is gated by Databricks Runtime version, with certain features unavailable depending on the runtime version (13.3 LTS, 15.3, 16.3).
* [Databricks Connect for Python](/concepts/databricks-connect-for-python.md) - A Databricks SDK feature enabling IDEs, notebook servers, and custom applications to connect to Databricks clusters using PySpark.
* [Databricks Connect for Python Volume File Operations](/concepts/databricks-connect-for-python-volume-file-operations.md) - Example pattern demonstrating how to use Databricks SDK's WorkspaceClient and dbutils.fs to create, read, and delete files in Unity Catalog volumes.
* [Databricks Connect for R](/concepts/databricks-connect-for-r.md) - A feature enabling R IDEs like RStudio Desktop to connect to Databricks clusters using sparklyr for remote Spark execution.
* [Databricks Connect for R (sparklyr)](/concepts/databricks-connect-for-r.md) - A feature that enables connecting R IDEs like RStudio Desktop to Databricks clusters using the sparklyr package, allowing remote execution of Spark code from local R environments.
* [Databricks Connect for Scala](/concepts/databricks-connect-for-scala.md) - A tool for connecting Scala IDEs, notebook servers, and custom applications to Databricks clusters, introduced for Databricks Runtime 13.3 LTS and above.
* [Databricks Connect for Scala Compatibility](/concepts/databricks-connect-for-scala-compatibility.md) - This version of Databricks Connect for Scala requires Databricks Runtime 13.3 LTS and above
* [Databricks Connect for Scala installation](/concepts/databricks-connect-for-scala-installation.md) - Installation process for setting up Databricks Connect for Scala on a local development machine.
* [Databricks Connect for Scala installation methods](/concepts/databricks-connect-for-scala-installation-methods.md) - The procedures for adding Databricks Connect as a dependency in Scala projects using sbt, Maven, or Gradle build tools.
* [Databricks Connect for Scala Runtime Compatibility](/concepts/databricks-connect-for-scala-compatibility.md) - The described Databricks Utilities integration works with Databricks Connect for Databricks Runtime 13.3 LTS and above.
* [Databricks Connect GitHub Examples](/concepts/databricks-connect-github-examples.md) - A GitHub repository by Databricks containing example applications demonstrating Databricks Connect usage, including ETL, Plotly visualizations, and PySpark AI.
* [Databricks Connect GitHub Examples Repository](/concepts/databricks-connect-github-examples.md) - A curated collection of example applications demonstrating Databricks Connect usage, including ETL, Plotly visualizations, and PySpark AI integrations
* [Databricks Connect GitHub repository](/concepts/databricks-connect-github-examples.md) - A GitHub repository (dbconnect-examples) containing example applications for Databricks Connect including ETL and Plotly-based interactive apps
* [Databricks Connect gRPC Communication](/concepts/databricks-connect-grpc-communication.md) - Databricks Connect communicates with Databricks Clusters via gRPC over HTTP/2, which is the underlying protocol for Spark Connect connections.
* [Databricks Connect gRPC Communication Layer](/concepts/databricks-connect-grpc-communication-layer.md) - Databricks Connect communicates with Databricks clusters via gRPC over HTTP/2, enabling advanced proxy and header customization
* [Databricks Connect gRPC Communication Protocol](/concepts/databricks-connect-grpc-communication.md) - Databricks Connect communicates with Databricks Clusters via gRPC over HTTP/2, which underlies the need for proxy configuration and certificate management.
* [Databricks Connect gRPC over HTTP/2 Communication](/concepts/databricks-connect-grpc-communication.md) - The underlying protocol used by Databricks Connect to communicate with Databricks clusters via gRPC over HTTP/2
* [Databricks Connect Hybrid Session Pattern](/concepts/databricks-connect-hybrid-session-pattern.md) - A design pattern where code checks for the DATABRICKS_RUNTIME_VERSION environment variable to decide whether to reuse an existing SparkSession (on-cluster) or create a new DatabricksSession with artifact uploads (local development).
* [Databricks Connect IDE Integration](/concepts/databricks-connect-ide-integration.md) - Integration of Databricks Connect with popular IDEs and tools including Visual Studio Code, PyCharm, JupyterLab, Classic Jupyter Notebook, RStudio Desktop, IntelliJ, Eclipse, PyDev, SBT, and Spark shell for remote development and debugging.
* [Databricks Connect IDE Integration Patterns](/concepts/databricks-connect-ide-integration.md) - The various methods for integrating Databricks Connect with popular development environments including JupyterLab, PyCharm, VS Code, IntelliJ, Eclipse, RStudio, and Spark shells.
* [Databricks Connect incompatibility with open source Apache Spark](/concepts/databricks-connect-incompatibility-with-open-source-apache-spark.md) - Databricks Connect is built against Databricks Runtime and cannot connect to open source Apache Spark servers due to dependency differences such as json4s library versions.
* [Databricks Connect Interactive Development](/concepts/databricks-connect-interactive-development.md) - The capability to interactively develop and debug Spark code from IDEs (VS Code, PyCharm, IntelliJ) and build interactive data applications using Databricks compute instead of local Spark sessions.
* [Databricks Connect Java 17 compatibility](/concepts/databricks-connect-java-17-compatibility.md) - Java 17's security restrictions block Apache Arrow reflection access needed by Databricks Connect; the fix requires setting --add-opens JVM flag.
* [Databricks Connect Language Support](/concepts/databricks-connect-language-support.md) - Databricks Connect is available for Python, R, and Scala, allowing developers to write code using Spark APIs in their preferred language and run it remotely on Databricks compute.
* [Databricks Connect library distribution via Maven Central](/concepts/databricks-connect-library-distribution-via-maven-central.md) - The Databricks Connect client library is distributed through Maven Central repository, with separate artifact coordinates for different Databricks Runtime versions.
* [Databricks Connect Limitations](/concepts/databricks-connect-limitations.md) - Known limitations of Databricks Connect including lack of support for Unity Catalog, Structured Streaming, certain Delta table operations, Koalas, table access control, and various dbutils modules.
* [Databricks Connect Limitations and Unsupported Features](/concepts/databricks-connect-limitations.md) - The documented set of features and APIs that are not supported when using Databricks Connect, including Unity Catalog, Structured Streaming, Delta table native APIs, dbutils subsets, and table access control.
* [Databricks Connect Local Environment Requirements](/concepts/databricks-connect-local-environment-requirements.md) - Local development prerequisites for Databricks Connect including Python version, authentication setup (OAuth U2M/M2M), and Scala compatibility.
* [Databricks Connect Local Environment Setup](/concepts/databricks-connect-local-environment-setup.md) - Local development environment prerequisites including Python 3 installation, Python version matching for UDFs, and Scala requirements
* [Databricks Connect Local Python Environment Requirements](/concepts/databricks-connect-local-python-environment-requirements.md) - Python 3 must be installed locally, and the minor version must satisfy the version compatibility table; the document references both Python and Scala APIs.
* [Databricks Connect Logging and Debug Logs](/concepts/databricks-connect-logging-and-debugging.md) - Using the SPARK_CONNECT_LOG_LEVEL environment variable and standard Python logging to enable debug-level log output from Databricks Connect.
* [Databricks Connect Logging and Debugging](/concepts/databricks-connect-logging-and-debugging.md) - Enabling debug logging in Databricks Connect for Python via the SPARK_CONNECT_LOG_LEVEL environment variable
* [Databricks Connect Logging Configuration](/concepts/databricks-connect-logging-and-debugging.md) - Enabling debug logs in Databricks Connect for Python by setting the `SPARK_CONNECT_LOG_LEVEL=debug` environment variable, which sends logs to stderr using standard Python logging.
* [Databricks Connect Long-lived Sessions](/concepts/databricks-connect-long-lived-sessions.md) - On serverless compute (DBR 16.4+), Databricks Connect best-effort preserves session state — including configs, temporary views, UDFs, temp variables, and uploaded files — across idle timeouts, so users don't need to re-run setup commands.
* [Databricks Connect MemoryUtil initialization error](/concepts/databricks-connect-memoryutil-initialization-error.md) - An error occurring when building a DatabricksSession due to Apache Arrow's reflection access to private Java methods being blocked in Java 17.
* [Databricks Connect migration](/concepts/databricks-connect-migration.md) - Process for migrating from Databricks Connect for Databricks Runtime 12.2 LTS and below to Databricks Connect for Databricks Runtime 13.3 LTS and above.
* [Databricks Connect MLlib Compatibility Limitations](/concepts/databricks-connect-mllib-limitation.md) - A limitation where Databricks Connect has limited compatibility with Apache Spark MLlib because MLlib uses RDDs while Databricks Connect only supports the DataFrame API.
* [Databricks Connect MLlib Limitation](/concepts/databricks-connect-mllib-limitation.md) - Databricks Connect has limited compatibility with Apache Spark MLlib because MLlib uses RDDs while Databricks Connect only supports the DataFrame API.
* [Databricks Connect MLlib limitations in R](/concepts/databricks-connect-mllib-limitation.md) - Spark MLlib functions that rely on RDDs are incompatible with Databricks Connect, which only supports the DataFrame API, limiting sparklyr's MLlib capabilities.
* [Databricks Connect Project Setup with renv](/concepts/databricks-connect-project-setup-with-renv.md) - A reproducible R project workflow using RStudio, renv, and environment variables (.Renviron) to configure Databricks Connect connections.
* [Databricks Connect public preview](/concepts/databricks-connect-public-preview.md) - The Public Preview status of Databricks Connect for Scala on Databricks Runtime 13.3 LTS and above.
* [Databricks Connect PySpark Shell](/concepts/databricks-connect-pyspark-shell.md) - A PySpark REPL (Spark shell) bundled with Databricks Connect for Python that connects to a remote Databricks cluster for interactive analysis.
* [Databricks Connect Python Environment Requirements](/concepts/databricks-connect-python-environment-requirements.md) - Local environment requires Python 3 with correct minor version matching the Databricks Runtime, especially for UDF compatibility.
* [Databricks Connect Python Version Matching for UDFs](/concepts/databricks-connect-python-version-matching-for-udfs.md) - When using user-defined functions (UDFs) with Databricks Connect, the local Python minor version must match the minor version on the target Databricks Runtime cluster or serverless compute.
* [Databricks Connect Query Interruption via interruptTag API](/concepts/databricks-connect-query-interruption-via-interrupttag-api.md) - Databricks Connect provides an interruptTag() API that allows users to tag DataFrames and then interrupt long-running queries by tag, useful for cost control.
* [Databricks Connect Query Timeout](/concepts/databricks-connect-query-timeout.md) - Long queries over 3600 seconds are not available in Databricks Connect for Databricks Runtime 13.3 LTS and below.
* [Databricks Connect Remote Execution Model](/concepts/databricks-connect-remote-execution-model.md) - The architectural pattern where logical representations of Spark commands (DataFrame operations) are sent to a remote Spark server on the Databricks cluster rather than executed locally.
* [Databricks Connect remote table operations](/concepts/databricks-connect-remote-table-operations.md) - The ability to read, create, query, and manage tables on a remote Databricks cluster using standard Spark DataFrame and SQL APIs through Databricks Connect.
* [Databricks Connect remote() method](/concepts/databricks-connect-remote-method.md) - The remote() method allows connecting to Databricks compute not attached to the notebook or job using explicit configuration
* [Databricks Connect Requirements](/concepts/databricks-connect-requirements.md) - Supported Databricks Runtime versions (7.3 LTS through 12.2 LTS), matching Python minor versions, matching databricks-connect package versions, and JRE 8.
* [Databricks Connect Resilient Query Execution](/concepts/databricks-connect-resilient-query-execution.md) - Databricks Connect (DBR 14.0+) automatically reconnects to running queries after network interruptions or OS pauses (up to 5 min), and supports longer query execution beyond the previous 1-hour limit.
* [Databricks Connect Serverless Compute](/concepts/databricks-connect-serverless-compute.md) - Feature allowing Databricks Connect to connect to serverless compute resources instead of classic clusters
* [Databricks Connect Serverless Compute Connection](/concepts/databricks-connect-serverless-compute-connection.md) - Configuration methods for connecting Databricks Connect to serverless compute instead of classic clusters, including environment variables, config profiles, and builder methods.
* [Databricks Connect Serverless Compute Support](/concepts/databricks-connect-serverless-compute-support.md) - Support for connecting Databricks Connect to serverless compute starting from version 15.1, with workspace-level prerequisites for serverless compute.
* [Databricks Connect Serverless UDF Limitations](/concepts/databricks-connect-serverless-udf-limitations.md) - UDF support on serverless compute always follows the initial corresponding minor release of Databricks Connect, with version compatibility defined in the official compatibility table.
* [Databricks Connect service disablement](/concepts/databricks-connect-service-disabling.md) - Method to disable Databricks Connect (Spark Connect) on a cluster by setting spark.databricks.service.server.enabled false in Spark configuration.
* [Databricks Connect Service Disabling](/concepts/databricks-connect-service-disabling.md) - How to disable the Databricks Connect (Spark Connect) service on a cluster using the spark.databricks.service.server.enabled Spark configuration property.
* [Databricks Connect setup and requirements](/concepts/databricks-connect-setup-and-requirements.md) - Prerequisites, installation steps, and compatibility requirements for using Databricks Connect in a local development environment.
* [Databricks Connect Spark Connect Connection String](/concepts/databricks-connect-spark-connect-connection-string.md) - Advanced method to connect Databricks Connect to a cluster using a Spark Connect connection string, either via the remote() function or the SPARK_REMOTE environment variable.
* [Databricks Connect Spark Connect Server](/concepts/databricks-connect-spark-connect-server.md) - Running Databricks Connect against an open source Spark Connect server instead of a Databricks cluster, with caveats about exclusive features
* [Databricks Connect SSL/TLS Certificate Configuration](/concepts/databricks-connect-ssltls-certificate-configuration.md) - Configuring custom SSL/TLS certificates for Databricks Connect by setting the GRPC_DEFAULT_SSL_ROOTS_FILE_PATH environment variable
* [Databricks Connect Streaming and Distributed ML Limitations](/concepts/databricks-connect-streaming-and-distributed-ml-limitations.md) - Streaming foreachBatch, distributed ML training, and ApplyInPandas/Cogroup with standard access mode are unsupported in certain Databricks Connect runtime versions.
* [Databricks Connect Support Lifecycle](/concepts/databricks-connect-support-lifecycle.md) - Databricks Connect follows the Databricks Runtime support lifecycles; end-of-support versions should be upgraded to supported versions.
* [Databricks Connect Table CRUD Operations](/concepts/databricks-connect-table-crud-operations.md) - Common patterns for reading, creating, querying, and dropping tables on a Databricks cluster through a remote Databricks Connect session.
* [Databricks Connect Table Operations](/concepts/databricks-connect-table-operations.md) - Code patterns for reading, creating, and managing tables in Databricks using Databricks Connect for Scala, including CRUD operations and SQL queries.
* [Databricks Connect Testing with pytest](/concepts/databricks-connect-testing-with-pytest.md) - How to write and run pytest tests for PySpark code using Databricks Connect, where tests execute on a remote Databricks cluster or serverless compute.
* [Databricks Connect Troubleshooting](/concepts/databricks-connect-troubleshooting.md) - Common issues encountered when using Databricks Connect such as Python version mismatches, conflicting PySpark installations, SPARK_HOME conflicts, serialization settings, and Windows-specific problems.
* [Databricks Connect troubleshooting and limitations](/concepts/databricks-connect-troubleshooting-and-limitations.md) - Known troubleshooting guidance and limitations when using Databricks Connect.
* [Databricks Connect tutorial (IntelliJ IDEA)](/concepts/databricks-connect-tutorial-intellij-idea.md) - A walkthrough guide for running Scala code from IntelliJ IDEA on classic Databricks compute using Databricks Connect.
* [Databricks Connect UDF Custom Library Restriction on Serverless Compute](/concepts/databricks-connect-udf-custom-library-restriction-on-serverless-compute.md) - On serverless compute with Databricks Connect for Runtime 16.3 and below, UDFs cannot include custom libraries.
* [Databricks Connect UDF Serialization](/concepts/databricks-connect-udf-serialization.md) - The mechanism by which locally-defined user-defined functions (UDFs) are serialized and transmitted to the remote Databricks cluster for execution, along with related APIs like foreach, foreachBatch, and transformWithState.
* [Databricks Connect Unsupported APIs and Libraries](/concepts/databricks-connect-unsupported-apis-and-libraries.md) - Several Spark APIs and third-party libraries are unsupported in Databricks Connect, including RDDs, Spark Context, data frame display, certain Databricks Utilities, and libraries like Mosaic, GraphFrames, and GreatExpectations.
* [Databricks Connect usage requirements](/concepts/databricks-connect-usage-requirements.md) - Prerequisites and compatibility requirements for workspace and local development environment to use Databricks Connect.
* [Databricks Connect usage workflow](/concepts/databricks-connect-usage-workflow.md) - The typical three-step workflow for getting started with Databricks Connect: confirming requirements, installing the package, and walking through a tutorial.
* [Databricks Connect Version Compatibility](/concepts/databricks-connect-version-compatibility.md) - The version alignment rules between Databricks Connect packages and Databricks Runtime versions, including support lifecycles and end-of-support policies.
* [Databricks Connect Version Pinning with Dot-Asterisk Notation](/concepts/databricks-connect-version-pinning-with-dot-asterisk-notation.md) - The recommended practice of specifying databricks-connect versions using X.Y.* notation to ensure the latest patch version is installed.
* [Databricks Connect Windows path space restriction](/concepts/databricks-connect-windows-path-space-restriction.md) - Databricks Connect fails on Windows when installed into a directory path containing spaces; workarounds include using short name forms or space-free paths.
* [Databricks Connect with Classic Compute](/concepts/databricks-connect-with-classic-compute.md) - Support for running code from IDEs like PyCharm on classic (non-serverless) Databricks compute clusters
* [Databricks Connect with Open Source Spark Connect Server](/concepts/databricks-connect-spark-connect-server.md) - Running Databricks Connect against a local open source Spark Connect server instead of a Databricks cluster, with the caveat that some Databricks-exclusive features may fail.
* [Databricks Connect with Serverless Compute](/concepts/databricks-connect-with-serverless-compute.md) - Support for running Python code on Databricks serverless compute via Databricks Connect
* [Databricks Connect Workaround for CREATE TABLE AS SELECT](/concepts/databricks-connect-workaround-for-create-table-as-select.md) - In Databricks Connect, CREATE TABLE AS SELECT is not supported; the workaround is to use spark.sql('SELECT ...').write.saveAsTable('table') instead.
* [Databricks Connect Workspace Requirements](/concepts/databricks-connect-workspace-requirements.md) - Prerequisites for using Databricks Connect including Unity Catalog enablement, compute version compatibility, and cluster access mode requirements.
* [Databricks Dashboard Customization and Sharing](/concepts/databricks-dashboard-customization-and-sharing.md) - User-editable parameters for the entire dashboard and individual charts, allowing customization of date range, data slices, models, and modifications to charts.
* [Databricks Dashboard Import/Export](/concepts/databricks-dashboard-importexport.md) - The ability to import and export Databricks dashboards as JSON files, enabling sharing and reproduction of dashboard definitions across workspaces.
* [Databricks Data Classification](/concepts/databricks-data-classification.md) - An AI agent in Unity Catalog that automatically classifies and tags sensitive data across tables using large language models and intelligent scanning.
* [Databricks Data Preparation for Distributed ML](/concepts/databricks-data-preparation-for-distributed-ml.md) - The overall workflow and best practices for preparing data specifically on the Databricks platform to support distributed machine learning training.
* [Databricks Data Profile (Data Profiling in Unity Catalog)](/concepts/databricks-data-profile-data-profiling-in-unity-catalog.md) - A feature in Databricks Unity Catalog that tracks statistical properties and quality metrics of Delta tables over time, supporting three profile types: TimeSeries, InferenceLog, and Snapshot.
* [Databricks Data Profile Metric Tables](/concepts/databricks-data-profile-metric-tables.md) - Unity Catalog tables created by data profiling that store computed statistics and can be queried via SQL or Catalog Explorer
* [Databricks Data Profile Refresh Management](/concepts/databricks-data-profile-refresh-management.md) - Creating, listing, and monitoring refresh runs for data profiles using the API
* [Databricks Data Profile Scheduling and Notifications](/concepts/databricks-profile-scheduling-and-notifications.md) - Configuring cron-based schedules and email notifications for data profile refreshes
* [Databricks Data Profiling](/concepts/databricks-data-profiling.md) - A feature in Databricks Unity Catalog that analyzes table statistics, distributions, and quality metrics to help users understand their data.
* [Databricks Data Profiling (API)](/concepts/databricks-data-profiling-api.md) - Creating and managing data profiles on Unity Catalog tables using the Databricks SDK or REST API to monitor data quality and distribution over time.
* [Databricks Data Profiling (Unity Catalog)](/concepts/databricks-data-profiling-unity-catalog.md) - API-driven data profiling service for Delta tables in Unity Catalog, supporting TimeSeries, InferenceLog, and Snapshot profile types
* [Databricks Data Profiling Dashboard](/concepts/databricks-data-profiling-dashboard.md) - An automatically generated, customizable dashboard in Databricks that visualizes profiling metric and drift results.
* [Databricks Data Profiling Metric Tables](/concepts/databricks-data-profiling-metric-tables.md) - Two metric tables (profile metrics and drift metrics) created when a data profile runs on a Databricks table, containing summary statistics and distribution-change tracking data.
* [Databricks Data Profiling SDK and API](/concepts/databricks-sdk-data-profiling-api.md) - Python SDK and REST API for creating and managing data profiles on Unity Catalog tables programmatically.
* [Databricks Default Storage vs Customer-Provisioned Managed Storage](/concepts/databricks-default-storage-vs-customer-provisioned-managed-storage.md) - Two flavors of managed storage: Databricks-provisioned default storage and customer-provided cloud storage; same lifecycle but different billing and post-purge retention behavior.
* [Databricks Delta Lake documentation](/concepts/databricks-delta-lake-documentation.md) - The Databricks-specific documentation and tutorials for using Delta Lake on the Databricks platform on AWS.
* [Databricks Designated Service with Geos](/concepts/databricks-designated-service-with-geos.md) - Foundation Model APIs is a Databricks Designated Service that uses Databricks Geos to manage data residency when processing customer content.
* [Databricks Development Workflow for Distributed ML](/concepts/databricks-development-workflow-for-distributed-ml.md) - Recommended methodology on Databricks for iteratively developing machine learning models by starting with single-node training before scaling to distributed execution
* [Databricks DevOps for ML](/concepts/databricks-devops-for-ml.md) - Production and automation capabilities including model serving, Jobs for scheduling, Git folders for version control, and infrastructure-as-code tools like Bundles and Terraform.
* [Databricks Disk Cache](/concepts/databricks-disk-cache.md) - Local caching layer on Databricks compute nodes that accelerates data access by storing frequently read data files in SSD-based local storage.
* [Databricks Distributed Deep Learning Deprecation](/concepts/databricks-distributed-deep-learning-deprecation.md) - Databricks deprecated Horovod and HorovodRunner after ML 15.4 LTS, recommending TorchDistributor for PyTorch and tf.distribute.Strategy for TensorFlow.
* [Databricks distributed training on AWS](/concepts/databricks-distributed-training-on-aws.md) - Reference architecture for running distributed TensorFlow training on Databricks clusters in AWS
* [Databricks Documentation Lifecycle](/concepts/databricks-documentation-lifecycle.md) - The practice of Databricks marking documentation as 'retired' when products or services are no longer supported or superseded, serving as an archival notice for users.
* [Databricks Documentation Structure for Model Serving](/concepts/databricks-documentation-structure-for-model-serving.md) - The URL path convention used by Databricks to organize model-serving documentation under /aws/en/machine-learning/model-serving/.
* [Databricks Embedding Model Querying](/concepts/databricks-embedding-model-querying.md) - How to write and send query requests to foundation models optimized for embeddings tasks via Databricks model serving endpoints.
* [Databricks Environment Configuration for MLflow Tracing](/concepts/databricks-environment-configuration-for-mlflow-tracing.md) - Required environment setup including DATABRICKS_HOST, DATABRICKS_TOKEN, and LLM provider API keys for MLflow Tracing with Smolagents
* [Databricks Error Class System](/concepts/databricks-error-class-system.md) - Databricks organizes runtime errors into named error classes with unique identifiers for structured error handling
* [Databricks Error Classes](/concepts/databricks-error-classes.md) - A system in Databricks that organizes error conditions into classes and sub-conditions, each with specific SQLSTATE codes and resolution guidance.
* [Databricks error classes pattern](/concepts/databricks-error-classes.md) - The structured error classification system used in Databricks, where error classes group multiple specific error conditions under a single identifier with SQLSTATE codes
* [Databricks Error Message Documentation](/concepts/databricks-error-message-documentation.md) - Databricks provides structured documentation for error conditions, including error classes and their arguments, to help users troubleshoot issues on the Databricks platform
* [Databricks Error Message Framework](/concepts/databricks-error-message-framework.md) - The structured error classification system used by Databricks that organizes runtime errors by class name (e.g., DELTA_UNIFORM_REFRESH_INVALID_ARGUMENT) with associated cause and resolution information.
* [Databricks Error Message Reference](/concepts/databricks-error-message-reference.md) - The curated collection of documented error conditions in Databricks, each with a unique error class, description, cause, and remediation
* [Databricks Feature Engineering Client](/concepts/databricks-feature-engineering-client.md) - Python API (FeatureEngineeringClient) for creating, managing, and storing feature tables in Unity Catalog for use in machine learning workflows.
* [Databricks Feature Engineering Compatibility Matrix](/concepts/feature-engineering-compatibility-matrix.md) - A reference mapping package versions (databricks-feature-engineering and databricks-feature-store) to Databricks Runtime ML versions.
* [Databricks Feature Engineering package installation](/concepts/databricks-feature-engineering-package-installation.md) - Instructions for installing databricks-feature-engineering via pip in Databricks Runtime or local environments
* [Databricks Feature Engineering Python API](/concepts/databricks-feature-engineering-client.md) - The Python client library (databricks-feature-engineering) for working with feature tables and online stores on Databricks
* [Databricks Feature Engineering Python Client](/concepts/databricks-feature-engineering-client.md) - The databricks-feature-engineering Python package that provides APIs for working with feature tables and online stores in Unity Catalog and Workspace Feature Store.
* [Databricks Feature Serving](/concepts/databricks-feature-serving.md) - A managed serving infrastructure that exposes Databricks feature data (from Unity Catalog) to external models and applications via high-availability, low-latency, auto-scaling endpoints.
* [Databricks Feature Store](/concepts/databricks-feature-store.md) - A managed feature store for batch and real-time serving of ML features with a single governed source of truth under Unity Catalog.
* [Databricks Feature Store Access Control (Legacy)](/concepts/feature-store-access-control-legacy.md) - Granular permission system for feature table metadata in Databricks workspaces not enabled for Unity Catalog
* [Databricks Feature Store for Covariate Lookups](/concepts/databricks-feature-store-for-covariate-lookups.md) - Using Databricks Feature Engineering Client to create and store feature tables that can be joined with primary training data in AutoML experiments.
* [Databricks Feature Store for Covariate Management](/concepts/databricks-feature-store-for-covariates.md) - Using the Databricks Feature Store to store, version, and serve covariate feature tables for AutoML forecasting
* [Databricks Feature Store for Covariates](/concepts/databricks-feature-store-for-covariates.md) - Using Databricks Feature Store (FeatureEngineeringClient) to register covariate feature tables that can be joined with primary training data for AutoML forecasting.
* [Databricks Feature Store Lineage](/concepts/databricks-feature-store-lineage.md) - Automatic tracking of relationships between feature tables, functions, and models used to build ML models in Databricks
* [Databricks Feature Store Workflow](/concepts/databricks-feature-store-workflow.md) - The end-to-end pipeline from raw data transformation through feature table creation, model training with feature logging, model registration, to batch and real-time inference.
* [Databricks Forecasting Advanced Options](/concepts/databricks-forecasting-advanced-options.md) - Additional configuration parameters for forecasting experiments including holiday region covariates, weight columns, split columns, training framework selection, and primary metrics.
* [Databricks Foundation Model Access Enforcement Tiers](/concepts/databricks-foundation-model-access-enforcement-tiers.md) - The three tiers of model access in Databricks — pay-per-token, batch inference (AI Functions), and provisioned throughput — each with different permission enforcement behaviors.
* [Databricks Foundation Model Agent Tracing Pattern](/concepts/databricks-foundation-model-agent-tracing-pattern.md) - A design pattern for building tool-calling agents with Databricks Foundation Models where MLflow Tracing automatically captures the full agent loop including model invocations, tool function execution (decorated with @mlflow.trace), and final responses.
* [Databricks Foundation Model API Pattern](/concepts/databricks-foundation-model-api-pattern.md) - The 'databricks:/model-name' URI pattern used to reference Databricks-hosted foundation models for LLM-as-a-judge scoring
* [Databricks Foundation Model API Tracing via MLflow](/concepts/databricks-foundation-model-api-tracing-via-mlflow.md) - Using MLflow's OpenAI autolog to trace calls to Databricks Foundation Model APIs by configuring an OpenAI client with Databricks host and token, then querying models like Llama 4 Maverick.
* [Databricks Foundation Model API Tracing via OpenAI Client](/concepts/databricks-foundation-model-api-tracing-via-mlflow.md) - Technique of using the OpenAI Python client configured with Databricks credentials and endpoints to query Databricks-hosted models (e.g., Llama 4 Maverick) while MLflow automatically traces those calls.
* [Databricks Foundation Model APIs](/concepts/databricks-foundation-model-apis.md) - Databricks-hosted foundation models made available through APIs that can be governed via Unity Catalog permissions as securable AI objects.
* [Databricks Foundation Model Endpoints](/concepts/databricks-foundation-model-endpoints.md) - Databricks-hosted model endpoints accessed via 'databricks:/' URIs for use as evaluation judges
* [Databricks Function Calling](/concepts/databricks-function-calling.md) - An OpenAI-compatible mechanism on Databricks that lets LLMs generate structured JSON function-call outputs based on user-defined tool schemas, without the model executing the functions itself.
* [Databricks Function Calling Limitations](/concepts/databricks-function-calling-limitations.md) - Current Public Preview restrictions: single-turn optimization for most models, no parallel function calling, max 32 functions, no support on existing provisioned-throughput endpoints, and no 'id' field for Gemini models.
* [Databricks Function Calling Workflow](/concepts/databricks-function-calling-workflow.md) - The four-step sequence for using function calling on Databricks: submit query with tools, model returns JSON arguments, parse and execute function, append results for final summary.
* [Databricks Gateway Model as Judge](/concepts/databricks-gateway-model-as-judge.md) - Using Databricks-hosted LLM models (e.g., databricks-gpt-5-mini) as the evaluator model for scoring LLM outputs
* [Databricks GenAI Agent Evaluation Workflow](/concepts/databricks-genai-agent-evaluation-workflow.md) - End-to-end workflow for evaluating GenAI agents on Databricks, including creating agents, defining judges, building datasets, and running evaluations.
* [Databricks GenAI notebook development](/concepts/databricks-genai-notebook-development.md) - A development workflow pattern where Databricks notebooks serve as the IDE for building GenAI applications with built-in MLflow experiment tracking, auto-linked experiments, and inline trace visualization.
* [Databricks GenAI Tracing for Local Models](/concepts/databricks-genai-tracing-for-local-models.md) - Pattern for tracing locally-run Ollama models within Databricks by setting the MLflow tracking URI to 'databricks' and enabling OpenAI autologging on serverless compute clusters.
* [Databricks Genie Code](/concepts/databricks-genie-code.md) - An AI assistant within Databricks that performs automated exploratory data analysis, natural language AutoML, code generation for data preparation and feature pipelines, and diagnosis of model serving endpoints.
* [Databricks Genie Code for Deep Learning](/concepts/genie-code-for-deep-learning.md) - An AI assistant feature that supports deep learning workloads on AI Runtime by generating training code, resolving library errors, suggesting optimizations, and debugging issues.
* [Databricks Geos and Data Residency](/concepts/databricks-geos-and-data-residency.md) - Databricks' framework for managing data residency when processing customer content, applicable to Foundation Model APIs as a Designated Service.
* [Databricks Git Folders and CI/CD Integration](/concepts/databricks-git-folders-for-cicd.md) - Git integration in the Databricks workspace enabling version control, code sharing, and automated CI/CD pipelines with tools like GitHub Actions and Jenkins.
* [Databricks Git Folders for CI/CD](/concepts/databricks-git-folders-for-cicd.md) - Built-in Git integration in the Databricks workspace that enables version control, code sharing, and automated CI/CD workflows via APIs and third-party tools.
* [Databricks Inbound IP Allowlisting](/concepts/databricks-inbound-ip-allowlisting.md) - The practice of adding Databricks-managed IP addresses to a firewall allowlist for the provider's cloud and region to enable SecureConnect access
* [Databricks Lakebase Autoscaling](/concepts/databricks-lakebase-autoscaling.md) - The underlying infrastructure for Online Feature Stores—a Lakebase Autoscaling project that provisions highly available managed compute for real-time feature serving.
* [Databricks Legacy Feature Store (databricks-feature-store)](/concepts/databricks-legacy-feature-store-databricks-feature-store.md) - The deprecated legacy Python client package (databricks-feature-store) for workspace-level feature store operations, superseded by databricks-feature-engineering as of version 0.17.0.
* [Databricks LLM Endpoint Benchmarking Methodology](/concepts/llm-endpoint-benchmarking-methodology.md) - Databricks recommends maximizing throughput within a given latency budget using a benchmarking harness that measures total latency, throughput, and the throughput-latency curve across varying concurrency levels.
* [Databricks LLM Serving Endpoint Autoscaling](/concepts/databricks-llm-serving-endpoint-autoscaling.md) - Databricks' strategy for dynamically scaling LLM serving endpoints that balances latency and throughput, where throughput plateaus at provisioned capacity limits as concurrency increases.
* [Databricks LLM Serving Endpoints](/concepts/databricks-llm-serving-endpoints.md) - An OpenAI-compatible API for accessing Databricks-hosted LLMs (e.g., Claude Sonnet) using Databricks credentials, enabling seamless integration with MLflow evaluation.
* [Databricks LLM Serving Endpoints via OpenAI SDK](/concepts/databricks-llm-serving-endpoints.md) - Using the OpenAI Python SDK client configured with Databricks credentials to call Databricks-hosted LLMs (e.g., Claude Sonnet) as serving endpoints.
* [Databricks ML Deprecation Policy](/concepts/databricks-ml-deprecation-policy.md) - The practice of removing pre-installed packages like Horovod after specific ML runtime versions (15.4 LTS ML and later).
* [Databricks ML Lifecycle Platform](/concepts/databricks-ml-platform.md) - A unified platform that spans the full data science and machine learning lifecycle from raw data ingestion through feature engineering, model training, deployment, and production monitoring, integrating open-source frameworks with enterprise governance and MLOps tools.
* [Databricks ML Platform](/concepts/databricks-ml-platform.md) - Unified platform for the full data science and machine learning lifecycle on Databricks, integrating open-source frameworks with enterprise MLOps tooling.
* [Databricks MLflow 3 for GenAI](/concepts/databricks-mlflow-3-for-genai.md) - Databricks' MLflow 3 platform tailored for Generative AI development, providing tracing, evaluation with LLM-as-a-judge, human feedback collection, and production monitoring capabilities.
* [Databricks MLflow Agent Evaluation Workflow](/concepts/databricks-mlflow-agent-evaluation-workflow.md) - End-to-end workflow on Databricks for evaluating Claude agent responses using MLflow datasets, custom scorers, and experiment tracking.
* [Databricks MLflow Credential Propagation](/concepts/databricks-mlflow-credential-propagation.md) - Method for passing Databricks MLflow tracking credentials to remote Ray workers using get_databricks_env_vars and environment variable updates.
* [Databricks MLflow Experiment Setup](/concepts/databricks-mlflow-experiment-setup.md) - Required setup: install mlflow-tracing, create a GenAI Experiment in Databricks, configure authentication via environment variables, and initialize the SDK with a tracking URI and experiment ID.
* [Databricks MLflow GenAI Evaluation Framework](/concepts/databricks-mlflow-genai-evaluation-framework.md) - A comprehensive evaluation framework on Databricks that includes built-in LLM judges, custom judge creation, and the mlflow.genai.evaluate() API for running evaluations.
* [Databricks MLflow OTLP Endpoint](/concepts/databricks-mlflow-otlp-endpoint.md) - A Databricks REST API endpoint at /api/2.0/otel/v1/traces that ingests OpenTelemetry Protocol (OTLP) spans and routes them to a specified Unity Catalog table via the X-Databricks-UC-Table-Name header.
* [Databricks MLflow PySpark ML autologging](/concepts/mlflow-pyspark-ml-autologging.md) - The replacement for deprecated MLlib automated MLflow tracking, enabled by default in Databricks Runtime 10.2 ML and above via mlflow.pyspark.ml.autolog() and Databricks Autologging.
* [Databricks MLflow REST API 2.0](/concepts/databricks-mlflow-rest-api-20.md) - Databricks' managed version of the MLflow API, documented separately from the open-source reference
* [Databricks MLflow Serverless Workloads](/concepts/databricks-mlflow-serverless-workloads.md) - Serverless workloads managed by MLflow on Databricks, including scheduled scorers, synthetic evaluation set generation, and agent evaluation.
* [Databricks MLflow Tracking Integration](/concepts/databricks-mlflow-tracking-integration.md) - Configuration to route MLflow experiment data to a Databricks workspace using mlflow.set_tracking_uri('databricks') and mlflow.set_experiment().
* [Databricks MLlib MLflow Tracking Deprecation](/concepts/databricks-mllib-mlflow-tracking-deprecation.md) - MLlib automated MLflow tracking is deprecated on Databricks Runtime 10.1 ML and above, disabled by default from 10.2 ML onwards, with mlflow.pyspark.ml.autolog() as the recommended replacement.
* [Databricks MLlib Support](/concepts/databricks-mllib-support.md) - Databricks supports Apache Spark MLlib on serverless, standard, and dedicated compute, with integration for Python, Scala, Java, and R.
* [Databricks MLOps Stacks](/concepts/databricks-mlops-stacks.md) - Templates for enabling automated, repeatable model promotion from development to production using infrastructure-as-code, forming part of Databricks' MLOps and governance suite.
* [Databricks model endpoint references in evaluators](/concepts/databricks-model-endpoint-references-in-evaluators.md) - The pattern of referencing Databricks-hosted foundation models (e.g., databricks-gpt-5-mini) via the 'databricks:/' prefix as the evaluation model for Phoenix scorers in the MLflow GenAI evaluation framework.
* [Databricks Model Registry with Unity Catalog](/concepts/databricks-model-registry-with-unity-catalog.md) - A centralized model repository integrated with Unity Catalog providing governance, cross-workspace access, lineage tracking, and model discovery for managing the model deployment lifecycle.
* [Databricks Model Retirement Policy](/concepts/databricks-model-retirement-policy.md) - The overall policy governing how and when Databricks notifies customers about model retirements, covering transition periods, retirement dates, and migration recommendations.
* [Databricks Model Serving](/concepts/databricks-model-serving.md) - Databricks' real-time model serving solution, distinct from batch inference, for deploying models for low-latency predictions
* [Databricks Model Serving Acceptable Use Policy](/concepts/databricks-model-serving-acceptable-use-policy.md) - A policy framework governing which machine learning models can be served on Databricks Model Serving, though the source page contains no substantive policy content.
* [Databricks Model Serving Data Protection](/concepts/databricks-model-serving-data-protection.md) - Security controls for Model Serving including logical isolation per customer, AES-256 encryption at rest, TLS 1.2+ in transit, no training on user data, and temporary retention of logs and metrics.
* [Databricks Model Serving Endpoint](/concepts/databricks-model-serving-endpoint.md) - A REST API endpoint (POST /api/2.0/serving-endpoints) for deploying and managing machine learning models for inference on Databricks.
* [Databricks Model Serving Endpoint Load Testing](/concepts/databricks-model-serving-endpoint-load-testing.md) - Process of configuring and running load tests on Databricks Model Serving endpoints to optimize performance, latency, and throughput
* [Databricks Model Serving Endpoint Load Testing Methodology](/concepts/databricks-model-serving-endpoint-load-testing-methodology.md) - A step-by-step process for configuring, running, and analyzing load tests on Databricks custom model serving endpoints using Locust.
* [Databricks Model Serving Endpoints](/concepts/databricks-model-serving-endpoints.md) - The underlying serving infrastructure on Databricks that hosts Foundation Model APIs, supporting both pay-per-token and provisioned throughput endpoint modes.
* [Databricks Model Serving for computer vision](/concepts/databricks-model-serving-for-computer-vision.md) - Databricks supports deploying trained computer vision models (e.g., YOLO11n) to production using Model Serving after training on AI Runtime.
* [Databricks Model Serving Foundation Model APIs (FMAPI)](/concepts/databricks-model-serving-foundation-model-apis-fmapi.md) - A Databricks model serving capability for foundation models that is explicitly incompatible with express deployments, which require custom models instead.
* [Databricks Model Serving Monitoring](/concepts/databricks-model-serving-monitoring.md) - Overview of monitoring tools available for model quality and endpoint health on Databricks Model Serving.
* [Databricks Model Serving on AWS](/concepts/databricks-model-serving-on-aws.md) - A Databricks platform service for deploying and serving machine learning models on AWS infrastructure.
* [Databricks Model Training](/concepts/databricks-model-training.md) - A unified platform for training and deploying ML and AI models through AutoML and Foundation Model Fine-tuning workloads
* [Databricks model training with MLlib](/concepts/databricks-model-training-with-mllib.md) - Using Apache Spark MLlib on Databricks for binary classification, decision trees, GBT regression, structured streaming, and custom transformers.
* [Databricks model training with scikit-learn and MLflow](/concepts/databricks-model-training-with-scikit-learn-and-mlflow.md) - End-to-end machine learning on Databricks using scikit-learn with Unity Catalog, MLflow tracking, and automated hyperparameter tuning via Hyperopt/Optuna.
* [Databricks model training with XGBoost](/concepts/databricks-model-training-with-xgboost.md) - Training XGBoost models on Databricks using Python, PySpark, and Scala, supporting both single-node and distributed workloads.
* [Databricks Model URI for Scorers](/concepts/databricks-model-uri-for-scorers.md) - The use of databricks:/ model URIs (e.g., databricks:/databricks-gpt-5-mini) to specify LLM judge models for Phoenix scorers within Databricks-managed MLflow environments.
* [Databricks NLP Platform](/concepts/databricks-nlp-platform.md) - Databricks' overall support for natural language processing tasks using open-source libraries (Spark ML, Spark NLP, Hugging Face) and proprietary integrations (John Snow Labs), enabling distributed NLP workloads.
* [Databricks notebook code modularization](/concepts/databricks-notebook-code-modularization.md) - The practice of organizing reusable code across notebooks using %run and workspace files to encapsulate functions and logic.
* [Databricks notebook integration for MLflow](/concepts/databricks-notebook-integration-for-mlflow.md) - Databricks notebooks automatically connect to the MLflow server, support notebook and workspace experiments, enable autologging for MLflow tracking, and provide an inline tracing UI for GenAI interactive analysis.
* [Databricks notebook requirement for scorer registration](/concepts/databricks-notebook-requirement-for-scorer-registration.md) - Scorer functions used in production monitoring must be defined and registered from a Databricks notebook because the monitoring service serializes function code for remote execution.
* [Databricks notebook serialization requirement for scorers](/concepts/databricks-notebook-serialization-requirement-for-scorers.md) - Custom scorer functions used in production monitoring must be defined and registered from a Databricks notebook because the monitoring service requires the notebook environment for code serialization.
* [Databricks Notebook Widgets for Configuration](/concepts/databricks-notebook-widgets-for-configuration.md) - Using dbutils.widgets.text to create parameterized notebook inputs for Unity Catalog paths and model names, enabling reusable training workflows.
* [Databricks OIDC Profile Portal](/concepts/databricks-oidc-profile-portal.md) - A portal URL provided by the Databricks provider where recipients download an Iceberg profile file containing catalog endpoint, share name, and placeholders for OAuth credentials.
* [Databricks on AWS](/concepts/databricks-on-aws.md) - Databricks platform deployed on Amazon Web Services infrastructure for data analytics and machine learning
* [Databricks on AWS Platform](/concepts/databricks-on-aws.md) - Databricks' unified analytics platform deployed on Amazon Web Services infrastructure.
* [Databricks Online Feature Store](/concepts/databricks-online-feature-store.md) - A high-performance, low-latency service for serving ML feature data to real-time applications, powered by Databricks Lakebase.
* [Databricks Online Feature Store Cost Optimization](/concepts/databricks-online-feature-store-cost-optimization.md) - Best practices for managing costs including reusing online stores across projects, right-sizing capacity from CU_2, and deleting unused stores to avoid continuous charges.
* [Databricks Online Tables](/concepts/databricks-online-tables.md) - Read-only, serverless copies of Delta Tables stored in row-oriented format, optimized for low-latency online access from Model Serving, Feature Serving, and RAG applications.
* [Databricks Online Tables (Legacy)](/concepts/databricks-online-tables.md) - Serverless, read-only copies of Delta Tables stored in row-oriented format, optimized for low-latency online access and designed for use with Model Serving, Feature Serving, and RAG applications.
* [Databricks Open Source ML Ecosystem](/concepts/databricks-open-source-ml-ecosystem.md) - Databricks' full support for the open-source ML ecosystem including frameworks like scikit-learn, PyTorch, TensorFlow, and Hugging Face, with open formats for artifacts (MLflow), governance (Unity Catalog APIs), and storage (Delta Lake).
* [Databricks OpenAI Client for Embeddings](/concepts/databricks-openai-client-for-embeddings.md) - A Databricks-specific OpenAI-compatible client library (databricks_openai) used to send embedding requests to Databricks-hosted models using the OpenAI embeddings API format.
* [Databricks OpenAI Client for Foundation Models](/concepts/databricks-openai-client-for-foundation-models.md) - The databricks-openai package provides an OpenAI-compatible API client for calling Databricks-hosted Foundation Model endpoints, enabling standard OpenAI SDK patterns against Databricks models.
* [Databricks OpenAI client integration with MLflow](/concepts/databricks-openai-client-integration-with-mlflow.md) - Using Databricks-hosted OpenAI-compatible models (via DatabricksOpenAI client) within MLflow prompt optimization workflows for prediction and evaluation.
* [Databricks OpenSharing](/concepts/databricks-opensharing.md) - A Databricks feature that enables organizations to share data across platforms using Delta Sharing protocols
* [Databricks OpenSharing Audit Log System Table](/concepts/databricks-opensharing-audit-log-system-table.md) - The system.access.audit table stores OpenSharing audit events, accessible to account admins and authorized users when system tables are enabled.
* [Databricks OTLP Trace Endpoint](/concepts/databricks-otlp-trace-endpoint.md) - The /api/2.0/otel/v1/traces endpoint that accepts OpenTelemetry traces via a BatchSpanProcessor and OTLPSpanExporter, routing them to a Unity Catalog table via the X-Databricks-UC-Table-Name header.
* [Databricks Personal Access Token (PAT) Authentication](/concepts/databricks-personal-access-token-pat-authentication.md) - An authentication method for connecting local development tools to Databricks workspaces, using DATABRICKS_TOKEN and DATABRICKS_HOST environment variables.
* [Databricks personal access token authentication for R](/concepts/databricks-connect-authentication-for-r.md) - The only supported authentication method for Databricks Connect for R, requiring a PAT stored in the DATABRICKS_TOKEN environment variable.
* [Databricks Pipeline Task for Jobs](/concepts/databricks-pipeline-task-for-jobs.md) - Using the Pipeline task type in Databricks Jobs to include a Lakeflow Spark Declarative Pipeline as a step in a multi-task job workflow.
* [Databricks Pipelines REST API Update Endpoint](/concepts/databricks-pipelines-rest-api-update-endpoint.md) - The REST API endpoint at /api/2.0/pipelines/{pipeline_id}/updates used to trigger pipeline updates programmatically from external orchestrators.
* [Databricks Premium Plan Requirement](/concepts/databricks-premium-plan-requirement.md) - The pricing tier prerequisite stating that Hive metastore table access control requires the Premium plan or above on Databricks.
* [Databricks Profile Metrics 30-Day Lookback Window](/concepts/databricks-profile-metrics-30-day-lookback-window.md) - For TimeSeries and InferenceLog profiles, metrics look back 30 days from profile creation time, which may cause partial windows (e.g., mid-week/month boundaries) in the first analysis.
* [Databricks Profile Notifications](/concepts/databricks-profile-notifications.md) - Configurable email notifications that alert users when a data profiling refresh fails or times out.
* [Databricks Profile Scheduling and Notifications](/concepts/databricks-profile-scheduling-and-notifications.md) - Configuring cron-based schedules and email notifications for automated data profile refreshes in Databricks.
* [Databricks Profile Types: Snapshot, TimeSeries, InferenceLog](/concepts/databricks-profile-types-snapshot-timeseries-inferencelog.md) - Three analysis types exist: Snapshot (single point-in-time window), TimeSeries (windows based on timestamp granularities), and InferenceLog (adds model accuracy, fairness, and bias statistics).
* [Databricks Ray-Spark Integration](/concepts/databricks-ray-spark-integration.md) - Databricks provides native integration for running Ray and Spark in the same environment, including utilities like databricks.ray.data.from_spark for seamless data passing between the frameworks.
* [Databricks Release Stages: Public Preview vs Beta](/concepts/databricks-release-stages-public-preview-vs-beta.md) - Databricks distinguishes between Public Preview (feature-complete and ready for broader adoption) and Beta (still under active development) release stages for its AI Runtime capabilities.
* [Databricks Release Tiers: Public Preview vs Beta](/concepts/databricks-release-tiers-public-preview-vs-beta.md) - AI Runtime features are split across two release stages: single-node serverless GPU tasks are in Public Preview, while the distributed multi-GPU training API remains in Beta.
* [Databricks release types for ML features](/concepts/databricks-release-types-for-ml-features.md) - The distinction between Public Preview and Beta release stages for Databricks machine learning features, indicating maturity and support levels
* [Databricks Release Types: Public Preview and Beta](/concepts/databricks-release-types-public-preview-and-beta.md) - Databricks categorizes features by maturity: Public Preview (generally available for evaluation) and Beta (still under active development, for early adopters).
* [Databricks Runtime 13.3 LTS](/concepts/databricks-runtime-133-lts.md) - The minimum supported Databricks Runtime version (13.3 LTS and above) for the current generation of Databricks Connect for Scala.
* [Databricks Runtime 14.0 ML](/concepts/databricks-runtime-140-ml.md) - The ML runtime version on Databricks that introduced support for the DeepSpeed library, enabling advanced distributed training capabilities.
* [Databricks Runtime Compatibility](/concepts/databricks-runtime-compatibility.md) - Version compatibility requirements between Databricks Runtime and Databricks Connect client packages.
* [Databricks Runtime for Machine Learning](/concepts/databricks-runtime-for-machine-learning.md) - A pre-built runtime environment for deep learning that includes TensorFlow, PyTorch, Keras, pre-configured GPU support, and integrated MLflow tracking.
* [Databricks Runtime for Machine Learning (Databricks Runtime ML)](/concepts/databricks-runtime-for-machine-learning.md) - A pre-configured Databricks runtime environment that automates compute resource creation with built-in machine learning and deep learning libraries and tools.
* [Databricks Runtime for Machine Learning (ML)](/concepts/databricks-runtime-for-machine-learning-ml.md) - A pre-configured Databricks runtime that includes PyTorch and other ML libraries
* [Databricks Runtime for ML](/concepts/databricks-runtime-for-ml.md) - Pre-configured compute clusters for machine learning on Databricks, bundling libraries like scikit-learn, XGBoost, MLflow, and deep learning frameworks.
* [Databricks Runtime ML](/concepts/databricks-runtime-ml.md) - A Databricks runtime variant that includes pre-installed popular machine learning and deep learning libraries for ML workloads.
* [Databricks Runtime ML deprecation of Hyperopt](/concepts/databricks-runtime-ml-deprecation-of-hyperopt.md) - Hyperopt removed from Databricks Runtime for Machine Learning after version 16.4 LTS ML
* [Databricks Runtime ML Deprecation Policy](/concepts/databricks-runtime-ml-deprecation-policy.md) - Databricks Runtime for Machine Learning drops Hyperopt after version 16.4 LTS ML, reflecting a shift toward Optuna and RayTune.
* [Databricks Runtime ML for Deep Learning](/concepts/databricks-runtime-ml-for-deep-learning.md) - Databricks Runtime ML includes deep learning libraries such as TensorFlow and PyTorch, and supports featurization at scale by distributing computation across clusters.
* [Databricks Runtime ML for DeepSpeed](/concepts/databricks-runtime-ml-support.md) - Databricks Runtime 14.0 ML or above is required to use the DeepSpeed library and distributor for distributed training.
* [Databricks Runtime ML for Distributed Training](/concepts/databricks-runtime-ml-for-distributed-training.md) - Databricks Runtime 13.0 ML or above as the required environment for running TorchDistributor with Spark 3.4
* [Databricks Runtime ML for XGBoost](/concepts/databricks-runtime-ml-for-xgboost.md) - The pre-configured Databricks Runtime ML environment that includes XGBoost libraries and dependencies for Scala-based training.
* [Databricks Runtime ML Library Continuity Options](/concepts/databricks-runtime-ml-library-continuity-options.md) - Options available to customers after a library is removed from Databricks Runtime ML pre-installation: manually install the library or use an earlier runtime version.
* [Databricks Runtime ML Library Deprecation Policy](/concepts/databricks-runtime-ml-library-deprecation-policy.md) - The criteria and process by which Databricks removes pre-installed libraries from Databricks Runtime ML, including conditions like inactivity, low usage, or replacement.
* [Databricks Runtime ML Library Removal Notification](/concepts/databricks-runtime-ml-library-removal-notification.md) - The multi-channel notification process Databricks uses to inform customers before removing a library, including release notes, import warnings, and documentation updates.
* [Databricks Runtime ML model training examples](/concepts/databricks-runtime-ml-model-training-examples.md) - Databricks provides notebook examples for training models including hyperparameter tuning documentation within the Databricks Runtime ML ecosystem
* [Databricks Runtime ML Mosaic Streaming Integration](/concepts/databricks-runtime-ml-mosaic-streaming-integration.md) - Mosaic Streaming is pre-installed in Databricks Runtime 15.2 ML and higher; authentication for Unity Catalog volumes requires setting DATABRICKS_HOST and DATABRICKS_TOKEN environment variables, especially for distributed training with TorchDistributor.
* [Databricks Runtime ML on AWS Graviton](/concepts/databricks-runtime-ml-on-aws-graviton.md) - Support for Graviton instance types starting from Databricks Runtime 15.4 LTS ML, offering improved performance for Spark, Photon, feature engineering, and ML libraries like XGBoost and LightGBM with better price-to-performance.
* [Databricks Runtime ML Support](/concepts/databricks-runtime-ml-support.md) - DeepSpeed is available in Databricks Runtime 14.0 ML and above, enabling distributed training with DeepSpeed on the Databricks platform.
* [Databricks Runtime ML Top-tier Libraries](/concepts/databricks-runtime-ml-top-tier-libraries.md) - A designated subset of pre-installed ML/DL libraries in Databricks Runtime ML that receive faster updates, advanced testing, and embedded optimizations.
* [Databricks Runtime Support Lifecycles](/concepts/databricks-connect-support-lifecycle.md) - The policy that Databricks Connect follows Databricks Runtime support lifecycles, with end-of-support versions requiring upgrades to supported versions.
* [Databricks Runtime Support Lifecycles for Databricks Connect](/concepts/databricks-connect-support-lifecycle.md) - End-of-support policy where Databricks Connect versions follow Databricks Runtime support lifecycles, requiring upgrades when versions reach end-of-support.
* [Databricks Runtime Version Alignment for Model Deployment](/concepts/databricks-runtime-version-alignment-for-model-deployment.md) - The recommendation to run inference on the same Databricks Runtime version used during training, leveraging the saved databricks_runtime field in MLmodel metadata to ensure dependency compatibility.
* [Databricks Runtime Version Compatibility](/concepts/databricks-runtime-version-compatibility.md) - Annotations (DBR tags) indicating which Databricks Runtime ML version introduced each parameter, helping users determine feature availability.
* [Databricks S3 Commit Service and Multi-cluster Writes](/concepts/databricks-s3-commit-service-and-multi-cluster-writes.md) - Databricks provides an S3 commit service to support multi-cluster writes within a single workspace; certain features like SSE-C and STS-dependent credential paths are unsupported in this mode.
* [Databricks SDK Authentication Methods](/concepts/databricks-sdk-authentication-methods.md) - Multiple ways to authenticate the WorkspaceClient: hard-coded credentials, configuration profiles, or environment variables (DATABRICKS_HOST and DATABRICKS_TOKEN).
* [Databricks SDK Data Profiling API](/concepts/databricks-sdk-data-profiling-api.md) - The Python SDK and REST API for creating, refreshing, viewing, and deleting data profiles on Unity Catalog tables, including support for scheduling and notifications.
* [Databricks SDK for Python Authentication Methods](/concepts/databricks-sdk-authentication-methods.md) - Three ways to authenticate the WorkspaceClient: hard-coded host/token, configuration profiles, or environment variables (DATABRICKS_HOST and DATABRICKS_TOKEN). The SPARK_REMOTE environment variable is not supported.
* [Databricks SDK for Python Integration](/concepts/databricks-sdk-for-python-integration.md) - The Databricks SDK for Python, included with Databricks Connect, provides not only dbutils access via WorkspaceClient but also full access to any Databricks REST API at both workspace and account levels.
* [Databricks SDK for Python REST API Access](/concepts/databricks-sdk-for-python-rest-api-access.md) - The included Databricks SDK for Python can be used to access any available Databricks REST API, beyond just the dbutils utilities
* [Databricks SDK serving endpoint query](/concepts/databricks-sdk-serving-endpoint-query.md) - Using the Databricks Python SDK's WorkspaceClient.serving_endpoints_data_plane.query() method to directly query route-optimized endpoints with service principal credentials.
* [Databricks Secret Scopes](/concepts/databricks-secret-scopes.md) - A secure storage mechanism for credentials that can be referenced by model serving endpoints using the {{secrets/scope/key}} syntax
* [Databricks Secret Scopes for Model Serving](/concepts/databricks-secret-scopes-for-model-serving.md) - Creating and managing secret scopes (via CLI or API) that model serving endpoints read from at serving time to populate secrets-based environment variables.
* [Databricks Secrets for ML Model Credentials](/concepts/databricks-secrets-for-ml-model-credentials.md) - A pattern for securely injecting Hugging Face access tokens (or other sensitive credentials) into distributed training workloads by storing them as Databricks secrets and referencing them in the workload YAML rather than hard-coding them in scripts.
* [Databricks Secrets for Online Store Credentials](/concepts/databricks-secrets-for-online-store-credentials.md) - Storing read-only and read-write credentials for third-party online stores in Databricks secret scopes, organized by prefix-based naming conventions for DynamoDB and SQL stores.
* [Databricks Secrets for Third-Party Online Store Credentials](/concepts/databricks-secrets-for-online-store-credentials.md) - Storing read-only and read-write credentials (access keys, passwords, usernames) in Databricks secret scopes with a consistent prefix naming scheme for authenticating to online stores.
* [Databricks Security Perimeter](/concepts/databricks-security-perimeter.md) - A security boundary within which Databricks hosts third-party foundation model endpoints, ensuring data does not leave the Databricks-controlled environment.
* [Databricks Serverless Budget Policy 403 Error](/concepts/databricks-serverless-budget-policy-403-error.md) - A 403 Forbidden error (PERMISSION_DENIED) that occurs when no fallback budget policy is available and no policy is manually set on an MLflow experiment.
* [Databricks Serverless Forecasting](/concepts/databricks-serverless-forecasting.md) - A fully-managed AutoML forecasting capability on Databricks that automatically selects algorithms and hyperparameters for time-series prediction using serverless compute.
* [Databricks Serverless Forecasting with AutoML](/concepts/databricks-serverless-forecasting.md) - A UI-driven feature in Databricks that automates time series forecasting by selecting optimal algorithms and hyperparameters on fully-managed compute resources.
* [Databricks Serverless GPU](/concepts/databricks-serverless-gpu.md) - A serverless compute environment on Databricks that provides on-demand GPU resources (e.g., 8×H100) for distributed training workloads without requiring manual cluster management.
* [Databricks Serverless GPU Cluster](/concepts/databricks-serverless-gpu-cluster.md) - On-demand GPU compute infrastructure on Databricks used for training deep learning forecasting models
* [Databricks serverless GPU clusters](/concepts/databricks-serverless-gpu-clusters.md) - A compute environment on Databricks (AWS) that enables serverless GPU-accelerated workloads for classic machine learning tasks
* [Databricks Serverless GPU Compute](/concepts/databricks-serverless-gpu.md) - Databricks managed compute service that automatically scales GPU resources and handles provisioning, data distribution, and synchronization for distributed training workloads.
* [Databricks Serverless GPU Compute Configuration](/concepts/databricks-serverless-gpu-compute-configuration.md) - The workflow to configure a Databricks notebook with serverless GPU compute, including selecting accelerator type (8xH100) and environment (AI v5)
* [Databricks Serverless GPU Compute for LLM Training](/concepts/databricks-serverless-gpu-compute-for-llm-training.md) - Databricks AI Runtime environment providing serverless GPU compute with A10 accelerators, pre-installed with Unsloth and deep learning libraries for fine-tuning large language models.
* [Databricks Serverless GPU Compute for ML](/concepts/databricks-serverless-gpu-compute-for-ml.md) - A Databricks compute environment (e.g., 1xA10 GPU) optimized for running GPU-accelerated machine learning workloads with Unity Catalog integration.
* [Databricks Serverless GPU Distributed Training](/concepts/databricks-serverless-gpu-distributed-training.md) - A Databricks compute paradigm that provisions on-demand GPU clusters (e.g., 8xH100) for distributed training using the @distributed decorator, handling data parallelism automatically.
* [Databricks Serverless GPU for Deep Learning](/concepts/databricks-serverless-gpu-for-deep-learning.md) - A cloud compute environment on Databricks that provides on-demand GPU acceleration (e.g., 1xA10) for training deep learning models without managing infrastructure.
* [Databricks Serverless GPU with @distributed Decorator](/concepts/databricks-serverless-gpu-with-distributed-decorator.md) - Databricks serverless GPU compute with a Python decorator (`@distributed`) that transparently launches training across multiple H100 GPUs on a single node.
* [Databricks Spark Configuration for MLflow Tracking](/concepts/databricks-spark-configuration-for-mlflow-tracking.md) - Spark configuration flags spark.databricks.mlflow.trackMLlib.enabled and spark.databricks.mlflow.autologging.enabled used to enable or disable legacy MLlib automated MLflow tracking in Databricks Runtime 10.2 ML and above.
* [Databricks Spark Job Project Format](/concepts/databricks-spark-job-project-format.md) - A specialized MLflow Project type for Databricks Spark Jobs clusters (Runtime 13.0+), using databricks_spark_job section with python_file or entry_points settings and python_libraries dependencies.
* [Databricks SQL Alert Configuration for Anomaly Detection](/concepts/databricks-sql-alert-configuration-for-anomaly-detection.md) - A method for creating anomaly detection alerts by querying the system.data_quality_monitoring.table_results system table with custom SQL, supporting advanced filtering, trigger conditions, and custom notification templates.
* [Databricks SQL Alert on Anomaly Detection Results](/concepts/databricks-sql-alert-on-anomaly-detection-results.md) - Method for creating advanced alerts by querying the system.data_quality_monitoring.table_results system table with Databricks SQL, supporting custom filtering and notification templates.
* [Databricks SQL Alerts](/concepts/databricks-sql-alerts.md) - The underlying alerting framework used to create profile alerts, supporting scheduled evaluation of queries and multi-channel notifications.
* [Databricks SQL Anomaly Alert Configuration](/concepts/databricks-sql-anomaly-alert-configuration.md) - An approach to configure anomaly detection alerts by querying the system.data_quality_monitoring.table_results system table with custom SQL, supporting advanced filtering and custom notification templates.
* [Databricks SQL Views for Trace Data](/concepts/databricks-sql-views-for-trace-data.md) - Databricks SQL views (trace_unified and trace_metadata) that transform OpenTelemetry table data into MLflow format for SQL-based trace querying.
* [Databricks SQLSTATE 0AKDC](/concepts/databricks-sqlstate-0akdc.md) - A SQLSTATE code in Databricks indicating a 'feature not supported' condition, associated with the DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE error class.
* [Databricks SQLSTATE KD00C](/concepts/databricks-sqlstate-kd00c.md) - The SQLSTATE class KD00C that categorizes datasource-specific errors within Databricks, including the DELTA_VERSIONS_NOT_CONTIGUOUS error.
* [Databricks table history operations](/concepts/databricks-table-history-operations.md) - Broader topic covering how to work with Delta table history, including DESCRIBE HISTORY and related tools, documented separately in the Databricks guide.
* [Databricks Token-Based Authentication for External API Calls](/concepts/databricks-token-based-authentication-for-external-api-calls.md) - Using a Databricks personal access token (DATABRICKS_TOKEN) as a Bearer token in HTTP headers to authenticate requests made to Databricks serving endpoints from external clients or SDKs.
* [Databricks Trace Archival](/concepts/databricks-trace-archival.md) - Mechanism for persisting MLflow traces and their assessments to Unity Catalog Delta tables for long-term storage and advanced analysis.
* [Databricks Unified ML Platform](/concepts/databricks-ml-platform.md) - A single, integrated platform covering the full data science and machine learning lifecycle from raw data ingestion through feature engineering, model training, deployment, and production monitoring.
* [Databricks Unity Catalog Data Classification](/concepts/databricks-unity-catalog-data-classification.md) - The data classification subsystem within Unity Catalog that uses system tags to identify and label sensitive data across tables.
* [Databricks Utilities (dbutils) via Connect](/concepts/databricks-utilities-dbutils-via-connect.md) - Accessing dbutils.fs and dbutils.secrets through Databricks Connect, including file copying between local and remote filesystems and secret management.
* [Databricks Utilities (dbutils) via Databricks Connect](/concepts/databricks-utilities-dbutils-via-connect.md) - The supported subset of Databricks Utilities commands accessible through Databricks Connect, including dbutils.fs for file operations and dbutils.secrets for secrets management.
* [Databricks Utilities Availability via Databricks Connect](/concepts/limited-dbutils-availability-in-databricks-connect.md) - Only dbutils.fs (file system) and dbutils.secrets are available through WorkspaceClient.dbutils in Databricks Connect, not the full set of Databricks Utilities
* [Databricks Utilities for Scala (dbutils-scala)](/concepts/databricks-utilities-for-scala-dbutils-scala.md) - A Scala library (com.databricks:databricks-dbutils-scala_2.12) that provides access to DBFS and secrets utilities via DBUtils.getDBUtils in Databricks Connect for Scala projects.
* [Databricks Utilities for Scala library](/concepts/databricks-utilities-for-scala-library.md) - A published library (com.databricks/databricks-dbutils-scala_2.12) that provides Scala bindings for Databricks Utilities, with a dependency automatically declared by Databricks Connect for Scala.
* [Databricks Utilities with Databricks Connect](/concepts/databricks-utilities-with-databricks-connect.md) - Using Databricks Utilities (dbutils) within a Databricks Connect session for Python.
* [Databricks Utilities with Unity Catalog Volumes](/concepts/databricks-utilities-with-unity-catalog-volumes.md) - Using the DBUtils.fs API in Scala to programmatically create, read, and delete files in Unity Catalog volume paths within Databricks Connect projects.
* [Databricks Visual Studio Code Extension](/concepts/databricks-visual-studio-code-extension.md) - A VS Code extension that uses Databricks Connect to provide built-in debugging of user code on Databricks compute.
* [Databricks vs OSS MLflow Search Differences](/concepts/databricks-vs-oss-mlflow-search-differences.md) - Databricks-managed MLflow shares most search query syntax with open-source MLflow but has field-level differences in supported filters and capabilities.
* [Databricks Workspace Feature Store (Deprecated)](/concepts/databricks-workspace-feature-store-deprecated.md) - The legacy databricks-feature-store Python package, deprecated as of version 0.17.0, with all modules moved to databricks-feature-engineering.
* [Databricks Workspace Feature Store (Legacy)](/concepts/databricks-workspace-feature-store-legacy.md) - A legacy feature store UI in Databricks on AWS for managing, searching, and tracking feature tables used in machine learning workflows.
* [Databricks Workspace Feature Store UI](/concepts/workspace-feature-store-ui.md) - A user interface within Databricks on AWS for managing, searching, and governing feature tables used in machine learning workflows.
* [Databricks Workspace-level Table Access Control Setting](/concepts/databricks-workspace-level-table-access-control-setting.md) - An admin setting on the Security tab of the Databricks workspace settings page that must be enabled before users can configure Python and SQL table access control on clusters.
* [databricks-feature-store (legacy) deprecation](/concepts/databricks-feature-store-legacy-deprecation.md) - The legacy databricks-feature-store package is deprecated as of version 0.17.0 in favor of databricks-feature-engineering
* [databricks-gpt-5-mini Model](/concepts/databricks-gpt-5-mini-model.md) - A Databricks-hosted GPT model used as the judge/grader for Arize Phoenix scorers in MLflow evaluations
* [Databricks-hosted Claude model integration](/concepts/databricks-hosted-claude-model-integration.md) - Use of Databricks-hosted Claude models (via databricks:/ paths) as judges or generation models within MLflow GenAI evaluation workflows.
* [Databricks-hosted foundation models](/concepts/databricks-hosted-foundation-models.md) - State-of-the-art open models hosted and managed by Databricks, available as preconfigured endpoints through Foundation Model APIs for pay-per-token and provisioned throughput modes.
* [Databricks-hosted foundation models via OpenAI client](/concepts/databricks-hosted-foundation-models-via-openai-client.md) - The databricks_openai.DatabricksOpenAI client allows calling Databricks-hosted LLMs (e.g., Claude Sonnet 4) using the standard OpenAI SDK interface.
* [Databricks-Hosted GPT-5 Foundation Models](/concepts/databricks-hosted-gpt-5-foundation-models.md) - A family of pay-per-token foundation model endpoints (databricks-gpt-5 through databricks-gpt-5-5-pro) hosted on Databricks that support the OpenAI Responses API with various model sizes and capabilities.
* [Databricks-Hosted Judge LLM](/concepts/databricks-hosted-judge-llm.md) - The default LLM used by built-in judges in Databricks, designed specifically for performing GenAI quality assessments and accessible via Databricks serving endpoints.
* [Databricks-hosted LLM for Judges](/concepts/databricks-hosted-llm-for-judges.md) - The default LLM model provided by Databricks that powers built-in GenAI quality assessment judges, with the option to switch to other models via the 'model' argument.
* [Databricks-hosted LLM Serving](/concepts/databricks-hosted-llm-serving.md) - Using Databricks-hosted foundation models (e.g., Claude Sonnet) via OpenAI-compatible API endpoints with Databricks credentials for GenAI application development.
* [Databricks-hosted LLMs](/concepts/databricks-hosted-llms.md) - Large Language Models served directly from Databricks workspaces, accessible via the databricks-openai library which provides a compatible OpenAI client interface for models like databricks-claude-sonnet-4.
* [Databricks-hosted LLMs with MLflow](/concepts/databricks-hosted-llms-with-mlflow.md) - Integration of Databricks-hosted foundation models (via the databricks-openai package) with MLflow Tracing and Evaluation for GenAI applications.
* [Databricks-hosted LLMs with OpenAI-compatible API](/concepts/databricks-hosted-llms-with-openai-compatible-api.md) - Using the databricks-openai package to create an OpenAI-compatible client that connects to Databricks-hosted foundation models for GenAI application development.
* [Databricks-hosted MLflow tracking server](/concepts/databricks-hosted-mlflow-tracking-server.md) - Default managed MLflow tracking server provided by Databricks that stores experiment data in the workspace with no additional setup required.
* [Databricks-Managed MLflow](/concepts/databricks-managed-mlflow.md) - Databricks' managed version of the open-source MLflow platform providing experiment tracking, model registry, and audit trails integrated with Unity Catalog and Git.
* [Databricks-to-Databricks OpenSharing Protocol](/concepts/databricks-to-databricks-opensharing-protocol.md) - A Databricks-managed secure data sharing protocol that does not require credential files, relying instead on Unity Catalog metastore identifiers for establishing connections between provider and recipient workspaces.
* [Databricks-to-Databricks Sharing](/concepts/databricks-to-databricks-sharing.md) - A sharing model where the recipient must be a Databricks user with Unity Catalog; the provider uses the recipient's metastore identifier to create a secure connection.
* [Databricks-to-Databricks Sharing Model](/concepts/databricks-to-databricks-sharing.md) - A sharing model where both the data provider and recipient use Databricks workspaces with Unity Catalog, using a metastore sharing identifier to establish a secure connection.
* [Databricks-to-Databricks View Sharing](/concepts/databricks-to-databricks-view-sharing.md) - A Delta Sharing mode that allows sharing views between Databricks workspaces while restricting SQL functions for security
* [Databricks-to-Databricks vs Databricks-to-Open Sharing](/concepts/databricks-to-databricks-vs-databricks-to-open-sharing.md) - Two OpenSharing protocols: Databricks-to-Databricks (DATABRICKS auth) for cross-Databricks sharing, and Databricks-to-Open (TOKEN/OAUTH/OIDC auth) for sharing with non-Databricks recipients.
* [Databricks-to-Open Sharing](/concepts/databricks-to-open-sharing.md) - A sharing model where the recipient can use any tool (including Databricks) to access shared data via an activation URL, portal link, credential file, or OIDC federation.
* [Databricks-to-Open Sharing (Bearer Token Authentication)](/concepts/databricks-to-open-sharing-bearer-token-authentication.md) - An authentication flow in Delta Sharing where data providers create recipient objects using bearer tokens for non-Databricks users to access shared data without a Unity Catalog workspace.
* [Databricks-to-Open Sharing Model](/concepts/databricks-to-open-sharing.md) - A sharing model where recipients can use any tool (including Databricks) to access shared data via an activation URL or portal link to download a credential file or URL.
* [Databricks-to-Open Sharing Protocol](/concepts/databricks-to-open-sharing-protocol.md) - Delta Sharing protocol variant on Databricks that enables data providers to share data with recipients using the open-source Delta Sharing standard, supporting features like IP access lists and SecureConnect.
* [Databricks-to-Open Sharing vs Databricks-to-Databricks Sharing](/concepts/databricks-to-open-sharing-vs-databricks-to-databricks-sharing.md) - Distinction between sharing data to recipients without a Unity Catalog-enabled Databricks workspace (Open Sharing) and recipients with such a workspace, with OIDC federation being an alternative to bearer tokens for the former.
* [databricks.automl.classify](/concepts/databricksautomlclassify.md) - AutoML method that trains classification models, returning an AutoMLSummary with trial notebooks and model metrics.
* [databricks.automl.forecast](/concepts/databricksautomlforecast.md) - AutoML method that trains forecasting models with required time_col, frequency, horizon, and optional identity_col for time series data.
* [databricks.automl.import_notebook](/concepts/databricksautomlimport_notebook.md) - Method to import a notebook saved as an MLflow artifact, returning an ImportNotebookResult with path and URL.
* [databricks.automl.regress](/concepts/databricksautomlregress.md) - AutoML method that trains regression models, returning an AutoMLSummary with per-trial notebooks and metrics; primary metric defaults to r2.
* [databricks.migrations.migrate_traces_to_uc](/concepts/databricksmigrationsmigrate_traces_to_uc.md) - A Python SDK module providing the run() function used to perform the migration from a source MLflow experiment to a target Unity Catalog-backed experiment.
* [DatabricksSession](/concepts/databrickssession.md) - The primary entry point for creating a SparkSession when using Databricks Connect, bridging local Python code to a remote Databricks compute environment.
* [DatabricksSession API](/concepts/databrickssession.md) - The primary entry-point class in Databricks Connect that creates a SparkSession bound to a remote Databricks cluster.
* [DatabricksSession API Portability](/concepts/databrickssession-api-portability.md) - Databricks Connect APIs are available in Databricks notebooks as part of the runtime, enabling seamless code transitions
* [DatabricksSession Authentication for DBUtils](/concepts/databrickssession-authentication-for-dbutils.md) - Authentication for the Databricks Utilities for Scala library is determined through initialization of the DatabricksSession class in a Databricks Connect for Scala project.
* [DatabricksSession authentication for Scala Utilities](/concepts/databrickssession-authentication-for-dbutils.md) - Authentication for the Databricks Utilities for Scala library is determined through initializing the DatabricksSession class in the Databricks Connect project for Scala.
* [DatabricksSession behavior differences](/concepts/databrickssession-behavior-differences.md) - The behavior of DatabricksSession.builder.getOrCreate() and create() differs between local development environments and Databricks workspace notebooks/jobs, particularly regarding default session handling and error behavior.
* [DatabricksSession Builder API](/concepts/databrickssession-builder-api.md) - The programmatic API (remote(), profile(), serverless(), sdkConfig(), getOrCreate()) for configuring and initializing a Databricks Connect session.
* [DatabricksSession builder configuration chain](/concepts/databrickssession-builder-configuration-chain.md) - The ordered precedence of configuration methods for DatabricksSession: remote() method, config profile, DATABRICKS_CONFIG_PROFILE env var, individual env vars, and DEFAULT profile.
* [DatabricksSession Builder Configuration Methods](/concepts/databrickssession-builder-configuration-methods.md) - The DatabricksSession class provides builder methods (remote(), profile(), sdkConfig(), serverless(), getOrCreate()) to configure connections to Databricks compute.
* [DatabricksSession Builder Patterns](/concepts/databrickssession-builder-patterns.md) - The getOrCreate() and create() methods on DatabricksSession.builder behave differently depending on whether code runs in a local IDE (Databricks Connect) or in the Databricks workspace (notebooks/jobs).
* [DatabricksSession class](/concepts/databrickssession-class.md) - The class used to initialize authentication for Databricks Connect projects in Scala, which also determines authentication for the Databricks Utilities for Scala library.
* [DatabricksSession initialization for authentication](/concepts/databrickssession-initialization-for-authentication.md) - Authentication for the Databricks Utilities for Scala library is determined through initializing the DatabricksSession class in a Databricks Connect for Scala project.
* [DatabricksSession local development behavior](/concepts/databrickssession-local-development-behavior.md) - In local IDE environments, getOrCreate() reuses existing sessions while create() always creates new sessions with connection params from code, env vars, or config files
* [DatabricksSession remote() Method](/concepts/databrickssession-remote-method.md) - A method that allows DatabricksSession to connect to Databricks compute not attached to the current notebook or job by specifying configuration kwargs or individual configuration methods like host() or token().
* [DatabricksSession workspace behavior](/concepts/databrickssession-workspace-behavior.md) - In Databricks notebooks/jobs, getOrCreate() returns the default pre-configured Spark session, while create() requires explicit connection parameters
* [DatabricksSession Workspace vs Local Behavior](/concepts/databrickssession-workspace-vs-local-behavior.md) - Key behavioral differences of DatabricksSession.getOrCreate() and create() when run in a local IDE versus a Databricks notebook or job, including default session handling and connection parameter requirements.
* [DatabricksSession.getOrCreate() vs create()](/concepts/databrickssessiongetorcreate-vs-create.md) - Behavioral differences between getOrCreate() and create() methods when running locally vs in Databricks workspace
* [DataCollatorWithPadding](/concepts/datacollatorwithpadding.md) - A Hugging Face data collator that dynamically pads input sequences to the longest sequence in each batch during training, providing a memory-efficient alternative to static padding.
* [DataOps on Databricks](/concepts/dataops-on-databricks.md) - Practice of applying CI/CD principles to data pipelines within the Databricks platform, ensuring reliable and secure data for ML models.
* [Dataset Lineage Tracking](/concepts/dataset-lineage-tracking.md) - The mechanism for tracking provenance of evaluation dataset records using source fields that identify whether a record came from a human, a document, or a production trace.
* [Dataset Record Tagging](/concepts/dataset-record-tagging.md) - The ability to add and edit metadata tags on individual evaluation dataset records through the MLflow UI for organization and filtering.
* [Dataset Size Estimation with load_dataset_builder](/concepts/dataset-size-estimation-with-load_dataset_builder.md) - Using Hugging Face's load_dataset_builder to inspect download and dataset sizes before committing to a large download
* [Dataset Source Lineage](/concepts/dataset-source-lineage.md) - The source field in evaluation datasets for tracking record provenance via human, document, or trace origin types.
* [Dataset Synchronization](/concepts/dataset-synchronization.md) - A mechanism to sync collected Expectations from labeling sessions to Evaluation Datasets via an intelligent upsert operation, using trace inputs as unique keys.
* [db_repl and brickster Package](/concepts/db_repl-and-brickster-package.md) - The brickster R package provides a db_repl function as an alternative to sparklyr for using full Spark MLlib capabilities within Databricks.
* [DBFS Access Patterns for Shared Clusters](/concepts/dbfs-access-patterns-for-shared-clusters.md) - DBFS FUSE paths fail on shared clusters in Unity Catalog; alternatives include using dbutils, spark readers, or Unity Catalog Volumes.
* [DBFS Migration to Unity Catalog Volumes](/concepts/dbfs-migration-to-unity-catalog-volumes.md) - The process of migrating raw files, notebooks, scripts, and init scripts from DBFS root and mounts to governed Unity Catalog volumes and workspace files.
* [DBFS root and DBFS mounts deprecation](/concepts/dbfs-root-and-dbfs-mounts-deprecation.md) - DBFS root storage and DBFS mounts will no longer be available in new workspaces created after September 30, 2026, requiring migration of files to volumes and external locations.
* [DBFS root location](/concepts/dbfs-root-location.md) - A sibling path under a workspace's default storage path that does not create a conflict when used as an external location, as it is not the same as the internal storage path.
* [dbt Python Models on Databricks](/concepts/dbt-python-models-on-databricks.md) - Running dbt Python models (beta, requires dbt >= 1.3) as part of a Databricks job, noting that SQL warehouses are not supported for Python models and all-purpose compute must be used.
* [dbt Task Artifact and Output Retrieval](/concepts/dbt-task-artifact-and-output-retrieval.md) - Retrieving dbt task output, inline logs, and archived artifacts (logs, results, manifests, configuration) programmatically via the Databricks CLI or Jobs API.
* [dbt Task Configuration via UI and API](/concepts/dbt-task-configuration-via-ui-and-api.md) - Step-by-step configuration of a dbt task in the Databricks Jobs UI and via the Jobs API, including source repo, commands, SQL warehouse, compute, and environment settings.
* [dbt Task in Databricks Jobs](/concepts/dbt-task-in-databricks-jobs.md) - Running dbt Core projects as a task within Databricks Lakeflow Jobs, enabling automation, monitoring, multi-step workflows, and artifact archiving.
* [dbt-databricks Adapter Package](/concepts/dbt-databricks-adapter-package.md) - The recommended dbt adapter package for Databricks, a fork of dbt-spark optimized for Databricks, used to connect dbt Core to Databricks compute and SQL warehouses.
* [dbutils.fs and dbutils.secrets in Databricks Connect](/concepts/dbutilsfs-and-dbutilssecrets-in-databricks-connect.md) - The only two Databricks Utilities accessible through the dbutils variable in Databricks Connect for Python
* [dbutils.fs via Databricks Connect](/concepts/dbutilsfs-via-databricks-connect.md) - The file system utility from Databricks Utilities accessible through WorkspaceClient.dbutils.fs for file operations on Databricks volumes.
* [dbutils.fs Volume Operations Example](/concepts/dbutilsfs-volume-operations-example.md) - A concrete Python example demonstrating file operations (put, head, rm) on Unity Catalog volumes using dbutils.fs via Databricks Connect's WorkspaceClient.
* [DBUtils.getDBUtils](/concepts/dbutilsgetdbutils.md) - The Scala API entry point in the Databricks Utilities for Scala library used to access DBFS and secrets functionality within Databricks Connect projects.
* [DBUtils.getDBUtils for Scala](/concepts/dbutilsgetdbutils.md) - The method to access Databricks Utilities (DBFS and secrets) in Scala via Databricks Connect
* [DBUtils.getDBUtils in Databricks Connect for Scala](/concepts/dbutilsgetdbutils.md) - The entry point for accessing Databricks Utilities (DBFS and Secrets) in Scala-based Databricks Connect projects via the DBUtils.getDBUtils() method from the com.databricks.sdk.scala.dbutils package.
* [DBUtils.getDBUtils() method](/concepts/dbutilsgetdbutils.md) - The entry point method in the dbutils-scala library that returns a DBUtils instance for accessing Databricks File System (DBFS) and secrets utilities within a Databricks Connect for Scala project.
* [dbutils.notebook.run()](/concepts/dbutilsnotebookrun.md) - An API method to call a Databricks notebook as a separate job, supporting parameter passing via widgets and return values via dbutils.notebook.exit().
* [dbutils.secrets via Databricks Connect](/concepts/dbutilssecrets-via-databricks-connect.md) - The secrets utility from Databricks Utilities accessible through WorkspaceClient.dbutils.secrets for managing sensitive information.
* [DDP (Distributed Data Parallel) on Databricks](/concepts/ddp-distributed-data-parallel-on-databricks.md) - A parallelism technique for multi-GPU training that replicates the model on each GPU and synchronizes gradients across devices.
* [DDP Applicability Criteria](/concepts/ddp-applicability-criteria.md) - Conditions under which DDP is appropriate: model fits in single GPU memory, scaling data throughput is the goal, and simplest distributed approach is desired
* [DDP Gradient Checkpointing for Distributed LLM Training](/concepts/ddp-gradient-checkpointing-for-distributed-llm-training.md) - The requirement to use non-reentrant gradient checkpointing (use_reentrant=False) when combining Unsloth with Distributed Data Parallel (DDP) to avoid variable-ready errors during multi-GPU training.
* [DDP gradient checkpointing with non-reentrant mode](/concepts/ddp-gradient-checkpointing-with-non-reentrant-mode.md) - A distributed training fix required when combining PyTorch DDP with gradient checkpointing to avoid 'mark a variable ready only once' errors.
* [Debugging Databricks Connect R Code in RStudio](/concepts/debugging-databricks-connect-r-code-in-rstudio.md) - The ability to set breakpoints and debug R code that runs against Databricks clusters through Databricks Connect within RStudio Desktop.
* [Decision Framework: DDP vs FSDP vs DeepSpeed](/concepts/decision-framework-ddp-vs-fsdp-vs-deepspeed.md) - Guidelines for choosing between DDP (simpler use cases), FSDP (PyTorch-native large model training), and DeepSpeed (advanced memory optimization with ZeRO stages) on Databricks.
* [Declarative Automation Bundles](/concepts/declarative-automation-bundles.md) - A collection of source files that serves as the end-to-end definition of a Databricks project, including ML resources like workspaces and pipelines defined as code for version control and testing.
* [Declarative Automation Bundles (Databricks)](/concepts/declarative-automation-bundles-databricks.md) - Databricks' recommended tool for building and deploying Scala JARs, providing correct Scala, JDK, and Databricks Connect versions pre-configured for serverless compute.
* [Declarative Automation Bundles for Scala JARs](/concepts/declarative-automation-bundles-databricks.md) - Databricks recommends Declarative Automation Bundles over manual JAR building; bundles handle correct version configuration, templating, and easy deployment to the workspace.
* [Declarative Feature Engineering API](/concepts/declarative-feature-engineering-api.md) - A Python-based API for defining, registering, and using machine learning features declaratively (vs. imperatively) with Unity Catalog on Databricks.
* [Declarative Feature Engineering APIs](/concepts/declarative-feature-engineering-apis.md) - A Databricks framework for defining and computing ML features using declarative objects rather than imperative data pipelines
* [Declarative Feature Lifecycle Workflows](/concepts/declarative-feature-lifecycle-workflows.md) - The end-to-end workflow covering feature development, model training, and feature materialization/serving with declarative APIs
* [Declarative Feature Limitations and Constraints](/concepts/declarative-feature-limitations-and-constraints.md) - Key constraints of declarative features including restrictions on entity column types, supported functions, source column uniqueness, and request source capabilities
* [Declarative Feature Serving Limitation](/concepts/declarative-feature-serving-limitation.md) - Feature Serving endpoints are not supported for Declarative Feature Engineering; models must be deployed via model serving endpoints using models logged through Unity Catalog.
* [Declarative Features Limitations and Constraints](/concepts/declarative-features-limitations-and-constraints.md) - Known limitations of the declarative feature APIs including entity column type restrictions, RequestSource scalar-only support, globally unique column names across sources, and limited UDAF function support.
* [Declarative YAML Job Configuration for ML Training](/concepts/declarative-yaml-job-configuration-for-ml-training.md) - The pattern of defining ML training jobs declaratively in YAML configuration files that can be checked into source control
* [Decorator Ordering Best Practices](/concepts/decorator-ordering-best-practices.md) - When combining @mlflow.trace with other decorators (e.g., from web frameworks), @mlflow.trace must be the outermost decorator to ensure complete observability of the function's execution.
* [Decorator Ordering for @mlflow.trace](/concepts/decorator-ordering-for-mlflowtrace.md) - The rule that @mlflow.trace must be the outermost decorator when combined with other decorators to ensure complete observability of function execution.
* [Decorator ordering with @mlflow.trace](/concepts/decorator-ordering-for-mlflowtrace.md) - The @mlflow.trace decorator must be the outermost decorator when combined with other decorators to ensure complete observability of function execution.
* [Decoupled Client-Server Architecture](/concepts/decoupled-client-server-architecture.md) - The architectural pattern underlying Databricks Connect where the client runs general code locally while Spark DataFrame operations are executed remotely on Databricks compute via Spark Connect.
* [Dedicated Access Mode for Databricks Runtime ML](/concepts/dedicated-access-mode-for-databricks-runtime-ml.md) - A required access mode for Unity Catalog access on Databricks Runtime ML, automatically set when selecting Machine Learning compute, which can be scoped to a single user or group.
* [Dedicated Access Mode for ML Compute](/concepts/dedicated-access-mode-for-ml-compute.md) - Databricks Runtime ML requires Dedicated access mode for Unity Catalog data access, assigning compute to a single user or group with automatic permission down-scoping, with fine-grained access control available from Runtime 15.4 LTS ML.
* [Dedicated access mode with fine-grained access control](/concepts/dedicated-access-mode-with-fine-grained-access-control.md) - Restrictions and caveats when querying tables with row filters or column masks from dedicated access compute resources, including serverless FGAC enforcement and Cloud Fetch storage implications.
* [Deep Clone](/concepts/deep-clone.md) - A cloning strategy that creates a complete, independent copy of a source table, including all data and metadata.
* [Deep Clone (Databricks)](/concepts/deep-clone.md) - A complete, independent copy of a source Delta, Iceberg, or Parquet table, copying both data and metadata to a target location.
* [Deep Clone (Delta Lake)](/concepts/deep-clone.md) - A complete, independent copy of a source Delta, Iceberg, or Parquet table, including both data and metadata.
* [Deep Clone (Delta Table)](/concepts/deep-clone.md) - A complete, independent copy of a source Delta, Iceberg, or Parquet table, including both data and metadata.
* [Deep Clone Limitations for Streaming Tables](/concepts/deep-clone-limitations-for-streaming-tables.md) - Streaming tables have specific restrictions when used as the source of a deep clone operation
* [Deep Clone of Streaming Tables](/concepts/deep-clone-of-streaming-tables.md) - The operation of creating a deep copy of a Delta streaming table, which has specific constraints and requirements in Databricks
* [Deep clone restrictions for streaming tables](/concepts/deep-clone-restrictions-for-streaming-tables.md) - Databricks imposes multiple restrictions on deep cloning streaming tables: no custom LOCATION, only default publishing mode, WITH HISTORY required, no scheduled tables, no time travel.
* [Deep Clone Restrictions on Streaming Tables](/concepts/deep-clone-restrictions-on-streaming-tables.md) - A set of constraints preventing deep clone operations on Delta streaming tables, including location, architecture, history, scheduling, and time travel limitations.
* [Deep learning based recommender systems](/concepts/deep-learning-based-recommender-systems.md) - Notebook examples for building recommendation systems on Databricks AI Runtime using modern deep learning approaches like two-tower models.
* [Deep Learning Based Recommender Systems on Databricks](/concepts/deep-learning-based-recommender-systems-on-databricks.md) - The overall pattern of building modern recommendation systems using deep learning approaches within the Databricks platform, leveraging AI Runtime and GPU acceleration.
* [Deep learning best practices on Databricks](/concepts/deep-learning-best-practices-on-databricks.md) - General guidelines and optimization strategies for developing and fine-tuning deep learning models on the Databricks platform.
* [Deep Learning Data Loading with Delta Lake](/concepts/deep-learning-data-loading-with-delta-lake.md) - Best practices for using Delta Lake tables and streaming approaches (PyTorch IterableDataset, Hugging Face datasets, Ray Data) to optimize data throughput for deep learning workloads on Databricks.
* [Deep Learning Inference on Databricks](/concepts/deep-learning-inference-on-databricks.md) - Strategies for online, batch, and streaming inference including Model Serving, Pandas UDFs, and considerations for CPU vs GPU selection.
* [Deep Learning on Databricks](/concepts/deep-learning-on-databricks.md) - Overview of using PyTorch, TensorFlow, and distributed training to develop and fine-tune deep learning models on the Databricks platform.
* [Deep Learning Performance Optimization on Databricks](/concepts/deep-learning-performance-optimization-techniques.md) - Techniques including early stopping, batch size tuning, transfer learning, and GPU monitoring to optimize deep learning training performance.
* [Deep Learning Performance Optimization Techniques](/concepts/deep-learning-performance-optimization-techniques.md) - Key optimization strategies for deep learning on Databricks including early stopping, batch size tuning (with learning rate adjustment by sqrt(n)), transfer learning, and GPU scheduling to maximize utilization.
* [Deep Learning Performance Optimization Techniques on Databricks](/concepts/deep-learning-performance-optimization-techniques.md) - Techniques including early stopping, batch size tuning, learning rate adjustment, and transfer learning to optimize deep learning model training on Databricks
* [Deep Learning Recommender Systems (Databricks)](/concepts/deep-learning-recommender-systems-databricks.md) - Examples for building recommendation systems using deep learning approaches like two-tower models on AI Runtime
* [Deep learning recommender systems on Databricks](/concepts/deep-learning-recommender-systems-databricks.md) - Examples for building recommendation systems using modern deep learning approaches like two-tower models on AI Runtime.
* [Deep Learning Training Best Practices on Databricks](/concepts/deep-learning-training-best-practices-on-databricks.md) - Key training optimization techniques including early stopping, batch size tuning with learning rate adjustment, transfer learning, and moving to distributed training.
* [DeepAR](/concepts/deepar.md) - A recurrent neural network (RNN) model for probabilistic forecasting that produces prediction intervals alongside point forecasts.
* [DeepAR Estimator](/concepts/deepar.md) - A probabilistic forecasting model in GluonTS using recurrent neural networks (RNNs) to produce prediction intervals and point forecasts for time series data.
* [DeepAR Model](/concepts/deepar-model.md) - A probabilistic forecasting model from GluonTS used for time-series prediction of electricity consumption
* [DeepAR Probabilistic Forecasting Model](/concepts/deepar-probabilistic-forecasting-model.md) - A neural network-based model from GluonTS that produces probabilistic forecasts for time series data, demonstrated in an electricity consumption forecasting workflow.
* [DeepEval Evaluation Framework](/concepts/deepeval-evaluation-framework.md) - A comprehensive evaluation framework for LLM applications that provides metrics for RAG systems, agents, conversational AI, and safety evaluation.
* [DeepEval Framework](/concepts/deepeval-framework.md) - A comprehensive evaluation framework for LLM applications that provides metrics for RAG systems, agents, conversational AI, and safety evaluation.
* [DeepEval Integration with MLflow](/concepts/deepeval-integration-with-mlflow.md) - MLflow integrates with DeepEval so that DeepEval metrics can be used as scorers for LLM evaluation
* [DeepEval Metric Categories](/concepts/deepeval-metric-categories.md) - DeepEval provides five categories of metrics: RAG metrics, Agentic metrics, Conversational metrics, Safety metrics, and Non-LLM metrics.
* [DeepEval Scorer API](/concepts/deepeval-scorer-api.md) - Python API for invoking DeepEval scorers directly or via mlflow.genai.evaluate(), with threshold, model and metric-specific parameters
* [DeepEval Scorer Categories](/concepts/deepeval-metric-categories.md) - DeepEval provides five categories of scorers: RAG metrics, agentic metrics, conversational metrics, safety metrics, and non-LLM metrics.
* [DeepEval Scorer Configuration](/concepts/deepeval-scorer-configuration.md) - DeepEval scorers accept metric-specific keyword parameters including model, threshold, include_reason, window_size, and strict_mode for customizing evaluation behavior.
* [DeepEval Scorer Configuration Parameters](/concepts/deepeval-scorer-configuration.md) - DeepEval scorers accept metric-specific parameters like threshold, model, include_reason, window_size, and strict_mode to customize evaluation behavior.
* [DeepEval Scorers Integration](/concepts/deepeval-scorers-integration.md) - MLflow integration with DeepEval offering the broadest metric coverage across RAG, agents, conversational AI, and safety including agent plan quality and multi-turn conversation completeness
* [DeepEval-MLflow Integration](/concepts/deepeval-mlflow-integration.md) - MLflow natively integrates DeepEval scorers for evaluating LLM applications within the MLflow evaluation framework.
* [DeepSpeed](/concepts/deepspeed.md) - A deep learning optimization library by Microsoft that provides advanced memory optimization techniques for distributed training of large models
* [DeepSpeed Distributor](/concepts/deepspeed-distributor.md) - A Databricks distributed training solution built on TorchDistributor, using Microsoft's DeepSpeed library for optimized memory usage, reduced communication overhead, and advanced pipeline parallelism.
* [DeepSpeed Distributor (Databricks)](/concepts/deepspeed-distributor.md) - A Databricks solution for distributed PyTorch training built on top of TorchDistributor, using the DeepSpeed library to overcome GPU memory constraints and scale large model training.
* [DeepSpeed Distributor on Databricks](/concepts/deepspeed-distributor.md) - A Microsoft open-source library integrated with Databricks for distributed training with optimized memory usage, reduced communication overhead, and pipeline parallelism.
* [DeepSpeed Library](/concepts/deepspeed-library.md) - An open-source library developed by Microsoft that offers optimized memory usage, reduced communication overhead, and advanced pipeline parallelism for scaling model training on standard hardware.
* [DeepSpeed on Databricks](/concepts/deepspeed-on-databricks.md) - A parallelism technique from Microsoft that provides ZeRO optimization stages and other memory-saving strategies for large-scale training on Databricks.
* [DeepSpeed Use Cases](/concepts/deepspeed-use-cases.md) - The DeepSpeed distributor is beneficial for scenarios with low GPU memory, large model training, and large input data such as batch inference.
* [DeepSpeed ZeRO Stage 1](/concepts/deepspeed-zero-stage-2.md) - ZeRO stage that shards optimizer states across GPUs, reducing memory consumption while gradients and model parameters remain fully replicated.
* [DeepSpeed ZeRO Stage 2](/concepts/deepspeed-zero-stage-2.md) - ZeRO stage that shards both optimizer states and gradients across GPUs, further reducing memory compared to Stage 1.
* [DeepSpeed ZeRO Stage 3](/concepts/deepspeed-zero-stage-3.md) - A memory optimization technique that partitions model parameters, gradients, and optimizer states across GPUs to enable training of large models
* [DeepSpeed ZeRO Stage 3 Optimization](/concepts/deepspeed-zero-stage-3.md) - A memory optimization technique that partitions model parameters, gradients, and optimizer states across all GPUs to reduce per-GPU memory consumption, enabling training of large models that wouldn't fit on a single GPU.
* [Default Budget Policy Fallback Failure](/concepts/default-budget-policy-fallback-failure.md) - When a Databricks workspace disables the default serverless budget policy, MLflow fails to register scorers or run evaluations unless a policy is explicitly set on the experiment.
* [Default Catalog Configuration in Unity Catalog](/concepts/default-catalog-configuration-in-unity-catalog.md) - When a workspace is enabled for Unity Catalog, a default catalog is configured; automatic enablement sets the workspace catalog as default, while manual enablement sets `hive_metastore` as default, affecting how unqualified queries resolve.
* [DEFAULT Configuration Profile Constraint](/concepts/default-configuration-profile-constraint.md) - A constraint that pytest with Databricks Connect only works with the DEFAULT configuration profile, which must include the target cluster or serverless compute.
* [Default Email Destinations](/concepts/default-email-destinations.md) - A metastore-wide setting that routes access requests to the object owner's email when no explicit destination is configured.
* [DEFAULT Expression in UPDATE](/concepts/default-expression-in-update.md) - Sets a column to its defined default value (or NULL if no default is defined) during an UPDATE operation. Available in Databricks Runtime 11.3 LTS and above.
* [Default Feature Table Permissions](/concepts/default-feature-table-permissions.md) - Upon creation, the creator and workspace admins automatically receive CAN MANAGE permission on a feature table, while all other users have NO PERMISSIONS.
* [DEFAULT Keyword in MERGE Actions](/concepts/default-keyword-in-merge-actions.md) - The DEFAULT keyword can be used as an expression in UPDATE SET and INSERT VALUES clauses to explicitly set a column to its default value (Databricks Runtime 11.3 LTS+).
* [Default Permission Behavior on Feature Table Creation](/concepts/default-permission-behavior-on-feature-table-creation.md) - Default permissions assigned when a feature table is created: creator and workspace admins get CAN MANAGE, all others get NO PERMISSIONS
* [Default Permissions for Feature Tables](/concepts/default-permissions-for-feature-tables.md) - By default, the creator and workspace admins get CAN MANAGE permission while all other users have no permissions on a new feature table
* [Default Permissions on Feature Table Creation](/concepts/default-permissions-on-feature-table-creation.md) - When a feature table is created in Databricks Feature Store, the creator and workspace admins get CAN MANAGE, while other users get NO PERMISSIONS by default.
* [Default Serverless Budget Policy Fallback Error](/concepts/databricks-serverless-budget-policy-403-error.md) - A 403 Forbidden error that occurs when a workspace has disabled the default serverless budget policy and MLflow cannot automatically select a policy for an experiment.
* [Default Spark Session in Databricks Workspace](/concepts/default-spark-session-in-databricks-workspace.md) - In Databricks notebooks and jobs, the `spark` variable provides a pre-configured default Spark session that connects to the attached compute, accessible via DatabricksSession.builder.getOrCreate() without extra configuration.
* [Default values for feature lookups](/concepts/default-values-for-feature-lookups.md) - A mechanism in FeatureLookup that supplies fallback values when a feature column has no value, with the restriction that renamed outputs must use the new name in default_values.
* [Default vs AI Environments in AI Runtime](/concepts/default-vs-ai-environments-in-ai-runtime.md) - Two base environment options: 'None' (default environment) and 'AI v4/v5' (Databricks AI environment), configurable in notebook and job definitions.
* [Default workspace admin privileges (Unity Catalog)](/concepts/default-workspace-admin-privileges-on-metastore.md) - A set of metastore-level privileges (CREATE CATALOG, CREATE EXTERNAL LOCATION, CREATE SHARE, etc.) auto-granted to workspace admins in automatically-enabled Unity Catalog workspaces, represented via a system group.
* [Default Workspace Admin Privileges on Metastore](/concepts/default-workspace-admin-privileges-on-metastore.md) - The set of privileges (CREATE CATALOG, CREATE EXTERNAL LOCATION, etc.) automatically granted to workspace admins on the attached metastore in auto-enabled workspaces.
* [Degree and Degree Distribution](/concepts/degree-and-degree-distribution.md) - Degree is the number of edges incident to a node (with in-degree and out-degree for directed graphs); degree distribution describes the count of nodes at each degree level, revealing network structure and organization.
* [DELETE FROM (Delta Lake)](/concepts/delete-from-delta-lake.md) - SQL statement to delete rows from Delta Lake tables, optionally filtered by a predicate
* [DELETE FROM for Delta Lake](/concepts/delete-from-for-delta-lake.md) - The DELETE FROM statement in Databricks SQL is used to delete rows from Delta Lake tables, with optional predicate filtering and support for subqueries in the WHERE clause.
* [Delete Operations on Prompt Registry](/concepts/delete-operations-on-prompt-registry.md) - Deleting prompts requires deleting all versions first for Unity Catalog registries; other registry types allow direct prompt deletion without version cleanup.
* [Deleting Materialized Features](/concepts/deleting-materialized-features.md) - Process for removing materialized features by passing the appropriate materialized feature object; aggregation features require deleting the offline version to propagate to both stores, while ColumnSelection features are deleted directly via online materialized feature.
* [Deletion Vector constraints for Iceberg compat](/concepts/deletion-vectors-and-icebergcompat.md) - IcebergCompatV requires that Deletion Vectors be either disabled or fully purged from a Delta table before Uniform Iceberg compatibility can be enabled.
* [Deletion Vector corruption detection](/concepts/deletion-vector-corruption-detection.md) - Capability of FSCK REPAIR TABLE to detect corruption in deletion vectors of Delta Lake tables
* [Deletion Vector Integrity Checking](/concepts/deletion-vector-integrity-checking.md) - Detection of missing or corrupted deletion vector files associated with data files in Delta Lake tables, surfaced by FSCK REPAIR TABLE commands.
* [Deletion Vectors](/concepts/deletion-vectors.md) - A Delta Lake feature for marking rows as deleted without rewriting data files; conflicts with symlink manifest generation and has Parquet-only persistence constraints
* [Deletion Vectors and Column Mapping in Shared Tables](/concepts/deletion-vectors-and-column-mapping-in-shared-tables.md) - Storage optimization features (deletion vectors for efficient deletes/updates, column mapping for renaming/dropping columns) that can be enabled on shared Delta tables. Requires history sharing and recipients must use Databricks Runtime 14.1+ or open source delta-sharing-spark 3.1+ to query such tables.
* [Deletion Vectors and IcebergCompat](/concepts/deletion-vectors-and-icebergcompat.md) - IcebergCompatV requires Deletion Vectors to be either disabled or fully purged from a Delta table before compatibility can be enabled.
* [Deletion Vectors and IcebergCompat conflict](/concepts/deletion-vectors-and-icebergcompat.md) - IcebergCompatV requires Deletion Vectors to be either disabled or completely purged from a Delta table before enabling Iceberg compatibility, with specific REORG TABLE APPLY (PURGE) commands needed.
* [Deletion Vectors in Delta Lake](/concepts/deletion-vectors-in-delta-lake.md) - A Delta Lake table feature that must be purged or disabled before enabling IcebergCompat, as unresolved deletion vectors cause compatibility violations.
* [Deletion Vectors in Delta-Iceberg Compatibility](/concepts/deletion-vectors-and-icebergcompat.md) - Deletion Vectors must be purged or disabled before enabling IcebergCompatV on a Delta table, as they are incompatible with the Iceberg read protocol.
* [Delta ACID Transactions](/concepts/delta-acid-transactions.md) - Delta Lake provides ACID transaction support to data lakes, ensuring data reliability and consistency.
* [Delta Cache](/concepts/delta-cache.md) - A Databricks caching feature for Delta Lake tables that materializes selected query results in the disk cache to avoid rescanning source files.
* [Delta Cache Supported Data Sources](/concepts/delta-cache-supported-data-sources.md) - The Delta Cache (disk cache) supports Delta tables, Parquet tables, and views, with views restricted to simple queries upon expansion.
* [Delta Change Data Feed (CDF)](/concepts/delta-change-data-feed-cdf.md) - A Delta table feature that records row-level changes and is required for Triggered and Continuous online table sync modes.
* [Delta Change Data Feed for Streaming](/concepts/delta-change-data-feed-for-streaming.md) - Using the Delta Lake change data feed to capture and stream all types of row-level changes (inserts, updates, deletes) from a source table, providing the most robust approach for handling upstream modifications.
* [Delta Clone](/concepts/delta-clone.md) - A Databricks operation to clone an existing table (including Iceberg or other formats) into a Delta table, with specific compatibility constraints.
* [Delta Clone (Databricks)](/concepts/delta-clone.md) - A Databricks operation that creates a deep or shallow clone of a Delta table; certain source table features like indexes or partition evolution can cause incompatibility errors.
* [Delta Clone with History](/concepts/delta-clone-with-history.md) - A Databricks Delta Lake operation that clones a Delta table including its full version history, which has restrictions on source table types.
* [Delta Clone with History (Deep Clone)](/concepts/delta-clone-with-history-deep-clone.md) - A Databricks operation that creates a deep copy of a Delta table including its full commit history, as opposed to a shallow clone.
* [Delta CLONE with history operation](/concepts/delta-clone-with-history.md) - A Delta Lake operation that clones a table including its full version history, which has restrictions on supported source types and time travel methods
* [Delta CLONE WITH HISTORY semantics](/concepts/delta-clone-with-history.md) - The broader Databricks/Delta Lake feature of cloning a table including its full change history, which imposes restrictions on source format and access methods.
* [Delta Column Mask Compatibility with External Metadata](/concepts/delta-column-mask-compatibility-with-external-metadata.md) - Tables with Column Mask (CM) policies are unsupported sources for Delta external metadata operations.
* [Delta Conversion Caveats and Constraints](/concepts/delta-conversion-caveats-and-constraints.md) - Important limitations: file visibility after conversion, shared external directories across tables, and metadata mismatch exceptions when catalog and log diverge.
* [Delta Conversion Metadata and Partitioning](/concepts/delta-conversion-metadata-and-partitioning.md) - How CONVERT TO DELTA handles partition specifications from the metastore or explicit PARTITIONED BY clauses, with validation against directory structure.
* [Delta Conversion Metadata Check](/concepts/delta-conversion-metadata-check.md) - A validation mechanism in CONVERT TO DELTA that compares catalog metadata with existing Delta Lake transaction log metadata, throwing a convertMetastoreMetadataMismatchException on mismatch, which can be overridden via Spark configuration.
* [Delta Conversion Statistics Collection](/concepts/delta-conversion-statistics-collection.md) - The automatic process during CONVERT TO DELTA that collects statistics to improve query performance on the converted Delta table, which can be bypassed with the NO STATISTICS option.
* [Delta Deep Clone for Cross-Region Replication](/concepts/delta-deep-clone-for-cross-region-replication.md) - Technique using DEEP CLONE to replicate Delta tables to external locations across regions, enabling incremental updates and avoiding egress costs when recipients access local replicas.
* [Delta error SQLSTATE codes](/concepts/delta-error-sqlstate-codes.md) - Standardized SQLSTATE codes used by Databricks Delta to classify error conditions, such as class 0A for feature not supported
* [Delta Error Sub-Conditions](/concepts/delta-error-sub-conditions.md) - The pattern of categorized sub-conditions (NON_CLUSTERING_COLUMN, NO_CLUSTERING_COLUMNS, UNRESOLVED_COLUMN, UNSUPPORTED_EXPRESSION) within Delta error classes to provide granular error diagnostics.
* [Delta External Metadata Projection Restrictions](/concepts/delta-external-metadata-projection-restrictions.md) - The reconciliation query for external metadata operations has unsupported projection expressions that trigger a PROJECTION_NOT_SUPPORTED error.
* [Delta External Metadata Support for Streaming Tables and Materialized Views](/concepts/delta-external-metadata-support-for-streaming-tables-and-materialized-views.md) - Only streaming tables and materialized views are supported sources for Delta external metadata operations; other table types trigger an error.
* [Delta Iceberg Partition Spec Constraints](/concepts/delta-iceberg-partition-spec-constraints.md) - A restriction that replacing partitioned tables with differently-named partition specs is not supported under IcebergCompatV due to Iceberg-Spark 1.1.0 limitations.
* [Delta Lake](/concepts/delta-lake.md) - An open source storage layer that brings reliability, ACID transactions, scalable metadata handling, and unified streaming/batch processing to data lakes.
* [Delta Lake ACL Restrictions on Schema Migration](/concepts/delta-lake-acl-restrictions-on-schema-migration.md) - When Table ACLs are enabled on a cluster, automatic schema migration via DataFrameWriter options is blocked; users must use ALTER TABLE instead.
* [Delta Lake Apache Spark Compatibility](/concepts/apache-spark-compatibility.md) - Delta Lake runs on top of existing data lakes and is fully compatible with Apache Spark APIs.
* [Delta Lake API](/concepts/delta-lake-api.md) - The programming interface for interacting with Delta Lake, with API references available for Scala, Java, and Python on the Delta Lake website.
* [Delta Lake API Reference](/concepts/delta-lake-api.md) - The official API documentation for Delta Lake, providing Scala, Java, and Python programming interfaces for interacting with Delta Lake tables and operations.
* [Delta Lake APIs](/concepts/delta-lake-api.md) - Programmatic interfaces for Scala, Java, and Python that allow developers to interact with Delta Lake tables using Apache Spark APIs.
* [Delta Lake Archival Support](/concepts/delta-lake-archival-support.md) - A Databricks feature that enables Delta Lake tables to work with cloud-based lifecycle policies by ignoring files older than a specified threshold, preventing query failures against archived data.
* [Delta Lake as a Streaming Sink](/concepts/delta-lake-as-a-streaming-sink.md) - Using Delta Lake tables as sinks for Spark Structured Streaming with exactly-once processing guarantees
* [Delta Lake as a Streaming Source and Sink](/concepts/delta-lake-as-a-streaming-source-and-sink.md) - Overview of using Delta Lake tables with Spark Structured Streaming readStream/writeStream, including exactly-once guarantees and performance benefits.
* [Delta Lake Automated Metadata Management](/concepts/delta-lake-automated-metadata-management.md) - Delta Lake automatically handles REFRESH TABLE, partition management, and data file tracking via the transaction log, making manual operations unnecessary and dangerous.
* [Delta Lake Automatic Partition Management](/concepts/delta-lake-automatic-partition-management.md) - Delta Lake automatically tracks partitions and returns up-to-date information, eliminating the need for manual REFRESH TABLE, ALTER TABLE ADD/DROP PARTITION, or MSCK commands.
* [Delta Lake Bucket Versioning Limitations on S3](/concepts/delta-lake-s3-bucket-versioning-limitations.md) - Databricks recommends against enabling S3 bucket versioning for Delta Lake tables because it interferes with Delta Lake's own versioning and garbage collection, leading to retained metadata and data files.
* [Delta Lake Change Data Feed (CDF)](/concepts/delta-lake-change-data-feed-cdf.md) - The underlying Delta Lake feature that table_changes relies on, which only works on Delta tables, not views.
* [Delta Lake Cloning Constraints from External Sources](/concepts/delta-lake-cloning-constraints-from-external-sources.md) - The concept that Delta Lake imposes specific compatibility constraints when cloning from external table formats like Iceberg or tables with database indexes, requiring certain conditions to be met.
* [Delta Lake CLUSTER BY Duplicate Constraint](/concepts/delta-lake-cluster-by-duplicate-constraint.md) - A constraint in Delta Lake that the CLUSTER BY clause must not contain duplicate column references, enforced by the CLUSTER_BY sub-reason of DELTA_DUPLICATE_COLUMNS_FOUND.
* [Delta Lake Clustering Columns](/concepts/delta-lake-clustering-columns.md) - Columns designated as clustering columns in Liquid tables; only these columns may be referenced in certain command predicates.
* [Delta Lake Column Mask (CM) Policies](/concepts/delta-lake-column-mask-cm-policies.md) - A column-level security feature in Delta Lake that restricts visibility of sensitive columns, and is unsupported for external metadata operations.
* [Delta Lake Column Masking](/concepts/delta-lake-column-masking.md) - A Delta table security feature that masks column values; incompatible with external metadata sources in Databricks.
* [Delta Lake column masks](/concepts/delta-lake-column-masks.md) - A Delta Lake table feature that restricts column visibility; incompatible with REFRESH SYNC UNIFORM.
* [Delta Lake Compatibility with Apache Spark](/concepts/delta-lake-compatibility-with-apache-spark.md) - Delta Lake runs on top of existing data lakes and is fully compatible with Apache Spark APIs.
* [Delta Lake Concurrent Operation Retry Strategy](/concepts/delta-lake-concurrent-operation-retry-strategy.md) - The recommended resolution pattern for DELTA_CONCURRENT_APPEND errors: retrying the failed operation after a concurrent modification completes.
* [Delta Lake Conversion from Apache Iceberg](/concepts/delta-lake-conversion-from-apache-iceberg.md) - Process and limitations for converting Iceberg tables to Delta Lake, including version requirements and unsupported features like partition evolution
* [Delta Lake data ingestion duplicate detection](/concepts/delta-lake-data-ingestion-duplicate-detection.md) - Sub-conditions under DELTA_DUPLICATE_COLUMNS_FOUND that detect duplicate columns during data ingestion operations, including DATA (saving data) and CONVERT_TO_DELTA (converting formats to Delta).
* [Delta Lake Deletion Vectors](/concepts/delta-lake-deletion-vectors.md) - A Delta Lake feature that marks rows as logically deleted rather than physically removing them, enabling efficient operations like MERGE and UPDATE without rewriting entire Parquet files.
* [Delta Lake DML Operations](/concepts/delta-lake-dml-operations.md) - Delta Lake supports standard DML operations including DELETE, INSERT, MERGE, and UPDATE with SQL syntax.
* [Delta Lake DML Restrictions](/concepts/delta-lake-dml-restrictions.md) - Constraints on Delta Lake DML operations including DELETE restrictions on temporal specifications, foreign tables, and subquery nesting.
* [Delta Lake DML statements](/concepts/delta-lake-dml-operations.md) - Set of data modification operations supported on Delta Lake tables, including DELETE, INSERT, MERGE, UPDATE, and COPY INTO.
* [Delta Lake DML Support](/concepts/delta-lake-dml-support.md) - DELETE FROM is only supported for Delta Lake tables in Databricks, not other table formats
* [Delta Lake Duplicate Columns in Data Ingestion](/concepts/delta-lake-duplicate-columns-in-data-ingestion.md) - Scenarios where duplicate columns are detected during data ingestion operations such as saving data or converting to Delta format.
* [Delta Lake Duplicate Columns in Partitioning](/concepts/delta-lake-duplicate-columns-in-partitioning.md) - Scenarios where duplicate columns are detected in PARTITION BY, CLUSTER BY, or partition schema specifications in Delta Lake.
* [Delta Lake Duplicate Columns in Schema Operations](/concepts/delta-lake-duplicate-columns-in-schema-operations.md) - Scenarios where duplicate columns are detected during schema-related Delta Lake operations including adding columns, replacing columns, reading schemas, and updating metadata.
* [Delta Lake External Metadata](/concepts/delta-lake-external-metadata.md) - A mechanism for storing Delta table metadata externally; has compatibility constraints with certain Delta features like column masking, RLS, and unsupported table types.
* [Delta Lake File Compaction](/concepts/delta-lake-file-compaction.md) - The practice of running OPTIMIZE to compact small files into larger ones, improving read throughput and query performance
* [Delta Lake File Compaction (OPTIMIZE)](/concepts/delta-lake-file-compaction.md) - Best practice of running OPTIMIZE to compact small files into larger ones for improved read throughput, with VACUUM to remove old files.
* [Delta Lake for Deep Learning Data Loading](/concepts/delta-lake-for-deep-learning-data-loading.md) - Using Delta Lake tables to optimize data throughput for deep learning workloads, particularly for image ETL and large-scale training.
* [Delta Lake General Best Practices](/concepts/delta-lake-general-best-practices.md) - High-level recommendations for Delta Lake including Unity Catalog managed tables, predictive optimization, liquid clustering, and safe table replacement patterns.
* [Delta Lake Integration for Ray via Third-Party Libraries](/concepts/delta-lake-integration-for-ray-via-third-party-libraries.md) - Use the deltalake (delta-rs) and deltaray (Delta Incubator) libraries to write Ray Core data to Delta Lake tables, though deltalake currently works only with Hive metastore tables (not Unity Catalog) and is not officially supported by Databricks.
* [Delta Lake Legacy Configuration Removal](/concepts/legacy-delta-configuration-removal.md) - Recommendation to remove explicit legacy Delta configurations from Spark configs and table properties when upgrading Databricks Runtime to enable new optimizations and default values
* [Delta Lake Limitations on S3](/concepts/delta-lake-limitations-on-s3.md) - Warning against simultaneously modifying the same Delta table stored on S3 from multiple workspaces or systems due to concurrency limitations.
* [Delta Lake Liquid Clustering](/concepts/delta-lake-liquid-clustering.md) - A data organization technique in Delta Lake where data is clustered by specified columns, and operations like certain commands may only reference those clustering columns.
* [Delta Lake manifest files](/concepts/delta-lake-manifest-files.md) - External metadata files generated from a Delta table that list its data files for use by query engines that do not natively support Delta Lake.
* [Delta Lake Manifest Files for External Query Engines](/concepts/delta-lake-manifest-files-for-external-query-engines.md) - Generated files that allow external query engines like Presto and Amazon Athena to read Delta Lake tables without native Delta Lake support.
* [Delta Lake MERGE INTO (Upsert)](/concepts/delta-lake-merge-into-upsert.md) - A SQL statement that merges updates, insertions, and deletions from a source table into a target Delta table, supporting WHEN MATCHED, WHEN NOT MATCHED, and WHEN NOT MATCHED BY SOURCE clauses.
* [Delta Lake MERGE Performance Optimization](/concepts/delta-lake-merge-performance-tuning.md) - Strategies to speed up MERGE operations including reducing search space with partition constraints, file compaction, controlling shuffle partitions, enabling optimized writes, and tuning file sizes.
* [Delta Lake MERGE Performance Tuning](/concepts/delta-lake-merge-performance-tuning.md) - Techniques to reduce merge operation time including reducing search space, compacting files, controlling shuffle partitions, enabling optimized writes, and tuning file sizes
* [Delta Lake metadata update duplicate detection](/concepts/delta-lake-metadata-update-duplicate-detection.md) - A sub-condition under DELTA_DUPLICATE_COLUMNS_FOUND specifically for detecting duplicate columns when updating table metadata, distinct from schema or partition validation.
* [Delta Lake multi-cluster write configuration](/concepts/delta-lake-multi-cluster-write-configuration.md) - The spark.databricks.delta.multiClusterWrites.enabled setting controls whether multiple clusters can concurrently write to the same Delta Lake table; disabling it requires all writes to originate from a single cluster.
* [Delta Lake multi-cluster write restrictions on S3](/concepts/delta-lake-multi-cluster-write-restrictions-on-s3.md) - S3's eventual consistency model limits multi-cluster writes to a single Databricks workspace; cross-workspace writes to the same Delta table on S3 risk data corruption.
* [Delta Lake Multi-Cluster Writes Configuration](/concepts/delta-lake-multi-cluster-write-configuration.md) - The spark.databricks.delta.multiClusterWrites.enabled setting controls multi-cluster writes; when disabled, writes to a single Delta table must originate from exactly one cluster to avoid corruption.
* [Delta Lake Named Table References in DDL](/concepts/delta-lake-named-table-references-in-ddl.md) - The DROP BLOOMFILTER INDEX command requires a table name without temporal or options specifications, and fails if the table or specified columns do not exist.
* [Delta Lake on Databricks](/concepts/delta-lake-on-databricks.md) - Databricks' integration and documentation of Delta Lake, including tutorials for creating and managing Delta Lake tables within the Databricks environment on AWS.
* [Delta Lake on Databricks SQL](/concepts/delta-lake-on-databricks-sql.md) - Databricks SQL and Databricks Runtime support for Delta Lake DELETE operations with full SQL DML semantics.
* [Delta Lake Optimistic Concurrency Control](/concepts/delta-lake-optimistic-concurrency-control.md) - The concurrency model underlying Delta Lake that detects conflicts at commit time rather than locking resources upfront, leading to error conditions like DELTA_CONCURRENT_DELETE_DELETE.
* [Delta Lake Optimized Writes](/concepts/delta-lake-optimized-writes.md) - A feature that reduces the number of small files produced by MERGE operations on partitioned tables by reducing the number of files each shuffle task writes across multiple partitions
* [Delta Lake partition duplicate detection](/concepts/delta-lake-partition-duplicate-detection.md) - Sub-conditions under DELTA_DUPLICATE_COLUMNS_FOUND that detect duplicate columns in partitioning contexts, including PARTITION_COLUMNS, PARTITION_SCHEMA, and CLUSTER_BY clauses.
* [Delta Lake Partition Spec Immutability in IcebergCompatV1](/concepts/delta-lake-partition-spec-immutability-in-icebergcompatv1.md) - The restriction that replacing a partitioned table with a differently-named partition spec is disallowed when IcebergCompatV1 is enabled, mirroring a limitation in Iceberg-Spark 1.1.0.
* [Delta Lake Partitioning Considerations](/concepts/delta-lake-partitioning-considerations.md) - Guidance on avoiding over-partitioned tables when migrating from Parquet to Delta Lake, as over-partitioning is a common cause of slow workloads.
* [Delta Lake Partitioning Constraints](/concepts/delta-lake-partitioning-constraints.md) - The requirement that partition columns and partition schemas in Delta Lake must not contain duplicate column definitions, enforced by the PARTITION_COLUMNS and PARTITION_SCHEMA sub-reasons of DELTA_DUPLICATE_COLUMNS_FOUND.
* [Delta Lake Predictive Optimization](/concepts/delta-lake-predictive-optimization.md) - A Databricks feature that automatically runs OPTIMIZE and VACUUM commands on Unity Catalog managed tables to maintain performance
* [Delta Lake Protocol Change Conflicts](/concepts/delta-lake-protocol-change-conflicts.md) - A transaction conflict triggered when a concurrent operation upgrades the table protocol in Delta Lake, requiring the conflicting operation to be retried.
* [Delta Lake protocol changes and table metadata conflicts](/concepts/delta-lake-protocol-changes-and-table-metadata-conflicts.md) - A concurrency conflict type where a concurrent operation upgrades the table protocol or changes table metadata (schema, partitioning), invalidating an in-progress transaction.
* [Delta Lake Protocol Upgrade Conflicts](/concepts/delta-lake-protocol-change-conflicts.md) - A transaction conflict triggered when a concurrent operation upgrades the Delta Lake table protocol version, forcing other transactions to retry.
* [Delta Lake Reader Table Features](/concepts/delta-lake-reader-table-features.md) - Extensibility features in Delta Lake that affect how a table is read, and can cause incompatibility with UniForm metadata refresh.
* [Delta Lake Reconciliation Queries](/concepts/delta-lake-reconciliation-queries.md) - Queries used to compare or reconcile data between sources in Delta Lake, with restrictions on projections and column aliasing when external metadata is involved.
* [Delta Lake Row-Level Conflict Detection](/concepts/delta-lake-row-level-conflict-detection.md) - A mechanism used by Delta Lake to detect and resolve transaction conflicts at the granularity of individual rows, with constraints on time, predicates, and metadata.
* [Delta Lake Row-Level Conflict Resolution](/concepts/row-level-conflict-resolution.md) - A mechanism in Delta Lake that detects and resolves conflicts at the row level when concurrent transactions delete data being read by another transaction.
* [Delta Lake row-level security](/concepts/delta-lake-row-level-security.md) - A Delta Lake table feature that restricts row visibility; incompatible with REFRESH SYNC UNIFORM.
* [Delta Lake Row-Level Security (RLS)](/concepts/delta-lake-row-level-security-rls.md) - A Delta table security feature that restricts row access; incompatible with external metadata sources in Databricks.
* [Delta Lake Row-Level Security (RLS) Policies](/concepts/delta-lake-row-level-security-rls.md) - A row-level security feature in Delta Lake that restricts data access at the row level, and is unsupported for external metadata operations.
* [Delta Lake S3 bucket versioning conflict](/concepts/delta-lake-s3-bucket-versioning-conflict.md) - S3 bucket versioning should not be enabled for Delta Lake tables because it conflicts with Delta Lake's own versioning and garbage collection, causing retained data files and performance degradation.
* [Delta Lake S3 bucket versioning limitations](/concepts/delta-lake-s3-bucket-versioning-limitations.md) - Databricks recommends against enabling S3 bucket versioning for Delta Lake tables due to conflicts with Delta's own versioning and garbage collection, suggesting a max of 3 versions retained for ≤7 days if enabled.
* [Delta Lake Schema Migration](/concepts/delta-lake-schema-migration.md) - The process of evolving a Delta table's schema to match incoming data, controlled via DataFrameWriter options (mergeSchema, overwriteSchema) or session configurations.
* [Delta Lake schema modification duplicate detection](/concepts/delta-lake-schema-modification-duplicate-detection.md) - Sub-conditions under DELTA_DUPLICATE_COLUMNS_FOUND that detect duplicate columns during schema-altering operations such as ADDING_COLUMNS, REPLACING_COLUMNS, or when validating EXISTING_SCHEMA, TABLE_SCHEMA, SPECIFIED_COLUMNS, and READ_SCHEMA.
* [Delta Lake Schema Operations and Duplicate Column Detection](/concepts/delta-lake-schema-operations-and-duplicate-column-detection.md) - The set of Delta Lake operations (schema changes, data ingestion, partitioning, clustering, table conversion) that trigger duplicate column validation and raise DELTA_DUPLICATE_COLUMNS_FOUND.
* [Delta Lake schema validation rules for empty schemas](/concepts/delta-lake-schema-validation-rules-for-empty-schemas.md) - The four specific sub-conditions (STRUCT_ALL_VOID_FIELDS, STRUCT_NO_FIELDS, TABLE_ALL_VOID_COLUMNS, TABLE_NO_COLUMNS) under which Delta Lake refuses to write because the schema is effectively empty.
* [Delta Lake SQLSTATE 2D521](/concepts/delta-lake-sqlstate-2d521.md) - The SQL state code (2D521) indicating an invalid transaction termination due to a transaction conflict in Delta Lake.
* [Delta Lake Stream Input Rate Limiting](/concepts/delta-lake-stream-input-rate-limiting.md) - Options maxFilesPerTrigger and maxBytesPerTrigger to control how much data each micro-batch processes, helping manage memory, latency, and cloud storage costs
* [Delta Lake Streaming and Batch Unification](/concepts/delta-lake-streaming-and-batch-unification.md) - Delta Lake unifies streaming and batch data processing within a single storage layer.
* [Delta Lake streaming retention window and failure handling](/concepts/delta-lake-streaming-retention-window-and-failure-handling.md) - Streaming queries from Delta tables must run within the VACUUM retention window (7 days for data files, 30 days for transaction logs) or they fail with DELTA_FILE_NOT_FOUND_DETAILED.
* [Delta Lake Streaming Sink Modes](/concepts/delta-lake-streaming-sink-modes.md) - Using Delta Lake tables as streaming sinks with append mode (adding new records) and complete mode (replacing the entire table each batch)
* [Delta Lake Table](/concepts/delta-lake-table.md) - The storage format required for UPDATE, DELETE, and MERGE operations in Databricks; standard tables do not support UPDATE.
* [Delta Lake Table Deletion on S3](/concepts/delta-lake-table-deletion-on-s3.md) - Using rm -rf to drop a Delta Lake table on S3 can result in stale data; users should use proper table drop operations instead.
* [Delta Lake table deletion risks on S3](/concepts/delta-lake-table-deletion-on-s3.md) - Using rm -rf or direct file deletion to drop Delta Lake tables on S3 causes stale data issues; proper drop or replace table operations should be used instead.
* [Delta Lake Table Feature Dependencies](/concepts/delta-lake-table-feature-dependencies.md) - The concept that Delta Lake table features have dependency relationships where enabling a compatibility mode like IcebergCompatV1 requires certain features to be enabled and prohibits others.
* [Delta Lake table features and Iceberg compatibility](/concepts/delta-lake-table-features-compatibility.md) - IcebergWriterCompatV requires specific Delta Lake table features to be enabled and supported, and is incompatible with certain features
* [Delta Lake Table Features Compatibility](/concepts/delta-lake-table-features-compatibility.md) - The set of table features that are required, incompatible, or must be enabled/disabled for IcebergCompatV to function correctly.
* [Delta Lake table history retention](/concepts/delta-lake-table-history-retention.md) - Delta Lake retains table history metadata for 30 days by default
* [Delta Lake Table Protocol Changes](/concepts/delta-lake-table-protocol-changes.md) - A concurrent operation upgrading the table protocol version, causing transaction conflicts that require retrying the operation
* [Delta Lake Table Protocol Upgrade Conflicts](/concepts/delta-lake-protocol-change-conflicts.md) - A conflict scenario where a concurrent operation upgrades a Delta table's protocol version, requiring the conflicting transaction to retry.
* [Delta Lake Table Validation Strategies](/concepts/delta-lake-table-validation-strategies.md) - Three modes of table validation: METADATA ONLY (lightweight CRC check), DRY RUN (file existence checks), and VERIFY ALL FILES (deep data validation including file readability and deletion vector integrity).
* [Delta Lake Time Travel](/concepts/delta-lake-time-travel.md) - Ability to query, restore, or access previous versions of a Delta table using timestamps or version numbers
* [Delta Lake Time Travel and VACUUM](/concepts/delta-lake-time-travel-and-vacuum.md) - Running VACUUM on a Delta table removes the ability to time travel to versions older than the specified data retention period.
* [Delta Lake Transaction Log](/concepts/delta-lake-transaction-log.md) - The metadata structure created during CONVERT TO DELTA that tracks all Parquet files, infers schema from file footers, and enables Delta Lake features on the table.
* [Delta Lake Transaction Log Generation](/concepts/delta-lake-transaction-log-generation.md) - During conversion, CONVERT TO DELTA scans existing Parquet files (or Iceberg manifests) and creates a Delta Lake transaction log that tracks those files, with no data rewriting required.
* [Delta Lake Transaction Log Integrity](/concepts/delta-lake-transaction-log-integrity.md) - Delta Lake uses a transaction log to commit changes atomically; users must never manually modify, add, or delete Parquet data files directly as it can cause data loss or corruption.
* [Delta Lake Transaction Protocol Changes](/concepts/delta-lake-table-protocol-changes.md) - A conflict scenario where a concurrent operation upgrades the table protocol, which can invalidate ongoing transactions and require retries.
* [Delta Lake Universal Format (delta.universalFormat.enabledFormats)](/concepts/delta-lake-universal-format-deltauniversalformatenabledformats.md) - A Delta Lake table property that enables multi-format support, requiring explicit opt-in for features like SYNC UNIFORM
* [Delta Lake Universal Format (UniForm)](/concepts/delta-lake-universal-format-uniform.md) - A Databricks feature that enables Delta Lake tables to be read by Apache Iceberg clients by automatically generating Iceberg metadata asynchronously without rewriting data.
* [Delta Lake VACUUM and Garbage Collection](/concepts/delta-lake-vacuum-and-garbage-collection.md) - Delta Lake's VACUUM command permanently deletes data files and transaction logs; enabling S3 bucket versioning causes S3 to retain these files, undermining Delta's own garbage collection.
* [Delta Lake VACUUM and garbage collection on S3](/concepts/delta-lake-vacuum-and-garbage-collection.md) - Delta Lake implements its own versioning and garbage collection (including VACUUM), which conflicts with S3 bucket versioning and lifecycle policies that retain deleted files.
* [Delta Lake VACUUM and S3 Garbage Collection](/concepts/delta-lake-vacuum-and-garbage-collection.md) - Delta Lake's VACUUM command permanently deletes old data files as part of its garbage collection, but S3 bucket versioning retains these deleted files, undermining the cleanup process.
* [Delta Lake VACUUM and S3 Versioning Interaction](/concepts/delta-lake-vacuum-and-garbage-collection.md) - When S3 bucket versioning is enabled, VACUUM operations and automatic transaction log cleanup in Delta Lake do not permanently delete files; S3 retains older versions.
* [Delta Lake view restrictions](/concepts/delta-lake-view-restrictions.md) - Constraints on which views can be used with table_changes, including requiring single-relation, deterministic, Delta-based views without subqueries
* [Delta Lake vs Parquet on Apache Spark](/concepts/delta-lake-vs-parquet-on-spark.md) - Key differences where Delta Lake automatically handles REFRESH TABLE, partition management, partition reading, and data file modifications that must be done manually with Parquet
* [Delta Lake vs Parquet on Spark](/concepts/delta-lake-vs-parquet-on-spark.md) - Key operational differences: Delta Lake automatically handles REFRESH TABLE, partition management, partition loading, and data file modifications, whereas Parquet on Spark requires manual management.
* [Delta Lake vs Parquet Operational Differences](/concepts/delta-lake-vs-parquet-operational-differences.md) - Key operational differences where Delta Lake automates table management tasks that require manual steps with Parquet, including partition management, data file integrity, and caching.
* [Delta Lake-Iceberg column mapping constraints](/concepts/delta-lake-iceberg-column-mapping-constraints.md) - IcebergWriterCompatV requires column mapping physical names to follow the 'col-[fieldId]' convention, enforced via the FIELD_ID_DOES_NOT_MATCH_PHYSICAL_NAME condition
* [Delta Lake-Iceberg map key constraints](/concepts/delta-lake-iceberg-map-key-constraints.md) - IcebergWriterCompatV prohibits changing map keys that are structs, enforced via the CANNOT_CHANGE_MAP_STRUCT_KEY condition
* [Delta Log File Removal](/concepts/delta-log-file-removal.md) - Manual deletion of transaction log files from the Delta log directory, a primary cause of the non-contiguous version error across all cloud platforms.
* [Delta log gap](/concepts/delta-log-gap.md) - A missing version sequence number in the Delta transaction log, often caused by manual file deletion or S3 eventual consistency issues.
* [Delta Log Location](/concepts/delta-log-location.md) - The metadata directory (_delta_log) for a Delta table; errors occur when this location is missing or unexpected during Uniform Ingress operations.
* [Delta log non-contiguity due to manual file deletion](/concepts/delta-log-non-contiguity-due-to-manual-file-deletion.md) - Manually removing files from the Delta transaction log can create gaps in version numbering, triggering the non-contiguous error.
* [Delta log non-contiguity due to S3 eventual consistency](/concepts/delta-log-non-contiguity-due-to-s3-eventual-consistency.md) - On AWS, eventual consistency of S3 when a Delta table is deleted and recreated at the same location can cause non-contiguous version gaps.
* [Delta Log Version Mismatch on Concurrent Rollback](/concepts/deltalog-version-mismatch-in-managed-rollback.md) - A concurrency conflict where the managed and external DeltaLog versions diverge, typically because a concurrent ALTER TABLE UNSET MANAGED command already succeeded.
* [Delta Row Filter (RLS) Compatibility with External Metadata](/concepts/delta-row-filter-rls-compatibility-with-external-metadata.md) - Tables with Row-level Security (RLS) policies are unsupported sources for Delta external metadata operations.
* [Delta Schema Migration via mergeSchema](/concepts/delta-schema-migration-via-mergeschema.md) - The mechanism to enable automatic schema migration when writing to Delta tables using the mergeSchema option or session configuration.
* [Delta Sharing](/concepts/delta-sharing.md) - An open protocol for secure real-time data sharing across platforms, foundational to Databricks-to-Databricks view sharing
* [Delta Sharing Asset Types](/concepts/delta-sharing-asset-types.md) - Types of assets that can be shared via OpenSharing including tables, views, materialized views, streaming tables, and volumes
* [Delta Sharing Clean Rooms](/concepts/delta-sharing-clean-rooms.md) - A mechanism within Delta Sharing where securable objects are shared via clean rooms, with central clean room IDs tracked as a dependency that blocks deletion.
* [Delta Sharing on AWS Databricks](/concepts/delta-sharing-on-aws-databricks.md) - An open protocol for secure data sharing across Databricks workspaces on AWS, enabling cross-workspace collaboration without copying data.
* [Delta Sharing Open Sharing](/concepts/delta-sharing-open-sharing.md) - An authentication flow for sharing data via Delta Sharing with recipients who do not have access to a Unity Catalog-enabled Databricks workspace, using either bearer tokens or OIDC federation.
* [Delta Sharing Provider/Recipient Error Resolution](/concepts/delta-sharing-providerrecipient-error-resolution.md) - The bilateral error resolution approach in Delta Sharing where providers verify recipient property existence and recipients contact the data provider when property mismatches occur.
* [Delta Sharing Recipient Object](/concepts/delta-sharing-recipient-object.md) - A named object in Unity Catalog that represents the identity of a user or group who consumes shared data, with authentication types depending on the recipient's environment.
* [Delta Sharing Recipient Permissions Model](/concepts/delta-sharing-recipient-permissions-model.md) - The hierarchy of privileges required to create, describe, manage, and grant access to recipients and shares in Unity Catalog, including metastore admin, USE RECIPIENT, CREATE RECIPIENT, and share-level permissions.
* [Delta Sharing recipient properties](/concepts/delta-sharing-recipient-properties.md) - Configurable properties on Delta Sharing recipients that can restrict or control data access
* [Delta Sharing Resource Limits (Active Files & RemoveFile Actions)](/concepts/delta-sharing-resource-limits-active-files-removefile-actions.md) - Hard limits on shared Delta tables: 400K active files (AddFile) and 100K RemoveFile entries in the Delta log, which if exceeded produce RESOURCE_LIMIT_EXCEEDED errors.
* [Delta Sharing Securable Protection](/concepts/delta-sharing-securable-protection.md) - The security principle that securable objects (tables, views) actively shared via OpenSharing or used in clean rooms cannot be deleted until sharing is revoked.
* [Delta Sharing Security Model](/concepts/delta-sharing-security-model.md) - The overarching security approach in Delta Sharing that uses function whitelisting, column-level security, and view definitions rather than raw table access to protect shared data.
* [Delta Source Streaming Offsets](/concepts/delta-source-streaming-offsets.md) - Conceptual mechanism by which Delta streaming sources track and skip processing positions via offset ranges
* [Delta Streaming Offset Validation Constraints](/concepts/delta-streaming-offset-validation-rules.md) - Validations that streaming offsets must pass when used with skip offset ranges in Delta sources, including restrictions on event time, initial snapshot status, and index format
* [Delta Streaming Offset Validation Rules](/concepts/delta-streaming-offset-validation-rules.md) - Three validation constraints for Delta streaming skip offsets: no event time allowed, no initial snapshot offsets permitted, and offsets must reference a valid BASE_INDEX.
* [Delta Streaming Skip Offsets](/concepts/delta-streaming-skip-offsets.md) - A mechanism in Databricks Delta Lake streaming that allows users to specify an offset range to skip when reading from a Delta source, enabling selective processing of data.
* [Delta Streaming Tables and Materialized Views](/concepts/delta-streaming-tables-and-materialized-views.md) - The only table types supported by Delta external metadata; other table types trigger the DELTA_EXTERNAL_METADATA_UNSUPPORTED_SOURCE error.
* [Delta Table as Feature Table](/concepts/delta-table-as-feature-table.md) - Any Delta table in Unity Catalog with a primary key constraint can be used as a feature table for machine learning.
* [Delta Table Caching](/concepts/delta-table-caching.md) - The application of CACHE SELECT specifically to Delta Lake tables, enabling faster subsequent queries by avoiding full file scans.
* [Delta Table Cloning](/concepts/delta-table-cloning.md) - The ability to create deep or shallow copies of Delta tables, copying data and/or metadata to a target location for purposes like archiving, ML workflows, and data migration.
* [Delta Table Column Mapping](/concepts/delta-table-column-mapping.md) - A Delta Lake feature that must be enabled to use column aliases in reconciliation queries; related to external metadata constraints.
* [Delta Table Features and IcebergCompat dependencies](/concepts/delta-table-features-and-icebergcompat-dependencies.md) - IcebergCompatV requires specific Delta table features to be enabled and supported, and prevents dropping required features or enabling incompatible features simultaneously.
* [Delta table features compatibility with Iceberg](/concepts/delta-table-features-compatibility-with-iceberg.md) - Constraints enforced by IcebergWriterCompatV that require specific Delta table features to be enabled and prohibit incompatible features
* [Delta Table for Image Metadata Management](/concepts/delta-table-for-image-metadata-management.md) - Using Delta tables to store and manage image metadata, file paths, and inference results, enabling ACID transactions and schema enforcement for ML image workloads.
* [Delta Table History Retention](/concepts/delta-table-history-retention.md) - The period (30 days) for which Delta Lake retains write history for a table.
* [Delta table name restrictions for index commands](/concepts/delta-table-name-restrictions-for-index-commands.md) - The table name in DROP BLOOM FILTER INDEX must not include temporal or options specifications
* [Delta Table Partitioning Constraints](/concepts/delta-table-partitioning-constraints.md) - Delta Lake enforces that partition columns specified during a write must match the existing table's partition columns; a mismatch raises the PARTITIONING_MISMATCH error.
* [Delta Table Partitioning Mismatch](/concepts/delta-table-partitioning-mismatch.md) - An error subtype indicating that the partition columns provided during a write do not match the existing partition columns of the Delta table.
* [Delta Table Provenance](/concepts/delta-table-provenance.md) - Metadata captured for every write operation on a Delta table, including operation type, user, and timestamp.
* [Delta Table Repair](/concepts/delta-table-repair.md) - The recommended remediation for a non-contiguous Delta log, requiring contacting Databricks support to repair the table.
* [Delta table repair via Databricks support](/concepts/delta-table-repair.md) - The recommended remediation for a Delta table with non-contiguous log versions, requiring contacting Databricks support to repair the table.
* [Delta table schema requirements](/concepts/delta-table-schema-requirements.md) - The requirement that a Delta table must have at least one non-void column and any struct columns must have at least one non-void field to be writable.
* [Delta Table Streaming Start Position](/concepts/delta-table-streaming-start-position.md) - Using startingVersion and startingTimestamp options to define the initial read position for a streaming query on a Delta table
* [Delta table versioning](/concepts/delta-table-versioning.md) - The mechanism by which Delta Lake maintains sequential, atomic versions of a table through ordered commits in the transaction log.
* [Delta Tables with Primary Key as Feature Tables](/concepts/delta-tables-with-primary-key-as-feature-tables.md) - Any Delta table in Unity Catalog with a primary key constraint can serve as a feature table
* [Delta transaction log](/concepts/delta-transaction-log.md) - The commit log that records all changes made to a Delta table in sequential version order.
* [Delta transaction log archival constraint](/concepts/delta-transaction-log-archival-constraint.md) - Archiving any files in the _delta_log/ directory makes the entire Delta table inaccessible; cloud lifecycle policies must exclude the transaction log path from archival rules.
* [Delta transaction log archival prohibition](/concepts/delta-transaction-log-archival-prohibition.md) - The Delta transaction log (_delta_log/) must never be archived by cloud lifecycle policies, or the table becomes entirely inaccessible.
* [Delta transaction log archival restriction](/concepts/delta-transaction-log-archival-restriction.md) - The Delta transaction log (_delta_log/ directory) must never be archived; doing so makes the entire table inaccessible.
* [Delta Uniform](/concepts/delta-uniform.md) - A Databricks feature that enables Delta Lake tables to be readable as Apache Iceberg tables, requiring specific table properties like tableId, snapshotId, and metadataLocation.
* [Delta Uniform (Apache Iceberg Ingress)](/concepts/delta-uniform-apache-iceberg-ingress.md) - A Databricks feature that allows reading Apache Iceberg tables using Delta Lake via a uniform ingress table configuration
* [Delta Uniform (Databricks)](/concepts/delta-uniform-databricks.md) - A Databricks feature enabling Delta Lake tables to be read by other query engines through a unified format
* [Delta Uniform (Delta Lake Uniform)](/concepts/delta-uniform.md) - A Databricks feature that provides interoperability between Delta Lake and Apache Iceberg formats, allowing tables to be read using Iceberg readers while maintaining Delta Lake as the primary format.
* [Delta UniForm (Delta Universal Format)](/concepts/delta-uniform-universal-format.md) - A Databricks feature that enables Delta Lake tables to be read by Iceberg, Hudi, and other formats through automatic metadata conversion and refresh operations.
* [Delta Uniform (UniForm)](/concepts/delta-uniform.md) - A Databricks feature that allows Delta Lake tables to be read natively as Apache Iceberg or Apache Hudi tables without conversion.
* [Delta UniForm (Universal Format)](/concepts/delta-uniform-universal-format.md) - A Databricks Delta Lake feature enabling interoperability with other table formats like Apache Iceberg and Apache Hive by writing metadata in those formats alongside Delta metadata.
* [Delta Uniform (Universal Format) Compatibility](/concepts/delta-uniform-universal-format-compatibility.md) - A Databricks feature that enables Delta tables to be read by non-Delta engines through a compatibility format, requiring a dedicated empty directory location
* [Delta Uniform Compatibility Format](/concepts/delta-uniform-compatibility-format.md) - A Delta Lake feature in Databricks requiring a dedicated empty directory for compatibility format configuration.
* [Delta UniForm compatibility location requirements](/concepts/delta-uniform-compatibility-location-requirements.md) - The set of conditions an error location must satisfy for Delta UniForm compatibility: must exist, must be a directory, must not be blank, must be empty, and must be accessible.
* [Delta Uniform compatibility requirement](/concepts/delta-uniform-compatibility-requirement.md) - The delta.universalFormat.enabledFormats config must include 'compatibility' for REFRESH SYNC UNIFORM to work
* [Delta Uniform Delta Log Location](/concepts/delta-uniform-delta-log-location.md) - The delta_log directory within a Delta Uniform table that is critical for ingress operations; errors occur when it is missing, unexpected, or when the metadata path cannot be found.
* [Delta Uniform delta_log location errors](/concepts/delta-uniform-delta_log-location-errors.md) - Errors related to missing, unexpected, or incorrect delta_log location metadata when working with Delta Uniform ingress tables
* [Delta Uniform Iceberg Metadata Location File Naming Convention](/concepts/delta-uniform-iceberg-metadata-location-file-naming-convention.md) - The expected file naming convention for Apache Iceberg metadata files that must be followed by the Iceberg writer for Delta Uniform to correctly parse version information from metadata locations.
* [Delta Uniform Iceberg metadata versioning](/concepts/delta-uniform-iceberg-metadata-versioning.md) - The requirement that Iceberg metadata locations for a Delta Uniform table must follow a monotonically increasing version sequence, and refreshes must point to a higher version than the existing metadata.
* [Delta Uniform Ingress](/concepts/delta-uniform-ingress.md) - The pattern of reading or importing Apache Iceberg tables through Delta Uniform on Databricks, supporting only CREATE and REFRESH operations.
* [Delta Uniform Ingress Table](/concepts/delta-uniform-ingress.md) - A Databricks table concept that supports reading Apache Iceberg metadata via Delta Uniform, requiring Unity Catalog and supporting only CREATE and REFRESH operations.
* [Delta Uniform Metadata Conversion](/concepts/delta-uniform-metadata-conversion.md) - The process of converting metadata between source formats and Delta Lake during Delta Uniform ingress operations, which is the central failure point for the DELTA_UNIFORM_INGRESS_VIOLATION error.
* [Delta Uniform Metadata Conversion Failure](/concepts/delta-uniform-metadata-conversion-failure.md) - A failure scenario where converting source format metadata to Delta format fails during a Delta Uniform read operation, reported under the DELTA_UNIFORM_INGRESS_VIOLATION error class.
* [Delta Uniform metadata versioning constraints](/concepts/delta-uniform-metadata-versioning-constraints.md) - The requirement that Iceberg metadata locations being refreshed must have a strictly higher version number than the existing metadata location.
* [Delta Uniform Refresh](/concepts/delta-uniform-refresh.md) - A Databricks operation to synchronize or update the metadata of a Delta table's Uniform representation so external engines can read the latest data.
* [Delta Uniform Required Table Properties](/concepts/delta-uniform-required-table-properties.md) - The set of Delta table properties (tableId, snapshotId, metadataLocation) that must be present for Delta Uniform to function correctly when bridging between Delta Lake and Apache Iceberg.
* [Delta Uniform Sub-Error Conditions](/concepts/delta-uniform-sub-error-conditions.md) - A set of specific sub-errors under DELTA_UNIFORM_INGRESS_VIOLATION: DELTA_LOG_LOCATION_NOT_FOUND, NOT_UNIFORM_INGRESS_TABLE, OPERATION_NOT_SUPPORTED, UNEXPECTED_DELTA_LOG_LOCATION, and UNITY_CATALOG_NOT_ENABLED.
* [Delta Uniform Unity Catalog Requirement](/concepts/delta-uniform-unity-catalog-requirement.md) - Unity Catalog must be enabled to read Apache Iceberg tables using Delta Uniform; absence triggers the UNITY_CATALOG_NOT_ENABLED error.
* [Delta Uniform with IcebergCompat](/concepts/delta-uniform-with-icebergcompat.md) - The Uniform feature in Delta Lake enables Apache Iceberg compatibility, and upgrading or enabling it requires all existing files to be Iceberg-compatible, enforced via IcebergCompatV.
* [Delta Universal Format (Uniform)](/concepts/delta-universal-format-uniform.md) - A Delta Lake feature for format compatibility that requires a valid empty directory location configuration
* [Delta Unsupported Clustering Column Predicates Error Class](/concepts/delta_unsupported_clustering_column_predicates-error-class.md) - A Delta Lake error class (DELTA_UNSUPPORTED_CLUSTERING_COLUMN_PREDICATES) raised when a command on a Liquid table uses predicates that reference non-clustering columns, missing clustering columns, unresolved columns, or unsupported expressions.
* [DELTA_ALTER_TABLE_SET_MANAGED_FAILED](/concepts/delta_alter_table_set_managed_failed.md) - A Databricks error class that occurs when ALTER TABLE ... SET MANAGED fails, with multiple sub-error types indicating specific failure modes during external-to-managed table migration.
* [DELTA_ALTER_TABLE_SET_MANAGED_FAILED Error Class](/concepts/delta_alter_table_set_managed_failed.md) - Error class raised when ALTER TABLE SET MANAGED fails on a Delta table in Databricks
* [DELTA_ALTER_TABLE_SET_MANAGED_TABLE_NOT_MIGRATABLE](/concepts/delta_alter_table_set_managed_table_not_migratable.md) - A Databricks error raised when ALTER TABLE SET MANAGED fails to migrate a table to managed storage.
* [DELTA_ALTER_TABLE_SET_MANAGED_TABLE_NOT_MIGRATABLE error](/concepts/delta_alter_table_set_managed_table_not_migratable.md) - A Databricks error condition that occurs when ALTER TABLE SET MANAGED fails to migrate a table to managed storage
* [DELTA_ALTER_TABLE_SET_MANAGED_TABLE_NOT_MIGRATABLE error class](/concepts/delta_alter_table_set_managed_table_not_migratable.md) - A Databricks error class indicating that ALTER TABLE SET MANAGED failed to migrate the given table to a managed table.
* [DELTA_ALTER_TABLE_UNSET_MANAGED_FAILED](/concepts/delta_alter_table_unset_managed_failed.md) - Databricks error class raised when a managed Delta table cannot be rolled back to an external table
* [DELTA_ALTER_TABLE_UNSET_MANAGED_FAILED error class](/concepts/delta_alter_table_unset_managed_failed.md) - A Databricks error class (SQLSTATE 42809) raised when a table cannot be rolled back from managed to external in Unity Catalog.
* [DELTA_CANNOT_WRITE_EMPTY_SCHEMA](/concepts/delta_cannot_write_empty_schema.md) - A Delta Lake error raised when attempting to write to a table whose schema is empty or contains only unsupported (VOID) types.
* [DELTA_CANNOT_WRITE_EMPTY_SCHEMA error](/concepts/delta_cannot_write_empty_schema.md) - A Delta Lake error raised when a table schema is empty or consists entirely of unsupported VOID-type columns or empty structs, preventing writes.
* [DELTA_CANNOT_WRITE_EMPTY_SCHEMA error class](/concepts/delta_cannot_write_empty_schema.md) - A Databricks Delta Lake error raised when a table cannot be written because its schema is empty or composed entirely of VOID-type columns
* [DELTA_CLONE_INCOMPATIBLE_SOURCE](/concepts/delta_clone_incompatible_source.md) - A Databricks error class raised when attempting to clone a table into Delta format from a source that has valid format but unsupported features.
* [DELTA_CLONE_INCOMPATIBLE_SOURCE Error](/concepts/delta_clone_incompatible_source.md) - A Databricks error that occurs when cloning a Delta table from a source that has a valid format but unsupported features, including sub-classes like HAS_INDEXES, ICEBERG_MISSING_PARTITION_SPECS, and ICEBERG_UNDERGONE_PARTITION_EVOLUTION.
* [DELTA_CLONE_INCOMPATIBLE_SOURCE error class](/concepts/delta_clone_incompatible_source.md) - A Databricks error condition raised when a Delta clone operation fails because the source table has a valid format but uses unsupported features.
* [DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE error class](/concepts/delta_clone_with_history_unsupported_target.md) - A Databricks error class indicating that a Delta table CLONE operation with history is attempted on an unsupported source table
* [DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_TARGET](/concepts/delta_clone_with_history_unsupported_target.md) - A Databricks error raised when cloning a Delta table with history into an unsupported target table type.
* [DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_TARGET error class](/concepts/delta_clone_with_history_unsupported_target.md) - A Databricks error that occurs when attempting to clone a Delta table with history to an unsupported target type.
* [DELTA_CONCURRENT_APPEND Error](/concepts/delta_concurrent_append-error.md) - A Delta Lake error indicating a transaction conflict was detected because a concurrent operation added data to the table.
* [DELTA_CONCURRENT_APPEND Error Class](/concepts/delta_concurrent_append-error.md) - A Delta Lake error class indicating transaction conflicts when concurrent operations modify the same table or partition
* [DELTA_CONCURRENT_APPEND Error Condition](/concepts/delta_concurrent_append-error.md) - A Delta Lake transaction conflict detected when a concurrent operation adds data to a table, described by SQLSTATE 2D521 with multiple sub-conditions.
* [DELTA_CONCURRENT_DELETE_DELETE Error Class](/concepts/delta_concurrent_delete_delete-error-class.md) - A Databricks error class for transaction conflicts where a concurrent delete operation conflicts with another delete on the same Delta table.
* [DELTA_CONCURRENT_DELETE_READ Error](/concepts/delta_concurrent_delete_read-error.md) - A transaction conflict error in Delta Lake on Databricks triggered when a concurrent operation deletes data that another transaction is reading.
* [DELTA_CONCURRENT_DELETE_READ Error Class](/concepts/delta_concurrent_delete_read-error.md) - A Delta Lake transaction conflict error that occurs when a concurrent operation deletes data that another transaction has read, with multiple sub-types describing specific conflict scenarios.
* [DELTA_DEEP_CLONE_STREAMING_TABLE_ERROR](/concepts/delta_deep_clone_streaming_table_error.md) - An error class in Databricks that occurs when attempting to deep clone a streaming table, with multiple specific sub-error conditions.
* [DELTA_DUPLICATE_COLUMNS_FOUND](/concepts/delta_duplicate_columns_found.md) - A Delta Lake error class raised when duplicate column names are detected in a table operation, with sub-reasons identifying the specific operation context.
* [DELTA_DUPLICATE_COLUMNS_FOUND Error (Databricks)](/concepts/delta_duplicate_columns_found.md) - A Delta Lake error class raised when duplicate column names are detected during table operations, with sub-classifications for different operation contexts.
* [DELTA_DUPLICATE_COLUMNS_FOUND error class](/concepts/delta_duplicate_columns_found.md) - A Databricks Delta Lake error (SQLSTATE 42711) raised when duplicate column names are detected in a Delta operation, with multiple sub-condition variants pinpointing the exact operation context.
* [DELTA_DUPLICATE_COLUMNS_FOUND error condition](/concepts/delta_duplicate_columns_found.md) - A Delta Lake error (SQLSTATE 42711) raised when duplicate column names are detected during various table operations such as schema changes, data writes, or metadata updates.
* [DELTA_EXTERNAL_METADATA_UNSUPPORTED_SOURCE](/concepts/delta_external_metadata_unsupported_source.md) - A Databricks error condition raised when external metadata operations are attempted on unsupported Delta table sources or configurations.
* [DELTA_EXTERNAL_METADATA_UNSUPPORTED_SOURCE error](/concepts/delta_external_metadata_unsupported_source.md) - A Databricks error class raised when external metadata operations are attempted on unsupported Delta table features.
* [DELTA_ICEBERG_COMPAT_V1_VIOLATION](/concepts/delta_iceberg_compat_v1_violation.md) - A Delta Lake error class indicating that validation of IcebergCompatV1 has failed on a table operation
* [DELTA_ICEBERG_COMPAT_V1_VIOLATION error class](/concepts/delta_iceberg_compat_v1_violation.md) - A Delta Lake error class raised when IcebergCompatV1 validation fails on a table operation
* [DELTA_ICEBERG_COMPAT_VIOLATION error class](/concepts/delta_iceberg_compat_v1_violation.md) - A Databricks Delta Lake error class (SQLSTATE KD00E) raised when IcebergCompatV validation fails during table operations.
* [DELTA_ICEBERG_WRITER_COMPAT_VIOLATION](/concepts/delta_iceberg_writer_compat_violation.md) - A Databricks error class raised when Delta Lake write operations violate Iceberg-compatible writer constraints
* [DELTA_ICEBERG_WRITER_COMPAT_VIOLATION error class](/concepts/delta_iceberg_writer_compat_violation.md) - A Databricks error class raised when IcebergWriterCompatV validation fails on a Delta table, with multiple specific sub-error conditions.
* [DELTA_METADATA_MISMATCH Error Class](/concepts/delta_metadata_mismatch-error-class.md) - A Databricks error condition triggered when a metadata mismatch is detected during writes to a Delta table, with several specific subtypes.
* [DELTA_METADATA_MISMATCH Error Condition](/concepts/delta_metadata_mismatch-error-class.md) - A Databricks error condition (SQLSTATE 42KDG) raised when a metadata mismatch is detected while writing to a Delta table, with multiple subtypes for different root causes.
* [DELTA_SHARING_CURRENT_RECIPIENT_PROPERTY_UNDEFINED](/concepts/delta_sharing_current_recipient_property_undefined.md) - A Databricks error condition raised when a Delta Sharing recipient property referenced in a query does not apply to the current recipient in the session.
* [DELTA_SHARING_CURRENT_RECIPIENT_PROPERTY_UNDEFINED error](/concepts/delta_sharing_current_recipient_property_undefined.md) - A Databricks error raised when a Delta Sharing recipient property restricting data access does not apply to the current recipient session
* [DELTA_SHARING_CURRENT_RECIPIENT_PROPERTY_UNDEFINED Error Condition](/concepts/delta_sharing_current_recipient_property_undefined.md) - A Databricks error (SQLSTATE 42704) raised when a Delta Sharing query references a recipient property that does not apply to the current recipient in the session.
* [DELTA_SHARING_SECURABLE_DELETE_BLOCKED Error](/concepts/delta_sharing_securable_delete_blocked-error.md) - A Databricks error (SQLSTATE 55006) raised when a securable object cannot be deleted because it is actively shared via Delta Sharing (OpenSharing).
* [DELTA_SHARING_SECURABLE_DELETE_BLOCKED Error Class](/concepts/delta_sharing_securable_delete_blocked-error.md) - A Databricks error that prevents deletion of a securable object (e.g., table, view) that is currently being shared via Delta Sharing's OpenSharing protocol.
* [DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE](/concepts/delta_streaming_skip_offsets_invalid_offset_range.md) - A Databricks error class (SQLSTATE 42616) raised when a user-specified skip offset range for a Delta streaming source is invalid.
* [DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE Error Class](/concepts/delta_streaming_skip_offsets_invalid_offset_range.md) - A Databricks error class raised when an invalid skip offset range is specified for a Delta streaming source, with SQLSTATE 42616
* [DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED](/concepts/delta_table_changes_view_unsupported.md) - An error class in Databricks that occurs when table_changes is called on a view that does not meet the required criteria (single relation, deterministic, references a Delta table, etc.)
* [DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED error](/concepts/delta_table_changes_view_unsupported.md) - A Databricks error raised when the table_changes function is used on a view instead of a Delta table.
* [DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED error class](/concepts/delta_table_changes_view_unsupported.md) - A Databricks error indicating that the table_changes function cannot be applied to views, with multiple specific sub-reasons.
* [DELTA_UNIFORM_COMPATIBILITY_MISSING_OR_INVALID_LOCATION](/concepts/delta_uniform_compatibility_missing_or_invalid_location.md) - A Databricks error class raised when the Delta UniForm compatibility location configuration is missing, empty, inaccessible, or not a valid empty directory.
* [DELTA_UNIFORM_COMPATIBILITY_MISSING_OR_INVALID_LOCATION Error Class](/concepts/delta_uniform_compatibility_missing_or_invalid_location.md) - A Databricks error class raised when the delta.universalFormat.compatibility.location configuration is missing, blank, inaccessible, or points to an invalid path
* [DELTA_UNIFORM_ICEBERG_INGRESS_VIOLATION](/concepts/delta_uniform_iceberg_ingress_violation.md) - A Databricks error class that occurs when reading Apache Iceberg via Delta Uniform fails due to version parsing problems in metadata locations.
* [DELTA_UNIFORM_ICEBERG_INGRESS_VIOLATION error class](/concepts/delta_uniform_iceberg_ingress_violation.md) - A Databricks error class (SQLSTATE KD00E) that occurs when reading an Apache Iceberg table via Delta Uniform fails, typically due to metadata parsing failures or misconfigured table properties.
* [DELTA_UNIFORM_INGRESS_VIOLATION](/concepts/delta_uniform_ingress_violation.md) - An error class in Databricks raised when reading Delta Uniform tables fails due to metadata conversion or configuration issues.
* [DELTA_UNIFORM_INGRESS_VIOLATION Error Class](/concepts/delta_uniform_ingress_violation.md) - A Databricks error class (SQLSTATE KD00E) raised when reading a Delta Uniform table fails due to metadata conversion or ingress configuration issues.
* [DELTA_UNIFORM_REFRESH_INVALID_ARGUMENT Error](/concepts/delta_uniform_refresh_invalid_argument-error.md) - A Databricks error class raised when invalid arguments are provided during a Delta Uniform refresh operation
* [DELTA_UNIFORM_REFRESH_INVALID_ARGUMENT Error Class](/concepts/delta_uniform_refresh_invalid_argument-error.md) - A Databricks error condition raised when an invalid argument is passed to a Delta UniForm refresh operation on AWS.
* [DELTA_UNIFORM_REFRESH_INVALID_ARGUMENT Error Condition](/concepts/delta_uniform_refresh_invalid_argument-error.md) - A Databricks error class indicating an invalid argument was provided during a Delta Uniform refresh operation
* [DELTA_UNIFORM_REFRESH_NOT_SUPPORTED error class](/concepts/delta_uniform_refresh_not_supported-error-class.md) - A Databricks error class raised when REFRESH SYNC UNIFORM cannot be performed due to various incompatibilities
* [DELTA_UNSUPPORTED_CLUSTERING_COLUMN_PREDICATES error class](/concepts/delta_unsupported_clustering_column_predicates-error-class.md) - A Delta Lake error class raised when commands on Liquid tables use predicates that are not supported, involving clustering column restrictions.
* [DELTA_VERSIONS_NOT_CONTIGUOUS error](/concepts/delta_versions_not_contiguous-error.md) - A Databricks Delta Lake error that occurs when versions in the delta transaction log are not sequential, indicating a gap in the log.
* [DELTA_VERSIONS_NOT_CONTIGUOUS error condition](/concepts/delta_versions_not_contiguous-error.md) - A Delta Lake error that occurs when version numbers in the transaction log are not sequential, indicating a gap in the delta log.
* [DELTA_VIOLATE_TABLE_PROPERTY_VALIDATION_FAILED](/concepts/delta_violate_table_property_validation_failed.md) - A Delta Lake error class indicating that table property validation has been violated due to incompatible feature combinations
* [DELTA_VIOLATE_TABLE_PROPERTY_VALIDATION_FAILED Error Class](/concepts/delta_violate_table_property_validation_failed-error-class.md) - A Databricks error class (SQLSTATE 0A000) triggered when table property validation constraints are violated, with specific sub-cases related to deletion vectors and manifest generation.
* [Delta-Iceberg Column Mapping Field Physical Name Constraint](/concepts/delta-iceberg-column-mapping-field-physical-name-constraint.md) - A requirement of IcebergWriterCompatV that column mapping field physical names must follow the pattern 'col-[fieldId]' for Iceberg-compatible Delta writes.
* [Delta-Iceberg column mapping field physical name convention](/concepts/delta-iceberg-column-mapping-field-physical-name-constraint.md) - The requirement enforced by IcebergWriterCompatV that column mapping field physical names follow the pattern 'col-[fieldId]'
* [Delta-Iceberg Map Key Struct Restriction](/concepts/delta-iceberg-map-key-struct-restriction.md) - A restriction enforced by IcebergWriterCompatV that prevents changing map keys that are structs, because Iceberg does not support struct-typed map keys.
* [Delta-Iceberg Schema Compatibility Constraints](/concepts/delta-iceberg-schema-compatibility-constraints.md) - A set of restrictions on Delta table schemas when using IcebergCompatV, including unsupported data types, partition column types, and schema evolution rules.
* [Delta-Iceberg Table Feature Compatibility](/concepts/delta-lake-table-features-compatibility.md) - Constraints enforced by IcebergWriterCompatV requiring specific table features to be enabled or disabled, and preventing incompatible features from coexisting on a Delta table intended for Iceberg compatibility.
* [Delta-Iceberg Unsupported Data Types and Properties](/concepts/delta-iceberg-unsupported-data-types-and-properties.md) - The set of data types and Apache Iceberg table properties that IcebergWriterCompatV rejects, ensuring only Iceberg-compatible schema types and table configurations are used in Delta tables.
* [Delta-Iceberg Version Alignment and Metadata Tracking](/concepts/delta-iceberg-version-alignment-and-metadata-tracking.md) - Delta Lake and Iceberg table versions do not directly align by commit timestamp or version ID; metadata fields track the relationship, accessible via DESCRIBE EXTENDED or Catalog Explorer.
* [Delta-Iceberg writer compatibility constraints](/concepts/delta-iceberg-writer-compatibility-constraints.md) - The set of specific schema and table property constraints enforced by IcebergWriterCompatV, including field naming, data types, table features, and configuration requirements.
* [Delta-Uniform Managed Table Requirement](/concepts/delta-uniform-managed-table-requirement.md) - The IcebergCompatV feature can only be enabled on managed Delta tables, not external or unmanaged tables.
* [delta.deletedFileRetentionDuration Table Property](/concepts/deltadeletedfileretentionduration-table-property.md) - A Delta table property that controls the retention window for VACUUM, overriding the default 7-day threshold to support longer time travel capabilities.
* [delta.timeUntilArchived](/concepts/deltatimeuntilarchived.md) - A Delta table property that specifies the archival interval, telling Databricks to ignore files older than the given time period when querying.
* [delta.timeUntilArchived table property](/concepts/deltatimeuntilarchived.md) - A Delta table property that specifies the archival threshold, telling Databricks to ignore files older than a given duration.
* [delta.universalFormat.compatibility.location](/concepts/deltauniversalformatcompatibilitylocation.md) - A configuration property in Databricks Delta Lake that must point to an empty directory for Delta UniForm compatibility format to work.
* [delta.universalFormat.compatibility.location Configuration](/concepts/deltauniversalformatcompatibilitylocation.md) - A Databricks Delta Lake configuration property that must point to an empty directory for Uniform compatibility format
* [delta.universalFormat.enabledFormats configuration](/concepts/deltauniversalformatenabledformats-configuration.md) - A Spark configuration that controls which UniForm formats (Iceberg, Hudi) are enabled for a Delta table.
* [DeltaLog Version Mismatch](/concepts/deltalog-version-mismatch.md) - A condition where the internal managed DeltaLog and external DeltaLog have diverging versions, preventing table rollback
* [DeltaLog version mismatch during managed migration](/concepts/deltalog-version-mismatch-during-managed-migration.md) - A version conflict in the managed DeltaLog when a concurrent ALTER TABLE SET MANAGED command has already migrated the table, leading to a VERSION_MISMATCH error.
* [DeltaLog Version Mismatch in Managed Rollback](/concepts/deltalog-version-mismatch-in-managed-rollback.md) - A VERSION_MISMATCH error occurs when the managed DeltaLog and external DeltaLog versions diverge, often due to a concurrent ALTER TABLE UNSET MANAGED command succeeding first.
* [DeltaTableSource](/concepts/deltatablesource.md) - An ephemeral Python object defining how features are computed from a Delta table in Unity Catalog; supports filter_condition, transformation_sql, schema_json, and helper methods like from_sql() and to_dataframe().
* [DeltaTableSource and RequestSource](/concepts/deltatablesource-and-requestsource.md) - Two data source types for feature computation: DeltaTableSource reads from Unity Catalog Delta tables (with optional filtering and transformation), and RequestSource defines inference-time schema for pass-through features.
* [Dependency management for Databricks Connect Scala Utilities](/concepts/dependency-management-for-databricks-connect-scala-utilities.md) - Databricks Connect for Scala automatically declares a dependency on the databricks-dbutils-scala library, so users do not need to add it explicitly in build.sbt, pom.xml, or build.gradle.
* [Dependency Management for Databricks Model Serving](/concepts/dependency-management-for-databricks-model-serving.md) - Best practices for managing dependencies in model serving containers, including custom libraries, flash-attn, and version pinning.
* [Dependency Management in Databricks Connect](/concepts/dependency-management-in-databricks-connect.md) - The dual dependency model where application dependencies are installed locally while UDF dependencies must be managed separately on the Databricks cluster.
* [Dependency Management Strategies for Serverless JARs](/concepts/dependency-management-strategies-for-serverless-jars.md) - Three strategies to make libraries available on serverless compute: declare as provided, attach as environment libraries, or use JDBC connections for external databases.
* [Dependency Management with uv](/concepts/dependency-management-with-uv.md) - Python dependency specification for AI Runtime workloads using `environment.dependencies`, supporting pip-style specs, requirements files, wheels, index URLs, and install flags, all installed via the uv package manager.
* [Dependency Version Matching for Serverless Compute](/concepts/dependency-version-matching-for-serverless-compute.md) - JAR Scala and JDK versions must exactly match the serverless runtime versions; mismatches cause failures. Example uses Scala 2.13, JDK 17, Databricks Connect 17.3.
* [Deploy Code Pattern (recommended)](/concepts/deploy-code-pattern-recommended.md) - A model deployment pattern where training code is promoted through development, staging, and production environments, and the model is retrained in each environment using that environment's data.
* [Deploy Code, Not Models](/concepts/deploy-code-not-models.md) - A MLOps principle recommending that during ML development, code (rather than models) should be promoted from one environment to the next, ensuring all code undergoes code review and integration testing.
* [Deploy Models Pattern](/concepts/deploy-models-pattern.md) - A model deployment pattern where the model artifact is trained in development, then tested in staging and promoted to production without retraining, separating model artifact lifecycle from code lifecycle.
* [Deploying Code Instead of Models](/concepts/deploying-code-instead-of-models.md) - The MLOps Stacks philosophy of deploying the code that builds the model rather than the model artifact itself, simplifying retraining by automating model construction from source.
* [Deploying Non-Natively Supported Frameworks with MLflow](/concepts/deploying-non-natively-supported-frameworks-with-mlflow.md) - Using MLflow pyfunc to deploy models whose frameworks are not natively supported by MLflow's built-in flavors
* [Deployment Container and Dependencies](/concepts/deployment-container-and-dependencies.md) - Production-grade container built during deployment includes MLflow-captured or specified libraries; dependencies can be added via pip_requirements, conda_env, extra_pip_requirements, and code_path.
* [Deployment container and dependency packaging](/concepts/deployment-container-and-dependency-packaging.md) - Model Serving builds a production-grade container from the MLflow model log; native flavor models capture dependencies automatically while pyfunc models require explicit specification via pip_requirements, conda_env, extra_pip_requirements, or code_path parameters.
* [Deployment Job Approval Mechanism](/concepts/deployment-job-approval-mechanism.md) - An experimental human-in-the-loop approval process for deployment jobs that uses Unity Catalog tags to gate progression from evaluation to deployment, with automatic job repair on approval.
* [Deployment Job Auto-Triggering and Credential Security](/concepts/deployment-job-auto-triggering-and-credential-security.md) - Deployment jobs auto-trigger using the model owner's credentials when a new model version is created, with recommendations to use service principals to prevent privilege escalation.
* [Deployment Job Connection to Unity Catalog Models](/concepts/deployment-job-connection-to-unity-catalog-models.md) - The process of linking a deployment job to a Unity Catalog registered model via the MLflow client or UI, enabling automatic triggering when new model versions are created.
* [Deployment Job Template Notebooks](/concepts/deployment-job-template-notebooks.md) - Reference notebooks for evaluation, approval, deployment, and job creation that serve as customizable templates for building deployment jobs on Databricks.
* [Deployment Jobs for model lifecycle](/concepts/deployment-jobs-for-model-lifecycle.md) - Automated jobs that trigger on new model versions to manage evaluation, approval, and deployment workflows in Unity Catalog, replacing stage transitions from the Workspace Model Registry.
* [Deprecated Data (Unity Catalog)](/concepts/deprecated-data-unity-catalog.md) - System tags applied to data assets to mark them as outdated or unsuitable for new use in Unity Catalog.
* [Deprecation and Migration Guidance (Databricks)](/concepts/deprecation-and-migration-guidance-databricks.md) - Databricks documentation providing migration paths away from deprecated Bloom filter indexes toward recommended alternatives like predictive I/O and liquid clustering.
* [Deprecation Comments and Genie Code Integration](/concepts/deprecation-comments-and-genie-code-integration.md) - Mechanism to annotate deprecated Hive tables with structured comments, triggering strikethrough display and Quick Fix links in notebooks, plus Genie Code auto-replacement of table references.
* [Deprecation Notification Process](/concepts/deprecation-notification-process.md) - The multi-channel notification process Databricks follows when planning to remove a library, including release notes warnings, import-time notifications, and documentation updates.
* [Deprecation of MLlib Automated MLflow Tracking](/concepts/deprecation-of-mllib-automated-mlflow-tracking.md) - MLlib automated MLflow tracking is deprecated on Databricks Runtime 10.1 ML and above, disabled by default on 10.2 ML+, with a configuration escape hatch to re-enable the old behavior
* [Deprecation of sparkdl.xgboost in Favor of xgboost.spark](/concepts/deprecation-of-sparkdlxgboost-in-favor-of-xgboostspark.md) - The sparkdl.xgboost module is deprecated since Databricks Runtime 12.0 ML, with recommendation to migrate to the xgboost.spark module
* [DESC EXTENDED diagnostic command](/concepts/desc-extended-diagnostic-command.md) - A Databricks SQL command used to inspect a table's metadata and verify whether an ALTER TABLE SET MANAGED migration has already completed.
* [DESCRIBE HISTORY](/concepts/describe-history.md) - A Delta Lake SQL command that returns provenance information (operation, user, etc.) for each write to a Delta table.
* [DESCRIBE HISTORY Command](/concepts/describe-history-command.md) - Command used to retrieve the version history of a Delta table, enabling selection of a specific version for restore
* [DESCRIBE HISTORY syntax constraints](/concepts/describe-history-syntax-constraints.md) - The table name used with DESCRIBE HISTORY must not include a temporal specification or options specification
* [Detection Exclusion](/concepts/detection-exclusion.md) - A beta feature allowing users to exclude individual column detections, which removes existing tags, prevents reapplication, and provides feedback to improve future classification accuracy.
* [Detection Exclusion and Feedback Loop](/concepts/detection-exclusion.md) - A Beta feature allowing users to exclude individual column detections, which removes existing tags, prevents reapplication, and provides feedback to improve future classification accuracy.
* [Detection Exclusion Feedback Loop](/concepts/detection-exclusion.md) - A Beta feature allowing users to exclude individual column detections, which removes existing classification tags, prevents future reapplication, and provides feedback that improves the accuracy of future classification results.
* [Detection Exclusions](/concepts/detection-exclusion.md) - A beta feature that allows users to exclude individual column detections, removing tags and preventing reapplication while providing feedback to improve future classification accuracy.
* [Detection Exclusions in Data Classification](/concepts/detection-exclusion.md) - A Beta feature allowing users to exclude individual column detections, removing existing tags, preventing reapplication, and providing feedback to improve future classification accuracy.
* [DetectJailbreak Scorer](/concepts/detectjailbreak-scorer.md) - A Guardrails AI scorer for detecting jailbreak attempts in LLM interactions, configurable with custom sensitivity thresholds.
* [DetectPII Scorer](/concepts/detectpii-scorer.md) - A Guardrails AI scorer for detecting personally identifiable information such as credit card numbers, SSNs, and email addresses in LLM outputs.
* [Deterministic and error-safe expressions in policy UDFs](/concepts/deterministic-and-error-safe-expressions-in-policy-udfs.md) - Using DETERMINISTIC functions and error-safe alternatives (try_divide, try_cast, try_to_number) in policy UDFs enables the optimizer to cache results, apply constant folding, and safely rearrange query plans without risking side-channel data leakage.
* [Developer Annotations in MLflow UI](/concepts/developer-annotations-on-traces.md) - Adding developer feedback, ratings, and notes directly through the MLflow UI by creating assessments on trace spans with custom names, numeric values, and rationales
* [Developer Annotations on Traces](/concepts/developer-annotations-on-traces.md) - Interactive workflow for developers to add custom feedback assessments (e.g., accuracy scores) directly in the MLflow UI on individual trace spans.
* [Developer Annotations via MLflow UI](/concepts/developer-annotations-on-traces.md) - Interactive method for developers to add feedback scores and rationale directly in the MLflow UI's Assessments tab on individual trace spans.
* [Developer iteration workflow for scorers](/concepts/developer-iteration-workflow-for-scorers.md) - A 4-step workflow to rapidly iterate on code-based scorers without rerunning the entire AI application by generating, storing, and reusing traces.
* [Development and Production Workflow for dbt on Databricks](/concepts/development-and-production-workflow-for-dbt-on-databricks.md) - Recommended workflow of developing dbt projects against a Databricks SQL warehouse for testing and debugging, then running in production via the dbt task in a Databricks job.
* [Development Workflow for Distributed Training](/concepts/development-workflow-for-distributed-training.md) - Recommended workflow of first training a model on a single node, then adapting the code for distributed training using tools like HorovodRunner
* [Development Workflow for HorovodRunner](/concepts/development-workflow-for-horovodrunner.md) - The recommended development workflow on Databricks for preparing data and code before running distributed training with HorovodRunner.
* [Development-Production Unified Tracing](/concepts/development-production-unified-tracing.md) - A property of MLflow Tracing where instrumentation is written once and works consistently across development environments (IDE, notebooks) and production monitoring dashboards.
* [Development-time Annotation](/concepts/development-time-annotation.md) - Adding assessment labels (feedback or expectations) to MLflow traces during GenAI application development, before formal evaluation is set up.
* [DevOps for ML on Databricks](/concepts/devops-for-ml-on-databricks.md) - Production and automation practices including model serving, jobs orchestration, Git folders, declarative automation bundles, and Terraform infrastructure-as-code.
* [Difference Between ML, Deep Learning, and GenAI](/concepts/difference-between-ml-deep-learning-and-genai.md) - Conceptual boundaries between classic machine learning, deep learning, and generative AI, noting that deep learning and GenAI are technically types of ML with fuzzy boundaries.
* [Direct Evaluation](/concepts/direct-evaluation.md) - An evaluation mode where MLflow calls the GenAI app directly via a predict_fn, generates traces, and applies scorers in parallel to assess quality.
* [Direct Evaluation Mode](/concepts/direct-evaluation.md) - Evaluation mode where MLflow calls the GenAI app directly via a predict_fn to generate and evaluate traces, recommended for reusing scorers in production monitoring.
* [Direct Grant vs GRANT Policy Interaction](/concepts/direct-grant-vs-grant-policy-interaction.md) - The effective privileges on an object are the union of direct grants and applicable GRANT policies; mixing both for the same privilege complicates reasoning about access.
* [Direct Model Query](/concepts/direct-model-query.md) - The ability to bypass traffic splitting rules and query a specific served model directly on a multi-model endpoint using a dedicated invocation URL.
* [Directed and Undirected Graphs](/concepts/directed-and-undirected-graphs.md) - Classification of graphs based on whether edges represent one-way or two-way relationships between nodes
* [Directed vs Undirected Networks](/concepts/directed-vs-undirected-networks.md) - Classification of graphs based on edge directionality: directed graphs represent one-way relationships (e.g., follower/followee), while undirected graphs represent two-way relationships (e.g., coworkers).
* [DIRECTORY_NOT_EMPTY (Delta Uniform error)](/concepts/directory_not_empty-delta-uniform-error.md) - A sub-error of DELTA_UNIFORM_COMPATIBILITY_MISSING_OR_INVALID_LOCATION indicating the specified directory already contains files.
* [DIRECTORY_NOT_EMPTY sub-error (Delta Uniform Compatibility)](/concepts/directory_not_empty-sub-error-delta-uniform-compatibility.md) - A specific reason code under the DELTA_UNIFORM_COMPATIBILITY_MISSING_OR_INVALID_LOCATION error class indicating the configured directory must be empty
* [Disable legacy features setting](/concepts/disable-legacy-features-setting.md) - An account console setting that allows organizations to test the new UC-only default behavior before the September 30, 2026 deadline by disabling legacy features in new workspaces.
* [Disable Legacy Hive Metastore Access](/concepts/disable-legacy-hive-metastore-access.md) - Workspace admin setting to prevent users from bypassing Unity Catalog and directly accessing the legacy Hive metastore in Databricks.
* [Disable Legacy Hive Metastore Access (Workspace Setting)](/concepts/disable-legacy-hive-metastore-access.md) - A workspace admin setting that prevents users from bypassing Unity Catalog and directly accessing tables registered in the legacy Hive metastore.
* [disable_databricks_trace_archival API](/concepts/disable_databricks_trace_archival-api.md) - An MLflow API function used to stop archiving traces for a specific experiment.
* [Disabling Databricks Connect on a Cluster](/concepts/disabling-databricks-connect-on-a-cluster.md) - Disabling the Databricks Connect and Spark Connect services on a cluster by setting spark.databricks.service.server.enabled to false.
* [Disabling Direct Access to the Legacy Hive Metastore](/concepts/disabling-direct-access-to-the-legacy-hive-metastore.md) - After migrating tables to Unity Catalog, Databricks recommends explicitly disabling direct Hive metastore access (workspace-wide or per-cluster) to avoid unnecessary connections and resource consumption.
* [Disabling Legacy Features After Unity Catalog Migration](/concepts/disabling-legacy-features-after-unity-catalog-migration.md) - Post-migration steps to disable legacy DBFS root and mounts, the Hive metastore, and no-isolation shared compute resources to enforce Unity Catalog governance.
* [Disabling Legacy Hive Metastore Access in Databricks](/concepts/disable-legacy-hive-metastore-access.md) - A workspace admin setting that prevents users from bypassing Unity Catalog and accessing tables registered in the Hive metastore
* [Disabling MLflow Auto Tracing](/concepts/disabling-mlflow-auto-tracing.md) - Auto-tracing for Amazon Bedrock can be disabled globally by calling mlflow.bedrock.autolog(disable=True) or mlflow.autolog(disable=True).
* [Disabling MLflow Autologging](/concepts/disabling-mlflow-autologging.md) - The mechanism to globally disable MLflow auto-tracing for Ollama/OpenAI interactions by calling mlflow.openai.autolog(disable=True) or mlflow.autolog(disable=True).
* [DISABLING_REQUIRED_TABLE_FEATURE](/concepts/missing_required_table_feature.md) - Error sub-type indicating a required table feature for IcebergCompatV1 cannot be dropped without first disabling IcebergCompatV1
* [DISABLING_REQUIRED_TABLE_FEATURE (Delta Iceberg error)](/concepts/missing_required_table_feature.md) - A sub-error of DELTA_ICEBERG_COMPAT_V1_VIOLATION that occurs when attempting to drop a table feature required by IcebergCompatV1
* [DISABLING_REQUIRED_TABLE_FEATURE error](/concepts/missing_required_table_feature.md) - A sub-error of DELTA_ICEBERG_COMPAT_V1_VIOLATION indicating an attempt to drop a table feature that IcebergCompatV1 requires.
* [Disk Cache Compatibility](/concepts/disk-cache-compatibility.md) - The Databricks disk cache (and therefore CACHE SELECT) is applicable only to Delta tables and Parquet tables; views are supported but with expanded query restrictions.
* [Distributed Checkpointing with PyTorch DCP](/concepts/distributed-checkpointing-with-pytorch-dcp.md) - Saving and loading model and optimizer state during distributed training using PyTorch's distributed checkpoint (dcp) API, enabling fault tolerance and continued training or inference.
* [Distributed Data Parallel (DDP)](/concepts/distributed-data-parallel-ddp.md) - A parallelism technique for distributed training where the full model is replicated on each GPU and data batches are split across GPUs.
* [Distributed Data Parallel (DDP) on Databricks](/concepts/distributed-data-parallel-ddp-on-databricks.md) - PyTorch's standard multi-GPU data parallelism integrated with the serverless_gpu @distributed API for distributed training.
* [Distributed Data Parallel (DDP) Training](/concepts/distributed-data-parallel-ddp.md) - A parallelism technique for distributed training where the full model is replicated on each GPU and data batches are split across GPUs
* [Distributed data parallelism (DDP) on Databricks](/concepts/distributed-data-parallelism-ddp-on-databricks.md) - Training approach that shards mini-batches across multiple GPUs with synchronized gradients, automated via Databricks' @distributed decorator
* [Distributed Data Parallelism Training](/concepts/distributed-data-parallelism-training.md) - Training technique that distributes model training across multiple GPUs using data parallelism, handling gradient synchronization and batching across devices
* [Distributed Data Parallelism with @distributed Decorator](/concepts/distributed-data-parallelism-with-distributed-decorator.md) - Pattern for distributing training across multiple GPUs using Databricks' serverless_gpu library, handling data parallelism, environment variables, and process coordination
* [Distributed data-parallel fine-tuning of LLMs with Ray Train](/concepts/distributed-data-parallel-fine-tuning-of-llms-with-ray-train.md) - End-to-end pattern for fine-tuning a large language model (e.g., Qwen2.5-3B) using Ray Train's TorchTrainer across multiple GPUs with DDP, covering model loading, dataset preparation, training loop, and metric reporting.
* [Distributed Deep Learning Data Preparation on Databricks](/concepts/distributed-deep-learning-data-preparation-on-databricks.md) - Strategies for preparing data specifically for distributed deep learning training workloads on Databricks clusters.
* [Distributed deep learning on Databricks](/concepts/distributed-deep-learning-on-databricks.md) - The practice of scaling deep learning model training across multiple nodes using HorovodRunner within the Databricks platform.
* [Distributed Deep Learning Training on Databricks](/concepts/distributed-deep-learning-training-on-databricks.md) - Best practices and examples for developing and fine-tuning deep learning models at scale using distributed training on the Databricks platform.
* [Distributed execution architecture on Databricks Serverless GPU](/concepts/distributed-execution-architecture-on-databricks-serverless-gpu.md) - The underlying compute manager, runtime environment, and launcher components that orchestrate distributed training across multiple GPUs.
* [Distributed Featurization with Pandas UDFs](/concepts/distributed-featurization-with-pandas-udfs.md) - A pattern for scaling featurization computations across a Spark cluster by using pandas UDFs (including Scalar Iterator pandas UDFs) to apply pre-trained deep learning models to large datasets in a distributed manner.
* [Distributed Fine-tuning with Databricks Serverless GPU](/concepts/distributed-fine-tuning-with-databricks-serverless-gpu.md) - A training pattern that uses the @distributed decorator to automatically provision multiple H100 GPUs, handle data distribution, and orchestrate synchronized distributed training of LLMs.
* [Distributed Image Inference on Databricks](/concepts/distributed-image-inference-on-databricks.md) - Architecture and workflow for scaling deep learning model inference across large image collections using Spark, pandas UDFs, and DL frameworks like PyTorch and TensorFlow on Databricks.
* [Distributed Multi-GPU Training API](/concepts/distributed-multi-gpu-training-api.md) - A Databricks API (currently in Beta) for orchestrating training workloads across multiple GPUs, enabling larger-scale model training.
* [Distributed Multi-GPU Training for LLMs](/concepts/distributed-multi-gpu-training-for-llms.md) - Training approach that distributes LLM workloads across multiple GPUs using Databricks' distributed training API
* [Distributed Multi-GPU Training on Databricks](/concepts/distributed-multi-gpu-training-api.md) - A Beta API in AI Runtime for distributing training workloads across multiple GPUs, separate from the single-node preview.
* [Distributed PyTorch Training Development Workflow](/concepts/distributed-pytorch-training-development-workflow.md) - A recommended workflow pattern for distributed deep learning that starts with single-node training before adapting code to distributed execution using HorovodRunner
* [Distributed PyTorch Training on Databricks](/concepts/distributed-pytorch-training-on-databricks.md) - Approaches and best practices for running distributed PyTorch training workloads on Databricks clusters
* [Distributed PyTorch training with Horovod](/concepts/distributed-pytorch-training-with-horovod.md) - Technique for scaling PyTorch model training across multiple nodes using Horovod, demonstrated via an MNIST example on Databricks.
* [Distributed training adaptation workflow](/concepts/distributed-training-adaptation-workflow.md) - The recommended development workflow of first training a model on a single node, then adapting it using HorovodRunner for distributed training.
* [Distributed training API (Databricks)](/concepts/distributed-training-api-databricks.md) - A Beta API for multi-GPU distributed training workloads on Databricks, scoped to the Serverless GPU API.
* [Distributed training API for multi-GPU workloads](/concepts/distributed-training-api-for-multi-gpu-workloads.md) - Beta-stage API in Databricks AI Runtime that enables scaling training workloads across multiple GPUs and nodes using the Serverless GPU API.
* [Distributed Training Development Workflow](/concepts/distributed-training-development-workflow.md) - A four-step process for converting single-node PyTorch code to distributed training using TorchDistributor on Databricks notebooks.
* [Distributed training development workflow on Databricks](/concepts/distributed-training-development-workflow-on-databricks.md) - The recommended workflow pattern for developing and running distributed deep learning jobs on Databricks using HorovodRunner, including data preparation and notebook-based execution.
* [Distributed Training for Large Models and Data](/concepts/distributed-training-for-large-models-and-data.md) - A strategy on Databricks for training neural networks when model or data exceeds single-machine memory capacity, using distributed frameworks
* [Distributed Training Import Placement for Pickling](/concepts/distributed-training-import-placement-for-pickling.md) - Best practice of placing all imports inside the training function to avoid pickling errors when distributing code to executors
* [Distributed Training on AI Runtime](/concepts/distributed-training-on-ai-runtime.md) - A Beta API using the @distributed decorator from the serverless_gpu Python package that enables multi-GPU training on a single node with PyTorch DDP, FSDP, or DeepSpeed and minimal configuration.
* [Distributed Training on Databricks](/concepts/distributed-training-on-databricks.md) - Moving from single-node to distributed deep learning training using built-in tools like TorchDistributor, DeepSpeed, and Ray.
* [Distributed Training Strategies Comparison](/concepts/distributed-training-strategies-comparison.md) - A taxonomy of distributed training approaches including DDP (Data Distributed Parallel), FSDP (Fully Sharded Data Parallel), and DeepSpeed, each suited to different model sizes and use cases.
* [Distributed Training Tools on Databricks](/concepts/distributed-training-tools-on-databricks.md) - Built-in distributed training support via TorchDistributor, DeepSpeed, and Ray, plus adaptive hyperparameter tuning with Optuna.
* [Distributed Training with @distributed Decorator](/concepts/distributed-training-with-distributed-decorator.md) - A Python API (Beta) for multi-GPU distributed training on AI Runtime using PyTorch DDP, FSDP, or DeepSpeed via a @distributed decorator
* [Distributed training with Ray Train](/concepts/distributed-training-with-ray-train.md) - A distributed data-parallel fine-tuning pattern using Ray Train's TorchTrainer across 8 H100 GPUs on a single node, with one worker per GPU.
* [Distributed training with Ray Train on Databricks](/concepts/distributed-training-with-ray-train.md) - Distributed data-parallel fine-tuning using Ray Train's TorchTrainer across 8 H100 GPUs on a single node, with one worker per GPU.
* [Distributed training with TensorFlow 2](/concepts/distributed-training-with-tensorflow-2.md) - Approach to scaling TensorFlow model training across multiple workers using tf.distribute.Strategy
* [Distributed Training with the @distributed Decorator](/concepts/distributed-training-with-the-distributed-decorator.md) - A Databricks Serverless GPU Compute API that uses the @distributed decorator to orchestrate multi-GPU training across a specified number of accelerators, handling distributed setup coordination automatically.
* [Distributed Training with xgboost.spark](/concepts/distributed-training-with-xgboostspark.md) - Distributed XGBoost training on Spark using the num_workers parameter, with restrictions on autoscaling and MLflow autologging
* [Distributed Training Workflow for Notebooks](/concepts/distributed-training-workflow-for-notebooks.md) - A four-step workflow to convert single-node PyTorch training code for distributed execution using TorchDistributor on Databricks
* [Distributed Training Workloads on Databricks](/concepts/distributed-training-workloads-on-databricks.md) - Multi-GPU training workloads submitted and managed via the AI Runtime CLI or the in-notebook Python API (@distributed and @ray_launch).
* [Distributed XGBoost Training on Databricks](/concepts/distributed-xgboost-training-on-databricks.md) - Running XGBoost model training at scale across a Spark cluster using Scala on the Databricks platform
* [Distributed XGBoost Training on Databricks with Scala](/concepts/distributed-xgboost-training-on-databricks.md) - Using Scala API on Databricks to perform distributed training of XGBoost models
* [Distributed XGBoost Training with num_workers](/concepts/distributed-xgboost-training-with-num_workers.md) - Distributed training capability in sparkdl.xgboost using the num_workers parameter, which can be set to use all Spark task slots via sc.defaultParallelism
* [Distributed XGBoost training with Scala on Databricks](/concepts/distributed-xgboost-training-on-databricks.md) - Using XGBoost for distributed model training within Scala notebooks on Databricks, integrated with Apache Spark MLlib.
* [DistributedSampler Pattern for Sharded Data Loading](/concepts/distributedsampler-pattern-for-sharded-data-loading.md) - A PyTorch pattern using DistributedSampler with DataLoader to partition dataset batches across multiple GPU workers in a distributed training setup, with epoch-based shuffling via sampler.set_epoch().
* [DL Training Scenarios on Databricks](/concepts/dl-training-scenarios-on-databricks.md) - The three supported deep learning training scenarios on Databricks: single-node training, distributed hyperparameter tuning, and distributed training.
* [Document Extraction Accuracy Evaluation](/concepts/document-extraction-accuracy-evaluation.md) - Patterns for evaluating document extraction applications with guidelines judges, covering field extraction completeness, numerical handling, entity recognition, and document-type-specific criteria.
* [Documentation archive on Databricks](/concepts/documentation-archive-on-databricks.md) - The archive section of Databricks documentation containing legacy or older versions of machine learning and training guides.
* [Downstream Impact Metrics in Anomaly Detection](/concepts/downstream-impact-metrics-in-anomaly-detection.md) - Concept of measuring the impact of data quality issues by tracking the number of queries affected on unhealthy tables, used as a trigger threshold for alerts.
* [Downstream Impact Tracking in Data Quality](/concepts/downstream-impact-tracking-in-data-quality.md) - A metric (num_queries_on_affected_tables) that measures how many queries are affected by unhealthy tables, used to prioritize alerting based on business impact.
* [downstream_impact Struct](/concepts/downstream_impact-struct.md) - A struct column in the anomaly detection results table that captures downstream impact information for each scanned table.
* [Drift detection (data quality)](/concepts/drift-detection-data-quality.md) - A monitoring capability that identifies statistical drift between current data and a known baseline, between successive time windows, or in ML model inputs and predictions over time.
* [Drift Metrics](/concepts/drift-metrics.md) - Statistics produced by data profiling that quantify how a table's data distribution changes over time or relative to a baseline table, used to detect data and model drift.
* [Drift Metrics Table](/concepts/drift-metrics-table.md) - A table that stores statistics tracking changes in data distribution over time, enabling alerts on data drift.
* [Drift Metrics: Consecutive vs Baseline Drift](/concepts/drift-metrics-consecutive-vs-baseline-drift.md) - Two types of drift computed for monitoring data distribution changes: consecutive drift compares a window to the previous time window, while baseline drift compares a window to a baseline distribution from a baseline table.
* [DROP BLOOM FILTER INDEX Command](/concepts/drop-bloom-filter-index-syntax.md) - SQL syntax and parameters for dropping a Bloom filter index on a Delta table column in Databricks
* [DROP BLOOM FILTER INDEX syntax](/concepts/drop-bloom-filter-index-syntax.md) - SQL command syntax and parameters for dropping Bloom filter indexes on Delta tables in Databricks
* [Drop Published Online Table](/concepts/drop-published-online-table.md) - The drop_online_table() operation to delete a published table from an online store provider, along with irreversible warning about downstream dependencies and the suggested key-rotation safety practice.
* [Dropping Delta Tables with rm -rf](/concepts/dropping-delta-tables-with-rm-rf.md) - Using rm -rf to drop a Delta Lake table on S3 leads to stale data issues; Databricks recommends using proper table drop or replace operations instead.
* [DRY RUN mode](/concepts/dry-run-mode.md) - Preview mode for FSCK REPAIR TABLE that reports detected issues without making any changes
* [Dual Export (MLflow + OpenTelemetry)](/concepts/dual-export-mlflow-opentelemetry.md) - A mode where traces are sent simultaneously to both the MLflow Tracking Server and an OpenTelemetry Collector, enabled via the MLFLOW_ENABLE_DUAL_EXPORT environment variable.
* [Dual Run Model (Job Run + MLflow Run)](/concepts/dual-run-model-job-run-mlflow-run.md) - Each `air run` submission creates both a Databricks job run (execution tracking) and an MLflow run (experiment tracking), with retries creating new MLflow runs.
* [Duplicate column error scenarios in Delta Lake](/concepts/duplicate-column-scenarios-in-delta-lake.md) - The specific operational contexts — ADDING_COLUMNS, CLUSTER_BY, CONVERT_TO_DELTA, DATA, EXISTING_SCHEMA, PARTITION_COLUMNS, PARTITION_SCHEMA, READ_SCHEMA, REPLACING_COLUMNS, SPECIFIED_COLUMNS, TABLE_SCHEMA — in which duplicate column detection can trigger the DELTA_DUPLICATE_COLUMNS_FOUND error.
* [Duplicate column scenarios in Delta Lake](/concepts/duplicate-column-scenarios-in-delta-lake.md) - The specific operational contexts in which the DELTA_DUPLICATE_COLUMNS_FOUND error can be triggered, including adding columns, partitioning, schema operations, data reads/writes, and table conversions.
* [Dynamic ABAC Policy Evaluation](/concepts/dynamic-abac-policy-evaluation.md) - ABAC policies evaluate at query time based on user identity, group memberships, and object tags, requiring tools like SHOW EFFECTIVE POLICIES and documentation for transparency.
* [Dynamic Partition Overwrites with partitionOverwriteMode](/concepts/dynamic-partition-overwrites-with-partitionoverwritemode.md) - Legacy approach using spark.sql.sources.partitionOverwriteMode=dynamic to replace only affected partitions during INSERT OVERWRITE operations.
* [Dynamic Policy Evaluation and Transparency in ABAC](/concepts/dynamic-policy-evaluation-and-transparency-in-abac.md) - ABAC policies evaluate at query time based on user identity, group memberships, and object tags; tools like SHOW EFFECTIVE POLICIES help data consumers understand which rules apply to a given table.
* [Dynamic Policy Evaluation in ABAC](/concepts/dynamic-policy-evaluation-in-abac.md) - ABAC policies evaluate at query time based on user identity, group memberships, and object tags, making transparency a challenge
* [Dynamic Scorer Creation](/concepts/dynamic-scorer-creation.md) - DeepEval scorers can be created dynamically at runtime using get_scorer() by passing the metric name as a string, enabling flexible evaluation pipelines.
* [Dynamic Scorer Creation via get_scorer](/concepts/dynamic-scorer-creation.md) - DeepEval scorers can be created dynamically at runtime by passing the metric name as a string to the get_scorer function.
* [Dynamic Scorer Creation with get_scorer](/concepts/dynamic-scorer-creation-with-get_scorer.md) - A method to dynamically create Guardrails AI scorers at runtime by passing the validator name as a string to the get_scorer function.
* [Dynamic Scorer Factory (get_scorer)](/concepts/dynamic-scorer-factory-get_scorer.md) - MLflow's get_scorer() function enables dynamic creation of DeepEval scorers by passing the metric name as a string parameter.
* [Dynamic Scorer Resolution in MLflow](/concepts/dynamic-scorer-resolution-in-mlflow.md) - MLflow's get_scorer factory function allows creating DeepEval scorers dynamically by passing a metric name as a string parameter.
* [Dynamic Views for Column/Row-Level Permissions](/concepts/dynamic-views-for-columnrow-level-permissions.md) - Using dynamic view functions (current_user() and is_member()) to implement column-level permissions, row-level permissions, and data masking in Hive metastore-managed views.
* [Dynamic Views for Fine-Grained Access](/concepts/dynamic-views-for-fine-grained-access.md) - Dynamic views can be created to restrict MLflow system table access to specific experiment IDs or other custom criteria, providing finer-grained control than granting full table access.
* [Dynamic Views for Fine-Grained Access Control](/concepts/dynamic-views-for-fine-grained-access-control.md) - Using dynamic views with is_member() and current_user() functions in view definitions to enforce column-level, row-level, and data-masking permissions in the Hive metastore.
* [Dynamic Views for Recipient-Scoped Access Control](/concepts/dynamic-views-for-recipient-scoped-access-control.md) - Using dynamic views with the CURRENT_RECIPIENT() function to configure fine-grained access control at row and column levels based on recipient properties. Enables data masking and security by restricting recipient access according to properties specified in the recipient definition. Requires Databricks Runtime 14.2+.
* [Dynamic Views with CURRENT_RECIPIENT()](/concepts/dynamic-views-with-current_recipient.md) - Using dynamic views with the CURRENT_RECIPIENT() SQL function to enforce row-level and column-level security based on recipient properties, enabling fine-grained access control in shared data.
* [DynamoDB Permission Requirements for Databricks Feature Store](/concepts/dynamodb-permission-requirements-for-databricks-feature-store.md) - The comprehensive set of DynamoDB IAM permissions (e.g., DeleteItem, PutItem, Query, Scan) that an instance profile or IAM user must have to publish features to a DynamoDB online store.
* [DynamoDB Permissions for Feature Store Operations](/concepts/dynamodb-permissions-for-feature-store-operations.md) - Complete set of 24 required DynamoDB IAM permissions needed for publishing feature tables to third-party DynamoDB online stores via Databricks Feature Store.
* [DynamoDB Permissions Required for Feature Store Operations](/concepts/dynamodb-permissions-for-feature-store-operations.md) - Complete list of DynamoDB IAM permissions needed for publishing feature tables and looking up features from DynamoDB online stores.
* [DynamoDB Required IAM Permissions for Feature Store](/concepts/dynamodb-required-iam-permissions-for-feature-store.md) - The complete set of IAM permissions (dynamodb: actions) required for write authentication when publishing feature tables to DynamoDB online stores from Databricks.
* [DynamoDB Required Permissions for Feature Store](/concepts/dynamodb-required-permissions-for-feature-store.md) - The comprehensive list of DynamoDB IAM permissions (e.g., PutItem, GetItem, CreateTable, Query) required for a Databricks Feature Store integration with DynamoDB as an online store.
* [Early failure and error reporting for archived queries](/concepts/early-failure-and-error-reporting-for-archived-queries.md) - Archival support enables queries that need archived files to fail early with descriptive errors, reducing wasted compute.
* [Early failure for archived queries](/concepts/early-failure-for-archived-queries.md) - A behavior where queries requiring archived data fail immediately with clear error messages, avoiding wasted compute and enabling rapid recovery via file restoration.
* [Early Failure for Queries Requiring Archived Data](/concepts/early-failure-for-queries-requiring-archived-data.md) - A Databricks optimization that causes queries needing archived files to fail immediately with clear error messages, rather than waiting for unavailable data.
* [Early Failure on Archived Queries](/concepts/early-failure-on-archived-queries.md) - An optimization where queries that must scan archived files fail early with clear error messages instead of running into missing-data issues, reducing wasted compute.
* [Effective Policy Evaluation in Unity Catalog](/concepts/effective-policy-evaluation-in-unity-catalog.md) - Policies applied at a catalog or schema scope are inherited by child objects (tables); SHOW EFFECTIVE POLICIES reveals all applicable policies including those from parent scopes, enabling table admins to see rules without requiring parent-level read access.
* [Effective Policy Visibility via SHOW EFFECTIVE POLICIES](/concepts/effective-policy-visibility-via-show-effective-policies.md) - The ability to view all policies affecting a securable object including inherited policies from parent scopes (catalog → schema → table) without requiring permissions on parent objects.
* [Effective Privileges Union Model](/concepts/effective-privileges-union-model.md) - The principle that effective privileges on a securable object are the union of direct grants and any applicable GRANT policies, meaning a more selective policy does not override broader direct grants.
* [Effects of Disabling Legacy Hive Metastore](/concepts/effects-of-disabling-legacy-hive-metastore.md) - The observable changes in a Databricks workspace after disabling legacy Hive metastore access, including job failures, UI changes, and SQL command impacts.
* [Electricity Consumption Forecasting](/concepts/electricity-consumption-forecasting.md) - Use case of applying probabilistic time-series forecasting to predict electricity demand
* [Embarrassingly Parallel Image Inference](/concepts/embarrassingly-parallel-image-inference.md) - Characterization of image model inference as an embarrassingly parallel workload that can be easily distributed across Spark cluster nodes for near-linear scaling.
* [Embedding Models for RAG](/concepts/text-embedding-models.md) - Text embedding models like GTE, BGE, and Qwen3-Embedding that map text to dense vectors for semantic search, retrieval, and RAG use cases, available alongside generative LLMs in Foundation Model APIs.
* [Embedding Models Response Format](/concepts/embedding-models-response-format.md) - Standard JSON response structure for embedding model queries including object type, embedding vector, model name, and token usage metadata.
* [Embedding Normalization Check](/concepts/embedding-normalization-check.md) - A technique to verify whether embeddings generated by a model are normalized by checking if the L2 norm is approximately equal to 1.
* [Embeddings API](/concepts/embeddings-api.md) - An API endpoint on Databricks for mapping input strings into embedding vectors, supporting batched inputs and model-specific instruction strings.
* [Embeddings API (Databricks)](/concepts/embeddings-api.md) - API for mapping input strings into embedding vectors with batching support, where instructions are model-specific (e.g., BGE, Qwen3-Embedding, Instructor-XL).
* [Empty DataFrame Creation Without sc.emptyRDD](/concepts/empty-dataframe-creation-without-scemptyrdd.md) - Creating empty DataFrames by passing an empty ArrayList (Scala) or empty list (Python) to spark.createDataFrame instead of using sc.emptyRDD.
* [Empty Read Predicates and Whole Table Scan Conflicts](/concepts/empty-read-predicates-and-whole-table-scan-conflicts.md) - A conflict scenario where a transaction reads an entire table without filters while a concurrent operation deletes data, requiring filter predicates to narrow the read scope and resolve the conflict.
* [Empty Source Query Semantics in Delta Overwrites](/concepts/empty-source-query-semantics-in-delta-overwrites.md) - Behavioral differences between REPLACE WHERE (may delete data), REPLACE USING (no deletion), and REPLACE ON (no deletion) when the source query produces zero rows.
* [enable_databricks_trace_archival API](/concepts/enable_databricks_trace_archival-api.md) - An MLflow API function used to begin archiving traces for an experiment by specifying a target Unity Catalog Delta table and optional experiment ID.
* [ENABLE_FEATURE_TRACING for Inference Tables](/concepts/feature-tracing-for-inference-tables.md) - A special environment variable that logs automatic feature lookup DataFrames to inference tables on a model serving endpoint, requiring MLflow 2.14.0 or above.
* [ENABLE_LIQUID sub-error](/concepts/enable_liquid-sub-error.md) - A DELTA_METADATA_MISMATCH sub-error that occurs when trying to enable liquid clustering on an existing table without using overwrite mode
* [Enabling Liquid Clustering on Existing Delta Tables](/concepts/enabling-liquid-clustering-on-existing-delta-tables.md) - The requirement to use overwrite mode with overwriteSchema option to enable clustering on an existing Delta table.
* [Encryption of Classification Metadata](/concepts/encryption-of-classification-metadata.md) - Custom classifier configuration and detection metadata are encrypted at rest, and a customer-managed key (CMK) on the system catalog can be used to manage the encryption key.
* [Encryption of custom classifier configuration](/concepts/encryption-of-custom-classifier-configuration.md) - Custom classifier metadata and detection patterns are encrypted at rest, with optional customer-managed key (CMK) support via the system catalog.
* [End User Feedback Collection Patterns](/concepts/end-user-feedback-collection-patterns.md) - Two approaches for linking user feedback to GenAI traces: using MLflow's auto-generated trace IDs or custom client request IDs.
* [End-to-End Forecasting Pipeline](/concepts/end-to-end-forecasting-pipeline.md) - A complete workflow for time-series forecasting covering data ingestion, resampling, model training, prediction, visualization, and evaluation stages.
* [End-to-end forecasting workflow](/concepts/end-to-end-forecasting-pipeline.md) - A complete pipeline for time series forecasting covering data ingestion, resampling, training, prediction, visualization, and evaluation
* [End-to-End Forecasting Workflow on Databricks](/concepts/end-to-end-forecasting-workflow-on-databricks.md) - Complete pipeline from data ingestion to visualization and evaluation for time-series forecasting on Databricks
* [End-to-End Latency Components](/concepts/end-to-end-latency-components.md) - Total latency for a model inference request consists of two main parts: network overhead (communication time) and model execution time (inference computation), both of which must be accounted for when setting realistic performance requirements.
* [End-to-End ML Workflow on Databricks](/concepts/end-to-end-ml-workflow-on-databricks.md) - A complete machine learning pipeline pattern covering data ingestion, preprocessing, model training, prediction, visualization, and evaluation within the Databricks environment.
* [End-to-End Trace Recording](/concepts/end-to-end-trace-recording.md) - The practice of capturing complete execution flows including inputs, outputs, intermediate steps, and metadata for GenAI applications.
* [End-User Feedback Collection via SDK](/concepts/end-user-feedback-collection-via-sdk.md) - Using mlflow.log_feedback() to programmatically collect end-user feedback (like thumbs up/down) and associate it with specific traces for quality monitoring
* [Endpoint Configuration for Load Testing](/concepts/endpoint-configuration-for-load-testing.md) - Specific configuration recommendations for starting load tests: Small CPU endpoint with concurrency of 4, route optimization enabled, and scale to zero disabled
* [Endpoint Configuration Lifecycle](/concepts/endpoint-configuration-lifecycle.md) - The process of creating, modifying, and updating Databricks model serving endpoints, including immutable properties, in-progress update cancellation, and safe rollback to previous configuration on failure.
* [Endpoint Configuration Restrictions](/concepts/endpoint-configuration-restrictions.md) - Constraints on endpoint configuration updates: external_model endpoints can only have one served entity, and the external_model type cannot be added or removed after creation.
* [Endpoint Configuration Updates and Rollback Behavior](/concepts/endpoint-configuration-updates-and-rollback-behavior.md) - Process of modifying an existing endpoint's compute configuration; old configuration continues serving traffic until new config is ready, with cancel capability
* [Endpoint Creation Interfaces](/concepts/endpoint-creation-interfaces.md) - Three options for creating model serving endpoints: Serving UI (graphical), REST API, and MLflow Deployments SDK (programmatic).
* [Endpoint Creation, Update, and Maintenance Expectations](/concepts/endpoint-creation-update-and-maintenance-expectations.md) - Key operational characteristics: ~10-minute deployment time, zero-downtime updates by keeping old config until new one is ready, 597-second request timeout, and occasional zero-downtime system maintenance where models are reloaded.
* [Endpoint Health Metrics](/concepts/endpoint-health-metrics.md) - Infrastructure metrics including latency, request rate, error rate, CPU usage, and memory usage for model serving endpoints.
* [Endpoint Lifecycle and Instance Profile Staleness](/concepts/endpoint-lifecycle-and-instance-profile-staleness.md) - Changes to an instance profile IAM role or deletion of the profile from Databricks settings do not affect running endpoints until the endpoint is updated or restarted.
* [Endpoint Lifecycle Management](/concepts/endpoint-lifecycle-management.md) - Operations to stop, start, and delete model serving endpoints, including constraints on when each operation is allowed.
* [Endpoint Sizing and Concurrency Planning](/concepts/endpoint-sizing-and-concurrency-planning.md) - Methodology for testing on small endpoints and scaling up based on linear relationships between concurrency, latency, and RPS to determine production endpoint size.
* [Endpoint Sizing from Load Test Results](/concepts/endpoint-sizing-from-load-test-results.md) - A methodology that uses a small concurrency endpoint for load testing, then calculates the production endpoint size needed to meet target requests-per-second and latency requirements based on observed metrics.
* [Endpoint Sizing via Load Test Results](/concepts/endpoint-sizing-via-load-test-results.md) - Methodology for calculating required endpoint concurrency from load test results by correlating client-side concurrency with latency percentiles and RPS targets to recommend final endpoint size.
* [Endpoint Telemetry Configuration](/concepts/endpoint-telemetry-configuration.md) - Process for enabling telemetry on custom model serving endpoints through either the Databricks UI or REST API, including specifying Unity Catalog destination and table prefix.
* [Endpoint Telemetry Limitations and Constraints](/concepts/endpoint-telemetry-limitations-and-constraints.md) - Boundaries of the telemetry feature including no schema evolution, managed Delta tables only, region restrictions, at-least-once delivery, size limits, and QPS thresholds.
* [Endpoint Update and Configuration Constraints](/concepts/endpoint-update-and-configuration-constraints.md) - Rules governing updates to model serving endpoints, including compute configuration changes, the inability to add/remove external_model from an endpoint, and cancellation of in-progress updates.
* [Endpoint Update Semantics](/concepts/endpoint-update-semantics.md) - Rules governing how model serving endpoints can be updated, including constraints on external_model presence and configuration change handling.
* [Enforce Table Access Control](/concepts/enforce-table-access-control.md) - The practice of restricting users to clusters with table access control enabled by denying cluster creation permissions and CAN ATTACH TO permissions for non-compliant clusters.
* [Enforcing Table Access Control in Databricks](/concepts/enforcing-table-access-control-in-databricks.md) - Workspace-level enforcement mechanisms to ensure users only access data through clusters that have table access control enabled.
* [Enforcing Table Access Control via Cluster Permissions](/concepts/enforcing-table-access-control-via-cluster-permissions.md) - The practice of denying users the ability to create clusters or attach to clusters without table access control enabled, to ensure data access restrictions are consistently applied.
* [Enforcing Table Access Control via Workspace Settings](/concepts/enforcing-table-access-control-via-workspace-settings.md) - Workspace-level administrative controls to enable table access control and restrict users to secured clusters with table access control enabled.
* [Entity Columns vs Filter Conditions](/concepts/entity-columns-vs-filter-conditions.md) - A design pattern decision guide for choosing between entity columns (for different aggregation levels) and filter_condition on DeltaTableSource (for row filtering at the same aggregation level).
* [Entity vs Filter Condition Decision Guide](/concepts/entity-vs-filter-condition-design-pattern.md) - Design pattern distinguishing when to use entity columns for different aggregation levels vs filter conditions for row-level filtering at the same aggregation level
* [Entity vs Filter Condition Design Pattern](/concepts/entity-vs-filter-condition-design-pattern.md) - Decision guide for structuring feature definitions: use entity columns for different aggregation levels (e.g., customer-level vs customer-merchant) and filter_condition for filtering rows within the same aggregation level.
* [Entra ID Configuration for Delta Sharing Recipients](/concepts/entra-id-configuration-for-delta-sharing-recipients.md) - Configuration steps for Microsoft Entra ID users to provide issuer, subject claim, subject, and audience values for OIDC federation setup
* [Entry Points in Python Wheels](/concepts/entry-points-in-python-wheels.md) - A mechanism in setup.py that maps a named entry point to a specific function in the package, used to define which code runs when a wheel task executes.
* [env_pack Parameter](/concepts/env_pack-parameter.md) - A parameter used with mlflow.register_model() to trigger packaging of model artifacts and the serverless notebook environment during model registration for express deployments.
* [Environment and Version Tracking in MLflow Traces](/concepts/environment-and-version-tracking-in-mlflow-traces.md) - Tracking execution environment (dev/staging/prod) and application version metadata for debugging and performance analysis across deployments
* [Environment and Version Tracking in Traces](/concepts/environment-and-version-tracking-in-traces.md) - Capturing execution environment (development, staging, production) and application version metadata via environment variables and standardized MLflow fields like mlflow.source.type.
* [Environment Packaging (env_pack)](/concepts/environment-packaging-env_pack.md) - A mechanism in MLflow's register_model function that packages and stages model artifacts along with the serverless notebook environment during model registration, enabling express deployments.
* [Environment Variable Configuration for Databricks Connect](/concepts/environment-variable-configuration-for-databricks-connect.md) - Using environment variables such as DATABRICKS_CLUSTER_ID, DATABRICKS_HOST, DATABRICKS_TOKEN, etc. to configure Databricks Connect without code changes.
* [Environment Variable Injection for App Resources](/concepts/environment-variable-injection-for-app-resources.md) - Databricks Apps automatically injects resource metadata (such as experiment ID) into the app runtime via environment variables, configured using the valueFrom field in app.yaml.
* [Environment Variable Injection for Databricks App Resources](/concepts/environment-variable-injection-for-app-resources.md) - The mechanism by which Databricks exposes resource identifiers (such as experiment IDs) as environment variables in deployed apps, accessible via the valueFrom field in app.yaml.
* [Environment Variable Injection from App Resources](/concepts/environment-variable-injection-for-app-resources.md) - The mechanism by which Databricks exposes resource IDs (e.g., MLflow experiment IDs) as environment variables to app code via the 'valueFrom' field in app.yaml configuration.
* [Environment Variable Injection from Databricks App Resources](/concepts/environment-variable-injection-from-databricks-app-resources.md) - When a Databricks App is deployed with a resource like an MLflow experiment, the resource ID is exposed via environment variables for use in application code.
* [Environment variables for Databricks Connect authentication](/concepts/environment-variables-for-databricks-connect-authentication.md) - Using environment variables (DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET, DATABRICKS_CLUSTER_ID) to configure connections without code-level config.
* [EnvPackConfig](/concepts/envpackconfig.md) - A configuration class with an install_dependencies parameter that controls whether model dependencies are installed during the env_pack process to validate the environment.
* [Ephemeral Service Logs](/concepts/ephemeral-service-logs.md) - Real-time stdout/stderr logs from model serving endpoints, useful for debugging during deployment.
* [Equal Opportunity](/concepts/equal-opportunity.md) - A fairness metric that measures whether a label is predicted equally well for both protected and unprotected groups (true positive rate parity)
* [Error handling and chaining in MLflow scorers](/concepts/error-handling-and-chaining-in-mlflow-scorers.md) - Patterns for handling scorer errors gracefully via AssessmentError and try/except, and for chaining evaluation results to iteratively refine datasets and applications.
* [Error handling in code-based scorers](/concepts/error-handling-in-scorers.md) - Two approaches for handling scorer errors: letting exceptions propagate for automatic capture, or handling exceptions explicitly with Feedback objects and AssessmentError.
* [Error handling in Databricks notebook workflows](/concepts/error-handling-in-databricks-notebook-workflows.md) - Strategies for handling errors and implementing retry logic when using dbutils.notebook.run(), which throws WorkflowException on failure.
* [Error handling in MLflow scorers](/concepts/error-handling-in-scorers.md) - Two approaches for handling scorer errors: letting exceptions propagate (recommended) for automatic capture, or handling exceptions explicitly with Feedback and AssessmentError.
* [Error handling in scorers](/concepts/error-handling-in-scorers.md) - Two approaches for handling scorer errors: letting exceptions propagate (recommended) or handling exceptions explicitly with AssessmentError and Feedback
* [Error handling patterns for MLflow scorers](/concepts/error-handling-patterns-for-mlflow-scorers.md) - Approaches for handling failures in custom scorers: returning AssessmentError via Feedback for graceful degradation, or raising exceptions to let MLflow handle errors, ensuring evaluation continues even when some scorers fail.
* [Error handling patterns for scorers](/concepts/error-handling-in-scorers.md) - MLflow supports two error handling approaches: letting exceptions propagate (recommended) or catching exceptions and returning Feedback with AssessmentError details.
* [Error handling strategies for MLflow scorers](/concepts/error-handling-in-scorers.md) - Two approaches for handling errors in custom scorers: returning structured AssessmentError within a Feedback object, or letting exceptions propagate for MLflow to handle gracefully
* [Error Monitoring with MLflow Traces](/concepts/error-monitoring-with-mlflow-traces.md) - Pattern for detecting and analyzing errors in production GenAI traces using mlflow.search_traces() with status and time-window filters.
* [Evaluation Chaining and Conditional Guidelines](/concepts/evaluation-chaining-and-conditional-guidelines.md) - A workflow pattern where evaluation results are reused to filter problematic traces and re-evaluate with different scorers or guidelines based on context.
* [Evaluation Dataset](/concepts/evaluation-dataset.md) - A curated, versioned collection of test cases with inputs and optional ground truth expectations, used for systematically testing and validating a GenAI application's quality.
* [Evaluation Dataset Constraints](/concepts/evaluation-dataset-limitations.md) - Technical limitations for MLflow evaluation datasets including a maximum of 2000 rows per dataset, 20 expectations per record, and incompatibility with CMK-encrypted Unity Catalog catalogs.
* [Evaluation Dataset for GenAI Agents](/concepts/evaluation-dataset-for-genai-agents.md) - Dataset structure for evaluation containing inputs (conversation messages), optional expectations, and test cases for scoring GenAI agent responses
* [Evaluation Dataset for GenAI Applications](/concepts/evaluation-dataset-for-genai-applications.md) - Structured input datasets used in MLflow evaluation, containing template-based examples with multiple test cases for assessing GenAI app performance.
* [Evaluation Dataset for Prompts](/concepts/evaluation-dataset-for-prompts.md) - Structured dataset containing input examples and expected facts used to objectively measure prompt quality and fact coverage across different versions.
* [Evaluation Dataset Limitations](/concepts/evaluation-dataset-limitations.md) - Known constraints on MLflow evaluation datasets including a maximum of 2000 rows per dataset, 20 expectations per record, and incompatibility with CMK-encrypted catalogs.
* [Evaluation Dataset Schema](/concepts/evaluation-dataset-schema.md) - The structured schema for MLflow evaluation datasets, defining core fields like inputs, expectations, and lineage metadata for GenAI app evaluation.
* [Evaluation Dataset SDK](/concepts/evaluation-dataset-sdk.md) - The programmatic Python API (mlflow.genai.datasets) for creating, retrieving, and deleting evaluation datasets, including the EvaluationDataset class.
* [Evaluation Dataset Source Lineage](/concepts/dataset-source-lineage.md) - A field tracking the origin of each dataset record, supporting exactly one source type: human, document, or trace.
* [Evaluation Dataset Storage in Unity Catalog](/concepts/evaluation-dataset-storage-in-unity-catalog.md) - MLflow evaluation datasets are stored as tables in Unity Catalog, which provides versioning, lineage tracking, sharing across workspaces, and governance controls.
* [Evaluation Dataset Structure for GenAI](/concepts/evaluation-dataset-structure-for-genai.md) - A structured dataset format with inputs, outputs, and expectations (context) fields used for MLflow GenAI evaluation
* [Evaluation Dataset Sync](/concepts/evaluation-dataset-sync.md) - A process that synchronizes Expectations collected in a labeling session to an Evaluation Dataset using an intelligent upsert operation keyed on trace inputs.
* [Evaluation Dataset UI](/concepts/evaluation-dataset-ui.md) - A visual interface in MLflow experiments for managing evaluation datasets and records without writing code, including create, edit, delete, tagging, and trace viewing.
* [Evaluation Dataset with Expected Facts](/concepts/evaluation-dataset-with-expected-facts.md) - A structured dataset containing input-output pairs with expected facts that allow measuring correctness of LLM responses by checking fact coverage.
* [Evaluation Datasets](/concepts/evaluation-datasets.md) - Structured test data used as input to `mlflow.genai.evaluate()`, supporting EvaluationDataset objects, lists of dicts, Pandas DataFrames, and Spark DataFrames.
* [Evaluation Datasets and Runs](/concepts/evaluation-datasets-and-runs.md) - Evaluation datasets are curated test cases (with optional ground-truth expectations) and evaluation runs are the results of testing an app version against them using scorers, producing traces and aggregated metrics.
* [Evaluation Datasets for GenAI](/concepts/evaluation-datasets-for-genai.md) - Structured datasets used to evaluate GenAI applications, consisting of input templates or prompts organized as records with consistent schema for batch evaluation.
* [Evaluation Datasets for GenAI Agents](/concepts/evaluation-datasets-for-genai-agents.md) - Structured test datasets containing inputs (conversation messages) and optional expectations fields used to evaluate GenAI agents with custom judges.
* [Evaluation Datasets with Expected Facts](/concepts/evaluation-datasets-with-expected-facts.md) - Building structured evaluation datasets that pair input content with expected facts to serve as ground truth for prompt performance measurement.
* [Evaluation Feedback](/concepts/evaluation-feedback.md) - Quality assessments produced by scorers and attached to individual traces within an evaluation run.
* [Evaluation Harness](/concepts/evaluation-harness.md) - Testing framework for evaluating traces during the development phase of GenAI applications.
* [Evaluation Harness (MLflow GenAI)](/concepts/evaluation-harness-mlflow-genai.md) - SDK (mlflow.genai.evaluate()) for systematically testing an app version against an evaluation dataset using scorers, producing traces with feedback assessments.
* [Evaluation Harness and Production Monitoring](/concepts/evaluation-harness-and-production-monitoring.md) - The evaluation harness (mlflow.genai.evaluate()) runs applications against evaluation datasets with scorers for iterative quality improvement; production monitoring schedules scorers on live traces for ongoing quality oversight.
* [Evaluation Run](/concepts/evaluation-run.md) - The results of testing an application version against an evaluation dataset using scorers, containing the generated traces, their assessments, and aggregated quality metrics.
* [Evaluation Run (MLflow)](/concepts/evaluation-run-mlflow.md) - An MLflow run that organizes and stores the results of evaluating a GenAI application, including traces, feedback, metrics, and metadata.
* [Evaluation Run Structure](/concepts/evaluation-run-structure.md) - The hierarchical organization of an evaluation run comprising run info, traces with feedback, aggregate metrics, and configuration parameters.
* [Evaluation Runs](/concepts/evaluation-runs.md) - Test reports that capture aggregate metrics and all per-row traces annotated with scorer feedback, viewable in the MLflow UI for investigation.
* [Evaluation Runs and Results UI](/concepts/evaluation-runs-and-results-ui.md) - The structured test report that captures all evaluation results, including per-row traces annotated with feedback from scorers, viewable in the Databricks UI with pass/fail assessments.
* [Evaluation Traces](/concepts/evaluation-traces.md) - Per-input records within an evaluation run, each containing inputs, outputs, and associated feedback for one example in the evaluation dataset.
* [Evaluation-Production Consistency](/concepts/evaluation-production-consistency.md) - The principle that same evaluation logic and scorers used in offline development can be reused in production monitoring, providing consistent quality views across the AI lifecycle.
* [EVENT_TIME_PRESENT](/concepts/event_time_present.md) - A sub-condition of the DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE error stating that skip offsets must not contain event timestamps
* [EVENT_TIME_PRESENT sub-condition](/concepts/event_time_present.md) - A specific reason for the DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE error, indicating that offsets in the skip range must not have event time metadata (eventTimeMillis must be empty).
* [EVENT_TIME_PRESENT Subcondition](/concepts/event_time_present.md) - A sub-error of DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE indicating skip offset range is invalid because the offset contains an event time (eventTimeMillis must be empty).
* [Event-Time Ordered Initial Snapshot Processing](/concepts/event-time-ordered-initial-snapshot-processing.md) - The withEventTimeOrder option processes initial Delta snapshot data in event-time order buckets to prevent watermark-based data drops in stateful streaming queries.
* [Eventual Consistency Impact on Delta Lake on S3](/concepts/s3-eventual-consistency-impact-on-delta-lake.md) - Amazon S3's eventually consistent model can cause data corruption and data loss when multiple systems or clusters modify the same Delta Lake table simultaneously, especially across workspaces.
* [Eventually consistent storage impact on Delta Lake](/concepts/eventually-consistent-storage-impact-on-delta-lake.md) - Amazon S3's eventually consistent model introduces risks for Delta Lake operations like concurrent writes and table modifications, requiring workarounds and single-workspace isolation.
* [Example applications for Databricks Connect](/concepts/example-applications-for-databricks-connect.md) - Reference code examples, including a simple ETL application and charts visualizations with JFreeChart, demonstrating Databricks Connect usage.
* [Example Column (Custom Classifier)](/concepts/example-column-custom-classifier.md) - A column selected from a Unity Catalog table that provides representative values for training a custom classifier's detection rules in Databricks Data Classification.
* [Example column selection for custom classifiers](/concepts/example-column-selection-for-custom-classifiers.md) - Process of choosing 1-10 representative table columns per classifier to train detection rules, where broader and more varied examples improve accuracy.
* [Example Columns Pattern](/concepts/example-columns-pattern.md) - The technique of providing 1–10 representative example columns from existing tables to teach a custom classifier what values to detect; broader and more varied examples produce more accurate detection rules.
* [EXCEPT clause in Unity Catalog access policies](/concepts/except-clause-in-unity-catalog-access-policies.md) - Using EXCEPT in policy definitions to exempt specific groups from masking or filtering rules while applying the policy to all other principals, enabling owner-group exceptions.
* [Expectation vs Feedback Label Types](/concepts/expectation-vs-feedback-labels.md) - Two distinct label types in MLflow labeling: 'expectation' labels capture ground truth / correct answers for evaluation datasets, while 'feedback' labels capture subjective ratings, classifications, or assessments.
* [Expectation vs Feedback Labels](/concepts/expectation-vs-feedback-labels.md) - Two types of labeling schemas: expectation type captures ground truth or correct answers (usable in evaluation datasets), while feedback type captures subjective assessments, ratings, or classifications.
* [Expectations in MLflow evaluation](/concepts/expectations-in-mlflow-evaluation.md) - Ground-truth values or labels provided in evaluation datasets (via an 'expectations' column) that scorers can use for offline evaluation, with graceful degradation for production monitoring where expectations are absent.
* [Expectations Reserved Keys](/concepts/expectations-reserved-keys.md) - Built-in reserved keys in the expectations field (guidelines, expected_facts, expected_response) used by MLflow's LLM judges for structured ground-truth evaluation.
* [Expectations-Based Evaluation](/concepts/expectations-based-evaluation.md) - A pattern for offline evaluation where ground-truth 'expectations' are supplied alongside inputs and passed to custom scorers for comparison.
* [Expectations-based LLM Evaluation Dataset](/concepts/expectations-based-llm-evaluation-dataset.md) - An evaluation dataset structure for MLflow GenAI that includes 'inputs', 'outputs', and 'expectations' (with 'context') to provide ground-truth reference for scorers.
* [ExpectationsGuidelines() Judge (Per-Row Guidelines)](/concepts/expectationsguidelines-judge-per-row-guidelines.md) - A built-in MLflow scorer that evaluates against row-specific guidelines provided by domain experts in the evaluation dataset, enabling different criteria for each example.
* [Expected Behaviors Evaluation](/concepts/expected-behaviors-evaluation.md) - Evaluation technique where judges verify that agent responses demonstrate predefined expected behaviors by comparing outputs against expectations.
* [expected_facts vs expected_response](/concepts/expected_facts-vs-expected_response.md) - Two alternative ground-truth parameters for the Correctness judge: expected_facts allows flexible fact-checking without requiring word-for-word matching, while expected_response requires a closer match to a reference answer.
* [Experiment Artifact Storage Locations](/concepts/experiment-artifact-storage-locations.md) - MLflow experiment artifacts can be stored in MLflow-managed DBFS (default), Unity Catalog volumes (recommended), or S3 (not recommended). S3-stored artifacts don't appear in the MLflow UI and cannot be registered in Model Registry.
* [Experiment-Level Trace Archival Configuration](/concepts/experiment-level-trace-archival-configuration.md) - The pattern of enabling or disabling trace archival on a per-experiment basis using experiment_id parameters.
* [Expert Labeling Sessions for GenAI Review](/concepts/expert-labeling-sessions-for-genai-review.md) - Creating structured expert review workflows using create_labeling_session() and label schemas (categorical and text inputs) to collect authoritative ground-truth feedback on trace responses
* [Expert Trace Labeling Workflow](/concepts/expert-trace-labeling-workflow.md) - The end-to-end process of identifying traces for expert review, creating labeling sessions with schemas, adding traces, having domain experts label them via the Review App, and retrieving labels for evaluation datasets.
* [Explicit Scorer Selection](/concepts/explicit-scorer-selection.md) - MLflow 3 requires users to explicitly list all scorers for evaluation, unlike MLflow 2.x which automatically selected applicable judges based on data fields.
* [Exploratory Data Analysis (EDA) on Databricks](/concepts/exploratory-data-analysis-eda-on-databricks.md) - The process of summarizing and visualizing datasets to surface distributions, correlations, missing values, and outliers, supported by Databricks tools including notebooks, dashboards, Genie Chat, and Genie Code.
* [Express Deployment Requirements and Constraints](/concepts/express-deployment-requirements-and-constraints.md) - The set of conditions necessary for express deployments including UC model registry, CPU serving, custom models (not FMAPI), max 1GB environment size, mlflow>=3.1, and serverless notebooks.
* [Express Deployments (Databricks Model Serving)](/concepts/express-deployments-databricks.md) - A deployment method for model serving endpoints that dramatically reduces deployment times by packaging and staging model artifacts during model registration in serverless notebooks.
* [Express Deployments (Databricks)](/concepts/express-deployments-databricks.md) - A deployment approach for Databricks model serving that accelerates endpoint creation and ensures environment consistency between training and serving by packaging model artifacts during registration.
* [Express Deployments for Model Serving](/concepts/express-deployments-databricks.md) - A Databricks feature that packages and stages model artifacts during model registration to dramatically reduce endpoint deployment times and maintain environment consistency between training and serving.
* [Extending Autologged Runs with Additional Tracking](/concepts/extending-autologged-runs-with-additional-tracking.md) - Pattern for combining automatic autologging with manual MLflow tracking calls (log_param, log_metric) by setting exclusive=False and wrapping training code inside mlflow.start_run().
* [Extending lifecycle management transition rules](/concepts/extending-lifecycle-management-transition-rules.md) - A risk scenario: extending the cloud archival interval without caution may cause errors because cloud providers do not auto-restore already-archived files.
* [Extensible Optimization Framework](/concepts/extensible-optimization-framework.md) - MLflow's prompt optimization framework allows plugging in custom optimization algorithms by extending the base class, making it framework-agnostic and compatible with any agent system.
* [External and Foreign Object Deletion Behavior](/concepts/external-and-foreign-object-deletion-behavior.md) - Deleting external tables/volumes removes metadata but leaves data files intact in cloud storage; deleting foreign/federated catalogs removes connection metadata without affecting source data.
* [External API Bottlenecks in Model Inference](/concepts/external-api-bottlenecks-in-model-inference.md) - Performance bottlenecks caused by models calling external APIs during inference, manifested as high latency, throughput limits, and error rates. Mitigation includes caching frequently accessed data and monitoring external API performance.
* [External assets in Unity Catalog](/concepts/external-assets-in-unity-catalog.md) - Data assets (tables and volumes) where Unity Catalog controls governance only and the file storage lifecycle is controlled by the user
* [External JAR Upload for Databricks Connect](/concepts/external-jar-upload-for-databricks-connect.md) - Uploading private or third-party JARs from local directories (e.g., a lib/ folder) to Databricks clusters using the addCompiledArtifacts() API to resolve ClassNotFoundException during UDF execution.
* [External Lineage](/concepts/external-lineage.md) - Extension of Unity Catalog lineage beyond Databricks by registering external assets like Salesforce, MySQL, Tableau, or Power BI to appear alongside Unity Catalog tables in a single lineage graph.
* [External Lineage (Unity Catalog)](/concepts/external-lineage-unity-catalog.md) - The ability to manually or automatically add lineage metadata for data sources and consumers outside of Databricks, enabling end-to-end data lineage views in Unity Catalog.
* [External Lineage API](/concepts/external-lineage-api.md) - A REST API for programmatically creating lineage relationships between external metadata objects and Unity Catalog objects (tables, models, paths, or other external metadata), with optional column-level mappings and custom metadata.
* [External Lineage APIs and SDK](/concepts/external-lineage-apis-and-sdk.md) - Programmatic interfaces including the External Metadata API, External Lineage API, and Databricks SDK for Python for managing external lineage without the UI.
* [External Lineage APIs and SDKs](/concepts/external-lineage-apis-and-sdk.md) - Programmatic interfaces for adding external lineage, including the External Metadata API, External Lineage API, and Databricks SDK for Python, in addition to the manual Catalog Explorer UI.
* [External Lineage in Unity Catalog](/concepts/external-lineage-unity-catalog.md) - A feature in Databricks Unity Catalog that allows users to add lineage metadata for workloads running outside Databricks (e.g., ETL tools, BI tools) to augment automatically captured runtime lineage.
* [External Lineage Limitations](/concepts/external-lineage-limitations.md) - Constraints on external lineage including non-recording in system lineage tables, and resource limits of 10,000 external metadata objects and 100,000 relationships per metastore.
* [External Lineage Limitations and Quotas](/concepts/external-lineage-limitations.md) - Resource limits and constraints for external lineage: a maximum of 10,000 external metadata objects and 100,000 external lineage relationships per metastore, and external lineage is not recorded in lineage system tables.
* [External Lineage Privileges (Unity Catalog)](/concepts/external-lineage-privileges-unity-catalog.md) - The specific Unity Catalog privileges required to create external metadata objects and define lineage relationships, including CREATE EXTERNAL METADATA and MODIFY privileges.
* [External Lineage Relationship Patterns](/concepts/lineage-relationship-patterns.md) - Patterns for modeling complex external lineage: connecting two Unity Catalog tables via an intermediary external metadata object, adding multiple levels of lineage across systems, and specifying column-level lineage.
* [External location](/concepts/external-location.md) - A Unity Catalog object that defines a cloud storage path and its access credentials, which must not overlap with any workspace's default storage path in the same account.
* [External Location Creation via AWS Quickstart](/concepts/external-location-creation-via-aws-quickstart.md) - A streamlined method using AWS CloudFormation Quickstart templates to create external locations and storage credentials in Unity Catalog for S3 bucket access, involving token-based authentication between Databricks and AWS.
* [External locations and credentials in COPY INTO](/concepts/external-locations-and-credentials-in-copy-into.md) - Mechanisms for providing storage access to COPY INTO via Unity Catalog external locations, named credentials, or inline temporary credentials.
* [External Metadata API](/concepts/external-metadata-api.md) - A REST API for programmatically creating and managing external metadata objects in Unity Catalog, including specifying names, system types, entity types, URLs, column definitions, and custom properties.
* [External metadata in Databricks](/concepts/external-metadata-in-databricks.md) - A Databricks concept for managing table metadata externally, with limited support only for streaming tables and materialized views.
* [External Metadata Object](/concepts/external-metadata-object.md) - A securable object in Unity Catalog that represents an entity in an external system (e.g., a Tableau dashboard, MySQL table, or Power BI report), which can be connected to Unity Catalog objects via lineage relationships.
* [External Model Configuration](/concepts/external-model-configuration.md) - Configuration details required to set up an external model endpoint, including model provider selection, task type (chat/completion/embeddings), API keys via secrets, and model name selection.
* [External Model Credential Management via Environment Variables](/concepts/external-model-credential-management-via-environment-variables.md) - Pattern of storing API keys and tokens as Databricks secrets and exposing them as environment variables to serve OpenAI, LangChain, and other SaaS models.
* [External Model Endpoint Constraints](/concepts/external-model-endpoint-constraints.md) - Restrictions on external model endpoints: only one served_entity per endpoint when external_model is present, and external_model cannot be added or removed after creation.
* [External Model Multi-Serving](/concepts/external-model-multi-serving.md) - Serving multiple external (third-party provider) models such as OpenAI GPT-4 and Anthropic Claude on the same endpoint, provided they share the same task type.
* [External Model Routing via Unified AI Gateway](/concepts/external-model-routing-via-unified-ai-gateway.md) - The pattern of routing requests to external model providers (such as Google Gemini) through Databricks serving endpoints, enabling centralized auth, logging, and governance.
* [External Models](/concepts/external-models.md) - Foundation models hosted outside Databricks (e.g., GPT-4, Claude) served through Databricks endpoints with centralized governance, rate limits, and access control.
* [External Models (Databricks)](/concepts/external-models.md) - A Model Serving feature allowing centrally governed access to foundation models hosted outside Databricks, such as GPT-4 from OpenAI and Anthropic, streamlining multi-provider LLM management.
* [External Models in Databricks](/concepts/external-models.md) - Foundation models hosted outside Databricks (e.g., GPT-4, Claude) served through centrally governed endpoints with rate limits and access control.
* [External Models in Databricks Responses API](/concepts/external-models-in-databricks-responses-api.md) - Support for externally hosted models (OpenAI provider, Azure OpenAI provider) that can be queried via the Responses API with full parameter and tool support, unlike Databricks-hosted pay-per-token models.
* [External Models on Databricks](/concepts/external-models-on-databricks.md) - Foundation models hosted outside of Databricks but integrated through Databricks model serving endpoints.
* [External Models vs Foundation Model APIs on Databricks](/concepts/external-models-vs-foundation-model-apis-on-databricks.md) - The distinction between Databricks-hosted foundation models (with specific image handling and token billing) and externally hosted models (where provider documentation governs behavior).
* [External Ray Application Connectivity to Databricks](/concepts/external-ray-applications-connecting-to-databricks.md) - Connect Ray applications outside Databricks to Unity Catalog tables using ray.data.read_databricks_tables() for SQL warehouse queries (Ray 2.8.0+) and ray.data.read_delta_sharing_tables() for Delta Sharing tables (Ray 2.33+).
* [External Ray Applications Connecting to Databricks](/concepts/external-ray-applications-connecting-to-databricks.md) - Connecting Ray applications running outside of Databricks to Unity Catalog tables using ray.data.read_databricks_tables() or Delta Sharing with ray.data.read_delta_sharing_tables().
* [External Ray Applications to Databricks](/concepts/external-ray-applications-to-databricks.md) - Connecting Ray applications running outside Databricks to Databricks data sources using ray.data.read_databricks_tables (SQL warehouse) and ray.data.read_delta_sharing_tables (Delta Sharing) APIs.
* [External Ray Applications to Databricks Connectivity](/concepts/external-ray-applications-to-databricks.md) - Connecting Ray applications running outside Databricks to Unity Catalog tables via SQL warehouses (ray.data.read_databricks_tables) and Delta Sharing (ray.data.read_delta_sharing_tables)
* [External Ray Applications with Databricks](/concepts/external-ray-applications-to-databricks.md) - Connecting external Ray applications to Databricks using ray.data.read_databricks_tables() (via SQL warehouse) and ray.data.read_delta_sharing_tables() (via Delta Sharing/OpenSharing).
* [External Sharing Restriction for Mounted Genie Spaces](/concepts/external-sharing-restriction-for-mounted-genie-spaces.md) - Mounted Genie Spaces cannot be shared with users outside the recipient's organization using OpenSharing; OpenSharing recipients cannot re-share data assets received from a provider.
* [External Table Conversion Caveats](/concepts/external-table-conversion-caveats.md) - Risks and limitations when converting external tables, including shared directory visibility issues and data integrity concerns during the conversion process.
* [External Tables in Unity Catalog](/concepts/external-tables-in-unity-catalog.md) - Tables whose data lifecycle, file layout, and storage are not managed by Unity Catalog; support multiple formats and allow direct non-Databricks compute access to data.
* [External User Authentication for AI Systems](/concepts/external-user-authentication-for-ai-systems.md) - Requirements for authenticating or monitoring users outside an organization who interact with AI applications, including multi-factor authentication and activity logging.
* [Fail-Closed Access Control in Unity Catalog](/concepts/fail-closed-access-control-in-unity-catalog.md) - A security design principle where users not found in an access mapping table, or whose access has expired, see no rows by default rather than all rows.
* [Fail-Closed Design for ABAC Policies](/concepts/fail-closed-design-for-abac-policies.md) - Security model where Databricks denies access to ABAC-secured tables if it cannot safely verify and enforce all applicable policies, including on unsupported compute, specific operations, or when policy dependencies are removed.
* [Fairness and Bias Monitoring for Classification Models](/concepts/fairness-and-bias-monitoring-for-classification-models.md) - Using data profiling to detect whether a classification model performs differently across demographic or protected groups
* [Faithfulness Scorer](/concepts/faithfulness-scorer.md) - A RAGAS scorer that measures the factual consistency between the generated answer and the retrieved context, producing a score between 0.0 and 1.0 with an explanation.
* [Fallback Behavior After Hive Metastore Disablement](/concepts/fallback-behavior-after-hive-metastore-disablement.md) - When legacy Hive metastore access is disabled, fallback to hive_metastore catalog is disabled and jobs referencing legacy tables fail.
* [Fallback from DatabricksSession to SparkSession](/concepts/fallback-from-databrickssession-to-sparksession.md) - The pattern of using SparkSession.builder().getOrCreate() directly when the DatabricksSession class is unavailable, relying on the SPARK_REMOTE environment variable for authentication.
* [Fallback Rules for Unclassified Data](/concepts/fallback-rules-for-unclassified-data.md) - Applying default restrictive tags and policies to unclassified or unverified data objects as a safety net
* [Fallback Rules for Unclassified Data in ABAC](/concepts/fallback-rules-for-unclassified-data.md) - Implementing default restrictive tags and policies for unclassified or untagged data objects to prevent unprotected data exposure.
* [Fallback Rules for Untagged Data in ABAC](/concepts/fallback-rules-for-untagged-data-in-abac.md) - Applying default restrictive tags like 'classification:unverified' to new objects and creating policies that restrict access to untagged data until a data steward reviews the classification.
* [Feature Construction and Registration Workflows](/concepts/feature-construction-and-registration-workflows.md) - Two complementary approaches to creating features: two-step (local Feature construction + register_feature) for experimentation, and single-step (create_feature) for direct registration in Unity Catalog.
* [Feature Definition and Registration Lifecycle](/concepts/feature-definition-and-registration-workflow.md) - Workflow for defining Feature objects locally or via create_feature, then persisting them to Unity Catalog using register_feature for reuse in training, serving, and model governance.
* [Feature Definition and Registration Workflow](/concepts/feature-definition-and-registration-workflow.md) - Workflow for creating Feature objects locally or directly in Unity Catalog using create_feature and register_feature, which can then be used in training and serving.
* [Feature Discovery in Unity Catalog](/concepts/feature-discovery-in-unity-catalog.md) - The ability to browse and search for feature tables by name, feature, comment, or tag within Unity Catalog.
* [Feature Discovery with Genie Code](/concepts/feature-discovery-with-genie-code.md) - Using Databricks' Genie Code assistant to find features and feature tables via natural language /findTables queries
* [Feature Engineering Compatibility Matrix](/concepts/feature-engineering-compatibility-matrix.md) - A reference mapping Databricks Runtime ML versions to the corresponding pre-installed versions of databricks-feature-engineering and databricks-feature-store packages.
* [Feature Engineering for Time Series Forecasting](/concepts/feature-engineering-for-time-series-forecasting.md) - Creating derived features from date/time columns — such as is_weekend binary flags using dayofweek — to improve forecasting model accuracy.
* [Feature Engineering in Unity Catalog](/concepts/feature-engineering-in-unity-catalog.md) - Databricks' feature store solution that integrates feature tables with Unity Catalog for governance, lineage, and discovery
* [Feature Engineering on Databricks](/concepts/feature-engineering-on-databricks.md) - Databricks feature store service for creating, managing, and serving ML features with automated data pipelines and feature discovery.
* [Feature Engineering Python API (FeatureEngineeringClient)](/concepts/feature-engineering-python-api-featureengineeringclient.md) - The Python client library (databricks-feature-engineering) providing methods like write_table, read_table, get_table, create_table, drop_table, and tag management for interacting with feature tables in Unity Catalog.
* [Feature Engineering with Unity Catalog](/concepts/feature-engineering-with-unity-catalog.md) - Integration of Databricks Feature Store with Unity Catalog to provide built-in governance, lineage, point-in-time joins, and cross-workspace feature sharing.
* [Feature flag A/B performance analysis using traces](/concepts/feature-flag-ab-performance-analysis-using-traces.md) - Pattern for comparing latency distributions between control and treatment groups in feature flag experiments by filtering traces on metadata tags.
* [Feature Flag Performance Analysis with Traces](/concepts/feature-flag-performance-analysis-with-traces.md) - Pattern for comparing GenAI application performance across different feature flag states by filtering and aggregating trace latency data.
* [Feature Freshness Monitoring](/concepts/feature-freshness-monitoring.md) - Tracking the last write time and scheduled job status for feature tables to verify data recency.
* [Feature Function](/concepts/feature-function.md) - A user-defined scalar function registered in Unity Catalog that computes feature values on-demand at inference time using both stored and context-provided inputs.
* [Feature Function (UDF)](/concepts/feature-function.md) - A user-defined function in Unity Catalog that computes on-demand features at inference time using input bindings from retrieved and context features
* [Feature Function Integration](/concepts/feature-function-integration.md) - The capability to define and include Python UDFs within a FeatureSpec to compute derived features on the fly, using input bindings from other features or request parameters.
* [Feature Governance and Lineage](/concepts/feature-governance-and-lineage.md) - Built-in tracking of the origin, transformations, and usage of features across model training and inference workflows, enabled by Unity Catalog.
* [Feature Lifecycle Workflows](/concepts/feature-lifecycle-workflows.md) - The end-to-end pipeline covering feature development (create/register), model training (create_training_set), and materialization/serving in Databricks.
* [Feature Lineage and Online Workflows](/concepts/feature-lineage-and-online-workflows.md) - Models trained using Databricks features automatically track lineage to those features; when deployed as serving endpoints, they use Unity Catalog to resolve and look up the correct features in online stores for real-time inference.
* [Feature Lineage and Training/Serving Skew Elimination](/concepts/feature-lineage-and-trainingserving-skew-elimination.md) - Models automatically track lineage to training features and look up latest values at inference, ensuring identical feature computations between training and serving.
* [Feature Lineage in Databricks](/concepts/feature-lineage-in-databricks.md) - Lineage tracking for feature tables that records data sources used to create the table and links to models, notebooks, jobs, and endpoints that consume each feature.
* [Feature Lineage in Unity Catalog](/concepts/feature-lineage-in-unity-catalog.md) - Tracking of data sources used to create feature tables and tracking of downstream consumers including models, notebooks, jobs, and endpoints that use each feature.
* [Feature Lineage Tracking](/concepts/feature-lineage-tracking.md) - Automatic tracking of which features from the Feature Store were used to train a model, enabling governance and reproducibility.
* [Feature Lookup](/concepts/feature-lookup.md) - A mechanism to retrieve precomputed features from a feature table by primary key at serving time
* [Feature Lookup Keys](/concepts/feature-lookup-keys.md) - When joining a feature table to an AutoML training dataset, each feature table primary key must be mapped to a corresponding lookup key column in the training dataset.
* [Feature Lookup with Timestamp Keys](/concepts/feature-lookup-with-timestamp-keys.md) - Using FeatureLookup with a timestamp_lookup_key parameter to perform point-in-time lookups of feature values based on a time column in the label DataFrame.
* [Feature Materialization](/concepts/feature-materialization.md) - The process of persisting computed features to offline or online stores for efficient reuse in training and serving
* [Feature Materialization for Offline and Online Serving](/concepts/feature-materialization-for-offline-and-online-serving.md) - Process of materializing defined features to offline stores for efficient batch reuse or online stores for low-latency model serving using materialize_features.
* [Feature Materialization with Online and Offline Stores](/concepts/feature-materialization-with-online-and-offline-stores.md) - The process of persisting computed features to offline stores (for batch training) or online stores (for low-latency serving) using materialize_features with scheduling triggers like CronSchedule or TableTrigger.
* [Feature Naming and Auto-Generated Names](/concepts/feature-naming-and-auto-generated-names.md) - A naming convention system where features can be explicitly named or auto-named based on function, window, and column properties for traceability.
* [Feature Object and Registration Workflow](/concepts/feature-object-and-registration-workflow.md) - Two-step workflow for defining features locally with the Feature class and persisting them to Unity Catalog via register_feature() or create_feature(), supporting experimentation before registration.
* [Feature Publishing to Online Stores](/concepts/feature-publishing-to-online-stores.md) - The process of publishing Databricks feature tables to third-party online stores (DynamoDB, SQL stores) for low-latency, real-time serving of machine learning features.
* [Feature Registration and Lifecycle](/concepts/feature-registration-and-lifecycle.md) - Two workflows for persisting features to Unity Catalog: a two-step approach (construct Feature locally, then register_feature) and a single-step approach (create_feature that validates and registers immediately).
* [Feature Registration and Lifecycle in Unity Catalog](/concepts/feature-registration-and-lifecycle-in-unity-catalog.md) - Process of persisting locally-defined Feature objects into Unity Catalog using register_feature or create_feature, enabling governance, discoverability, and reuse across teams.
* [Feature Registration Workflows](/concepts/feature-registration-workflows.md) - Two approaches for registering features in Unity Catalog: a two-step workflow (construct Feature locally, then register_feature) for experimentation, or single-step create_feature for immediate registration
* [Feature Serving and Model Serving Integration](/concepts/feature-serving-and-model-serving-integration.md) - Online feature stores integrate with Feature Serving Endpoints and Model Serving Endpoints to automatically look up and serve features for real-time inference and applications.
* [Feature Serving Default Values](/concepts/feature-serving-default-values.md) - A mechanism to supply fallback values for feature lookups when keys are not found in the online store, requiring renamed feature names when using rename_outputs.
* [Feature Serving Endpoint](/concepts/feature-serving-endpoint.md) - A Databricks endpoint that serves precomputed and on-demand features for real-time inference using a FeatureSpec
* [Feature Serving Endpoint Lifecycle](/concepts/feature-serving-endpoint-lifecycle.md) - The create/read/update/delete lifecycle for Feature Serving endpoints, emphasizing update-over-recreate to avoid downtime, and supporting UI, REST API, Python API, and Databricks SDK.
* [Feature Serving endpoint lifecycle management](/concepts/feature-serving-endpoint-lifecycle-management.md) - APIs and UI workflows for creating, retrieving, updating (never delete-and-recreate), and deleting Feature Serving endpoints, with warnings against downtime caused by deletion.
* [Feature Serving Endpoints (Databricks)](/concepts/feature-serving-endpoints-databricks.md) - Feature and function serving endpoints on Databricks that support route optimization using FeatureSpecs.
* [Feature Serving Endpoints for Online Tables](/concepts/feature-serving-endpoints-for-online-tables.md) - A REST API endpoint that serves features from online tables at low latency for models and applications hosted outside Databricks, created via a Feature Spec linked to a Delta table.
* [Feature Serving vs Model Serving](/concepts/feature-serving-vs-model-serving.md) - The distinction between serving raw features directly via Feature Serving versus serving ML models that autonomously look up features via Model Serving, as depicted in the document's decision diagram.
* [Feature Serving with Online Tables](/concepts/feature-serving-with-online-tables.md) - Architecture for serving online table data via REST API endpoints using a feature serving endpoint, enabling low-latency feature lookups for models and applications hosted outside Databricks.
* [Feature Store](/concepts/feature-store.md) - A governed system for defining, managing, and reusing ML features as assets, with the same feature definitions used for both training and serving, supporting batch and real-time ingestion.
* [Feature Store (Machine Learning)](/concepts/feature-store-machine-learning.md) - A centralized repository that enables data scientists to find, share, and reuse features, ensuring the same feature computations are used during model training and inference.
* [Feature Store Access Control (Legacy)](/concepts/feature-store-access-control-legacy.md) - A permission system for controlling access to feature table metadata in Databricks workspaces not enabled for Unity Catalog
* [Feature Store Access Control Permission Levels](/concepts/feature-store-access-control-permission-levels.md) - Three permission levels for feature table metadata: CAN VIEW METADATA, CAN EDIT METADATA, and CAN MANAGE.
* [Feature Store Access Control Permissions (Legacy)](/concepts/feature-store-access-control-permissions-legacy.md) - Three permission levels (CAN VIEW METADATA, CAN EDIT METADATA, CAN MANAGE) that control granular access to feature table metadata in Databricks Feature Store workspaces not using Unity Catalog.
* [Feature Store Access Control vs Unity Catalog](/concepts/legacy-access-control-vs-unity-catalog.md) - Legacy access control for feature tables in workspaces not enabled for Unity Catalog; workspaces with Unity Catalog should use UC privileges instead.
* [Feature Store and Feature Engineering](/concepts/feature-store-and-feature-engineering.md) - Managed Feature Store in Unity Catalog providing a single governed source of truth for features, supporting both batch and real-time serving with Genie Code-assisted discovery and transformation.
* [Feature Store Client API for Remote Operations](/concepts/feature-store-client-api-for-remote-operations.md) - The set of FeatureStoreClient methods (create_table, read_table, write_table, publish_table, create_training_set, get_table) that operate on remote feature stores when configured with a feature_store_uri.
* [Feature Store Client URI (databricks://)](/concepts/feature-store-client-uri-databricks.md) - A URI scheme of the form 'databricks://<scope>:<prefix>' used to specify a remote feature store or model registry when instantiating a FeatureStoreClient in Databricks.
* [Feature Store Lookup Keys](/concepts/feature-lookup-keys.md) - The mechanism for joining feature tables to training datasets by mapping feature table primary keys to columns in the training dataset
* [Feature Store Lookups in AutoML](/concepts/feature-store-lookups-in-automl.md) - The mechanism for passing Databricks Feature Store tables as covariates to AutoML forecasting experiments using the feature_store_lookups parameter.
* [Feature Store Online Store Authentication](/concepts/feature-store-online-store-authentication.md) - Overview of authentication methods (instance profiles and Databricks secrets) for publishing to and reading from third-party online stores in Databricks Feature Store.
* [Feature Store Permission Levels (Legacy)](/concepts/feature-table-permission-levels.md) - Three permission levels for feature table metadata in Databricks Feature Store: CAN VIEW METADATA, CAN EDIT METADATA, and CAN MANAGE.
* [Feature Store Resource Cleanup](/concepts/feature-store-resource-cleanup.md) - Background asynchronous cleanup process that removes tables, pipelines, and jobs when the last materialized feature referencing those resources is deleted, using a Databricks-managed system service principal.
* [Feature Store-Wide Permission Configuration](/concepts/feature-store-wide-permission-configuration.md) - Workspace administrators and users with CAN MANAGE on the Feature Store can set default permission levels on all current and future feature tables from the Feature Store page.
* [Feature Store-Wide Permission Inheritance](/concepts/feature-store-wide-permission-inheritance.md) - Permissions set at the Feature Store level apply to all feature tables (current and future) and can be overridden per-table to add but not restrict permissions
* [Feature Store-Wide Permissions](/concepts/feature-store-wide-permissions.md) - Workspace administrators can set permissions for all current and future feature tables from the Feature Store UI.
* [Feature Table](/concepts/feature-table.md) - An organized collection of features backed by a Delta table with a primary key and additional metadata tracking data sources, notebooks, and jobs.
* [Feature Table Access Control](/concepts/feature-table-access-control.md) - A mechanism within the Databricks Workspace Feature Store for restricting which users or groups can view, modify, or use feature tables.
* [Feature Table Access Control (Legacy)](/concepts/feature-table-access-control-legacy.md) - Mechanisms to control user access to feature tables within the Databricks Workspace Feature Store, documented in a separate dedicated page.
* [Feature Table Default Permissions](/concepts/feature-table-default-permissions.md) - By default, the creator and workspace admins get CAN MANAGE, while other users have NO PERMISSIONS on a new feature table.
* [Feature Table Governance](/concepts/feature-table-governance.md) - Unity Catalog governs feature tables, functions, and models; models inherit permissions from the data they were trained on.
* [Feature Table Governance and Tagging](/concepts/feature-table-governance-and-tagging.md) - Feature tables in Unity Catalog can be browsed, searched, and managed via the Features UI and Catalog Explorer, with support for tags (key-value pairs) for categorization.
* [Feature Table Lookup Keys](/concepts/feature-lookup-keys.md) - Mechanism for joining feature tables to AutoML training datasets by mapping primary keys from feature tables to corresponding lookup key columns in the training data.
* [Feature table management and update strategies](/concepts/feature-table-management-and-update-strategies.md) - Methods for updating feature tables including merge mode for upserts, streaming pipelines for real-time updates, scheduled jobs for periodic refreshes, and composite primary keys (date + ID) for storing historical feature values.
* [Feature Table Permission Levels](/concepts/feature-table-permission-levels.md) - Databricks Feature Store defines three fine-grained permission levels for feature table metadata: CAN VIEW METADATA, CAN EDIT METADATA, and CAN MANAGE.
* [Feature Table Producers](/concepts/feature-table-producers.md) - Notebooks and jobs that write to a feature table, visible in the Producers table on the feature table detail page.
* [Feature Table Producers Table](/concepts/feature-table-producers-table.md) - A UI component in the feature table page that lists all notebooks and jobs that write to a feature table, enabling freshness and job status verification.
* [Feature Table Publishing](/concepts/feature-table-publishing.md) - The process of synchronizing offline feature tables to an online store using the publish_table API, with support for CONTINUOUS, TRIGGERED, and SNAPSHOT publish modes.
* [Feature Table Search](/concepts/feature-table-search.md) - A case-insensitive search capability in the Databricks Feature Store UI allowing users to find feature tables by name, feature name, data source, or tag key-value pairs.
* [Feature Table Search and Discovery](/concepts/feature-table-search.md) - Capability to search feature tables by name, feature name, data source, or tag with case-insensitive matching.
* [Feature Table Tagging](/concepts/feature-table-tagging.md) - Adding key-value tag pairs to feature tables for organization and searchability, with edit and delete capabilities.
* [Feature table tagging and governance](/concepts/feature-table-tags-and-governance.md) - Using key-value tags to categorize, search, and manage feature tables and individual features in Unity Catalog, supported via Catalog Explorer, SQL, and the Feature Engineering Python API.
* [Feature Table Tags and Governance](/concepts/feature-table-tags-and-governance.md) - Tags (key-value pairs) can be applied to feature tables and individual features via Catalog Explorer, SQL, or Python API for categorization and governance.
* [Feature Table Update Strategies](/concepts/feature-table-update-strategies.md) - Methods for updating feature tables including adding new features, merging specific rows, scheduling periodic jobs, and streaming feature computation pipelines.
* [Feature table upgrade to Unity Catalog](/concepts/feature-table-upgrade-to-unity-catalog.md) - Two-step process of first upgrading the underlying Delta table to Unity Catalog, then migrating feature table metadata using upgrade_workspace_table.
* [Feature Tables](/concepts/feature-table.md) - Organized collections of features backed by Delta tables with primary keys, tracked with metadata about data sources, notebooks, and jobs that created them.
* [Feature Tables in Unity Catalog](/concepts/feature-tables-in-unity-catalog.md) - Any Delta table with a primary key constraint in Unity Catalog can serve as a feature table for machine learning, accessible via a three-level namespace.
* [Feature Tracing for Inference Tables](/concepts/feature-tracing-for-inference-tables.md) - Logging automatic feature lookup DataFrames to inference tables by setting the ENABLE_FEATURE_TRACING environment variable on a model serving endpoint
* [Feature Tracing with ENABLE_FEATURE_TRACING](/concepts/feature-tracing-with-enable_feature_tracing.md) - Logging automatic feature lookup DataFrames to inference tables by setting the ENABLE_FEATURE_TRACING environment variable
* [Feature Tracing with Inference Tables](/concepts/feature-tracing-for-inference-tables.md) - Logging automatic feature lookup DataFrames to inference tables by setting ENABLE_FEATURE_TRACING environment variable to true, requiring MLflow 2.14.0 or above.
* [FEATURE_SOURCE Environment Variable](/concepts/feature_source-environment-variable.md) - An environment variable set on serving endpoints to switch their feature data source from legacy online tables to the Databricks Online Feature Store.
* [FeatureEngineeringClient (Python client)](/concepts/featureengineeringclient-python-api.md) - A Python client (databricks-feature-engineering package) for creating, reading, updating, and deleting feature tables in Unity Catalog programmatically.
* [FeatureEngineeringClient API](/concepts/featureengineeringclient-api.md) - The Python client class for creating, writing, and managing Feature Store tables in Databricks
* [FeatureEngineeringClient for Covariate Tables](/concepts/featureengineeringclient-for-covariate-tables.md) - Databricks API client used to create and register feature tables with a primary key (e.g., date) for use as covariates in AutoML
* [FeatureEngineeringClient Python API](/concepts/featureengineeringclient-python-api.md) - A Python client API (`FeatureEngineeringClient`) for creating, reading, updating, and managing feature tables in Unity Catalog, pre-installed in Databricks Runtime ML editions.
* [FeatureEngineeringClient vs FeatureStoreClient](/concepts/featureengineeringclient-vs-featurestoreclient.md) - Two distinct clients: FeatureEngineeringClient for Unity Catalog feature tables and FeatureStoreClient for the legacy Workspace Feature Store.
* [FeatureEngineeringClient.log_model](/concepts/featureengineeringclientlog_model.md) - API method that automatically captures lineage of features, feature tables, and Python UDFs when logging an ML model
* [FeatureEngineeringClient.log_model / FeatureStoreClient.log_model](/concepts/featureengineeringclientlog_model-featurestoreclientlog_model.md) - Models must be logged using FeatureEngineeringClient.log_model (Unity Catalog) or FeatureStoreClient.log_model (legacy) at v0.3.5+ to enable automatic feature lookup at serving time.
* [FeatureFunction](/concepts/featurefunction.md) - A mechanism for on-demand feature computation that combines real-time inference inputs with stored feature values at scoring time.
* [FeatureFunction (On-Demand Feature Computation)](/concepts/featurefunction-on-demand-feature-computation.md) - A Unity Catalog function used in FeatureSpecs to compute feature values at inference time using runtime-provided context data
* [FeatureFunction API](/concepts/featurefunction-api.md) - API construct in Databricks Feature Engineering that binds Python UDF inputs to columns/feature lookups for on-demand feature computation during training and inference.
* [FeatureFunction for On-Demand Features](/concepts/featurefunction-for-on-demand-features.md) - A construct used with FeatureEngineeringClient to register Python UDFs that compute features on-demand at inference time, with lineage tracked like feature tables.
* [FeatureLookup](/concepts/featurelookup.md) - A construct that specifies which features to select from a feature table and defines the join keys used to link feature tables with label data during training set creation.
* [FeatureLookup and FeatureFunction API](/concepts/featurelookup-and-featurefunction-api.md) - Features for a training set can be defined via FeatureLookup (table-based) or FeatureFunction (UDF-based) to compute on-demand features
* [FeatureLookup default values](/concepts/featurelookup-default-values.md) - Mechanism to specify default values for feature lookups when lookup keys are missing, with special handling when using rename_outputs.
* [Features UI and Catalog Explorer](/concepts/features-ui-and-catalog-explorer.md) - User interfaces for exploring and managing feature tables, including metadata like ownership, online stores, last write time, tags, and comments.
* [Features UI in Databricks](/concepts/features-ui-in-databricks.md) - A dedicated user interface for browsing, searching, and managing feature tables within Unity Catalog
* [FeatureSpec](/concepts/featurespec.md) - A Unity Catalog entity that bundles FeatureLookups and FeatureFunctions into a reusable logical unit for model training and Feature Serving endpoints.
* [FeatureSpec Default Values](/concepts/featurespec-default-values.md) - The ability to specify fallback default values for feature lookups within a FeatureSpec, which are used when a lookup key is not found in the feature table.
* [Featurization for Transfer Learning](/concepts/featurization-for-transfer-learning.md) - Using pre-trained deep learning models to compute feature vectors from data, which can then be used as input for downstream models in a different problem domain.
* [Feedback (MLflow Evaluation)](/concepts/feedback-mlflow-evaluation.md) - Quality assessments from scorers attached to individual traces during an evaluation run, representing per-example scores.
* [Feedback Analysis and Monitoring with MLflow SDK](/concepts/feedback-analysis-and-monitoring-with-mlflow-sdk.md) - Using the MLflow SDK to query traces, extract feedback assessments, calculate metrics like feedback rate and positive rate, and analyze multi-dimensional ratings over time windows.
* [Feedback Analysis via MLflow SDK](/concepts/feedback-analysis-via-mlflow-sdk.md) - Programmatic analysis of collected user feedback by querying traces, extracting assessments, and computing quality metrics
* [Feedback Collection Approaches: Trace IDs vs Client Request IDs](/concepts/feedback-collection-approaches-trace-ids-vs-client-request-ids.md) - Two methods for linking user feedback to specific traces: using MLflow's auto-generated trace IDs or generating unique client request IDs.
* [Feedback Collection Patterns in GenAI Applications](/concepts/feedback-collection-patterns-in-genai-applications.md) - Two approaches for linking user feedback to MLflow traces: using MLflow-generated trace IDs or custom client request IDs
* [Feedback Collection with Streaming Responses](/concepts/feedback-collection-with-streaming-responses.md) - Techniques for collecting user feedback when GenAI responses are streamed via SSE or WebSockets, where trace IDs are only available after stream completion.
* [Feedback in MLflow Evaluation](/concepts/feedback-mlflow-evaluation.md) - Quality assessments from scorers attached to each trace within an evaluation run, providing per-example quality scores.
* [Feedback Object](/concepts/feedback-object.md) - The return type of the Correctness judge containing a binary 'value' ('yes'/'no') indicating correctness and a 'rationale' string explaining which facts are supported or missing.
* [Feedback object for rich assessments](/concepts/feedback-object-for-rich-assessments.md) - Scorers can return Feedback objects with value, rationale, source, and metadata for detailed multi-metric evaluations.
* [Feedback Object in MLflow Evaluation](/concepts/feedback-object-in-mlflow-evaluation.md) - A structured result type carrying a value, rationale, optional name, error info, and assessment source metadata for scorer outputs.
* [Feedback Object Return Format](/concepts/feedback-object-return-format.md) - Guidelines judges return an `mlflow.entities.Feedback` object containing a pass/fail value ('yes'/'no'), detailed rationale, assessment name, and optional error details.
* [Feedback Object Schema](/concepts/feedback-object-schema.md) - The structured output format returned by MLflow LLM judges, containing a 'value' field (yes/no) and a 'rationale' field explaining the judge's decision.
* [Feedback Object Structure](/concepts/feedback-object-structure.md) - The output format of MLflow relevance judges, consisting of a 'value' field (yes/no) indicating relevance and a 'rationale' field explaining the judge's reasoning.
* [Feedback objects](/concepts/feedback-objects.md) - Rich return type for scorers enabling detailed assessments with scores, rationales, metadata, and error information, supporting multiple feedback per scorer.
* [Feedback Objects in MLflow](/concepts/feedback-objects-in-mlflow.md) - Return type of make_judge() judges, represented as mlflow.entities.Feedback objects containing evaluation results with typed ratings
* [Feedback Value Types](/concepts/feedback-value-types.md) - A typing mechanism using Python's Literal and bool to define the output format and possible values returned by custom judges, such as categorical labels or boolean assessments.
* [Feedback Value Types for Judges](/concepts/feedback-value-types-for-judges.md) - Custom judges can return various feedback value types including boolean (`bool`), string literals (e.g., `fully_resolved`, `partially_resolved`, `needs_follow_up`), and other types defined via Python's `Literal` type annotation.
* [Feedback Value Types in Judges](/concepts/feedback-value-types-in-judges.md) - The return types for custom judges, supporting Literal string types (e.g., 'fully_resolved') and bool types for different grading scales.
* [Feedback vs Expectation Assessment Types](/concepts/feedback-vs-expectation-assessment-types.md) - Two distinct assessment types in MLflow: Feedback for subjective quality ratings/comments, and Expectation for recording what output should have been produced (ground truth).
* [Feedback vs Expectation in MLflow](/concepts/feedback-vs-expectation-in-mlflow.md) - Two distinct assessment types: Feedback evaluates the app's actual outputs (ratings, comments on what was generated), while Expectation defines the desired or correct outcome (ground truth) that the app should have produced, useful for creating evaluation datasets.
* [Feedback vs Expectation Labels](/concepts/feedback-vs-expectation-labels.md) - The two types of human-generated assessments captured in labeling sessions — Feedback (subjective ratings) and Expectation (ground-truth annotations) — used to improve GenAI apps through systematic evaluation.
* [Feedback vs Expectation schema types](/concepts/feedback-vs-expectation-schema-types.md) - Two distinct schema types: feedback for subjective assessments (ratings, preferences) and expectation for objective ground truth (correct answers, required facts).
* [feedback_value_type](/concepts/feedback_value_type.md) - Parameter in make_judge() that defines the return type of a custom judge, supporting Literal string unions or bool
* [Feedback-Driven Evaluation with MLflow Correctness Scorer](/concepts/feedback-driven-evaluation-with-mlflow-correctness-scorer.md) - Using expert-provided expected_response labels to quantitatively evaluate GenAI app outputs against ground truth using MLflow's Correctness scorer within mlflow.genai.evaluate()
* [Feedback-to-score mapping for prompt optimization](/concepts/feedback-to-score-mapping-for-prompt-optimization.md) - A technique where categorical or qualitative feedback from LLM judges is mapped to numerical scores (e.g., high→1.0, medium→0.5, low→0.0) for use by prompt optimizers.
* [FILE_VALIDATION_FAILED](/concepts/file_validation_failed.md) - A sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED indicating that one or more files could not be migrated during the external-to-managed table transition.
* [FILE_VALIDATION_FAILED error](/concepts/file_validation_failed.md) - A sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED raised when one or more files cannot be migrated from external to managed storage during table conversion.
* [FILE_VALIDATION_FAILED error (Delta managed migration)](/concepts/file_validation_failed.md) - Sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED indicating one or more files could not be migrated during the external-to-managed transition
* [FILE_VALIDATION_FAILED Sub-Error](/concepts/file_validation_failed.md) - Sub-error raised when one or more files cannot be migrated during ALTER TABLE SET MANAGED
* [Filter Conditions and SQL Transformations](/concepts/filter-conditions-and-sql-transformations.md) - Mechanisms to filter source rows before aggregation (filter_condition as SQL WHERE clause) and transform columns (transformation_sql for renaming, casting, arithmetic) within DeltaTableSource, with schema_json for type safety.
* [Filter Conditions and SQL Transformations on Source Data](/concepts/filter-conditions-and-sql-transformations.md) - Mechanisms to filter rows (filter_condition) and transform columns (transformation_sql) at the DeltaTableSource level before aggregation, supporting row-wise operations like casts and arithmetic.
* [Filter Predicates in Conflict Detection](/concepts/filter-predicates-in-conflict-detection.md) - The requirement that filter predicates used by a transaction must be applicable for row-level conflict detection, otherwise the operation may fail with PREDICATES_NEED_REWRITE
* [Filter-Based Conflict Resolution Strategy](/concepts/filter-based-conflict-resolution-strategy.md) - The recommended approach to resolve certain Delta Lake concurrent delete conflicts by adding filter predicates to narrow the data scope of operations, reducing the chance of whole-table conflicts and enabling row-level detection.
* [Filtering and Sorting MLflow Runs](/concepts/filtering-and-sorting-mlflow-runs.md) - Ability to filter runs by parameter, metric, or tag values, and sort runs by selected parameters to control which runs appear on charts.
* [Filtering MLflow experiment data](/concepts/filtering-mlflow-experiment-data.md) - Using Spark DataFrame filter expressions on metrics and params columns (e.g., 'metrics.loss < 0.01') after loading experiment data via the mlflow-experiment data source.
* [Fine-tune Llama 2 with DeepSpeedTorchDistributor](/concepts/fine-tune-llama-2-with-deepspeedtorchdistributor.md) - An example notebook demonstrating fine-tuning of the Llama 2 7B Chat model using the DeepSpeedTorchDistributor on Databricks.
* [Fine-tuned and custom model support via Provisioned Throughput](/concepts/fine-tuned-and-custom-model-support-via-provisioned-throughput.md) - Provisioned throughput endpoints support base models of all sizes, fine-tuned variants, fully custom weights and tokenizers, and architectural variations like CodeLlama.
* [Fine-tuning large language models with AI Runtime](/concepts/fine-tuning-large-language-models-with-ai-runtime.md) - Notebook examples for fine-tuning LLMs on Databricks AI Runtime, including parameter-efficient methods.
* [Fine-tuning Llama 2 7B with DeepSpeed on Databricks](/concepts/fine-tuning-llama-2-7b-with-deepspeed-on-databricks.md) - An example notebook demonstrating how to fine-tune the Llama 2 7B Chat model using the DeepSpeedTorchDistributor on Databricks.
* [FineTome-100k Dataset](/concepts/finetome-100k-dataset.md) - A large-scale instruction-following dataset (100k examples) created by mlabonne, used for supervised fine-tuning of language models, formatted in the ShareGPT conversation format.
* [Fixed-Size vs Auto-Scaling Ray Clusters](/concepts/fixed-size-vs-auto-scaling-ray-clusters.md) - Ray cluster sizing strategies on Databricks, including fixed-size clusters with explicit worker counts and auto-scaling clusters that adjust capacity dynamically.
* [Flash Attention](/concepts/flash-attention.md) - An optimized attention mechanism that accelerates transformer training and inference by reducing memory reads/writes, distributed as a prebuilt wheel to avoid slow compilation from source.
* [Flash-Attention Model Logging for Serving](/concepts/flash-attention-model-logging-for-serving.md) - Best practices for logging MLflow models that require flash-attn, including custom wheel versions and compatible CUDA/pytorch dependency specification.
* [flash-attn Dependency Management in Model Serving](/concepts/flash-attn-dependency-management-in-model-serving.md) - Best practices for logging models requiring flash-attn, including using custom wheel versions, specifying compatible PyTorch/CUDA versions, and listing all pip requirements explicitly in mlflow.transformers.log_model.
* [foreachBatch for Idempotent and Upsert Writes](/concepts/foreachbatch-for-idempotent-and-upsert-writes.md) - Using foreachBatch with txnAppId/txnVersion for idempotent writes to multiple Delta tables, and employing MERGE within foreachBatch for continuous upserts from streaming queries.
* [Forecast](/concepts/forecast.md) - The databricks.automl.forecast method configures an AutoML run for training a forecasting model with support for Auto-ARIMA.
* [Forecast Frequency and Horizon](/concepts/forecast-frequency-and-horizon.md) - Two key parameters that define the granularity (frequency) and duration (horizon) of a time series forecast in Databricks AutoML.
* [Forecast Frequency and Horizon Configuration](/concepts/forecast-frequency-and-horizon-configuration.md) - Key parameters for defining a forecasting problem: forecast frequency (granularity like hours/days/months) and forecast horizon (number of future time units to predict).
* [Forecast Horizon and Frequency Configuration](/concepts/forecast-horizon-and-frequency-configuration.md) - Configuration parameters in AutoML forecasting that specify the number of future time periods to predict and the time units (e.g., days, weeks, months).
* [Foreign Asset Sharing via Lakehouse Federation](/concepts/foreign-asset-sharing-via-lakehouse-federation.md) - The ability to share foreign schemas and tables (data managed by external systems) through OpenSharing without copying data or transferring credentials. Data is queried and temporarily materialized on the provider's side. Requires Lakehouse Federation Sharing preview, serverless compute, and default storage. Recommended for ad-hoc exploratory queries with result sizes under 10 GB.
* [Foreign Iceberg Table Sharing](/concepts/foreign-iceberg-table-sharing.md) - Sharing Iceberg tables federated from foreign Iceberg catalogs via Lakehouse Federation, supporting both Databricks-to-Databricks and open Iceberg client recipients.
* [Foreign Schema and Table Sharing (Lakehouse Federation)](/concepts/foreign-table-sharing-via-lakehouse-federation.md) - Sharing data from external systems (foreign tables/schemas) via Lakehouse Federation without copying data into Databricks, using on-demand materialization on the provider side.
* [Foreign Table and Schema Sharing](/concepts/foreign-table-and-schema-sharing.md) - Sharing data from external systems via Lakehouse Federation without copying data into Databricks, using serverless compute for on-demand materialization on the provider side.
* [Foreign Table Sharing via Lakehouse Federation](/concepts/foreign-table-sharing-via-lakehouse-federation.md) - Securely sharing data from external database systems through Lakehouse Federation without copying data, using on-demand query materialization on the provider side.
* [Foundation Model Access Governance Alternatives](/concepts/foundation-model-access-governance-alternatives.md) - Alternative governance mechanisms for Databricks foundation models including system.billing for cost tracking, AI Gateway for rate limits, Private Link for secure connectivity, and egress controls — to be used for day-to-day governance rather than model-specific restriction.
* [Foundation Model Access Governance Use Cases](/concepts/foundation-model-access-governance-use-cases.md) - The distinction between when to use foundation model Unity Catalog permissions (export controls, vendor restrictions, corporate policies) versus other governance tools like billing, AI Gateway, and network controls.
* [Foundation Model API Multimodal Support](/concepts/foundation-model-api-multimodal-support.md) - The capability of Databricks-hosted foundation models to accept diverse input types including text, image, video, and audio across multiple model providers.
* [Foundation Model API Offerings](/concepts/foundation-model-api-offerings.md) - The three Databricks Foundation Model API service tiers — pay-per-token, provisioned throughput, and fine-tuning — each with distinct model retirement timelines and policies.
* [Foundation Model API Security Best Practices](/concepts/foundation-model-api-security-best-practices.md) - Security measures include workspace-level access controls, IP allowlists, PrivateLink, network policies, and automatic container rebuilding every 30 days for compliance.
* [Foundation Model API Service Offerings](/concepts/foundation-model-api-offerings.md) - Three tiers of Databricks generative AI model access — pay-per-token, provisioned throughput, and fine-tuning — each with distinct retirement timelines and policies.
* [Foundation Model APIs](/concepts/foundation-model-apis.md) - Databricks-curated set of state-of-the-art open foundation model architectures (e.g., Meta-Llama-3.3-70B-Instruct) supporting optimized inference with pay-per-token or provisioned throughput pricing.
* [Foundation Model APIs (Databricks)](/concepts/foundation-model-apis-databricks.md) - Databricks-managed API infrastructure for hosting and querying foundation models including vision models via Model Serving.
* [Foundation Model APIs Access Control (Databricks)](/concepts/foundation-model-apis-access-control-databricks.md) - Workspace-level access controls for Foundation Model API endpoints, including admin-only governance modifications, IP allowlists, PrivateLink support, and outbound network policy configuration.
* [Foundation Model APIs Compliance Standards](/concepts/foundation-model-apis-compliance-standards.md) - Enterprise compliance certifications (HIPAA, PCI-DSS, FedRAMP, IRAP, CCCS, UK Cyber Essentials Plus) supported by Databricks Foundation Model APIs, with availability depending on deployment mode (pay-per-token vs provisioned throughput).
* [Foundation Model APIs Limits and Compliance](/concepts/foundation-model-apis-limits-and-compliance.md) - Rate limits, quotas, and compliance standards specific to Foundation Model APIs workloads including provisioned throughput, pay-per-token, and batch inference.
* [Foundation Model APIs Pay-per-Token](/concepts/foundation-model-apis-pay-per-token.md) - A Databricks offering that provides access to generative AI models with pay-per-token pricing, governed by a specific model retirement policy with defined transition periods.
* [Foundation Model APIs Provisioned Throughput](/concepts/foundation-model-apis-provisioned-throughput.md) - A Databricks offering for dedicated model serving capacity with provisioned throughput, governed by a specific model retirement policy with defined transition periods.
* [Foundation Model APIs Rate Limits](/concepts/foundation-model-apis-rate-limits.md) - Rate limits enforced on Databricks Foundation Model APIs to ensure reliable performance and fair resource allocation, varying by workspace tier, model type, and deployment method.
* [Foundation Model APIs Security Best Practices](/concepts/foundation-model-apis-security-best-practices.md) - Security measures for Foundation Model APIs including workspace-level access controls, IP allowlists, PrivateLink configurations, network policy restrictions, and endpoint governance by workspace admins.
* [Foundation Model Context Windows](/concepts/foundation-model-context-windows.md) - The maximum token limits for model inputs and outputs across supported models, ranging from 128K to 1M tokens, with common configurations like 400K input / 128K output for OpenAI GPT models.
* [Foundation Model Fine-tuning (Databricks)](/concepts/foundation-model-fine-tuning-databricks.md) - A Databricks offering for fine-tuning foundation models, governed by a specific model retirement policy with defined transition periods and recommended replacement models.
* [Foundation Model Fine-tuning on Databricks](/concepts/foundation-model-fine-tuning-on-databricks.md) - A feature within Databricks Model Training that lets users customize large language models using their own data via instruction fine-tuning, continued pre-training, and chat completion.
* [Foundation Model Fine-tuning Retirement Policy](/concepts/foundation-model-fine-tuning-retirement-policy.md) - The specific model retirement timelines, migration windows, and replacement recommendations for Databricks' Foundation Model Fine-tuning offering.
* [Foundation Model Permission Enforcement Scope](/concepts/foundation-model-permission-enforcement-scope.md) - The principle that foundation model Unity Catalog permissions are automatically enforced for pay-per-token and batch inference (AI Functions) but not for provisioned throughput endpoints.
* [Foundation Model Retirement and Migration Policy](/concepts/foundation-model-retirement-and-migration-policy.md) - Policies and timelines for retired foundation models including migration guidance and temporary re-routing strategies.
* [Foundation Model Serving Endpoints](/concepts/foundation-model-serving-endpoints.md) - Databricks Model Serving endpoints that deploy and serve foundation models, supporting external models and Foundation Model APIs.
* [Foundation Model Serving Endpoints (Databricks)](/concepts/foundation-model-serving-endpoints.md) - Managed endpoints on Databricks that deploy and serve foundation models, supporting both externally-hosted and Databricks-curated models.
* [Foundation Model Serving Modes](/concepts/foundation-model-serving-modes.md) - Two modes for accessing models via Databricks Foundation Model APIs: pay-per-token (usage-based pricing) and provisioned throughput (dedicated capacity recommended for production workloads).
* [Foundation Model Unity Catalog Permissions](/concepts/foundation-model-unity-catalog-permissions.md) - A Databricks feature that allows account admins to restrict which Databricks-hosted foundation models users and groups can access using Unity Catalog permissions on the system.ai schema.
* [Foundation Models APIs on Databricks](/concepts/foundation-models-apis-on-databricks.md) - Databricks-hosted foundation models available via pay-per-token APIs for embedding tasks, also called Databricks-hosted foundation models.
* [Freshness (data quality metric)](/concepts/freshness-data-quality.md) - A metric measuring how recently a table has been updated, using historical commit analysis to predict expected next commit time and flagging tables as stale if commits are unusually late.
* [Freshness (data quality)](/concepts/freshness-data-quality.md) - A data quality metric that measures how recently a table has been updated; anomaly detection builds a per-table model to predict next commit time and flags stale tables.
* [Freshness and Completeness Quality Metrics](/concepts/freshness-and-completeness-quality-metrics.md) - Two core data quality dimensions: freshness measures how recently a table was updated compared to predicted commit patterns; completeness measures whether row counts fall within expected historical ranges over 24 hours.
* [Freshness Monitoring (Databricks Anomaly Detection)](/concepts/freshness-monitoring-databricks-anomaly-detection.md) - A quality dimension that monitors how recently a table has been updated, using historical commit patterns to predict when the next commit should occur and flagging tables as stale if commits are unusually late.
* [FSCK REPAIR TABLE](/concepts/fsck-repair-table.md) - Command to detect and repair metadata inconsistencies in Delta Lake tables on Databricks
* [FSCK REPAIR TABLE Command](/concepts/fsck-repair-table-command.md) - A Delta Lake SQL command that repairs a table by detecting and reporting various types of corruption including missing files, CRC corruption, unreadable files, and invalid partition values.
* [FSDP (Fully Sharded Data Parallel)](/concepts/fsdp-fully-sharded-data-parallel.md) - A distributed training strategy that shards model parameters, gradients, and optimizer states across multiple GPUs to enable training of large models beyond a single GPU's memory capacity.
* [FSDP (Fully Sharded Data Parallel) on Databricks](/concepts/fsdp-fully-sharded-data-parallel-on-databricks.md) - A parallelism technique that shards model parameters, gradients, and optimizer states across GPUs to reduce memory usage for large models.
* [FSDP Checkpoint Consolidation with FullStateDictConfig](/concepts/fsdp-checkpoint-consolidation-with-fullstatedictconfig.md) - A pattern for saving FSDP-sharded models that gathers the full state dict from all ranks onto CPU of rank 0 and writes a consolidated checkpoint (via save_pretrained) to shared storage like Unity Catalog volumes.
* [FSDP on Databricks AI Runtime](/concepts/fsdp-on-databricks-ai-runtime.md) - Notebook-based examples and integration for running FSDP training workloads on Databricks AI Runtime on AWS
* [FSDP vs DDP (Distributed Data Parallel)](/concepts/fsdp-vs-ddp-distributed-data-parallel.md) - Comparison between FSDP and DDP, where DDP is simpler and recommended for smaller models that fit in a single GPU's memory, while FSDP provides better memory efficiency for large models.
* [FSDP vs DeepSpeed](/concepts/fsdp-vs-deepspeed.md) - Comparison between FSDP and DeepSpeed, where DeepSpeed offers advanced memory optimization features as an alternative to FSDP for large model training.
* [Full Supervised Fine-Tuning](/concepts/full-supervised-fine-tuning.md) - Complete fine-tuning approach for LLMs where all model parameters are updated during training
* [Fully Sharded Data Parallel (FSDP)](/concepts/fully-sharded-data-parallel-fsdp.md) - A PyTorch-native distributed training strategy for large model training, presented as an alternative to DeepSpeed
* [Fully Sharded Data Parallel (FSDP) for Large Model Training](/concepts/fully-sharded-data-parallel-fsdp-for-large-model-training.md) - A distributed training technique that shards model parameters, gradients, and optimizer states across GPUs to enable training of large models that exceed single GPU memory capacity.
* [Fully Sharded Data Parallel (FSDP) on Databricks](/concepts/fully-sharded-data-parallel-fsdp-on-databricks.md) - Memory-efficient training for large models using PyTorch FSDP, supported by the serverless_gpu @distributed API.
* [Function Calling Limitations (Public Preview)](/concepts/function-calling-limitations-public-preview.md) - Current limitations include no parallel function calling, max 32 functions, single-turn optimization, and restricted provisioned throughput support for new endpoints only.
* [Function Calling Trace Integration](/concepts/function-calling-trace-integration.md) - MLflow auto-traces function calling responses from Databricks Foundation Models, highlights function instructions in the trace UI, and supports decorating tool functions with @mlflow.trace to create spans for tool execution.
* [Function Calling Workflow](/concepts/function-calling-workflow.md) - The four-step sequence of calling the model with tools, receiving JSON function arguments, executing the function in code, and returning results to the model for summarization.
* [Functions and Registered Models](/concepts/functions-and-registered-models.md) - Functions in Unity Catalog represent reusable executable logic including UDFs, stored procedures, and registered MLflow models, with models having additional privileges for version management and tagging.
* [GDPR Discovery and Deletion with Classification](/concepts/gdpr-discovery-and-deletion-with-classification.md) - A use case and example notebook showing how to leverage data classification results to assist with data discovery and deletion for GDPR compliance.
* [Gemini Foundation Models Supporting Multimodal Input on Databricks](/concepts/gemini-foundation-models-supporting-multimodal-input-on-databricks.md) - The list of Gemini pay-per-token foundation models on Databricks that support audio and video inputs, including databricks-gemini-3-1-pro, databricks-gemini-3-pro, and others.
* [GenAI Agent Evaluation Workflow](/concepts/genai-agent-evaluation-workflow.md) - End-to-end process of creating an agent, defining custom judges, building a test dataset, and running evaluations to compare agent configurations
* [GenAI Agent Observability](/concepts/genai-agent-observability.md) - The practice of monitoring, tracing, debugging, and evaluating GenAI agent behavior using MLflow tools
* [GenAI App Evaluation Workflow on Databricks](/concepts/genai-app-evaluation-workflow-on-databricks.md) - End-to-end workflow pattern: instrument a GenAI app with tracing, collect end-user feedback, add developer annotations, conduct expert review via labeling sessions, and evaluate with scorers.
* [GenAI App Lifecycle Management](/concepts/genai-app-lifecycle-management.md) - Versioning, tracking, and governance for GenAI applications including prompts, apps, and traces across development and production
* [GenAI Application Evaluation Lifecycle](/concepts/genai-application-evaluation-lifecycle.md) - The iterative workflow of developing, evaluating, and monitoring GenAI applications across multiple phases including development testing and production monitoring
* [GenAI Evaluation and Quality Assessment](/concepts/genai-evaluation-and-quality-assessment.md) - Reviewing assessment scores from human feedback, LLM judges, and programmatic checks; inspecting evaluation datasets and scorers for GenAI agents
* [GenAI Evaluation Dataset Design](/concepts/genai-evaluation-dataset-design.md) - Practice of designing structured evaluation datasets with diverse input templates to systematically test GenAI application behavior across different scenarios.
* [GenAI human feedback and review tools](/concepts/genai-human-feedback-and-review-tools.md) - Databricks provides a Review App for human feedback on GenAI outputs, including a Chat UI for vibe checks and an expert feedback UI for labeling traces, enabling structured human evaluation of generative AI.
* [GenAI Observability](/concepts/genai-observability.md) - The practice of monitoring, debugging, and optimizing generative AI applications through comprehensive trace data and metadata analysis.
* [GenAI production monitoring with LLM judges](/concepts/genai-production-monitoring-with-llm-judges.md) - A Databricks service that continuously evaluates a sample of production GenAI traffic using LLM judges and scorers, powered by production-scale trace ingestion stored in Unity Catalog tables.
* [GenAI Trace Analysis and Debugging](/concepts/genai-trace-analysis-and-debugging.md) - The capability to investigate failing traces, examine span trees, pinpoint root causes, analyze latency, and identify bottlenecks in GenAI agent execution flows within MLflow.
* [GenAI Tracing Autolog on Serverless Clusters](/concepts/genai-tracing-autolog-on-serverless-clusters.md) - On Databricks serverless compute clusters, genAI tracing autologging must be explicitly enabled per integration
* [GENERATE Command (Delta Lake)](/concepts/generate-command-delta-lake.md) - A SQL statement used to generate manifest files for Delta Lake tables, primarily for compatibility with external query engines.
* [GENERATE statement (Delta Lake)](/concepts/generate-command-delta-lake.md) - SQL command to generate manifest files for Delta tables, enabling external query engines to read Delta Lake data.
* [GENERATE Syntax Constraints on Delta Table Names](/concepts/generate-syntax-constraints-on-delta-table-names.md) - When using the GENERATE command, the table name must not include temporal specifications or options specifications.
* [generate_test_cases from Production Sessions](/concepts/generate_test_cases-from-production-sessions.md) - A utility function that extracts test case goals and personas from existing production conversation sessions to reflect real user behavior.
* [Generating Test Cases from Production Conversations](/concepts/generating-test-cases-from-production-conversations.md) - Extracting structured test cases (goals and personas) from existing production conversation sessions using mlflow.genai.simulators.generate_test_cases, enabling test scenarios that reflect real user behavior.
* [Genie Agent Sharing Preview](/concepts/genie-agent-sharing-preview.md) - An account-level preview feature that must be enabled for Genie Space sharing via OpenSharing to work.
* [Genie Code](/concepts/genie-code.md) - A natural language interface within Databricks/MLflow for conversational exploration of GenAI observability and evaluation data
* [Genie Code (AI Assistant for Data Science)](/concepts/genie-code-ai-assistant-for-data-science.md) - An AI assistant integrated across Databricks notebooks and workspace for development, debugging, and operations, drawing on specialized enterprise knowledge.
* [Genie Code (AI-Assisted Development)](/concepts/genie-code-ai-assisted-development.md) - An AI assistant integrated across notebooks and workspace on Databricks, used for development, debugging, and ongoing operations with specialized knowledge of enterprise context.
* [Genie Code (DS Agent)](/concepts/genie-code-ds-agent.md) - An AI-assisted agent on Databricks that performs automated EDA, feature discovery, code generation, AutoML, and model serving diagnostics using natural language.
* [Genie Code AI Assistant for ML](/concepts/genie-code-ai-assistant-for-ml.md) - An AI-powered assistant that accelerates the ML lifecycle by performing automated EDA, generating code for feature pipelines, acting as intelligent AutoML from natural language requests, and diagnosing model serving endpoint issues.
* [Genie Code Best Practice Guidance](/concepts/genie-code-best-practice-guidance.md) - Provides recommendations on scaling configurations, endpoint states, monitoring setup, and links to relevant documentation.
* [Genie Code Deployment Diagnostics](/concepts/genie-code-deployment-diagnostics.md) - Capability to diagnose deployment failures, analyze build logs and endpoint state, and assess pending configuration changes.
* [Genie Code for Agent Observability](/concepts/genie-code-for-agent-observability.md) - A natural language interface for analyzing traces, debugging errors, and exploring evaluation data in MLflow Tracing.
* [Genie Code for Deep Learning](/concepts/genie-code-for-deep-learning.md) - An AI assistant feature that supports deep learning workloads on AI Runtime with code generation, debugging, and optimization suggestions
* [Genie Code for Feature Search](/concepts/genie-code-for-feature-search.md) - A natural-language query feature in Databricks that allows users to find feature tables using `/findTables` commands mentioning 'features' or 'feature tables'.
* [Genie Code for Feature Table Discovery](/concepts/genie-code-for-feature-table-discovery.md) - Using the Genie Code assistant's /findTables command with 'features' or 'feature tables' queries to locate relevant feature tables.
* [Genie Code for Lineage](/concepts/genie-code-for-lineage.md) - A natural-language interface in Databricks that allows users to query table lineages and insights using commands such as /getTableLineages and /getTableInsights.
* [Genie Code for Lineage Queries](/concepts/genie-code-for-lineage-queries.md) - Natural-language assistant in Catalog Explorer that can answer lineage questions via commands like /getTableLineages and /getTableInsights to show dependencies and usage patterns.
* [Genie Code for ML](/concepts/genie-code-for-ml.md) - AI-assisted data science tool on Databricks that automates exploratory data analysis, feature engineering, model training (AutoML), deployment, and endpoint diagnostics via natural language.
* [Genie Code for Model Serving Observability](/concepts/genie-code-for-model-serving-observability.md) - An AI-powered read-only advisor that helps diagnose issues, analyze performance, and provide guidance for Databricks model serving endpoints.
* [Genie Code for Trace Analysis](/concepts/genie-code-for-trace-analysis.md) - A natural language interface that allows users to query, explore, and debug traces using plain English questions instead of writing queries or navigating UI pages.
* [Genie Code Instrumentation Guidance](/concepts/genie-code-instrumentation-guidance.md) - Genie Code provides help adding tracing to code with autolog(), @mlflow.trace, or manual spans, including runnable code snippets that can be pasted into Databricks notebooks.
* [Genie Code Latency and Error Analysis](/concepts/genie-code-latency-and-error-analysis.md) - Ability to investigate high latency, timeout issues, and error patterns using metrics, events, inference tables, and service logs.
* [Genie Code Requirements](/concepts/genie-code-requirements.md) - To use Genie Code for agent observability and evaluation, the workspace needs Partner-powered AI features enabled for both account and workspace, and the workspace must be in a supported region as Genie Code uses Geos for data residency.
* [Genie Space Mount Prerequisites](/concepts/genie-space-mount-prerequisites.md) - To mount a shared Genie Space, a user needs the USE PROVIDER privilege, CREATE CATALOG privilege on the Unity Catalog metastore, and access to a pro or serverless SQL warehouse.
* [Genie Space OpenSharing](/concepts/genie-space-opensharing.md) - A feature that allows sharing a Genie Space with users outside your organization via a point-in-time snapshot of data assets and instructions.
* [Genie Space OpenSharing (Beta)](/concepts/genie-space-opensharing-beta.md) - A Databricks feature (currently in Beta) that allows data providers to share Genie Spaces with recipients across workspaces using the OpenSharing protocol.
* [Genie Space Sharing Limitations](/concepts/genie-space-sharing-limitations.md) - Constraints on sharing Genie Spaces: snapshot-only (no live updates), 256 KB compressed size limit for configuration, and inability to share spaces containing metric views.
* [Genie Space Sharing Recipient Management](/concepts/genie-space-sharing-recipient-management.md) - The ability to add additional recipients to an existing shared Genie Space snapshot, but inability to modify data assets in the share after creation.
* [Genie Space Sharing Requirements](/concepts/genie-space-sharing-requirements.md) - Prerequisites for sharing a Genie Space, including Genie Agent Sharing preview, OpenSharing setup, CREATE SHARE privilege, CAN EDIT permission, SELECT on data assets, and at least one recipient.
* [Genie Space Snapshot](/concepts/genie-space-snapshot.md) - A point-in-time capture of a Genie Space's tables, instructions, curated SQL examples, SQL functions, and configuration, created when the Share button is clicked.
* [GEPA (Gradient-Free Evolutionary Prompt Algorithm)](/concepts/gepa-gradient-free-evolutionary-prompt-algorithm.md) - An evolutionary optimization algorithm used by MLflow to iteratively refine prompt templates without gradient computation
* [GEPA Optimization Algorithm](/concepts/gepa-optimization-algorithm.md) - Generative Evolutionary Prompt Algorithm (GEPA) - the prompt optimization algorithm implemented via GepaPromptOptimizer, researched by Databricks AI Research Team, which iteratively refines prompts using LLM-driven reflection and automated feedback.
* [GepaPromptOptimizer](/concepts/gepapromptoptimizer.md) - An MLflow prompt optimizer that uses a reflection model to iteratively improve prompt templates based on evaluation scores from custom scorers.
* [get_scorer Dynamic Creator](/concepts/get_scorer-dynamic-creator.md) - A function in mlflow.genai.scorers.ragas that allows creating a RAGAS scorer at runtime by passing the metric name as a string, enabling dynamic evaluation workflows.
* [Git Folder Limitation with MLflow 3 Tracing](/concepts/git-folder-limitation-with-mlflow-3-tracing.md) - Known limitation where MLflow 3 real-time tracing does not work by default when deploying agents from notebooks stored in Databricks Git folders, requiring explicit experiment assignment via mlflow.set_experiment() to a non-Git-associated experiment.
* [Git Snapshot Pinning](/concepts/git-snapshot-pinning.md) - Pin code versions in training jobs by git branch or commit SHA using the `git:` block inside `code_source`, with optional remote HEAD fetching via `git.remote`.
* [Git-based code workflows](/concepts/git-based-code-workflows.md) - Approach for submitting training workloads using code stored in git repositories.
* [Global Mode Ray Cluster](/concepts/global-mode-ray-cluster.md) - A shared Ray cluster on Databricks accessible by multiple users with no automatic inactivity timeout, started via setup_global_ray_cluster
* [Global mode Ray clusters](/concepts/global-mode-ray-cluster.md) - Shared Ray clusters accessible by all users attached to a Databricks cluster, created via setup_global_ray_cluster, with no automatic shutdown timeout
* [Global Tags](/concepts/global-tags.md) - Classification tags that detect sensitive data regardless of geographic origin and are available in all Databricks workspace regions.
* [Global vs Per-Row Guidelines](/concepts/global-vs-per-row-guidelines.md) - Two strategies for applying evaluation criteria: uniform Guidelines() across all rows vs row-specific ExpectationsGuidelines() for domain-expert-labeled datasets.
* [Global vs Per-Row Guidelines Judges](/concepts/global-vs-per-row-guidelines-judges.md) - MLflow provides two built-in guidelines judges: `Guidelines()` for uniform global rules across all rows, and `ExpectationsGuidelines()` for per-row guidelines labeled by domain experts in the evaluation dataset.
* [Global vs User-Specific Ray Clusters](/concepts/global-vs-user-specific-ray-clusters.md) - Two lifecycle modes for Ray clusters on Databricks: single-user clusters tied to notebook sessions and shared global clusters accessible by all workspace users.
* [Gloo Errors in Distributed CPU Training](/concepts/gloo-errors-in-distributed-cpu-training.md) - Connection refused errors when using the Gloo backend for distributed training on CPU instances, resolved by setting the GLOO_SOCKET_IFNAME environment variable.
* [GluonTS](/concepts/gluonts.md) - A library for probabilistic time-series forecasting, used with the DeepAR model on Databricks
* [GluonTS Evaluation Metrics](/concepts/gluonts-evaluator.md) - Built-in evaluation toolkit in GluonTS that computes standard forecasting metrics including MASE, RMSE, and quantile losses across individual time series and aggregated results.
* [GluonTS Evaluator](/concepts/gluonts-evaluator.md) - A component of the GluonTS library that computes standard forecasting metrics including MASE, RMSE, and quantile losses across multiple time series.
* [GluonTS for Time Series Forecasting](/concepts/gluonts-for-time-series-forecasting.md) - GluonTS is a probabilistic time-series forecasting toolkit used for end-to-end workflows including data ingestion, resampling, model training, prediction, visualization, and evaluation.
* [GluonTS PandasDataset](/concepts/gluonts-pandasdataset.md) - A GluonTS dataset wrapper that converts Pandas DataFrames into the time series format required by GluonTS models, enabling seamless integration with standard data science workflows.
* [Golden Dataset](/concepts/golden-dataset.md) - A 'golden set' of evaluation examples that must always work correctly, used to prevent regressions when testing fixes and changes to prompts, models, or app logic.
* [Golden Set](/concepts/golden-set.md) - A curated collection of test examples that must always pass correctly, used to prevent regressions when iterating on prompts, models, or application logic.
* [Google Gemini API (on Databricks)](/concepts/google-gemini-api-on-databricks.md) - Databricks-hosted access to Google's Gemini API for Gemini models, uniquely supporting text, image, video, and audio inputs.
* [Google Gemini API Integration on Databricks](/concepts/google-gemini-api-integration-on-databricks.md) - Using the Google Gemini API client on Databricks to send media as inlineData (base64) or fileData (URL reference) within the parts array, with Databricks-specific authentication.
* [Google GenAI SDK Custom Endpoint Configuration](/concepts/google-genai-sdk-custom-endpoint-configuration.md) - Using the google.genai Python SDK with custom base_url and headers in http_options to route API calls through a Databricks serving endpoint instead of directly to Google.
* [Governance of ML and AI Assets in Unity Catalog](/concepts/governance-of-ml-and-ai-assets-in-unity-catalog.md) - Unified governance of data, ML assets, model endpoints, and ML tooling through Unity Catalog, AI Gateway, and Databricks AI Security providing a single security and administration layer.
* [Governed Tag (Databricks)](/concepts/governed-tag-databricks.md) - A tag type in Databricks Unity Catalog used for governing and classifying sensitive data, with permissions, allowed values, and integration with ABAC column-level masks.
* [Governed Tag Permissions in Unity Catalog](/concepts/governed-tag-permissions-in-unity-catalog.md) - The ASSIGN permission on system-governed tags and APPLY TAG privilege required to apply certification status tags to Unity Catalog objects.
* [Governed Tags](/concepts/governed-tags.md) - User-defined or system-defined key-value metadata tags applied to securable objects in Unity Catalog that can be referenced in ABAC GRANT policy conditions to dynamically determine access.
* [Governed Tags (Databricks)](/concepts/governed-tag-databricks.md) - Tags managed under a governance policy that can be assigned to data assets; custom classifiers use governed tags to label detected sensitive data columns in Unity Catalog.
* [Governed Tags (Unity Catalog)](/concepts/governed-tags-unity-catalog.md) - Controlled, administrator-defined tag vocabularies that users can apply to catalog objects to organize and categorize data by topic, team, or domain.
* [Governed Tags and System Tags](/concepts/governed-tags-and-system-tags.md) - Tag metadata used in GRANT policy conditions — governed tags are user-defined, system tags are predefined by Databricks — to determine which securable objects a policy applies to.
* [Governed Tags for ABAC Policies](/concepts/governed-tags-for-abac-policies.md) - Tags applied to securable objects (catalogs, schemas, tables, columns) that policies use to match targets for row filtering or column masking.
* [Governed Tags for ABAC Policy Discovery](/concepts/governed-tags-for-abac-policy-discovery.md) - Using governed tags on columns to automatically identify which columns should participate in row filter and column mask policies via MATCH COLUMNS, enabling attribute-based access control.
* [Governed Tags for Advanced Approvals](/concepts/governed-tags-for-advanced-approvals.md) - Using Databricks governed tag policies to control who can approve deployment jobs, enabling multi-group approval workflows (e.g., separate Legal and Security gates) and preventing model owner self-approval.
* [Governed tags for column classification](/concepts/governed-tags-for-column-classification.md) - A governed tag system in Unity Catalog that associates key-value metadata (e.g., domain, sensitivity) with columns, which can be queried by policy MATCH COLUMNS clauses to dynamically target columns.
* [Governed Tags in Unity Catalog](/concepts/governed-tags.md) - Attributes/ tags associated with securable objects that are used in ABAC policy conditions to identify which data a policy should protect.
* [Governed Tags Integration](/concepts/governed-tags-integration.md) - Custom classifiers assign governed tags to detected columns, enabling ABAC column-level masks and auto-tagging for organization-specific data types.
* [Governed Tags Vocabulary](/concepts/governed-tags-vocabulary.md) - Controlled tag vocabularies that administrators create for users to organize and categorize catalog objects by topic, team, or domain, making data browseable and filterable.
* [GPT-OSS 120B Model Fine-Tuning](/concepts/gpt-oss-120b-model-fine-tuning.md) - The supervised fine-tuning of a 120 billion parameter GPT-OSS model using distributed training techniques on 8 H100 GPUs with Databricks Serverless GPU Compute.
* [GPU Build Failures and Flash-Attention Dependencies](/concepts/gpu-build-failures-and-flash-attention-dependencies.md) - Handling GPU-specific container build failures in model serving, including flash-attn dependency management with custom wheels and CUDA version compatibility.
* [GPU Capacity Limitations in Model Deployment](/concepts/gpu-capacity-limitations-in-model-deployment.md) - A known limitation where model deployment on Databricks may fail due to GPU capacity issues, causing timeout during endpoint creation or update.
* [GPU Memory Optimization for Large Language Models](/concepts/gpu-memory-optimization-for-large-language-models.md) - Techniques such as gradient accumulation fusion and CPU offloading used to reduce GPU memory footprint when training models ranging from 1B to 100B+ parameters.
* [GPU Optimization for Deep Learning on Databricks](/concepts/gpu-optimization-for-deep-learning-on-databricks.md) - Best practices for GPU selection (including A100 GPUs), GPU scheduling, and leveraging Single Node GPU clusters for cost-effective deep learning development.
* [GPU Resource Availability for Model Serving](/concepts/gpu-resource-availability-for-model-serving.md) - Constraints and failure modes related to GPU supply limitations when building and deploying GPU-backed model serving endpoints on Databricks.
* [GPU Resource Conservation](/concepts/gpu-resource-conservation.md) - Best practice of attaching notebooks to CPU clusters for non-GPU operations (e.g., Git cloning, data conversion, EDA) to preserve GPU resources.
* [GPU Resource Management for Deep Learning on Databricks](/concepts/gpu-resource-management-for-deep-learning-on-databricks.md) - Guidance on selecting GPU instances (including A100 GPUs), GPU scheduling optimization, and using cluster policies to guide resource choices for deep learning workloads on Databricks.
* [GPU Resource Management for Model Serving](/concepts/gpu-resource-management-for-model-serving.md) - Handling GPU availability, GPU container builds, and custom GPU serving configurations in Databricks Model Serving.
* [GPU resource preservation strategy](/concepts/gpu-resource-preservation-strategy.md) - Operations that don't require GPUs (e.g., Git cloning, data conversion, EDA) should use a CPU cluster instead of serverless GPU to preserve GPU resources.
* [GPU Resources Monitoring Pane](/concepts/gpu-resources-monitoring-pane.md) - A Databricks notebook side pane that displays per-GPU utilization, memory usage, and temperature metrics for AI Runtime workloads, polling every 10 seconds with 2 hours of history.
* [GPU Runtime Utilities (serverless_gpu.runtime)](/concepts/gpu-runtime-utilities-serverless_gpuruntime.md) - Runtime module providing get_local_rank() and get_global_rank() for managing multi-GPU process identity
* [GPU Training for XGBoost on Databricks](/concepts/gpu-accelerated-xgboost-training.md) - Support for GPU-accelerated XGBoost training on Databricks clusters using the use_gpu parameter, with compute capability requirements for different runtime versions
* [GPU Training for XGBoost on Spark](/concepts/gpu-training-for-xgboost-on-spark.md) - GPU-accelerated XGBoost training on Databricks clusters via the use_gpu parameter, supported from Databricks Runtime 9.1 LTS ML and above.
* [GPU Training with xgboost.spark](/concepts/gpu-training-with-xgboostspark.md) - Enabling GPU-accelerated training by setting use_gpu=True, with recommendation to keep spark.task.resource.gpu.amount=1 to avoid idle GPUs.
* [GPU Utilization Monitoring Dashboard](/concepts/gpu-utilization-monitoring-dashboard.md) - A Databricks dashboard for monitoring average GPU utilization across MLflow experiments, helping identify inefficient resource usage across a workspace.
* [GPU Utilization Monitoring with MLflow](/concepts/gpu-utilization-monitoring-dashboard.md) - Using MLflow system tables dashboards to monitor average GPU utilization across experiments and identify inefficient resource usage
* [GPU workload considerations for model serving](/concepts/gpu-workload-limitations-in-model-serving.md) - GPU serving has longer container build times, a 60-minute deployment timeout, storage constraints for large models, slower autoscaling than CPU, and no GPU capacity guarantee after scale-to-zero — large models should use Foundation Model APIs instead.
* [GPU Workload Limitations for Databricks Model Serving](/concepts/gpu-workload-limitations-in-model-serving.md) - Specific limitations when serving models on GPU: longer container image creation, potential timeouts for very large models exceeding 60-minute deployment, longer autoscaling, and no GPU capacity guarantee when scaling to zero.
* [GPU Workload Limitations for Model Serving](/concepts/gpu-workload-limitations-in-model-serving.md) - Specific limitations when deploying custom models on GPU: longer container build times, potential timeout for large models (60-minute limit), longer autoscaling, and no guaranteed GPU capacity when scaling to zero.
* [GPU Workload Limitations in Model Serving](/concepts/gpu-workload-limitations-in-model-serving.md) - GPU serving endpoints have longer container build times, potential timeout for large models exceeding 60-minute builds, longer autoscaling, and no GPU capacity guarantee when scaling to zero.
* [GPU Workload Type Selection Deprecation](/concepts/gpu-workload-type-selection-deprecation.md) - The move away from requiring customers to manually select GPU workload types for LLM serving endpoints, with legacy API-only support remaining for those who need to continue selecting GPU types.
* [GPU Workload Types for Model Serving](/concepts/gpu-workload-types-for-model-serving.md) - GPU compute configurations available for Databricks Model Serving endpoints, with concurrency determining replica count (replicas = concurrency / 4).
* [GPU Workload Types in Databricks Model Serving](/concepts/gpu-workload-types-in-databricks-model-serving.md) - GPU compute options for model serving endpoints supporting PyTorch 1.13-2.0.1, TensorFlow 2.5-2.13, and MLflow 2.4+; concurrency determines replica count (concurrency / 4).
* [GPU-Accelerated Computer Vision on Databricks](/concepts/gpu-accelerated-computer-vision-on-databricks.md) - Using GPU-optimized AI Runtime on Databricks for computer vision tasks including object detection and image classification.
* [GPU-Accelerated Recommendation Systems on Databricks](/concepts/gpu-accelerated-recommendation-systems-on-databricks.md) - The practice of building and training deep learning based recommendation models using GPU acceleration within the Databricks environment on AWS.
* [GPU-Accelerated XGBoost on Databricks](/concepts/gpu-accelerated-xgboost-training.md) - Enabling GPU training for XGBoost models on Databricks clusters by setting use_gpu=True, with compute capability constraints.
* [GPU-Accelerated XGBoost Training](/concepts/gpu-accelerated-xgboost-training.md) - Using GPU clusters for XGBoost training on Databricks by setting use_gpu=True in sparkdl.xgboost estimators.
* [Gradient Accumulation Fusion](/concepts/gradient-accumulation-fusion.md) - A DeepSpeed feature that fuses gradient accumulation steps to improve training efficiency and throughput
* [Gradient Boosted Trees Regression with MLlib](/concepts/gradient-boosted-trees-regression-with-mllib.md) - Using MLlib pipelines to perform regression with gradient boosted trees, demonstrated with bike rental count prediction.
* [Gradient Checkpointing for Memory-Efficient Training](/concepts/gradient-checkpointing-for-memory-efficient-training.md) - A memory optimization technique that trades computation for memory by not storing intermediate activations during forward pass, instead recomputing them during backward pass, enabling larger batch sizes on limited GPU memory.
* [Gradient Checkpointing with Liger Kernels in Distributed Training](/concepts/gradient-checkpointing-with-liger-kernels-in-distributed-training.md) - A memory optimization technique that trades computation for memory by recomputing intermediate activations during backpropagation, combined with Liger kernel optimizations to fit larger models on GPU.
* [GRANT policies (ABAC Beta)](/concepts/grant-policies-beta.md) - A beta feature in Unity Catalog ABAC supporting dynamic privilege grants through GRANT policies, currently scoped to EXECUTE on models.
* [GRANT Policies (Beta)](/concepts/grant-policies-beta.md) - ABAC policy type for dynamic privilege grants, currently scoped to EXECUTE permission on models in Unity Catalog
* [GRANT policies in Unity Catalog](/concepts/grant-policies-in-unity-catalog.md) - ABAC policy type (Beta) for dynamic privilege grants, currently scoped to EXECUTE on models.
* [GRANT Policies in Unity Catalog (Beta)](/concepts/grant-policies-in-unity-catalog.md) - A type of ABAC policy supporting dynamic privilege grants, currently scoped to EXECUTE on models in Unity Catalog.
* [GRANT Policies vs Direct Grants](/concepts/grant-policies-vs-direct-grants.md) - The interaction model between dynamic tag-based GRANT policies and explicit direct grants, where effective privileges are the union of both mechanisms.
* [GRANT Policy Audit Logging](/concepts/grant-policy-audit-logging.md) - GRANT policy create, alter, and drop operations are logged under the same audit actions (createPolicy, deletePolicy, getPolicy, listPolicies) as row filter and column mask policies.
* [GRANT Policy Best Practices](/concepts/grant-policy-best-practices.md) - Recommendations for designing maintainable GRANT policies: use groups not individuals, attach at smallest scope, use tag inheritance with EXCEPT for exceptions, avoid mixing with direct grants.
* [GRANT Policy Beta Limitations](/concepts/grant-policy-beta-limitations.md) - Current Beta constraints: only EXECUTE on models is supported, policies attach at catalog/schema level only, SHOW GRANTS does not reflect policy-granted privileges, and Delta Sharing is incompatible.
* [GRANT Policy Effective Privilege Union Model](/concepts/effective-privileges-union-model.md) - The effective privileges on a securable object are the union of direct grants and any applicable GRANT policies, meaning a more selective policy does not override existing direct grants.
* [GRANT Policy Interaction with Direct Grants](/concepts/grant-policy-interaction-with-direct-grants.md) - Effective privileges are the union of direct grants and GRANT policies; a principal holds EXECUTE if either mechanism grants it, making mixing the two approaches potentially confusing.
* [GRANT Policy Limitations and Beta Scope](/concepts/grant-policy-limitations-and-beta-scope.md) - Current Beta limitations: only EXECUTE privilege on models is supported, policies can only be attached at catalog/schema level (not model), SHOW GRANTS doesn't reflect policy-granted privileges, and Delta Sharing is not compatible.
* [GRANT Policy Management and Auditing](/concepts/grant-policy-management-and-auditing.md) - SQL commands (CREATE POLICY, ALTER POLICY, SHOW POLICIES, DESCRIBE POLICY, DROP POLICY), Catalog Explorer UI operations, and audit logging for managing ABAC GRANT policies in Unity Catalog.
* [GRANT Policy Management Operations](/concepts/grant-policy-management-operations.md) - SQL and UI operations for creating, editing, deleting, showing, and describing GRANT policies, including the USE CATALOG/SCHEMA prerequisites and SHOW EFFECTIVE POLICIES for inherited policies.
* [GRANT Policy Management via Catalog Explorer and SQL](/concepts/grant-policy-management-via-catalog-explorer-and-sql.md) - How to create, edit, delete, list, and describe GRANT policies using the Catalog Explorer UI, SQL statements (CREATE POLICY, DESCRIBE POLICY, SHOW POLICIES), and the Databricks Python SDK.
* [GRANT Policy Quotas and Limitations (Beta)](/concepts/grant-policy-quotas-and-limitations-beta.md) - The current Beta restrictions on GRANT policies including only EXECUTE on models, no direct model attachment, no SHOW GRANTS visibility, and separate quotas.
* [GRANT Policy SQL Syntax](/concepts/grant-policy-sql-syntax.md) - SQL syntax for creating, describing, showing, and deleting GRANT policies, including CREATE POLICY, DESCRIBE POLICY, SHOW POLICIES, and SHOW EFFECTIVE POLICIES.
* [GRANT Policy Syntax and Management](/concepts/grant-policy-syntax-and-management.md) - SQL syntax (CREATE POLICY, DESCRIBE POLICY, SHOW POLICIES, SHOW EFFECTIVE POLICIES, DROP POLICY) and Catalog Explorer UI for creating, editing, deleting, and inspecting ABAC GRANT policies.
* [GRANT Policy Syntax and Structure](/concepts/grant-policy-syntax-and-structure.md) - The CREATE POLICY SQL syntax for defining GRANT policies, including TO/EXCEPT principals, GRANT EXECUTE FOR MODELS, and WHEN conditions using functions like has_tag_value.
* [GRANT Policy vs Direct Grant](/concepts/grant-policy-vs-direct-grant.md) - Key distinction between dynamic tag-based GRANT policies that match by governed tags versus static direct GRANT statements that identify objects by their three-level namespace.
* [GRANT Policy vs Direct Grants](/concepts/grant-policy-vs-direct-grant.md) - Comparison between dynamic attribute-based GRANT policies that match on tags versus traditional explicit GRANT statements that name specific objects in their three-level namespace.
* [Grant Share to SAP BDC](/concepts/grant-share-to-sap-bdc.md) - The workflow of granting an SAP BDC recipient access to selected shares through the Databricks OpenSharing interface.
* [Granting Data Shares to SAP BDC Recipients](/concepts/granting-data-shares-to-sap-bdc-recipients.md) - The step-by-step process of granting an SAP BDC recipient access to specific OpenSharing shares in Databricks
* [Granting Share Access to Recipients](/concepts/granting-share-access-to-recipients.md) - The process of adding a data recipient to an OpenSharing share so they can access the shared data assets.
* [Graph and Network Analysis](/concepts/graph-and-network-analysis.md) - The practice of modeling and analyzing systems using graphs (vertices connected by edges) to understand relationships, structures, and patterns in domains such as social networks, transportation, fraud detection, and recommendations.
* [Graph and Network Analysis on Databricks](/concepts/graph-and-network-analysis-on-databricks.md) - Overview of Databricks capabilities for graph processing, combining Databricks Runtime ML with NetworkX, GraphFrames, and external tools
* [Graph Processing on Databricks](/concepts/graph-processing-on-databricks.md) - Databricks provides capabilities for graph analysis at any scale: NetworkX for single-node processing of small networks, GraphFrames for distributed processing of large networks, and the ability to install additional open-source or third-party tools.
* [Graph Properties (Centrality, Shortest Path, Diameter, Density)](/concepts/graph-properties-centrality-shortest-path-diameter-density.md) - Key analytical metrics for graphs: shortest path (minimum distance between nodes), centrality (node importance measures like degree centrality and betweenness centrality), diameter (maximum shortest path across the network), and density (actual edges vs possible edges).
* [Graph Properties: Density, Diameter, and Degree Distribution](/concepts/graph-properties-density-diameter-and-degree-distribution.md) - Fundamental structural properties of networks including density, diameter, and degree distribution used to characterize graph topology
* [GraphFrames](/concepts/graphframes.md) - Distributed graph processing library for Apache Spark used on Databricks for large-scale network analysis
* [Groq SDK Integration with MLflow](/concepts/groq-sdk-integration-with-mlflow.md) - Integration enabling MLflow to automatically trace Groq chat completion requests
* [Ground Truth Comparison Strategies](/concepts/ground-truth-comparison-strategies.md) - Two approaches for providing ground truth to the Correctness judge: expected_facts (list of atomic facts) for flexible matching, and expected_response (full expected answer) for stricter comparison.
* [Ground Truth Evaluation Strategies](/concepts/ground-truth-evaluation-strategies.md) - Two approaches for evaluating LLM response correctness: expected_facts allows flexible fact-level checking, while expected_response requires more exact response matching.
* [Ground Truth in LLM Evaluation](/concepts/ground-truth-in-llm-evaluation.md) - A distinguishing feature of LLM judges where some judges require ground truth (expectations) to evaluate correctness or sufficiency, while others operate without it by evaluating based on inputs and outputs alone.
* [Ground truth requirement in LLM evaluation](/concepts/ground-truth-requirement-in-llm-evaluation.md) - Design pattern where some LLM judges (Correctness, RetrievalSufficiency, ToolCallCorrectness) require ground truth expectations as input, while others (RelevanceToQuery, Safety, RetrievalGroundedness) do not.
* [Ground Truth Requirements in LLM Evaluation](/concepts/ground-truth-requirements-in-llm-evaluation.md) - A design distinction among built-in judges where some evaluators require ground truth labels (e.g., Correctness, RetrievalSufficiency) and others evaluate purely based on inputs and outputs.
* [Grounded and Topical AI Applications](/concepts/grounded-and-topical-ai-applications.md) - A mitigation requirement that AI chat and conversational agents must be grounded or topical, meaning users cannot freely query the model for unrestricted outputs.
* [Grouping Runs by Parameter](/concepts/grouping-runs-by-parameter.md) - Feature to group MLflow runs by one or more parameter values for comparative analysis in chart views.
* [gRPC over HTTP/2 in Databricks Connect](/concepts/grpc-over-http2-in-databricks-connect.md) - Databricks Connect communicates with Databricks clusters via gRPC over HTTP/2, which is the underlying transport protocol for Spark Connect.
* [gRPC Protocol in Databricks Connect](/concepts/grpc-protocol-in-databricks-connect.md) - The underlying communication protocol used by Spark Connect that transmits Spark unresolved logical plans and Apache Arrow data over gRPC between the client and the remote Databricks cluster.
* [Guardrails AI Framework](/concepts/guardrails-ai-framework.md) - A community-driven framework for validating LLM outputs using a hub of validators for safety, PII detection, content quality, and more.
* [Guardrails AI scorer configuration](/concepts/guardrails-ai-scorer-configuration.md) - Guardrails AI scorers accept validator-specific parameters as keyword arguments to the constructor for customization
* [Guardrails AI Scorers Integration](/concepts/guardrails-ai-scorers-integration.md) - MLflow integration with Guardrails AI for rule-based output validation without LLM calls such as toxicity detection, PII scanning, and jailbreak detection
* [Guideline Authoring Best Practices](/concepts/guideline-authoring-best-practices.md) - Effective natural-language guidelines for LLM judges should be specific and measurable, use clear pass/fail conditions, explicitly reference context variables, and structure complex requirements with bullet points.
* [Guideline Writing Best Practices](/concepts/guideline-writing-best-practices.md) - Principles for authoring effective natural language evaluation criteria: be specific and measurable, use clear pass/fail conditions, reference context explicitly, and structure complex requirements.
* [Guidelines and Tool Call Judges](/concepts/guidelines-and-tool-call-judges.md) - A category of built-in LLM judges that evaluate responses against natural language criteria (Guidelines, ExpectationsGuidelines) or assess tool call quality (ToolCallCorrectness, ToolCallEfficiency).
* [Guidelines Judge Request/Response Parsing](/concepts/guidelines-judge-requestresponse-parsing.md) - The automatic extraction rules by which the Guidelines judge parses inputs and outputs into request and response context variables from OpenAI-format traces or arbitrary dicts.
* [Guidelines LLM Judge](/concepts/guidelines-llm-judge.md) - A pass/fail LLM-based evaluator that uses natural language criteria to assess GenAI outputs for compliance, tone, accuracy, and requirements.
* [Guidelines LLM Judges](/concepts/guidelines-llm-judges.md) - Pass/fail natural language criteria judges that evaluate whether inputs and outputs comply with specified guidelines, configurable with Databricks-hosted or custom model endpoints.
* [Guidelines Scorer](/concepts/guidelines-scorer.md) - An MLflow scorer that evaluates whether LLM responses adhere to user-defined natural language guidelines (e.g., language, tone, safety constraints).
* [Guidelines Writing Best Practices](/concepts/guideline-writing-best-practices.md) - Principles for writing effective guidelines: be specific and measurable, use clear pass/fail conditions, reference context explicitly, and structure complex requirements.
* [Guidelines-based LLM evaluation](/concepts/guidelines-based-llm-evaluation.md) - Evaluation approach using natural language criteria (via Guidelines and ExpectationsGuidelines judges) to assess whether responses meet specified requirements, with or without per-example expectations.
* [Guidelines() Judge (Global Guidelines)](/concepts/guidelines-judge-global-guidelines.md) - A built-in MLflow scorer that applies uniform natural-language guidelines across all evaluation rows or production traces, automatically extracting request/response data.
* [H100 GPU distributed training on Databricks](/concepts/h100-gpu-distributed-training-on-databricks.md) - Pattern for leveraging H100 GPUs in multi-node distributed training workloads on Databricks AI Runtime, supporting both FSDP and Ray Train backends.
* [H100 vs A10 GPU Selection Guidance](/concepts/h100-vs-a10-gpu-selection-guidance.md) - Guidance on when to use H100 GPUs over A10 GPUs: H100s are recommended for large model training requiring high throughput and large GPU memory due to larger FLOPS and HBM capacity.
* [Hadoop Configuration in Databricks Connect](/concepts/hadoop-configuration-in-databricks-connect.md) - How Hadoop configurations are managed in Databricks Connect, including the distinction between spark.conf.set (per-session) and sparkContext configurations (cluster-wide).
* [Hallucination Scorer](/concepts/hallucination-scorer.md) - An Arize Phoenix scorer that detects hallucinations in LLM outputs by comparing them against a provided context
* [Hallucination scorer (Phoenix)](/concepts/hallucination-scorer-phoenix.md) - An Arize Phoenix scorer used within MLflow GenAI evaluation that detects factual inaccuracies or fabricated information in LLM outputs relative to provided context.
* [HAS_INDEXES](/concepts/has_indexes.md) - A subclass of DELTA_CLONE_INCOMPATIBLE_SOURCE error indicating the source table has indexes that must be dropped before cloning can succeed.
* [HAS_INDEXES (Delta Clone sub-error)](/concepts/has_indexes.md) - A specific DELTA_CLONE_INCOMPATIBLE_SOURCE sub-error indicating the source table has indexes that must be dropped before cloning.
* [HAS_INDEXES error condition](/concepts/has_indexes.md) - A sub-error of DELTA_CLONE_INCOMPATIBLE_SOURCE indicating the source table has indexes that must be dropped before a Delta clone operation can proceed.
* [HAS_INDEXES sub-error](/concepts/has_indexes.md) - A specific error under DELTA_CLONE_INCOMPATIBLE_SOURCE indicating the source table has indexes that must be dropped before cloning into Delta.
* [Health Indicators (Catalog Explorer)](/concepts/health-indicators-catalog-explorer.md) - Visual status badges displayed on schema and table overview pages that summarize table health (healthy/unhealthy/error) for data consumers without requiring navigation to the monitoring UI.
* [Health Indicators (Databricks Anomaly Detection)](/concepts/health-indicators-databricks.md) - Visual status indicators shown in Catalog Explorer for each table within a monitored schema, summarizing table health for data consumers
* [Health Indicators (Databricks)](/concepts/health-indicators-databricks.md) - Visual status indicators shown in Catalog Explorer that summarize table health for data consumers, requiring SELECT or BROWSE permissions to view.
* [Heterogeneous Ray Clusters on Databricks](/concepts/heterogeneous-ray-clusters-on-databricks.md) - Ray on Spark clusters allow different configurations between head and worker nodes, but all worker nodes must share the same configuration; Databricks supports this via cluster policies with different driver and worker instance types.
* [High Throughput vs Low Latency LLM Use Cases](/concepts/high-throughput-vs-low-latency-llm-use-cases.md) - Classification of LLM application scenarios: high throughput for batch/non-user-facing tasks where latency can be sacrificed, and low latency for real-time interactive applications.
* [Historical Trace Backfilling](/concepts/historical-trace-backfilling.md) - The pattern of retrospectively applying evaluation scorers to previously collected trace data over a date range using the backfill_scorers API.
* [History Sharing and Time Travel](/concepts/history-sharing-and-time-travel.md) - Sharing the Delta transaction log history of a table to enable recipients to perform time travel queries, read with Spark Structured Streaming, and access Change Data Feed.
* [History Sharing for Shared Tables](/concepts/history-sharing-for-shared-tables.md) - A prerequisite feature on OpenSharing shares that enables metadata history, enabling Structured Streaming and change data feed capabilities on shared tables.
* [Hive Metastore Coexistence with Unity Catalog](/concepts/hive-metastore-coexistence-with-unity-catalog.md) - The per-workspace legacy Hive metastore can continue to be used alongside Unity Catalog in Databricks, appearing as a top-level catalog named `hive_metastore` in the three-level namespace.
* [Hive metastore deprecation in new workspaces](/concepts/hive-metastore-deprecation-in-new-workspaces.md) - The Hive metastore will not be available in new workspaces created after September 30, 2026; existing Hive tables and views must be upgraded to Unity Catalog.
* [Hive Metastore Fallback](/concepts/hive-metastore-fallback.md) - A mechanism that allows queries to fall back to legacy Hive metastore tables when they cannot be resolved in Unity Catalog, which is disabled when legacy access is turned off.
* [Hive Metastore Federation](/concepts/hive-metastore-federation.md) - A technique to federate an existing Hive metastore or AWS Glue catalog as a foreign catalog in Unity Catalog, enabling table upgrades without data movement.
* [Hive Metastore Federation to Unity Catalog](/concepts/hive-metastore-federation-to-unity-catalog.md) - Hive Metastore federation creates a foreign catalog in Unity Catalog that mirrors the Hive metastore, offering a gradual migration path as an alternative to upgrading all tables at once.
* [Hive Metastore Migration Prerequisites](/concepts/hive-metastore-migration-prerequisites.md) - The criteria that must be met before disabling legacy Hive metastore access, including completing table migration to Unity Catalog and upgrading all jobs to Databricks Runtime 13.3 LTS or above.
* [Hive Metastore Object Ownership](/concepts/hive-metastore-object-ownership.md) - The ownership model where creators of objects become owners (or group membership confers ownership), with ability to transfer ownership via ALTER command.
* [Hive Metastore Privilege Model (Legacy)](/concepts/hive-metastore-privilege-model-legacy.md) - The legacy access control system for the built-in Hive metastore in Databricks workspaces, distinct from Unity Catalog's privilege model.
* [Hive Metastore Privilege Types](/concepts/hive-metastore-privilege-types.md) - The set of privileges (SELECT, CREATE, MODIFY, USAGE, READ_METADATA, CREATE_NAMED_FUNCTION, MODIFY_CLASSPATH, ALL PRIVILEGES) used with legacy Hive metastore access control.
* [Hive Metastore Privileges and Securable Objects](/concepts/hive-metastore-privileges-and-securable-objects.md) - The permission model and set of securable objects (e.g., tables, databases, ANY FILE) used in the legacy Hive metastore table access control system.
* [Hive Metastore Resource Limits and Concurrency](/concepts/hive-metastore-resource-limits-and-concurrency.md) - The Databricks-hosted legacy Hive metastore has resource limits on concurrent connections and connections per hour; exceeding these can cause metastore connection errors, and mitigation includes migration to Unity Catalog and smoothing workload concurrency.
* [Hive Metastore Securable Objects](/concepts/hive-metastore-securable-objects.md) - The hierarchical object model (CATALOG → SCHEMA → TABLE/VIEW/FUNCTION, plus ANONYMOUS FUNCTION and ANY FILE) to which privileges can be granted in the legacy Hive metastore.
* [Hive Metastore Securable Objects and Privileges](/concepts/hive-metastore-securable-objects-and-privileges.md) - The permission model defining which Hive metastore objects (tables, databases, views, etc.) can be secured and what privileges can be granted on them using table access control.
* [Hive Metastore Table Access Control](/concepts/hive-metastore-table-access-control.md) - A Databricks security feature that restricts access to Hive metastore tables and views on a cluster based on administrator-defined privileges.
* [Hive Metastore Table Access Control (Legacy)](/concepts/hive-metastore-table-access-control-legacy.md) - A legacy Databricks data governance model that enables programmatic grant and revoke of permissions on Hive metastore objects via Python and SQL, per cluster.
* [Hive Metastore Table Upgrade Methods](/concepts/hive-metastore-table-upgrade-methods.md) - Techniques for upgrading Hive metastore tables to Unity Catalog, including ALTER TABLE SET MANAGED/EXTERNAL, SYNC, and CREATE TABLE AS SELECT.
* [Hive Metastore to Unity Catalog Migration](/concepts/hive-metastore-to-unity-catalog-migration.md) - Strategies for upgrading legacy Hive metastore managed and external Parquet tables to Unity Catalog Delta Lake tables using CONVERT TO DELTA or CTAS.
* [Hive Metastore to Unity Catalog Table Migration](/concepts/hive-metastore-to-unity-catalog-migration.md) - Strategies for converting legacy Hive metastore managed and external Parquet tables to Unity Catalog Delta Lake tables using CTAS or upgrade wizard
* [Hive-to-Unity-Catalog Impact Considerations](/concepts/hive-to-unity-catalog-impact-considerations.md) - Key behavioral changes when migrating: Unity Catalog manages partitions differently (no direct partition manipulation commands), and table history is not migrated with CLONE operations.
* [Holiday Region Covariates](/concepts/holiday-region-covariates.md) - A configurable option in Databricks forecasting that adds holiday region data as covariates (external features) during model training to improve forecast accuracy.
* [Horovod](/concepts/horovod.md) - A distributed deep learning framework that can be used on Databricks for scaling PyTorch and other ML models across multiple nodes.
* [Horovod Custom Installation on Databricks](/concepts/horovod-custom-installation-on-databricks.md) - Process to install a different version of Horovod on Databricks Runtime ML by uninstalling, recompiling from source, and reinstalling as a wheel.
* [Horovod Deprecation on Databricks](/concepts/horovod-deprecation-on-databricks.md) - Horovod and HorovodRunner are deprecated in Databricks Runtime ML after version 15.4 LTS ML, with TorchDistributor and tf.distribute.Strategy recommended as replacements.
* [Horovod Distributed Training Framework](/concepts/horovod.md) - Horovod is a distributed deep learning training framework supporting TensorFlow, Keras, and PyTorch, now deprecated on Databricks.
* [Horovod Extension Build Error Troubleshooting](/concepts/horovod-extension-build-error-troubleshooting.md) - Common ImportError for horovod.{torch|tensorflow} caused by missing required libraries at install time, resolved by reinstalling Horovod after the required framework.
* [Horovod for Distributed Deep Learning on Databricks](/concepts/horovod-for-distributed-deep-learning-on-databricks.md) - The integration of Uber's Horovod distributed training framework within the Databricks environment for scaling deep learning workloads
* [Horovod ImportError Troubleshooting](/concepts/horovod-importerror-troubleshooting.md) - A common error where importing horovod.torch or horovod.tensorflow fails because the extension was not built during installation, typically due to missing required libraries at compile time.
* [Horovod Installation and Compilation Constraints](/concepts/horovod-installation-and-compilation-constraints.md) - Requirement to reinstall Horovod against the same compiler and CUDA version as the deep learning library (TensorFlow/PyTorch) after upgrading or downgrading those libraries.
* [Horovod Installation on Databricks](/concepts/horovod-installation-on-databricks.md) - The process of upgrading or downgrading Horovod on Databricks Runtime ML clusters, requiring manual uninstallation, recompilation with CUDA libraries, and wheel installation.
* [Horovod Installation with Deep Learning Libraries](/concepts/horovod-installation-with-deep-learning-libraries.md) - Horovod must be reinstalled after upgrading or downgrading TensorFlow, Keras, or PyTorch, using the same compiler that the deep learning library was built with, and specifying GPU allreduce backend (NCCL).
* [Horovod Network Interface Configuration](/concepts/horovod-network-interface-configuration.md) - Configuration of OMPI_MCA_btl_tcp_if_include and NCCL_SOCKET_IFNAME environment variables to resolve Open MPI TCP connection issues in multi-node clusters.
* [Horovod on Spark](/concepts/horovod-on-spark.md) - A distributed deep learning package providing an estimator API for training Keras and PyTorch models on Apache Spark clusters.
* [Horovod Spark Estimator API with Keras](/concepts/horovod-spark-estimator-api-with-keras.md) - Using the horovod.spark estimator API to run distributed deep learning training with Keras models on Databricks.
* [Horovod Spark Estimator API with PyTorch](/concepts/horovod-spark-estimator-api-with-pytorch.md) - Using the horovod.spark estimator API to run distributed deep learning training with PyTorch models on Databricks.
* [Horovod Timeline](/concepts/horovod-timeline.md) - A performance tracing tool for Horovod that records activity timelines, accessible via Chrome tracing, but with significant performance overhead (~40% throughput reduction).
* [horovod.spark](/concepts/horovodspark.md) - A Spark integration package for Horovod that provides an estimator API for Spark ML pipelines using Keras or PyTorch models.
* [horovod.spark Estimator API](/concepts/horovodspark.md) - Spark ML pipeline estimator API for using Horovod with Keras and PyTorch models in Spark pipelines.
* [horovod.spark Package](/concepts/horovodspark-package.md) - A Spark estimator API package for distributed deep learning training with Horovod, supporting Keras and PyTorch on Databricks.
* [HorovodRunner](/concepts/horovodrunner.md) - A Databricks API for distributed deep learning using Horovod that allows adapting single-node PyTorch code to distributed training with minimal changes.
* [HorovodRunner Deprecation](/concepts/horovodrunner-deprecation.md) - Horovod and HorovodRunner are deprecated in Databricks ML runtime 15.4 LTS and later, with recommended replacements being TorchDistributor for PyTorch and tf.distribute.Strategy for TensorFlow.
* [HorovodRunner deprecation on Databricks](/concepts/horovodrunner-deprecation.md) - Horovod and HorovodRunner are deprecated on Databricks as of ML releases after 15.4 LTS; users must migrate to alternative distributed training APIs.
* [HorovodRunner development workflow](/concepts/horovodrunner-development-workflow.md) - The recommended workflow for developing distributed deep learning models using HorovodRunner on Databricks.
* [HorovodRunner Pickling and Import Constraints](/concepts/horovodrunner-pickling-and-import-constraints.md) - Requirement that all import statements must be included inside the Horovod training method (and any called functions) to avoid pickling errors during distribution to Spark executors.
* [HorovodRunner Pickling and Import Requirements](/concepts/horovodrunner-pickling-and-import-requirements.md) - HorovodRunner pickles the training method on the driver and distributes it to Spark workers, requiring all import statements to be included inside the training method to avoid pickling errors.
* [Hosted Third-Party Foundation Model Catalog](/concepts/hosted-third-party-foundation-model-catalog.md) - A curated catalog of state-of-the-art foundation models from OpenAI, Google, Anthropic, Meta, and Alibaba Cloud, hosted within the Databricks security perimeter and accessible via named endpoints.
* [HTTP vs Job Registry Webhooks](/concepts/http-vs-job-registry-webhooks.md) - Two types of webhooks based on trigger targets: HTTP endpoints (send HTTP requests to a URL) and job triggers (run a Databricks job in the same or different workspace).
* [Hub-and-Spoke Medallion Design](/concepts/hub-and-spoke-medallion-design.md) - An enterprise deployment pattern that combines medallion architecture with a centralized data hub and domain-specific spokes, enabling shared organization-wide assets alongside domain-curated data products.
* [Hugging Face Dataset Caching on Databricks](/concepts/hugging-face-dataset-caching-on-databricks.md) - Strategies for persisting Hugging Face dataset caches across cluster lifecycles using Unity Catalog volumes and HF_DATASETS_CACHE environment variable
* [Hugging Face Datasets on Databricks](/concepts/hugging-face-datasets-on-databricks.md) - Using the Hugging Face Datasets library within Databricks environments to load, prepare, and manage data for fine-tuning LLMs
* [Hugging Face Model Caching on Databricks](/concepts/hugging-face-model-caching-on-databricks.md) - Caching Hugging Face models in DBFS or mount points to reduce model load times and ingress costs across clusters by setting the TRANSFORMERS_CACHE environment variable
* [Hugging Face Trainer](/concepts/hugging-face-trainer.md) - The Trainer utility from Hugging Face transformers library for fine-tuning models, including metrics, data collators, and training configuration.
* [Hugging Face Trainer Utility on Databricks](/concepts/hugging-face-trainer-utility-on-databricks.md) - How to use the Hugging Face Transformers Trainer class for fine-tuning on single-GPU Databricks clusters
* [Hugging Face Transformers Pipelines for Inference](/concepts/hugging-face-transformers-pipelines-for-inference.md) - Using the Hugging Face pipelines class to run pre-trained NLP models for tasks like translation and named-entity recognition on Databricks
* [Hugging Face Transformers Trainer](/concepts/hugging-face-transformers-trainer.md) - A high-level training utility from the Hugging Face transformers library that orchestrates model fine-tuning, evaluation, and checkpointing with minimal boilerplate.
* [Human Feedback Collection for Judge Alignment](/concepts/human-feedback-collection-for-judge-alignment.md) - Best practices and methods for collecting domain expert feedback on judge assessments, including UI review and programmatic feedback, with guidance on diverse reviewers and balanced examples.
* [Human Feedback Collection in MLflow](/concepts/human-feedback-collection-in-mlflow.md) - Mechanism to collect end-user feedback (thumbs up/down) on GenAI app outputs using mlflow.log_feedback with assessment metadata.
* [Human Feedback Collection with mlflow.log_feedback()](/concepts/human-feedback-collection-with-mlflowlog_feedback.md) - Programmatic API for collecting end-user feedback (e.g., thumbs up/down) on GenAI app responses, associated with trace IDs and including rationale and source provenance.
* [Human Feedback for LLM Judge Alignment](/concepts/human-feedback-for-llm-judge-alignment.md) - Human feedback collected via MLflow can be used to create datasets for LLM judges and scorers, keeping automated evaluation systems aligned with human expert judgment and nuanced business requirements.
* [Human Feedback in LLM Monitoring](/concepts/human-feedback-in-llm-monitoring.md) - Incorporating human feedback loops into LLM application monitoring and evaluation, managed as data with near real-time streaming
* [Human Labeling: Sessions and Schemas](/concepts/labeling-sessions-and-schemas.md) - Labeling sessions organize traces for domain expert review, while labeling schemas define the structured questions and response types (ratings, free text) for consistent label collection.
* [Hybrid Deploy Code Approach](/concepts/hybrid-deploy-code-approach.md) - A variation of the deploy code pattern where code is deployed to staging, the model is trained on full production data in staging, and then the trained model artifact is deployed to production, saving training costs in production.
* [Hybrid Reasoning Models](/concepts/hybrid-reasoning-models.md) - A class of AI models (e.g., Claude Opus 4.x, Gemini 2.5/3 Pro) that offer both near-instant responses and extended/deep thinking modes depending on task complexity, blending fast inference with deliberate reasoning.
* [Hybrid Spark and Ray Workload Resource Optimization](/concepts/hybrid-spark-and-ray-workload-resource-optimization.md) - Best practices for running concurrent Spark and Ray workloads on Databricks, including reducing Spark executor memory to avoid OOM errors and enabling Ray-on-Spark autoscaling to dynamically share cluster resources.
* [Hybrid Tracing Patterns](/concepts/hybrid-tracing-patterns.md) - Combining mlflow.start_span() context managers, @mlflow.trace function decorators, and automatic tracing (autolog) for comprehensive application observability.
* [Hyperopt](/concepts/hyperopt.md) - A deprecated Python library for distributed hyperparameter tuning, not included in Databricks Runtime ML after 16.4 LTS ML.
* [Hyperopt Conditional Hyperparameter Spaces](/concepts/hyperopt-conditional-hyperparameter-spaces.md) - Hyperopt supports conditional search spaces where certain hyperparameters are only explored for specific model flavors or algorithms, enabling more targeted optimization.
* [Hyperopt Deprecation and Alternatives](/concepts/hyperopt-deprecation-and-migration.md) - The open-source Hyperopt library is no longer maintained and is removed from Databricks Runtime ML after 16.4 LTS. Databricks recommends Optuna for single-node or RayTune for distributed tuning.
* [Hyperopt deprecation and migration](/concepts/hyperopt-deprecation-and-migration.md) - Hyperopt is no longer maintained and is excluded from Databricks Runtime ML after 16.4 LTS; Databricks recommends Optuna or RayTune as replacements.
* [Hyperopt Deprecation and Migration on Databricks](/concepts/hyperopt-deprecation-and-migration.md) - Hyperopt is deprecated in Databricks Runtime for ML 16.4 LTS and later; recommended replacements are Optuna for single-node optimization and RayTune for distributed hyperparameter tuning.
* [Hyperopt deprecation in Databricks Runtime ML](/concepts/hyperopt-deprecation-on-databricks.md) - Hyperopt is excluded from Databricks Runtime for Machine Learning starting from version 16.4 LTS ML, marking its phased removal from the Databricks ML ecosystem.
* [Hyperopt deprecation on Databricks](/concepts/hyperopt-deprecation-on-databricks.md) - Hyperopt is no longer maintained and is removed from Databricks Runtime for Machine Learning after version 16.4 LTS ML
* [Hyperopt Distributed Training Workflow](/concepts/hyperopt-distributed-training-workflow.md) - Guidance on using Hyperopt with distributed machine learning algorithms on Databricks, where trials run on the driver node so the algorithm itself can initiate distributed training.
* [Hyperopt fmin()](/concepts/hyperopt-fmin.md) - The core function in Hyperopt used to execute hyperparameter optimization runs by iteratively generating and evaluating trials.
* [Hyperopt fmin() Workflow](/concepts/hyperopt-fmin-workflow.md) - The standard Hyperopt optimization workflow: define an objective function, define a search space, select a search algorithm, and run fmin() to find optimal hyperparameters.
* [Hyperopt hp.choice() Index Return Behavior](/concepts/hyperopt-hpchoice-index-return-behavior.md) - When using hp.choice() in Hyperopt, the returned value is the index of the chosen option, not the option itself; hyperopt.space_eval() is needed to retrieve actual parameter values.
* [Hyperopt NaN Loss Handling](/concepts/hyperopt-nan-loss-handling.md) - A NaN loss from Hyperopt fmin() usually means the objective function returned NaN; it does not affect other trials and can be safely ignored or mitigated by adjusting the search space or objective function.
* [Hyperopt Search Algorithms](/concepts/hyperopt-search-algorithms.md) - Hyperopt provides two main search algorithms: TPE (Tree of Parzen Estimators) for adaptive Bayesian optimization and random search for non-adaptive sampling over the hyperparameter space.
* [Hyperopt SparkTrials MLflow integration](/concepts/hyperopt-sparktrials-mlflow-integration.md) - SparkTrials logs tuning results as nested MLflow runs, with fmin() calls as parent runs and individual trials as child runs.
* [Hyperopt Tree of Parzen Estimators (TPE) Algorithm](/concepts/hyperopt-tree-of-parzen-estimators-tpe-algorithm.md) - Bayesian hyperparameter optimization algorithm used by Hyperopt that is more efficient than grid search or random search.
* [Hyperopt with HorovodRunner on Databricks](/concepts/hyperopt-with-horovodrunner-on-databricks.md) - Integration of Hyperopt with HorovodRunner to tune distributed deep learning workloads on Databricks, leveraging Spark's barrier mode for stability.
* [Hyperopt with MLlib Distributed Training](/concepts/hyperopt-with-mllib-distributed-training.md) - Example pattern for using Hyperopt to tune hyperparameters of Apache Spark MLlib's distributed training algorithms using the Trials class.
* [Hyperopt-MLflow Integration](/concepts/hyperopt-mlflow-integration.md) - Automated MLflow tracking of Hyperopt tuning trials on Databricks, enabled by calling mlflow.start_run() before fmin() to log all trials as MLflow runs.
* [Hyperparameter Tuning](/concepts/hyperparameter-tuning.md) - The process of optimizing model hyperparameters using automated libraries like Optuna, Ray Tune, and Hyperopt to efficiently find optimal settings for ML models.
* [Hyperparameter tuning for model selection](/concepts/hyperparameter-tuning-for-model-selection.md) - The technique of using hyperparameter optimization frameworks like Hyperopt to compare different model types (e.g., different algorithm families) by tuning each with its own appropriate hyperparameter space to find the best overall model.
* [Hyperparameter Tuning of XGBoost using MLlib Cross-Validation](/concepts/hyperparameter-tuning-of-xgboost-using-mllib-cross-validation.md) - Using Spark MLlib's cross-validation framework to tune XGBoost model hyperparameters at scale
* [Hyperparameter tuning on Databricks](/concepts/hyperparameter-tuning-on-databricks.md) - Integration of Optuna hyperparameter optimization within the Databricks platform for scalable machine learning
* [IAM role for Unity Catalog storage](/concepts/iam-role-for-unity-catalog-storage.md) - An AWS IAM role that grants Unity Catalog access to an S3 bucket used for metastore-level managed storage, requiring a cross-account trust relationship with Databricks.
* [IAM Role Trust for Unity Catalog](/concepts/iam-role-trust-for-unity-catalog.md) - The cross-account IAM role and trust relationship that must be established in AWS to allow Unity Catalog to access an S3 bucket for managed storage.
* [IAM Trust Relationship for Serverless Compute](/concepts/trust-relationship-for-serverless-compute.md) - Requirement that the instance profile's IAM role must have a trust relationship configured for Databricks serverless compute, or the endpoint creation fails.
* [Iceberg Conversion Limitations](/concepts/iceberg-conversion-limitations.md) - Known restrictions when converting Apache Iceberg tables to Delta Lake, including unsupported features like partition evolution and metastore tables.
* [Iceberg data type restrictions in Delta tables](/concepts/iceberg-data-type-restrictions-in-delta-tables.md) - The constraint that IcebergWriterCompatV rejects Delta schemas containing data types not supported by Apache Iceberg
* [Iceberg Metadata Versioning in Delta Uniform](/concepts/iceberg-metadata-versioning-in-delta-uniform.md) - The requirement that Iceberg metadata locations used to refresh a Delta Uniform table must have monotonically increasing version numbers, and UUID consistency must be maintained.
* [Iceberg REST Catalog (IRC) Protocol](/concepts/iceberg-rest-catalog-irc-protocol.md) - The open protocol that allows clients like Spark and Snowflake to read Iceberg tables from a catalog endpoint, used here to access shared data from Databricks OpenSharing.
* [Iceberg REST Catalog Integration for OpenSharing](/concepts/iceberg-rest-catalog-integration-for-opensharing.md) - A method to access OpenSharing data using external Iceberg clients (Snowflake, Trino, Flink, Spark) via the Apache Iceberg REST Catalog API, using vended credentials.
* [Iceberg Table Conversion Limitations](/concepts/iceberg-conversion-limitations.md) - Restrictions and unsupported scenarios when converting Iceberg tables, including partition evolution, metastore tables, and truncated column type limitations.
* [Iceberg to Delta Conversion](/concepts/iceberg-to-delta-conversion.md) - Converting Apache Iceberg tables whose underlying file format is Parquet to Delta Lake, using the Iceberg table's native file manifest, schema and partitioning information.
* [Iceberg to Delta conversion limitations](/concepts/iceberg-conversion-limitations.md) - Technical constraints when converting Iceberg tables including partition evolution and truncated column type restrictions
* [Iceberg to Delta Lake Conversion](/concepts/iceberg-to-delta-lake-conversion.md) - Conversion of Apache Iceberg tables to Delta Lake using CONVERT TO DELTA, with specific version and feature limitations.
* [Iceberg to Delta Lake Conversion Limitations](/concepts/iceberg-conversion-limitations.md) - Constraints and unsupported scenarios when converting Apache Iceberg tables to Delta Lake, including partition evolution, metastore tables, and truncated column types.
* [Iceberg unsupported data types and table properties](/concepts/iceberg-unsupported-data-types-and-table-properties.md) - IcebergWriterCompatV restricts which data types and Iceberg table properties are allowed in the Delta table schema and configuration
* [ICEBERG_MISSING_PARTITION_SPECS](/concepts/iceberg_missing_partition_specs.md) - A subclass of DELTA_CLONE_INCOMPATIBLE_SOURCE error indicating the source Apache Iceberg table lacks partition specifications, making it incompatible for cloning.
* [ICEBERG_MISSING_PARTITION_SPECS (Delta Clone sub-error)](/concepts/iceberg_missing_partition_specs.md) - A specific DELTA_CLONE_INCOMPATIBLE_SOURCE sub-error indicating the source Iceberg table lacks partition specs.
* [ICEBERG_MISSING_PARTITION_SPECS error condition](/concepts/iceberg_missing_partition_specs.md) - A sub-error of DELTA_CLONE_INCOMPATIBLE_SOURCE raised when cloning an Apache Iceberg table that has no partition specs defined.
* [ICEBERG_MISSING_PARTITION_SPECS sub-error](/concepts/iceberg_missing_partition_specs.md) - A specific error under DELTA_CLONE_INCOMPATIBLE_SOURCE indicating the source Apache Iceberg table has no partition specs, making it incompatible for Delta cloning.
* [ICEBERG_UNDERGONE_PARTITION_EVOLUTION](/concepts/iceberg_undergone_partition_evolution.md) - A subclass of DELTA_CLONE_INCOMPATIBLE_SOURCE error indicating the source Apache Iceberg table has undergone partition evolution, making it incompatible for cloning.
* [ICEBERG_UNDERGONE_PARTITION_EVOLUTION (Delta Clone sub-error)](/concepts/iceberg_undergone_partition_evolution.md) - A specific DELTA_CLONE_INCOMPATIBLE_SOURCE sub-error indicating the source Iceberg table has undergone partition evolution, which is incompatible with cloning.
* [ICEBERG_UNDERGONE_PARTITION_EVOLUTION error condition](/concepts/iceberg_undergone_partition_evolution.md) - A sub-error of DELTA_CLONE_INCOMPATIBLE_SOURCE raised when cloning an Apache Iceberg table that has undergone partition evolution (changed partitioning over time).
* [ICEBERG_UNDERGONE_PARTITION_EVOLUTION sub-error](/concepts/iceberg_undergone_partition_evolution.md) - A specific error under DELTA_CLONE_INCOMPATIBLE_SOURCE indicating the source Apache Iceberg table has undergone partition evolution, which is unsupported for Delta cloning.
* [Iceberg-Delta schema and type compatibility constraints](/concepts/iceberg-delta-schema-and-type-compatibility-constraints.md) - IcebergCompatV imposes restrictions on schema changes, including unsupported data types for columns and partition columns, incompatibility with type widening, and a requirement that partition specs cannot be renamed during REPLACE TABLE operations.
* [IcebergCompat Version Mutual Exclusivity](/concepts/icebergcompat-version-mutual-exclusivity.md) - Only one version of IcebergCompat can be enabled on a Delta table at a time; other versions must be explicitly disabled before enabling a new one.
* [IcebergCompatV](/concepts/icebergcompatv.md) - A Delta Lake table setting in Databricks that enables compatibility with Apache Iceberg readers, subject to version-specific validation rules.
* [IcebergCompatV (Iceberg Compatibility Version)](/concepts/icebergcompatv.md) - A Delta Lake table setting that enforces compatibility with Apache Iceberg format, with version constraints between 1 and a maximum supported version.
* [IcebergCompatV managed-table requirement](/concepts/delta-uniform-managed-table-requirement.md) - The IcebergCompatV feature can only be enabled on managed tables in Databricks, not on external/unmanaged tables.
* [IcebergCompatV versioning](/concepts/icebergcompatv-versioning.md) - Delta Lake table property controlling Uniform Iceberg compatibility level (versions 1 to maxVersion), where only one version can be enabled at a time and upgrading requires REORG TABLE.
* [IcebergCompatV versioning in Delta Lake](/concepts/icebergcompatv-versioning.md) - A versioned compatibility feature in Databricks Delta Lake that enforces Apache Iceberg compatibility rules; only one version can be enabled at a time and versions range from 1 to a configurable maxVersion.
* [IcebergCompatV1](/concepts/icebergcompatv.md) - A Delta Lake table compatibility feature that enforces Apache Iceberg compatibility requirements
* [IcebergCompatV1 Incompatible Table Features](/concepts/icebergcompatv1-incompatible-table-features.md) - Some Delta table features cannot coexist with IcebergCompatV1 and will cause a validation failure if present.
* [IcebergCompatV1 Partition Spec Constraint](/concepts/icebergcompatv1-partition-spec-constraint.md) - IcebergCompatV1 does not support replacing a partitioned table with a differently-named partition spec, mirroring a limitation in Iceberg-Spark 1.1.0.
* [IcebergCompatV1 Required Table Features](/concepts/icebergcompatv1-required-table-features.md) - Certain Delta table features must be enabled and cannot be dropped while IcebergCompatV1 is active; disabling the compatibility mode is required first.
* [IcebergCompatV1 Required Table Properties](/concepts/icebergcompatv1-required-table-properties.md) - The requirement that certain table properties must be set to specific values for IcebergCompatV1 to be valid, and violations raise the WRONG_REQUIRED_TABLE_PROPERTY sub-error.
* [IcebergCompatV1 Schema Restrictions](/concepts/icebergcompatv1-schema-restrictions.md) - The constraint that IcebergCompatV1 does not support tables with MapType, ArrayType, or NullType columns in their schema, due to limitations in the Iceberg format.
* [IcebergCompatV1 Unsupported Data Types](/concepts/icebergcompatv1-unsupported-data-types.md) - Schemas containing MapType, ArrayType, or NullType are rejected under IcebergCompatV1 because Iceberg does not support these types.
* [IcebergCompatV2](/concepts/icebergcompatv2.md) - A Delta Lake write protocol feature that enables Iceberg read compatibility on Delta Lake tables, requiring column mapping and Databricks Runtime 14.3 LTS or above.
* [IcebergWriterCompatV](/concepts/icebergwritercompatv.md) - A versioned validation mechanism in Databricks Delta Lake that enforces compatibility constraints so Delta tables can be read by Apache Iceberg writers.
* [IcebergWriterCompatV versioning](/concepts/icebergwritercompatv.md) - A Databricks compatibility mechanism that enforces Delta table schemas and properties conform to Apache Iceberg writer expectations at a given version level
* [IDE Integration with Databricks Connect](/concepts/ide-integration-with-databricks-connect.md) - How to configure and use Databricks Connect with various development environments including JupyterLab, Classic Jupyter Notebook, Visual Studio Code, PyCharm, RStudio Desktop, IntelliJ, Eclipse, and PyDev.
* [Idempotency in data optimization](/concepts/idempotency-in-data-optimization.md) - The property that re-running an optimization on unchanged data produces no additional effect; bin-packing is idempotent while Z-ordering is not.
* [Idempotency of REORG TABLE](/concepts/idempotency-of-reorg-table.md) - The REORG TABLE command is idempotent, meaning running it twice on the same dataset has no effect on the second run.
* [Idempotent Data Loading](/concepts/idempotent-data-loading.md) - COPY INTO automatically skips files that have already been loaded, even if they were modified, ensuring safe retry and avoiding duplicate data.
* [Idempotent Data Loading (COPY INTO)](/concepts/idempotent-data-loading.md) - The property of COPY INTO that skips already-loaded files, making it safe to retry without duplicate data even if source files have been modified.
* [Idempotent Data Migration Pattern for UC Traces](/concepts/idempotent-data-migration-pattern-for-uc-traces.md) - Design pattern for migrating trace data between UC table formats where the migration is idempotent: already-migrated rows are skipped on rerun, making it resilient to partial failures like cluster timeouts
* [Idempotent file ingestion](/concepts/idempotent-file-ingestion.md) - The property that files already loaded via COPY INTO are skipped on subsequent runs, even if modified, enabling safe retries.
* [Idempotent foreachBatch Writes](/concepts/idempotent-foreachbatch-writes.md) - Pattern using txnAppId and txnVersion to make writes to multiple sinks within foreachBatch idempotent, preventing duplicate writes on failure recovery
* [Idempotent Job Submission](/concepts/idempotent-job-submission.md) - Pattern using --idempotency-key to make job submission safely retryable, returning the existing run if the same key was used before.
* [Idempotent Streaming Writes with foreachBatch](/concepts/idempotent-streaming-writes-with-foreachbatch.md) - Using txnAppId and txnVersion to make foreachBatch writes idempotent when writing to Delta Lake from streams
* [Idempotent Submission Pattern](/concepts/idempotent-submission-pattern.md) - Using the --idempotency-key flag with air run to safely retry submissions, returning the existing run if the same key was already used instead of creating a duplicate.
* [Idempotent Trace Migration](/concepts/idempotent-trace-migration.md) - A migration mechanism that automatically resumes from where it left off if interrupted, skipping already-migrated rows so the same command can be safely rerun.
* [Idempotent Writes with foreachBatch and Transaction Identifiers](/concepts/idempotent-writes-with-foreachbatch-and-transaction-identifiers.md) - Using txnAppId and txnVersion options with foreachBatch to make multi-table writes idempotent, allowing safe retry after failures without duplicating data.
* [Idempotent writes with foreachBatch using txnAppId and txnVersion](/concepts/idempotent-writes-with-foreachbatch-using-txnappid-and-txnversion.md) - Pattern for idempotent multi-sink writes within foreachBatch using Delta Lake's transaction app ID and version to identify and ignore duplicate writes after failures.
* [Identity Federation for Unity Catalog](/concepts/identity-federation-for-unity-catalog.md) - The centralization of user management at the Databricks account level, required as a prerequisite for using UCX and enabling Unity Catalog adoption.
* [Identity Resolution for Allowlisted Artifacts](/concepts/identity-resolution-for-allowlisted-artifacts.md) - The rule by which the identity used for allowlisted libraries and init scripts is determined — the library installer's identity in standard mode and the assigned principal's identity in dedicated access mode.
* [Idle Endpoint Warm-up Delay](/concepts/idle-endpoint-warm-up-delay.md) - Timeout risk when a model serving endpoint scaled to 0 receives a request and must warm up before serving.
* [Idle Endpoint Warm-Up Timeout](/concepts/idle-endpoint-warm-up-timeout.md) - Timeouts that can occur when a scaled-to-zero serving endpoint receives a request and must warm up, potentially exceeding the client-side timeout.
* [Idle Endpoint Warm-Up Timeouts](/concepts/idle-endpoint-warm-up-timeout.md) - Timeouts that can occur when a Model Serving endpoint that has scaled to 0 receives a request and takes too long to warm up.
* [ignoreDeletes and ignoreChanges Legacy Options](/concepts/ignoredeletes-and-ignorechanges-legacy-options.md) - Legacy streaming source options for Delta Lake that handle delete/change operations differently than skipChangeCommits, with specific behaviors for file re-emission and duplicate handling.
* [Image classification with CNN on Databricks](/concepts/image-classification-with-cnn-on-databricks.md) - A tutorial notebook demonstrating how to train a 2-D convolutional neural network for image classification on serverless GPUs using Databricks AI Runtime.
* [Image Classification with CNN on Serverless GPU](/concepts/image-classification-with-cnn-on-databricks.md) - Training a 2-D convolutional neural network for image classification using serverless GPUs in Databricks.
* [Image Classification with CNNs on Databricks](/concepts/image-classification-with-cnn-on-databricks.md) - A tutorial demonstrating how to train a 2-D convolutional neural network for image classification on serverless GPUs using Databricks AI Runtime.
* [Image ETL Pipeline Pattern](/concepts/image-etl-pipeline-pattern.md) - A two-stage reference pipeline for image applications: first ETL images into a Delta table using Auto Loader, then perform distributed inference using pandas UDFs with DL models.
* [Image Input Safety Restrictions for AI Models](/concepts/image-input-safety-restrictions-for-ai-models.md) - A mitigation requirement that image inputs to AI models must be restricted to low risk and topical images to prevent misuse.
* [Image-to-Token Conversion for Foundation Models](/concepts/image-to-token-conversion-for-foundation-models.md) - The mechanism by which each image in a request to a Databricks-hosted foundation model is converted to token usage for billing and pricing purposes.
* [Imbalanced Dataset Handling](/concepts/imbalanced-dataset-handling.md) - AutoML's strategy for reducing class imbalance by downsampling majority classes and applying inverse class weights during training.
* [Imbalanced Dataset Handling in AutoML](/concepts/imbalanced-dataset-handling.md) - Techniques AutoML uses to address class imbalance in classification datasets, including downsampling majority classes and applying class weights.
* [Importing Databricks Dashboards from JSON](/concepts/importing-databricks-dashboards-from-json.md) - The ability to import Databricks dashboards from a JSON file definition, enabling sharing and reuse of dashboard templates like the MLflow system tables dashboard.
* [Importing MLflow Dashboards from JSON](/concepts/importing-mlflow-dashboards-from-json.md) - The workflow of downloading a pre-built dashboard JSON file and importing it into a Databricks workspace to quickly visualize MLflow system table data.
* [ImportNotebook](/concepts/importnotebook.md) - The databricks.automl.import_notebook method imports a notebook saved as an MLflow artifact.
* [ImportNotebookResult](/concepts/importnotebookresult.md) - Result object returned by import_notebook containing the path and URL of the imported notebook.
* [Imputation of Missing Values in AutoML](/concepts/imputation-of-missing-values-in-automl.md) - How AutoML handles null value imputation, including default behavior and custom imputation methods configurable via UI or API.
* [In-memory DataFrame creation and table persistence](/concepts/in-memory-dataframe-creation-and-table-persistence.md) - The workflow of creating a Spark DataFrame from in-memory Scala data, defining a schema with StructType, saving it as a managed table, and querying it on the remote cluster.
* [In-memory Spark to Ray Data Transfer](/concepts/in-memory-spark-to-ray-data-transfer.md) - Using ray.data.from_spark() to efficiently transfer Spark DataFrames to Ray datasets without writing to disk, available on Databricks Runtime ML 15.0+.
* [In-memory Spark-to-Ray Data Transfer on Databricks](/concepts/in-memory-spark-to-ray-data-transfer.md) - Using ray.data.from_spark() to transfer Spark DataFrames to Ray datasets in-memory without writing to disk, available on Databricks Runtime ML 15.0+
* [In-place Table Conversion](/concepts/in-place-table-conversion.md) - The approach of converting Parquet or Iceberg tables to Delta Lake format without copying or rewriting data files, only generating a new transaction log.
* [In-place Table Format Conversion](/concepts/in-place-table-conversion.md) - The process of converting a table from one format (Parquet or Iceberg) to Delta Lake without copying or rewriting the underlying data files, only generating transaction metadata.
* [In-Product Access Request Surfaces](/concepts/in-product-access-request-surfaces.md) - Multiple Databricks surfaces (Catalog Explorer, SQL editor, notebooks, AI/BI dashboards, Genie Spaces) where users can request privileges on Unity Catalog objects.
* [Incident Management (Data Quality)](/concepts/incident-management-data-quality.md) - Workflow for handling detected data quality incidents including assigning ownership, marking false positives, and reviewing recently auto-resolved incidents.
* [Incident Management for Data Quality](/concepts/incident-management-for-data-quality.md) - Workflow for reviewing and managing unhealthy table incidents, including assigning ownership to indicate active investigation and marking incidents as false positives (not an issue), with assignments and dismissals persisting for 7 days.
* [INCOMPATIBLE_TABLE_FEATURE](/concepts/incompatible_table_feature.md) - Error sub-type indicating a table feature is incompatible with IcebergCompatV1
* [INCOMPATIBLE_TABLE_FEATURE (Delta Iceberg error)](/concepts/incompatible_table_feature.md) - A sub-error of DELTA_ICEBERG_COMPAT_V1_VIOLATION that occurs when a table has a feature incompatible with IcebergCompatV1
* [Incremental Clone of Parquet and Iceberg to Delta Lake](/concepts/incremental-clone-of-parquet-and-iceberg-to-delta-lake.md) - Alternative approach to incrementally clone Parquet or Iceberg tables to Delta Lake instead of one-time conversion.
* [Incremental Cloning of Parquet and Iceberg to Delta Lake](/concepts/incremental-clone-of-parquet-and-iceberg-to-delta-lake.md) - An alternative approach to CONVERT TO DELTA for incremental, ongoing conversion of Parquet or Iceberg tables to Delta Lake tables.
* [Incremental Cloning to Delta Lake](/concepts/incremental-cloning-to-delta-lake.md) - An alternative to one-time CONVERT TO DELTA for incrementally cloning Parquet and Apache Iceberg tables to Delta Lake.
* [Incremental Cloning vs One-Time Conversion](/concepts/incremental-cloning-vs-one-time-conversion.md) - Distinction between the one-time CONVERT TO DELTA command and incremental cloning for Parquet and Iceberg tables to Delta Lake
* [Incremental Intelligent Scanning](/concepts/incremental-intelligent-scanning.md) - The classification engine intelligently determines when to scan data by leveraging Unity Catalog metadata, performing incremental scans to classify new data without manual configuration, typically scanning new tables and columns within 24 hours.
* [Incremental Model Update Versioning](/concepts/incremental-model-update-versioning.md) - Databricks' approach to shipping model optimizations where endpoint URLs remain stable but the model ID in responses includes a date stamp reflecting the update.
* [Incremental Model Updates (Databricks)](/concepts/incremental-model-update-versioning.md) - Databricks practice of shipping minor model optimizations while keeping the endpoint URL unchanged, with the model ID in the response updated to reflect the date (e.g., 'model-name-030424').
* [Incremental Model Updates and Version History](/concepts/incremental-model-updates-and-version-history.md) - Databricks ships minor model updates with versioned model IDs appended with dates, while keeping endpoint URLs unchanged.
* [Incremental Symlink Manifest Generation](/concepts/incremental-symlink-manifest-generation.md) - A Delta Lake feature for generating symlink manifests incrementally, which is mutually exclusive with persistent deletion vectors.
* [Inference Deployment Strategies on Databricks](/concepts/inference-deployment-strategies-on-databricks.md) - Options for deploying deep learning models for online serving via Model Serving (custom, foundation, external models), batch inference with Spark Pandas UDFs, and streaming inference
* [Inference Monitoring](/concepts/inference-monitoring.md) - Tracking the performance of GenAI applications, machine learning models, and model-serving endpoints by monitoring inference tables that contain model inputs and predictions.
* [Inference Profile](/concepts/inference-profile.md) - A data profile type for monitoring ML model predictions, requiring problem type (classification/regression), prediction column, and model ID column.
* [Inference Profiling](/concepts/inference-profile.md) - A data profiling type for monitoring machine learning model predictions, requiring prediction column, model ID, and optionally label and problem type.
* [Inference Table Augmented DataFrame Logging](/concepts/inference-table-augmented-dataframe-logging.md) - Model serving endpoints created from February 2025 onwards can log the augmented DataFrame (containing looked-up features and function return values) to inference tables for monitoring.
* [Inference Table Logging for Feature Serving](/concepts/inference-tables-for-feature-logging.md) - The ability to log the augmented DataFrame (looked-up feature values and function results) to an inference table for monitoring and debugging, available for endpoints created starting February 2025.
* [Inference Table Monitoring](/concepts/inference-table-monitoring.md) - A use case of data profiling that tracks performance of GenAI apps, ML models, and model-serving endpoints by monitoring inference tables containing model inputs and predictions.
* [Inference Table Profiling](/concepts/inference-table-profiling.md) - A data profiling mode for tracking ML model performance over time by profiling inference tables that contain model inputs and predictions, supporting model version comparison.
* [Inference Table Profiling for ML Models](/concepts/inference-table-profiling.md) - Profiling inference tables that contain ML model inputs and predictions to track model performance over time, compare model versions, and detect data drift relative to training/validation data.
* [Inference Tables](/concepts/inference-tables.md) - A Databricks AI Gateway feature that automatically captures incoming requests and outgoing responses from model serving endpoints for logging and audit.
* [Inference Tables for Feature Logging](/concepts/inference-tables-for-feature-logging.md) - A capability to log the augmented DataFrame containing looked-up feature values and function return values to an inference table for monitoring and debugging, available for endpoints created starting February 2025.
* [Inference Tables for Request Root Cause Analysis](/concepts/inference-tables-for-root-cause-analysis.md) - Using inference tables in Unity Catalog to log, query, and analyze model serving endpoint requests and responses for debugging failures.
* [Inference Tables for Root Cause Analysis](/concepts/inference-tables-for-root-cause-analysis.md) - Using Unity Catalog inference tables to automatically log all requests and responses to model serving endpoints for query-based debugging of failed requests.
* [InferenceLog Analysis](/concepts/inferencelog-analysis.md) - A profile analysis type for ML model monitoring that computes model accuracy, fairness, bias statistics, and per-model-id metrics using label and prediction columns.
* [InferenceLog Analysis for Model Retraining](/concepts/inferencelog-analysis-for-model-retraining.md) - Using drift metrics detected via profile alerts on inference logs to determine when a machine learning model should be retrained.
* [InferenceLog Analysis Metrics](/concepts/inferencelog-analysis-metrics.md) - Additional statistics for model monitoring including model quality metrics (when label_col and prediction_col are provided), fairness and bias statistics for classification models, and automatic slicing by model_id.
* [InferenceLog Model Accuracy and Fairness Statistics](/concepts/inferencelog-model-accuracy-and-fairness-statistics.md) - Additional statistics computed for InferenceLog analysis including model quality metrics (when label_col and prediction_col are provided), automatic slicing by model_id, and fairness/bias statistics for classification models with Boolean slices.
* [InferenceLog Profile](/concepts/inferencelog-profile.md) - A data profile type for ML model monitoring that includes model quality metrics alongside time-series distribution analysis
* [INFORMATION_SCHEMA Grant Visibility Limitation](/concepts/information_schema-grant-visibility-limitation.md) - Users with the MANAGE privilege on an object cannot view all grants for that object via INFORMATION_SCHEMA — only their own grants are shown. This is a known behavior gap slated for correction.
* [Init Script Allowlisting](/concepts/init-script-allowlisting.md) - The process of adding cluster init scripts to the Unity Catalog allowlist from Volumes or object storage, where the identity of the cluster owner is used for permissions in standard access mode.
* [Init Script and JAR Allowlisting](/concepts/init-script-and-jar-allowlisting.md) - The process of adding init scripts or JAR files to the Unity Catalog allowlist via Catalog Explorer or REST API, supporting sources from Unity Catalog volumes or object storage.
* [Initial Snapshot Offset Restriction](/concepts/initial-snapshot-offset-restriction.md) - The constraint that offsets used in Delta skip offset ranges cannot be initial snapshot offsets (isInitialSnapshot must be false)
* [Initial snapshot processing with withEventTimeOrder](/concepts/initial-snapshot-processing-with-witheventtimeorder.md) - Feature to prevent data drops during initial Delta snapshot processing in stateful streaming queries by ordering data by event time into time-bucketed micro-batches.
* [Initial Table Version for Delta Streaming](/concepts/initial-table-version-for-delta-streaming.md) - Setting startingVersion or startingTimestamp to control where a Delta Lake streaming source begins reading
* [INITIAL_SNAPSHOT](/concepts/initial_snapshot.md) - A sub-condition of the DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE error stating that skip offsets must not be initial snapshot offsets
* [INITIAL_SNAPSHOT sub-condition](/concepts/initial_snapshot.md) - A specific reason for the DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE error, indicating that offsets in the skip range must not be initial snapshot offsets (isInitialSnapshot must be false).
* [INITIAL_SNAPSHOT Subcondition](/concepts/initial_snapshot.md) - A sub-error of DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE indicating skip offset range is invalid because the offsets are initial snapshot offsets (isInitialSnapshot must be false).
* [Inline Dependency Pinning for Ray Workloads](/concepts/inline-dependency-pinning-for-ray-workloads.md) - Declaring Python dependencies directly in the workload YAML environment section rather than in a separate requirements file, including pinning transitive dependencies like fsspec that conflict with newer library versions.
* [Inline Ray Cluster Bootstrap Pattern](/concepts/inline-ray-cluster-bootstrap-pattern.md) - A pattern for starting a Ray cluster directly within a workload command by running ray start on the head node and having worker nodes join the cluster, enabling distributed training without a separate cluster launcher.
* [Input Rate Limiting and Backlog Metrics](/concepts/input-rate-limiting-and-backlog-metrics.md) - Options (maxFilesPerTrigger, maxBytesPerTrigger) and metrics (numBytesOutstanding, numFilesOutstanding) to control and monitor streaming query processing rate from Delta sources.
* [Input Types for Custom Labeling Schemas](/concepts/input-types-for-custom-schemas.md) - Six supported input types for collecting feedback in custom schemas: InputCategorical, InputCategoricalList, InputText, InputTextList, and InputNumeric — each with configurable constraints.
* [Input types for custom schemas](/concepts/input-types-for-custom-schemas.md) - The supported input types for collecting feedback in custom labeling schemas: InputCategorical, InputCategoricalList, InputText, InputTextList, and InputNumeric.
* [Input types for labeling schemas](/concepts/input-types-for-custom-schemas.md) - Supported input types for custom labeling schemas: InputCategorical, InputCategoricalList, InputText, InputTextList, and InputNumeric, each with specific constraints.
* [Input/Output and Expectation Judges](/concepts/inputoutput-and-expectation-judges.md) - Judges that evaluate agent responses by analyzing conversation inputs/outputs or comparing outputs against predefined expected behaviors
* [Input/Output Tracking](/concepts/inputoutput-tracking.md) - The practice of capturing inputs, outputs, and metadata for each intermediate step in a GenAI application pipeline (retrieval, tool calls, LLM interactions) to enable debugging and analysis.
* [Input/Output-based Evaluation](/concepts/inputoutput-based-evaluation.md) - Evaluation approach where judges assess agent performance by analyzing conversation history (inputs) and agent responses (outputs).
* [Instance Profile (Databricks on AWS)](/concepts/instance-profile-databricks-on-aws.md) - An AWS IAM instance profile attached to a Databricks model serving endpoint to grant access to AWS resources such as S3, allowing models to read/write data during inference.
* [Instance Profile Attachment to Model Serving Endpoints](/concepts/instance-profile-for-model-serving-endpoints.md) - The process and configuration of attaching an AWS IAM instance profile to a Databricks model serving endpoint to grant access to AWS resources.
* [Instance Profile Authentication for Databricks Feature Store](/concepts/instance-profile-authentication-for-feature-store.md) - Using AWS instance profiles attached to Databricks clusters or serving endpoints to authenticate feature store operations with third-party online stores.
* [Instance Profile Authentication for Databricks Online Stores](/concepts/instance-profile-authentication-for-feature-store.md) - Using AWS IAM instance profiles attached to Databricks clusters or serving endpoints to provide read and write authentication to third-party online stores like DynamoDB.
* [Instance Profile Authentication for DynamoDB Online Stores](/concepts/instance-profile-authentication-for-feature-store.md) - Using AWS instance profiles attached to Databricks clusters (for write) or serving endpoints (for read) to authenticate with DynamoDB online stores without explicit secret credentials.
* [Instance Profile Authentication for Feature Store](/concepts/instance-profile-authentication-for-feature-store.md) - Using AWS instance profiles attached to Databricks clusters (for write) or serving endpoints (for read) to authenticate with DynamoDB online stores without explicit credentials.
* [Instance Profile for Databricks Model Serving](/concepts/instance-profile-databricks-on-aws.md) - An AWS IAM instance profile attached to a model serving endpoint to grant access to AWS resources (e.g., S3, feature stores) from a deployed model.
* [Instance Profile for Model Serving Endpoints](/concepts/instance-profile-for-model-serving-endpoints.md) - Attaching AWS IAM instance profiles to Databricks model serving endpoints to grant access to AWS resources such as S3 buckets and feature stores.
* [Instance Profile Lifecycle and Endpoint Behavior](/concepts/instance-profile-lifecycle-and-endpoint-behavior.md) - Modifying or deleting an instance profile in Databricks settings does not affect running endpoints until they are updated or restarted.
* [Instance Profile Lifecycle and Running Endpoints](/concepts/instance-profile-lifecycle-and-endpoint-behavior.md) - Running endpoints continue to use the originally attached instance profile IAM role even if the profile is edited or deleted from the Databricks UI; changes only take effect upon endpoint update.
* [Instance Profile Staleness on Endpoint Update](/concepts/instance-profile-staleness-on-endpoint-update.md) - When an instance profile's IAM role is edited or deleted in Databricks Settings, running serving endpoints continue using the old role until the endpoint is explicitly updated.
* [Intelligent Incremental Scanning](/concepts/intelligent-incremental-scanning.md) - The classification engine uses an agentic AI system with the Data Intelligence Engine to incrementally scan only new or changed data, optimizing cost and avoiding manual configuration.
* [Intelligent Scanning](/concepts/intelligent-scanning.md) - An automated scanning mechanism that prioritizes high-impact tables based on popularity and downstream usage while reducing scan frequency for less critical tables.
* [Intelligent Scanning (Databricks)](/concepts/intelligent-scanning.md) - An automated scan frequency system that prioritizes high-impact tables based on popularity and downstream usage, while reducing scan frequency for less critical tables
* [Intelligent Schema Scanning](/concepts/intelligent-schema-scanning.md) - An approach used by Databricks anomaly detection that prioritizes important tables and skips low-impact ones when monitoring all tables in a schema.
* [IntelliJ IDEA Databricks Integration](/concepts/intellij-idea-databricks-integration.md) - Using Databricks Connect for Scala from the IntelliJ IDEA IDE to run Spark code on Databricks compute.
* [Interactive Data Applications with Databricks Connect](/concepts/interactive-data-applications-with-databricks-connect.md) - The ability to embed Databricks Connect in applications (like JDBC but with full PySpark expressiveness) to build interactive data apps that leverage Databricks serverless compute.
* [Interactive Development with Databricks Connect](/concepts/interactive-development-with-databricks-connect.md) - The ability to develop and debug Spark code from any IDE (VS Code, PyCharm, IntelliJ) with code running remotely on Databricks compute while supporting local debugging.
* [Interactive IDE Development with Databricks Connect](/concepts/interactive-development-with-databricks-connect.md) - A workflow enabling developers to write Spark code in IDEs like VS Code, PyCharm, and IntelliJ IDEA while running and debugging that code on remote Databricks compute clusters.
* [Intermediate Step Instrumentation](/concepts/intermediate-step-instrumentation.md) - Capturing inputs, outputs, and metadata for each intermediate step in a GenAI pipeline (e.g., retrieval, tool calls, LLM interactions) along with user feedback and quality evaluation results.
* [Intermediate Step Trace Capture](/concepts/intermediate-step-trace-capture.md) - The practice of examining inputs, outputs, and metadata for each intermediate step in a GenAI pipeline (retrieval, tool calls, LLM interactions) for debugging and analysis.
* [Internal vs External Identity Provider Models](/concepts/internal-vs-external-identity-provider-models.md) - Two deployment modes for OIDC federation: provider-managed (internal IdP used when sharing within the same organization) and recipient-managed (external IdP where the recipient controls access policies and user identities).
* [Internal vs External Identity Provider Patterns](/concepts/internal-vs-external-identity-provider-models.md) - Two deployment patterns for OIDC federation: provider-managed IdP for intra-organization sharing and recipient-managed IdP for cross-organization sharing, each with different security and control characteristics.
* [Internal vs External Identity Provider Scenarios](/concepts/internal-vs-external-identity-provider-scenarios.md) - Two OIDC sharing models: provider-managed (internal IdP for sharing within large organizations) and recipient-managed (external IdP where recipient controls access).
* [Internal vs External Identity Providers](/concepts/internal-vs-external-identity-providers.md) - Two sharing scenarios: provider-managed (internal IdP for sharing within the same organization) and recipient-managed (external IdP where the recipient controls access policies and MFA).
* [Invalid Partition Value Detection](/concepts/invalid-partition-value-detection.md) - The ability of FSCK REPAIR TABLE to detect data files with partition column values that violate NOT NULL constraints, such as NULL values in non-nullable partition columns.
* [Invalid partition values detection](/concepts/invalid-partition-values-detection.md) - Detection of data files with partition values that violate the table's NOT NULL partition column constraints
* [INVALID_INDEX](/concepts/invalid_index.md) - A sub-condition of the DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE error stating that offset indices must match the expected base index
* [INVALID_INDEX sub-condition](/concepts/invalid_index.md) - A specific reason for the DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE error, indicating that the offset index must match the expected BASE_INDEX value.
* [INVALID_INDEX Subcondition](/concepts/invalid_index.md) - A sub-error of DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE indicating the offset index must be BASE_INDEX (<baseIndex>).
* [IP Access List Constraints](/concepts/ip-access-list-constraints.md) - Technical limitations of recipient IP access lists: maximum 100 IP/CIDR values per recipient, only IPv4 addresses supported, only allow lists (no deny lists) available.
* [IP Access Lists for OpenSharing Recipients](/concepts/ip-access-lists-for-opensharing-recipients.md) - A security feature that allows providers to restrict Databricks-to-Open sharing recipients to a defined set of IP addresses, limiting network-level access to shared data.
* [Irreversibility of Unity Catalog Enablement](/concepts/irreversibility-of-unity-catalog-enablement.md) - Once a workspace is enabled for Unity Catalog, the enablement cannot be reversed and identity management permanently shifts to account-level interfaces.
* [Iterative GenAI Development Lifecycle](/concepts/iterative-genai-development-lifecycle.md) - The continuous optimization workflow of developing, testing, monitoring, and collecting human feedback for GenAI applications.
* [Iterative Prompt Improvement Workflow](/concepts/iterative-prompt-improvement-workflow.md) - A workflow pattern for GenAI development: evaluate a prompt, review results, revise the prompt based on evaluation criteria, re-run evaluation, and compare runs in the MLflow UI.
* [Iterative Scorer Development Workflow](/concepts/iterative-scorer-development-workflow.md) - A four-step development process for building code-based scorers: define evaluation data, generate traces, store traces, and iterate on scorers using stored traces.
* [Ivy Dependency Resolution for Databricks Connect UDFs](/concepts/ivy-dependency-resolution-for-databricks-connect-udfs.md) - Using spark.addArtifact() with ivy:// URIs to download third-party Maven dependencies at runtime for UDFs that depend on libraries not present on the Databricks cluster.
* [JAR Build Compatibility for Databricks Serverless](/concepts/jar-build-compatibility-for-databricks-serverless.md) - Specific version requirements (Scala 2.13, JDK 17, Databricks Connect 17.3 for environment version 4) that JARs must match exactly to run on serverless compute.
* [JAR Dependency Management on Serverless](/concepts/jar-dependency-management-on-serverless.md) - Three strategies to make libraries available to JARs on serverless: use provided libraries, attach as environment libraries, or use JDBC connections for external databases.
* [JAR Dependency Strategy for Serverless](/concepts/jar-dependency-strategy-for-serverless.md) - Three approaches to manage library dependencies for JARs on serverless compute: provided libraries, environment libraries, and JDBC connections
* [JAR Library Allowlisting](/concepts/jar-library-allowlisting.md) - The process of adding JAR artifacts to the Unity Catalog allowlist from Volumes or object storage, where the identity of the library installer is used for permissions in standard access mode.
* [JAR Task Configuration in Databricks Jobs](/concepts/jar-task-configuration-in-databricks-jobs.md) - Step-by-step process for creating a Databricks job that runs a JAR on serverless compute: setting main class, selecting compute type, configuring serverless environment version, attaching JAR file, and passing parameters.
* [JAR Task in Lakeflow Jobs](/concepts/jar-task-in-lakeflow-jobs.md) - A Databricks workflow task type that runs a compiled Java/Scala JAR on serverless or classic compute, configured with main class, parameters, and serverless environment.
* [Job Registry Webhook Payload by Job Type](/concepts/job-registry-webhook-payload-by-job-type.md) - Different payload formats for job-triggering webhooks based on job type: notebook/Python wheel jobs get a params dict, Python/JAR/Spark submit jobs get a params list, multi-task jobs get all parameter types populated.
* [Job Reliability Configuration](/concepts/job-reliability-configuration.md) - Settings for workload resilience including `max_retries` (number of automatic retries on failure) and `timeout_minutes` (wall-clock budget per attempt) in AI Runtime YAML configs.
* [Job Webhooks for Manual Model Deployment Approval](/concepts/job-webhooks-for-manual-model-deployment-approval.md) - Using job notifications to call out to external CI/CD systems for manual approval gates before deploying production models, then using aliases to promote the model version
* [Jobs API and Databricks Asset Bundles for AI Runtime](/concepts/jobs-api-and-databricks-asset-bundles-for-ai-runtime.md) - Programmatic creation and management of AI Runtime jobs using the Databricks Jobs API or Asset Bundles with YAML configuration
* [John Snow Labs Healthcare NLP](/concepts/john-snow-labs-healthcare-nlp.md) - A proprietary library (Spark NLP for Healthcare) for clinical and biomedical text mining, offering pre-trained models for entities, drugs, risk factors, anatomy, and demographics, available via Databricks Partner Connect.
* [Joint Multi-Prompt Optimization](/concepts/joint-multi-prompt-optimization.md) - The ability to simultaneously refine multiple prompts together for best overall performance, enabled by passing multiple prompt_uris to the optimize_prompts API.
* [JSON Schema Limitations for Function Calling](/concepts/json-schema-limitations-for-function-calling.md) - Foundation Model APIs support only a subset of JSON schema for function definitions, excluding pattern, complex composition keywords, and limiting to 16 keys maximum.
* [JSON Schema Subset for Function Calling](/concepts/json-schema-subset-for-function-calling.md) - Databricks Foundation Model APIs support a simplified subset of JSON Schema for function definitions, excluding pattern, anyOf/oneOf/allOf, prefixItems, and $ref to improve generation quality.
* [JSON Schema Subset for Function Definitions](/concepts/json-schema-subset-for-function-definitions.md) - The restricted JSON schema specification that Databricks Foundation Model APIs accept for function call definitions, excluding pattern, anyOf, oneOf, allOf, $ref, and limiting keys to 16.
* [Judge Alignment](/concepts/judge-alignment.md) - A process that teaches LLM judges to match human evaluation standards through systematic feedback, improving agreement with human assessments by 30-50 percent.
* [Judge Alignment Three-Step Workflow](/concepts/judge-alignment-three-step-workflow.md) - The standard workflow for aligning judges: generate initial assessments, collect human feedback, then align and deploy using the judge's align() method.
* [Judge Alignment Validation](/concepts/alignment-validation.md) - The process of comparing original and aligned judge performance against human ground truth to measure improvement in accuracy after alignment.
* [Judge Alignment with Human Experts](/concepts/judge-alignment-with-human-experts.md) - A process where base LLM judges are refined by incorporating human expert feedback on application outputs to improve judge accuracy and alignment with human judgment.
* [Judge Instruction Best Practices](/concepts/judge-instruction-best-practices.md) - Guidelines for writing effective judge instructions including specificity about output formats (categorical, boolean, numeric) and breaking down complex evaluations.
* [Judge Model Customization](/concepts/judge-model-customization.md) - The ability to change the underlying LLM that powers a built-in judge by specifying a model in '<provider>:/<model-name>' format, supporting Databricks-hosted models and other LiteLLM-compatible providers.
* [Judge Prompt Template Variables](/concepts/judge-prompt-template-variables.md) - A restricted set of four template variables ({{ inputs }}, {{ outputs }}, {{ expectations }}, {{ trace }}) that can be used in custom judge prompts, with validation to prevent injection of arbitrary variables.
* [Judge Prompt Templates with Variables](/concepts/judge-prompt-templates-with-variables.md) - Custom judge instructions support templated variables such as `{{ inputs }}`, `{{ outputs }}`, `{{ expectations }}`, and `{{ trace }}` that dynamically inject evaluation context from the agent invocation.
* [Judge Registration and Deployment](/concepts/judge-registration-and-deployment.md) - The process of registering an aligned judge for production use with distinguishing metadata such as name, alignment date, and number of traces used.
* [Judge Validation](/concepts/judge-validation.md) - Methodology for comparing original and aligned judge performance against human ground truth to measure accuracy improvement.
* [KD00C SQLSTATE Class](/concepts/kd00c-sqlstate-class.md) - The SQLSTATE class 'KD' (Datasource-specific errors) that encompasses the DELTA_VERSIONS_NOT_CONTIGUOUS error condition in Databricks.
* [KD00E SQLSTATE](/concepts/kd00e-sqlstate.md) - A Databricks-specific SQLSTATE code (KD00E) under the KD class for datasource-specific errors, used by the DELTA_ICEBERG_WRITER_COMPAT_VIOLATION error condition.
* [Knowledge Assistant Limitation with Model Permissions](/concepts/knowledge-assistant-limitation-with-model-permissions.md) - A known limitation where Agent Bricks Knowledge Assistant does not support foundation model Unity Catalog permissions, requiring contact with Databricks account team before enabling the feature if Knowledge Assistant is actively used.
* [Label Mapping for Sequence Classification](/concepts/label-mapping-for-sequence-classification.md) - Mapping string labels to integer IDs and creating id2label/label2id dictionaries for Hugging Face AutoModelForSequenceClassification
* [Label Schema Types: Feedback vs Expectation](/concepts/assessment-types-feedback-vs-expectation.md) - Two fundamental assessment types for labeling schemas: Feedback (subjective opinions, ratings, preferences) and Expectation (objective ground truth, correct answers, expected behavior).
* [Label Schemas](/concepts/label-schemas.md) - Definition of structured feedback fields (categorical or text) used in labeling sessions to collect targeted expert assessments.
* [Label Schemas for Structured Feedback](/concepts/label-schemas-for-structured-feedback.md) - Defining structured feedback schemas using create_label_schema() with InputCategorical and InputText types to standardize expert review criteria and expected responses
* [Labeling and Review in MLflow](/concepts/labeling-and-review-in-mlflow.md) - The capability to view labeling sessions, inspect labeling schemas, and review feedback criteria such as ratings, comments, and expectations for GenAI agent traces.
* [Labeling and Review with Genie Code](/concepts/labeling-and-review-with-genie-code.md) - Genie Code capability to view labeling sessions, see who is assigned to review traces, and inspect labeling schemas to understand feedback criteria such as ratings, comments, and expectations.
* [Labeling Schema](/concepts/labeling-schema.md) - A structured definition of questions, input types, validation rules, and instructions that domain experts answer when labeling traces in the Review App for GenAI evaluation.
* [Labeling schema best practices](/concepts/labeling-schema-best-practices.md) - Guidelines for effective schema design including clear questions, specific prompts, reasonable limits, mutually exclusive options, descriptive naming, and lifecycle management.
* [Labeling schema CRUD management](/concepts/labeling-schema-management-api.md) - API-driven lifecycle management for labeling schemas: create (with overwrite), get, list, update, and delete operations scoped to MLflow experiments.
* [Labeling Schema Input Types](/concepts/labeling-schema-input-types.md) - The five supported input types for labeling schemas: InputCategorical, InputCategoricalList, InputText, InputTextList, and InputNumeric, each with configurable validation rules and constraints.
* [Labeling schema integration with sessions](/concepts/labeling-schema-integration-with-sessions.md) - How labeling schemas are associated with labeling sessions and presented in the Review App to collect structured feedback.
* [Labeling Schema Lifecycle Management](/concepts/labeling-schema-management-api.md) - CRUD operations for labeling schemas using the MLflow API: listing (get_label_schema), creating/updating (create_label_schema with overwrite=True), and deleting (delete_label_schema), with schemas scoped to experiments.
* [Labeling schema management API](/concepts/labeling-schema-management-api.md) - CRUD operations for labeling schemas via the MLflow API: create_label_schema, get_label_schema, and delete_label_schema, with overwrite support for updates.
* [Labeling schema scope and uniqueness](/concepts/labeling-schema-scope-and-uniqueness.md) - Labeling schemas are scoped to MLflow experiments, requiring unique names within each experiment.
* [Labeling Schema Scoping to Experiments](/concepts/labeling-schema-scoping-to-experiments.md) - Labeling schemas are scoped to MLflow experiments, requiring unique names within an experiment and enabling organized, experiment-specific feedback collection.
* [Labeling Schemas](/concepts/labeling-schemas.md) - Definitions of the questions and input types (categorical, numeric, free-text) that domain experts use to provide feedback or expectations on traces within the Review App.
* [Labeling schemas (MLflow Review App)](/concepts/labeling-schema.md) - Structured question definitions that domain experts answer when labeling existing traces in the MLflow Review App for GenAI evaluation.
* [Labeling schemas (MLflow)](/concepts/labeling-schema.md) - Structured question definitions that domain experts answer when labeling existing traces in the MLflow Review App for GenAI evaluation.
* [Labeling Session](/concepts/labeling-session.md) - A structured MLflow run container that collects human-generated assessments (feedback or expectations) from domain experts on GenAI application traces using the MLflow Review App.
* [Labeling session and schema integration](/concepts/labeling-session-and-schema-integration.md) - How labeling schemas are associated with labeling sessions to organize structured human review workflows in the Review App.
* [Labeling Session Dataset Synchronization](/concepts/labeling-session-dataset-synchronization.md) - The mechanism to sync Expectation data from labeling sessions to Evaluation Datasets via an intelligent upsert operation, using trace inputs as unique keys.
* [Labeling Session User Management](/concepts/labeling-session-user-management.md) - The policies, permissions model, and API methods for assigning and managing domain expert reviewers for labeling sessions, including SCIM integration and workspace access.
* [Labeling Session User Management and Permissions](/concepts/labeling-session-user-management-and-permissions.md) - Process for assigning users to labeling sessions, including automatic WRITE permission grants on the MLflow experiment, SCIM provisioning for non-workspace users, and APIs for adding or replacing assigned users.
* [Labeling Sessions](/concepts/labeling-sessions.md) - A special type of MLflow Run that organizes a set of traces for review by specific experts using selected labeling schemas, acting as a queue for the review process.
* [Labeling Sessions (MLflow)](/concepts/labeling-session.md) - Structured containers within MLflow that collect human-generated assessments (labels) on MLflow Traces from domain experts via the MLflow Review App.
* [Labeling Sessions and Labeling Schemas](/concepts/labeling-sessions-and-schemas.md) - Human annotation infrastructure where labeling sessions queue traces for domain expert review and labeling schemas define the structured questions and valid responses for consistent assessment collection.
* [Labeling Sessions and Schemas](/concepts/labeling-sessions-and-schemas.md) - Labeling sessions queue traces for human expert review, while labeling schemas define structured questions and valid responses to ensure consistent assessment collection across domain experts.
* [Labeling Sessions and Schemas (MLflow GenAI)](/concepts/labeling-sessions-and-schemas.md) - Infrastructure for collecting domain expert feedback, where sessions queue traces for review and schemas define structured questions for consistent labeling.
* [Labeling Sessions for Expert Review](/concepts/labeling-sessions-for-expert-review.md) - Structured expert review process where domain experts assess trace outputs using defined label schemas (e.g., categorical accuracy, ideal text responses) to establish ground truth.
* [Lakebase Autoscaling](/concepts/lakebase-autoscaling.md) - The latest version of Lakebase with autoscaling compute, scale-to-zero, branching, and instant restore, replacing the original Lakebase Provisioned offering.
* [Lakebase Autoscaling for Online Feature Stores](/concepts/lakebase-autoscaling-for-online-feature-stores.md) - The underlying managed infrastructure powering Databricks Online Feature Stores, providing scalability and high availability for real-time feature serving.
* [Lakebase Synced Table](/concepts/lakebase-synced-table.md) - A Unity Catalog read-only Postgres table that automatically synchronizes data from a Unity Catalog table to a Lakebase database instance for OLTP workloads.
* [Lakeflow Connect Source Lineage](/concepts/lakeflow-connect-source-lineage.md) - Automatic recording of source lineage from external source tables to destination tables in Unity Catalog via managed ingestion pipelines using Lakeflow Connect.
* [Lakeflow Jobs](/concepts/lakeflow-jobs.md) - Databricks' job orchestration platform for running automated workflows, including Python wheel tasks, on AWS.
* [Lakeflow Spark Declarative Pipelines](/concepts/lakeflow-spark-declarative-pipelines.md) - A Databricks pipeline framework supporting incremental (triggered or continuous) data processing, used here to read new OTel spans, apply transformations like PII redaction, and write results to target tables.
* [Lakeflow Spark Declarative Pipelines for Feature Tables](/concepts/lakeflow-spark-declarative-pipelines-for-feature-tables.md) - Streaming tables and materialized views created by Lakeflow Spark Declarative Pipelines can include primary key constraints and be used as feature tables in Unity Catalog.
* [Lakehouse Federation Data Sharing](/concepts/lakehouse-federation-data-sharing.md) - Sharing foreign schemas and tables from external data sources via Lakehouse Federation without copying data, using on-demand query materialization on the provider side.
* [Lakehouse Federation with Online Tables](/concepts/lakehouse-federation-with-online-tables.md) - The ability to query online tables using Lakehouse Federation and a Serverless SQL warehouse, supporting only SELECT operations for interactive or debugging purposes, not for production workloads.
* [Lakehouse Monitoring](/concepts/lakehouse-monitoring.md) - The former name for Data Profiling in Databricks Unity Catalog; now superseded by the data profiling capability within data quality monitoring.
* [Lakehouse Monitoring (Legacy Name)](/concepts/lakehouse-monitoring.md) - The former name of the Data Profiling feature in Databricks Unity Catalog; used for tracking summary statistics and historical metrics of data tables.
* [Landing Zone Architecture](/concepts/landing-zone-architecture.md) - An optional pre-ingestion layer in cloud object storage that receives raw files from external systems before processing into the bronze medallion layer, with patterns for triggers and governance.
* [LangChain Tool for Feature Serving](/concepts/langchaintool-with-feature-serving.md) - A LangChain tool wrapper that calls a Databricks feature serving endpoint to retrieve structured data augmenting the context passed to an LLM.
* [LangChainTool for Feature Lookup](/concepts/langchaintool-for-feature-lookup.md) - A LangChain tool created to query a Databricks feature serving endpoint for relevant structured data in a RAG pipeline
* [LangChainTool with Feature Serving](/concepts/langchaintool-with-feature-serving.md) - A LangChain tool that wraps a Databricks feature serving endpoint to enable an agent to look up structured data during a RAG workflow.
* [Langfuse @observe() Decorator](/concepts/langfuse-observe-decorator.md) - Langfuse's function decorator that automatically creates OpenTelemetry trace spans for instrumented functions, which can be exported to any OTel-compatible backend.
* [Langfuse OTEL Exporter Hijacking](/concepts/langfuse-otel-exporter-hijacking.md) - Setting Langfuse environment variables to dummy values and attaching a custom BatchSpanProcessor with an OTLPSpanExporter to intercept Langfuse-generated OTel spans and redirect them to Databricks instead of the Langfuse server.
* [Langfuse SDK Dummy Configuration](/concepts/langfuse-sdk-dummy-configuration.md) - A technique that disables Langfuse's own trace collection by setting LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, and LANGFUSE_SECRET_KEY to dummy values, so only an attached OTLP exporter receives spans.
* [Langfuse SDK Export Redirection](/concepts/langfuse-sdk-export-redirection.md) - Technique of setting dummy Langfuse environment variables (LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY) to prevent traces from being sent to Langfuse servers while still using the SDK's OpenTelemetry TracerProvider for custom export.
* [Langfuse-Databricks Trace Export](/concepts/langfuse-databricks-trace-export.md) - Configuring Langfuse to route OpenTelemetry trace spans to the Databricks MLflow OTLP endpoint for storage and governance in Unity Catalog.
* [Langfuse-to-Databricks Trace Export](/concepts/langfuse-databricks-trace-export.md) - A configuration pattern that redirects Langfuse-instrumented OpenTelemetry spans to Databricks MLflow's OTLP endpoint instead of Langfuse's own servers, enabling trace storage in Unity Catalog tables.
* [Language Support for Databricks Connect](/concepts/language-support-for-databricks-connect.md) - Databricks Connect is available for Python, R, and Scala, with each language having its own dedicated documentation and tutorials.
* [Large Artifact Logging Pattern](/concepts/large-artifact-logging-pattern.md) - Best practice for handling large artifacts in distributed Ray workloads by persisting them as files and logging with MLflow via file path references.
* [Large Dataset Sampling for AutoML](/concepts/large-dataset-sampling-for-automl.md) - AutoML's automatic memory estimation and stratified/random sampling of large datasets to fit single-worker-node training constraints.
* [Large Dataset Sampling in AutoML](/concepts/large-dataset-sampling-in-automl.md) - How Databricks AutoML samples large datasets for classification and regression to fit within single-node memory constraints, using stratified sampling for classification.
* [Large Image File Handling Strategy](/concepts/large-image-file-handling-strategy.md) - Recommendation to store only metadata in Delta tables and load large images (>100 MB on average) directly from object store by path when needed, to avoid performance bottlenecks.
* [Large language models (LLMs) on Databricks](/concepts/large-language-models-llms-on-databricks.md) - Notebook examples for fine-tuning large language models on Databricks AI Runtime, including parameter-efficient methods.
* [Latency-Throughput Trade-off](/concepts/latency-throughput-trade-off.md) - The fundamental tension in LLM serving where increasing concurrent requests improves throughput at the cost of higher per-request latency, requiring workload-specific balancing.
* [Latency-Throughput Trade-off in LLM Serving](/concepts/latency-throughput-trade-off.md) - The fundamental tension in LLM serving where increasing concurrent requests improves throughput at the cost of higher latency, requiring careful control of parallelism based on use case.
* [Learning from Sets Dataset (GroupLens)](/concepts/learning-from-sets-dataset-movielens.md) - A movie rating dataset from GroupLens (Learning from Sets 2019) used for training the two-tower recommendation model, preprocessed with label binarization.
* [Learning from Sets Dataset (MovieLens)](/concepts/learning-from-sets-dataset-movielens.md) - A public MovieLens-derived dataset used for training and evaluating recommendation models, preprocessed with rating binarization
* [Legacy Access Control vs Unity Catalog](/concepts/legacy-access-control-vs-unity-catalog.md) - The access control model described applies only to Databricks workspaces not enabled for Unity Catalog; Unity Catalog-enabled workspaces use Unity Catalog privileges instead.
* [Legacy Anomaly Detection Configurations](/concepts/legacy-anomaly-detection-configurations.md) - Earlier version features including a data quality dashboard at /Shared/Databricks Quality Monitoring/ and customizable job parameters for freshness/completeness evaluation thresholds.
* [Legacy Data Governance Models in Databricks](/concepts/legacy-data-governance-models-in-databricks.md) - Older data governance approaches on Databricks that are superseded by Unity Catalog, including Hive metastore table access control.
* [Legacy data profiling billing query pattern](/concepts/legacy-data-profiling-billing-query-pattern.md) - Pre-February 2026 query pattern for data profiling expenses using sku_name containing JOBS_SERVERLESS and custom_tags['LakehouseMonitoring'] instead of table_id filter.
* [Legacy Delta Configuration Migration](/concepts/legacy-delta-configuration-removal.md) - Recommendation to remove explicit legacy Delta configurations from Spark configurations and table properties when upgrading to a new Databricks Runtime version to enable new optimizations and defaults.
* [Legacy Delta Configuration Removal](/concepts/legacy-delta-configuration-removal.md) - Best practice of removing explicit legacy Delta configurations from Spark configs and table properties when upgrading Databricks Runtime versions to avoid blocking new optimizations.
* [Legacy Feature Store Access Control vs Unity Catalog](/concepts/legacy-access-control-vs-unity-catalog.md) - Databricks Feature Store offers two access control models: legacy fine-grained permissions for workspaces without Unity Catalog, and Unity Catalog privileges for workspaces with Unity Catalog enabled.
* [Legacy Foundation Model Families](/concepts/legacy-foundation-model-families.md) - Legacy model families (e.g., Meta Llama 2/3, DBRX, Mistral) on Databricks that provision inference capacity based on tokens per second ranges rather than model units.
* [Legacy MLflow Model Serving](/concepts/legacy-mlflow-model-serving.md) - The older MLflow-based model serving experience on Databricks that reaches end of life on September 15, 2025, with new endpoint creation blocked after August 22, 2025.
* [Legacy Online Tables](/concepts/legacy-online-tables.md) - The original Databricks online table product being deprecated; users are instructed to migrate to Online Feature Store or Lakebase synced tables.
* [Legacy replaceWhere Partition-Only Overwrite](/concepts/legacy-replacewhere-partition-only-overwrite.md) - Older Delta Lake behavior where replaceWhere only works on partition columns, enabled by setting spark.databricks.delta.replaceWhere.dataColumns.enabled to false.
* [Legacy vs Unity Catalog Access Control for Feature Store](/concepts/legacy-vs-unity-catalog-access-control-for-feature-store.md) - Databricks Feature Store offers two access control paradigms: a legacy permission system with three levels (used in workspaces without Unity Catalog) and Unity Catalog privileges (used in workspaces enabled for Unity Catalog).
* [Legacy Workspace Feature Store](/concepts/legacy-workspace-feature-store.md) - The older feature store system in Databricks, separate from Unity Catalog, still supported for AutoML integration.
* [Library Deprecation and Removal Policy](/concepts/library-deprecation-policy.md) - Policies governing when pre-installed or top-tier libraries are deprecated and removed from Databricks Runtime ML, including triggers such as inactivity, low usage, or replacement by superior packages.
* [Library deprecation criteria](/concepts/library-deprecation-criteria.md) - Conditions under which Databricks removes a top-tier or pre-installed library, including inactivity metrics, lack of maintenance, and low usage.
* [Library Deprecation Policy](/concepts/library-deprecation-policy.md) - Defines conditions under which Databricks removes libraries from top-tier or pre-installed status, including inactivity, low usage, or replacement.
* [Library Maintenance Activity Criteria](/concepts/library-maintenance-activity-criteria.md) - Specific metrics used by Databricks to determine if a library is actively maintained, based on commit frequency and release cadence (e.g., no commits in 2-3 months, no releases in 6-9 months).
* [Library removal notification process](/concepts/library-removal-notification-process.md) - The multi-channel customer notification workflow Databricks follows before removing a library, including release notes, import warnings, and documentation updates.
* [Lifecycle Policy Extension Problem](/concepts/lifecycle-policy-extension-problem.md) - The risk and complications that arise when extending the cloud lifecycle management transition rule to a longer interval, due to the mismatch between the table property and actual archival state during the lag period.
* [Lifecycle policy extension risks with archival support](/concepts/lifecycle-policy-extension-risks-with-archival-support.md) - Extending the cloud lifecycle management transition rule (e.g., from 60 to 90 days) can cause errors because previously archived files are not automatically restored; delta.timeUntilArchived must never exceed the actual age of the most recently archived data.
* [Lifecycle policy extension risks with Delta Lake archival](/concepts/lifecycle-policy-extension-risks-with-delta-lake-archival.md) - Extending the archival interval in cloud lifecycle policies can lead to errors because previously archived files are not automatically restored.
* [Lifecycle Policy Transition Rule Changes](/concepts/lifecycle-policy-transition-rule-changes.md) - Procedures and risks associated with changing the cloud lifecycle management transition rule interval and keeping the delta.timeUntilArchived property in sync.
* [Liger Kernels](/concepts/liger-kernels.md) - GPU-optimized Triton kernels that fuse multiple operations (e.g., linear + loss) to reduce memory usage by up to 80%, optimized for transformer operations like RMSNorm, RoPE, SwiGLU, and CrossEntropy.
* [LightningModule with AUROC Metrics for Recommendation](/concepts/lightningmodule-with-auroc-metrics-for-recommendation.md) - Using PyTorch Lightning's LightningModule pattern with torchmetrics AUROC to encapsulate training, validation, and logging for binary relevance prediction in recommender systems.
* [Limitations of Databricks Autologging](/concepts/limitations-of-databricks-autologging.md) - Known limitations including driver-node-only execution, unsupported XGBoost scikit-learn integration, and interaction with Spark MLlib and Hyperopt integrations
* [Limitations of Databricks Connect Long-lived Sessions](/concepts/limitations-of-databricks-connect-long-lived-sessions.md) - Long-lived sessions are only available on serverless compute (not standard/dedicated), are best-effort (not guaranteed), have state size limits, expire after 2 days of inactivity, and do not preserve streaming query state or EXECUTE IMMEDIATE SQL scripts.
* [Limitations of Databricks Feature Engineering Client](/concepts/limitations-of-databricks-feature-engineering-client.md) - The client library can only run on Databricks (Databricks Runtime or Databricks Runtime ML) and does not support calling Feature Engineering APIs from external/local environments.
* [Limitations of Databricks Utilities in Scala Connect](/concepts/limitations-of-databricks-utilities-in-scala-connect.md) - When using Databricks Connect for Scala, only DBFS and secrets utilities are available through DBUtils.getDBUtils; no other Databricks Utilities functionality is accessible.
* [Limitations of dbutils in Databricks Connect](/concepts/restricted-dbutils-in-databricks-connect.md) - Only the fs and secrets Databricks Utilities are available through dbutils in Databricks Connect; all other dbutils utilities are not accessible
* [Limitations of Delta archival support](/concepts/limitations-of-delta-archival-support.md) - Key constraints include: transaction log must never be archived, no support for access-time or tag-based policies, DROP COLUMN and PURGE restrictions, and LIMIT query incompatibility.
* [Limitations of Distributed XGBoost](/concepts/limitations-of-distributed-xgboost.md) - Restrictions when using distributed XGBoost on Databricks: no mlflow.xgboost.autolog, no baseMarginCol, and incompatibility with autoscaling clusters.
* [Limitations of Distributed XGBoost on Databricks](/concepts/limitations-of-distributed-xgboost.md) - Key restrictions when using distributed XGBoost training: mlflow.xgboost.autolog is unavailable, baseMarginCol cannot be used, and autoscaling clusters are not supported
* [Limitations of External Lineage](/concepts/limitations-of-external-lineage.md) - Constraints on external lineage in Unity Catalog including: lineage not recorded in system.access lineage tables, and resource limits of 10,000 external metadata objects and 100,000 lineage relationships per metastore.
* [Limitations of Multimodal Model Serving on Databricks](/concepts/limitations-of-multimodal-model-serving-on-databricks.md) - Constraints for audio and video inputs on Databricks: only available on Gemini pay-per-token models (not provisioned throughput), and large files increase latency and token usage.
* [Limitations of Route Optimization](/concepts/limitations-of-route-optimization.md) - Route optimization is only available for custom model and feature serving endpoints, not Foundation Model APIs or external models.
* [Limitations of row filters and column masks](/concepts/limitations-of-row-filters-and-column-masks.md) - Comprehensive set of constraints including runtime version requirements, unsupported operations (MERGE, time travel, clones, views), and dedicated access mode restrictions.
* [Limited Databricks Utilities Support in Scala Connect](/concepts/limitations-of-databricks-utilities-in-scala-connect.md) - Only DBFS and secrets utilities are available through Databricks Connect for Scala; no other Databricks Utilities (like widgets, notebook, etc.) are supported for Scala projects.
* [Limited dbutils availability in Databricks Connect](/concepts/limited-dbutils-availability-in-databricks-connect.md) - Only the `fs` and `secrets` Databricks Utilities are available via dbutils in Databricks Connect; all other utilities are unavailable.
* [Limited Utility Availability in Databricks Connect for Scala](/concepts/limited-utility-availability-in-databricks-connect-for-scala.md) - Only DBFS filesystem and secrets utilities are available through Databricks Utilities for Scala; no other utilities are supported
* [Lineage Graph in Catalog Explorer](/concepts/lineage-graph-in-catalog-explorer.md) - Interactive visual interface in Databricks Catalog Explorer for browsing upstream and downstream dependencies of tables, jobs, pipelines, and dashboards with expandable nodes and edge details.
* [Lineage Limitations and Boundary Conditions](/concepts/lineage-limitations-and-boundary-conditions.md) - Notable gaps in lineage capture including renamed objects, RDDs, global temp views, path-referenced tables, UDFs, Jobs API runs_submit workflows, Spark SQL checkpointing, and incomplete coverage for PRIVATE tables in Lakeflow pipelines.
* [Lineage Limitations and Edge Cases](/concepts/lineage-limitations-and-edge-cases.md) - Known gaps in Unity Catalog lineage capture including renamed objects, Spark SQL checkpointing, RDDs, global temp views, path-referenced tables, UDFs, and Jobs API runs_submit workflows.
* [Lineage Limitations in Unity Catalog](/concepts/lineage-limitations-in-unity-catalog.md) - Known constraints including no lineage before September 2024, no preservation on renamed objects, no capture for RDDs, global temp views, spark-submit jobs, and incomplete column coverage for UDFs and path-based references.
* [Lineage Permissions and Access Control](/concepts/lineage-permissions-and-access-control.md) - Lineage visibility is governed by Unity Catalog's BROWSE permission on catalog objects and workspace-level permissions on notebooks, jobs, and dashboards, with cross-workspace masking.
* [Lineage Permissions and Masked Nodes](/concepts/lineage-permissions-and-masked-nodes.md) - Access control model where lineage visibility is governed by Unity Catalog BROWSE/SELECT privileges, and unauthorized tables appear as masked nodes in the lineage graph.
* [Lineage Permissions Model](/concepts/lineage-permissions-model.md) - Access control for lineage data requiring BROWSE privilege on catalog objects and workspace-level permissions on notebooks, jobs, and dashboards, with masked nodes for unauthorized objects.
* [Lineage Relationship Patterns](/concepts/lineage-relationship-patterns.md) - Design patterns for modeling external lineage, including chaining multiple external metadata objects, connecting two Unity Catalog tables through an intermediary, and adding column-level lineage.
* [Lineage Retention and Time Range](/concepts/lineage-retention-and-time-range.md) - Lineage data in Catalog Explorer is retained indefinitely for events after September 1, 2024, with an 'All time' option for newer metastores; lineage system tables retain a rolling one-year window.
* [Lineage Retention Policy](/concepts/lineage-retention-policy.md) - Lineage data displayed in Catalog Explorer is retained indefinitely from September 1, 2024 onward, while system tables retain a rolling one-year window.
* [Lineage System Tables](/concepts/lineage-system-tables.md) - Programmatic access to lineage data via system.access.table_lineage and system.access.column_lineage tables, which retain a rolling one-year window of lineage events.
* [Liquid Clustering](/concepts/liquid-clustering.md) - A Databricks data layout optimization technique recommended for Delta Lake tables to improve query performance.
* [Liquid Clustering (Databricks)](/concepts/liquid-clustering-databricks.md) - A Databricks table clustering feature recommended as a replacement for deprecated Bloom filter indexes.
* [Liquid Clustering Enablement on Existing Tables](/concepts/liquid-clustering-enablement-on-existing-tables.md) - Enabling liquid clustering on an existing Delta table requires using overwrite mode with the overwriteSchema option set to true.
* [Liquid tables](/concepts/liquid-tables.md) - Delta Lake tables that support clustering columns and impose restrictions on certain commands requiring predicates that reference only clustering columns.
* [Liquid tables (Databricks)](/concepts/liquid-tables-databricks.md) - A Databricks table type that supports clustering columns and restricts certain command predicates to only reference those clustering columns.
* [List of Dictionaries Evaluation Pattern](/concepts/list-of-dictionaries-evaluation-pattern.md) - Using a simple list of Python dictionaries for quick prototyping and small-scale GenAI evaluation without formal dataset creation.
* [LitTwoTower: Lightning Module Wrapping TwoTowerModel](/concepts/littwotower-lightning-module-wrapping-twotowermodel.md) - A PyTorch Lightning module that wraps the TwoTowerModel with BCE loss, AUROC metrics, and batch transformation from DataFrame rows to TorchRec Batch objects.
* [LiveSpan Input/Output Management](/concepts/livespan-inputoutput-management.md) - Manual specification of span inputs and outputs via LiveSpan.set_inputs() and LiveSpan.set_outputs() when using context managers, unlike automatic capture with function decorators.
* [Llama 3.2 1B Fine-tuning on Databricks](/concepts/llama-32-1b-fine-tuning-on-databricks.md) - A reference workflow for fully fine-tuning the Llama 3.2 1B Instruct model on 8xH100 GPUs using AI Runtime, DeepSpeed, and TRL, with bfloat16 precision and gradient checkpointing.
* [LLM Endpoint Benchmarking Harness](/concepts/llm-endpoint-benchmarking-harness.md) - A systematic approach to benchmarking LLM endpoints by measuring total latency, throughput, and the throughput-versus-latency curve across varying numbers of parallel requests.
* [LLM Endpoint Benchmarking Methodology](/concepts/llm-endpoint-benchmarking-methodology.md) - A Databricks-recommended approach using a benchmarking harness notebook that measures total latency vs. throughput across varying concurrency levels to characterize endpoint performance.
* [LLM endpoint governance](/concepts/llm-endpoint-governance.md) - Controlling access to hosted and external LLM endpoints, enforcing rate limits, and tracking usage and costs across providers via Unity AI Gateway.
* [LLM Endpoint Latency vs Throughput Trade-off](/concepts/llm-endpoint-latency-vs-throughput-trade-off.md) - The balancing act in LLM serving systems where increasing concurrent requests raises throughput at the cost of higher latency, controlled by managing parallel request load.
* [LLM Evaluation Dataset Schema](/concepts/llm-evaluation-dataset-schema.md) - The structured dataset format used in mlflow.genai.evaluate, requiring inputs, outputs, and expectations (with context) fields
* [LLM Evaluation Quality Dimensions](/concepts/llm-evaluation-quality-dimensions.md) - Specific quality aspects evaluated by LLM judges including relevance, retrieval relevance, safety, groundedness, correctness, sufficiency, guidelines compliance, tool call correctness, and tool call efficiency.
* [LLM Fine-tuning in Production](/concepts/llm-fine-tuning-in-production.md) - Fine-tuning existing LLMs on custom data to improve performance for specific scenarios, represented as distinct pipeline jobs in production architecture
* [LLM fine-tuning on Databricks](/concepts/llm-fine-tuning-on-databricks.md) - Notebook examples for fine-tuning large language models on AI Runtime, including parameter-efficient methods.
* [LLM Foundry YAML Configuration](/concepts/llm-foundry-yaml-configuration.md) - A structured YAML-based configuration system for LLM Foundry that defines model architecture, FSDP settings, training hyperparameters, dataset configuration, MLflow logging, callbacks, and checkpointing in a declarative format.
* [LLM Function Calling](/concepts/llm-function-calling.md) - A technique that lets LLMs generate structured JSON output describing function arguments, enabling reliable integration with external APIs and code.
* [LLM Inference Latency](/concepts/llm-inference-latency.md) - Total response time calculated as TTFT plus the product of TPOT and number of generated tokens, representing end-to-end user-perceived delay.
* [LLM Inference Phases: Prefill and Decoding](/concepts/llm-inference-prefill-and-decoding.md) - LLMs perform inference in two distinct phases: prefill (parallel processing of input tokens) and decoding (auto-regressive generation of output tokens one at a time).
* [LLM Inference Prefill and Decoding](/concepts/llm-inference-prefill-and-decoding.md) - The two-step process by which LLMs perform inference: prefill processing input tokens in parallel, followed by auto-regressive decoding that generates one token at a time.
* [LLM Inference Two-Step Process: Prefill and Decoding](/concepts/llm-inference-prefill-and-decoding.md) - LLM inference consists of two sequential phases: parallel prompt processing (prefill) followed by auto-regressive token generation (decoding).
* [LLM Judge Customization](/concepts/llm-judge-customization.md) - The ability to change the underlying model that powers a built-in LLM judge by specifying a different model via the 'model' argument using LiteLLM-compatible format.
* [LLM Judge Customization Paths](/concepts/llm-judge-customization-paths.md) - Alternatives to built-in judges, including custom LLM judges for more control and Python code-based scorers, enabling users to tailor evaluation to their specific use cases.
* [LLM Judge Model Customization](/concepts/llm-judge-model-customization.md) - The ability to change the underlying model powering a guidelines judge using the model argument with a provider/model-name format compatible with LiteLLM.
* [LLM Judge Model Infrastructure and Privacy](/concepts/llm-judge-model-infrastructure-and-privacy.md) - LLM judges for guidelines evaluation use specially-tuned models hosted by region (EU or US), support custom model selection via LiteLLM-compatible providers, and have privacy controls including opt-out of Abuse Monitoring and partner-powered AI feature disabling.
* [LLM Judge Model Trust and Data Privacy](/concepts/llm-judge-model-trust-and-data-privacy.md) - Operational and trust considerations for LLM judges, including third-party model hosting (Azure OpenAI), regional data residency, abuse monitoring opt-out, and restrictions on using judge outputs for model training.
* [LLM Judge Model Trust and Privacy](/concepts/llm-judge-model-trust-and-privacy.md) - Information about the models powering LLM judges, including Azure OpenAI hosting, regional data residency, opt-out of abuse monitoring, and the ability to disable partner-powered models.
* [LLM Judges](/concepts/llm-judges.md) - A type of MLflow Scorer that uses Large Language Models to assess quality across dimensions like relevance, safety, and correctness.
* [LLM Judges and Scorers](/concepts/llm-judges-and-scorers.md) - Built-in or custom evaluation components used to assess LLM and agent outputs, reusable across both development testing and production monitoring.
* [LLM Model Selection for Conversation Simulation](/concepts/llm-model-selection-for-conversation-simulation.md) - Configuring which LLM powers the simulated user in conversation simulation, supporting provider-prefixed model names and regional hosting constraints.
* [LLM Throughput vs Latency Trade-off](/concepts/llm-throughput-vs-latency-trade-off.md) - The fundamental tension in LLM serving where increasing concurrent requests boosts throughput but also raises latency, requiring careful load control based on use case.
* [LLM-as-a-Judge](/concepts/llm-as-a-judge.md) - The pattern of using a large language model to evaluate the quality of outputs from another LLM-based system, serving as an automated evaluator for dimensions like relevance and safety.
* [LLM-as-a-Judge Evaluation](/concepts/llm-as-a-judge-evaluation.md) - Using an LLM to automatically evaluate the quality of another LLM's outputs based on criteria like safety, correctness, and custom guidelines.
* [LLM-as-a-Judge Evaluation Metrics](/concepts/llm-as-a-judge-evaluation-metrics.md) - DeepEval provides LLM-based evaluation metrics that require a model parameter to act as the judge for scoring outputs
* [LLM-as-a-Judge for Relevance](/concepts/llm-as-a-judge-for-relevance.md) - The pattern of using a configurable LLM (defaulting to a Databricks-hosted model) to automatically score the relevance of responses and retrieved documents in GenAI applications.
* [LLM-as-a-Judge for Relevance Evaluation](/concepts/llm-as-a-judge-for-relevance-evaluation.md) - The paradigm of using a secondary LLM (judge model) to automatically assess the relevance of outputs or retrieved documents in GenAI applications, as implemented by MLflow's built-in scorers.
* [LLM-as-a-judge in MLflow evaluation](/concepts/llm-as-a-judge-in-mlflow-evaluation.md) - Pattern where a scorer integrates a custom or externally hosted LLM to evaluate response quality, using structured prompts and parsing the LLM's JSON output into Feedback objects with AssessmentSource metadata.
* [LLM-as-a-Judge Paradigm](/concepts/llm-as-a-judge-paradigm.md) - The pattern of using large language models themselves as evaluators to assess quality dimensions of other LLM outputs, as implemented by Databricks-hosted judge models.
* [LLM-as-a-Judge Scoring](/concepts/llm-as-a-judge-scoring.md) - DeepEval scorers use an LLM-as-a-judge approach where a language model (e.g., databricks-gpt-5-mini or gpt-4o) evaluates outputs against configurable thresholds.
* [LLM-as-judge evaluation paradigm](/concepts/llm-as-judge-evaluation-paradigm.md) - The methodology of using one LLM to evaluate the outputs of another LLM-powered application, scoring along quality dimensions such as relevance, safety, groundedness, and correctness.
* [LLM-based vs Non-LLM RAGAS Scorers](/concepts/llm-based-vs-non-llm-ragas-scorers.md) - RAGAS scorers divide into LLM-based metrics that require a model parameter (e.g., Faithfulness) and non-LLM deterministic metrics that need no model (e.g., ExactMatch).
* [LLMOps vs MLOps](/concepts/llmops-vs-mlops.md) - Key differences and commonalities between traditional MLOps and LLMOps workflows for large language models
* [Load Test Cluster Configuration on Databricks](/concepts/load-test-cluster-configuration-on-databricks.md) - Recommended compute configuration for running Locust-based load tests: single-node cluster with Databricks Runtime 15.4 LTS ML and a CPU-optimized instance type with at least 32 cores.
* [Load Test Payload Specification](/concepts/load-test-payload-specification.md) - Guidelines for choosing representative payloads in load tests to accurately reflect production usage patterns and validate them via the Serving UI query interface
* [Load Testing for ML Serving Endpoints](/concepts/load-testing-for-ml-serving-endpoints.md) - The practice of simulating real-world traffic on model serving infrastructure to measure performance, identify bottlenecks, and correctly size endpoints for production workloads.
* [Load Testing for Model Serving Endpoints](/concepts/load-testing-for-model-serving-endpoints.md) - Testing methodology that measures endpoint performance under realistic traffic conditions to determine optimal provisioned concurrency, identify bottlenecks, and validate latency and throughput requirements.
* [load_context and predict Model Lifecycle](/concepts/load_context-and-predict-model-lifecycle.md) - Custom MLflow pyfunc models require two functions: load_context for one-time initialization and predict for per-request inference logic.
* [Loading and Using Prompts from the Registry](/concepts/loading-and-using-prompts-from-the-registry.md) - Prompts can be loaded by name and version using URI syntax (prompts:/catalog.schema.name/version) or explicitly, then formatted with variables and used in LLM applications.
* [Loading and Using Prompts in Applications](/concepts/loading-and-using-prompts-in-applications.md) - The pattern of loading prompt versions from the registry using URI syntax (prompts:/catalog.schema.name/version) and formatting them with runtime variables for use in LLM applications.
* [Loading Data for Machine Learning on Databricks](/concepts/loading-data-for-machine-learning-on-databricks.md) - Best practices and patterns for loading data specifically for ML and DL workloads on Databricks, as distinct from general data ingestion.
* [Loading Fine-tuned Models for Inference via MLflow](/concepts/loading-fine-tuned-models-for-inference-via-mlflow.md) - Loading logged Hugging Face models from MLflow as Spark UDFs for batch inference on Databricks
* [Loading MLflow experiments by ID](/concepts/loading-mlflow-experiments-by-id.md) - Loading MLflow experiment run data into a Spark DataFrame by passing one or more experiment IDs as a comma-separated string to the mlflow-experiment data source.
* [Loading MLflow experiments by name](/concepts/loading-mlflow-experiments-by-name.md) - Resolving an MLflow experiment name to an ID using mlflow.get_experiment_by_name() and then passing that ID to the mlflow-experiment data source.
* [Loading Parquet Files Directly with Petastorm](/concepts/loading-parquet-files-directly-with-petastorm.md) - The alternative (less preferred) workflow for using Petastorm by saving data in Parquet format to DBFS and loading it via the DBFS mount point, bypassing the Spark Converter API.
* [Loading TFRecords into Spark DataFrames](/concepts/loading-tfrecords-into-spark-dataframes.md) - The process of reading TFRecord files into Apache Spark DataFrames using the spark-tensorflow-connector library.
* [Local Development Environment Integration with Databricks MLflow](/concepts/local-development-environment-integration-with-databricks-mlflow.md) - The workflow for connecting a local IDE (VS Code, PyCharm, Cursor, Jupyter) to a Databricks workspace for MLflow experiment tracking, using environment variables and authentication tokens.
* [Local Model Serving Simulation](/concepts/local-model-serving-simulation.md) - Manual approach to test model serving behavior by loading the model on an all-purpose cluster or local PC with a matching environment before endpoint deployment.
* [Local vs Databricks Environment Constraints for Feature Engineering](/concepts/local-vs-databricks-environment-constraints-for-feature-engineering.md) - The limitation that the client library can only run on Databricks (Runtime or ML Runtime) for actual API calls, but can be installed locally for code development, IDE support, and unit testing.
* [Local vs Remote Execution Model in Databricks Connect](/concepts/local-vs-remote-execution-model-in-databricks-connect.md) - The execution model where all Python code runs locally while PySpark DataFrame operations run on the remote Databricks cluster, with responses sent back to the local caller.
* [LOCATION_NOT_SUPPORTED](/concepts/location_not_supported.md) - Sub-error indicating that specifying a LOCATION is not supported when deep cloning a streaming table because the cloned table uses managed storage.
* [LOCATION_NOT_SUPPORTED sub-error](/concepts/location_not_supported.md) - Sub-error of DELTA_DEEP_CLONE_STREAMING_TABLE_ERROR indicating that specifying a custom LOCATION is forbidden because the cloned streaming table uses managed storage.
* [Locust CPU Resource Constraints](/concepts/locust-cpu-resource-constraints.md) - Understanding that Locust performance is CPU-bound (roughly 4000 requests per second per CPU core) and that auto-detection of CPU cores via the --processes -1 flag maximizes throughput
* [Locust Load Testing for ML Endpoints](/concepts/locust-load-testing-for-ml-endpoints.md) - Using the open-source Locust framework to run distributed load tests against Databricks Model Serving endpoints, configured to auto-detect CPU cores for maximum request throughput.
* [Locust Load Testing Framework](/concepts/locust-load-testing-framework.md) - An open-source framework for load testing production-grade endpoints, capable of modifying connection count and spawn rate while measuring endpoint performance.
* [Locust Load Testing Framework for Model Serving](/concepts/locust-load-testing-framework-on-databricks.md) - Using the open-source Locust framework to run automated, parameterized load tests against Databricks Model Serving endpoints, with support for custom concurrency, spawn rates, and CPU-aware parallel execution.
* [Locust Load Testing Framework on Databricks](/concepts/locust-load-testing-framework-on-databricks.md) - Using the open-source Locust framework to run automated, configurable load tests against Databricks Model Serving endpoints, supporting parameter tuning for client connections, spawn rates, and multi-CPU utilization.
* [Log Probability for Chat Completion](/concepts/log-probability-for-chat-completion.md) - The logprobs parameter for chat completion tasks that provides token log probabilities for classification, model uncertainty assessment, and evaluation metrics.
* [Logged Model Evaluation with MLflow](/concepts/logged-model-evaluation-with-mlflow.md) - Loading logged MLflow models with pyfunc and wrapping them for evaluation using mlflow.genai.evaluate() with named parameters.
* [Logged Models](/concepts/logged-models.md) - A concept in MLflow 3 that tracks a model's progress throughout its lifecycle by persisting metadata, metrics, parameters, and artifacts across environments and runs for comparison and debugging.
* [LoggedModel](/concepts/loggedmodel.md) - MLflow abstraction that packages model artifacts together with metrics, parameters, and lineage; when registered to Unity Catalog, all metadata becomes queryable in the UI and API.
* [Long-Term Trace Analysis and Dashboards](/concepts/long-term-trace-analysis-and-dashboards.md) - Archiving traces to Delta tables enables building custom dashboards, performing in-depth analytics on trace data, and maintaining a durable record of application behavior beyond ephemeral tracking.
* [Lookback window for feature lookups](/concepts/lookback-window-for-feature-lookups.md) - A timedelta parameter in FeatureLookup that excludes feature values older than a specified duration from the training set, applied during training and batch inference but not during online serving.
* [Lookup table optimization for ABAC policy UDFs](/concepts/lookup-table-optimization-for-abac-policy-udfs.md) - Using small lookup tables in policy UDFs enables broadcast hash joins where the lookup table is copied to each executor for fast filtering, but large lookup tables or complex predicates force slower shuffle joins.
* [Lookup tables in ABAC policy UDFs](/concepts/lookup-tables-in-abac-policy-udfs.md) - Using small lookup tables to express user-specific access rules that cannot be captured by the TO/EXCEPT clauses alone, leveraging broadcast hash joins for performance.
* [LoRA (Low-Rank Adaptation)](/concepts/lora-low-rank-adaptation.md) - A parameter-efficient fine-tuning technique that trains small low-rank adapter matrices while freezing the base model weights, drastically reducing the number of trainable parameters.
* [LoRA (Low-Rank Adaptation) for Efficient Finetuning](/concepts/lora-low-rank-adaptation.md) - Parameter-efficient fine-tuning technique that adapts pre-trained models by training low-rank matrices targeting specific attention modules (q_proj, k_proj, v_proj, etc.), reducing memory footprint.
* [LoRA (Low-Rank Adaptation) for LLM fine-tuning](/concepts/lora-low-rank-adaptation.md) - Parameter-efficient fine-tuning technique that trains small adapter layers while freezing the base model to reduce compute and memory requirements.
* [LoRA (Low-Rank Adaptation) for LLMs](/concepts/lora-low-rank-adaptation.md) - A parameter-efficient fine-tuning technique that trains small rank-decomposition matrices on targeted modules (q_proj, k_proj, v_proj, etc.) while keeping the base model frozen, supported by Unsloth.
* [LoRA Adapter Merging and Model Export](/concepts/lora-adapter-merging-and-model-export.md) - Process of merging learned LoRA adapter weights into the base model (merge_and_unload) and exporting both tokenizer and merged model for inference and registration.
* [LoRA Adapter Merging Pattern](/concepts/lora-adapter-merging-pattern.md) - The process of merging a trained LoRA adapter with the base model to produce a standalone combined model for deployment
* [LoRA Adapter Merging Workflow](/concepts/lora-adapter-merging-workflow.md) - The process of loading a trained LoRA adapter, merging it with the base model using merge_and_unload(), and registering the combined model for deployment
* [LoRA Configuration Parameters](/concepts/lora-configuration-parameters.md) - Key hyperparameters for configuring LoRA including rank (r=8) balancing performance vs parameters, alpha (32) as scaling factor, dropout (0.1) for regularization, and target module selection for attention and MLP layers.
* [LoRA Finetuning with Unsloth on Llama 3.2](/concepts/lora-finetuning-with-unsloth-on-llama-32.md) - The specific recipe for applying Low-Rank Adaptation (LoRA) to Llama-3.2-3B-Instruct using Unsloth's FastLanguageModel API, including hyperparameters like r=16, lora_alpha=16, and target module selection.
* [Low Shuffle Merge](/concepts/low-shuffle-merge.md) - An optimized implementation of MERGE in Delta Lake that provides better performance for most common workloads and preserves existing data layout optimizations like liquid clustering on unmodified data
* [Low-Rank Adaptation (LoRA)](/concepts/low-rank-adaptation-lora.md) - A parameter-efficient fine-tuning technique that adds small, trainable rank-decomposition matrices to attention and feed-forward layers while keeping the base model frozen, reducing memory and compute requirements.
* [Low-Rank Adaptation (LoRA) for Parameter-Efficient Fine-Tuning](/concepts/low-rank-adaptation-lora.md) - A technique that reduces the number of trainable parameters by adding small adapter layers to a frozen base model, making fine-tuning of large language models more memory and compute efficient.
* [M2M (Machine-to-Machine) Authentication Flow](/concepts/machine-to-machine-m2m-authentication.md) - Authentication flow for automated workloads using service principals registered in the recipient's IdP, using the OAuth Client Credentials Grant flow with no user interaction required.
* [Machine-to-Machine (M2M) Authentication](/concepts/machine-to-machine-m2m-authentication.md) - Authentication flow for automated workloads where a service principal registered in the recipient's IdP enables applications or scripts to access shared data programmatically using the OAuth Client Credentials Grant flow without user interaction.
* [Machine-to-Machine (M2M) Authentication Flow](/concepts/machine-to-machine-m2m-authentication.md) - OIDC authentication flow for automated workloads where a service principal registered in the IdP programmatically accesses shared data using the OAuth Client Credentials Grant, with no user interaction required.
* [MANAGE ALLOWLIST Privilege](/concepts/manage-allowlist-privilege.md) - A Unity Catalog privilege required to add or remove items from the allowlist; recommended to be granted only to metastore admins and trusted platform administrators.
* [MANAGE Permission in Unity Catalog](/concepts/manage-permission-in-unity-catalog.md) - A privilege required for all policy operations (create, edit, delete, describe) on securable objects in Unity Catalog, distinct from ownership but an alternative requirement.
* [MANAGE Permission Requirement](/concepts/manage-permission-requirement.md) - All ABAC policy operations (create, edit, delete, show, describe) require MANAGE permission on the securable object or object ownership, ensuring only authorized users can modify data access rules.
* [MANAGE Privilege](/concepts/manage-privilege.md) - A special privilege in Unity Catalog that allows users to manage (grant/revoke) privileges on an object without owning it, though currently MANAGE users cannot see all grants via INFORMATION_SCHEMA.
* [MANAGE Privilege for Data Quality Monitoring Alerts](/concepts/manage-privilege-for-data-quality-monitoring-alerts.md) - The MANAGE privilege on a schema or catalog is required to create or manage alert rules for data quality monitoring in Unity Catalog.
* [MANAGE privilege in Unity Catalog](/concepts/manage-privilege-in-unity-catalog.md) - A Unity Catalog privilege that allows users to assign/revoke privileges, transfer ownership, and delete an object without being the owner
* [Managed and Foreign Iceberg Table Sharing](/concepts/foreign-iceberg-table-sharing.md) - Sharing managed Iceberg tables (native to Databricks) and foreign Iceberg tables (federated from external Iceberg catalogs) with Delta Uniform support, history sharing, and specific limitations.
* [Managed assets in Unity Catalog](/concepts/managed-assets-in-unity-catalog.md) - Data assets (tables and volumes) where Unity Catalog controls both governance and the underlying file storage lifecycle
* [Managed Iceberg Table Clone Limitations](/concepts/managed-iceberg-table-cloning.md) - Managed Iceberg tables support only deep cloning and cannot be shallow cloned or have their table format changed during cloning.
* [Managed Iceberg Table Clone Restrictions](/concepts/managed-iceberg-table-cloning.md) - Managed Iceberg tables support only deep cloning (not shallow), and the table format cannot be changed during cloning.
* [Managed Iceberg Table Cloning](/concepts/managed-iceberg-table-cloning.md) - Managed Iceberg tables on Databricks support only deep cloning, not shallow cloning, and the table format cannot be changed during the clone operation.
* [Managed MLflow on Databricks](/concepts/managed-mlflow-on-databricks.md) - Enterprise extension of open source MLflow with governance, managed hosting, production scaling, and integration with Databricks lakehouse and Unity Catalog
* [Managed Storage for Streaming Tables](/concepts/managed-storage-for-streaming-tables.md) - Databricks-managed storage location for streaming tables, where users cannot specify a custom LOCATION during deep clone
* [Managed Storage in Unity Catalog](/concepts/managed-storage-in-unity-catalog.md) - The S3 bucket storage location, configurable at metastore, catalog, or schema level, where Unity Catalog stores managed tables and managed volumes.
* [Managed storage location](/concepts/managed-storage-location.md) - The storage location within a user's cloud account that Unity Catalog controls for organizing, optimizing, and deleting data files of managed assets
* [Managed table requirement for Uniform Iceberg](/concepts/managed-tables-requirement-for-icebergcompat.md) - The IcebergCompatV feature can only be enabled on Managed Tables in Databricks, not on external or unmanaged tables.
* [Managed Tables in Databricks](/concepts/managed-tables-in-databricks.md) - A table type requirement for enabling IcebergCompatV; the feature can only be enabled on managed tables, not external or unmanaged tables.
* [Managed Tables in Unity Catalog](/concepts/managed-tables-in-unity-catalog.md) - Tables whose lifecycle, file layout, storage location, and performance optimization are fully managed by Unity Catalog; always use Delta format and reside in reserved managed storage.
* [Managed Tables requirement for IcebergCompat](/concepts/managed-tables-requirement-for-icebergcompat.md) - IcebergCompatV can only be enabled on managed Delta tables, not external or unmanaged tables.
* [Managed to External Table Rollback](/concepts/managed-to-external-table-rollback.md) - The operation of reverting a Unity Catalog managed table back to an external table, which is time-limited and has specific failure modes.
* [Managed to External Table Rollback in Delta Lake](/concepts/managed-to-external-table-rollback.md) - The operation of converting a Unity Catalog managed table back to an external table via ALTER TABLE UNSET MANAGED, which has constraints and failure modes.
* [Managed to External Table Rollback in Unity Catalog](/concepts/managed-to-external-table-rollback.md) - The operation of reverting a Unity Catalog managed table back to an external table, which may fail under certain conditions
* [Managed vs External Assets in Unity Catalog](/concepts/managed-vs-external-assets-in-unity-catalog.md) - Fundamental distinction between managed objects (Unity Catalog controls storage lifecycle) and external objects (user controls storage and lifecycle) in Unity Catalog.
* [Managed vs External tables in Unity Catalog](/concepts/managed-vs-external-tables-in-unity-catalog.md) - Distinction between tables where Unity Catalog controls storage location and file lifecycle versus tables where the user specifies the storage location
* [Managed vs External volumes in Unity Catalog](/concepts/managed-vs-external-volumes-in-unity-catalog.md) - Distinction between volumes where Unity Catalog determines the storage location and manages file lifecycle versus volumes where the user specifies the storage location
* [Managing MLflow Experiments: Permissions, Rename, Delete](/concepts/managing-mlflow-experiments-permissions-rename-delete.md) - Experiment owners can rename, delete, or change permissions from either the Experiments page or the experiment details page. Notebook experiments are tied to notebooks — deleting one deletes both. Git-folder notebook experiments must be managed at the Git folder level.
* [Managing MLflow Runs](/concepts/managing-mlflow-runs.md) - Capabilities to delete, compare, and tag MLflow runs from the chart view interface for experiment organization.
* [Managing Reasoning Across Multi-Turn Conversations](/concepts/managing-reasoning-across-multi-turn-conversations.md) - Technique for handling reasoning blocks in multi-turn conversations where only the last assistant turn's reasoning is visible to the model; optionally omitting or including reasoning blocks depending on whether the model needs to reason over prior steps.
* [Manual Delta log tampering](/concepts/manual-delta-log-tampering.md) - Manually deleting or removing files from the Delta transaction log directory can corrupt the table by creating version gaps.
* [Manual Iceberg Metadata Conversion (MSCK REPAIR TABLE SYNC METADATA)](/concepts/manual-iceberg-metadata-conversion-msck-repair-table-sync-metadata.md) - A synchronous operation that manually triggers Iceberg metadata generation for the latest Delta Lake table version, used when automatic generation fails due to cluster termination, errors, or writes from non-UniForm clients.
* [Manual Library Installation on Databricks](/concepts/manual-library-installation-on-databricks.md) - The fallback option to continue using a library after it has been removed from Databricks Runtime ML by installing it manually or using an earlier runtime version.
* [Manual MLflow Logging with Hyperopt Trials](/concepts/manual-mlflow-logging-with-hyperopt-trials.md) - When using the Trials class with distributed training algorithms on Databricks, automatic MLflow logging is not supported; users must manually call MLflow to log trials.
* [Manual OAuth token fetch with authorization_details](/concepts/manual-oauth-token-fetch-with-authorization_details.md) - The process of manually requesting an OAuth token from the OIDC token endpoint with an authorization_details payload specifying the endpoint ID, object type, and action permission.
* [Manual Span Instrumentation in LangGraph Nodes](/concepts/manual-span-instrumentation-in-langgraph-nodes.md) - Adding child spans inside LangGraph nodes or tools using mlflow.start_span() to gain detailed per-step insights
* [Manual Tracing](/concepts/manual-tracing.md) - An MLflow tracing approach designed for custom logic and complex workflows, offering Function Decorator APIs and low-level APIs for fine-grained control over what gets traced.
* [Manual Tracing (MLflow)](/concepts/manual-tracing.md) - Approach for tracing custom logic and complex workflows using Function Decorator APIs or low-level APIs for fine-grained control.
* [Manual Tracing APIs](/concepts/manual-tracing-apis.md) - Four manual tracing approaches: trace() for functions, @trace decorator for class methods, withSpan() for code blocks, and explicit startSpan/endSpan for full lifecycle control.
* [Manual Tracing for MLflow](/concepts/manual-tracing.md) - Manual approach to tracing designed for custom logic and complex workflows using function decorators or low-level APIs
* [Manual UDF Dependency Management with withDependencies](/concepts/manual-udf-dependency-management-with-withdependencies.md) - Databricks Connect allows explicit specification of Python dependencies for UDFs via the withDependencies() API, supporting PyPI packages, Unity Catalog volumes, and local packages.
* [Mapping Table Pattern for Access Control](/concepts/mapping-table-pattern-for-access-control.md) - A single lookup table that drives both row-level filtering and column-level masking, replacing complex group-based approaches with row updates for access changes.
* [Mapping Tables for Access Control](/concepts/mapping-tables-for-access-control.md) - A pattern using dedicated mapping tables (access-control lists) to encode which users or groups can access specific rows, integrated via row filters that join against the mapping table.
* [Marian Transformer for Machine Translation on Databricks](/concepts/marian-transformer-for-machine-translation-on-databricks.md) - A practical example of using Spark NLP's Marian Transformer for batch machine translation on Databricks clusters, including pipeline construction and inference.
* [MATCH COLUMNS with AND conditions](/concepts/match-columns-with-and-conditions.md) - Using logical AND inside MATCH COLUMNS to target columns that satisfy multiple tag criteria simultaneously, enabling fine-grained policy targeting across tag dimensions.
* [Materialization Trigger Types](/concepts/materialization-trigger-types.md) - CronSchedule for periodic materialization of aggregation features and TableTrigger for incremental materialization of column-selection features
* [Materialization Triggers](/concepts/materialization-triggers.md) - CronSchedule (for aggregation features, scheduled via Quartz cron) and TableTrigger (for ColumnSelection features, triggered on upstream Delta table commits) control when materialization pipelines run.
* [materialize_features() API](/concepts/materialize_features-api.md) - API function that materializes a list of registered declarative features into offline Delta tables and/or an Online Feature Store, controlled by a trigger (CronSchedule or TableTrigger).
* [Materialized Feature Pipeline Scheduling](/concepts/materialized-feature-pipeline-scheduling.md) - Mechanisms for scheduling feature materialization jobs using CronSchedule (time-based, for aggregation features) and TableTrigger (event-driven, for column selection features), with support for both offline and online store configurations.
* [Materialized Features (Declarative)](/concepts/materialized-features-declarative.md) - Precomputed feature representations in Unity Catalog produced from declarative feature definitions via Lakeflow Spark Declarative Pipelines, used for model training, batch scoring, and online serving.
* [Materialized views in Databricks](/concepts/materialized-views-in-databricks.md) - Pre-computed views that refresh on a schedule or on demand, one of the two table types supported for external metadata.
* [MaterializedFeaturePipelineSchedule](/concepts/materializedfeaturepipelineschedule.md) - Configuration for scheduling feature materialization pipelines using CronSchedule (for aggregation features) or TableTrigger (for column selection features), with ACTIVE/PAUSED state management.
* [Maven Coordinate Allowlisting](/concepts/maven-coordinate-allowlisting.md) - The process of adding Maven artifacts to the Unity Catalog allowlist using groupId:artifactId:version format, supporting wildcard patterns for all versions or all artifacts in a group.
* [Maven Coordinates in Allowlist](/concepts/maven-coordinate-allowlisting.md) - The ability to add Maven coordinates to the Unity Catalog allowlist using groupId:artifactId:version format, with support for wildcards to include all versions or all artifacts in a group.
* [MCP Client Configuration for Databricks](/concepts/mcp-client-configuration-for-databricks.md) - Configuration patterns for connecting MCP-compatible clients (VS Code, Cursor, Claude Desktop) to the MLflow MCP server with Databricks authentication
* [MCP Server Governance](/concepts/mcp-server-governance.md) - Managing access to managed, external, and custom MCP (Model Context Protocol) servers alongside LLM endpoints through the Unity AI Gateway.
* [MCP server governance in Unity AI Gateway](/concepts/mcp-server-governance.md) - Management of access to managed, external, and custom MCP (Model Context Protocol) servers alongside LLM endpoints through Unity AI Gateway.
* [MDS (Mosaic Data Shard) Format](/concepts/mds-mosaic-data-shard-format.md) - A storage format produced by converting Apache Spark DataFrames using Mosaic Streaming, optimized for distributed deep learning training with support for compression and hashing.
* [Medallion Architecture](/concepts/medallion-architecture.md) - A data organization pattern that structures data into bronze (raw), silver (refined), and gold (business-ready) layers to progressively improve data quality as it flows through a lakehouse pipeline.
* [MemAlign Optimizer](/concepts/memalign-optimizer.md) - The default alignment optimizer used by MLflow's judge alignment system, automatically applied when calling align() without specifying an optimizer.
* [Memory Efficiency in Distributed Training](/concepts/memory-efficiency-in-distributed-training.md) - The comparative memory efficiency of FSDP over DDP as a key consideration for choosing training strategies
* [MERGE INTO (Delta Lake)](/concepts/merge-into-delta-lake.md) - Alternative SQL statement used instead of unsupported UPDATE...FROM...JOIN syntax to update a table based on a join with another table or subquery.
* [Merge/Update/Delete restrictions on archived Delta data](/concepts/mergeupdatedelete-restrictions-on-archived-delta-data.md) - MERGE, UPDATE, and DELETE operations fail if they impact data in archived files; files must be restored first.
* [mergeSchema Option](/concepts/mergeschema-option.md) - A Spark/Delta Lake DataFrameWriter option that enables automatic schema migration by merging the provided schema with the existing table schema during writes.
* [mergeSchema Option Pattern](/concepts/mergeschema-option.md) - A Delta Lake DataFrameWriter option (.option('mergeSchema', 'true')) used to enable automatic schema migration when writing, allowing new columns to be added to the table.
* [Metadata Column Access in Unity Catalog](/concepts/metadata-column-access-in-unity-catalog.md) - Using _metadata.file_name and _metadata.file_path as an alternative to input_file_name() for shared clusters in Unity Catalog.
* [Metadata Immutability in MLflow Traces](/concepts/metadata-immutability-in-mlflow-traces.md) - The constraint that trace metadata is immutable once set — attempting to update an existing metadata key is silently ignored.
* [METADATA ONLY mode](/concepts/metadata-only-mode.md) - FSCK REPAIR TABLE mode that only checks metadata integrity without verifying actual data file contents
* [METADATA ONLY verification](/concepts/metadata-only-verification.md) - A verification mode for FSCK REPAIR TABLE that checks only the Delta transaction log metadata (checkpoint CRCs, file listings) without examining actual data file contents.
* [Metadata Quality and Discoverability](/concepts/metadata-quality-and-discoverability.md) - The principle that the quality of admin-side metadata directly determines how useful the data discovery experience is for users.
* [Metadata Quality and Discovery Experience](/concepts/metadata-quality-and-discovery-experience.md) - The quality of admin-side metadata directly determines how useful the discovery experience is for users; consistent certification standards and governed tag vocabularies enable confident filtering and searching.
* [Metadata upgrade scope](/concepts/metadata-upgrade-scope.md) - The set of feature table metadata upgraded to Unity Catalog: primary keys, time series columns, comments, tags, and notebook/job lineage.
* [Metadata vs Tags Immutability in MLflow Traces](/concepts/metadata-vs-tags-immutability-in-mlflow-traces.md) - A key distinction: once a trace is logged, metadata fields are immutable (ideal for stable identifiers like user and session IDs), while tags remain mutable (suitable for updatable annotations).
* [METADATA_CLEANUP_ERROR](/concepts/metadata_cleanup_error.md) - A specific sub-error under DELTA_ALTER_TABLE_SET_MANAGED_TABLE_NOT_MIGRATABLE indicating failure to create a checkpoint or clean up old metadata files before migration.
* [METADATA_CLEANUP_ERROR sub-error](/concepts/metadata_cleanup_error.md) - A specific sub-error of DELTA_ALTER_TABLE_SET_MANAGED_TABLE_NOT_MIGRATABLE indicating failure to create checkpoint or clean up old metadata files before migrating the table.
* [Metastore](/concepts/metastore.md) - The top-level securable object in Unity Catalog that contains all registered objects (catalogs, credentials, shares, connections, etc.) within a single cloud region.
* [Metastore Admin (Databricks)](/concepts/metastore-admin-databricks.md) - Privileged role required to create, edit, or delete custom classifiers in Databricks Unity Catalog data classification.
* [Metastore Admin (Unity Catalog)](/concepts/metastore-admin-role.md) - An optional role governing data access, ownership, and top-level Unity Catalog securable objects within a single metastore, with privileges from both the role and ownership of the metastore.
* [Metastore admin ownership privileges](/concepts/metastore-admin-ownership-privileges.md) - Default and ownership-based privileges granted to metastore admins as owners of the metastore, including the ability to manage any object in the metastore.
* [Metastore Admin Role](/concepts/metastore-admin-role.md) - An optional, highly privileged Unity Catalog role that governs data access, ownership, and top-level securable objects; required for specific actions like changing ownership or removing default workspace admin permissions.
* [Metastore Admin Role (Unity Catalog)](/concepts/metastore-admin-role.md) - An optional but highly privileged Unity Catalog role that governs data access, ownership, and top-level securable objects within a metastore.
* [Metastore admin role assignment](/concepts/metastore-admin-role-assignment.md) - Process and considerations for assigning the metastore admin role via the account console, with recommendations to use groups and awareness of caching delays.
* [Metastore Admin Role in Unity Catalog](/concepts/metastore-admin-role-in-unity-catalog.md) - The administrative role automatically assigned to the creator of a metastore, with privileges to create catalogs and manage access, and a recommendation to reassign it to a group.
* [Metastore admins (Unity Catalog)](/concepts/metastore-admin-role.md) - Optional but highly privileged role governing data access, ownership, and top-level Unity Catalog securable objects within a single metastore.
* [Metastore Admins in Unity Catalog](/concepts/metastore-admin-role.md) - Optional but highly privileged role governing data access, ownership, and top-level Unity Catalog securable objects within a single metastore; needed for operations like changing ownership or removing default workspace admin permissions.
* [Metastore Connection Limit for SAP BDC](/concepts/metastore-connection-limit-for-sap-bdc.md) - A maximum of five SAP BDC connections can exist per Unity Catalog metastore.
* [Metastore Connection Limits](/concepts/metastore-connection-limits.md) - A limitation restricting the number of SAP BDC connections to five per metastore in Databricks
* [Metastore Connection Limits for SAP BDC](/concepts/metastore-connection-limit-for-sap-bdc.md) - A limit of five SAP BDC connections per Unity Catalog metastore, enforced as a hard upper bound.
* [Metastore Metadata Mismatch](/concepts/metastore-metadata-mismatch.md) - An error (convertMetastoreMetadataMismatchException) thrown when the catalog metadata differs from the Delta Lake transaction log metadata during conversion, which can be overridden with a configuration flag.
* [Metastore Metadata Mismatch in Delta Conversion](/concepts/metastore-metadata-mismatch.md) - An error condition (convertMetastoreMetadataMismatchException) that occurs when catalog metadata differs from the Delta Lake transaction log metadata during or after CONVERT TO DELTA.
* [Metastore Recipient Token Lifetime Configuration](/concepts/metastore-recipient-token-lifetime-configuration.md) - The default token lifetime setting for a Unity Catalog metastore that applies to new recipients when no explicit token lifetime is set, configurable only by account admins.
* [Metastore Storage Root Removal](/concepts/metastore-storage-root-removal.md) - The process of removing the metastore-level storage root to enforce catalog- or schema-level data isolation, which pushes the root location down to existing catalogs and requires new catalogs to specify their own storage location.
* [Metastore-Level Connection Limit](/concepts/metastore-level-connection-limit.md) - A single Unity Catalog metastore can have at most five SAP BDC connections.
* [Metastore-level managed storage](/concepts/metastore-level-managed-storage.md) - An optional S3 bucket used as the default storage location for managed tables and managed volumes in a Unity Catalog metastore, which can be overridden at catalog and schema levels.
* [Metastore-Level Quality Dashboard](/concepts/metastore-level-quality-dashboard.md) - A downloadable Lakeview dashboard template that provides a workspace-wide view of all data quality results across the metastore using the system.data_quality_monitoring.table_results table.
* [Metastore-Level Scoping of Custom Classifiers](/concepts/metastore-level-scoping-of-custom-classifiers.md) - Custom classifiers apply to all catalogs in the metastore that have Data Classification enabled; per-catalog or per-schema scoping is not supported, making this a metastore-wide governance mechanism.
* [Metastore-level SecureConnect enrollment](/concepts/metastore-level-secureconnect-enrollment.md) - A metastore administrator can configure a metastore so that all new recipients automatically use SecureConnect, controlled via Catalog Explorer settings.
* [Methods for Creating Model Serving Endpoints](/concepts/methods-for-creating-model-serving-endpoints.md) - Databricks Model Serving supports four creation methods: Serving UI, REST API, MLflow Deployments SDK, and Workspace Client.
* [Metric Backfill](/concepts/metric-backfill.md) - backfill_scorers() function to evaluate historical traces by applying scorers to past time windows, returning a job ID for tracking
* [Metric naming behavior](/concepts/metric-naming-behavior.md) - Rules for how scorer metric names are derived from function names, Feedback.name fields, and Scorer.name fields
* [Metric naming behavior in scorers](/concepts/metric-naming-behavior.md) - Rules for how metric names are derived from scorers: Feedback.name takes precedence over function/Scorer.name, and all metrics must have distinct names.
* [Metric naming conventions for MLflow scorers](/concepts/metric-naming-behavior.md) - Rules for how evaluation metric names are derived from scorers: function name for decorator-based scorers, Feedback.name for named feedback, class name for class-based scorers, with unique names required when returning multiple Feedback objects.
* [Metric naming conventions for scorers](/concepts/metric-naming-behavior.md) - Metric names in evaluation results are determined by Feedback.name fields (if present) or the scorer function/class name, with distinct naming requirements.
* [Metric naming in MLflow scorers](/concepts/metric-naming-behavior.md) - Rules for how metric names are derived from scorers, with Feedback.name fields taking precedence over function/class names, requiring distinct names for multiple feedbacks.
* [Metric Slicing Expressions](/concepts/metric-slicing-expressions.md) - A technique to define subsets of a table for targeted profiling, generating separate slices for each predicate and its complement, such as 'col_2 > 10' or unique values in a column.
* [Metric Table Location and Querying](/concepts/metric-table-location-and-querying.md) - Convention for naming and locating profile and drift metric tables using output schema and table name, along with direct SQL querying patterns.
* [Metric Tables (Databricks Data Profiling)](/concepts/metric-tables-databricks-data-profiling.md) - Unity Catalog tables created by data profiles that store computed statistics; can be queried via notebooks, SQL query explorer, or Catalog Explorer.
* [Metric Tables and Profile Outputs](/concepts/metric-tables-and-profile-outputs.md) - Unity Catalog tables automatically created by data profiling to store computed metrics, which can be queried via SQL, notebooks, or Catalog Explorer and controlled via Unity Catalog privileges.
* [Metric tables vs dashboard separation](/concepts/metric-tables-vs-dashboard-separation.md) - The architectural separation where profile updates write to metric tables, but the dashboard runs separate SQL queries over those tables, requiring independent refresh operations.
* [Metrics and Performance Analysis for GenAI Agents](/concepts/metrics-and-performance-analysis-for-genai-agents.md) - The ability to compute latency percentiles (P50/P95/P99), track error rates and throughput over time, analyze token usage patterns and costs, and compare performance across different time periods or filters.
* [Metrics and Performance Analysis via Genie Code](/concepts/metrics-and-performance-analysis-via-genie-code.md) - Genie Code capability to compute latency percentiles (P50/P95/P99), track error rates and throughput over time, analyze token usage patterns and costs, and compare performance across time periods or filters.
* [Migrating existing workspaces to Unity Catalog](/concepts/migrating-existing-workspaces-to-unity-catalog.md) - The recommended process of upgrading existing Databricks workspaces to Unity Catalog, including migrating Hive tables, DBFS files, notebooks, jobs, compute access modes, and Runtime versions.
* [Migrating External Distributed Training to Databricks](/concepts/migrating-external-distributed-training-to-databricks.md) - Process to import an external training repository as a Databricks Git folder and launch distributed training via TorchDistributor
* [Migrating from databricks-feature-store to databricks-feature-engineering](/concepts/migrating-from-databricks-feature-store-to-databricks-feature-engineering.md) - The migration path from the deprecated legacy package to the new unified package, requiring only a pip install change without code modifications.
* [Migrating from MLflow 2.x to MLflow 3](/concepts/migrating-from-mlflow-2x-to-mlflow-3.md) - Key differences and migration guide between MLflow 2.x and 3.0, including changes to model logging, artifact storage, and model registry.
* [Migrating LLM Serving Endpoints](/concepts/migrating-llm-serving-endpoints.md) - The process of updating existing Databricks LLM serving endpoints to use provisioned throughput by selecting a new model version, which automatically switches the UI to the new experience.
* [Migrating Single-Node to Distributed Deep Learning with Horovod](/concepts/migrating-single-node-to-distributed-deep-learning-with-horovod.md) - A six-step workflow for converting single-node deep learning code (TensorFlow, Keras, PyTorch) to distributed training using Horovod, then wrapping it for HorovodRunner.
* [Migration from Databricks Agent Evaluation to MLflow 3](/concepts/migration-from-databricks-agent-evaluation-to-mlflow-3.md) - The process and code changes required to migrate from the legacy databricks.agents.evals API to the new mlflow.genai API.
* [Migration from databricks-feature-store to databricks-feature-engineering](/concepts/migration-from-databricks-feature-store-to-databricks-feature-engineering.md) - The process and compatibility notes for migrating from the deprecated databricks-feature-store package to the new databricks-feature-engineering package, with backward compatibility for import statements.
* [Migration from External Repositories to Databricks](/concepts/migration-of-external-repositories-to-databricks.md) - Process of importing existing distributed training code from external repos into Databricks Git folders and launching it with TorchDistributor
* [Migration from sparkdl.xgboost to xgboost.spark](/concepts/migration-from-sparkdlxgboost-to-xgboostspark.md) - Guide for replacing deprecated sparkdl.xgboost estimators with xgboost.spark equivalents, including parameter name changes (camelCase to snake_case) and removal of use_external_storage.
* [Migration of Databricks Connect from Runtime 12.2 LTS to 13.3 LTS and above](/concepts/migration-of-databricks-connect-from-runtime-122-lts-to-133-lts-and-above.md) - The process of upgrading a Python project or environment from Databricks Connect for Databricks Runtime 12.2 LTS and below to 13.3 LTS and above.
* [Migration of External Repositories to Databricks](/concepts/migration-of-external-repositories-to-databricks.md) - Process for importing external distributed training repositories as Databricks Git folders and launching them via TorchDistributor.
* [MirroredStrategyRunner](/concepts/mirroredstrategyrunner.md) - Spark-based runner for TensorFlow MirroredStrategy that coordinates distributed training across Spark executors
* [Missing data files and deletion vectors](/concepts/missing-data-files-and-deletion-vectors.md) - Detection of referenced data files or deletion vectors that are missing from storage in Delta Lake tables
* [Missing value handling in on-demand features](/concepts/missing-value-handling-in-on-demand-features.md) - Difference in how missing feature lookup values are returned depending on the environment: None for batch scoring vs float('nan') for online serving.
* [Missing Value Imputation](/concepts/missing-value-imputation.md) - Configurable strategies for imputing null values in AutoML, including default auto-detection and user-specified methods via UI or API.
* [Missing Value Imputation for Forecasting](/concepts/missing-value-imputation-for-forecasting.md) - AutoML allows specifying imputation methods for null values per column; defaults are chosen by column type and content, and custom imputation disables semantic type detection.
* [Missing Value Imputation in AutoML](/concepts/missing-value-imputation-in-automl.md) - Configurable strategies for handling null values in forecasting data, including default automatic selection based on column type and content.
* [Missing Value Imputation in AutoML Forecasting](/concepts/missing-value-imputation-for-forecasting.md) - How Databricks AutoML handles null values in forecasting datasets, including configurable imputation methods via UI or API, with automatic selection based on column type and content.
* [Missing Value Semantics in Spark XGBoost](/concepts/missing-value-semantics-in-spark-xgboost.md) - Difference in how the 'missing' parameter is handled in sparkdl.xgboost vs standard xgboost, particularly regarding sparse matrix zero values not being treated as missing by default.
* [MISSING_REQUIRED_TABLE_FEATURE](/concepts/missing_required_table_feature.md) - Error sub-type indicating a required table feature for IcebergCompatV1 is not supported or enabled
* [MISSING_REQUIRED_TABLE_FEATURE (Delta Iceberg error)](/concepts/missing_required_table_feature.md) - A sub-error of DELTA_ICEBERG_COMPAT_V1_VIOLATION that occurs when a required table feature for IcebergCompatV1 is not enabled
* [MISSING_UNIFORM_TBL_PROPERTIES](/concepts/missing_uniform_tbl_properties.md) - A sub-error of DELTA_UNIFORM_ICEBERG_INGRESS_VIOLATION indicating required Delta table properties (tableId, snapshotId, metadataLocation) are missing, possibly due to manual changes to the _delta_log.
* [MISSING_UNIFORM_TBL_PROPERTIES error](/concepts/missing_uniform_tbl_properties.md) - A sub-error of DELTA_UNIFORM_ICEBERG_INGRESS_VIOLATION indicating that one or more required Delta Uniform properties (tableId, snapshotId, metadataLocation) are missing from the Delta table properties, possibly due to manual changes to _delta_log.
* [Mistral AI Python SDK](/concepts/mistral-ai-python-sdk.md) - The official Mistral AI Python client library (mistralai) that provides a chat completion API for interacting with Mistral's large language models.
* [Mixture-of-Experts (MoE) Architecture](/concepts/mixture-of-experts-moe-architecture.md) - A neural network architecture where only a subset of parameters (e.g., 10B active out of 122B total) are activated per inference, enabling compute-efficient reasoning and coding, exemplified by models like Qwen3.5.
* [ML Code as Templates](/concepts/ml-code-as-templates.md) - Standardized notebook templates for ML project components (training, batch inference, testing) created by MLOps Stacks to unify project structure and enforce modularized, test-ready code.
* [ML Environment Stages (Dev/Staging/Prod)](/concepts/ml-environment-stages-devstagingprod.md) - The separation of ML workloads into development, staging, and production environments, each corresponding to a Unity Catalog catalog, with distinct purposes for model development, integration testing, and final serving.
* [ML Lifecycle](/concepts/ml-lifecycle.md) - The end-to-end journey for taking an ML project from initial scoping to production, encompassing eight stages: scoping, data exploration, data preparation, model training, evaluation, staging, deployment, and monitoring.
* [ML Lifecycle on Databricks](/concepts/ml-lifecycle-on-databricks.md) - The end-to-end journey from raw data to production model, covering eight key stages: scoping, EDA, data/feature preparation, training, evaluation, registration, deployment, and monitoring/retraining.
* [ML Model Monitoring and Automated Retraining](/concepts/ml-model-monitoring-and-automated-retraining.md) - The practice of profiling input data and model predictions for drift and performance, publishing metrics to dashboards, setting up alerts, and triggering automated retraining cycles via scheduled or event-driven jobs.
* [ML Model Performance Profiling](/concepts/ml-model-performance-profiling.md) - Using Databricks data profiling on inference tables to track ML model inputs, predictions, performance trends, and drift over time.
* [ML Platform](/concepts/ml-platform.md) - The combined infrastructure, tooling, and governance layer that supports the full ML lifecycle on Databricks
* [ML Platform (Databricks)](/concepts/ml-platform.md) - Combined infrastructure, tooling, and governance layer supporting the full ML lifecycle — including data assets, experimentation tools, training infrastructure, deployment/monitoring infrastructure, and MLOps/governance tools.
* [ML vs Deep Learning vs GenAI Distinctions](/concepts/ml-vs-deep-learning-vs-genai-distinctions.md) - The overlapping boundaries between classical machine learning, deep learning, and generative AI, with platform features like Model Serving, ai_query, AI Runtime, MLflow, and AI Search supporting all three paradigms.
* [ML vs Deep Learning vs GenAI on Databricks](/concepts/ml-vs-deep-learning-vs-genai-on-databricks.md) - The relationship and shared platform support between classic ML, deep learning, and generative AI paradigms on Databricks
* [MLflow](/concepts/mlflow.md) - An open source platform for managing the end-to-end machine learning lifecycle, supporting tracking for model tuning in Python, R, and Scala, and used as the tracking backend for MLlib automated logging.
* [MLflow 3](/concepts/mlflow-3.md) - The latest major version of MLflow on Databricks, delivering observability, evaluation, prompt management, and new concepts like Logged Models and Deployment Jobs.
* [MLflow 3 Agent Evaluation SDK](/concepts/mlflow-3-managed-evaluation.md) - The `mlflow[databricks]>=3.1` SDK that provides Agent Evaluation methods as part of the managed MLflow 3 platform.
* [MLflow 3 API Changes and Deprecations](/concepts/mlflow-3-api-changes-and-deprecations.md) - Notable API changes in MLflow 3 including the deprecation of artifact_path (replaced by name), removal of run_uuid (use run_id), and deprecation of mlflow.evaluate in favor of mlflow.models.evaluate.
* [MLflow 3 Artifact Storage Architecture](/concepts/mlflow-3-artifact-storage-architecture.md) - In MLflow 3, model artifacts are stored under a dedicated model artifact path (models/<model_id>/artifacts/) rather than under the run's artifact path, changing how models are loaded and referenced.
* [MLflow 3 Deep Learning Workflow](/concepts/mlflow-3-deep-learning-workflow.md) - A workflow for training deep learning models with PyTorch on Databricks, using MLflow 3 to log checkpoints, track metrics, and register models to Unity Catalog.
* [MLflow 3 Dependency for GenAI Tracing](/concepts/mlflow-3-dependency-for-genai-tracing.md) - Requirement for MLflow 3.x (specifically mlflow[databricks]>=3.1) to access the latest GenAI tracing capabilities, including automatic and manual tracing features.
* [MLflow 3 deployment job](/concepts/mlflow-3-deployment-job.md) - A Databricks job that evaluates and deploys model versions, creating additional metrics viewable on the Catalog Explorer model version page for approver review before deployment approval.
* [MLflow 3 Deployment Jobs](/concepts/mlflow-3-deployment-jobs.md) - Lakeflow Jobs-based workflows that manage the model lifecycle including evaluation, approval, and deployment steps, governed by Unity Catalog with full activity logging.
* [MLflow 3 Evaluation Data Schema](/concepts/mlflow-3-evaluation-data-schema.md) - MLflow 3 uses a new data field mapping: inputs (replaces request), outputs (replaces response), and expectations (replaces expected_response, expected_facts, etc.), organized as structured dicts within each evaluation row.
* [MLflow 3 Experiment Page Models Tab](/concepts/mlflow-3-experiment-page-models-tab.md) - A UI view in MLflow 3 experiments that displays all logged checkpoint models from a run, with metrics, parameters, and visualization tools for model comparison.
* [MLflow 3 for GenAI](/concepts/mlflow-3-for-genai.md) - A comprehensive set of features in MLflow 3 for generative AI application development including tracing, evaluation with LLM judges, human feedback collection, and a centralized Prompt Registry.
* [MLflow 3 for Models](/concepts/mlflow-3-for-models.md) - State-of-the-art experiment tracking, performance evaluation, and production management for traditional machine learning and deep learning models on Databricks.
* [MLflow 3 GenAI Evaluation API](/concepts/mlflow-3-genai-evaluation-api.md) - The new mlflow.genai.evaluate() API that replaces mlflow.evaluate() for agent and LLM evaluation, with cleaner parameter names and explicit scorer selection.
* [MLflow 3 GenAI Module](/concepts/mlflow-3-genai-module.md) - The new mlflow.genai package that unifies GenAI evaluation functionality, replacing the separate Databricks Agent Evaluation API.
* [MLflow 3 GenAI Tracing Platform](/concepts/mlflow-3-genai-tracing-platform.md) - MLflow 3 provides the recommended platform for GenAI tracing with capabilities for automatic and manual tracing of LLM calls, spans, and multi-step workflows on Databricks.
* [MLflow 3 Guidelines and ExpectationsGuidelines Scorers](/concepts/mlflow-3-guidelines-and-expectationsguidelines-scorers.md) - Global guidelines from MLflow 2.x are replaced by explicit Guidelines() scorer instances, and per-example guidelines use the ExpectationsGuidelines scorer with expectations.guidelines fields in the evaluation data.
* [MLflow 3 Human Feedback and Labeling](/concepts/mlflow-3-human-feedback-and-labeling.md) - The human feedback system in MLflow 3 moves from databricks.agents.review_app to mlflow.genai.labeling and mlflow.genai.label_schemas, with explicit schema creation and new dataset management via mlflow.genai.datasets.
* [MLflow 3 Installation](/concepts/mlflow-3-installation.md) - Instructions for installing MLflow 3 by upgrading to version 3.0 or above using pip.
* [MLflow 3 LoggedModel](/concepts/mlflow-3-loggedmodel.md) - A new first-class object in MLflow 3 that elevates models from run artifacts to dedicated tracked entities, capturing metrics, parameters, and traces across multiple development phases and environments.
* [MLflow 3 Managed Evaluation](/concepts/mlflow-3-managed-evaluation.md) - Managed Agent Evaluation integrated with MLflow 3 using the mlflow[databricks]>=3.1 SDK, superseding MLflow 2 Agent Evaluation.
* [MLflow 3 Migration from MLflow 2.x](/concepts/mlflow-3-migration-from-mlflow-2x.md) - Key changes when migrating from MLflow 2.x to 3.0 including new log_model API, artifact storage location changes, default Unity Catalog registry, and deprecated APIs.
* [MLflow 3 Model Artifact Storage](/concepts/mlflow-3-model-artifact-storage.md) - Change in storage location for model artifacts from run-based paths to model-based paths under experiments.
* [MLflow 3 Model Object](/concepts/mlflow-3-model-object.md) - In MLflow 3, models are first-class objects distinct from run artifacts, with code snippets for loading and using them for predictions on Spark and Pandas DataFrames.
* [MLflow 3 Model Registry](/concepts/mlflow-3-model-registry.md) - Enhanced model registry in MLflow 3 that exposes metrics and parameters for registered models, with unified cross-experiment and cross-workspace visibility.
* [MLflow 3 Model Registry in Unity Catalog](/concepts/mlflow-3-model-registry-in-unity-catalog.md) - MLflow 3 defaults to the Unity Catalog-backed Model Registry (databricks-uc) with three-level model naming (<catalog>.<schema>.<model>) and inferred defaults for Unity Catalog workspaces.
* [MLflow 3 Model Registry with Unity Catalog](/concepts/mlflow-3-model-registry-in-unity-catalog.md) - Default registry URI in MLflow 3 is now databricks-uc, using Unity Catalog for model lifecycle management with three-level naming.
* [MLflow 3 Model Visualization](/concepts/mlflow-3-model-visualization.md) - New capabilities in MLflow 3 that extend chart view features from experiment runs to models on the Models tab, allowing visual comparison of logged models.
* [MLflow 3 Models tab in experiment page](/concepts/mlflow-3-models-tab-in-experiment-page.md) - A view showing all Logged Models produced by an experiment's runs, with a Charts tab for visual comparison of models before registering them to Unity Catalog for deployment.
* [MLflow 3 Models Tab Visualization](/concepts/mlflow-models-tab-visualizations.md) - Extension of chart view and visualization features (previously only for runs) to models from the Models tab in MLflow 3.
* [MLflow 3 Prerequisites for GenAI Tracing](/concepts/mlflow-3-prerequisites-for-genai-tracing.md) - System and package requirements for using MLflow 3's GenAI tracing capabilities, including mlflow[databricks]>=3.1 and framework-specific SDKs.
* [MLflow 3 Serverless Compute Autologging Requirement](/concepts/serverless-compute-autologging-requirement.md) - On Databricks serverless compute clusters, autologging for genAI tracing frameworks is not automatically enabled; users must explicitly call the appropriate `mlflow.<library>.autolog()` function.
* [MLflow 3 traditional ML workflow](/concepts/mlflow-3-traditional-ml-workflow.md) - A Databricks notebook-based workflow demonstrating model training as an MLflow Run, producing a Logged Model, and exploring results via the MLflow UI and Unity Catalog.
* [MLflow active experiment](/concepts/mlflow-active-experiment.md) - The mechanism for controlling which experiment MLflow logs runs to, which can be set using mlflow.set_experiment(), mlflow.start_run(), or environment variables.
* [MLflow Active Run Management with fit()](/concepts/active-mlflow-run-management-with-fit.md) - Best practice of wrapping CrossValidator.fit() or TrainValidationSplit.fit() calls inside a 'with mlflow.start_run():' block to ensure tuning results are logged under a dedicated parent run and to simplify adding custom tags, parameters, or metrics.
* [MLflow add_libraries_to_model Utility](/concepts/mlflow-add_libraries_to_model-utility.md) - MLflow utility that prepackages all model dependencies as Python wheel files alongside the model, ensuring exact library versions from the training environment
* [MLflow Agent Evaluation](/concepts/mlflow-agent-evaluation.md) - Comprehensive evaluation framework for AI agents, LLMs, RAG systems, and GenAI applications covering the entire development lifecycle from testing through production monitoring.
* [MLflow Aggregate Metric Analysis Across Experiments](/concepts/mlflow-aggregate-metric-analysis-across-experiments.md) - Using system tables to compute summary statistics (e.g., average GPU utilization) across all experiments in a workspace within a given time window.
* [MLflow and Unity Catalog Model Registration](/concepts/mlflow-and-unity-catalog-model-registration.md) - The workflow for logging fine-tuned LLM artifacts with MLflow and registering them in Unity Catalog for governance, versioning, and deployment as an 'llm/v1/chat' task.
* [MLflow and Unity Catalog Model Registration Pipeline](/concepts/mlflow-and-unity-catalog-model-registration.md) - Workflow for logging fine-tuned models to MLflow Tracking and registering them in Databricks Unity Catalog with versioning, metadata, and deployment readiness.
* [MLflow Annotation Data Formats](/concepts/mlflow-annotation-data-formats.md) - The otel_annotations table structure for storing MLflow tracing entities (metadata, tags, assessments, run links) as typed rows with soft-delete semantics.
* [MLflow Anthropic Autolog](/concepts/mlflow-anthropic-autolog.md) - Automatic trace capture for Anthropic LLM SDK calls via mlflow.anthropic.autolog()
* [MLflow Anthropic Autolog Disable](/concepts/mlflow-anthropic-autolog-disable.md) - Mechanism to globally disable automatic tracing for Anthropic via mlflow.anthropic.autolog(disable=True) or mlflow.autolog(disable=True)
* [MLflow Anthropic Autologging](/concepts/mlflow-anthropic-autolog.md) - Automatic tracing of Anthropic Claude SDK calls via mlflow.anthropic.autolog() for capturing agent interactions and tool calls in MLflow traces.
* [MLflow Application Versioning](/concepts/mlflow-application-versioning.md) - Version management for GenAI apps through logged models (snapshots linking traces, prompts, and code) and prompts (version-controlled templates with variables and aliases like 'production').
* [MLflow Artifact Packaging for Model Serving](/concepts/mlflow-artifact-packaging-for-model-serving.md) - Using MLflow's log_model() artifacts parameter to bundle file dependencies (e.g., model weights, tokenizer caches) directly into the model artifact for deployment on Databricks Model Serving.
* [MLflow Assessment and AssessmentSource API](/concepts/mlflow-assessment-and-assessmentsource-api.md) - MLflow SDK entities (Assessment, AssessmentSource, AssessmentSourceType) used to log structured human feedback on traces with rationale and source provenance.
* [MLflow Assessment and Feedback API](/concepts/mlflow-assessment-and-feedback-api.md) - Programmatic API for logging feedback on traces using mlflow.log_feedback() with AssessmentSource and AssessmentSourceType, enabling structured attribution of human evaluations.
* [MLflow Assessments (Feedback and Expectation)](/concepts/mlflow-assessments-feedback-and-expectation.md) - The human-generated labels captured during a labeling session, stored as Assessments on MLflow Traces; can be either Feedback or Expectation data and later synced to evaluation datasets.
* [MLflow Authentication Methods](/concepts/mlflow-authentication-methods.md) - Authentication approaches for connecting to remote MLflow tracking servers, including Personal Access Tokens (PAT) and OAuth with service principals.
* [MLflow Authentication Methods for Remote Tracking](/concepts/mlflow-authentication-methods.md) - Authentication approaches for remote MLflow tracking server connections, including Personal Access Tokens (PAT) and OAuth using service principals.
* [MLflow Auto-Tracing Limitations](/concepts/mlflow-auto-tracing-limitations.md) - MLflow auto-tracing only supports synchronous calls; async and streaming methods are not traced
* [MLflow Autolog Configuration Parameters](/concepts/mlflow-autolog-configuration-parameters.md) - A set of configurable parameters (log_input_examples, log_model_signatures, log_models, disable, exclusive, disable_for_unsupported_versions, silent) that control what and how Databricks Autologging captures during model training.
* [MLflow Autolog Disable Mechanism](/concepts/mlflow-autolog-disable-mechanism.md) - Method to globally disable automatic tracing for Groq via mlflow.groq.autolog(disable=True) or mlflow.autolog(disable=True)
* [MLflow Autolog for GenAI Tracing](/concepts/autologging-for-genai-tracing.md) - A single-line code pattern (mlflow.<library>.autolog()) that automatically traces generative AI applications across 20+ supported libraries and frameworks.
* [MLflow autolog for generative AI](/concepts/mlflow-autolog-for-generative-ai.md) - The mechanism in MLflow that automatically captures traces for supported GenAI frameworks (including txtai) when the autolog function is called
* [MLflow autolog() Configuration](/concepts/mlflow-autolog-configuration.md) - The default configuration parameters and customization options for mlflow.autolog() including log_models, log_input_examples, log_model_signatures, disable, exclusive, disable_for_unsupported_versions, and silent flags.
* [MLflow Autologging](/concepts/mlflow-autologging.md) - A framework feature that automatically logs ML model parameters, metrics, and artifacts from supported libraries without requiring explicit logging calls in user code.
* [MLflow Autologging for GenAI](/concepts/mlflow-autologging-for-genai.md) - Automatic instrumentation via mlflow.openai.autolog() that captures application execution traces (inputs, outputs) for use in MLflow Evaluation and monitoring.
* [MLflow Automatic Tracing](/concepts/mlflow-automatic-tracing.md) - One-line automatic instrumentation for GenAI libraries to capture LLM calls, tool usage, and agent interactions without significant code changes.
* [MLflow Automatic Tracing (autolog)](/concepts/mlflow-automatic-tracing-autolog.md) - One-line code pattern using mlflow.<library>.autolog() to automatically trace generative AI applications across 20+ supported libraries and frameworks.
* [MLflow Automatically Populated Trace Metadata](/concepts/automatically-populated-trace-metadata.md) - Standard metadata fields that MLflow automatically sets based on the execution environment, which can be overridden using mlflow.update_current_trace().
* [MLflow Bedrock Autolog](/concepts/mlflow-bedrock-autolog.md) - A function (mlflow.bedrock.autolog) that enables automatic tracing of Amazon Bedrock LLM invocations, capturing prompts, responses, latencies, model names, metadata, function calls, and exceptions.
* [MLflow Chart Interaction and Customization](/concepts/mlflow-chart-interaction-and-customization.md) - Interactive features of MLflow charts including hover details, move/resize/fullscreen controls, and edit/delete/download options via a kebab menu.
* [MLflow Chart View](/concepts/mlflow-chart-view.md) - A page in the MLflow UI that shows a collection of charts comparing runs of an experiment, with capabilities to select, filter, sort, group runs and customize or create new charts.
* [MLflow Child Run Pattern](/concepts/mlflow-child-run-pattern.md) - A hierarchical logging approach where a parent run is created in the driver process and each Ray task logs metrics to a nested child run under that parent.
* [MLflow Code-based Scorers](/concepts/mlflow-code-based-scorers.md) - Custom Python functions (decorator-based or class-based) that define flexible evaluation metrics for GenAI applications within MLflow Evaluation.
* [MLflow Combined Tracing](/concepts/mlflow-combined-tracing.md) - Hybrid approach mixing automatic and manual tracing to achieve complete coverage of application traces
* [MLflow Compatibility with databricks-feature-engineering](/concepts/mlflow-compatibility-with-databricks-feature-engineering.md) - A compatibility note that databricks-feature-engineering versions <=0.7.0 are incompatible with mlflow >=2.18.0, requiring an upgrade to databricks-feature-engineering 0.8.0 or above.
* [MLflow Correctness Scorer](/concepts/mlflow-correctness-scorer.md) - An evaluation scorer that quantitatively compares GenAI app outputs against expert-provided expected responses to measure factual alignment.
* [MLflow CrewAI Autolog](/concepts/mlflow-crewai-autolog.md) - Automatic tracing for CrewAI multi-agent workflows via mlflow.crewai.autolog()
* [MLflow Dashboard for GPU Utilization Monitoring](/concepts/mlflow-dashboard-for-gpu-utilization-monitoring.md) - A dashboard pattern for monitoring average GPU utilization and other system metrics across MLflow experiments within a time window, using system tables to identify inefficient resource usage.
* [MLflow Dashboard for Run Details](/concepts/mlflow-dashboard-for-run-details.md) - A dashboard pattern that reproduces MLflow UI run details (tags, parameters, metric graphs) from system table data using customizable SQL queries.
* [MLflow Dashboarding on Databricks](/concepts/mlflow-dashboarding-on-databricks.md) - The practice of building Dashboards in Databricks using MLflow metadata from system tables to analyze experiments and runs across an entire workspace.
* [MLflow Dashboards from System Tables](/concepts/mlflow-dashboards-from-system-tables.md) - Building and using Databricks dashboards that query MLflow system tables to visualize experiment results, run details, and system metrics across a workspace.
* [MLflow Databricks Tracking](/concepts/mlflow-databricks-tracking.md) - Setting up MLflow to use the Databricks tracking URI and experiments for capturing and visualizing agent traces
* [MLflow deep learning tracking](/concepts/mlflow-deep-learning-tracking.md) - Using MLflow on Databricks to track deep learning training runs and model development, providing experiment tracking essential for iterative deep learning workflows.
* [MLflow Deployment Jobs](/concepts/mlflow-deployment-jobs.md) - Automated jobs that manage the full lifecycle of ML models on Databricks by automating evaluation, approval, and deployment tasks when new model versions are created.
* [MLflow Deployment SDK](/concepts/mlflow-deployment-sdk.md) - A Python SDK (MLflow 2.9+) for interacting with Databricks Model Serving endpoints programmatically, used to send scoring requests alongside REST API and PowerBI connectors
* [MLflow deployment with CI/CD and MLOps](/concepts/mlflow-deployment-with-cicd-and-mlops.md) - Infrastructure-as-code management of MLflow experiments and models using Declarative Automation Bundles and MLOps Stacks, plus automated staged model deployment through MLflow 3 deployment jobs integrated with Databricks Workflows and Unity Catalog.
* [MLflow Deployments SDK](/concepts/mlflow-deployments-sdk.md) - A Python SDK (mlflow.deployments) for programmatically creating and managing model serving endpoints on Databricks.
* [MLflow Document Entity](/concepts/mlflow-document-entity.md) - A data class in mlflow.entities representing a retrieved document with an id, page_content, and metadata dictionary.
* [MLflow Environment and Version Tracking](/concepts/mlflow-environment-and-version-tracking.md) - Standardized metadata fields for tracking execution environment (development, staging, production) and application version in traces, enabling environment-specific analysis and regression detection.
* [MLflow Eval Harness](/concepts/mlflow-eval-harness.md) - Development-phase evaluation system that uses LLM judges and scorers to assess traces and measure the quality of GenAI applications
* [MLflow evaluate with Judges](/concepts/mlflow-evaluate-with-judges.md) - The pattern of using mlflow.genai.evaluate() with a dataset and one or more scorer/judge instances to perform comprehensive GenAI application evaluation.
* [MLflow Evaluation Dataset](/concepts/mlflow-evaluation-dataset.md) - A curated set of example inputs (labeled or unlabeled) used to systematically test and improve GenAI applications, stored in Unity Catalog with versioning, lineage, sharing, and governance.
* [MLflow Evaluation Dataset for Conversation Testing](/concepts/mlflow-evaluation-dataset-for-conversation-testing.md) - Persisting conversation test cases as reproducible MLflow Evaluation Datasets using create_dataset and merge_records.
* [MLflow Evaluation Dataset for Reproducible Testing](/concepts/mlflow-evaluation-dataset-for-conversation-testing.md) - A mechanism to persist test cases as MLflow Evaluation Datasets using create_dataset and get_dataset, enabling reproducible conversation simulation runs.
* [MLflow Evaluation Dataset for Versioning](/concepts/mlflow-evaluation-dataset-for-versioning.md) - Recommended practice of packaging labeled traces into an MLflow Evaluation Dataset for version tracking and lineage when evaluating GenAI apps.
* [MLflow Evaluation Dataset Integration for Conversation Simulation](/concepts/mlflow-evaluation-dataset-integration-for-conversation-simulation.md) - Mechanism to persist conversation test cases as MLflow Evaluation Datasets for reproducible testing, using create_dataset and get_dataset APIs.
* [MLflow Evaluation Dataset Limitations](/concepts/evaluation-dataset-limitations.md) - Constraints on evaluation datasets including a maximum of 2000 rows per dataset, 20 expectations per record, and incompatibility with customer-managed key (CMK) encrypted catalogs.
* [MLflow Evaluation Dataset UI](/concepts/evaluation-dataset-ui.md) - A visual interface in the MLflow experiment Datasets tab for creating, editing, searching, and managing evaluation datasets and their records without writing code.
* [MLflow Evaluation Datasets](/concepts/evaluation-datasets.md) - Structured test data used as input for GenAI evaluations, available in formats like EvaluationDataset, list of dicts, Pandas DataFrame, or Spark DataFrame.
* [MLflow Evaluation Results Comparison](/concepts/mlflow-evaluation-results-comparison.md) - The ability to compare multiple evaluation runs side-by-side in the MLflow UI to assess the impact of prompt changes on GenAI application quality.
* [MLflow Evaluation Run](/concepts/mlflow-evaluation-run.md) - An MLflow run that organizes and stores the results of evaluating a GenAI app, including traces, feedback, metrics, and metadata.
* [MLflow Evaluation Runs](/concepts/mlflow-evaluation-runs.md) - Test report capturing everything about how a GenAI app performed on a specific dataset, including per-row traces annotated with feedback from each judge.
* [MLflow Evaluation UI](/concepts/mlflow-evaluation-ui.md) - Interactive user interface within MLflow for reviewing, comparing, and analyzing GenAI evaluation runs across different prompt versions.
* [MLflow Experiment](/concepts/mlflow-experiment.md) - A named container that organizes and groups all artifacts related to a single GenAI application, including traces, evaluation runs, app versions, prompts, and quality assessments.
* [MLflow Experiment as a Databricks App Resource](/concepts/mlflow-experiment-as-a-databricks-app-resource.md) - The ability to add an MLflow experiment as a resource within a Databricks App, enabling experiment tracking for AI applications directly from the app.
* [MLflow Experiment as Trace Storage Backend](/concepts/mlflow-experiment-as-trace-storage-backend.md) - MLflow experiments serve as the primary storage location for production traces, with optional custom artifact locations (e.g., Unity Catalog volumes) enabling Unity Catalog volume privilege governance for trace data access.
* [MLflow Experiment Best Practices for Databricks Apps](/concepts/mlflow-experiment-best-practices-for-databricks-apps.md) - Recommended practices for organizing, naming, and managing MLflow experiments used as resources in Databricks Apps.
* [MLflow Experiment Budget Policy Binding](/concepts/mlflow-experiment-budget-policy-binding.md) - The practice of assigning a serverless budget policy to an MLflow experiment so that all serverless workloads spawned from that experiment use the designated policy instead of the workspace default.
* [MLflow Experiment Budget Policy Configuration](/concepts/mlflow-experiment-budget-policy-configuration.md) - The mechanism to assign a serverless budget policy to an MLflow experiment via UI or API to control serverless workloads.
* [MLflow Experiment Configuration for Air Runs](/concepts/mlflow-experiment-configuration-for-air-runs.md) - The YAML fields `experiment_name` (required) and `mlflow_run_name` (optional) control how runs are organized in MLflow experiments.
* [MLflow Experiment Environment Variable Injection](/concepts/mlflow-experiment-environment-variable-injection.md) - Databricks automatically exposes the experiment ID via environment variables when an MLflow experiment is added as an app resource, accessible through the valueFrom field in app.yaml.
* [MLflow Experiment for GenAI](/concepts/mlflow-experiment-for-genai.md) - A container in Databricks used to organize and manage GenAI application traces, runs, and experiment metadata, accessible via the Experiments UI under 'GenAI apps & agents'.
* [MLflow Experiment Integration for AI Runtime](/concepts/mlflow-integration-in-ai-runtime.md) - Each AI Runtime CLI run creates a run within a named MLflow experiment, capturing metrics, parameters, artifacts, and logs viewable in the Databricks workspace MLflow UI.
* [MLflow experiment output schema](/concepts/mlflow-experiment-output-schema.md) - The fixed schema returned by the mlflow-experiment data source, including run_id, experiment_id, metrics (map), params (map), tags (map), start_time, end_time, status, and artifact_uri.
* [MLflow Experiment Permission Levels for Apps](/concepts/mlflow-experiment-permission-levels-for-apps.md) - Three permission levels (Can read, Can edit, Can manage) that control what a Databricks App's service principal can do with a linked MLflow experiment resource.
* [MLflow Experiment Resource](/concepts/mlflow-experiment-resource.md) - A Databricks Apps resource type that connects an MLflow experiment to an app for tracking runs, parameters, metrics, and artifacts.
* [MLflow Experiment Resource (Databricks Apps)](/concepts/mlflow-experiment-resource.md) - A Databricks Apps resource type that binds an MLflow experiment to an app, granting the app's service principal permissions to track runs, log parameters/metrics, and access experiment data via the MLflow Tracking API.
* [MLflow Experiment Resource Best Practices](/concepts/mlflow-experiment-resource-best-practices.md) - Recommended practices for organizing experiments by project or model type, using consistent naming conventions for runs and parameters, and considering retention policies and storage management.
* [MLflow Experiment Resource in Databricks Apps](/concepts/mlflow-experiment-resource-in-databricks-apps.md) - A configuration pattern that allows Databricks Apps to reference and interact with MLflow experiments as first-class resources, enabling experiment tracking capabilities within applications.
* [MLflow Experiment Resource Lifecycle](/concepts/mlflow-experiment-resource-lifecycle.md) - The process of adding and removing MLflow experiment resources from Databricks apps, including how removal revokes service principal access while leaving the experiment itself intact.
* [MLflow Experiment Resource Lifecycle in Databricks Apps](/concepts/mlflow-experiment-resource-lifecycle.md) - The process of adding an MLflow experiment resource to an app (granting service principal access) and removing it (revoking service principal access, with the experiment itself remaining unchanged).
* [MLflow Experiment Scoped Access](/concepts/mlflow-experiment-scoped-access.md) - When an MLflow experiment is added as a resource, the app's access is scoped exclusively to that specific experiment; other experiments must be added as separate resources.
* [MLflow Experiment Trace Location Configuration](/concepts/mlflow-experiment-trace-location-configuration.md) - The API and configuration pattern for setting a Unity Catalog trace location when creating or configuring an MLflow experiment, including catalog_name, schema_name, and table_prefix parameters
* [MLflow Experiment Tracking](/concepts/mlflow-experiment-tracking.md) - The MLflow capability to create, list, get, and manage experiments and runs programmatically via the REST API
* [MLflow Experiment Tracking for Distributed Training](/concepts/mlflow-experiment-tracking-for-distributed-training.md) - Integrating MLflow to log metrics (loss values), artifacts (checkpoints), and run metadata during distributed multi-GPU training sessions for experiment tracking and reproducibility.
* [MLflow Experiment Tracking Lifecycle](/concepts/mlflow-experiment-tracking-lifecycle.md) - Structured workflow for organizing AI/ML development lifecycle using MLflow experiments — including logging runs, parameters, metrics, artifacts, debugging agent traces, evaluating LLM quality, and versioning prompt templates.
* [MLflow Experiment Tracking on Databricks AI Runtime](/concepts/mlflow-experiment-tracking-on-databricks-ai-runtime.md) - Integration of MLflow for logging metrics, parameters, and artifacts from distributed training workloads submitted via the air CLI on Databricks
* [MLflow Experiment Tracking on Serverless GPU](/concepts/mlflow-experiment-tracking-on-serverless-gpu.md) - Logging training metrics (loss, avg_loss), artifacts (checkpoints), and run metadata to MLflow during distributed training on serverless GPU compute for experiment tracking and reproducibility.
* [MLflow Experiment Types: Workspace vs Notebook](/concepts/mlflow-experiment-types-workspace-vs-notebook.md) - MLflow experiments on Databricks come in two flavors — workspace experiments (standalone, created via UI or API, not tied to a notebook) and notebook experiments (auto-created when mlflow.start_run() is called without an active experiment, bound to a specific notebook).
* [MLflow Experiment Workload Creation Policy Tag](/concepts/mlflow-experiment-workload-creation-policy-tag.md) - The mlflow.workload_creation_policy_id experiment tag used to bind a serverless budget policy to an MLflow experiment via API.
* [MLflow experiments for GenAI applications](/concepts/mlflow-experiment-for-genai.md) - An MLflow experiment serves as the container for GenAI application traces and runs, automatically linked when using a Databricks notebook.
* [MLflow Feedback Data Model](/concepts/mlflow-feedback-data-model.md) - A structured entity (a type of Assessment) for capturing user feedback on traces, storing value, source, rationale, and metadata
* [MLflow Feedback Entity (Assessment System)](/concepts/mlflow-feedback-entity-assessment-system.md) - Structured data model for capturing user feedback as assessments on traces, including value, source, rationale, and metadata fields.
* [MLflow Feedback Entity and Assessment Data Model](/concepts/mlflow-feedback-entity-assessment-system.md) - A structured data model where user feedback is captured as Assessment entities attached to traces or spans, containing value, source, rationale, and metadata fields.
* [MLflow Feedback Object](/concepts/mlflow-feedback-object.md) - The structured output returned by MLflow LLM judges containing a 'value' field ('yes' or 'no') indicating relevance and a 'rationale' field explaining the reasoning for the assessment.
* [MLflow for Deep Learning Model Lifecycle](/concepts/mlflow-for-deep-learning-model-lifecycle.md) - Integrated MLflow support in Databricks for tracking model development, autologging, model deployment, and serving — enabling management of deep learning models including custom preprocessing logic across batch, streaming, and online inference.
* [MLflow for Deep Learning Model Management](/concepts/mlflow-for-deep-learning-model-management.md) - Using MLflow for tracking experiments, autologging, model deployment, and model serving for deep learning models on Databricks.
* [MLflow for Deep Learning Tracking](/concepts/mlflow-deep-learning-tracking.md) - Using MLflow on Databricks to track iterative deep learning training runs, model development, and experiment logging.
* [MLflow for experiment tracking](/concepts/mlflow-for-experiment-tracking.md) - Using MLflow to log, track, and compare hyperparameter tuning runs across different model types
* [MLflow for Experiment Tracking and Model Registry](/concepts/mlflow-for-experiment-tracking-and-model-registry.md) - Databricks-managed MLflow provides reproducible, auditable ML development with tracking, registry, and full audit trails linking model versions to training runs, datasets, environments, and git commits.
* [MLflow for Model Lifecycle Tracking](/concepts/mlflow-for-model-lifecycle-tracking.md) - Using MLflow to track model development, log code snapshots, parameters, metrics, artifacts, and manage the model lifecycle across environments.
* [MLflow Function Calling Agent Tracing](/concepts/mlflow-function-calling-agent-tracing.md) - Combining MLflow's @mlflow.trace decorator with mlflow.bedrock.autolog enables tracing of custom ReAct-style function-calling agents, capturing the full chain of LLM invocations and tool executions as a single trace.
* [MLflow Gemini Autolog](/concepts/mlflow-gemini-autolog.md) - Automatic MLflow tracing for Google Gemini SDK calls via mlflow.gemini.autolog(), capturing prompts, responses, latencies, model names, metadata, function calls, and exceptions.
* [MLflow GenAI App Tracing with OpenAI](/concepts/mlflow-genai-app-tracing-with-openai.md) - Pattern for instrumenting Python GenAI applications using MLflow's automatic OpenAI client tracing and the @mlflow.trace decorator for custom functions like retrievers
* [MLflow GenAI Assessment](/concepts/mlflow-genai-assessment.md) - Quality measurements and ground truth labels (feedback or expectations) attached to a trace, added by end users, domain experts, or automated scorers.
* [MLflow GenAI Data Model](/concepts/mlflow-genai-data-model.md) - Hierarchical data model for organizing generative AI application data within experiments, covering traces, evaluations, human labeling, and versioning.
* [MLflow GenAI Datasets SDK](/concepts/mlflow-genai-datasets-sdk.md) - Programmatic API (mlflow.genai.datasets) for creating, retrieving, deleting, and interacting with evaluation datasets via Python.
* [MLflow GenAI Evaluate API](/concepts/mlflow-genai-evaluate-api.md) - The mlflow.genai.evaluate() API for running evaluation datasets through multiple scorers including third-party integrations like Phoenix
* [MLflow genai evaluate() with DeepEval Scorers](/concepts/mlflow-genai-evaluate-with-deepeval-scorers.md) - DeepEval scorers can be used as part of the mlflow.genai.evaluate() pipeline to evaluate datasets of inputs and outputs.
* [MLflow GenAI Evaluation](/concepts/mlflow-genai-evaluation.md) - A framework within MLflow for evaluating generative AI applications, including built-in judges like RetrievalGroundedness for automated scoring of LLM outputs.
* [MLflow GenAI Evaluation Dataset](/concepts/mlflow-genai-evaluation-dataset.md) - A curated, versioned collection of test cases (inputs and optional ground-truth expectations) used to systematically evaluate and improve a GenAI application's quality.
* [MLflow GenAI Evaluation Dataset Format](/concepts/mlflow-genai-evaluation-dataset-format.md) - Structure for evaluation datasets in MLflow GenAI, requiring inputs, outputs, and expectations (with context) fields
* [MLflow GenAI Evaluation Datasets](/concepts/mlflow-genai-evaluation-datasets.md) - Structured evaluation data for LLM applications, typically a list of dictionaries with an 'inputs' key containing message histories for the AI agent.
* [MLflow GenAI Evaluation Framework](/concepts/mlflow-genai-evaluation-framework.md) - A framework within MLflow for evaluating generative AI applications using structured datasets, custom scorers, and automated tracing.
* [MLflow GenAI Evaluation Harness](/concepts/mlflow-genai-evaluation-harness.md) - A structured framework using `mlflow.genai.evaluate()` to systematically test GenAI app quality against curated datasets using scorers, with support for both direct and answer-sheet modes.
* [MLflow GenAI Evaluation Pipeline](/concepts/mlflow-genai-evaluation-pipeline.md) - The workflow for evaluating GenAI applications using MLflow, combining evaluation datasets, built-in or custom scorers/judges, and the mlflow.genai.evaluate() function to produce quality assessments.
* [MLflow GenAI Evaluation Results Comparison](/concepts/mlflow-evaluation-results-comparison.md) - The ability to compare multiple evaluation runs side-by-side in the MLflow UI to measure the impact of prompt changes or model updates.
* [MLflow GenAI Evaluation Scorers (Judges)](/concepts/mlflow-genai-evaluation-scorers-judges.md) - A framework within MLflow for using LLM-based judges (scorers) to programmatically evaluate the quality of GenAI application outputs, including relevance, groundedness, safety, and correctness.
* [MLflow GenAI Evaluation with LLM-as-Judge](/concepts/mlflow-genai-evaluation-with-llm-as-judge.md) - Using MLflow's make_judge to create LLM-based evaluators that assess agent outputs for relevance and other criteria, integrated with tracing.
* [MLflow GenAI evaluation with third-party scorers](/concepts/mlflow-genai-evaluation-with-third-party-scorers.md) - The integration pattern for using external evaluation scorers, such as Arize Phoenix, within MLflow's mlflow.genai.evaluate() function by passing them in the scorers list parameter.
* [MLflow GenAI Evaluation Workflow](/concepts/mlflow-genai-evaluation-workflow.md) - End-to-end process of defining custom judges, building evaluation datasets, and running evaluations with mlflow.genai.evaluate() to assess GenAI agent quality
* [MLflow GenAI Experiment](/concepts/mlflow-experiment.md) - A named container that organizes all artifacts related to a single GenAI application, including traces, evaluations, prompts, and model versions.
* [MLflow GenAI Feedback Data Model](/concepts/mlflow-genai-feedback-data-model.md) - The structured Feedback entity in MLflow that captures user assessments on traces, including value, source, rationale, and metadata fields.
* [MLflow GenAI Feedback Object](/concepts/mlflow-genai-feedback-object.md) - A structured output returned by LLM judges containing a binary value (yes/no) and a detailed rationale explaining the evaluation result.
* [MLflow GenAI Labeling and Label Schemas](/concepts/mlflow-genai-labeling-and-label-schemas.md) - New mlflow.genai.labeling and mlflow.genai.label_schemas modules for defining and managing evaluation label schemas.
* [MLflow GenAI Labeling Session](/concepts/mlflow-genai-labeling-session.md) - An organized queue of traces prepared for human review by domain experts, using labeling schemas to structure the assessments collected.
* [MLflow genai optimize_prompts](/concepts/mlflow-genai-optimize_prompts.md) - An MLflow function that orchestrates prompt optimization by combining a prediction function, training data, prompt URIs, an optimizer, custom scorers, and an aggregation function.
* [MLflow GenAI Predefined Scorers](/concepts/mlflow-genai-predefined-scorers.md) - A set of built-in evaluation scorers in MLflow 3 including Correctness, Guidelines, ExpectationsGuidelines, RelevanceToQuery, Safety, RetrievalGroundedness, RetrievalRelevance, and RetrievalSufficiency.
* [MLflow GenAI Production Monitoring](/concepts/mlflow-genai-production-monitoring.md) - Automated continuous quality assessment of GenAI app traces using scheduled scorers in production, with configurable sampling and multi-turn conversation evaluation.
* [MLflow GenAI Prompt Optimization](/concepts/mlflow-genai-prompt-optimization.md) - A framework within MLflow for automatically optimizing prompt templates using evolutionary algorithms and evaluation scorers
* [MLflow genai Python SDK for Prompt Management](/concepts/mlflow-genai-python-sdk-for-prompt-management.md) - Python SDK functions (register_prompt, load_prompt, search_prompts) for programmatic management of prompts in the Prompt Registry.
* [MLflow GenAI Scorer](/concepts/mlflow-genai-scorer.md) - A function that evaluates a trace's quality by extracting relevant data fields and passing them to evaluation logic (deterministic code or LLM judge), returning feedback assessments.
* [MLflow GenAI Scorers](/concepts/mlflow-genai-scorers.md) - Modular scoring components (built-in like Safety, and custom like Guidelines) used to evaluate GenAI outputs programmatically within MLflow Evaluation workflows.
* [MLflow GenAI Scorers API](/concepts/mlflow-genai-scorers-api.md) - Python API within Databricks for registering, starting, and backfilling scorers on MLflow traces, including both built-in and custom scoring functions.
* [MLflow GenAI Trace](/concepts/mlflow-genai-trace.md) - A record of the complete execution of a GenAI application, capturing inputs, outputs, and every intermediate step such as LLM calls, retrievals, and tool use.
* [MLflow GenAI Tracing](/concepts/mlflow-genai-tracing.md) - An MLflow subsystem that captures and visualizes the end-to-end execution traces of generative AI models and LLM API calls for debugging and observability.
* [MLflow GenAI Tracing on Databricks](/concepts/mlflow-genai-tracing-on-databricks.md) - The integration that captures full traces of GenAI agent workflows—LLM calls, tool invocations, and multi-agent message passing—and records them to a Databricks experiment.
* [MLflow genai.evaluate Function](/concepts/mlflow-genaievaluate-function.md) - The mlflow.genai.evaluate() function runs a GenAI application against an evaluation dataset and applies scorers to judge outputs, logging results to the active MLflow experiment.
* [MLflow Groq Autolog](/concepts/mlflow-groq-autolog.md) - Automatic trace generation for Groq SDK calls via mlflow.groq.autolog()
* [MLflow Guardrails AI scorer integration](/concepts/mlflow-guardrails-integration.md) - MLflow integrates Guardrails AI validators as scorers for rule-based LLM evaluation without requiring LLM calls
* [MLflow Guardrails Integration](/concepts/mlflow-guardrails-integration.md) - MLflow integrates Guardrails AI validators as scorers for rule-based LLM output evaluation without requiring additional LLM calls.
* [MLflow Guidelines Scorer](/concepts/mlflow-guidelines-scorer.md) - A scorer that evaluates LLM outputs against custom natural-language guidelines such as language consistency, humor, child safety, and template adherence.
* [MLflow HTTP Client Timeout Configuration](/concepts/mlflow-http-timeout-configuration.md) - Configuration of MLflow environment variables (MLFLOW_HTTP_REQUEST_TIMEOUT, MLFLOW_HTTP_REQUEST_MAX_RETRIES) to control client-side HTTP request timeouts for model serving.
* [MLflow HTTP Request Timeout Configuration](/concepts/mlflow-http-timeout-configuration.md) - Client-side timeout configuration via MLflow environment variables MLFLOW_HTTP_REQUEST_TIMEOUT and MLFLOW_HTTP_REQUEST_MAX_RETRIES for model serving requests.
* [MLflow HTTP Timeout Configuration](/concepts/mlflow-http-timeout-configuration.md) - MLflow environment variables (MLFLOW_HTTP_REQUEST_TIMEOUT, MLFLOW_HTTP_REQUEST_MAX_RETRIES) that control client-side HTTP request timeouts for MLflow operations.
* [MLflow Human Feedback Assessments](/concepts/mlflow-human-feedback-assessments.md) - MLflow stores human feedback as Assessments — structured objects (Feedback or Expectation types) attached to individual MLflow Traces or spans within a trace, enabling collection of ratings, comments, and ground-truth expectations from developers, domain experts, and end users.
* [MLflow Human Feedback Collection](/concepts/mlflow-human-feedback-collection.md) - System for collecting expert or human feedback on GenAI application traces using labeling schemas, labeling sessions, and the Review App UI.
* [MLflow Human Feedback Collection (Review App)](/concepts/mlflow-human-feedback-collection.md) - A system for collecting structured feedback from domain experts on GenAI traces using labeling schemas, labeling sessions, and the Review App UI.
* [MLflow Instrumentation for GenAI](/concepts/mlflow-instrumentation-for-genai.md) - Methods for adding tracing to GenAI code using autolog(), @mlflow.trace decorator, or manual span creation with runnable code snippets
* [MLflow Instrumentation Guidance](/concepts/mlflow-instrumentation-guidance.md) - The ability to get help adding tracing to GenAI agent code using autolog(), @mlflow.trace decorator, or manual spans, with runnable code snippets for Databricks notebooks.
* [MLflow Integration for AI Runtime](/concepts/mlflow-integration-in-ai-runtime.md) - Automatic tracking of training workload metrics, parameters, artifacts, and logs in MLflow experiments and runs, viewable in the Databricks workspace UI
* [MLflow integration for AI Runtime CLI](/concepts/mlflow-integration-in-ai-runtime.md) - Integration with MLflow to track and manage training runs submitted via the AI Runtime CLI.
* [MLflow Integration for Computer Vision Models](/concepts/mlflow-integration-for-computer-vision-models.md) - AI Runtime computer vision workflows integrate MLflow for experiment tracking and Databricks Model Serving for deployment of trained models.
* [MLflow Integration for Distributed Training Runs](/concepts/mlflow-integration-for-distributed-training-runs.md) - Integration between the AI Runtime CLI and MLflow for tracking distributed training runs, combined with visibility in the Databricks Jobs run page
* [MLflow integration for hyperparameter tuning](/concepts/mlflow-integration-for-hyperparameter-tuning.md) - The practice of using MLflow to track and compare hyperparameter tuning runs across different model types, enabling systematic model selection and experiment tracking.
* [MLflow Integration for LLM Training](/concepts/mlflow-integration-for-llm-training.md) - The integration of MLflow experiment tracking with LLM Foundry to log training metrics, system metrics, and model checkpoints, enabling experiment comparison and reproducibility.
* [MLflow Integration for XGBoost Spark Models](/concepts/mlflow-integration-for-xgboost-spark-models.md) - Limitations of MLflow autologging with distributed XGBoost and recommended use of mlflow.spark.log_model for model logging
* [MLflow Integration in AI Runtime](/concepts/mlflow-integration-in-ai-runtime.md) - Automatic tracking of training runs as MLflow experiments, capturing metrics, parameters, artifacts, and logs visible in the Databricks workspace MLflow UI.
* [MLflow integration in AI Runtime CLI](/concepts/mlflow-integration-in-ai-runtime.md) - AI Runtime CLI integrates with MLflow for tracking runs of distributed training workloads.
* [MLflow Integration in Ray Train on Databricks](/concepts/mlflow-integration-in-ray-train-on-databricks.md) - Automatic MLflow logging integration where Databricks injects MLFLOW_RUN_ID and configures the tracking URI so distributed Ray Train jobs can log metrics without explicit credentials.
* [MLflow Integration on Databricks](/concepts/mlflow-integration-on-databricks.md) - Databricks AI Runtime integrates MLflow for tracking model training experiments, as demonstrated in computer vision tutorials.
* [MLflow Integration with AI Runtime](/concepts/mlflow-integration-in-ai-runtime.md) - AI Runtime natively integrates with MLflow for experiment tracking, metric visualization, and model logging, with support for autologging, custom run names, and environment variable configuration.
* [MLflow Integration with AI Runtime CLI](/concepts/mlflow-integration-in-ai-runtime.md) - Integration between the AI Runtime CLI and MLflow for tracking and managing training runs.
* [MLflow Integration with Axolotl](/concepts/mlflow-integration-with-axolotl.md) - The integration between Axolotl training framework and MLflow for experiment tracking, logging training metrics, and registering models to Unity Catalog
* [MLflow Integration with Hugging Face](/concepts/mlflow-integration-with-hugging-face.md) - Logging Hugging Face models to MLflow using mlflow.transformers.log_model, including automatic metric logging via MLflowCallback and model artifact management.
* [MLflow Integration with Hugging Face Transformers](/concepts/mlflow-integration-with-hugging-face.md) - The integration between MLflow and Hugging Face that automatically logs training metrics via MLflowCallback and provides mlflow.transformers.log_model for logging trained models and pipelines to the MLflow registry.
* [MLflow integration with Hyperopt on Databricks](/concepts/mlflow-integration-with-hyperopt-on-databricks.md) - Using MLflow to track and compare hyperparameter tuning experiments across different model types in Databricks
* [MLflow Integration with Ray on Databricks](/concepts/mlflow-integration-with-ray-on-databricks.md) - Using MLflow tracking with Ray Tune, Ray Train, or customized Ray tasks on Databricks by setting authentication environment variables
* [MLflow integration with Ray Train on Databricks](/concepts/mlflow-integration-in-ray-train-on-databricks.md) - Automatic MLflow logging setup where AI Runtime injects MLFLOW_RUN_ID and configures the Databricks tracking URI, enabling metric and parameter logging without explicit credentials.
* [MLflow Judge Feedback Object](/concepts/mlflow-feedback-object.md) - The output schema returned by MLflow LLM judges containing a 'value' field ('yes' or 'no') indicating relevance and a 'rationale' field providing an explanation for the judge's decision.
* [MLflow Labeling Schemas](/concepts/mlflow-labeling-schemas.md) - Predefined or custom templates that define the questions, format, and structure for collecting feedback or expectations during labeling sessions.
* [MLflow Labeling Schemas and Sessions](/concepts/mlflow-labeling-schemas-and-sessions.md) - Programmatic API (create_label_schema, create_labeling_session) to define what feedback to collect from human reviewers and organize traces for review.
* [MLflow Labeling Sessions](/concepts/labeling-session.md) - A structured mechanism for gathering human-generated assessments (labels) from domain experts on GenAI application traces, implemented as a special type of MLflow run.
* [MLflow Labeling Sessions and Schemas](/concepts/mlflow-labeling-sessions-and-schemas.md) - Primitives for defining structured human feedback collection — create_label_schema defines what feedback to collect (e.g., categorical ratings), and create_labeling_session groups schemas and traces for reviewer access.
* [MLflow Labeling Sessions for Expert Review](/concepts/mlflow-labeling-sessions-for-expert-review.md) - Structured workflow for sending GenAI app traces to domain experts for review via labeling sessions, enabling authoritative ground-truth feedback collection.
* [MLflow LabelingSession API](/concepts/mlflow-labelingsession-api.md) - The programmatic Python API (mlflow.genai.labeling) for creating, retrieving, deleting labeling sessions, managing users, adding traces, and syncing expectations to evaluation datasets.
* [MLflow LangGraph Tracing](/concepts/mlflow-langgraph-tracing.md) - Automatic capture of LangGraph agent executions as MLflow traces via the LangChain autolog integration
* [MLflow LLM Judge Architecture](/concepts/mlflow-llm-judge-architecture.md) - The framework of built-in LLM judges in MLflow that use a judge LLM (defaulting to a Databricks-hosted model) to automatically assess GenAI application quality across dimensions like relevance, groundedness, safety, and correctness.
* [MLflow LLM Judge Feedback Object](/concepts/mlflow-llm-judge-feedback-object.md) - The structured return type from MLflow LLM judges containing a binary 'value' (yes/no) indicating correctness and a 'rationale' providing detailed explanation of which facts were supported or missing.
* [MLflow LLM Judges](/concepts/mlflow-llm-judges.md) - Built-in LLM-powered evaluators that assess quality dimensions (relevance, groundedness, safety, correctness) of GenAI application outputs
* [MLflow LLM-as-a-Judge Evaluation](/concepts/mlflow-llm-as-a-judge-evaluation.md) - Automated evaluation of GenAI application outputs using built-in and custom scorers that leverage an LLM to judge metrics like Safety, Correctness, and custom guidelines.
* [MLflow log_feedback API](/concepts/mlflow-log_feedback-api.md) - The API for programmatically logging user feedback against traces, supporting boolean, numeric, text, and structured feedback values.
* [MLflow Logged Model](/concepts/mlflow-logged-model.md) - A trained model tracked as a first-class entity in MLflow 3, capturing parameters, metrics, metadata, source runs, datasets, and model versions separately from MLflow Runs.
* [MLflow LoggedModel](/concepts/mlflow-loggedmodel.md) - A tracked model artifact in MLflow 3 that represents a specific checkpoint or version of a model, with associated metrics, parameters, and metadata.
* [MLflow Logging (Parameters, Metrics, Artifacts)](/concepts/mlflow-logging-parameters-metrics-artifacts.md) - The MLflow capability to log parameters, metrics, and artifacts to runs via the REST API
* [MLflow Logging in Ray Train Workers](/concepts/mlflow-logging-in-ray-train-workers.md) - Pattern for conditionally starting MLflow runs inside Ray Train workers using the MLFLOW_RUN_ID environment variable injected by the Databricks AI Runtime, enabling per-worker metric logging without explicit credentials.
* [MLflow Logging Limitations for Distributed XGBoost](/concepts/mlflow-logging-limitations-for-distributed-xgboost.md) - Inability to use mlflow.xgboost.autolog with distributed XGBoost; alternative use of mlflow.spark.log_model for logging xgboost.spark models
* [MLflow make_judge](/concepts/mlflow-make_judge.md) - A function in MLflow's GenAI module that allows users to create custom judges/scorers for evaluating LLM outputs based on custom instructions and criteria.
* [MLflow Manual Tracing](/concepts/manual-tracing.md) - Manual approach using function decorator APIs or low-level APIs for custom logic and complex workflow tracing.
* [MLflow MCP Server](/concepts/mlflow-mcp-server.md) - An MCP (Model Context Protocol) server that exposes MLflow trace management operations to AI applications and coding assistants
* [MLflow Mistral Auto Tracing](/concepts/mlflow-mistral-auto-tracing.md) - Automatic tracing of Mistral AI API calls via mlflow.mistral.autolog() that captures request/response telemetry without manual instrumentation.
* [MLflow Model Comparison](/concepts/mlflow-model-comparison.md) - Using MLflow to track, compare, and evaluate performance across multiple model types and hyperparameter configurations during model selection.
* [MLflow Model Dependency Logging](/concepts/mlflow-model-dependency-logging.md) - The practice of logging a machine learning model along with its Python and non-Python dependencies as artifacts, ensuring reproducibility and portability across environments.
* [MLflow Model Dependency Management](/concepts/mlflow-model-dependency-management.md) - Process of updating pip requirements of a logged MLflow model in-place without re-logging the model.
* [MLflow Model Flavors](/concepts/mlflow-model-flavors.md) - Built-in conventions that allow a single MLflow Model to be saved in multiple framework-specific formats (python-function, pytorch, sklearn, etc.) for compatibility with different serving and inference platforms.
* [MLflow Model Format](/concepts/mlflow-model-format.md) - A standard format for packaging machine learning models for use across downstream tools such as batch inference on Spark or real-time REST API serving.
* [MLflow Model Logging for GPU Serving with flash-attn](/concepts/mlflow-model-logging-for-gpu-serving-with-flash-attn.md) - Best practices for logging transformer models that require flash-attn, including specifying compatible CUDA, PyTorch, Torchvision versions and using custom wheel references in pip_requirements for mlflow.transformers.log_model.
* [MLflow Model Logging for Serving](/concepts/mlflow-model-logging-for-serving.md) - Methods for logging ML models in MLflow format for deployment: autologging, built-in flavors, pyfunc custom logging, with required signatures and input examples for Unity Catalog.
* [MLflow model logging methods for serving](/concepts/mlflow-model-logging-methods-for-serving.md) - Three approaches to log ML models for Databricks Model Serving: autologging, MLflow built-in flavors, and custom pyfunc logging — each with different trade-offs for manual control and flexibility.
* [MLflow Model Registration and PyFunc Wrapping for Recommendation Models](/concepts/mlflow-model-registration-and-pyfunc-wrapping-for-recommendation-models.md) - Registering a trained two-tower model in MLflow with a PyFunc wrapper to simplify serving by converting dict inputs into prediction lists.
* [MLflow Model Registry](/concepts/mlflow-model-registry.md) - A centralized model store with UI and APIs for managing the full lifecycle of MLflow Models, including registration with Unity Catalog or Workspace Model Registry.
* [MLflow Model Registry (Unity Catalog)](/concepts/mlflow-model-registry-unity-catalog.md) - Databricks' hosted version of the open-source MLflow Model Registry, integrated with Unity Catalog, using the 'databricks-uc' registry URI and three-level naming (catalog.schema.model).
* [MLflow Model Registry in Unity Catalog](/concepts/mlflow-model-registry-in-unity-catalog.md) - A governed model registry in Unity Catalog for managing model versions, aliases (Staging, Production), staging-to-production promotion, rollback, and audit trails.
* [MLflow Model Serving (Databricks)](/concepts/mlflow-model-serving-databricks.md) - Databricks platform feature for deploying custom ML models as serving endpoints, with support for MLflow pyfunc models registered to Unity Catalog or Workspace Registry.
* [MLflow Model Serving and Deployment](/concepts/mlflow-model-serving-and-deployment.md) - Deploying MLflow models as REST endpoints via Model Serving on Databricks, including validation before deployment and automatic updates based on model version availability.
* [MLflow Model Validation Before Deployment](/concepts/model-validation-before-deployment.md) - MLflow provides mlflow.models.predict to validate custom pyfunc models locally before deploying them to Model Serving to ensure they are capable of being served.
* [MLflow Models Tab Visualizations](/concepts/mlflow-models-tab-visualizations.md) - Chart view features available for comparing models from the Models tab in MLflow 3, extending the same visualization tools used for experiment runs.
* [MLflow Nested Run Structure for Hyperparameter Tuning](/concepts/mlflow-nested-run-structure-for-hyperparameter-tuning.md) - How MLlib automated MLflow tracking organizes tuning results: a parent run for the CrossValidator/TrainValidationSplit and child runs for each hyperparameter setting tested.
* [MLflow on Databricks](/concepts/mlflow-on-databricks.md) - Integrated MLflow platform on Databricks for experiment tracking, model management in Unity Catalog, model deployment, and evaluation throughout the ML lifecycle.
* [MLflow OpenAI Autolog](/concepts/mlflow-openai-autolog.md) - Automatic instrumentation of OpenAI API calls by MLflow to capture traces, inputs, and outputs without manual logging code.
* [MLflow OpenAI Autolog for Ollama Tracing](/concepts/mlflow-openai-autolog-for-ollama-tracing.md) - Technique using mlflow.openai.autolog() to automatically capture and record Ollama LLM interactions as traces in MLflow, leveraging Ollama's OpenAI API compatibility.
* [MLflow OpenAI Autolog for Tracing](/concepts/mlflow-openai-autolog.md) - Automatic instrumentation of OpenAI client calls via mlflow.openai.autolog() to capture traces without manual instrumentation, enabling trace-based evaluation workflows.
* [MLflow OpenAI Autologging](/concepts/mlflow-openai-autolog.md) - Automatic instrumentation of OpenAI client calls via mlflow.openai.autolog() to capture traces for GenAI evaluation.
* [MLflow OpenAI Autologging and Tracing](/concepts/mlflow-openai-autolog.md) - Automatic instrumentation of OpenAI API calls via mlflow.openai.autolog() and @mlflow.trace decorator for capturing LLM request/response traces during evaluation.
* [MLflow OpenAI Autologging for Tracing](/concepts/mlflow-openai-autolog.md) - Automatically instrumenting OpenAI-based applications with MLflow Tracing via mlflow.openai.autolog(), which captures traces for use as scorer inputs during evaluation.
* [MLflow OpenTelemetry Compatibility](/concepts/opentelemetry-compatibility.md) - MLflow traces conform to the OpenTelemetry trace specification, enabling export to any OpenTelemetry-compatible observability solution.
* [MLflow OpenTelemetry Metrics Export](/concepts/mlflow-opentelemetry-metrics-export.md) - MLflow can export OpenTelemetry metrics such as span durations to a configured metrics endpoint using OTLP protocol.
* [MLflow Parallel Coordinates Plot](/concepts/parallel-coordinates-plot.md) - A chart type in MLflow that visualizes relationships between multiple parameters and metrics across runs, helping identify parameter settings that influence model performance.
* [MLflow Production Monitoring](/concepts/mlflow-production-monitoring.md) - Production-phase monitoring system that reuses the same judges and scorers from development evaluation to ensure consistent quality assessment
* [MLflow Production Monitoring for Long-term Trace Storage](/concepts/mlflow-production-monitoring-for-long-term-trace-storage.md) - Beta feature for storing GenAI agent traces in Delta tables via Production Monitoring, offering durable storage, no trace size limits, automated quality assessment, and ~15-minute sync intervals.
* [MLflow Production Tracing for External Deployments](/concepts/mlflow-production-tracing-for-external-deployments.md) - Framework for capturing execution traces from GenAI agents deployed outside Databricks and sending them to a Databricks workspace for observability in the MLflow UI.
* [MLflow Production Tracing on Databricks](/concepts/mlflow-production-tracing-on-databricks.md) - Automatic capture of GenAI application traces when deployed on Databricks, with traces logged to MLflow experiments for real-time viewing and optional long-term Delta table storage via Production Monitoring.
* [MLflow Production Tracing with Unity Catalog](/concepts/mlflow-production-tracing-with-unity-catalog.md) - For large-scale trace analysis in production, use production monitoring to log traces to Delta tables in Unity Catalog, removing the 1,000-trace and 100,000-trace limits.
* [MLflow Project Entry Points](/concepts/mlflow-project-entry-points.md) - Executable commands within an MLflow Project defined in the MLproject file, with optional parameters declared with types and defaults, and commanded via 'command' field in YAML.
* [MLflow Project Environment Specification](/concepts/mlflow-project-environment-specification.md) - The software environment for an MLflow Project is defined via python_env.yaml for virtualenv, conda_env for Conda, or docker_env for Docker; if absent, a minimal Python virtualenv is used.
* [MLflow Projects](/concepts/mlflow-projects.md) - A format for packaging data science code in a reusable and reproducible way, including API and CLI tools for running projects with automatic parameter and git commit tracking.
* [MLflow Projects Deprecation on Databricks](/concepts/mlflow-projects-deprecation-on-databricks.md) - MLflow Projects is no longer supported on Databricks; documentation is retired and users must migrate to alternative approaches, especially for Databricks Runtime 13.0 ML and above.
* [MLflow Prompt Optimization API](/concepts/mlflow-prompt-optimization-api.md) - The `mlflow.genai.optimize_prompts()` API for automatically improving prompts using evaluation metrics and training data, part of MLflow's GenAI capabilities.
* [MLflow prompt registration and versioning](/concepts/mlflow-prompt-registration-and-versioning.md) - The process of registering prompt templates in MLflow's prompt registry, creating versioned prompts that can be loaded and used in prediction functions.
* [MLflow Prompt Registry](/concepts/prompt-registry.md) - A centralized system within MLflow for storing, versioning, and managing prompt templates used in GenAI applications, backed by Unity Catalog.
* [MLflow pyfunc (Python Function)](/concepts/mlflow-pyfunc-python-function.md) - MLflow's Python function format that allows deploying any Python code or model as a serveable model on Databricks Model Serving
* [MLflow pyfunc Custom Python Model](/concepts/mlflow-pyfunc-custom-python-model.md) - MLflow's Python function (pyfunc) format allows deploying arbitrary Python code as a model, including preprocessing, postprocessing, branching logic, or unsupported frameworks.
* [MLflow pyfunc load_context Pattern](/concepts/mlflow-pyfunc-load_context-pattern.md) - The design pattern of separating one-time initialization (load_context) from per-request prediction logic (predict) in custom MLflow pyfunc models
* [MLflow PyFunc Model Validation](/concepts/mlflow-pyfunc-model-validation.md) - The practice of using mlflow.models.predict to validate custom pyfunc models before deploying them to a serving endpoint, ensuring the model is capable of being served.
* [MLflow PyFunc Wrapper for Model Serving](/concepts/mlflow-pyfunc-wrapper-for-model-serving.md) - Wrapping a TwoTower model in an MLflow PythonModel (PyFunc) to abstract away TorchRec internals and enable simple dictionary-based serving
* [MLflow PyFunc Wrapping for Model Serving](/concepts/mlflow-pyfunc-wrapping-for-model-serving.md) - Wrapping a PyTorch recommendation model in an MLflow PythonModel to simplify the inference interface from tensor-based batches to dictionary-based list inputs for serving.
* [MLflow PySpark ML Autologging](/concepts/mlflow-pyspark-ml-autologging.md) - The recommended replacement for MLlib automated MLflow tracking, enabled by calling mlflow.pyspark.ml.autolog() and enabled by default with Databricks Autologging on Databricks Runtime 10.1 ML and above.
* [MLflow Python SDK for Tracing](/concepts/mlflow-python-sdk-for-tracing.md) - Python-based APIs in MLflow for instrumenting generative AI applications with tracing, supporting autolog, decorator, and low-level patterns.
* [MLflow REST API](/concepts/mlflow-rest-api.md) - The open-source REST API for MLflow that enables creating, listing, and getting experiments and runs, plus logging parameters, metrics, and artifacts
* [MLflow Review App](/concepts/mlflow-review-app.md) - A structured UI tool within MLflow for collecting domain expert feedback and expectations on GenAI application traces via labeling sessions and schemas.
* [MLflow Review App for Domain Expert Feedback](/concepts/mlflow-review-app-for-domain-expert-feedback.md) - MLflow's Review App provides two approaches for collecting domain expert feedback: interactive testing with a Chat UI (real-time conversational testing of deployed apps) and labeling existing traces (systematic review of captured traces to assess outputs and define ground truth).
* [MLflow Run](/concepts/mlflow-run.md) - A single execution of model code in MLflow that records parameters, metrics, tags, artifacts, and source information for experiment tracking.
* [MLflow Run Comparison](/concepts/mlflow-run-comparison.md) - A feature to compare multiple MLflow runs from single or multiple experiments in tabular format with parameter and metric side-by-side views, diff toggling, and visualization support.
* [MLflow Run Download and Export](/concepts/mlflow-run-download-and-export.md) - The ability to download MLflow run data as CSV files from the UI or programmatically, and to copy runs between workspaces using the MLflow Export-Import community tool.
* [MLflow Run Filtering and Search](/concepts/mlflow-run-filtering-and-search.md) - The ability to search and filter MLflow runs using queries on parameters, metrics, tags, state, creation time, and datasets to locate specific experiment executions.
* [MLflow Run Filtering and Sorting](/concepts/mlflow-run-filtering-and-sorting.md) - Filtering MLflow runs based on parameter values, metric values, or tags, and sorting runs by selected parameters to control what appears in chart views.
* [MLflow Run Filtering, Sorting and Grouping](/concepts/mlflow-run-filtering-sorting-and-grouping.md) - Capabilities in the MLflow UI to filter runs by parameter/metric values or tags, sort runs by parameters, and group runs by parameter values for comparative analysis.
* [MLflow Run Grouping](/concepts/mlflow-run-grouping.md) - Grouping MLflow runs by one or more parameter values using the Group by dropdown to organize chart displays.
* [MLflow Run Lifecycle Management](/concepts/mlflow-run-lifecycle-management.md) - Operations for managing MLflow runs including renaming, deleting (with bulk deletion by creation time), restoring (with bulk restore by deletion time), and a 30-day retention period for deleted runs.
* [MLflow Run Management Actions](/concepts/mlflow-run-management-actions.md) - Operations for managing MLflow runs including deletion, comparison, and tag management through a checkbox-based selection interface in the chart view.
* [MLflow Run Selection and Batch Operations](/concepts/mlflow-run-selection-and-batch-operations.md) - The ability to select multiple runs in the MLflow UI to perform batch operations such as deletion, comparison, or tag management.
* [MLflow Run Selection and Filtering](/concepts/mlflow-run-selection-and-filtering.md) - Mechanisms in the MLflow UI to select, show/hide, filter, sort, and group runs for comparative analysis in charts, including search by parameter/metric values or tags.
* [MLflow Run Selection and Management](/concepts/mlflow-run-selection-and-management.md) - The ability to select, show/hide, delete, compare, and add/remove tags for individual runs within the MLflow experiment tracking UI.
* [MLflow Run Tagging System](/concepts/mlflow-run-tagging-system.md) - Key-value metadata tags that can be added, edited, and deleted on MLflow runs to enable searching and organizing experiment executions.
* [MLflow Run Tracking on AI Runtime](/concepts/mlflow-integration-in-ai-runtime.md) - The integration between AI Runtime CLI training jobs and MLflow for tracking experiment runs, accessible via the Jobs run page.
* [MLflow Run vs Logged Model distinction](/concepts/mlflow-run-vs-logged-model-distinction.md) - MLflow 3 separates the concept of a Run (training/evaluation job execution) from a Logged Model (the persistent model artifact with its own metadata lifecycle), enabling clearer tracking across training, evaluation, and deployment.
* [MLflow Safety Scorer](/concepts/mlflow-safety-scorer.md) - Built-in MLflow scorer that evaluates LLM outputs for safety and harmful content, used as part of GenAI application evaluation.
* [MLflow Scorer Error Handling](/concepts/mlflow-scorer-error-handling.md) - Two approaches for handling errors in custom scorers: returning AssessmentError in a Feedback object, or letting exceptions propagate for MLflow to handle gracefully.
* [MLflow Scorers](/concepts/mlflow-scorers.md) - Pluggable evaluation criteria in MLflow that assess LLM outputs against custom guidelines or built-in policies, including Guidelines scorer and Safety scorer.
* [MLflow Scorers (Guidelines & Safety)](/concepts/mlflow-scorers.md) - Modular evaluation criteria in MLflow for assessing GenAI outputs against custom guidelines and built-in safety checks.
* [MLflow Scorers (Guidelines and Safety)](/concepts/mlflow-scorers.md) - Built-in evaluation criteria in MLflow for assessing GenAI outputs, including Guidelines scorer for custom rules and Safety scorer for content safety checks.
* [MLflow Serverless Workloads](/concepts/mlflow-serverless-workloads.md) - Serverless workloads that MLflow can run against an experiment, including scheduled scorers, synthetic evaluation set generation, and agent evaluation.
* [MLflow Serverless Workloads for Experiments](/concepts/mlflow-serverless-workloads.md) - Serverless workloads triggered by MLflow experiments, including scheduled scorers, synthetic evaluation set generation, and agent evaluation.
* [MLflow set_experiment_trace_location API](/concepts/mlflow-set_experiment_trace_location-api.md) - An MLflow API that links a Unity Catalog schema location to an MLflow experiment, defining where incoming OTLP spans will be stored as trace tables.
* [MLflow Span Analysis and Hierarchy](/concepts/mlflow-span-analysis-and-hierarchy.md) - Techniques for analyzing span trees including parent-child relationships, critical path detection, and per-type timing statistics.
* [MLflow Span Type Labeling via gen_ai.operation.name](/concepts/mlflow-span-type-labeling-via-gen_aioperationname.md) - Setting the gen_ai.operation.name attribute on spans to classify operations (chat, etc.) so MLflow renders the correct span type instead of UNKNOWN.
* [MLflow Spans](/concepts/mlflow-spans.md) - Building blocks of traces representing individual operations with properties like span_id, name, type, timing, status, inputs/outputs, and parent-child relationships.
* [MLflow Standard Metadata Fields for Tracing](/concepts/mlflow-standard-metadata-fields-for-tracing.md) - Predefined metadata keys like mlflow.trace.user and mlflow.trace.session that MLflow recognizes for filtering, grouping, and UI integration in trace analytics.
* [MLflow Supported Tracing Libraries](/concepts/mlflow-supported-tracing-libraries.md) - The collection of 20+ supported libraries that can be automatically instrumented with mlflow.autolog() for tracing.
* [MLflow System Metrics](/concepts/mlflow-system-metrics.md) - System-level metrics (CPU, memory, GPU utilization) recorded by MLflow during runs and exposed via system tables for workspace-wide monitoring and analysis.
* [MLflow System Metrics Monitoring](/concepts/mlflow-system-metrics.md) - Recording and analyzing system-level metrics such as CPU, memory, and GPU utilization across MLflow experiments to detect inefficiencies
* [MLflow System Tables](/concepts/mlflow-system-tables.md) - System tables in Databricks that capture MLflow experiment metadata across all workspaces in a region, enabling large-scale analytics via SQL and dashboards.
* [MLflow System Tables Dashboards](/concepts/mlflow-system-tables-dashboards.md) - Building dashboards from MLflow metadata stored in Databricks system tables to analyze experiments and runs across an entire workspace
* [MLflow System Tables for Experiment Analysis](/concepts/mlflow-system-tables-for-experiment-data.md) - The availability of MLflow experiment and run metadata in Databricks system tables, enabling use of Databricks SQL and lakehouse tooling for custom visualization and analysis.
* [MLflow System Tables for Experiment Data](/concepts/mlflow-system-tables-for-experiment-data.md) - MLflow metadata stored in Databricks system tables, enabling SQL-based querying and lakehouse visualization of experiment data.
* [MLflow System Tables for Visualization](/concepts/mlflow-system-tables-for-visualization.md) - Databricks system tables that expose MLflow metadata for experiments and runs, enabling custom visualizations using Databricks SQL and lakehouse tooling.
* [MLflow Third-Party Scorer Architecture](/concepts/mlflow-third-party-scorer-architecture.md) - MLflow's integration layer that wraps evaluation metrics from third-party frameworks as MLflow scorers usable with mlflow.genai.evaluate()
* [MLflow to Unity Catalog Trace Migration](/concepts/mlflow-to-unity-catalog-trace-migration.md) - A process for copying traces, spans, assessments, tags, and metadata from an MLflow experiment into Unity Catalog Delta tables, removing trace storage limits and enabling fine-grained access control.
* [MLflow Token Usage Tracking](/concepts/mlflow-token-usage-tracking.md) - Mechanism for tracking token consumption in LLM calls within traces, aggregating input_tokens, output_tokens, and total_tokens from provider APIs at both trace and span level.
* [MLflow Trace Analysis](/concepts/mlflow-trace-analysis.md) - Capability to investigate failing traces, examine span trees, pinpoint root causes, analyze latency, and identify bottlenecks in GenAI agent execution
* [MLflow Trace and Autologging](/concepts/mlflow-trace-and-autologging.md) - Automatic tracing of GenAI application calls (e.g., OpenAI) via mlflow.openai.autolog() to capture inputs, outputs, and metadata for evaluation and debugging.
* [MLflow Trace Assessment](/concepts/mlflow-trace-assessment.md) - Structured feedback, scores, or ground truth attached to traces and spans for quality evaluation in MLflow, with configurable type, name, data type, value, and rationale.
* [MLflow Trace Assessments](/concepts/trace-assessments.md) - Evaluations and feedback attached to traces, searchable by name, type (feedback/expectation), span ID, and source, with support for overrides and rationales.
* [MLflow Trace Captured Data for Anthropic](/concepts/mlflow-trace-captured-data-for-anthropic.md) - The specific telemetry automatically captured by MLflow Traces from Anthropic SDK calls, including prompts, responses, latencies, model names, metadata, function calls, and exceptions
* [MLflow Trace Captured Metadata](/concepts/mlflow-trace-captured-metadata.md) - Information automatically captured by MLflow traces for Databricks Foundation Model calls including prompts, completions, latencies, model name/endpoint, temperature/max_tokens parameters, function calls, and exceptions.
* [MLflow Trace Collection Verification](/concepts/mlflow-trace-collection-verification.md) - Programmatic method to verify that traces are being collected from an external deployment by checking MLflow experiment connectivity and searching for recent traces.
* [MLflow Trace Context](/concepts/mlflow-trace-context.md) - The mechanism for adding metadata, tags, and contextual information to MLflow traces to enable debugging, monitoring, and analysis of GenAI applications.
* [MLflow Trace Context Best Practices](/concepts/mlflow-trace-context-best-practices.md) - Guidelines for effectively adding context to traces, including consistent ID formats, session boundaries, environment variables, combining context types, and regular analysis.
* [MLflow Trace Context Metadata](/concepts/mlflow-trace-context-metadata.md) - Standardized metadata fields (mlflow.trace.user, mlflow.trace.session, mlflow.source.type, etc.) that can be attached to traces to track users, sessions, environments, and versions in GenAI applications.
* [MLflow Trace Data Captured for Bedrock](/concepts/mlflow-trace-data-captured-for-bedrock.md) - MLflow automatically captures prompts, completion responses, latencies, model names, metadata (temperature, max_tokens), function calling info, and exceptions from Amazon Bedrock API calls.
* [MLflow Trace Data Export](/concepts/mlflow-trace-data-export.md) - Methods for converting MLflow Trace objects to dictionaries, JSON strings, and Pandas DataFrame rows for serialization and analysis.
* [MLflow trace decorator](/concepts/mlflow-trace-decorator.md) - A decorator (@mlflow.trace) for instrumenting functions with span types (e.g., RETRIEVER) to enable observability and tracing in GenAI application evaluation.
* [MLflow Trace Detail View](/concepts/mlflow-trace-detail-view.md) - A deep-dive view into an individual trace with two main tabs: Summary (root span inputs/outputs, exceptions) and Details & Timeline (span hierarchy/waterfall, per-span details including Chat, Inputs/Outputs, Attributes, Events tabs, and an Assessments panel).
* [MLflow Trace Error Monitoring](/concepts/mlflow-trace-error-monitoring.md) - Patterns for programmatically monitoring and analyzing trace errors using mlflow.search_traces() with status filters, time windows, and error aggregation by function name.
* [MLflow Trace Experiment Linking](/concepts/mlflow-trace-experiment-linking.md) - Using mlflow.tracing.enablement.set_experiment_trace_location to bind an MLflow experiment to a Unity Catalog catalog/schema, defining where incoming OTel traces will be stored.
* [MLflow Trace Export Modes](/concepts/mlflow-trace-export-modes.md) - MLflow supports three trace export modes: MLflow Tracking Server only, OpenTelemetry Collector only, or dual export to both simultaneously.
* [MLflow Trace Generation and Querying](/concepts/mlflow-trace-generation-and-querying.md) - The process of using mlflow.genai.evaluate() to generate traces from an AI application, then retrieving them as a Pandas DataFrame via mlflow.search_traces() for downstream evaluation.
* [MLflow Trace Immutability Model](/concepts/mlflow-trace-immutability-model.md) - Distinction between mutable tags and immutable metadata in MLflow tracing, where metadata keys are set-once and ignored on update attempts.
* [MLflow Trace List View](/concepts/mlflow-trace-list-view.md) - A tabular overview of all captured traces in an MLflow Experiment, displaying sortable columns such as Trace ID, Request, Response, Session, User, Execution time, State, Source, and custom assessment/tag columns.
* [MLflow Trace Management Operations](/concepts/mlflow-trace-management-operations.md) - The set of programmatic operations exposed by the MLflow MCP server for searching, retrieving, analyzing, and managing MLflow traces
* [MLflow Trace Metadata](/concepts/trace-metadata.md) - Immutable, write-once key-value pairs attached to traces at logging time for fixed information such as model version and environment.
* [MLflow Trace Object](/concepts/mlflow-trace-object.md) - Core entity representing a complete record of an LLM or application execution, composed of TraceInfo (metadata) and TraceData (execution data).
* [MLflow Trace object in scoring](/concepts/mlflow-trace-object-in-scoring.md) - Accessing full MLflow Trace objects within custom scorers to retrieve spans, timing, inputs, outputs, and attributes for fine-grained metric calculation.
* [MLflow Trace Object Structure](/concepts/mlflow-trace-object-structure.md) - The MLflow Trace object is composed of two main components: TraceInfo (metadata about the trace) and TraceData (actual execution data including spans and request/response payloads).
* [MLflow Trace Query and Reuse](/concepts/mlflow-trace-query-and-reuse.md) - The pattern of using mlflow.search_traces() to query stored evaluation traces and pass the resulting DataFrame directly to mlflow.genai.evaluate() for rapid metric iteration.
* [MLflow Trace Requirements for Evaluation](/concepts/mlflow-trace-requirements-for-judges.md) - Prerequisites for built-in judges to work: RelevanceToQuery requires 'inputs' and 'outputs' on the Trace's root span, while RetrievalRelevance requires at least one span with span_type 'RETRIEVER'.
* [MLflow Trace Requirements for Judges](/concepts/mlflow-trace-requirements-for-judges.md) - Built-in judges require specific MLflow Trace span data, such as inputs/outputs on the root span for RelevanceToQuery or RETRIEVER-type spans for RetrievalRelevance
* [MLflow Trace Requirements for LLM Judges](/concepts/mlflow-trace-requirements-for-llm-judges.md) - Technical prerequisites for MLflow LLM judges including that inputs and outputs must be on the Trace's root span (for RelevanceToQuery) and that the Trace must contain at least one span with span_type set to RETRIEVER (for RetrievalRelevance).
* [MLflow Trace Search and Filtering](/concepts/mlflow-trace-search-and-filtering.md) - Capabilities to narrow down traces using a search bar (by Request content), structured Filters dropdown (by attributes, assessments, state, tags), Sort dropdown, and Columns dropdown to customize the trace list.
* [MLflow Trace Search and Retrieval API](/concepts/mlflow-trace-search-and-retrieval-api.md) - Using MLflow Python SDK (search_traces, get_trace) with SQL warehouse configuration to query OpenTelemetry traces stored in Unity Catalog.
* [MLflow Trace Search Limitations](/concepts/mlflow-trace-search-limitations.md) - The trace list returns only the most recent 1,000 traces by default; older traces require narrowed time filters or migration to Unity Catalog to remove limits.
* [MLflow Trace Span for Retrieval](/concepts/mlflow-trace-span-for-retrieval.md) - A pattern using @mlflow.trace(span_type='RETRIEVER') to instrument and trace document retrieval functions for observability during RAG evaluation.
* [MLflow Trace Storage in Unity Catalog](/concepts/mlflow-trace-storage-in-unity-catalog.md) - Mechanism for storing MLflow AI/agent traces in Unity Catalog tables, supporting structured querying, annotations, and multiple trace destinations per schema
* [MLflow Trace Structure](/concepts/mlflow-trace-structure.md) - The hierarchical structure of a trace consisting of root and child spans, each containing attributes (model name, token counts, timing), inputs, and outputs.
* [MLflow Trace Tag Management via UI](/concepts/mlflow-ui-trace-tag-editing.md) - Graphical interface in the MLflow UI for updating or deleting tags on traces by navigating to the trace tab and using the pencil icon.
* [MLflow Trace Tags](/concepts/mlflow-trace-tags.md) - Mutable key-value pairs attached to MLflow traces for dynamic metadata like user feedback, review status, and data quality assessments.
* [MLflow Trace Timeout Mechanism](/concepts/mlflow-trace-timeout-mechanism.md) - The MLFLOW_TRACE_TIMEOUT_SECONDS environment variable can halt stuck traces with ERROR status to prevent infinite 'in progress' traces.
* [MLflow Trace Token Tracking](/concepts/mlflow-trace-token-tracking.md) - Token usage tracking for LLM calls within MLflow traces, accessible via span attributes and trace info
* [MLflow Trace UI Label Column](/concepts/mlflow-trace-ui-label-column.md) - Labels added to traces through the MLflow Experiment UI appear as new columns in the Traces tab, enabling trace-level filtering and organization by annotation value.
* [MLflow Trace-based Evaluation](/concepts/mlflow-trace-based-evaluation.md) - Technique of passing a Pandas DataFrame of precomputed MLflow traces directly to mlflow.genai.evaluate() instead of a predict_fn, enabling rapid scorer iteration without re-running the application.
* [MLflow Trace-based Evaluation Requirements](/concepts/mlflow-trace-based-evaluation-requirements.md) - The structural requirements that MLflow traces must meet for built-in judges to function, including specific span types (e.g., RETRIEVER) and root span input/output fields.
* [MLflow Trace-Based Evaluation Workflow](/concepts/mlflow-trace-based-evaluation-workflow.md) - End-to-end workflow: instrument GenAI apps with MLflow tracing, collect human feedback on traces, route traces to expert labeling sessions, and use labeled traces for automated evaluation.
* [MLflow Trace-based Scorer Development Workflow](/concepts/iterative-scorer-development-workflow.md) - A four-step iterative workflow for developing scorers: define evaluation data, generate traces, store traces, then iterate on scorers without rerunning the application.
* [MLflow Tracing](/concepts/mlflow-tracing.md) - Framework for recording and analyzing trace data from LLM and AI application calls
* [MLflow Tracing Approach Selection Guide](/concepts/tracing-approach-selection-guide.md) - Guidance on choosing between automatic, manual, or combined tracing, recommending starting with automatic tracing for fastest results
* [MLflow Tracing Archival API](/concepts/mlflow-tracing-archival-api.md) - Python API functions (enable_databricks_trace_archival and disable_databricks_trace_archival) in mlflow.tracing.archival that control whether traces from an experiment are saved to a Unity Catalog Delta table.
* [MLflow Tracing Concepts and Workflow](/concepts/mlflow-tracing-concepts-and-workflow.md) - The tracing workflow includes understanding how MLflow captures and organizes trace data, using the Trace UI to analyze application behavior, and setting up quality assessment for Gemini-powered applications.
* [MLflow Tracing Enable and Disable API](/concepts/mlflow-tracing-enable-and-disable-api.md) - mlflow.tracing.disable() ceases trace data collection and mlflow.tracing.enable() re-enables tracing for instrumented models.
* [MLflow Tracing Enablement via Autolog](/concepts/mlflow-tracing-enablement-via-autolog.md) - A mechanism to enable or disable tracing for generative AI framework integrations (OpenAI, LangChain, LlamaIndex, AutoGen) using autolog functions.
* [MLflow Tracing Environment Configuration](/concepts/mlflow-tracing-environment-configuration.md) - Required environment variables (DATABRICKS_HOST, DATABRICKS_TOKEN, MLFLOW_TRACKING_URI, MLFLOW_EXPERIMENT_NAME) to connect an external application to a Databricks workspace for trace collection.
* [MLflow Tracing for Databricks Foundation Models](/concepts/mlflow-tracing-for-databricks-foundation-models.md) - Automatic tracing capability that captures LLM invocation metadata, prompts, responses, latencies, and exceptions for Databricks Foundation Model API calls via mlflow.openai.autolog().
* [MLflow Tracing for GenAI](/concepts/mlflow-tracing-for-genai.md) - MLflow capability for adding traces to Python and TypeScript generative AI applications
* [MLflow Tracing for GenAI Applications](/concepts/mlflow-tracing-for-genai-applications.md) - Instrumenting generative AI applications with MLflow tracing using @mlflow.trace decorators and automatic OpenAI client tracing via mlflow.openai.autolog()
* [MLflow Tracing for GenAI Apps](/concepts/mlflow-tracing-for-genai-apps.md) - Automatic instrumentation of GenAI applications using MLflow's @mlflow.trace decorator and mlflow.openai.autolog() to capture and visualize traces of LLM calls and retrieval steps.
* [MLflow Tracing for GenAI Observability](/concepts/mlflow-tracing-for-genai-observability.md) - Databricks' tracing framework for monitoring, analyzing, and debugging GenAI application behavior using MLflow
* [MLflow Tracing for GenAI on Databricks](/concepts/mlflow-tracing-for-genai-on-databricks.md) - Databricks MLflow 3 provides automatic tracing for GenAI frameworks including CrewAI, LangChain, and others
* [MLflow Tracing for Generative AI](/concepts/mlflow-tracing-for-generative-ai.md) - An MLflow feature that uses autolog to enable or disable tracing for generative AI model integrations including OpenAI, LangChain, LangGraph, LlamaIndex, and AutoGen.
* [MLflow Tracing for LLMs](/concepts/mlflow-tracing-for-llms.md) - Automatic instrumentation of LLM API calls using mlflow.openai.autolog() and @mlflow.trace decorator to capture inputs, outputs, and metadata for observability.
* [MLflow Tracing for txtai](/concepts/mlflow-tracing-for-txtai.md) - Automatic tracing capability for txtai workflows via MLflow's autolog function, capturing traces for LLM invocations, embeddings, and AI Search
* [MLflow Tracing for TypeScript](/concepts/mlflow-tracing-for-typescript.md) - MLflow tracing support for TypeScript generative AI applications, alongside Python support
* [MLflow Tracing in Databricks](/concepts/mlflow-tracing-in-databricks.md) - An autolog-based tracing feature for generative AI workloads that enables trace logging for LLM frameworks such as OpenAI, LangChain, LangGraph, LlamaIndex, and AutoGen.
* [MLflow Tracing in Databricks Notebooks](/concepts/mlflow-tracing-in-databricks-notebooks.md) - Seamless integration that displays interactive trace exploration directly in notebook cell outputs when MLflow Tracking URI is set to 'databricks', available in MLflow 2.20+, with programmatic enable/disable controls.
* [MLflow Tracing Integrations](/concepts/mlflow-tracing-integrations.md) - The set of popular GenAI libraries and frameworks that MLflow Tracing natively supports for automatic observability.
* [MLflow Tracing integrations library](/concepts/mlflow-tracing-integrations.md) - A collection of 20+ pre-built library integrations that provide automatic tracing for popular frameworks used in GenAI applications.
* [MLflow Tracing Latency Overhead](/concepts/mlflow-tracing-latency-overhead.md) - Traces are written asynchronously but still add minimal latency proportional to trace size; testing is recommended before production deployment.
* [MLflow Tracing Limitations and Scaling](/concepts/mlflow-tracing-limitations-and-scaling.md) - The trace list returns at most 1,000 traces; experiments not in Unity Catalog are capped at 100,000 traces; migrating to Unity Catalog removes both limits and enables full time-range search across all traces.
* [MLflow Tracing Limitations for Anthropic](/concepts/mlflow-tracing-limitations-for-anthropic.md) - Current constraints of the MLflow-Anthropic tracing integration: sync-only tracing (async added in 2.21.0), no multi-modal input recording, and serverless cluster autolog limitations
* [MLflow Tracing Multi-threading Behavior](/concepts/mlflow-tracing-multi-threading-behavior.md) - MLflow Tracing uses Python ContextVar, causing each thread to have its own trace context by default; additional steps are needed to combine multi-threaded traces into a single trace.
* [MLflow Tracing Prerequisites (Databricks)](/concepts/mlflow-tracing-prerequisites-databricks.md) - Required software setup for MLflow tracing on Databricks: mlflow[databricks]>=3.1.0, openai, and databricks-connect>=16.1, plus creating an MLflow experiment.
* [MLflow Tracing Prerequisites and Setup for Anthropic](/concepts/mlflow-tracing-prerequisites-and-setup-for-anthropic.md) - Required environment setup including MLflow version (>=3.1 recommended), Anthropic SDK installation, Databricks environment variables, and API key configuration for the Anthropic tracing integration
* [MLflow Tracing Rate Limits and Quotas](/concepts/mlflow-tracing-rate-limits-and-quotas.md) - Databricks workspaces impose rate limits and quotas on MLflow Tracing to ensure service stability and fair usage.
* [MLflow Tracing Recommendation Strategy](/concepts/mlflow-tracing-recommendation-strategy.md) - Recommended workflow of starting with automatic tracing and adding manual tracing when more control is needed.
* [MLflow Tracing Span Hierarchy](/concepts/mlflow-tracing-span-hierarchy.md) - The parent-child span structure generated by MLflow Tracing, where a parent function span contains child spans for individual LLM calls and sub-functions.
* [MLflow Tracing TypeScript SDK](/concepts/mlflow-tracing-typescript-sdk.md) - An npm package (mlflow-tracing) that brings MLflow Tracing observability to TypeScript and JavaScript Node.js applications with minimal code changes.
* [MLflow Tracing UI Panels](/concepts/mlflow-tracing-ui-panels.md) - MLflow provides a Chat tab for rich message visualization (supported by converse/converse_stream APIs) and an Inputs/Outputs tab for raw payload inspection, including configuration parameters.
* [MLflow Tracking](/concepts/mlflow-tracking.md) - Experiment tracking system for logging parameters, metrics, artifacts, and model provenance (dataset, code, environment) across training runs, enabling reproducibility, comparison, and deployment.
* [MLflow Tracking for Deep Learning](/concepts/mlflow-tracking-for-deep-learning.md) - Using MLflow on Databricks to track iterative deep learning training runs and model development as part of the MLflow ecosystem.
* [MLflow tracking URI](/concepts/mlflow-tracking-uri.md) - A configuration parameter that specifies which tracking server MLflow should log experiment data to, set via mlflow.set_tracking_uri() with a URI like 'databricks://remote-workspace-url'.
* [MLflow tracking URI configuration](/concepts/mlflow-tracking-uri-configuration.md) - Setting the tracking URI to point to a remote MLflow tracking server using mlflow.set_tracking_uri() to direct where experiment data is logged.
* [MLflow Transformers Flavor](/concepts/mlflow-transformers-flavor.md) - Logging Hugging Face models to MLflow using the MLflow transformers flavor for model management and versioning
* [MLflow TypeScript SDK for Tracing](/concepts/mlflow-typescript-sdk-for-tracing.md) - TypeScript-based APIs in MLflow for instrumenting generative AI applications with tracing, providing similar capabilities to the Python SDK.
* [MLflow TypeScript Tracing SDK](/concepts/mlflow-typescript-tracing-sdk.md) - MLflow tracing support for TypeScript generative AI applications alongside Python.
* [MLflow UI Trace Tag Editing](/concepts/mlflow-ui-trace-tag-editing.md) - Graphical interface in the MLflow UI for updating or deleting tags on traces via the trace tab, using a pencil icon.
* [MLflow UI Trace Tag Management](/concepts/mlflow-ui-trace-tag-editing.md) - Graphical method within the MLflow UI to view, update, and delete tags on existing traces using a pencil icon in the trace tab.
* [MLflow Unified Evaluation Interface](/concepts/mlflow-unified-evaluation-interface.md) - The ability to combine scorers from multiple third-party frameworks in a single mlflow.genai.evaluate() call with unified results visualization
* [MLflow User and Session Tracking](/concepts/mlflow-user-and-session-tracking.md) - Standardized metadata fields mlflow.trace.user and mlflow.trace.session for associating traces with specific users and grouping traces into multi-turn conversations.
* [MLflow UUID Name Conflict Resolution](/concepts/mlflow-uuid-name-conflict-resolution.md) - MLflow's behavior when fit() is called multiple times within the same active run: it appends a UUID to parameter and tag names to resolve conflicts.
* [MLFLOW_LOCK_MODEL_DEPENDENCIES](/concepts/mlflow_lock_model_dependencies.md) - An environment variable in MLflow 3 that, when set to 'true', causes MLflow to capture both direct and transitive Python dependencies when logging a model.
* [MLflow-DeepEval Integration](/concepts/mlflow-deepeval-integration.md) - MLflow integrates DeepEval metrics as scorers for use with mlflow.genai.evaluate(), enabling standardized LLM evaluation within the MLflow ecosystem.
* [MLflow-DeepEval Scorer Integration](/concepts/mlflow-deepeval-integration.md) - MLflow integrates with DeepEval to allow using DeepEval metrics as MLflow scorers for LLM evaluation workflows.
* [mlflow-experiment data source](/concepts/mlflow-experiment-data-source.md) - A Spark DataFrameReader API on Databricks that loads MLflow experiment run data into a DataFrame for analysis, comparison, and dashboarding.
* [mlflow-skinny vs full mlflow for Model Serving](/concepts/mlflow-skinny-vs-mlflow-on-databricks.md) - A Databricks-specific consideration where Databricks Runtime ML includes mlflow-skinny by default, requiring explicit specification of mlflow in pip_requirements to avoid container build failures in Model Serving.
* [Mlflow-Skinny vs Full Mlflow in Databricks Runtime ML](/concepts/mlflow-skinny-vs-mlflow-on-databricks.md) - DBR ML ships with mlflow-skinny by default, but Model Serving requires the full mlflow package in conda.yaml; pip_requirements must specify mlflow explicitly.
* [MLflow-Skinny vs Full MLflow on Databricks](/concepts/mlflow-skinny-vs-mlflow-on-databricks.md) - Databricks Runtime ML includes mlflow-skinny by default, not full mlflow; custom pyfunc models must explicitly specify mlflow in pip_requirements to ensure Model Serving container compatibility.
* [mlflow-skinny vs mlflow Compatibility on Databricks](/concepts/mlflow-skinny-vs-mlflow-on-databricks.md) - The critical requirement that Databricks Model Serving needs the full mlflow package (not mlflow-skinny) in conda.yaml when deploying custom pyfunc models from Databricks Runtime ML
* [mlflow-skinny vs mlflow on Databricks](/concepts/mlflow-skinny-vs-mlflow-on-databricks.md) - Databricks Runtime ML includes mlflow-skinny by default, but Model Serving requires the full mlflow package; custom pyfunc models must explicitly specify mlflow in pip_requirements
* [mlflow-skinny vs mlflow Serving Compatibility](/concepts/mlflow-skinny-vs-mlflow-serving-compatibility.md) - Databricks Runtime ML ships with mlflow-skinny by default; custom pyfunc models must explicitly specify the full mlflow package in pip_requirements to be deployable on Model Serving.
* [mlflow-tracing Package](/concepts/mlflow-tracing-package.md) - Production-optimized Python package for MLflow tracing with minimal dependencies and better performance, requiring MLflow 3
* [mlflow-tracing Production Package](/concepts/mlflow-tracing-production-package.md) - A lightweight, optimized pip package for production deployments of MLflow tracing, distinct from the development package and required for feedback collection in MLflow 3.
* [mlflow-txtai extension](/concepts/mlflow-txtai-extension.md) - A Python package (mlflow-txtai) that provides the integration bridge between MLflow tracing and txtai workflows
* [mlflow.ag2.autolog()](/concepts/mlflowag2autolog.md) - A one-line API call that enables automatic tracing of all AG2 agent interactions, including tool calls, LLM calls, and message exchanges.
* [mlflow.autolog() Configuration](/concepts/mlflowautolog-configuration.md) - The Python API function used to enable, disable, and customize Databricks Autologging behavior with parameters like log_models, log_input_examples, and log_model_signatures.
* [mlflow.autolog() Default Configuration](/concepts/mlflowautolog-default-configuration.md) - Default parameter values for mlflow.autolog() on Databricks, including log_models, log_model_signatures, and disabling of input examples
* [mlflow.autolog() for GenAI Tracing](/concepts/autologging-for-genai-tracing.md) - One-line automatic tracing for generative AI applications using mlflow.<library>.autolog(), supporting 20+ libraries and frameworks out of the box.
* [mlflow.autolog() for Generative AI Tracing](/concepts/autologging-for-genai-tracing.md) - One-line automatic tracing of generative AI applications using mlflow.<library>.autolog(), supporting 20+ frameworks out of the box.
* [mlflow.genai.evaluate()](/concepts/mlflowgenaievaluate.md) - The Python API function that automatically creates an evaluation run when called with a dataset, prediction function, and scorers.
* [mlflow.langchain.autolog() for LangGraph](/concepts/mlflowlangchainautolog-for-langgraph.md) - The API call that enables automatic trace capture for LangGraph workflows, built on the LangChain autolog integration
* [mlflow.models.predict API](/concepts/mlflowmodelspredict-api.md) - MLflow API for testing offline model predictions in a simulated serving environment before endpoint deployment.
* [mlflow.openai.autolog()](/concepts/mlflowopenaiautolog.md) - Automatic instrumentation for OpenAI SDK calls that captures request details, responses, token counts, and timing without manual instrumentation code.
* [mlflow.pyfunc.log_model for Dependency Management](/concepts/mlflowpyfunclog_model-for-dependency-management.md) - Using mlflow.pyfunc.log_model to log models from libraries without built-in MLflow flavors, with automatic dependency inference and support for extra_pip_requirements and pip_requirements parameters.
* [mlflow.search_traces() API](/concepts/mlflowsearch_traces-api.md) - Primary Python API for programmatically querying and filtering MLflow traces stored in tracking servers, inference tables, or Unity Catalog tables.
* [mlflow.set_trace_tag and mlflow.delete_trace_tag](/concepts/mlflowset_trace_tag-and-mlflowdelete_trace_tag.md) - Post-hoc API methods to set or remove tags on already-completed MLflow traces stored in the backend, using the trace ID.
* [mlflow.set_trace_tag and mlflow.delete_trace_tag APIs](/concepts/mlflowset_trace_tag-and-mlflowdelete_trace_tag.md) - Methods to set or remove tags on a trace that has already been completed and logged, operating on the backend store.
* [mlflow.smolagents.autolog](/concepts/mlflowsmolagentsautolog.md) - The API function that enables automatic trace capture for Smolagents agent runs
* [mlflow.start_span() Context Manager](/concepts/mlflowstart_span-context-manager.md) - A context manager API in MLflow for creating spans around arbitrary code blocks, providing fine-grained control over tracing beyond function-level decorators.
* [mlflow.update_current_trace API](/concepts/mlflowupdate_current_trace-api.md) - Function used during trace execution to add metadata or tags to the currently active trace via fluent or automatic tracing APIs.
* [mlflow.workload_creation_policy_id Tag](/concepts/mlflowworkload_creation_policy_id-tag.md) - An MLflow experiment tag used to specify the serverless budget policy ID that MLflow will use for serverless workloads created from that experiment.
* [MlflowClient Trace Lifecycle Management](/concepts/mlflowclient-trace-lifecycle-management.md) - Low-level API for explicit, fine-grained control over trace creation, span management, and trace completion using the MlflowClient class.
* [MLlib Automated MLflow Tracking](/concepts/mllib-automated-mlflow-tracking.md) - A deprecated Databricks feature that automatically logs hyperparameters and evaluation metrics to MLflow when tuning Apache Spark MLlib models using CrossValidator or TrainValidationSplit, without requiring explicit API calls.
* [MLlib Cross-Validation for XGBoost](/concepts/mllib-cross-validation-for-xgboost.md) - Using MLlib cross-validation tools to tune hyperparameters of XGBoost models in Scala
* [MLlib Cross-Validation for XGBoost Hyperparameter Tuning](/concepts/mllib-cross-validation-for-xgboost-hyperparameter-tuning.md) - Using Spark MLlib's cross-validation functionality to tune XGBoost model hyperparameters
* [MLlib ML Pipeline for XGBoost](/concepts/mllib-ml-pipeline-for-xgboost.md) - Embedding XGBoost models into MLlib ML pipelines to enable distributed model training workflows
* [MLlib Pipelines API](/concepts/mllib-pipelines-api.md) - A high-level API in Apache Spark MLlib for constructing and tuning machine learning workflows as composable pipeline stages.
* [MLOps (Machine Learning Operations)](/concepts/mlops-machine-learning-operations.md) - A set of processes and automated steps for managing code, data, and models to improve performance, stability, and long-term efficiency of ML systems, combining DevOps, DataOps, and ModelOps.
* [MLOps Stacks](/concepts/mlops-stacks.md) - Infrastructure-as-code templates for automating repeatable ML pipeline promotion from development to production on Databricks.
* [MLOps Stacks for Automated ML Pipelines](/concepts/mlops-stacks.md) - Infrastructure-as-code templates enabling automated, repeatable promotion from development to production, integrated with Unity Catalog and AI Gateway for full governance.
* [MLOps Stacks on Databricks](/concepts/mlops-stacks.md) - Infrastructure-as-code templates on Databricks for automating repeatable ML model promotion from development to production with CI/CD.
* [MLOps Three-Workspace Architecture](/concepts/mlops-three-workspace-architecture.md) - The recommended Databricks MLOps pattern using separate development, staging, and production Databricks workspaces to isolate model development, testing, and production deployment.
* [MLproject File Format](/concepts/mlproject-file-format.md) - A YAML-based configuration file that defines an MLflow Project's name, software environment, entry points, and parameters for more explicit project specification.
* [MNIST Dataset](/concepts/mnist-dataset.md) - A large database of handwritten digits used as the example dataset for HorovodRunner distributed training examples.
* [MNIST dataset for distributed training examples](/concepts/mnist-dataset.md) - The MNIST handwritten digit database is used as the example dataset in Databricks' HorovodRunner distributed training demonstrations.
* [MNIST Dataset Loading with torchvision](/concepts/mnist-dataset-loading-with-torchvision.md) - Loading and transforming the MNIST handwritten digit dataset using torchvision.datasets.MNIST with ToTensor and Normalize transformations.
* [MNIST handwritten digit classification](/concepts/mnist-handwritten-digit-classification.md) - A classic benchmark dataset and problem for handwritten digit recognition using deep learning, used as a demonstration example for HorovodRunner with TensorFlow and Keras.
* [MNIST on Databricks with TensorFlow and Keras](/concepts/mnist-on-databricks-with-tensorflow-and-keras.md) - Reference example demonstrating distributed deep learning using TensorFlow, Keras, and HorovodRunner on Databricks for the MNIST digit classification task.
* [MNIST PyTorch on Databricks](/concepts/mnist-pytorch-on-databricks.md) - A reference notebook example showing how to train an MNIST model using PyTorch on Databricks, first on a single node then distributed with HorovodRunner
* [MNIST TensorFlow Keras Example](/concepts/mnist-tensorflow-keras-example.md) - A reference notebook demonstrating distributed deep learning on the MNIST dataset using TensorFlow, Keras, and HorovodRunner on Databricks.
* [MNIST TensorFlow Keras example on Databricks](/concepts/mnist-tensorflow-keras-example.md) - A reference notebook demonstrating deep learning with TensorFlow, Keras, and HorovodRunner for the MNIST dataset on Databricks.
* [MNIST with PyTorch on Databricks](/concepts/mnist-pytorch-on-databricks.md) - Example of training an MNIST model using PyTorch on Databricks, serving as a reference for distributed deep learning
* [MNIST with TensorFlow and Keras on Databricks](/concepts/mnist-with-tensorflow-and-keras-on-databricks.md) - A reference pattern for training an MNIST digit classifier using TensorFlow and Keras, runnable with HorovodRunner for distributed execution
* [Model Access Control in Unity Catalog](/concepts/model-access-control-in-unity-catalog.md) - Using Unity Catalog's GRANT ON FUNCTION syntax and Catalog Explorer to set permissions on registered models, including USE SCHEMA, USE CATALOG, CREATE MODEL, and EXECUTE privileges.
* [Model Accuracy Metrics in InferenceLog Profiling](/concepts/model-accuracy-metrics-in-inferencelog-profiling.md) - For InferenceLog analysis, model quality metrics (confusion_matrix, precision, recall, f1_score, roc_auc_score) are computed when label_col and prediction_col are provided; fairness/bias stats for classification models.
* [Model Aliases](/concepts/model-aliases.md) - Mutable, named references assigned to specific model versions for indicating deployment status (e.g., 'Champion'), enabling decoupled deployment workflows.
* [Model Aliases for Production Rollouts](/concepts/model-aliases-for-production-rollouts.md) - Using aliases like 'Champion' on registered models to manage production model versions and promote models across environments in Unity Catalog
* [Model Allow-List via Unity Catalog Permissions](/concepts/model-allow-list-via-unity-catalog-permissions.md) - A security pattern where admins remove default EXECUTE from system.ai schema and selectively grant EXECUTE only on approved models, creating an explicit allow-list of accessible models.
* [Model Allow-Listing via EXECUTE Grants](/concepts/model-allow-listing-via-execute-grants.md) - The pattern of removing default EXECUTE on the system.ai schema and then selectively granting EXECUTE on approved individual model objects to create an allow-list of permitted foundation models.
* [Model Checkpoint Resume Training](/concepts/model-checkpoint-resume-training.md) - A technique for saving model state during training (via PyTorch Lightning's ModelCheckpoint) and later resuming training from a saved checkpoint.
* [Model Checkpointing and Resume Training](/concepts/model-checkpointing-and-resume-training.md) - A technique to save model state during training (e.g., after each epoch) and later resume training from a saved checkpoint, enabling long-running or interrupted training workflows.
* [Model Checkpointing for Long-Running AI Runtime Jobs](/concepts/model-checkpointing-for-long-running-ai-runtime-jobs.md) - Recommended mechanism using Unity Catalog volumes via UCVolumeWriter/UCVolumeReader for workloads exceeding the 7-day maximum runtime.
* [Model Checkpointing with Unity Catalog](/concepts/model-checkpointing-with-unity-catalog.md) - Using UCVolumeWriter and UCVolumeReader from serverless_gpu.data to implement manual checkpointing for long-running AI Runtime jobs that may exceed the 7-day maximum runtime.
* [Model Checkpointing with Unity Catalog for AI Runtime](/concepts/model-checkpointing-with-unity-catalog.md) - Recommended checkpointing strategy for long-running AI Runtime jobs using UCVolumeWriter and UCVolumeReader from serverless_gpu.data
* [Model Checkpointing with Unity Catalog Volumes](/concepts/model-checkpointing-with-unity-catalog-volumes.md) - Distributed training on AI Runtime uses UCVolumeWriter and UCVolumeReader for asynchronous checkpoint saves/loads to Unity Catalog volumes, leveraging NVMe-backed local staging for performance.
* [Model Conversion from sparkdl.xgboost to xgboost.spark](/concepts/model-conversion-from-sparkdlxgboost-to-xgboostspark.md) - Utility function to convert saved sparkdl.xgboost models into xgboost.spark format, handling parameter mapping and booster serialization.
* [Model Dependency Management in MLflow](/concepts/model-dependency-management-in-mlflow.md) - Techniques for capturing and restoring model dependencies using requirements.txt, conda.yaml, virtualenv, and PySpark UDF environment managers to ensure reproducible model loading and scoring.
* [Model Deployment Timeout](/concepts/model-deployment-timeout.md) - Timeout during container build and model deployment, with retry limits and duration limits based on CPU/GPU workload size.
* [Model Deployment Timeouts](/concepts/model-deployment-timeouts.md) - Timeouts that occur during container build and model deployment for Databricks Model Serving endpoints, including retry limits and workload-specific time limits.
* [Model Hub in LLMOps](/concepts/model-hub-in-llmops.md) - Using internal or external model hubs to select pretrained foundation models for LLM applications, reducing time and cost of building custom models
* [Model Inference Best Practices on Databricks](/concepts/model-inference-best-practices-on-databricks.md) - Strategies for online, batch, and streaming inference including MLflow deployment, Model Serving, Foundation Model APIs, external models, and Pandas UDFs.
* [Model Inference Configuration from Registry](/concepts/model-inference-configuration-from-registry.md) - Capability to configure batch, streaming (via Lakeflow Spark Declarative Pipelines), and real-time inference (via Model Serving) directly from registered models in the Workspace Model Registry.
* [Model Lineage in Unity Catalog](/concepts/model-lineage-in-unity-catalog.md) - Tracking the upstream datasets used to train or evaluate a model by logging input tables with mlflow.log_input, visible in Catalog Explorer's lineage graph.
* [Model Optimization for Inference Serving](/concepts/model-optimization-for-inference-serving.md) - Techniques to improve inference speed and resource efficiency, including reducing model size and complexity via quantization or pruning, enabling client-side batching, and offloading pre/post-processing from serving infrastructure.
* [Model Optimization Information API](/concepts/model-optimization-information-api.md) - API endpoint that returns whether a model is optimizable for provisioned throughput and its throughput_chunk_size, used to determine suitable throughput ranges.
* [Model Packaging](/concepts/model-packaging.md) - The process of logging a model with feature metadata so it retains references to its features and can automatically retrieve feature values during batch or real-time inference.
* [Model Packaging with Feature Metadata](/concepts/model-packaging.md) - The process of logging a model with references to its feature tables so it can automatically retrieve feature values during batch or real-time inference.
* [Model Preprocessing and Postprocessing with pyfunc](/concepts/model-preprocessing-and-postprocessing-with-pyfunc.md) - Pattern of wrapping model inference with custom preprocessing (input formatting) and postprocessing (output transformation) logic within a single MLflow pyfunc model
* [Model Redirect Bridge](/concepts/model-redirect-bridge.md) - A Databricks technique where a retiring model is temporarily redirected to a compatible replacement (same price, backwards-compatible) to give customers the full transition period even when a partner's retirement timeline is shorter than Databricks' standard.
* [Model Redirection Mechanism](/concepts/model-redirection-mechanism.md) - Databricks practice of temporarily redirecting requests from a retired partner model to a similar replacement model to ensure customers receive full transition time.
* [Model Registration and Deployment from AutoML](/concepts/model-registration-and-deployment-from-automl.md) - Process of registering the best AutoML model to Unity Catalog or Model Registry and deploying it to a custom model serving endpoint on Databricks.
* [Model Registration Methods](/concepts/model-registration-methods.md) - Multiple approaches to register models in the Workspace Model Registry: via the UI from MLflow runs, programmatically via mlflow.log_model() during experiments, via mlflow.register_model() post-experiment, and via the MLflow Client API create_registered_model().
* [Model Registry Permissions and Access Control](/concepts/model-registry-permissions-and-access-control.md) - Role-based access control system for the Workspace Model Registry where users must have at least CAN MANAGE to configure permissions, and model versions inherit permissions from their parent model.
* [Model Requirements for Trace-based Judges](/concepts/model-requirements-for-trace-based-judges.md) - Trace-based judges require a capable model served via Foundation Model APIs, External Model Serving Endpoints, or Custom Model Serving Endpoints, with specific recommended models listed.
* [Model Retirement and Deprecation Policy](/concepts/model-retirement-and-deprecation-policy.md) - Databricks Foundation Model APIs have a formal retirement policy for models, with specific retirement dates and recommended replacement models for migrating workloads.
* [Model Retirement and Migration Policy](/concepts/model-retirement-and-migration-policy.md) - Databricks publishes retirement dates for hosted models with recommended replacement models and migration guidance, including temporary traffic redirection periods.
* [Model Retirement Policy (Databricks)](/concepts/model-retirement-policy-databricks.md) - Databricks' official policy for notifying customers, managing transitions, and retiring deprecated generative AI models across Foundation Model API offerings.
* [Model Retirement Policy on Databricks](/concepts/model-retirement-policy-on-databricks.md) - A lifecycle management policy for Databricks-hosted foundation models that includes scheduled retirement dates, recommended replacement models, and migration guidance for deprecated endpoints.
* [Model Selection for Conversation Simulation](/concepts/model-selection-for-conversation-simulation.md) - Configuring which LLM powers the simulated user, including provider format patterns, region-specific hosting for EU vs US, and the ability to disable partner-powered models.
* [Model selection for throughput optimization](/concepts/model-selection-for-throughput-optimization.md) - Choosing smaller models (e.g., Llama 3.1 8B) for high-volume throughput tasks and larger models (e.g., Llama 3.1 405B) for tasks requiring maximum capability.
* [Model Selection for User Simulation](/concepts/model-selection-for-user-simulation.md) - Choosing which LLM powers the simulated user in conversation simulation via the user_model parameter, supporting provider-prefixed model formats (e.g., 'anthropic:/claude-sonnet-4-20250514'), with specific data residency and privacy considerations for Azure OpenAI and EU workspaces.
* [Model Selection Strategy for Throughput](/concepts/model-selection-strategy-for-throughput.md) - Guidance on selecting smaller models (e.g., Llama 3.1 8B) for high-volume tasks and larger models (e.g., Llama 3.1 405B) for complex tasks to optimize within rate limits.
* [Model Server Debugging Logs](/concepts/model-server-debugging-logs.md) - Two types of logs for troubleshooting model serving endpoints: container build logs (initialization phase) and model server logs (runtime phase).
* [Model Serving](/concepts/model-serving.md) - Databricks' unified solution for deploying, governing, and querying AI/ML models via REST APIs for real-time and batch inference with automatic scaling.
* [Model Serving and Batch Inference for Forecasts](/concepts/model-serving-and-batch-inference-for-forecasts.md) - Post-training capabilities in Databricks Serverless Forecasting, including viewing predictions, generating batch inference notebooks, and deploying to Model Serving endpoints.
* [Model Serving Authentication](/concepts/model-serving-authentication.md) - Authentication mechanism for Databricks Model Serving endpoints requiring a Databricks REST API token, sent via HTTP basic auth with 'token:' as the username and the API token as the password
* [Model Serving Build Logs Troubleshooting](/concepts/model-serving-build-logs-troubleshooting.md) - Process of diagnosing package installation failures during Model Serving deployment by inspecting build logs in the Databricks Serving UI
* [Model Serving Compute Types](/concepts/model-serving-compute-types.md) - Variety of CPU and GPU compute options including CPU_MEDIUM and CPU_LARGE for trading concurrency for more memory; GPU requires explicit code setup for predictions.
* [Model Serving Connection Timeouts](/concepts/model-serving-connection-timeouts.md) - Timeouts related to establishing network connections between clients and model serving endpoints, such as JDBC SocketTimeout configurations.
* [Model Serving Container Build Debugging](/concepts/model-serving-container-build-debugging.md) - Process of diagnosing and resolving failures during container image creation for Databricks Model Serving endpoints, including event log inspection and common build errors.
* [Model Serving Container Security (Databricks)](/concepts/model-serving-container-security-databricks.md) - Security practices for model serving containers including automatic 30-day rebuilds for compliance, no backporting of security patches to existing images, and patching via new model versions.
* [Model Serving Data Protection](/concepts/model-serving-data-protection.md) - Security controls including logical isolation, AES-256 encryption at rest, TLS 1.2+ in transit, and policies against using customer data for model training
* [Model Serving Dependency Management](/concepts/model-serving-dependency-management.md) - Ensuring all required libraries and custom artifacts (whl files) are properly declared as model dependencies to prevent container build and runtime failures.
* [Model Serving Deployment Container and Dependencies](/concepts/model-serving-deployment-container-and-dependencies.md) - How a production-grade container is built during deployment, including how to specify package dependencies via pip_requirements, conda_env, and extra_pip_requirements, and code dependencies via code_path for custom pyfunc models.
* [Model Serving Deployment Timeouts](/concepts/model-deployment-timeouts.md) - Timeouts that occur during the container build and model deployment phases of Databricks Model Serving, including retry logic and duration limits per workload type.
* [Model Serving Endpoint](/concepts/model-serving-endpoint.md) - A REST API endpoint on Databricks that serves custom ML models for real-time inference, accessed via a URL pattern like https://<instance>/serving-endpoints/<endpoint-name>/invocations
* [Model Serving Endpoint (Custom Models)](/concepts/model-serving-endpoint-custom-models.md) - A Databricks endpoint that serves custom machine learning models via UI, REST API, MLflow Deployments SDK, or Workspace Client
* [Model Serving Endpoint (Databricks)](/concepts/model-serving-endpoint-databricks.md) - A serverless endpoint that hosts one or more ML models for inference, configurable with instance profiles, workload sizes, and scale-to-zero settings.
* [Model Serving Endpoint Concurrency Scaling](/concepts/model-serving-endpoint-concurrency-scaling.md) - The relationship between endpoint concurrency, client-side connections, and latency percentiles, where required concurrency scales linearly with concurrent connections for achieving target latency.
* [Model Serving Endpoint Configuration API](/concepts/model-serving-endpoint-configuration-api.md) - The REST API endpoints and payload format for creating (POST /api/2.0/serving-endpoints) and updating (PUT /api/2.0/serving-endpoints/{name}/config) model serving endpoints with an instance_profile_arn field.
* [Model Serving Endpoint Configuration for Load Testing](/concepts/model-serving-endpoint-configuration-for-load-testing.md) - Recommended endpoint configuration for load testing: start with a Small CPU endpoint with min/max concurrency of 4, enable route optimization, and disable scale-to-zero.
* [Model Serving Endpoint Configuration for Performance Testing](/concepts/model-serving-endpoint-configuration-for-performance-testing.md) - Recommended Databricks endpoint configuration settings including CPU instance type, concurrency levels, route optimization, and scale-to-zero settings for load testing.
* [Model Serving Endpoint Configuration Methods](/concepts/model-serving-endpoint-configuration-methods.md) - The four interfaces for configuring environment variables on model serving endpoints: Serving UI, REST API, WorkspaceClient SDK, and MLflow Deployments SDK.
* [Model Serving Endpoint Configuration with Resource Access](/concepts/model-serving-endpoint-configuration-with-resource-access.md) - The overall pattern of configuring model serving endpoints to access external or private resources such as SaaS APIs and data storage locations
* [Model Serving Endpoint Configuration Workflow](/concepts/model-serving-endpoint-configuration-workflow.md) - The overall process of creating or updating model serving endpoints with environment variables through UI, REST API, WorkspaceClient SDK, or MLflow Deployments SDK
* [Model Serving Endpoint Creation Methods](/concepts/model-serving-endpoint-creation-options.md) - Three ways to create model serving endpoints in Databricks: the Serving UI, REST API, and MLflow Deployments SDK.
* [Model Serving Endpoint Creation Options](/concepts/model-serving-endpoint-creation-options.md) - Three methods to create model serving endpoints: Serving UI, REST API, and MLflow Deployments SDK.
* [Model Serving Endpoint Creator Identity](/concepts/model-serving-endpoint-creator-identity.md) - The calling identity recorded as the endpoint's creator, used for Unity Catalog access, immutable after creation, and continuously validated for workspace membership and grants.
* [Model Serving Endpoint Debugging Workflow](/concepts/model-serving-endpoint-debugging-workflow.md) - Systematic approach to debugging Databricks Model Serving endpoints, starting with pre-deployment validation, then container build logs, followed by build-success or build-failure debugging paths.
* [Model Serving Endpoint Deployment](/concepts/model-serving-endpoint-deployment.md) - The process of registering a custom pyfunc model to Unity Catalog or Workspace Registry and serving it via a Model Serving endpoint on Databricks.
* [Model Serving Endpoint Deployment Constraints](/concepts/model-serving-endpoint-deployment-constraints.md) - Databricks Model Serving endpoints require that all model dependencies be statically captured at deployment time; network-based lazy loading and direct Unity Catalog volume access at inference time are not supported for real-time workloads.
* [Model Serving Endpoint Deployment for Custom Models](/concepts/model-serving-endpoint-deployment.md) - After logging a custom pyfunc model, it can be registered to Unity Catalog or Workspace Registry and served via a Databricks Model Serving endpoint.
* [Model Serving Endpoint Deployment Timeouts](/concepts/model-serving-endpoint-deployment-timeouts.md) - Timeouts that occur when deploying or updating a model serving endpoint, including container build retries and post-build deployment duration limits for CPU and GPU workloads.
* [Model Serving Endpoint for LangChain](/concepts/model-serving-endpoint-for-langchain.md) - A Databricks model serving endpoint used to deploy a LangChain application that uses feature lookups for RAG
* [Model Serving Endpoint Scaling](/concepts/model-serving-endpoint-scaling.md) - Automatic scaling of serving endpoints based on traffic and provisioned concurrency, including scale-to-zero after 30 minutes of inactivity, route optimization for high QPS, express deployments, and scaling behavior (immediate scale-up, 5-minute scale-down intervals).
* [Model Serving endpoint scaling behavior](/concepts/model-serving-endpoint-scaling-behavior.md) - Serving endpoints auto-scale based on traffic: scale up near-instantly under load, scale down every five minutes, with optional scale-to-zero after 30 minutes of inactivity (cold start adds latency). Provisioned concurrency = QPS × execution time.
* [Model Serving Endpoint Status](/concepts/model-serving-endpoint-status.md) - The operational states of a Databricks model serving endpoint, including Ready, Not Ready, and Update Failed indicators.
* [Model Serving Endpoint Telemetry](/concepts/model-serving-endpoint-telemetry.md) - A Databricks feature that persists OpenTelemetry logs, traces, and metrics from custom model serving endpoints to Unity Catalog tables for monitoring and compliance.
* [Model Serving Endpoint Timeout Types](/concepts/model-serving-endpoint-timeout-types.md) - Three categories of timeouts encountered when using Databricks Model Serving: model deployment timeouts, server-side timeouts, and client-side timeouts.
* [Model Serving Endpoint Timeouts](/concepts/model-serving-endpoint-timeouts.md) - Overview of the various timeout categories (deployment, server-side, client-side) encountered when using Databricks Model Serving endpoints.
* [Model Serving Endpoints (Databricks)](/concepts/model-serving-endpoints-databricks.md) - Custom model serving endpoints on Databricks that can be configured with route optimization for improved performance.
* [Model Serving Endpoints for Feature Store Models](/concepts/model-serving-endpoints-for-feature-store-models.md) - How to deploy and query models trained using Databricks Feature Store as model serving endpoints using Unity Catalog for online feature lookup.
* [Model Serving Environment Variables](/concepts/model-serving-environment-variables.md) - Mechanism for configuring access to external and private resources from Databricks model serving endpoints using plain text or secrets-based environment variables.
* [Model Serving Error Classification](/concepts/model-serving-error-classification.md) - Classification of error codes in Databricks Model Serving, distinguishing between customer-caused errors (4xx) and platform errors (5xx).
* [Model Serving Error Code Semantics](/concepts/model-serving-error-code-semantics.md) - Databricks Model Serving maps MlflowException model code errors to 4xx (customer-caused) responses and reserves 5xx for Databricks infrastructure errors, helping users determine responsibility.
* [Model Serving Error Code Semantics (4xx vs 5xx)](/concepts/model-serving-error-code-semantics.md) - Mapping of MLflowException to 4xx customer-caused errors and 5xx Databricks-responsibility errors in model serving endpoints.
* [Model Serving Error Response Mapping](/concepts/model-serving-error-code-semantics.md) - Databricks convention mapping MlflowException model code errors to 4xx responses (customer-caused) while reserving 5xx errors for Databricks infrastructure faults.
* [Model Serving for LLMs](/concepts/model-serving-for-llms.md) - Unified interface to deploy, govern, and query AI models, including handling external API calls, latency, and credential management for third-party LLM APIs
* [Model Serving Instance Profile](/concepts/model-serving-instance-profile.md) - An AWS IAM instance profile attached to a Databricks model serving endpoint, allowing models to access AWS resources (e.g., S3, Feature Store) during inference.
* [Model Serving Instance Profile Lifecycle Constraints](/concepts/model-serving-instance-profile-lifecycle-constraints.md) - Operational constraints around serving endpoint instance profiles: endpoints retain the old IAM role if the profile is edited in Databricks Settings, and deleting a profile does not affect running endpoints.
* [Model Serving Migration](/concepts/model-serving-migration.md) - The process of transitioning from Legacy MLflow Model Serving to the new Model Serving experience, including URL changes, scoring format changes, and endpoint reconfiguration.
* [Model Serving Networking and Security Limitations](/concepts/model-serving-networking-and-security-limitations.md) - Security restrictions for Model Serving endpoints including access control, PrivateLink support, outbound network policies, and security patching policies.
* [Model Serving on Databricks](/concepts/model-serving-on-databricks.md) - Databricks platform supports deploying trained models (e.g., YOLO) to production via Model Serving, a capability demonstrated in AI Runtime tutorials.
* [Model Serving on Databricks: Batch and Real-Time](/concepts/model-serving-on-databricks-batch-and-real-time.md) - Two model deployment patterns: batch inference for efficient large-scale scoring and real-time serving via low-latency API endpoints, with governance through Unity Catalog inference tables.
* [Model Serving Patterns](/concepts/model-serving-patterns.md) - Two primary deployment patterns on Databricks: real-time serving via low-latency REST endpoints (Model Serving) and batch inference via ai_query, Spark UDFs, or mlflow.pyfunc for periodic jobs.
* [Model Serving Pre-deployment Testing Workflow](/concepts/model-serving-pre-deployment-testing-workflow.md) - Databricks-recommended validation steps to catch model issues before endpoint deployment, including offline prediction testing, dependency checks, input validation, and local serving simulation.
* [Model Serving Region Availability](/concepts/model-serving-region-availability.md) - Regional limitations for Databricks Model Serving, including control plane dependencies and per-region feature availability for custom models, AI agents, and Foundation Model APIs.
* [Model Serving Resource and Payload Limits](/concepts/model-serving-resource-and-payload-limits.md) - Default limits that Databricks Model Serving imposes on custom model and AI agent endpoints to ensure reliable performance.
* [Model Serving Security and Data Protection](/concepts/model-serving-security-and-data-protection.md) - Security controls for Model Serving including logical isolation, AES-256 encryption at rest, TLS 1.2+ in transit, no training on customer data, container build log retention (30 days), metrics retention (14 days), and abuse monitoring for Foundation Model APIs.
* [Model Serving Traffic Split](/concepts/model-serving-traffic-split.md) - Ability to route a fraction of inference traffic to models in Unity Catalog while gradually increasing the proportion for controlled rollouts
* [Model Serving with Automatic Feature Lookup](/concepts/model-serving-with-automatic-feature-lookup.md) - Mechanism where models trained using FeatureLookup automatically retrieve feature values from online tables during inference without additional configuration.
* [Model serving with on-demand features](/concepts/model-serving-with-on-demand-features.md) - Workflow for serving models that use on-demand features, covering batch scoring (score_batch) and real-time Model Serving with automatic UDF evaluation.
* [Model sharding](/concepts/model-sharding.md) - The technique of partitioning model parameters, gradients, and optimizer states across multiple devices to overcome single-GPU memory limitations.
* [Model Signature and Input Examples in MLflow](/concepts/model-signature-and-input-examples-in-mlflow.md) - Recommended practice of adding signatures (via infer_signature) and input examples when logging models; signatures are required for logging to Unity Catalog.
* [Model signature requirement for Unity Catalog](/concepts/model-signature-requirement-for-unity-catalog.md) - Models registered in Unity Catalog require a signature; if none exists, the MLFLOW_SKIP_SIGNATURE_CHECK_FOR_UC_REGISTRY_MIGRATION environment variable (MLflow ≥3.4.0) can bypass the requirement during migration.
* [Model Signatures in Unity Catalog](/concepts/model-signatures-in-unity-catalog.md) - Required schema definitions for new model versions in Unity Catalog, inferred automatically via autologging or input examples, and needed for input enforcement, AI functions, and model serving.
* [Model Tags in Unity Catalog](/concepts/model-tags-in-unity-catalog.md) - Key-value pairs applied to registered models and model versions for labeling and categorization, managed via MLflow Client API or Catalog Explorer.
* [Model Traces in Unity Catalog](/concepts/model-traces-in-unity-catalog.md) - Traces panel on the model version page that consolidates inference traces from development, evaluation (mlflow.evaluate()), and online serving endpoints for full observability.
* [Model Truncation for Feature Extraction](/concepts/truncated-model-feature-extraction.md) - The technique of removing the final classification layers of a pre-trained deep learning model so that it outputs feature tensors instead of predictions, enabling use as a feature extractor.
* [Model type comparison with Hyperopt SparkTrials](/concepts/model-type-comparison-with-hyperopt-sparktrials.md) - Using Hyperopt with SparkTrials to tune hyperparameters for and compare multiple model types to select the best overall model
* [Model Type Restriction on Endpoints](/concepts/model-type-restriction-on-endpoints.md) - The constraint that a single Databricks model serving endpoint cannot mix different model types (custom, foundation, external) — all served models must be of the same type.
* [Model Units](/concepts/model-units.md) - A unit of throughput that determines how much work a provisioned throughput endpoint can handle per minute, used to allocate inference capacity for foundation models.
* [Model Validation Before Deployment](/concepts/model-validation-before-deployment.md) - Using mlflow.models.predict to validate custom pyfunc models before deploying them to Model Serving endpoints
* [Model Validation Pipeline](/concepts/model-validation-pipeline.md) - A structured pipeline that loads a model from Unity Catalog, runs validation checks (format, metadata, performance, compliance), and determines whether a model should proceed to deployment by assigning aliases.
* [Model version page in Catalog Explorer](/concepts/model-version-page-in-catalog-explorer.md) - A Unity Catalog page providing an overview of all model performance and evaluation results across linked environments including different workspaces, endpoints, and experiments, used for monitoring, deployment, and approval workflows.
* [Model Version Promotion Across Environments](/concepts/model-version-promotion-across-environments.md) - Copying model versions across registered models in Unity Catalog (e.g., staging to prod) using the copy_model_version API, combined with aliases for deployment tracking.
* [Model Versioning and Stage Transitions](/concepts/model-versioning-and-stage-transitions.md) - A system within the Workspace Model Registry that tracks distinct versions of a registered model and allows transitioning them through lifecycle stages: None, Staging, Production, and Archived, with optional approval workflows.
* [Model-Specific vs Registry-Wide Webhooks](/concepts/model-specific-vs-registry-wide-webhooks.md) - Two scope-based webhook types: model-specific (applies to a single registered model, requires CAN MANAGE permission) and registry-wide (triggered by events on any model, requires workspace admin permissions).
* [Modeling External Lineage Relationships](/concepts/modeling-external-lineage-relationships.md) - Patterns for representing complex data flows through external metadata objects, including connecting two Unity Catalog tables via an intermediary object, modeling multi-hop pipelines, and adding column-level lineage with source-to-target mappings.
* [ModelOps on Databricks](/concepts/modelops-on-databricks.md) - Model development and lifecycle management using MLflow tracking and Model Registry, with monitoring of model performance and input data changes.
* [Models in Unity Catalog](/concepts/models-in-unity-catalog.md) - Databricks' integration of MLflow Model Registry with Unity Catalog for centralized model management, access control, auditing, and lineage.
* [Models Tab vs Catalog Explorer Model Version Page](/concepts/models-tab-vs-catalog-explorer-model-version-page.md) - The Models tab on the MLflow experiment page focuses on comparing logged models for selection, while the Catalog Explorer model version page provides a holistic view for monitoring and deployment across environments.
* [Monitoring dashboard parameters and widgets](/concepts/monitoring-dashboard-parameters-and-widgets.md) - User-editable widgets at the top of the dashboard that control data filtering by date range, data slices, models, and other parameters, varying by analysis type.
* [Monitoring Deep Learning Training with TensorBoard and Cluster Metrics](/concepts/monitoring-deep-learning-training-with-tensorboard-and-cluster-metrics.md) - Using pre-installed TensorBoard and Databricks cluster metrics (network, processor, memory, GPU) to inspect bottlenecks and monitor training processes.
* [Monitoring GPU Utilization with MLflow System Metrics](/concepts/monitoring-gpu-utilization-with-mlflow-system-metrics.md) - Using MLflow-recorded system metrics (GPU, CPU, memory usage) and Databricks dashboards to monitor compute efficiency across experiments, particularly identifying underutilized GPUs.
* [Mosaic LLM Foundry](/concepts/mosaic-llm-foundry.md) - An open-source framework for training, fine-tuning, evaluating, and deploying large language models with built-in support for distributed training strategies like FSDP.
* [Mosaic Streaming](/concepts/mosaic-streaming.md) - An open-source data loading library for single-node or distributed training/evaluation of deep learning models, compatible with PyTorch, PyTorch Lightning, Mosaic Composer, and TorchDistributor.
* [Mosaic Streaming (MDS) format](/concepts/mosaic-streaming-mds-format.md) - A data format used for preparing and streaming training data, demonstrated in the context of distributed training on Databricks.
* [Mount a Shared Genie Space](/concepts/mount-a-shared-genie-space.md) - The process by which recipients of a shared Genie Space can mount the share to create a local Genie Space pre-loaded with the provider's data and instructions.
* [Mounted Genie Space Ownership and Isolation](/concepts/mounted-genie-space-ownership-and-isolation.md) - The recipient has full ownership of the mounted Genie Space — changes made locally do not affect the provider's original space, and the recipient can add data, modify instructions, and reconfigure freely.
* [Multi-Agent Workflow Termination](/concepts/multi-agent-workflow-termination.md) - The pattern of using a TERMINATE keyword in agent messages to signal the end of a multi-agent conversation
* [Multi-cluster Write Limitations for Delta Lake on S3](/concepts/multi-cluster-write-limitations-on-s3.md) - Delta Lake tables on S3 support multi-cluster writes only within a single Databricks workspace due to S3's eventual consistency model; cross-workspace writes risk data loss or corruption.
* [Multi-cluster Write Limitations on S3](/concepts/multi-cluster-write-limitations-on-s3.md) - Delta Lake tables on S3 have multi-cluster write guarantees limited to a single Databricks workspace due to S3's eventual consistency model; modifying the same table from different workspaces risks data corruption.
* [Multi-cluster writes feature toggle](/concepts/multi-cluster-writes-feature-toggle.md) - The spark.databricks.delta.multiClusterWrites.enabled setting controls whether Delta Lake tables on S3 support concurrent writes from multiple clusters; disabling it requires single-cluster access.
* [Multi-Dimensional Access Control with Single Lookup Table](/concepts/multi-dimensional-access-control-with-single-lookup-table.md) - Replacing N groups per access dimension with a single table where each row represents a user's permissions across multiple dimensions (region, department, clearance level), eliminating combinatorial group explosion.
* [Multi-Dimensional Feedback Assessment](/concepts/multi-dimensional-feedback-assessment.md) - Collecting separate ratings for different quality dimensions (accuracy, helpfulness, relevance) and logging each as an individual assessment on a single trace.
* [Multi-Dimensional Feedback Collection](/concepts/multi-dimensional-feedback-collection.md) - Collecting feedback across multiple rating dimensions (e.g., accuracy, helpfulness, relevance) by logging each as a separate assessment with normalized scores, enabling granular quality analysis.
* [Multi-Dimensional User Feedback](/concepts/multi-dimensional-user-feedback.md) - Logging separate assessments for different quality dimensions (accuracy, helpfulness, relevance) to enable granular analysis
* [Multi-domain column masking with sensitivity tiers](/concepts/multi-domain-column-masking-with-sensitivity-tiers.md) - A pattern for applying column-level masks that vary by both domain ownership and sensitivity classification, using AND conditions in MATCH COLUMNS to target specific combinations of governed tags.
* [Multi-Experiment Run Comparison](/concepts/multi-experiment-run-comparison.md) - Ability to display and compare runs from multiple experiments in the MLflow Chart View, enabling cross-experiment analysis.
* [Multi-Experiment Run Comparison in MLflow](/concepts/multi-experiment-run-comparison.md) - The capability to display and compare runs across multiple experiments simultaneously in the MLflow UI.
* [Multi-Feedback Scorer Pattern](/concepts/multi-feedback-scorer-pattern.md) - A single custom scorer returning a list of Feedback objects to assess multiple quality facets in one pass, each with a unique metric name.
* [Multi-framework Auto-tracing](/concepts/multi-framework-auto-tracing.md) - Ability to enable automatic tracing for multiple AI frameworks (e.g., OpenAI and LangChain) simultaneously in the same agent workflow, producing unified traces.
* [Multi-Framework Auto-Tracing in MLflow](/concepts/multi-framework-auto-tracing.md) - The ability to enable automatic tracing for multiple frameworks (e.g., OpenAI and LangChain) simultaneously in the same agent, producing a single unified trace.
* [Multi-Framework Trace Consolidation on Databricks](/concepts/multi-framework-trace-consolidation-on-databricks.md) - Architectural benefit of routing traces from Langfuse and other frameworks into a single Unity Catalog-backed storage, enabling unified querying, comparison, and governance.
* [Multi-GPU distributed training (Databricks)](/concepts/multi-gpu-distributed-training-databricks.md) - Notebook examples for scaling training across multiple GPUs and nodes on Databricks using the Serverless GPU API.
* [Multi-GPU distributed training API](/concepts/multi-gpu-distributed-training-api.md) - Beta-stage API on Databricks for scaling training workloads across multiple GPUs and nodes.
* [Multi-GPU Distributed Training on Databricks](/concepts/multi-gpu-distributed-training-on-databricks.md) - Scaling model training across multiple GPUs and nodes using Databricks AI Runtime, with support for H100 GPUs, parallelism techniques (DDP, FSDP, DeepSpeed), and serverless GPU Python library tutorials.
* [Multi-GPU Distributed Training with Serverless GPU API](/concepts/multi-gpu-distributed-training-with-serverless-gpu-api.md) - API for scaling deep learning training workloads across multiple GPUs and nodes on Databricks, currently in Beta.
* [Multi-GPU Training APIs on Databricks](/concepts/multi-gpu-training-apis-on-databricks.md) - The two parallel approaches for multi-GPU workload training on Databricks: the AI Runtime CLI for laptop-based submission and the in-notebook Python API with @distributed and @ray_launch decorators
* [Multi-GPU workload Python API](/concepts/multi-gpu-workload-python-api.md) - In-notebook Python API using `@distributed` and `@ray_launch` decorators as an alternative to the AI Runtime CLI for distributed training.
* [Multi-Image Input in Vision Model Queries](/concepts/multi-image-input-in-vision-model-queries.md) - The capability to send multiple images in a single Chat Completions API request, allowing vision models to analyze and compare images side by side.
* [Multi-Integration Auto Tracing in MLflow](/concepts/multi-integration-auto-tracing-in-mlflow.md) - Enabling auto-tracing for several GenAI libraries simultaneously to produce a single cohesive trace across combined toolchains.
* [Multi-Judge Evaluation Framework](/concepts/multi-judge-evaluation-framework.md) - The practice of combining multiple custom judges (e.g., issue resolution, expected behaviors, tool call correctness) in a single `mlflow.genai.evaluate()` call to assess different quality dimensions of a GenAI agent simultaneously.
* [Multi-Judge Evaluation Pipelines](/concepts/multi-judge-evaluation-pipelines.md) - The pattern of passing multiple custom judges as scorers to mlflow.genai.evaluate() to evaluate different aspects of an agent (issue resolution, expected behaviors, tool correctness) in a single run.
* [Multi-Level Human Feedback in GenAI Pipelines](/concepts/multi-level-human-feedback-in-genai-pipelines.md) - A pattern for collecting feedback at three distinct stages: end-user (in-app thumbs up/down), developer (UI annotations), and domain expert (structured labeling sessions)
* [Multi-LLM Call Tracing Pattern](/concepts/multi-llm-call-tracing-pattern.md) - Design pattern where automatic tracing captures multiple sequential LLM calls within a single workflow, illustrated by an analysis-then-response pattern with conditional model routing.
* [Multi-Model Serving](/concepts/multi-model-serving.md) - The ability to serve multiple models (e.g., different sizes of Llama) within a single serving endpoint, each with its own configuration.
* [Multi-Model Serving Endpoint](/concepts/multi-model-serving-endpoint.md) - A Databricks model serving endpoint that hosts multiple models simultaneously to enable A/B testing, traffic splitting, and side-by-side comparison.
* [Multi-Model Strategy in Agent Systems](/concepts/multi-model-strategy-in-agent-systems.md) - The practice of using different LLMs for different stages of a multi-prompt pipeline, such as a stronger model for planning and a more cost-efficient model for execution.
* [Multi-Module Artifact Upload for Databricks Connect](/concepts/multi-module-artifact-upload-for-databricks-connect.md) - Strategy for handling UDFs in multi-module SBT projects where each module's classes must be uploaded separately using addCompiledArtifacts() with getClass references from each module.
* [Multi-node FSDP fine-tuning on AI Runtime](/concepts/multi-node-fsdp-fine-tuning-on-ai-runtime.md) - Supervised fine-tuning of LLMs (e.g., Llama-3.1-8B) across multiple nodes using PyTorch Fully Sharded Data Parallel on H100 GPUs
* [Multi-node FSDP Fine-tuning on Databricks](/concepts/multi-node-llm-fine-tuning-with-fsdp.md) - Pattern for supervised fine-tuning of large language models (e.g., Llama-3.1-8B) across multiple nodes (e.g., 16 H100 GPUs on 2 nodes) using PyTorch FSDP and torchrun on Databricks AI Runtime
* [Multi-node LLM fine-tuning with FSDP](/concepts/multi-node-llm-fine-tuning-with-fsdp.md) - A distributed-training pattern using PyTorch FSDP and torchrun to fine-tune Llama-3.1-8B across 16 H100 GPUs (2 nodes), with MLflow logging and Unity Catalog checkpointing.
* [Multi-node LLM fine-tuning with FSDP on Databricks](/concepts/multi-node-llm-fine-tuning-with-fsdp.md) - Distributed supervised fine-tuning of Llama-3.1-8B across 16 H100 GPUs (2 nodes) using torchrun and PyTorch Fully Sharded Data Parallel (FSDP), with MLflow logging and Unity Catalog checkpointing.
* [Multi-Prompt Optimization](/concepts/multi-prompt-optimization.md) - The technique of jointly optimizing multiple chained prompts in an agent system using GEPA (GepaPromptOptimizer) to improve overall pipeline performance.
* [Multi-Series Forecasting](/concepts/multi-series-forecasting.md) - A forecasting approach where data is grouped by identifier columns into multiple independent time series, and Databricks trains a separate model for each series.
* [Multi-Series Forecasting in AutoML](/concepts/multi-series-forecasting-in-automl.md) - Technique in Databricks AutoML where time series identifiers group data into multiple independent time series, with each series trained independently.
* [Multi-Series Forecasting with Time Series Identifiers](/concepts/multi-series-forecasting-with-time-series-identifiers.md) - The ability to forecast multiple independent time series simultaneously by designating columns that identify each series, with a model trained per series.
* [Multi-Tenant App for Delta Sharing](/concepts/multi-tenant-app-for-delta-sharing.md) - Databricks-registered multi-tenant OAuth app (DeltaSharing) with client ID 64978f70-f6a6-4204-a29e-87d74bfea138 used by Power BI and Tableau recipients
* [Multi-threaded tracing in MLflow](/concepts/multi-threaded-tracing-in-mlflow.md) - MLflow Tracing is thread-safe with ContextVar isolation per thread, requiring manual context copying via contextvars.copy_context() and ctx.run() to propagate traces across threads.
* [Multi-threaded Tracing with ContextVar](/concepts/multi-threaded-tracing-in-mlflow.md) - MLflow Tracing is thread-safe using Python's ContextVar mechanism, but requires manual context copying via contextvars.copy_context() when spawning traces across threads with ThreadPoolExecutor.
* [Multi-threading with MLflow Tracing](/concepts/multi-threading-with-mlflow-tracing.md) - Thread-safe tracing in MLflow using ContextVar, requiring manual context copying across threads via contextvars.copy_context() for correct parent-child span relationships.
* [Multi-turn Assessment Storage](/concepts/multi-turn-assessment-storage.md) - Design pattern where multi-turn assessments are stored on the first trace (chronologically) in each session to ensure stability and easy retrieval.
* [Multi-turn Conversation Evaluation](/concepts/multi-turn-conversation-evaluation.md) - A paradigm for assessing conversational AI systems at the session level rather than individual turns, capturing user frustration, conversation completeness, knowledge retention, and dialogue coherence.
* [Multi-turn Conversation Judges](/concepts/multi-turn-conversation-judges.md) - Scorers that evaluate entire conversations by grouping traces via session tags, assessing quality patterns like user frustration and conversation completeness across multiple interactions.
* [Multi-turn Evaluation with Simulated Conversations](/concepts/multi-turn-evaluation-with-simulated-conversations.md) - Using the ConversationSimulator as input to mlflow.genai.evaluate() alongside multi-turn scorers (ConversationCompleteness) and single-turn scorers (Safety) applied per turn.
* [Multi-turn Function Calling](/concepts/multi-turn-function-calling.md) - Function calling across multiple conversational turns, recommended for Claude models on Databricks; Llama models currently optimized for single-turn only.
* [Multi-turn Judge](/concepts/multi-turn-judge.md) - A type of LLM judge that evaluates entire conversational histories across multiple interaction turns rather than scoring individual response turns in isolation.
* [Multi-turn Judges](/concepts/multi-turn-judge.md) - LLM judges that evaluate entire conversations rather than individual turns, analyzing the complete conversation history for quality patterns over multiple interactions.
* [Multi-turn LLM Judges](/concepts/multi-turn-llm-judges.md) - LLM-based evaluators that analyze full conversation history using built-in scorers like ConversationCompleteness and UserFrustration, or custom judges via make_judge with the {{ conversation }} template variable.
* [Multi-turn Scorers in MLflow](/concepts/multi-turn-scorers-in-mlflow.md) - Evaluation scorers that assess conversational AI agents across multiple turns, such as ConversationCompleteness and Safety, applicable within the conversation simulation evaluation workflow.
* [Multi-User Collaboration on AI Runtime](/concepts/multi-user-collaboration-on-ai-runtime.md) - Best practices for sharing code, using Git folders, and collaborating on notebooks in Databricks AI Runtime.
* [Multi-User Collaboration on Databricks](/concepts/multi-user-collaboration-on-ai-runtime.md) - Best practices for sharing code and notebooks on Databricks, including using /Workspace/Shared for shared artifacts and Git folders with remote repos for active development.
* [Multi-user Collaboration on Databricks AI Runtime](/concepts/multi-user-collaboration-on-ai-runtime.md) - Best practices for sharing code via /Workspace/Shared, using Git folders for active development, and collaborating on notebooks for multi-user ML projects.
* [Multilingual Chain-of-Thought Fine-Tuning](/concepts/multilingual-chain-of-thought-fine-tuning.md) - Fine-tuning language models on multilingual datasets like HuggingFaceH4/Multilingual-Thinking to enable chain-of-thought reasoning across languages
* [Multilingual chain-of-thought reasoning](/concepts/multilingual-chain-of-thought-reasoning.md) - Capability of a language model to perform internal reasoning in one language while generating responses in another, enabled by multilingual CoT datasets
* [Multimodal Foundation Models](/concepts/multimodal-foundation-models.md) - Foundation models that accept multiple input types (text, image, video, audio) alongside text, enabling cross-modal reasoning tasks like image understanding, video analysis, and visual Q&A.
* [Multiple ABAC Policy Resolution Rules](/concepts/multiple-abac-policy-resolution-rules.md) - Only one distinct row filter or column mask can resolve per user per table or column at runtime, otherwise Databricks blocks access
* [Multiple Client Options for Embedding Queries](/concepts/multiple-client-options-for-embedding-queries.md) - Various client interfaces supported for querying embedding models on Databricks including OpenAI client, SQL, REST API, MLflow Deployments SDK, Databricks Python SDK, and LangChain.
* [Multiple feedback from a single scorer](/concepts/multiple-feedback-from-a-single-scorer.md) - A pattern where a single custom scorer returns a list of Feedback objects, each with a unique name, to assess multiple quality facets simultaneously in one evaluation pass
* [Multiple Feedback objects from a single scorer](/concepts/multiple-feedback-objects-from-a-single-scorer.md) - A single custom scorer can return a list of Feedback objects, each with a unique name, to assess multiple quality facets (e.g., PII, sentiment, conciseness) simultaneously.
* [Multiple Framework Auto-Tracing](/concepts/multi-framework-auto-tracing.md) - Pattern enabling simultaneous automatic tracing of multiple AI frameworks (e.g., OpenAI and LangChain) in a single agent workflow to produce unified, debuggable traces.
* [Multiple Judge Evaluation](/concepts/multiple-judge-evaluation.md) - Using multiple custom judges together to evaluate different aspects of a GenAI agent simultaneously within a single evaluation run.
* [Multiple Row Filter and Column Mask Conflict Resolution](/concepts/multiple-row-filter-and-column-mask-conflict-resolution.md) - Rules ensuring only one distinct row filter per table and one distinct column mask per column can apply at runtime; conflicts block access and require diagnosis via SHOW EFFECTIVE POLICIES and INFORMATION_SCHEMA views.
* [Multiple Served Entities per Endpoint](/concepts/multiple-served-entities-per-endpoint.md) - A single serving endpoint can serve multiple models or model versions with configurable traffic splits between them.
* [Multiple Source Row Match Error in MERGE](/concepts/multiple-source-row-match-error-in-merge.md) - A MERGE operation fails with DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE error if more than one source row matches the same target row, requiring preprocessing of the source table to eliminate ambiguity.
* [MULTIPLE_RELATIONS constraint for table_changes](/concepts/multiple_relations-constraint-for-table_changes.md) - A validation rule requiring that views used with table_changes reference exactly one relation (table or view), not multiple
* [MULTIPLE_RELATIONS error reason](/concepts/multiple_relations-sub-error.md) - A sub-reason of DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED indicating the view references more than one relation.
* [MULTIPLE_RELATIONS sub-error](/concepts/multiple_relations-sub-error.md) - A specific cause of DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED where the view references more than one relation
* [MUST_REFRESH_SAME_TABLE](/concepts/must_refresh_same_table.md) - A sub-error of DELTA_UNIFORM_ICEBERG_INGRESS_VIOLATION that occurs when attempting to refresh an Iceberg table with metadata from a different table UUID where the version must increase monotonically.
* [MUST_REFRESH_SAME_TABLE error](/concepts/must_refresh_same_table.md) - A sub-error of DELTA_UNIFORM_ICEBERG_INGRESS_VIOLATION that occurs when attempting to refresh an existing Apache Iceberg table UUID with metadata from a different Iceberg table UUID, or when the refresh metadata location has a lower version than the existing location.
* [Mutability Rules for Trace Tags vs Metadata](/concepts/mutability-rules-for-trace-tags-vs-metadata.md) - The distinction that tags are mutable after trace logging while metadata becomes immutable once a trace is logged, influencing how context should be attached.
* [Mutable Tags vs Immutable Metadata](/concepts/mutable-tags-vs-immutable-metadata.md) - In MLflow tracing, tags on logged traces can be mutated after logging while metadata becomes immutable once the trace is logged, making metadata suitable for immutable identifiers like user IDs.
* [Mutable Tags vs Immutable Metadata in MLflow Traces](/concepts/mutable-tags-vs-immutable-metadata.md) - Distinction between tags (mutable after trace logging) and metadata (immutable after trace logging) in MLflow trace context
* [MXFP4 Quantization](/concepts/mxfp4-quantization.md) - Microscaling 4-bit floating point format that reduces memory requirements during model training and inference by representing weights in a compact 4-bit format with per-group scaling factors.
* [Mxfp4 Quantization for Large Model Loading](/concepts/mxfp4-quantization-for-model-loading.md) - A 4-bit quantization format (Mxfp4Config) used to reduce memory footprint when loading large models, with dequantization happening during forward passes to maintain training fidelity.
* [Mxfp4 Quantization for Model Loading](/concepts/mxfp4-quantization-for-model-loading.md) - A 4-bit floating point quantization format used to reduce memory footprint when loading large models (e.g., 120B parameters) into GPU memory during fine-tuning.
* [Named Entity Recognition with Spark NLP and MLflow](/concepts/named-entity-recognition-with-spark-nlp-and-mlflow.md) - Training a named entity recognition (NER) model using Spark NLP and tracking/saving it with MLflow for reusable inference on Databricks.
* [Navigating to the Data Profiling Dashboard](/concepts/accessing-the-data-profiling-dashboard.md) - Two methods to access the data profiling dashboard in Databricks: via the Catalog Explorer UI (Quality tab on a table) or via the Dashboards sidebar listing.
* [NCCL and Gloo Environment Variable Fixes for Distributed Training](/concepts/nccl-and-gloo-environment-variable-fixes-for-distributed-training.md) - Environment variable configurations (NCCL_P2P_DISABLE, NCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME) to resolve common GPU/CPU distributed training failures on Databricks
* [NCCL Errors in Multi-Node GPU Training](/concepts/nccl-errors-in-multi-node-gpu-training.md) - Common NCCL failures on Databricks including peer access issues on G5 GPUs and network interface misconfiguration, with environment variable workarounds.
* [NCCL Network Communication Failure in Multi-Node GPU Training](/concepts/nccl-network-communication-failure-in-multi-node-gpu-training.md) - Troubleshooting NCCL failures during multi-node GPU training by setting spark.executorEnv.NCCL_SOCKET_IFNAME to eth for correct network interface selection.
* [NCCL Network Configuration for Distributed GPU XGBoost](/concepts/nccl-network-configuration-for-distributed-gpu-xgboost.md) - Troubleshooting NCCL communication failures in multi-node GPU training by setting NCCL_SOCKET_IFNAME environment variable to eth
* [NCCL Network Configuration for Distributed XGBoost](/concepts/nccl-network-configuration-for-distributed-xgboost.md) - Troubleshooting NCCL communication failures in distributed XGBoost training by setting the NCCL_SOCKET_IFNAME environment variable on Databricks clusters.
* [NCCL Network Error Troubleshooting for XGBoost](/concepts/nccl-network-troubleshooting-for-xgboost.md) - Troubleshooting NCCL failures in multi-node GPU training by setting the spark.executorEnv.NCCL_SOCKET_IFNAME configuration to eth
* [NCCL Network Troubleshooting for XGBoost](/concepts/nccl-network-troubleshooting-for-xgboost.md) - Troubleshooting NCCL network failures in multi-node GPU training by setting spark.executorEnv.NCCL_SOCKET_IFNAME=eth to fix inter-GPU communication issues.
* [NCCL Socket Configuration for Spark XGBoost](/concepts/nccl-socket-configuration-for-xgboost.md) - Troubleshooting NCCL communication failures during multi-node GPU training by setting spark.executorEnv.NCCL_SOCKET_IFNAME to eth.
* [NCCL Socket Configuration for XGBoost](/concepts/nccl-socket-configuration-for-xgboost.md) - Troubleshooting NCCL failures during multi-node GPU training by setting spark.executorEnv.NCCL_SOCKET_IFNAME=eth to fix network communication issues.
* [Nested Function Tracing](/concepts/nested-function-tracing.md) - When traced functions call other traced functions, MLflow generates a hierarchical trace where parent-child span structure mirrors the call graph.
* [Nested MLflow Runs for Hyperparameter Tuning](/concepts/nested-mlflow-runs-for-tuning.md) - A pattern where CrossValidator or TrainValidationSplit logs tuning results as a parent run (containing the estimator metadata) and child runs (one per hyperparameter setting tested, with its evaluation metric).
* [Nested MLflow Runs for Model Tuning](/concepts/nested-mlflow-runs-for-tuning.md) - The logging structure where CrossValidator or TrainValidationSplit tuning results are organized as parent (main) runs and child runs per hyperparameter setting tested.
* [Nested MLflow Runs for Tuning](/concepts/nested-mlflow-runs-for-tuning.md) - Pattern where CrossValidator or TrainValidationSplit log tuning results as nested MLflow runs with a parent run for the estimator and child runs for each hyperparameter setting tested
* [Nested span tracing for complex workflows](/concepts/nested-span-tracing-for-complex-workflows.md) - Using @mlflow.trace decorators in combination with mlflow.start_span() context managers to create hierarchical parent-child span structures for multi-step pipelines.
* [Nested Span Workflows with Context Managers](/concepts/nested-span-workflows-with-context-managers.md) - Using mlflow.start_span() as a context manager within a @mlflow.trace-decorated function to create nested spans for complex, multi-step workflows like data pipelines.
* [Network Access for Serverless Materialization in Delta Sharing](/concepts/network-access-for-serverless-materialization-in-delta-sharing.md) - Shared view materialization uses serverless compute in the provider's region, which may be blocked by storage-layer firewalls or private links; requires allowlisting serverless compute IPs.
* [NetworkX](/concepts/networkx.md) - Python library for graph and network analysis suitable for small to medium-sized networks on a single compute node
* [No Isolation Shared Clusters](/concepts/no-isolation-shared-clusters.md) - A cluster type that does not respect the legacy Hive metastore disablement setting, requiring the Enforce User Isolation setting to prevent users from bypassing Unity Catalog governance.
* [No Isolation Shared Clusters and Legacy Access](/concepts/no-isolation-shared-clusters-limitation.md) - No Isolation shared clusters do not respect the legacy Hive metastore disablement setting; Enforce User Isolation must be enabled instead.
* [No Isolation Shared Clusters Limitation](/concepts/no-isolation-shared-clusters-limitation.md) - No Isolation shared clusters do not respect the legacy Hive metastore disablement setting, requiring the Enforce User Isolation setting to prevent users from bypassing the disablement.
* [NO_CLUSTERING_COLUMNS error subtype](/concepts/no_clustering_columns-error-subtype.md) - A specific error raised when a command requiring clustering columns is run on a Liquid table that has no clustering columns defined.
* [NO_HINT Sub-Error](/concepts/no_hint-sub-error.md) - A sub-condition of DELTA_SHARING_SECURABLE_DELETE_BLOCKED with no additional context, simply advising the user to use ALTER VIEW instead of deleting.
* [No-isolation shared clusters deprecation](/concepts/no-isolation-shared-clusters-deprecation.md) - No-isolation shared cluster access mode will not be available in new workspaces after September 30, 2026, requiring migration of compute to Unity Catalog access modes.
* [Nodes and Edges](/concepts/nodes-and-edges.md) - Fundamental building blocks of a graph: nodes (vertices) represent entities such as people or cities, and edges (links, relationships, arcs) represent connections or relationships between them.
* [NON_CLUSTERING_COLUMN error subtype](/concepts/non_clustering_column-error-subtype.md) - A specific error within DELTA_UNSUPPORTED_CLUSTERING_COLUMN_PREDICATES raised when a predicate references a column that is not a clustering column.
* [NON_DELTA error sub-condition](/concepts/non_delta-error-sub-condition.md) - A specific error sub-condition under DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE, raised when the source table is not a Delta format table.
* [NON_DELTA source error condition](/concepts/non_delta-error-sub-condition.md) - A specific sub-error of DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE raised when the source table format is not Delta (e.g., Parquet, CSV, JSON)
* [NON_DELTA source format error](/concepts/non_delta-source-sub-error.md) - A sub-error of DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE indicating the source table is not in Delta format, so cloning with history is unsupported.
* [NON_DELTA source restriction for clone with history](/concepts/non_delta-source-restriction-for-clone-with-history.md) - The error condition raised when cloning a Delta table with history from a source that is not in Delta format (e.g., Parquet, CSV, JSON).
* [NON_DELTA source sub-error](/concepts/non_delta-source-sub-error.md) - A specific sub-condition of DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE raised when the source table is not in Delta format.
* [NON_DELTA target error](/concepts/non_delta-target-error.md) - A subtype of DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_TARGET indicating the target table is not a Delta format table.
* [NON_DELTA target table restriction](/concepts/non_delta-target-table-restriction.md) - Delta clone with history does not support target tables that are not in Delta format.
* [NON_DETERMINISTIC_EXPRESSIONS constraint for table_changes](/concepts/non_deterministic_expressions-constraint-for-table_changes.md) - A validation rule requiring that views used with table_changes must not contain non-deterministic expressions
* [NON_DETERMINISTIC_EXPRESSIONS sub-error](/concepts/non_deterministic_expressions-sub-error.md) - A specific cause of DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED where the view contains non-deterministic expressions
* [Non-Delta target restriction for clone with history](/concepts/non-delta-target-restriction-for-clone-with-history.md) - Cloning a Delta table with history to a non-Delta format target table is not supported in Databricks.
* [Non-Delta target tables in Delta Clone with History](/concepts/non-delta-target-tables-in-delta-clone-with-history.md) - An error sub-condition (NON_DELTA) where the target table for a Delta Clone with History operation is not in Delta format
* [Non-Express Deployments (Databricks)](/concepts/non-express-deployments-databricks.md) - The traditional model serving deployment approach where model artifacts and environments are packaged into containers at deployment time, which may result in environment mismatches with training.
* [Non-Express vs Express Deployments](/concepts/non-express-vs-express-deployments.md) - The contrast between traditional deployments (packaging artifacts at deployment time, risking environment drift) and express deployments (packaging at registration time for consistency).
* [Non-linear Token Work Scaling](/concepts/non-linear-token-work-scaling.md) - The principle that computational work required for inference grows non-linearly as input or output token counts increase, rather than scaling linearly.
* [Non-LLM Metrics in DeepEval](/concepts/non-llm-metrics-in-deepeval.md) - A category of DeepEval metrics that do not require an LLM as a judge, enabling evaluation without calling a language model.
* [Non-modifying Monitoring](/concepts/non-modifying-monitoring.md) - Design principle stating that data quality monitoring does not modify any tables it monitors nor adds overhead to jobs that populate those tables.
* [Non-Python Package Dependency Logging](/concepts/non-python-package-dependency-logging.md) - The challenge and recommended approach for logging non-Python dependencies (Java, R, native/Linux packages) as artifacts alongside MLflow models, since MLflow does not automatically capture them.
* [NOT EXISTS vs NOT IN Performance for DELETE](/concepts/not-exists-vs-not-in-performance-for-delete.md) - Databricks recommends using NOT EXISTS over NOT IN in DELETE WHERE clauses because NOT IN subqueries can be slow.
* [NOT IN subquery performance warning](/concepts/not-in-subquery-performance-warning.md) - DELETE with NOT IN subqueries can be slow; Databricks recommends using NOT EXISTS instead whenever possible.
* [NOT IN subquery restrictions in Databricks DELETE](/concepts/not-in-subquery-restrictions-in-databricks-delete.md) - In Databricks SQL DELETE FROM statements, NOT IN subqueries are not supported when nested inside an OR expression, and can be slow; NOT EXISTS is recommended as a faster alternative.
* [NOT IN vs NOT EXISTS for DELETE](/concepts/not-in-vs-not-exists-for-delete.md) - Preference for NOT EXISTS over NOT IN subqueries in DELETE statements due to performance considerations
* [NOT_DELTA_TABLE constraint for table_changes](/concepts/not_delta_table-constraint-for-table_changes.md) - A validation rule requiring that views used with table_changes must ultimately reference a Delta table
* [NOT_DELTA_TABLE error reason](/concepts/not_delta_table-sub-error.md) - A sub-reason of DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED indicating the underlying view does not reference a Delta table.
* [NOT_DELTA_TABLE restriction](/concepts/not_delta_table-restriction.md) - A restriction that table_changes requires the target to be a Delta table; views referencing non-Delta tables are rejected.
* [NOT_DELTA_TABLE sub-error](/concepts/not_delta_table-sub-error.md) - A specific cause of DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED where the view does not reference a Delta table
* [NOT_SHARED_VIEW error reason](/concepts/not_shared_view-error-reason.md) - A sub-reason of DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED indicating the view is not an OpenSharing view.
* [NOT_UNIFORM_INGRESS_TABLE Error](/concepts/not_uniform_ingress_table-error.md) - An error raised when a table referenced in a Delta Uniform ingress operation is not actually configured as a uniform ingress table.
* [Notebook Development Workflow for Distributed PyTorch Training](/concepts/notebook-development-workflow-for-distributed-pytorch-training.md) - A four-step workflow to convert single-node PyTorch code to distributed training using TorchDistributor in notebooks
* [Notebook environment requirement](/concepts/notebook-environment-requirement.md) - @scorer-decorated functions used in production monitoring must be defined and registered from a Databricks notebook due to serialization constraints
* [Notebook experiment in Databricks](/concepts/notebook-experiment-in-databricks.md) - Each Databricks notebook has an associated MLflow experiment; its run data can be loaded by calling load() with no arguments on the mlflow-experiment data source.
* [Notebook experiment in Databricks MLflow](/concepts/notebook-experiment-in-databricks-mlflow.md) - The default experiment for MLflow runs when no experiment is explicitly set, where runs are automatically logged to the notebook's associated experiment.
* [Notebook Experiment Lifecycle and Restrictions](/concepts/notebook-experiment-lifecycle-and-restrictions.md) - Notebook experiments share name/ID with their notebook and cannot be independently deleted or renamed. Deleting a notebook experiment via UI or API also deletes the notebook. Git-folder notebook experiments can only be managed at the Git folder level.
* [Notebook-required function serialization](/concepts/notebook-required-scorer-serialization.md) - @scorer-decorated functions for production monitoring must be defined and registered from a Databricks notebook due to serialization requirements
* [Notebook-Required Scorer Serialization](/concepts/notebook-required-scorer-serialization.md) - @scorer-decorated functions used in production monitoring must be defined and registered from a Databricks notebook because the monitoring service serializes function code for remote execution and requires the notebook environment.
* [Notebook-Scoped Libraries and Cluster Policies for Deep Learning](/concepts/notebook-scoped-libraries-and-cluster-policies-for-deep-learning.md) - Customizing deep learning environments at notebook, cluster, and job levels using notebook-scoped libraries, cluster-level libraries, and cluster policies to enforce consistent configurations
* [NVIDIA H100 80GB HBM3 on Databricks](/concepts/nvidia-h100-80gb-hbm3-on-databricks.md) - Configuration and use of NVIDIA H100 80GB HBM3 GPUs via Databricks Serverless GPU compute for large model training
* [NVIDIA H100 GPU on Databricks](/concepts/nvidia-h100-gpu-on-databricks.md) - The NVIDIA H100 80GB HBM3 accelerator available as a serverless GPU option on Databricks, offering 8 chips per node with 80GB memory each for large model training.
* [OAuth Client Credentials Grant (M2M) Flow](/concepts/oauth-client-credentials-grant-m2m-flow.md) - A machine-to-machine authentication flow where an application (nightly job, VM) uses client credentials to obtain short-lived JWTs from an IdP, used as OAuth tokens to access Databricks Delta Sharing endpoints.
* [OAuth Profile File (oauth_config.share)](/concepts/oauth-profile-file-oauth_configshare.md) - A JSON configuration file downloaded from the OIDC profile portal containing endpoint, tokenEndpoint, client credentials, and scope, used to configure the Delta Sharing Python client for OIDC-based access.
* [OAuth Token Authentication for Route-Optimized Endpoints](/concepts/oauth-token-authentication-for-route-optimized-endpoints.md) - Route-optimized endpoints require Databricks in-house OAuth tokens instead of personal access tokens for authentication.
* [OAuth token authentication for serving endpoints](/concepts/oauth-token-authentication-for-serving-endpoints.md) - OAuth tokens are the only supported authentication method for querying route-optimized endpoints; personal access tokens are not supported.
* [Object detection with RetinaNet on Databricks](/concepts/object-detection-with-retinanet-on-databricks.md) - A tutorial notebook demonstrating how to train an object detection model using RetinaNet on serverless GPU with Databricks AI Runtime.
* [Object detection with YOLO11n on Databricks](/concepts/object-detection-with-yolo11n-on-databricks.md) - A tutorial notebook demonstrating how to train a YOLO11n object detection model on the COCO128 dataset using serverless GPU, with MLflow tracking and Model Serving deployment.
* [Object Ownership in Hive Metastore](/concepts/object-ownership-in-hive-metastore.md) - The model where a user who creates a schema, table, view, or function becomes its owner with all privileges and grant authority; workspace admins can transfer ownership.
* [Object Ownership in Unity Catalog](/concepts/object-ownership-in-unity-catalog.md) - Every securable object has an owner who possesses all privileges on that object and can grant privileges to others. Ownership can be transferred by the current owner, a metastore admin, or a user with MANAGE privilege.
* [Objective Function Pattern for Hyperopt](/concepts/objective-function-pattern-for-hyperopt.md) - The pattern for writing Hyperopt objective functions: define a function taking hyperparameters as arguments, train/evaluate a model, and return a dictionary with 'loss' (to minimize) and 'status'.
* [Offline Feature Store](/concepts/offline-feature-store.md) - The storage layer for feature tables materialized as Delta tables, used for feature discovery, model training, and batch inference.
* [Offline-to-Production Evaluation Continuity](/concepts/offline-to-production-evaluation-continuity.md) - The principle that the same evaluation logic and scorers used during development can also run in production monitoring, providing consistent quality assessment across the AI lifecycle.
* [Offline-to-Production Monitoring Consistency](/concepts/offline-to-production-evaluation-continuity.md) - The principle that the same scorers and evaluation logic used in offline development can be reused in production monitoring, providing consistent quality metrics across the AI lifecycle.
* [OfflineStoreConfig](/concepts/offlinestoreconfig.md) - Configuration specifying the Unity Catalog catalog, schema, and table name prefix where materialized features are stored as Delta tables for offline use (training, batch scoring).
* [OIDC Federation for Data Sharing](/concepts/oidc-federation-for-data-sharing.md) - An authentication method using OpenID Connect federation for Databricks-to-Open sharing, supporting both user-to-machine (U2M) and machine-to-machine (M2M) flows, used with tools like Tableau and Power BI.
* [OIDC Federation for Delta Sharing](/concepts/oidc-federation-for-delta-sharing.md) - Authentication method using Open ID Connect federation to allow recipients without a Databricks workspace to access OpenSharing shares
* [OIDC Federation for M2M Delta Sharing](/concepts/oidc-federation-for-m2m-delta-sharing.md) - Using Open ID Connect federation with the OAuth Client Credentials grant flow to let non-Databricks recipients authenticate to Delta Sharing shares without long-lived bearer tokens.
* [OIDC Federation for OpenSharing](/concepts/oidc-federation-for-opensharing.md) - Authentication method that allows data providers to federate identity verification to a recipient's IdP using OIDC, enabling short-lived JWT tokens instead of long-lived bearer tokens for OpenSharing.
* [OIDC Federation Policy](/concepts/oidc-federation-policy.md) - A configuration policy specifying issuer URL, subject claim (oid, groups, azp), subject identifier, and audience(s) that the JWT must match for authentication to succeed.
* [OIDC Federation Policy Configuration](/concepts/oidc-federation-policy-configuration.md) - A policy object in Databricks that maps JWT claims (issuer, subject, audience) from a recipient's IdP to access permissions, specifying the issuer URL, subject claim type (oid, groups, or azp), subject value, and allowed audiences.
* [OIDC Profile Generation Portal](/concepts/oidc-profile-generation-portal.md) - Portal URL provided by Databricks providers that recipients use to obtain profile files or OAuth sign-in endpoints for U2M access
* [OIDC Token Federation (Delta Sharing)](/concepts/oidc-token-federation-delta-sharing.md) - An alternative authentication flow for Open Sharing that offers security and convenience advantages over the bearer token method, recommended as a replacement for long-lived tokens.
* [OIDC Token Federation for Open Sharing](/concepts/oidc-token-federation-for-opensharing.md) - An alternative authentication method to bearer tokens for Databricks-to-Open sharing, offering advantages in security and convenience for recipients without Databricks workspaces.
* [OIDC Token Federation for OpenSharing](/concepts/oidc-token-federation-for-opensharing.md) - An alternative authentication flow to bearer tokens for Databricks-to-Open sharing that offers improved security and convenience by using OpenID Connect federation.
* [OIDC vs Bearer Token Authentication](/concepts/oidc-vs-bearer-token-authentication.md) - Comparison between OIDC federation and long-lived Databricks-issued bearer tokens for OpenSharing, highlighting trade-offs in security, MFA support, and credential management.
* [OLD_ARCHITECTURE_NOT_SUPPORTED](/concepts/old_architecture_not_supported.md) - Sub-error indicating that only streaming tables using the default publishing mode are supported for deep clone operations.
* [OLD_ARCHITECTURE_NOT_SUPPORTED sub-error](/concepts/old_architecture_not_supported.md) - Sub-error indicating that only streaming tables using the default publishing mode are supported for deep clone; legacy architectures are rejected.
* [Ollama](/concepts/ollama.md) - An open-source platform for running large language models (LLMs) locally on personal devices, supporting models like Llama 3.2, Gemma 2, and Mistral.
* [Ollama OpenAI API Compatibility](/concepts/ollama-openai-api-compatibility.md) - Ollama's local endpoint is compatible with the OpenAI API format, allowing the OpenAI SDK client to query it by setting a custom base_url and dummy api_key.
* [Olmo3 7B Instruct Model](/concepts/olmo3-7b-instruct-model.md) - An open-source 7 billion parameter instruction-tuned language model from Allen AI, used as the base for fine-tuning
* [On-Demand Feature Computation for Real-Time Inference](/concepts/on-demand-feature-computation-for-real-time-inference.md) - Databricks Feature Store supports on-demand computation of features for real-time applications, handling all feature computation tasks to simplify client-side code.
* [On-demand Feature Computation with Python UDFs](/concepts/on-demand-feature-computation-with-python-udfs.md) - Pattern for computing features at inference time using registered Python UDFs that transform raw input data into feature values on the fly
* [On-Demand Feature Computation with UDFs](/concepts/on-demand-feature-computation-with-udfs.md) - Python UDFs registered in Unity Catalog can be used as FeatureFunctions to compute features on-demand during model inference
* [On-demand features (Databricks)](/concepts/on-demand-features-databricks.md) - Features computed at inference time using Python UDFs in Unity Catalog, used when feature values depend on request-time inputs.
* [On-demand features limitations](/concepts/on-demand-features-limitations.md) - Constraints on on-demand features: MapType and ArrayType output types are unsupported, and older library versions require extra Unity Catalog permissions.
* [On-Device LLM Inference](/concepts/on-device-llm-inference.md) - The paradigm of running large language models locally on user devices rather than relying on cloud-hosted inference APIs, with privacy and latency benefits.
* [One-vs-All Fairness Scoring for Multi-Class Models](/concepts/one-vs-all-fairness-scoring-for-multi-class-models.md) - A technique for computing fairness metrics across all predicted classes in a multi-class classification model by treating each class in a one-vs-all manner
* [Online Feature Computation and Serving](/concepts/online-feature-computation-and-serving.md) - On-demand computation of features for real-time applications with millisecond-latency serving endpoints.
* [Online Feature Serving](/concepts/online-feature-serving.md) - Real-time feature computation and lookup with millisecond latency, including on-demand computation, available via single-click endpoints in Databricks.
* [Online Feature Store](/concepts/online-feature-store.md) - A high-performance, low-latency serving layer for real-time ML inference, powered by Databricks Lakebase, that syncs feature tables from the offline store.
* [Online Feature Store (Databricks)](/concepts/online-feature-store-databricks.md) - A managed feature store that serves feature tables to model and feature serving endpoints, replacing legacy online tables for ML serving.
* [Online Feature Store Capacity and Compute Sizing](/concepts/online-feature-store-capacity-and-compute-sizing.md) - The capacity setting (CU_1, CU_2, CU_4, CU_8) that controls compute power for an online store, corresponding to Lakebase Provisioned capacity levels.
* [Online Feature Store Capacity and Cost Optimization](/concepts/online-feature-store-cost-optimization.md) - Best practices for managing online store capacity settings (CU_1 through CU_8) and minimizing costs by reusing stores, right-sizing, and deleting unused stores.
* [Online Feature Store Cost Optimization](/concepts/online-feature-store-cost-optimization.md) - Best practices for managing costs including reusing online stores across projects, right-sizing capacity starting with CU_2, and deleting unused stores to avoid continuous charges.
* [Online Feature Store Lifecycle Management](/concepts/online-feature-store-lifecycle-management.md) - Full CRUD operations for online stores and tables including creation, listing, retrieval, capacity updates, and deletion via the FeatureEngineeringClient and Databricks SDK.
* [Online Feature Store Publish Modes](/concepts/online-feature-store-publish-modes.md) - The publish_mode parameter (TRIGGERED, CONTINUOUS, etc.) that controls how and when online tables are synced with offline feature tables, replacing the legacy streaming parameter.
* [Online Feature Store Read Replicas](/concepts/online-feature-store-read-replicas.md) - Read replicas that distribute read traffic across multiple compute instances, reducing latency and improving scalability for high-concurrency workloads, with support for up to 3 replicas.
* [Online Feature Table Publishing](/concepts/online-feature-table-publishing.md) - The process of synchronizing offline feature tables to online stores using the publish_table API, including prerequisites like primary keys and Change Data Feed.
* [Online Store Authentication for SQL Stores](/concepts/online-store-authentication-for-sql-stores.md) - The pattern of storing user/password credentials in Databricks secrets for SQL-based third-party online stores, as opposed to access keys used for DynamoDB.
* [Online Store Authentication via Secret Prefixes](/concepts/online-store-authentication-via-secret-prefixes.md) - Using write_secret_prefix and read_secret_prefix parameters in online store specifications to reference Databricks secrets for authentication.
* [Online Store Capacity and Cost Optimization](/concepts/online-feature-store-cost-optimization.md) - Best practices for managing online store costs including right-sizing capacity (CU_1 through CU_8), reusing stores across multiple projects, and deleting unused stores that continuously incur costs.
* [Online Store Capacity and Read Replicas](/concepts/online-store-capacity-and-read-replicas.md) - Online stores support configurable compute capacity (CU_1 through CU_8) and up to 3 read replicas for distributing read traffic and improving high-concurrency performance.
* [Online Store Feature Publishing](/concepts/online-store-feature-publishing.md) - The mechanism by which feature values are published to online stores for low-latency serving, with ArrayType and MapType features stored in JSON format.
* [Online Store Publishing](/concepts/online-store-publishing.md) - Tracking which online stores a feature table has been published to, visible in the Workspace Feature Store UI.
* [Online Store Publishing of Features](/concepts/online-store-publishing-of-features.md) - Features published to online stores for real-time serving with ArrayType and MapType stored in JSON format, providing millisecond-latency serving endpoints.
* [Online Table (Databricks)](/concepts/online-table-databricks.md) - Read-only, row-oriented, serverless copy of a Delta Table optimized for low-latency online access, used with Model Serving, Feature Serving, and RAG applications.
* [Online Table Integration with Lakehouse Federation](/concepts/online-table-integration-with-lakehouse-federation.md) - Online tables can be queried via Lakehouse Federation using Serverless SQL warehouses, but only for SELECT operations and intended for interactive/debugging use, not production.
* [Online Table Limitations](/concepts/online-table-limitations.md) - Constraints on online tables including 1 per source table, 1000 column max, 2TB metastore-wide preview limit, 750 MB/sec max read throughput, and restrictions on primary key data types.
* [Online Table Limitations and Constraints](/concepts/online-table-limitations.md) - Comprehensive set of constraints for Databricks online tables including metastore-wide size limits, column count limits, data type restrictions, primary key requirements, and throughput caps.
* [Online Table Overwrite and Merge Modes](/concepts/online-table-overwrite-and-merge-modes.md) - Two modes for updating online feature tables: 'overwrite' completely replaces the online table from offline data, while 'merge' allows conditional updates using filter_condition for selective row replacement.
* [Online Table Permission & Service Principal Model](/concepts/online-table-permission-service-principal-model.md) - A unique auto-generated service principal for each serving endpoint that allows autonomous data access from online tables, independent of the user who created the resource, with the principal lifetime tied to the endpoint lifetime.
* [Online Table Permission Model](/concepts/online-table-permission-model.md) - Permission requirements for creating and managing online tables, including SELECT/USE_CATALOG/CREATE_TABLE privileges, REFRESH privilege, and an auto-created service principal for endpoints.
* [Online Table Permissions and Security Model](/concepts/online-table-permission-model.md) - Permission requirements for creating and managing online tables, including Unity Catalog privileges and the automatic service principal model that enables endpoints to access data independently of the creating user.
* [Online Table Permissions Model](/concepts/online-table-permission-model.md) - Required Unity Catalog privileges (SELECT, USE CATALOG, USE SCHEMA, CREATE TABLE, REFRESH) and the automatic service principal created for endpoints.
* [Online Table Sync Modes](/concepts/online-table-sync-modes.md) - Three synchronization modes for updating online tables from source Delta tables: Snapshot (full copy), Triggered (manual/periodic incremental updates), and Continuous (real-time streaming sync).
* [Online Table Troubleshooting](/concepts/online-table-troubleshooting.md) - Common issues with online tables including missing Create option, unavailable sync modes, pipeline failures due to deleted source tables, firewall blocks, and exceeded metastore size limits.
* [Online Tables](/concepts/online-tables.md) - Unity Catalog tables optimized for low-latency serving of feature data, used to host structured data that RAG applications need to query in real time.
* [Online Tables (Databricks)](/concepts/online-table-databricks.md) - Read-only, row-oriented, serverless copies of Delta Tables optimized for low-latency online access, used with Model Serving, Feature Serving, and RAG applications.
* [Online Tables for RAG Applications](/concepts/online-tables-for-rag-applications.md) - Using online tables to host structured data for retrieval-augmented generation (RAG) applications, where a feature serving endpoint provides low-latency lookups integrated with LangChain agents.
* [Online Tables in Unity Catalog](/concepts/online-tables-in-unity-catalog.md) - Real-time synchronised tables in Databricks Unity Catalog that enable low-latency lookups of structured data for serving in applications.
* [Online Tables Limitations](/concepts/online-table-limitations.md) - Boundaries including 2TB metastore-wide limit, 1000 column max, 64KB string limit, 2MB row max, 750 MB/s read throughput, and restrictions on certain source table types.
* [OnlineStoreConfig](/concepts/onlinestoreconfig.md) - Configuration for the online feature store used by model serving, creating Delta tables streamed to an Online Feature Store with the same name prefix.
* [Open Sharing (Databricks-to-Open Sharing)](/concepts/open-sharing-databricks-to-open.md) - An authentication flow in Databricks Delta Sharing that allows data providers to share data with recipients who do not have access to a Unity Catalog-enabled Databricks workspace, using bearer tokens or OIDC federation.
* [Open Sharing (Databricks-to-Open)](/concepts/open-sharing-databricks-to-open.md) - A Delta Sharing authentication flow that allows data providers to share data with recipients who do not have access to a Unity Catalog-enabled Databricks workspace, using bearer tokens or OIDC federation.
* [Open Sharing views in Databricks](/concepts/open-sharing-views-in-databricks.md) - A view type in Databricks that supports Open Sharing protocol, required by table_changes (views that are not Open Sharing views trigger the NOT_SHARED_VIEW error)
* [Open Source Commitment in Databricks ML](/concepts/open-source-commitment-in-databricks-ml.md) - Databricks' commitment to the open-source ML ecosystem by supporting frameworks like scikit-learn, PyTorch, TensorFlow, and Hugging Face, and storing artifacts in open formats based on open-source MLflow, Unity Catalog, and Delta Lake.
* [Open source MLflow data portability](/concepts/open-source-mlflow-data-portability.md) - The core MLflow data model, API, and SDK are fully open source, ensuring users can export and use their MLflow data anywhere despite using the managed Databricks platform.
* [Open Source vs Databricks-Managed MLflow](/concepts/open-source-vs-databricks-managed-mlflow.md) - Key differences between the open-source MLflow and the Databricks-managed version, including exclusive enterprise features, Unity Catalog integration, and differences in telemetry collection defaults.
* [Open-Source Foundation of Databricks ML](/concepts/open-source-foundation-of-databricks-ml.md) - Databricks ML platform built on open-source technologies including MLflow, Unity Catalog APIs, and Delta Lake, ensuring portability of models, features, and data.
* [Open-Source ML Ecosystem on Databricks](/concepts/open-source-ml-ecosystem-on-databricks.md) - Full support for popular open-source ML frameworks (scikit-learn, PyTorch, TensorFlow, etc.) and open formats (Delta Lake, MLflow, Unity Catalog APIs) ensuring portability.
* [OpenAI and LangChain Credential Injection for Model Serving](/concepts/openai-and-langchain-credential-injection-for-model-serving.md) - Using secrets-based environment variables to securely pass credentials when deploying OpenAI and LangChain MLflow model flavors to Databricks serving endpoints
* [OpenAI API Compatibility in Databricks](/concepts/openai-compatible-api-interface.md) - Foundation Model APIs are compatible with the OpenAI client SDK and API, allowing users to use familiar OpenAI tooling to query models on Databricks.
* [OpenAI client compatibility](/concepts/openai-client-compatibility.md) - Foundation Model APIs are compatible with the OpenAI client SDK and API, allowing users to query Databricks-hosted models using familiar OpenAI tooling.
* [OpenAI High Risk Use Case Mitigation Framework](/concepts/openai-high-risk-use-case-mitigation-framework.md) - A structured set of mitigation requirements that end users must implement when deploying OpenAI models for high-risk use cases on Databricks.
* [OpenAI Responses API (on Databricks)](/concepts/openai-responses-api-on-databricks.md) - Databricks-hosted access to OpenAI's Responses API for GPT-5 and GPT-4o models, supporting text and image inputs.
* [OpenAI-compatible API (Databricks)](/concepts/openai-compatible-api-interface.md) - Databricks Foundation Model APIs are compatible with the OpenAI client SDK and API, allowing users to use familiar OpenAI tooling to query Databricks-hosted models.
* [OpenAI-Compatible API Auto-Tracing](/concepts/openai-compatible-api-auto-tracing.md) - MLflow leverages the OpenAI SDK to automatically trace Databricks Foundation Model calls because these models expose an OpenAI-compatible API, enabling reuse of mlflow.openai.autolog().
* [OpenAI-compatible API interface](/concepts/openai-compatible-api-interface.md) - Foundation Model APIs are compatible with the OpenAI client SDK and API, allowing users to use the OpenAI client for querying supported models.
* [OpenAI-Compatible Client for Databricks](/concepts/openai-compatible-client-for-databricks.md) - Using the OpenAI Python client to query Databricks-hosted chat models with a familiar chat completions interface.
* [OpenSharing](/concepts/opensharing.md) - An open standard for secure data sharing, supporting both Databricks-to-Databricks and Databricks-to-Open sharing models.
* [OpenSharing (Databricks Delta Sharing)](/concepts/opensharing-databricks-delta-sharing.md) - Databricks' implementation of the Delta Sharing protocol for sharing data across workspaces and platforms using Unity Catalog.
* [OpenSharing (Databricks-to-Open) Protocol](/concepts/opensharing-databricks-to-open-protocol.md) - An open sharing protocol that uses bearer tokens and credential files to provide secure read-only access to shared data, with providers managing credential expiration and rotation.
* [OpenSharing (Delta Sharing)](/concepts/opensharing-delta-sharing.md) - An open protocol for sharing datasets across platforms; objects shared via OpenSharing cannot be dropped — only altered — to prevent breaking downstream consumers.
* [OpenSharing Access Control Scope](/concepts/opensharing-access-control-scope.md) - The three specific endpoints affected by IP access lists: OpenSharing OSS Protocol REST API access, OpenSharing activation URL access, and OpenSharing credential file download.
* [OpenSharing Apache Spark Connector](/concepts/opensharing-apache-spark-connector.md) - A connector that enables reading shared data via Apache Spark using the delta-sharing-spark library, supporting batch queries, change data feed (CDF), structured streaming, deletion vectors, column mapping, and row tracking.
* [OpenSharing Audit Log Fields for Pre-signed URL Queries](/concepts/opensharing-audit-log-fields-for-pre-signed-url-queries.md) - Provider audit logs for deltaSharingQueriedTable events contain fields like scannedAddFileSize, numRecords, tableVersion, limitHint, and userAgent to track what was shared.
* [OpenSharing Audit Log Fields for STS Token Queries](/concepts/opensharing-audit-log-fields-for-sts-token-queries.md) - Provider audit logs for generateTemporaryTableCredentials events capture recipient_name, share_name, credential_type, operation, and table_url to track STS-token-based sharing.
* [OpenSharing Audit Log System](/concepts/opensharing-audit-log-system.md) - The system for auditing and monitoring data sharing events in Databricks Delta Sharing, capturing provider and recipient actions via system tables or audit log delivery.
* [OpenSharing Audit Logging Overview](/concepts/opensharing-audit-logging-overview.md) - Audit logging infrastructure for monitoring Delta Sharing (OpenSharing) events, covering both provider and recipient perspectives with logged events and error messages.
* [OpenSharing Audit Logs](/concepts/opensharing-audit-logs.md) - Audit logs that record actions taken by data providers and recipients in Delta Sharing, stored in system.access.audit or delivered to an S3 bucket
* [OpenSharing BI Tool Connectors (Power BI and Tableau)](/concepts/opensharing-bi-tool-connectors-power-bi-and-tableau.md) - Specialized connectors for Power BI Desktop and Tableau that allow users to discover, analyze, and visualize OpenSharing datasets through a graphical interface using credential files.
* [OpenSharing Catalog](/concepts/opensharing-catalog.md) - A special catalog type in Unity Catalog created from a share, providing read-only access to shared data assets (tables, views, volumes, notebooks) with a 3-level namespace structure (catalog.schema.table).
* [OpenSharing Credential File](/concepts/opensharing-credential-file.md) - A downloadable file used in Databricks-to-Open sharing with bearer tokens to authenticate and read shared data; can only be downloaded once and must be stored securely.
* [OpenSharing Egress Cost Management](/concepts/opensharing-egress-cost-management.md) - Strategies and tools provided by Databricks to monitor, manage, and minimize cloud vendor data egress fees when sharing data across regions using OpenSharing.
* [OpenSharing Egress Pipeline Notebooks](/concepts/opensharing-egress-pipeline-notebooks.md) - Two Databricks Marketplace notebooks (IP Ranges Mapping Pipeline and Egress Cost Analysis Pipeline) that use Lakeflow Spark Declarative Pipelines to monitor egress usage patterns and costs by joining logs with cloud provider IP ranges and OpenSharing system tables.
* [OpenSharing Error Handling](/concepts/opensharing-error-handling.md) - Categorized error messages and error codes (e.g., PERMISSION_DENIED, SHARE_DOES_NOT_EXIST, TABLE_DOES_NOT_EXIST) logged for failed OpenSharing actions on both provider and recipient sides.
* [OpenSharing Error Logging](/concepts/opensharing-error-logging.md) - Standardized error messages logged for failed OpenSharing operations in both provider and recipient audit logs, including errors for missing entities, permission denials, and configuration issues
* [OpenSharing Error Message Catalog](/concepts/opensharing-error-message-catalog.md) - Standardized error messages logged in OpenSharing audit logs for both providers and recipients, covering permission, existence, and configuration failures.
* [OpenSharing for SAP BDC](/concepts/opensharing-for-sap-bdc.md) - The Delta Sharing-based framework used to enable bidirectional data sharing between Databricks and SAP Business Data Cloud, requiring the metastore to have OpenSharing enabled.
* [OpenSharing in Delta Sharing](/concepts/opensharing-in-delta-sharing.md) - The sharing protocol underlying Databricks Delta Sharing that, when active on a securable object, blocks its deletion to prevent breaking active sharing recipients.
* [OpenSharing IP Access List](/concepts/opensharing-ip-access-list.md) - An IP-based allow list that data providers can assign to recipients using the Databricks-to-Open sharing protocol to restrict access to shared data from specific IP addresses or CIDR ranges.
* [OpenSharing IP Access List Audit Logging](/concepts/opensharing-ip-access-list-audit-logging.md) - Audit events triggered by IP access list operations, including recipient create/update operations and denial of access to OpenSharing REST API, activation URL, and credential file download.
* [OpenSharing Limitations and Constraints](/concepts/opensharing-limitations-and-constraints.md) - A collection of known limitations across OpenSharing clients, including memory constraints, unsupported features for streaming tables and materialized views, column mapping/CDF limitations, and connector-specific restrictions.
* [OpenSharing Profile Path Syntax](/concepts/opensharing-profile-path-syntax.md) - The URL-like path format '<profile-path>#<share-name>.<schema-name>.<table-name>' used to locate shared tables in Spark DataFrame read operations.
* [OpenSharing Protocol](/concepts/opensharing-protocol.md) - Databricks protocol for sharing data with recipients who don't have a Unity Catalog-enabled Databricks workspace
* [OpenSharing Provider Object](/concepts/opensharing-provider-object.md) - A securable object in a recipient's Unity Catalog metastore that represents a data provider organization sharing data via OpenSharing on Databricks.
* [OpenSharing Python and Pandas Connector](/concepts/opensharing-python-and-pandas-connector.md) - A Python connector (delta-sharing library) that enables listing and reading shared tables as pandas DataFrames, with support for change data feed queries.
* [OpenSharing Python Client for Delta Sharing](/concepts/opensharing-python-client-for-delta-sharing.md) - The open-source Python library (delta-sharing >=1.3.1) used by non-Databricks recipients to list shared tables and read shared data via the SharingClient, configured with the oauth_config.share profile.
* [OpenSharing Recipient](/concepts/opensharing-recipient.md) - A person or group outside the data provider's organization who receives access to shared data; represented as a 'recipient' object in the provider's Databricks account.
* [OpenSharing Recipient Activation Link](/concepts/opensharing-recipient-activation-link.md) - A URL generated by Databricks when creating a recipient object that allows the recipient to download a credential file once for authenticating and accessing shared Delta Sharing data.
* [OpenSharing Recipient Creation Workflow](/concepts/opensharing-recipient-creation-workflow.md) - Step-by-step process for data providers to create an OIDC-based recipient in Databricks, configure federation policy, and share an OIDC profile portal URL with the recipient.
* [OpenSharing Recipient Object](/concepts/opensharing-recipient-object.md) - A named object in Databricks Unity Catalog that represents the identity of a user or group with whom data is shared via Delta Sharing/OpenSharing. It supports multiple authentication types: DATABRICKS (Databricks-to-Databricks), TOKEN (bearer token for non-Databricks users), OAUTH_CLIENT_CREDENTIALS, and OIDC_FEDERATION.
* [OpenSharing SecureConnect](/concepts/opensharing-secureconnect.md) - A Databricks feature that allows providers to share data from cloud storage behind firewalls or private endpoints without needing to allowlist each recipient's network individually.
* [OpenSharing Share](/concepts/opensharing-share.md) - A securable object in Unity Catalog used to share data and AI assets (tables, views, models, notebooks, volumes, etc.) with one or more recipients.
* [OpenSharing Share (Unity Catalog)](/concepts/opensharing-share.md) - A securable object in Unity Catalog that bundles data assets (tables, views, volumes, notebooks, AI models) for sharing with recipients via Delta Sharing / OpenSharing.
* [OpenSharing Share Object](/concepts/opensharing-share.md) - A securable object in Unity Catalog used to share data and AI assets (tables, views, volumes, notebooks, models, FeatureSpecs, Genie Spaces) with one or more recipients. Shares can contain assets from only one metastore and support dynamic addition/removal of assets.
* [OpenSharing Sharing Identifier](/concepts/opensharing-sharing-identifier.md) - A unique string in the format '<cloud>:<region>:<uuid>' that identifies a Unity Catalog metastore, used to establish Databricks-to-Databricks sharing connections.
* [OpenSharing using OIDC Federation](/concepts/opensharing-using-oidc-federation.md) - Pattern for data recipients to use OpenID Connect (OIDC) federation with an Iceberg REST Catalog client to access Delta Sharing/OpenSharing shares created in Databricks.
* [OpenSharing view compatibility](/concepts/opensharing-view-compatibility.md) - table_changes on views is only supported for OpenSharing views; non-OpenSharing views trigger NOT_SHARED_VIEW.
* [OpenSharing views](/concepts/opensharing-views.md) - A specific type of view in Databricks that supports open sharing protocols; referenced as a requirement for table_changes on views
* [OpenSharing with ABAC Policies](/concepts/opensharing-with-abac-policies.md) - The capability to share Databricks tables and views protected by Attribute-Based Access Control (ABAC) row filters and column masks through OpenSharing, requiring Databricks Runtime 16.4+ or serverless compute.
* [OpenSharing with ABAC-Secured Tables](/concepts/opensharing-with-abac-secured-tables.md) - How Delta Sharing (OpenSharing) works with tables governed by ABAC row filter and column mask policies
* [OpenSharing with OIDC Federation](/concepts/opensharing-with-oidc-federation.md) - An authentication method for Databricks-to-Open sharing using OpenID Connect (OIDC) token federation, supporting both User-to-Machine (U2M) and Machine-to-Machine (M2M) flows.
* [OpenTelemetry Collector Configuration for MLflow](/concepts/opentelemetry-collector-configuration-for-mlflow.md) - MLflow uses the standard OTLP exporter to send traces and metrics to an OpenTelemetry Collector, supporting configuration of endpoint URL, protocol (gRPC vs HTTP), headers, and service name through environment variables.
* [OpenTelemetry Compatibility](/concepts/opentelemetry-compatibility.md) - MLflow Tracing's compatibility with OpenTelemetry, an industry-standard observability specification, enabling export of trace data to existing observability stacks.
* [OpenTelemetry Compatibility in MLflow Tracing](/concepts/opentelemetry-compatibility.md) - MLflow Tracing's integration with the OpenTelemetry standard, allowing export of trace data to existing observability stacks.
* [OpenTelemetry Compatibility with MLflow Tracing](/concepts/opentelemetry-compatibility.md) - MLflow Tracing's compatibility with the OpenTelemetry observability standard, enabling export of trace data to existing observability stacks and services.
* [OpenTelemetry Export for MLflow Traces](/concepts/opentelemetry-export-for-mlflow-traces.md) - Compatibility between MLflow Tracing and the OpenTelemetry industry-standard observability specification, allowing trace data to be exported to existing observability stacks.
* [OpenTelemetry for Model Serving](/concepts/opentelemetry-for-model-serving.md) - Persists system logs, application logs, metrics, and traces to Unity Catalog Delta tables using OpenTelemetry for long-term retention and compliance.
* [OpenTelemetry Instrumentation for Custom Models](/concepts/opentelemetry-instrumentation-for-custom-models.md) - Using the OpenTelemetry SDK to instrument MLflow pyfunc models with custom counters, spans, attributes, and logging for endpoint telemetry capture.
* [OpenTelemetry MLflow Span Attribute Mapping](/concepts/opentelemetry-mlflow-span-attribute-mapping.md) - How OpenTelemetry GenAI semantic convention attributes map to MLflow span types and UI rendering in Databricks managed MLflow.
* [OpenTelemetry Protocol (OTLP) Endpoint in Databricks MLflow](/concepts/opentelemetry-protocol-otlp-endpoint-in-databricks-mlflow.md) - Databricks exposes an OTLP-compatible API at /api/2.0/otel/v1/traces that accepts OpenTelemetry spans and stores them in Unity Catalog tables.
* [OpenTelemetry Trace Span Schema in Unity Catalog](/concepts/opentelemetry-trace-span-schema-in-unity-catalog.md) - The detailed schema of the trace_unified view including span structures, events, links, status, resource, and instrumentation scope fields.
* [OpenTelemetry Trace Storage in Unity Catalog](/concepts/opentelemetry-trace-storage-in-unity-catalog.md) - The foundational capability to store OTel trace data (spans, logs, annotations) as Delta tables in Unity Catalog, enabling SQL-based querying and integration with Databricks governance and processing tools.
* [OpenTelemetry Traces in Unity Catalog](/concepts/opentelemetry-traces-in-unity-catalog.md) - Storage of OTel spans, logs, and annotations as managed Delta tables in Unity Catalog with a defined schema and trace_unified view.
* [Optimized Writes for Partitioned Tables](/concepts/optimized-writes-for-partitioned-tables.md) - Technique to reduce the number of small output files produced by MERGE on partitioned tables by enabling optimized writes, addressing a common performance bottleneck.
* [Optuna](/concepts/optuna.md) - A lightweight Python framework for hyperparameter tuning with dynamic search spaces, parallelization via Joblib, and MLflow integration.
* [Optuna as Hyperopt replacement on Databricks](/concepts/optuna-as-hyperopt-replacement-on-databricks.md) - Databricks recommends Optuna for single-node hyperparameter optimization as a replacement for Hyperopt
* [Optuna for Hyperparameter Tuning](/concepts/optuna-for-hyperparameter-tuning.md) - Optuna provides adaptive hyperparameter tuning for machine learning models on Databricks, helping automate the search for optimal hyperparameters including batch size and learning rate combinations.
* [Optuna for single-node optimization on Databricks](/concepts/optuna-for-single-node-optimization-on-databricks.md) - Optuna is recommended by Databricks as the replacement for Hyperopt for single-node hyperparameter optimization workloads.
* [Optuna Hyperparameter Tuning on Databricks](/concepts/optuna-hyperparameter-tuning-on-databricks.md) - Using Optuna for adaptive hyperparameter tuning to optimize deep learning model performance, including batch size and learning rate tuning.
* [Optuna objective function](/concepts/optuna-objective-function.md) - A user-defined function that defines the hyperparameter search space and returns a metric value for Optuna to minimize or maximize
* [Optuna Trial object](/concepts/optuna-trial-object.md) - Core abstraction in Optuna that provides suggest methods to generate hyperparameter values during optimization
* [OSS Spark Iceberg REST Catalog Configuration](/concepts/oss-spark-iceberg-rest-catalog-configuration.md) - Configuration for open-source Apache Spark to connect to an Iceberg REST Catalog, requiring specific JARs (Iceberg Spark runtime and AWS bundle), SparkSession extensions, and OAuth2 authentication properties.
* [Output Reducers](/concepts/output-reducers.md) - Custom aggregation functions for streaming traces that consolidate yielded elements into a single span output, with patterns for token aggregation, metrics, and error collection.
* [Output Reducers for Streaming Traces](/concepts/output-reducers.md) - A mechanism to aggregate elements yielded by a traced generator or iterator into a single span output using a custom reducer function, with support for token aggregation, metrics aggregation, and error collection.
* [Override Feature Values in Online Scoring](/concepts/override-feature-values-in-online-scoring.md) - Users can override automatically looked-up feature values during REST API scoring by including the desired feature values in the API payload, as long as the data types match the model's expectations.
* [OVERWRITE_REQUIRED sub-error](/concepts/overwrite_required-sub-error.md) - A DELTA_METADATA_MISMATCH sub-error that occurs when schema or partitioning changes require setting overwriteSchema to true
* [overwriteSchema Option](/concepts/overwriteschema-option.md) - A Spark/Delta Lake DataFrameWriter option that allows overwriting the schema or changing partitioning of an existing Delta table.
* [overwriteSchema Option for Delta Tables](/concepts/overwriteschema-option.md) - A Spark DataFrameWriter option used to overwrite an existing Delta table's schema or change its partitioning.
* [overwriteSchema Option Pattern](/concepts/overwriteschema-option.md) - A Delta Lake DataFrameWriter option (.option('overwriteSchema', 'true')) used to force overwrite the table schema or change partitioning when writing, with restrictions when using replaceWhere.
* [Packaging Custom Libraries as Python Wheels for MLflow](/concepts/packaging-custom-libraries-as-python-wheels-for-mlflow.md) - Method of packaging custom or unavailable libraries as Python wheel files and including them when logging MLflow models via extra_pip_requirements or conda_env
* [Pandas Compatibility Issue in AutoML Model Serving](/concepts/pandas-dependency-compatibility-for-model-serving.md) - Known incompatibility between pandas version used by AutoML training and the model serving environment, resolved by editing the logged model's dependencies to include pandas==1.5.3.
* [Pandas Compatibility Issue with AutoML Model Serving](/concepts/pandas-dependency-compatibility-for-model-serving.md) - Common error and resolution for pandas version incompatibility when serving AutoML models with Databricks Model Serving
* [Pandas DataFrame Evaluation Pattern](/concepts/pandas-dataframe-evaluation-pattern.md) - Using Pandas DataFrames for GenAI evaluation when working with CSV files or existing data science workflows.
* [Pandas DataFrame Scoring Format](/concepts/pandas-dataframe-scoring-format.md) - A JSON-serialized format for scoring requests to Databricks Model Serving endpoints using Pandas DataFrame orientations, supporting both 'dataframe_split' (recommended) and 'dataframe_records' keys
* [Pandas Dependency Compatibility for AutoML Model Serving](/concepts/pandas-dependency-compatibility-for-model-serving.md) - Known pandas version incompatibility error when serving AutoML-built models, resolved by pinning pandas==1.5.3 via a helper script that edits the model's requirements.txt and conda.yaml.
* [Pandas Dependency Compatibility for Model Serving](/concepts/pandas-dependency-compatibility-for-model-serving.md) - Known issue where AutoML-built models can fail in Model Serving due to incompatible pandas versions, resolved by adding pandas==1.5.3 dependency via a provided script
* [pandas UDF for Distributed Inference](/concepts/pandas-udf-for-distributed-inference.md) - Using pandas User-Defined Functions (UDFs) in PySpark to parallelize deep learning model inference across a Spark cluster, treating inference as an embarrassingly parallel workload.
* [Pandas UDFs for Distributed Featurization](/concepts/pandas-udfs-for-distributed-featurization.md) - Using pandas UDFs (and Scalar Iterator pandas UDFs) on Databricks to apply pre-trained deep learning models across a cluster for scalable, distributed feature computation.
* [Pandas UDFs for Distributed Model Inference on Spark](/concepts/pandas-udfs-for-distributed-model-inference-on-spark.md) - Wrapping Hugging Face models in Pandas UDFs to distribute batch inference across worker nodes in a Spark cluster
* [Pandas Version Compatibility for AutoML Model Serving](/concepts/pandas-dependency-compatibility-for-model-serving.md) - Known issue where mismatched pandas versions between AutoML training and model serving endpoints cause 'No module named pandas.core.indexes.numeric' error, resolved by pinning pandas==1.5.3.
* [PandasDataset](/concepts/pandasdataset.md) - A GluonTS utility that converts a Pandas DataFrame into the internal dataset format required for model training and prediction.
* [Parallel Coordinates Chart in MLflow](/concepts/parallel-coordinates-chart-in-mlflow.md) - A visualization type in MLflow used to understand the effect of parameter settings on model performance and investigate relationships between parameters and metrics.
* [Parallel Coordinates Plot](/concepts/parallel-coordinates-plot.md) - A visualization type in MLflow for understanding the effect of parameter settings on model performance by investigating relationships between parameters and metrics.
* [Parallel Coordinates Plot for MLflow](/concepts/parallel-coordinates-plot.md) - A parallel coordinates visualization in MLflow that helps understand relationships between parameter settings and model performance metrics.
* [Parallel Requests Limits in Model Serving](/concepts/parallel-requests-limits-in-model-serving.md) - Workspace-enforced maximum on parallel inference requests, with mitigation via route optimized endpoints or client reduction and quota increase requests.
* [Parallelism vs adaptivity trade-off](/concepts/parallelism-vs-adaptivity-trade-off.md) - In Hyperopt, higher parallelism speeds up trials but reduces adaptivity since each iteration has fewer past results to inform new hyperparameter suggestions.
* [Parameter passing between Databricks notebooks](/concepts/parameter-passing-between-databricks-notebooks.md) - Techniques for passing string parameters and structured data (via temporary views, DBFS, or JSON) between notebooks using dbutils.notebook.run() and widgets.
* [Parameter Passing in Python Wheel Tasks](/concepts/parameter-passing-in-python-wheel-tasks.md) - How positional arguments (JSON array of strings) and keyword arguments (key-value pairs) are passed to a Python wheel task as command-line arguments.
* [Parameter-Efficient Fine-Tuning (PEFT)](/concepts/parameter-efficient-fine-tuning-peft.md) - A category of techniques that fine-tune LLMs by updating only a small subset of parameters, reducing computational cost
* [Parameter-Efficient Fine-Tuning (PEFT) for LLMs](/concepts/parameter-efficient-fine-tuning-peft.md) - Techniques for fine-tuning large language models with fewer computational resources by updating only a subset of model parameters, as demonstrated in Databricks AI Runtime notebooks.
* [Parameter-Efficient Fine-Tuning for LLMs](/concepts/parameter-efficient-fine-tuning-peft.md) - Techniques for fine-tuning large language models that update only a subset of parameters to reduce computational cost, as demonstrated in Databricks AI Runtime notebooks.
* [Parameterized Partition Sharing](/concepts/parameterized-partition-sharing.md) - A technique using the CURRENT_RECIPIENT() function to dynamically filter table partitions based on recipient properties, enabling a single share to serve multiple recipients with data boundaries maintained.
* [Parameterized Partition Sharing (Recipient Properties)](/concepts/parameterized-partition-sharing.md) - A technique for sharing table partitions that match data recipient properties (e.g., databricks.accountId, databricks.metastoreId, databricks.name, or custom properties). Enables a single share to dynamically deliver different data to different recipients while maintaining data boundaries, using the CURRENT_RECIPIENT() SQL function.
* [Parquet and Foreign Iceberg to Delta Clone](/concepts/parquet-and-foreign-iceberg-to-delta-clone.md) - The ability to incrementally clone Parquet and foreign Iceberg tables to Delta Lake format on Databricks, supporting both deep and shallow cloning.
* [Parquet Table Caching](/concepts/parquet-table-caching.md) - Caching support for Parquet tables in Databricks using the CACHE SELECT command.
* [Parquet to Delta Lake Conversion](/concepts/parquet-to-delta-lake-conversion.md) - Process of converting a directory of Parquet data files or a Parquet external table into a Delta Lake table using CONVERT TO DELTA.
* [Parquet to Delta Lake Migration](/concepts/parquet-to-delta-lake-migration.md) - The process and strategies for converting existing Apache Parquet data lakes into Delta Lake format on Databricks.
* [Parquet-based Delta Tables](/concepts/parquet-based-delta-tables.md) - Delta Lake tables stored in Parquet columnar format, which are required for features like persistent deletion vectors and serve as the standard storage format for Delta Lake.
* [Parquet-to-Delta Conversion](/concepts/parquet-to-delta-conversion.md) - The process of converting an existing Apache Parquet table to Delta Lake by listing files, creating a transaction log, and inferring schema by reading Parquet file footers.
* [Partial Column and Row Caching](/concepts/partial-column-and-row-caching.md) - The ability to cache only a subset of columns and rows from a table using column lists and WHERE predicates in the CACHE SELECT statement.
* [Partial reveal masking without regex](/concepts/partial-reveal-masking-without-regex.md) - Revealing parts of sensitive values using string operations like RIGHT and CONCAT instead of expensive regex-based masking, improving performance on large text fields.
* [Partial reveal without regex](/concepts/partial-reveal-without-regex.md) - A performance-optimized pattern for revealing part of a sensitive value (e.g. last digits of an SSN) using simple string functions like RIGHT and CONCAT instead of expensive regex-based masking on large text fields.
* [Partition Discovery in Delta Conversion](/concepts/partition-discovery-in-delta-conversion.md) - How CONVERT TO DELTA handles partitioning information, requiring PARTITIONED BY for path-based conversions and optionally loading from the metastore for registered tables.
* [Partition Evolution in Iceberg](/concepts/partition-evolution-in-iceberg.md) - A feature of Apache Iceberg tables where the partitioning scheme can change over time; tables that have undergone partition evolution are incompatible as sources for Delta clone operations.
* [Partition Hint and Predicate Rewrite Conflicts](/concepts/partition-hint-and-predicate-rewrite-conflicts.md) - Conflict sub-errors related to partition hints and filter predicates that could not be applied for row-level conflict detection in Delta Lake.
* [Partition Hints for Delta Lake Conflict Resolution](/concepts/partition-hints-for-delta-lake-conflict-resolution.md) - A technique using partition hints to narrow the scope of read operations in Delta Lake, reducing the chance of concurrent delete conflicts to specific partitions.
* [Partition Inference in CONVERT TO DELTA](/concepts/partition-inference-in-convert-to-delta.md) - CONVERT TO DELTA automatically infers partitioning information for Hive metastore tables but requires explicit partitioning for Unity Catalog external tables.
* [Partition Inference in Delta Conversion](/concepts/partition-inference-in-delta-conversion.md) - Automatic partitioning inference for Hive metastore tables during CONVERT TO DELTA, and the requirement to provide partitioning info for Unity Catalog external tables.
* [Partition Sharing and Parameterized Partition Filtering](/concepts/partition-sharing-and-parameterized-partition-filtering.md) - Sharing only a subset of a table's data via partition specifications, including dynamic filtering using recipient properties (CURRENT_RECIPIENT) to deliver different data to different recipients from a single share.
* [Partition Specification in CONVERT TO DELTA](/concepts/partition-specification-in-convert-to-delta.md) - The requirement and behavior of the PARTITIONED BY clause when converting partitioned Parquet tables to Delta Lake, including validation against actual directory structure.
* [Partition Value Validation in Delta Tables](/concepts/partition-value-validation-in-delta-tables.md) - Detection of data files with partition column values that violate table constraints, such as NULL values in NOT NULL partition columns, reported by FSCK REPAIR TABLE.
* [Partition-Aware Conflict Detection in Delta Lake](/concepts/partition-aware-conflict-detection-in-delta-lake.md) - Strategies for detecting and resolving concurrent delete conflicts on partitioned Delta tables, including partition hints and source table requirements for MERGE operations.
* [Partition-Aware Transaction Conflicts in Delta Lake](/concepts/partition-based-transaction-conflicts-in-delta-lake.md) - Conflict scenarios in Delta Lake specifically related to partitioned tables, including inability to detect conflicts on partitioned tables and conflicts limited to specific partition values.
* [Partition-based Conflict Detection in Delta Lake](/concepts/partition-based-conflict-detection-in-delta-lake.md) - Using partition hints and partition pruning to narrow the scope of transaction conflict detection, with specific error subclasses for partitioned tables.
* [Partition-based Transaction Conflicts in Delta Lake](/concepts/partition-based-transaction-conflicts-in-delta-lake.md) - Conflict detection scenarios specific to partitioned Delta tables, including cases where partition hints narrow the conflict scope or where partitioned tables lack merge source information for row-level detection.
* [Partition-Hinted vs Unhinted Conflict Resolution](/concepts/partition-hinted-vs-unhinted-conflict-resolution.md) - Two related sub-errors (WITH_PARTITION_HINT and WITHOUT_HINT) describing partition-level conflicts in Delta Lake concurrent operations, where the presence or absence of partition hints affects the error message.
* [Partition-Level Conflict Detection](/concepts/partition-level-conflict-detection.md) - Conflict detection that operates at the partition level, where concurrent modifications to specific partitions invalidate reads or writes targeting those partitions
* [Partition-Level Conflict Hints in Delta Lake](/concepts/partition-level-conflict-hints-in-delta-lake.md) - A mechanism where partition hints can narrow the scope of concurrent delete/read conflicts to specific partitions, but can still fail when those partitions are affected by concurrent deletions.
* [Partitioned Conversion and Metastore Metadata](/concepts/partitioned-conversion-and-metastore-metadata.md) - When converting partitioned data, PARTITIONED BY is required for path-based tables; mismatches between catalog metadata and the Delta transaction log throw a convertMetastoreMetadataMismatchException.
* [Partitioned Table Conflict Detection Limitations](/concepts/partitioned-table-conflict-detection-limitations.md) - A scenario in Delta Lake where row-level conflict detection cannot be performed on partitioned tables under certain conditions, requiring a retry of the operation.
* [Partitioned table merge conflicts in Delta Lake](/concepts/partitioned-table-merge-conflicts-in-delta-lake.md) - A specific conflict sub-error (PARTITIONED_TABLE_WITHOUT_MERGE_SOURCE) that occurs when row-level conflict detection cannot be performed on a partitioned table during a MERGE operation.
* [Partitioned Table Row-Level Conflict Detection](/concepts/partitioned-table-row-level-conflict-detection.md) - A limitation where row-level conflict detection cannot be performed on certain partitioned Delta tables, requiring a retry.
* [PARTITIONING_MISMATCH sub-error](/concepts/partitioning_mismatch-sub-error.md) - A DELTA_METADATA_MISMATCH sub-error that occurs when provided partition columns do not match the table's partition columns
* [Partner Model Retirement Policy](/concepts/partner-model-retirement-policy.md) - Special retirement policy for third-party models from OpenAI, Anthropic, and Google available through Databricks Foundation Model APIs, where Databricks may redirect models to bridge partner-led deprecation gaps.
* [Partner Model Retirement Redirection](/concepts/partner-model-retirement-redirection.md) - A Databricks policy that temporarily redirects deprecated third-party models (OpenAI, Anthropic, Google) to similar versions when partners provide shorter notice than Databricks' standard transition period.
* [Partner-powered AI Features on Databricks](/concepts/partner-powered-ai-features-on-databricks.md) - A prerequisite for using Genie Code, requiring partner-powered AI features to be enabled at both account and workspace level, with supported region and geo availability requirements.
* [Partner-powered AI Features Prerequisite](/concepts/partner-powered-ai-features-prerequisite.md) - Requirement that both the account and workspace must have partner-powered AI features enabled for Genie Code to function.
* [Passing structured data between Databricks notebooks](/concepts/passing-structured-data-between-databricks-notebooks.md) - Techniques for sharing non-string data between notebooks using temporary views, DBFS file storage, and JSON serialization.
* [PATH Configuration for Databricks Connect Binaries](/concepts/path-configuration-for-databricks-connect-binaries.md) - Databricks Connect requires its binaries to take precedence in the system PATH; missing or conflicting PATH entries can prevent commands like spark-shell from running correctly.
* [PATH_BASED target error](/concepts/path_based-target-error.md) - A subtype of DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_TARGET indicating the target table is a path-based table (not supported for clone with history).
* [PATH_BASED target table restriction](/concepts/path_based-target-table-restriction.md) - Delta clone with history does not support path-based (location-only) target tables.
* [Path-based target restriction for clone with history](/concepts/non-delta-target-restriction-for-clone-with-history.md) - Cloning a Delta table with history to a path-based target table is not supported in Databricks.
* [Path-based target tables in Delta Clone with History](/concepts/path-based-target-tables-in-delta-clone-with-history.md) - An error sub-condition (PATH_BASED) where the target table for a Delta Clone with History operation is path-based rather than metastore-registered
* [Pay-per-token Compliance](/concepts/pay-per-token-compliance.md) - HIPAA compliance only for pay-per-token Foundation Model API workloads; other standards (PCI-DSS, FedRAMP, IRAP, CCCS, UK Cyber Essentials Plus) not supported.
* [Pay-per-token deployment mode](/concepts/pay-per-token-deployment-mode.md) - A consumption-based pricing model for Foundation Model APIs that supports HIPAA compliance but not other standards like PCI-DSS, FedRAMP, IRAP, CCCS, or UK Cyber Essentials Plus.
* [Pay-per-token endpoint rate limits](/concepts/pay-per-token-endpoint-rate-limits.md) - Foundation Model APIs enforce token-based (ITPM, OTPM) and query-based (QPH) rate limits on pay-per-token endpoints, with the most restrictive limit applying at any time.
* [Pay-per-Token Endpoints](/concepts/pay-per-token-endpoints.md) - A consumption-based pricing model for foundation model inference where users are charged per token processed, suitable for exploration and variable workloads.
* [Pay-per-token Foundation Model APIs](/concepts/pay-per-token-foundation-model-apis.md) - Preconfigured endpoints for querying foundation models on a per-token pricing basis, recommended for getting started and low-throughput production workloads.
* [Pay-Per-Token Foundation Models on Databricks](/concepts/pay-per-token-foundation-models-on-databricks.md) - A pricing model for foundation models on Databricks where users are charged per token consumed, supported by the Anthropic Messages API for hosted Claude models.
* [Pay-per-Token Pricing](/concepts/pay-per-token-pricing.md) - Usage-based pricing model for Databricks Foundation Model APIs where users pay per token consumed for inference on curated model architectures.
* [Pay-per-Token Pricing (Databricks)](/concepts/pay-per-token-pricing-databricks.md) - A consumption-based pricing model for base foundation models available for immediate use without performance guarantees.
* [Pay-per-token Pricing Mode](/concepts/pay-per-token-pricing-mode.md) - A consumption-based pricing model for Foundation Model APIs where users are charged per token processed, suitable for experimentation and variable workloads.
* [Pay-per-token serving mode](/concepts/pay-per-token-serving-mode.md) - Preconfigured endpoints for accessing foundation models on a per-token billing basis, recommended for getting started and prototyping but not designed for high-throughput applications.
* [Pay-per-token vs Provisioned Throughput Deployment Modes](/concepts/pay-per-token-vs-provisioned-throughput-deployment-modes.md) - Two distinct deployment modes for Foundation Model APIs with different compliance support: pay-per-token supports only HIPAA, while provisioned throughput supports the full range of compliance standards.
* [Pay-per-Token vs Provisioned Throughput Endpoints](/concepts/pay-per-token-vs-provisioned-throughput-endpoints.md) - Two endpoint types in Databricks Foundation Model APIs: preconfigured pay-per-token endpoints and user-created provisioned throughput endpoints supporting A/B testing.
* [Pay-per-Token vs Provisioned Throughput Modes](/concepts/pay-per-token-vs-provisioned-throughput-modes.md) - Two pricing and deployment modes for Foundation Model APIs: pay-per-token for variable usage and provisioned throughput for production workloads, with the latter supporting all models in a model architecture family.
* [Pay-per-token vs Provisioned Throughput Serving Modes](/concepts/pay-per-token-vs-provisioned-throughput-modes.md) - Two distinct serving models in Databricks Foundation Model APIs: pay-per-token for consumption-based billing and provisioned throughput for production workloads with guaranteed capacity across model architecture families.
* [Per-Cluster Hive Metastore Disablement](/concepts/per-cluster-hive-metastore-disablement.md) - A gradual approach to disabling Hive metastore access on a cluster-by-cluster basis using Spark configurations, useful during incremental Unity Catalog migration.
* [Per-Table Access Control Configuration](/concepts/per-table-access-control-configuration.md) - Users with CAN MANAGE permission can configure permissions on an individual feature table via the table's drop-down menu in the Databricks Feature Store UI.
* [Percent Null](/concepts/percent-null.md) - A supplementary data quality metric that tracks the expected percentage of null values per column over the last 24 hours, flagging incompleteness when observed nulls exceed predicted bounds.
* [Percent Null (Completeness)](/concepts/percent-null.md) - An additional completeness sub-metric tracking the expected percentage of null values per column over 24 hours to detect anomalous null increases
* [Percent Null Completeness Check](/concepts/percent-null.md) - An additional quality detail of completeness that monitors the expected percentage of rows with null values for a given column over the last 24 hours, flagging a table as incomplete if the observed null percentage exceeds the predicted upper bound.
* [Performance and Cost Monitoring via Traces](/concepts/performance-and-cost-monitoring-via-traces.md) - Using MLflow Tracing to capture operational metrics like latency, cost, and resource utilization at each step of GenAI application execution to identify bottlenecks and optimize efficiency.
* [Performance and Cost Optimization via Traces](/concepts/performance-and-cost-monitoring-via-traces.md) - Using MLflow Tracing to capture operational metrics like latency, cost, and resource utilization at each execution step to identify bottlenecks and optimize cost efficiency in GenAI applications.
* [Performance and Cost Optimization via Tracing](/concepts/performance-and-cost-monitoring-via-traces.md) - Using MLflow Tracing to capture and monitor operational metrics like latency, cost, and resource utilization at each execution step to identify bottlenecks and optimize efficiency.
* [Performance monitoring with MLflow traces](/concepts/performance-monitoring-with-mlflow-traces.md) - Pattern for profiling GenAI trace performance by calculating latency percentiles (P50, P95, P99) and identifying outlier traces using execution_time_ms.
* [Performance optimization for row filters and column masks](/concepts/performance-optimization-for-row-filters-and-column-masks.md) - Best practices to minimize query performance impact when using row filters and column masks, including UDF simplicity, deterministic expressions, and avoiding Python UDFs.
* [Performance Profiling with Trace Percentiles](/concepts/performance-profiling-with-trace-percentiles.md) - Technique for analyzing GenAI trace performance by calculating latency percentiles (P50, P95, P99) and identifying outlier traces.
* [Performance Tuning for GPU-based Inference on Spark](/concepts/performance-tuning-for-gpu-based-inference-on-spark.md) - Strategies for optimizing Hugging Face inference performance on Spark clusters including batch size tuning, DataFrame repartitioning, and GPU utilization monitoring
* [Permission Inheritance and Override](/concepts/permission-inheritance-and-override.md) - Feature store-wide permissions are inherited by individual feature tables and can only be removed at the store level; per-table settings can add more permissive rules but cannot override inherited restrictions.
* [Permission Inheritance and Override in Feature Store](/concepts/permission-inheritance-in-feature-store.md) - Permissions set at the Feature Store level apply to all tables and can only be overridden to add more permissive access, not remove; inherited permissions are marked as such
* [Permission Inheritance in Feature Store](/concepts/permission-inheritance-in-feature-store.md) - Permissions set at the Feature Store level are inherited by individual feature tables; they can be overridden to add permissions but not to remove inherited permissions.
* [Permission Validation at Endpoint Creation](/concepts/permission-validation-at-endpoint-creation.md) - The endpoint creator's permission to the specified instance profile is validated at endpoint creation time, not at update time.
* [Permissions for Custom Classifiers](/concepts/permissions-for-custom-classifiers.md) - Role and privilege requirements for managing custom classifiers: metastore admin role, ASSIGN on the governed tag, and SELECT on example table columns.
* [Permissions for Governed Tags](/concepts/permissions-for-governed-tags.md) - The ASSIGN permission on system.certification_status governed tag required to apply certification status to objects, along with APPLY TAG, USE SCHEMA, and USE CATALOG privileges on the target object.
* [Persistent Deletion Vectors](/concepts/persistent-deletion-vectors.md) - A mode of deletion vectors in Delta Lake where the deletion metadata persists in the table metadata, supported only on Parquet-based Delta tables and incompatible with incremental symlink manifest generation.
* [Persistent Deletion Vectors Constraint](/concepts/persistent-deletion-vectors-constraint.md) - Persistent deletion vectors are only supported on Parquet-based Delta tables and cannot coexist with incremental symlink manifest generation.
* [Persisting Test Cases as MLflow Evaluation Datasets](/concepts/persisting-test-cases-as-mlflow-evaluation-datasets.md) - Storing conversation test cases in reusable MLflow Evaluation Datasets via mlflow.genai.datasets for reproducible testing and versioning across simulation runs.
* [Petastorm](/concepts/petastorm.md) - An open source data access library for converting Apache Spark DataFrames to TensorFlow Datasets or PyTorch DataLoaders, enabling single-node and distributed deep learning training from Parquet data.
* [Petastorm Cache Directory Configuration](/concepts/petastorm-cache-directory-configuration.md) - The mechanism to specify where Petastorm caches converted Spark DataFrames in Parquet format, configurable via Spark config or notebook, requiring a DBFS path starting with file:///dbfs/.
* [Petastorm Deprecation and Mosaic Streaming](/concepts/petastorm-deprecation-and-mosaic-streaming.md) - The petastorm package is deprecated on Databricks, with Mosaic Streaming recommended as the replacement for loading large datasets from cloud storage.
* [Photon Compatibility with Databricks Runtime ML](/concepts/photon-compatibility-with-databricks-runtime-ml.md) - Photon can be enabled on Databricks Runtime 15.2 ML+ to improve Spark SQL, DataFrames, feature engineering, and xgboost4j performance, but does not benefit Python-based ML packages like PyTorch and TensorFlow.
* [Photon Compatibility with ML Workloads](/concepts/photon-compatibility-with-ml-workloads.md) - Photon acceleration improves performance for Spark SQL, DataFrames, feature engineering, and xgboost4j but not for Spark RDDs, Pandas UDFs, Python packages like XGBoost/PyTorch/TensorFlow, and may cause memory issues with Spark MLlib.
* [Photon on Databricks Runtime ML](/concepts/photon-on-databricks-runtime-ml.md) - A performance optimization engine for Spark SQL, DataFrames, and feature engineering workloads, available on Databricks Runtime 15.2 ML and above
* [Pickling Errors in Distributed PyTorch](/concepts/pickling-errors-in-distributed-pytorch.md) - A common distributed training issue where library imports are not distributed to executors, solved by placing import statements inside the training function.
* [PII Redaction Pipeline for OpenTelemetry Traces](/concepts/pii-redaction-pipeline-for-opentelemetry-traces.md) - An incremental pipeline pattern that uses AI Functions and Lakeflow Spark Declarative Pipelines to redact personally identifiable information from OpenTelemetry trace data stored in Unity Catalog, writing sanitized results to separate tables with broader access controls.
* [Pipeline Parallelism in DeepSpeed](/concepts/pipeline-parallelism-in-deepspeed.md) - An advanced parallelism technique in DeepSpeed that partitions model layers across multiple devices to enable training of models that exceed single-device memory capacity.
* [Pipeline Update Retry Behavior in Orchestrated Workflows](/concepts/pipeline-update-retry-behavior-in-orchestrated-workflows.md) - How retry configurations on both the pipeline level and the orchestrator level interact, potentially causing cascading retries.
* [Placeholder Model Version Technique](/concepts/placeholder-model-version-technique.md) - A technique using a dummy PythonModel to fill gaps in version numbers, ensuring version parity between workspace and Unity Catalog model registries
* [Placeholder Scorer Pattern](/concepts/placeholder-scorer-pattern.md) - Using a minimal temporary scorer (e.g., returning a constant value) as a required argument to mlflow.genai.evaluate() to bootstrap trace generation before developing real metrics.
* [Placeholder Scorers](/concepts/placeholder-scorers.md) - Minimal scorer functions used solely to satisfy mlflow.genai.evaluate() requirements during initial trace generation, later replaced with real evaluation logic.
* [Placeholder Scorers for Trace Generation](/concepts/placeholder-scorers-for-trace-generation.md) - The pattern of defining a minimal placeholder scorer (e.g., a no-op function returning a constant) when initially generating traces via mlflow.genai.evaluate(), since evaluate() requires at least one scorer.
* [Plain Text Environment Variables in Databricks Model Serving](/concepts/plain-text-environment-variables-in-databricks-model-serving.md) - Setting non-sensitive environment variables directly in model serving endpoint configurations via the UI, REST API, or SDK.
* [Plain Text Environment Variables in Model Serving](/concepts/plain-text-environment-variables-in-model-serving.md) - Setting non-sensitive configuration variables directly on a model serving endpoint via UI, REST API, or SDK without encryption
* [Poetry Tilde Notation for Databricks Connect](/concepts/poetry-tilde-notation-for-databricks-connect.md) - When using Poetry, the '~X.Y' (tilde) notation is recommended to pin the major.minor version while allowing the latest patch of databricks-connect.
* [Point-in-Time Correct Training Sets](/concepts/point-in-time-correct-training-sets.md) - The mechanism by which create_training_set computes point-in-time aggregated features, ensuring temporal consistency between labels and features
* [Point-in-time correctness](/concepts/point-in-time-correctness.md) - A join strategy that ensures training datasets reflect only feature values that existed as of the timestamp of each label observation, preventing data leakage.
* [Point-in-Time Feature Joins](/concepts/point-in-time-feature-joins.md) - The ability to join feature values based on temporal alignment, ensuring historical consistency for training datasets.
* [Point-in-time Joins](/concepts/point-in-time-joins.md) - A feature of Databricks Feature Store that enables time-accurate feature lookups, ensuring features are joined to training or inference data at the correct historical timestamp.
* [Point-in-Time Lookups](/concepts/point-in-time-lookups.md) - A time-aware join technique for time series feature tables that ensures training data uses only feature values known up to the timestamp of each target observation.
* [Point-in-Time Training Sets](/concepts/point-in-time-training-sets.md) - Using create_training_set to compute point-in-time aggregated features for ML model training by joining labeled data with feature definitions across entity keys and time.
* [Policies (ABAC)](/concepts/policies-abac.md) - Core mechanism in Unity Catalog ABAC that uses governed tag attributes in conditions to determine access; attached at catalog, schema, or table level and evaluated dynamically.
* [Policies in Unity Catalog ABAC](/concepts/policies-in-unity-catalog-abac.md) - Reusable access rule definitions attached to catalog, schema, or table levels that dynamically evaluate governed tags to determine access
* [Policy Function Inputs](/concepts/policy-function-inputs.md) - Parameters provided to row filter or column mask UDFs, which can be column values matched by tags, columns matched by custom expressions, or constant values.
* [Policy Lifecycle Management (CREATE/ALTER/DROP/SHOW/DESCRIBE)](/concepts/policy-lifecycle-management-createalterdropshowdescribe.md) - SQL statements and UI workflows for creating, editing, deleting, listing, and describing ABAC row filter and column mask policies on securable objects.
* [Policy Quotas and Audit Logging](/concepts/policy-quotas-and-audit-logging.md) - GRANT policies have separate quotas from row filter/column mask policies, and CRUD operations are logged under standard policy audit actions (createPolicy, deletePolicy, getPolicy, listPolicies).
* [Policy Retention on Table Replacement](/concepts/policy-retention-on-table-replacement.md) - The behavior where row filters and column masks are automatically retained when running REPLACE TABLE, preventing accidental loss of data access policies unless explicitly dropped or updated.
* [Policy Scope and Attachment Levels](/concepts/policy-scope-and-attachment-levels.md) - GRANT policies in Unity Catalog can be attached at the catalog or schema level (not at the individual model level) and apply to all securable objects within that scope that match the condition.
* [Policy Scope and Inheritance](/concepts/policy-scope-and-inheritance.md) - ABAC policies can be attached at the catalog, schema, or table level, and effective policies include inherited policies from parent scopes (e.g., catalog-level policies affecting a table).
* [Policy Scoping at Catalog or Schema Level](/concepts/policy-scoping-at-catalog-or-schema-level.md) - Attaching ABAC policies at the highest applicable scope (catalog or schema) rather than at the table level
* [Policy-based column masking with UDFs](/concepts/policy-based-column-masking-with-udfs.md) - Using User-Defined Functions (UDFs) as the masking logic in Unity Catalog column mask policies, enabling reusable masking strategies such as partial masking and full redaction.
* [Population Stability Index (PSI)](/concepts/population-stability-index-psi.md) - A numeric metric in [0, inf) measuring how different two distributions are: PSI < 0.1 indicates no significant change, < 0.2 moderate change, and >= 0.2 significant population change.
* [Population Stability Index (PSI) for Data Drift](/concepts/population-stability-index-psi.md) - PSI is a numeric metric in [0, ∞) comparing two distributions; <0.1 no significant change, <0.2 moderate change, >=0.2 significant change. Used in Databricks drift metrics tables.
* [Portability of Databricks Connect APIs](/concepts/portability-of-databricks-connect-apis.md) - All Databricks Connect APIs are available in Databricks notebooks as part of the corresponding Databricks Runtime, enabling seamless code transition from local development to deployment without code changes.
* [Portable Spark Session Pattern](/concepts/portable-spark-session-pattern.md) - A try/except pattern for writing code that works in environments both with and without the DatabricksSession class, falling back to standard SparkSession.
* [Portable SparkSession Pattern](/concepts/portable-sparksession-pattern.md) - A coding pattern that gracefully falls back from DatabricksSession to a regular SparkSession when Databricks Connect is not available, enabling portable PySpark code.
* [Portable SparkSession pattern for Databricks Connect](/concepts/portable-sparksession-pattern.md) - A code pattern that falls back from DatabricksSession to SparkSession when Databricks Connect is unavailable, enabling portable code
* [Post-completion Tag Management (mlflow.set_trace_tag / mlflow.delete_trace_tag)](/concepts/post-completion-tag-management-mlflowset_trace_tag-mlflowdelete_trace_tag.md) - MLflow API methods for setting or deleting tags on traces that have already been completed and logged in the backend store.
* [Post-hoc Tag Management](/concepts/post-hoc-tag-management.md) - Setting, updating, and deleting tags on already-completed and logged traces using mlflow.set_trace_tag and mlflow.delete_trace_tag.
* [Post-hoc trace tag management](/concepts/post-hoc-tag-management.md) - The ability to set or delete tags on an MLflow trace that has already been completed and logged, using mlflow.set_trace_tag and mlflow.delete_trace_tag functions or the MLflow UI.
* [Pre-admission checking and token credit-back](/concepts/pre-admission-checking-and-token-credit-back.md) - Databricks pre-admits requests by verifying input tokens against ITPM and reserving max_tokens against OTPM; unused reserved tokens are credited back after the response.
* [Pre-deployment Validation for Model Serving](/concepts/pre-deployment-validation-for-model-serving.md) - Practices for validating model serving configurations before deployment to catch common issues early, preventing build and runtime failures.
* [Pre-generated Conversation Evaluation](/concepts/pre-generated-conversation-evaluation.md) - Approach for evaluating existing conversation traces (from production or pre-recorded test data) by retrieving traces grouped by session ID and passing them to mlflow.genai.evaluate.
* [Pre-Installed Library Removal Criteria](/concepts/pre-installed-library-removal-criteria.md) - Conditions under which a pre-installed library is removed from Databricks Runtime ML, including lack of active maintenance (no commits in 3 months, no releases in 9 months, archived repository, or announced maintenance stop) or no functional stable release.
* [Pre-signed URL Sharing](/concepts/pre-signed-url-sharing.md) - A Delta Sharing mechanism that provides temporary read access via pre-signed URLs, logged as deltaSharingQueriedTable and deltaSharingQueriedTableChanges events
* [Pre-signed URL Sharing Audit Details](/concepts/pre-signed-url-sharing-audit-events.md) - Audit log fields for pre-signed URL-based sharing, including deltaSharingQueriedTableChanges and deltaSharingQueriedTable events with detailed metadata about shared files.
* [Pre-signed URL Sharing Audit Events](/concepts/pre-signed-url-sharing-audit-events.md) - Audit events (deltaSharingQueriedTableChanges, deltaSharingQueriedTable) logged when data recipients query shared data via pre-signed URL-based sharing, with detailed response.result fields.
* [Pre-signed URL Sharing in OpenSharing](/concepts/pre-signed-url-sharing.md) - A credential mechanism for OpenSharing that provides temporary read access to shared data via pre-signed URLs, logged through deltaSharingQueriedTable and deltaSharingQueriedTableChanges events.
* [Pre-signed URL vs STS Token Auditing](/concepts/pre-signed-url-vs-sts-token-auditing.md) - OpenSharing supports two credential types for temporary data access — pre-signed URLs and scoped-down STS tokens — each producing distinct audit log fields in provider logs.
* [Pre-trained Model Truncation for Feature Extraction](/concepts/truncated-model-feature-extraction.md) - The technique of removing the final classification layers from a pre-trained deep learning model so that the model outputs feature tensors instead of predictions, which can then be used as inputs to new models.
* [Predicate-based row deletion](/concepts/predicate-based-row-deletion.md) - Core mechanism of DELETE FROM that filters which rows to remove using a WHERE predicate; deleting all rows when no predicate is provided.
* [Predict Function Contract for Optimization](/concepts/predict-function-contract-for-optimization.md) - The requirement that predict_fn must use mlflow.entities.model_registry.PromptVersion.format() to load prompts from the registry, rather than hardcoding prompt strings, for optimization to work correctly.
* [Predict Function Wrapper for Logged Models](/concepts/predict-function-wrapper-for-logged-models.md) - MLflow 3 requires wrapping logged MLflow models in a predict function that maps named input parameters to the model's expected input format, serving as a translation layer between the evaluation framework and the model.
* [predict_fn Patterns in MLflow Evaluation](/concepts/predict_fn-patterns-in-mlflow-evaluation.md) - Common patterns for defining or wrapping prediction functions used with mlflow.genai.evaluate(), including direct calls, wrappers, endpoint evaluation, and logged model evaluation.
* [Predictive Equality](/concepts/predictive-equality.md) - A fairness metric that compares false positive rates between protected and unprotected groups
* [Predictive I/O](/concepts/predictive-io.md) - A modern replacement for Bloom filter indexes in Databricks that optimizes data skipping during queries.
* [Predictive I/O (Databricks)](/concepts/predictive-io.md) - A Databricks optimization feature recommended as a replacement for deprecated Bloom filter indexes.
* [Predictive Optimization for Delta Lake](/concepts/predictive-optimization-for-delta-lake.md) - Automatic optimization feature on Databricks that runs OPTIMIZE and VACUUM commands on Unity Catalog managed tables without manual intervention.
* [Predictive Optimization for VACUUM](/concepts/predictive-optimization-for-vacuum.md) - When predictive optimization is enabled, Databricks automatically triggers VACUUM operations, making manual VACUUM commands unnecessary in most cases.
* [Predictive Parity](/concepts/predictive-parity.md) - A fairness metric that compares the model's precision (positive predictive value) between protected and unprotected groups
* [Prefill-Decode Architecture](/concepts/prefill-decode-architecture.md) - The two-phase LLM inference process where input tokens are processed in parallel (prefill) followed by auto-regressive token-by-token generation (decode).
* [Prefix-Based Secret Naming for Online Stores](/concepts/prefix-based-secret-naming-for-online-stores.md) - A naming convention using a unique prefix combined with -access-key-id/-secret-access-key (DynamoDB) or -user/-password (SQL stores) to organize credentials in Databricks secret scopes.
* [prepare_model and prepare_data_loader](/concepts/prepare_model-and-prepare_data_loader.md) - Ray Train utility functions that automatically wrap a PyTorch model in DDP (Distributed Data Parallel), move it to the correct device, and inject a DistributedSampler into DataLoaders for sharded training.
* [prepare_model and prepare_data_loader in Ray Train](/concepts/prepare_model-and-prepare_data_loader.md) - Ray Train utility functions that handle DDP wrapping, GPU device placement, and distributed dataset sampling for PyTorch training workflows.
* [Preprocessing and Postprocessing with Custom PyFunc](/concepts/preprocessing-and-postprocessing-with-pyfunc.md) - Using custom pyfunc models to add preprocessing (before model inference) and postprocessing (after model inference) logic to model serving pipelines
* [Preprocessing and Postprocessing with PyFunc](/concepts/preprocessing-and-postprocessing-with-pyfunc.md) - Using custom pyfunc models to add preprocessing (input formatting) and postprocessing (output formatting) logic around model inference.
* [Prerequisite Permission Validation](/concepts/prerequisite-permission-validation.md) - Automatic checking of prerequisite privileges (e.g., USE CATALOG, USE SCHEMA) when users request a privilege like SELECT, with missing prerequisites generating additional requests.
* [Prerequisites and Limitations for UniForm Iceberg Reads](/concepts/prerequisites-and-limitations-for-uniform-iceberg-reads.md) - Enabling Iceberg reads requires Unity Catalog registration, column mapping, specific Delta protocol versions (minReaderVersion >= 2, minWriterVersion >= 7), and Databricks Runtime 14.3+ — with limitations including no deletion vectors, read-only Iceberg client access, and no support for materialized views or streaming tables.
* [Prerequisites for Disabling Hive Metastore Access](/concepts/prerequisites-for-disabling-legacy-hive-metastore.md) - Conditions that must be met before disabling legacy Hive metastore, including completed migration, Databricks Runtime 13.3 LTS+, and user readiness.
* [Prerequisites for Disabling Legacy Hive Metastore](/concepts/prerequisites-for-disabling-legacy-hive-metastore.md) - Criteria that must be met before disabling the legacy Hive metastore, including completed Unity Catalog migration and Databricks Runtime 13.3 LTS or above
* [Prerequisites for MLflow 3 GenAI Tracing](/concepts/prerequisites-for-mlflow-3-genai-tracing.md) - Installation requirements including `mlflow[databricks]>=3.1`, provider SDKs, and LLM API key configuration for automatic tracing on Databricks.
* [Prerequisites for MLflow Bedrock Tracing](/concepts/prerequisites-for-mlflow-bedrock-tracing.md) - Using MLflow tracing with Amazon Bedrock requires installing mlflow[databricks]>=3.1 and boto3, configuring Databricks credentials (DATABRICKS_HOST, DATABRICKS_TOKEN), and configuring AWS credentials for Bedrock access.
* [Prerequisites for MLflow CrewAI Tracing](/concepts/prerequisites-for-mlflow-crewai-tracing.md) - Environment setup requirements including MLflow 3+, crewai library, API keys, and Databricks credentials
* [Prerequisites for Publishing Feature Tables to Online Stores](/concepts/prerequisites-for-publishing-feature-tables-to-online-stores.md) - Feature tables must satisfy primary key constraints, non-nullable primary key columns, and Change Data Feed (CDF) enablement before they can be published to an online store for real-time serving.
* [Prerequisites for Row Filters and Column Masks](/concepts/prerequisites-for-row-filters-and-column-masks.md) - The required privileges (EXECUTE, USE SCHEMA, USE CATALOG), compute resource constraints (SQL warehouse, shared/dedicated access mode versions), and UDF registration requirements for applying row filters and column masks.
* [Prerequisites for Unity Catalog Trace Migration](/concepts/prerequisites-for-unity-catalog-trace-migration.md) - The required environment setup including a Unity Catalog-enabled workspace, Databricks Runtime 15.3+, the databricks-agents package (>=1.10.1), and specific read/write permissions on source and destination.
* [Prerequisites for upgrading feature tables to Unity Catalog](/concepts/prerequisites-for-upgrading-feature-tables-to-unity-catalog.md) - The underlying workspace Delta table must be upgraded to Unity Catalog before calling upgrade_workspace_table, and users must notify producers/consumers of the new table name.
* [Presto and Athena Delta Lake integration](/concepts/presto-and-athena-delta-lake-integration.md) - Method for reading Delta Lake tables from Presto and Amazon Athena using generated manifest files as a bridge.
* [Prevent access until sensitive columns are tagged](/concepts/prevent-access-until-sensitive-columns-are-tagged.md) - A governance pattern that uses a default 'unverified' tag to block access to unclassified data, then automatically switches to column masking once a data steward updates the classification tag.
* [Primary Key Constraint for Feature Tables](/concepts/primary-key-constraint-for-feature-tables.md) - Feature tables require a primary key constraint, which can be defined at table creation or added to existing Delta tables via ALTER TABLE DDL statements.
* [Primary Key Constraints as Feature Tables](/concepts/primary-key-constraints-as-feature-tables.md) - Any Delta table in Unity Catalog with a primary key constraint is automatically treated as a feature table and appears in the Features UI.
* [Primary Key Constraints for Feature Tables](/concepts/primary-key-constraints-for-feature-tables.md) - Feature tables require a primary key constraint defined via NOT NULL columns and ALTER TABLE ADD CONSTRAINT DDL; only the table owner can declare it.
* [Primary Key Requirement for Shared Data](/concepts/primary-key-requirement-for-shared-data.md) - Requirement that any data asset shared with SAP BDC must have a primary key or identity column, which can be added before sharing or defined in the CSN schema.
* [Primary Key Requirements for Shared Data Assets](/concepts/primary-key-requirement-for-shared-data.md) - Any data asset shared via SAP BDC must have a primary key or identity column defined before sharing
* [Principal targeting in ABAC policies: TO/EXCEPT vs identity functions](/concepts/principal-targeting-in-abac-policies-toexcept-vs-identity-functions.md) - Two approaches for specifying which principals a policy applies to: using TO/EXCEPT clauses (simpler, eliminates UDF execution for exempt users) or identity functions like current_user() and is_account_group_member() inside the UDF (flexible, resolved once per query, not per row).
* [Principal Targeting with TO and EXCEPT Clauses](/concepts/principal-targeting-with-to-and-except-clauses.md) - Using policy TO and EXCEPT clauses to define which users/groups a row filter or column mask applies to, keeping UDF logic simple and avoiding complex conditional logic inside functions.
* [Private PyPI Mirror Configuration for Model Serving](/concepts/private-pypi-mirror-configuration-for-model-serving.md) - Workspace-level configuration of private package repositories (Nexus, Artifactory) as default package source for Model Serving deployments
* [Privilege Escalation Prevention for Views and Functions](/concepts/privilege-escalation-prevention-for-views-and-functions.md) - Only a metastore admin can transfer ownership of a view, function, or model to any user/group. Current owners and MANAGE users are restricted to transferring ownership to themselves or a group they belong to, preventing privilege escalation.
* [Privilege Grant and Revocation](/concepts/privilege-grant-and-revocation.md) - Granting, revoking, and inspecting privileges on Unity Catalog objects using Catalog Explorer and SQL
* [Privilege Hierarchy and Inheritance](/concepts/privilege-hierarchy-and-inheritance.md) - The hierarchical inheritance of privileges where granting on CATALOG cascades to all schemas and their objects, and DENY on a parent hides objects from listing.
* [Privilege Inheritance Hierarchy](/concepts/privilege-inheritance-hierarchy.md) - The downward inheritance model where privileges granted or denied on a parent object (e.g., CATALOG) automatically apply to all child objects (schemas, tables, etc.).
* [Privileges and Ownership](/concepts/privileges-and-ownership.md) - The traditional RBAC-style model controlling who can access what securable objects via grants and ownership in Unity Catalog.
* [Privileges and Ownership Model](/concepts/privileges-and-ownership.md) - Controls who can access what objects in Unity Catalog through grants on securable objects, with a hierarchy of privileges from account-level to table-level.
* [Privileges for External Lineage](/concepts/privileges-for-external-lineage.md) - The set of Unity Catalog privileges required to create external metadata objects and configure lineage relationships, including CREATE EXTERNAL METADATA, MODIFY, SELECT, and MODIFY privileges depending on the role of the object.
* [Privileges for External Lineage Management](/concepts/privileges-for-external-lineage.md) - The specific Unity Catalog privileges required to create external metadata objects (CREATE EXTERNAL METADATA), specify lineage relationships (MODIFY on the external object), and define upstream/downstream relationships to UC objects (write/read privileges).
* [Privileges for Prompt Management](/concepts/privileges-for-prompt-management.md) - Required Unity Catalog permissions (CREATE FUNCTION, EXECUTE, MANAGE) for creating prompts and evaluation datasets, ensuring proper access control.
* [Probabilistic Time Series Forecasting on Databricks](/concepts/probabilistic-time-series-forecasting-on-databricks.md) - An end-to-end workflow for probabilistic time series forecasting on Databricks, covering data ingestion, resampling, model training, prediction, visualization, and evaluation.
* [Probabilistic Time-Series Forecasting](/concepts/probabilistic-time-series-forecasting.md) - A forecasting approach that produces probability distributions over future values rather than point estimates
* [Producers and Consumers in Feature Store](/concepts/producers-and-consumers-in-feature-store.md) - A lineage concept distinguishing notebooks and jobs that write to a feature table (producers) from models, endpoints, notebooks, and jobs that read from it (consumers).
* [Production Feedback Collection with mlflow-tracing](/concepts/production-feedback-collection-with-mlflow-tracing.md) - The optimized production deployment pattern using the mlflow-tracing package with minimal dependencies and the log_feedback API, requiring MLflow 3 and supporting both trace ID and client request ID approaches.
* [Production ML Monitoring with Inference Tables](/concepts/production-ml-monitoring-with-inference-tables.md) - Continuous monitoring of production ML systems using inference tables to log model inputs/outputs automatically, combined with data quality monitoring, drift detection, and anomaly alerts.
* [Production Monitoring](/concepts/production-monitoring.md) - Capability to reuse LLM judges and scorers from development to monitor GenAI applications in production, ensuring consistent evaluation throughout the application lifecycle.
* [Production monitoring custom scorer functions](/concepts/production-monitoring-custom-scorer-functions.md) - The overall pattern for using user-defined scoring functions in Databricks production monitoring, including serialization and registration requirements
* [Production Monitoring for AI Agents](/concepts/production-monitoring-for-ai-agents.md) - Reusing the same judges and scorers from development in production to maintain consistent evaluation of AI application quality.
* [Production Monitoring for GenAI Applications](/concepts/production-monitoring-for-genai-applications.md) - The broader framework for setting up, managing, and backfilling production monitoring of GenAI applications using scorers in Databricks MLflow.
* [Production Monitoring for Long-term Trace Storage](/concepts/production-monitoring-for-long-term-trace-storage.md) - Optional beta feature that stores MLflow traces in Delta tables for durable long-term retention, removes trace size limits, enables automated quality assessment via MLflow scorers, and syncs approximately every 15 minutes.
* [Production Monitoring Scorer Restrictions](/concepts/production-monitoring-scorer-restrictions.md) - Class-based Scorer subclasses are not supported for production monitoring; only @scorer-decorated functions are allowed.
* [Production monitoring with custom scorer functions](/concepts/production-monitoring-with-custom-scorer-functions.md) - Production monitoring supports built-in LLM judges and @scorer-decorated functions, but not class-based Scorer subclasses
* [Production Quality Monitoring (MLflow GenAI)](/concepts/production-quality-monitoring-mlflow-genai.md) - Scheduled evaluation of production traces using scorers via mlflow.genai.Scorer.start() to detect quality issues in deployed applications.
* [Production Quality Monitoring on Databricks](/concepts/production-quality-monitoring-on-databricks.md) - Framework for continuous quality assessment of GenAI application traces using configurable scorers with sampling and filtering
* [Production Safety Monitoring](/concepts/production-safety-monitoring.md) - Continuous monitoring of safety for deployed AI applications, set up using MLflow's production monitoring capabilities to track safety evaluations over time.
* [Profile Alerts](/concepts/profile-alerts.md) - Databricks SQL alerts configured on profile metrics tables to notify when data statistics move out of expected ranges or when data drift is detected.
* [Profile analysis types](/concepts/profile-analysis-types.md) - Three types of data profiling analysis in Databricks: Snapshot, Timeseries, and InferenceLog, each with different dashboard selectors and default visualizations.
* [Profile and Dashboard Refresh Independence](/concepts/profile-and-dashboard-refresh-independence.md) - The mechanism by which profile metric tables and the profiling dashboard are updated independently — refreshing the profile updates metric tables but not the dashboard, and vice versa.
* [Profile Dashboard (Databricks)](/concepts/profile-dashboard-databricks.md) - An automatically generated, fully customizable dashboard in Databricks that visualizes data profiling results from profile and drift metric tables.
* [Profile Metric Table](/concepts/profile-metric-table.md) - An output table created by data profiling containing summary statistics for a table, stored as a Delta table in Unity Catalog, with metrics computed for time windows and data slices.
* [Profile Metric Tables](/concepts/profile-metric-tables.md) - Delta tables automatically created by Databricks data profiling that store summary statistics and drift metrics, queryable via SQL and usable for dashboards and alerts.
* [Profile Metrics Organization](/concepts/profile-metrics-organization.md) - Metrics computed by a profile run are organized into sections on the dashboard, with lists of metrics and statistics shown in tables and charts on the left side.
* [Profile Metrics Table](/concepts/profile-metrics-table.md) - A table storing data profiling statistics (e.g., fraction of missing values) that can be queried to trigger alerts on data quality changes.
* [Profile Refresh and Scheduling](/concepts/profile-refresh-and-scheduling.md) - Mechanisms to refresh data profiling metrics on-demand or on a cron schedule, with serverless compute execution.
* [Profile refresh mechanism](/concepts/profile-refresh-mechanism.md) - The two-step refresh process where profile calculations update metric tables, and a separate dashboard refresh runs queries over those tables to update visualizations.
* [Profile Refresh Operations](/concepts/profile-refresh-operations.md) - The process of triggering metric table creation/update via create_refresh API, running on serverless compute, with status monitoring via list_refreshes and get_refresh.
* [Profile Refresh Scheduling](/concepts/profile-refresh-scheduling.md) - Configuration for automated scheduled refreshes or manual refresh of data profiles, including frequency and time settings, with support for incremental processing via change data feed.
* [Profile Refresh vs Dashboard Refresh](/concepts/profile-refresh-vs-dashboard-refresh.md) - The distinction between refreshing the underlying profile metrics (via UI/API/schedule) and refreshing the dashboard's visualizations (via dashboard Refresh button); both must be done independently.
* [Profile Scheduling and Notifications](/concepts/profile-scheduling-and-notifications.md) - Configuration options for data profiles including scheduled refreshes using cron expressions and failure notifications via email (up to 5 addresses).
* [Profile Statistics Computation with Time Windows and Slices](/concepts/profile-statistics-computation-with-time-windows-and-slices.md) - Statistics are computed for specified time intervals (windows) and optionally for data slices defined by slicing expressions, with metrics aggregated across all combinations of time windows, slices, and grouping columns.
* [Profile Types in Databricks Data Profiling](/concepts/profile-types-in-databricks-data-profiling.md) - Databricks data profiling supports three analysis modes — Snapshot, Timeseries, and InferenceLog — each producing different visualization widgets and selectors on the dashboard.
* [Profiling Analysis Modes](/concepts/profiling-analysis-modes.md) - The three types of data profiling analysis in Databricks: time series, inference (for ML models), and snapshot, each suited to different monitoring use cases.
* [Profiling Granularity Options](/concepts/profiling-granularity-options.md) - The set of predefined aggregation granularities (from 5 minutes to 1 year) available for time-based profiles in Databricks data profiling.
* [Programmatic and Batch Evaluation Patterns](/concepts/programmatic-and-batch-evaluation-patterns.md) - DeepEval scorers can be used both programmatically as standalone scoring functions and in batch mode via mlflow.genai.evaluate() for dataset-level evaluation.
* [Programmatic Endpoint Configuration with Instance Profile ARN](/concepts/programmatic-endpoint-configuration-with-instance-profile-arn.md) - Using the REST API to attach an instance profile to a model serving endpoint by specifying the instance_profile_arn field in the served_entities configuration during endpoint creation or update.
* [Programmatic Instance Profile Assignment via Serving Endpoints API](/concepts/programmatic-instance-profile-assignment-via-serving-endpoints-api.md) - The Databricks Serving Endpoints API supports the instance_profile_arn field for attaching instance profiles during both endpoint creation and updates using REST API calls.
* [Programmatic Prompt Management with MLflow Python SDK](/concepts/programmatic-prompt-management-with-mlflow-python-sdk.md) - The MLflow Python SDK provides functions like mlflow.genai.register_prompt(), mlflow.genai.load_prompt(), and mlflow.genai.search_prompts() for programmatic prompt lifecycle management.
* [Prompt Aliases](/concepts/prompt-aliases.md) - Mutable pointers (e.g., 'production', 'staging') that reference specific prompt versions, enabling safe deployment strategies like A/B testing and gradual rollouts.
* [Prompt and Model Versioning](/concepts/prompt-and-model-versioning.md) - Version-controlled prompt templates with Git-like history and dynamic variables, plus logged models that snapshot application configurations, link to traces and evaluation runs, and can package code for deployment.
* [Prompt Chaining](/concepts/prompt-chaining.md) - An architectural pattern where the output of one prompt (e.g., a plan) is fed as input into a subsequent prompt (e.g., an answer) to decompose complex tasks into sequential steps.
* [Prompt Discovery and Search](/concepts/prompt-discovery-and-search.md) - Searching prompts using filter strings on catalog and schema with the mlflow.genai.search_prompts() function.
* [Prompt Iteration and Comparison Workflow](/concepts/prompt-iteration-and-comparison-workflow.md) - Workflow for iteratively improving GenAI application prompts, re-running evaluations, and comparing results side-by-side in the MLflow UI to measure improvement.
* [Prompt Iteration via Evaluation](/concepts/prompt-iteration-via-evaluation.md) - Iterative development workflow where GenAI prompts are refined based on quantitative evaluation results, with runs compared in the MLflow UI.
* [Prompt Iteration Workflow](/concepts/prompt-iteration-workflow.md) - A methodology for iteratively improving LLM prompts by defining evaluation criteria, running evaluations, reviewing results in the MLflow UI, refining prompts, and comparing runs.
* [Prompt Iteration Workflow for GenAI](/concepts/prompt-iteration-workflow-for-genai.md) - The iterative process of defining system prompts, evaluating outputs against criteria, refining prompts, and comparing results in the MLflow UI.
* [Prompt Optimization for Text Classification](/concepts/prompt-optimization-for-text-classification.md) - The pattern of using prompt optimization to align a generic classification prompt to a specific set of label categories and formatting constraints
* [Prompt Registry](/concepts/prompt-registry.md) - A centralized system for registering, versioning, and managing prompt templates used in GenAI applications, with support for tracking changes via commit messages.
* [Prompt Registry API (mlflow.genai)](/concepts/prompt-registry-api-mlflowgenai.md) - Python SDK functions including register_prompt(), load_prompt(), and search_prompts() for programmatic prompt lifecycle management.
* [Prompt Registry in Unity Catalog](/concepts/prompt-registry-in-unity-catalog.md) - A centralized prompt management system for browsing, versioning, and aliasing GenAI prompts stored in Unity Catalog
* [Prompt Registry load_prompt](/concepts/prompt-registry-load_prompt.md) - Loading specific prompt versions by URI ('prompts:/...') or by name and version number, with optional missing parameter handling.
* [Prompt Registry Location Tag](/concepts/prompt-registry-location-tag.md) - An MLflow experiment tag 'mlflow.promptRegistryLocation' that links an experiment to a Unity Catalog schema, allowing SDKs and tools to infer the prompt storage location automatically.
* [Prompt Registry register_prompt](/concepts/prompt-registry-register_prompt.md) - Registering a new prompt in the Prompt Registry with template variables, commit messages, and metadata tags.
* [Prompt Registry search_prompts with Unity Catalog constraints](/concepts/prompt-registry-search_prompts-with-unity-catalog-constraints.md) - Searching prompts in Unity Catalog registries is restricted to catalog.schema filter pairs; name patterns, tag filters, and combined filters are unsupported, requiring programmatic post-filtering.
* [Prompt Registry Version Control](/concepts/prompt-registry-version-control.md) - MLflow Prompt Registry's automatic registration of optimized prompts, enabling version control and retrieval of prompt versions via URIs like prompts:/{prompt_location}@latest.
* [Prompt Template Formats](/concepts/prompt-template-formats.md) - Two supported prompt storage formats in the Prompt Registry: simple string prompts and conversation-style (chat) prompts, both using double-brace syntax for template variables.
* [Prompt Template Variables](/concepts/prompt-template-variables.md) - Double-brace syntax ({{variable_name}}) for defining dynamic placeholders in prompt templates that are filled at runtime.
* [Prompt Templates with Variable Substitution](/concepts/prompt-templates-with-variable-substitution.md) - Prompts support double-brace syntax ({{variable_name}}) for defining template variables that are filled in at runtime using the prompt.format() method.
* [Prompt Templates with Variables](/concepts/prompt-templates-with-variables.md) - Prompts use double-brace syntax ({{variable_name}}) to define template variables that are filled in at runtime with dynamic data.
* [Prompt Types: Text vs Chat](/concepts/prompt-types-text-vs-chat.md) - Two prompt types supported in the MLflow Prompt Registry: Text (single completion-style template) and Chat (list of role-based messages for conversational models).
* [Prompt Version Aliases](/concepts/prompt-version-aliases.md) - Assigning aliases (e.g. 'production', 'staging') to specific prompt versions to manage deployment stages without referencing version numbers.
* [Prompt Version Evaluation](/concepts/prompt-version-evaluation.md) - A systematic methodology for comparing different prompt versions using consistent evaluation datasets, custom scorers, and composite metrics to identify the best-performing prompt.
* [Prompt Version Lifecycle](/concepts/prompt-version-lifecycle.md) - The end-to-end workflow for prompt management: creating versions, evaluating against datasets, selecting the best performer, and deploying to production using aliases.
* [Prompt Version Management](/concepts/prompt-version-management.md) - Registering, versioning, and iterating prompt templates in MLflow's prompt registry with semantic commit messages.
* [Prompt Version Management (MLflow GenAI)](/concepts/prompt-version-management-mlflow-genai.md) - Version-controlled templates for LLM prompts with Git-like history, variables, aliases, and links to evaluation runs for quality tracking.
* [Prompt Version Management in MLflow](/concepts/prompt-version-management.md) - Systematic approach to creating, registering, versioning, and comparing different prompt templates for GenAI applications using MLflow's prompt registry.
* [Prompt Versioning](/concepts/prompt-versioning.md) - Git-like version control for prompt templates, providing immutable snapshots with auto-incrementing version numbers, commit messages, and rollback capabilities.
* [Prompt Versioning and Immutability](/concepts/prompt-versioning-and-immutability.md) - Prompts in MLflow are immutable once created; edits produce new versions with full history preserved, enabling rollbacks and comparison.
* [Prompts and Logged Models](/concepts/prompts-and-logged-models.md) - Prompts are version-controlled LLM templates with variables and aliases; logged models are application snapshots linking traces, prompts, and evaluation runs, serving as a metadata hub or deployable artifact.
* [Protocol and metadata conflicts in Delta Lake](/concepts/protocol-and-metadata-conflicts-in-delta-lake.md) - Transaction conflicts triggered by concurrent operations that change the table protocol version, schema, or partitioning metadata, requiring a retry.
* [Provided Libraries on Databricks Serverless](/concepts/provided-libraries-on-databricks-serverless.md) - The curated set of libraries automatically available on serverless compute (environment version 4), which should not be bundled in JARs
* [Provided Libraries on Databricks Serverless Compute](/concepts/provided-libraries-on-databricks-serverless-compute.md) - A curated set of libraries (Databricks Connect, Scala, logging, HTTP, Jackson, Guava, etc.) are pre-installed on serverless compute; bundling duplicate versions causes NoSuchMethodError.
* [Provided Libraries on Serverless Compute](/concepts/provided-libraries-on-serverless-compute.md) - A curated set of required libraries (Databricks Connect, Scala libraries, Jackson, Guava, etc.) that are pre-installed on serverless compute and must be declared 'provided' in your build.
* [Provider and Recipient Auto-Creation](/concepts/provider-and-recipient-auto-creation.md) - When an SAP BDC connection is created on Databricks, two objects (a Provider and a Recipient) are automatically generated to represent the SAP BDC account for bidirectional sharing.
* [Provider Audit Log Events](/concepts/provider-audit-logs.md) - Audit events recorded for data providers in OpenSharing, including query response details, error messages, and actions taken by recipients on shared data.
* [Provider Audit Logs](/concepts/provider-audit-logs.md) - Audit logs recording actions taken by the data provider and actions taken by recipients on the provider's shared data, including query result details and error messages.
* [Provider Credential Rotation](/concepts/provider-credential-rotation.md) - The process of rotating bearer tokens or credentials for OpenSharing providers using Databricks REST API PATCH request, without dropping and recreating the provider object.
* [Provider Error Events in OpenSharing](/concepts/provider-error-events-in-opensharing.md) - OpenSharing logs specific error messages for providers, covering failures such as disabled features, missing catalogs, permission denials, invalid names, and duplicate resources.
* [Provider Internal ID Binding](/concepts/provider-internal-id-binding.md) - Catalogs bind to a provider's internal ID (not its name), so recreating a provider with the same name after deletion breaks the catalog's connection to shared data.
* [Provider Management Privileges in Unity Catalog](/concepts/provider-management-privileges-in-unity-catalog.md) - The permission hierarchy for managing provider objects: USE PROVIDER for viewing, CREATE PROVIDER for creating, and owner privileges for updating and deleting providers in Unity Catalog.
* [Provider Native APIs (Databricks)](/concepts/provider-native-apis-databricks.md) - Direct access to provider-specific API surfaces (OpenAI, Anthropic, Google) on Databricks Model Serving, bypassing the unified OpenAI-compatible API to use latest provider features.
* [Provider vs Recipient Audit Perspectives](/concepts/provider-vs-recipient-audit-perspectives.md) - Distinction between provider audit logs (recording both provider and recipient actions on shared data) and recipient audit logs (recording share access and provider object management)
* [Provider-Recipient Role Model in Delta Sharing](/concepts/provider-recipient-role-model-in-delta-sharing.md) - The two-party governance model in Delta Sharing where providers configure data access rules and recipients verify or contact providers when access issues arise.
* [Provider-Recipient roles in Delta Sharing error resolution](/concepts/provider-recipient-roles-in-delta-sharing-error-resolution.md) - The separation of responsibilities between data providers and data recipients when resolving Delta Sharing access errors
* [Provider-Side ABAC Policy Exemption via EXCEPT Clause](/concepts/provider-side-abac-policy-exemption-via-except-clause.md) - The mechanism by which share owners are exempted from ABAC row filter and column mask policies on the provider side using the EXCEPT clause in CREATE POLICY statements, enabling them to share protected data.
* [Provider-side OpenSharing Audit Events](/concepts/provider-side-opensharing-audit-events.md) - Audit events visible to data providers in OpenSharing, including query result details for pre-signed URL shares and STS-token shares, and error messages for failed operations.
* [Provider-side resolution for Delta Sharing access errors](/concepts/provider-side-resolution-for-delta-sharing-access-errors.md) - Troubleshooting guidance where the data provider must verify that the recipient or recipient property exists and is correctly configured.
* [Provisioned Concurrency](/concepts/provisioned-concurrency.md) - The amount of parallel processing capacity allocated to a model serving endpoint; increasing provisioned concurrency allows more requests to be processed simultaneously, directly impacting throughput.
* [Provisioned Concurrency and Parallel Request Limits](/concepts/provisioned-concurrency-and-parallel-request-limits.md) - Managing Databricks Model Serving concurrency quotas, parallel request limits, and the 'too many concurrent requests' 429 error, including mitigation strategies like autoscaling, route-optimized endpoints, and quota increases.
* [Provisioned Concurrency and Parallel Requests in Model Serving](/concepts/provisioned-concurrency-and-parallel-requests-in-model-serving.md) - Managing provisioned concurrency quotas and parallel request limits in Databricks Model Serving, including error handling and scaling strategies.
* [Provisioned Concurrency for Serving Endpoints](/concepts/provisioned-concurrency-for-serving-endpoints.md) - Controls how many simultaneous requests a model serving endpoint can process, with autoscaling support. Calculated as Target QPS times Average Latency. Used to handle baseline traffic and spikes while controlling costs.
* [Provisioned Concurrency Quotas in Model Serving](/concepts/provisioned-concurrency-quotas-in-model-serving.md) - Workspace-level limits on provisioned concurrency for model serving endpoints, including how to free quota by deleting/stopping endpoints and requesting increases.
* [Provisioned Throughput](/concepts/provisioned-throughput.md) - A deployment option for production workloads using base or fine-tuned models with performance guarantees, available through Foundation Model APIs.
* [Provisioned Throughput (Databricks)](/concepts/provisioned-throughput-databricks.md) - A deployment option for production workloads using base or fine-tuned models with performance guarantees, as opposed to pay-per-token pricing.
* [Provisioned Throughput and Tokens Per Second](/concepts/provisioned-throughput-and-tokens-per-second.md) - A model for provisioning inference capacity on Databricks based on tokens per second, with a throughput ceiling reached when the endpoint's provisioned capacity is saturated.
* [Provisioned Throughput Benchmarking](/concepts/provisioned-throughput-benchmarking.md) - The methodology of measuring tokens per second under varying concurrent request loads to find the saturation point where throughput plateaus due to provisioned capacity limits.
* [Provisioned Throughput Benchmarking for LLM Endpoints](/concepts/provisioned-throughput-benchmarking-for-llm-endpoints.md) - The practice of load-testing an LLM serving endpoint by measuring tokens per second across varying concurrency levels to understand capacity limits and latency-throughput curves.
* [Provisioned Throughput Compliance](/concepts/provisioned-throughput-compliance.md) - Provisioned throughput workloads support the full range of compliance standards (HIPAA, PCI-DSS, FedRAMP, IRAP, CCCS, UK Cyber Essentials Plus) in supported regions.
* [Provisioned throughput deployment mode](/concepts/provisioned-throughput-deployment-mode.md) - A reserved-capacity pricing model for Foundation Model APIs supporting the full range of compliance standards including HIPAA, PCI-DSS, FedRAMP, IRAP, CCCS, and UK Cyber Essentials Plus.
* [Provisioned Throughput Endpoint](/concepts/provisioned-throughput-endpoint.md) - A Databricks serving endpoint type that reserves dedicated inference capacity for foundation models, configured using model units or tokens per second.
* [Provisioned Throughput Endpoint Enforcement Gap](/concepts/provisioned-throughput-enforcement-gap.md) - A limitation where provisioned throughput endpoints do not automatically enforce foundation model Unity Catalog permissions and must be manually deleted by admins, unlike pay-per-token and batch inference endpoints which enforce automatically.
* [Provisioned Throughput Endpoint REST API](/concepts/provisioned-throughput-endpoint-rest-api.md) - Creating provisioned throughput serving endpoints via REST API by specifying min/max provisioned throughput fields and querying model optimization info.
* [Provisioned throughput endpoints](/concepts/provisioned-throughput-endpoints.md) - Alternative to pay-per-token endpoints offering no TPM restrictions, higher rate limits (up to 200 QPS), and predictable performance for production workloads.
* [Provisioned Throughput Enforcement Gap](/concepts/provisioned-throughput-enforcement-gap.md) - A limitation where provisioned throughput endpoints for disallowed models continue serving until manually deleted, unlike pay-per-token and batch inference endpoints which enforce permissions automatically.
* [Provisioned Throughput for LLM Endpoints](/concepts/provisioned-throughput-for-llm-endpoints.md) - A capacity planning model where inference capacity is provisioned based on tokens per second, with endpoint scaling limited by provisioned throughput leading to a throughput plateau.
* [Provisioned Throughput for LLM Inference on Databricks](/concepts/provisioned-throughput-for-llm-inference-on-databricks.md) - A capacity provisioning model for Databricks Foundation Model APIs where inference capacity is allocated in tokens per second for specific supported models (e.g., Meta Llama, GTE, BGE).
* [Provisioned Throughput for LLM Serving](/concepts/provisioned-throughput-for-llm-serving.md) - A Databricks offering that simplifies LLM serving endpoints by allowing scale-out configuration in tokens per second instead of concurrency, and removing the need for manual GPU workload type selection.
* [Provisioned Throughput Foundation Model APIs](/concepts/provisioned-throughput-foundation-model-apis.md) - Optimized inference mode for foundation model workloads requiring performance guarantees, compliance certifications like HIPAA, and support for fine-tuned models.
* [Provisioned Throughput Mode](/concepts/provisioned-throughput-mode.md) - A reserved-capacity deployment mode for foundation models, recommended by Databricks for production workloads, supporting all models of a given architecture family.
* [Provisioned Throughput Multi-Model Endpoint](/concepts/provisioned-throughput-multi-model-endpoint.md) - A Foundation Model API endpoint configured with provisioned throughput that serves multiple generative AI models with a shared traffic split.
* [Provisioned Throughput Permissions Enforcement Gap](/concepts/provisioned-throughput-enforcement-gap.md) - A limitation where provisioned throughput endpoints do not automatically enforce foundation model Unity Catalog permissions and require manual deletion of disallowed endpoints.
* [Provisioned throughput serving mode](/concepts/provisioned-throughput-serving-mode.md) - Dedicated endpoints with performance guarantees, HIPAA compliance, fine-tuned model support, and optimized inference for production workloads on Databricks.
* [Provisioned Throughput vs Pay-per-Token](/concepts/provisioned-throughput-vs-pay-per-token.md) - Two pricing models for Foundation Model APIs: pay-per-token for immediate use of base models, and provisioned throughput for production workloads with performance guarantees.
* [Publish Data Share to SAP BDC Notebook](/concepts/publish-data-share-to-sap-bdc-notebook.md) - A Databricks notebook that uses the SAP BDC SDK to describe a share in Core Schema Notation (CSN) for consumption by SAP BDC users
* [Publish Modes for Online Feature Stores](/concepts/publish-modes-for-online-feature-tables.md) - Synchronization modes (TRIGGERED, CONTINUOUS) that control how and when online tables are updated with changes from offline feature tables.
* [Publish Modes for Online Feature Tables](/concepts/publish-modes-for-online-feature-tables.md) - The publish_mode parameter (TRIGGERED, CONTINUOUS) determines how and when an online table is synced with changes from the offline feature table, replacing the older streaming parameter.
* [Publish Table API](/concepts/publish-table-api.md) - The API operation (publish_table) that creates an online table, syncs feature data from an offline feature table to the online store, and sets up infrastructure for ongoing synchronization.
* [Publish Table to Online Store](/concepts/publish-table-to-online-store.md) - The process of synchronizing a Delta table from Unity Catalog to a Databricks Online Feature Store for low-latency feature serving.
* [Publishing Feature Tables to Online Stores](/concepts/publishing-feature-tables-to-online-stores.md) - The publish_table API that synchronizes data from offline feature tables to an online store, including prerequisites like primary key constraints, non-nullable keys, and Change Data Feed.
* [Pull Request Driven ML Development](/concepts/pull-request-driven-ml-development.md) - Workflow where data scientists file pull requests to update ML code, triggering automated unit tests and integration tests in an isolated staging workspace before merging to main.
* [Purge Phase After Deletion](/concepts/purge-phase-after-deletion.md) - After the 7-day recovery window ends, Unity Catalog permanently deletes data files within 48 hours, and the object can no longer be recovered.
* [PyArrow Compatibility Constraint for horovod.spark](/concepts/pyarrow-compatibility-constraint-for-horovodspark.md) - horovod.spark does not support pyarrow versions 11.0 and above, requiring manual installation of a compatible version on Databricks Runtime 15.0 ML and above.
* [PyArrow Compatibility Constraint with horovod.spark](/concepts/pyarrow-compatibility-constraint-for-horovodspark.md) - horovod.spark does not support pyarrow versions 11.0 and above, requiring manual installation of pyarrow below version 11.0 for Databricks Runtime 15.0 ML and above.
* [PyFunc Code Path Sharing](/concepts/pyfunc-code-path-sharing.md) - Using the code_path parameter in mlflow.pyfunc.log_model to include shared utility modules that can be imported and used inside custom pyfunc model implementations.
* [PyFunc Context Artifact Access Pattern](/concepts/pyfunc-context-artifact-access-pattern.md) - Custom PyFunc models access packaged artifacts through the context.artifacts dictionary inside load_context(), enabling static loading of model weights, tokenizers, and other files at deployment time.
* [PyFunc Model Preprocessing and Postprocessing](/concepts/pyfunc-model-preprocessing-and-postprocessing.md) - Using MLflow pyfunc's predict method to encapsulate preprocessing logic (input formatting) and postprocessing logic (output transformation) within a custom model class.
* [PyFunc Preprocessing and Postprocessing Patterns](/concepts/pyfunc-preprocessing-and-postprocessing-patterns.md) - Custom MLflow pyfunc models can encapsulate preprocessing logic (e.g., tokenization), model inference, and postprocessing (e.g., sigmoid transformation) within the predict function, enabling end-to-end serving.
* [PySpark and Databricks Connect Conflict](/concepts/pyspark-and-databricks-connect-conflict.md) - The databricks-connect package conflicts with PySpark, requiring PySpark to be uninstalled before installing Databricks Connect.
* [PySpark and Databricks Connect Mutual Exclusivity](/concepts/pyspark-and-databricks-connect-mutual-exclusivity.md) - The databricks-connect package conflicts with PySpark; both cannot be installed in the same environment without causing stream corruption or class-not-found errors, requiring full uninstall and reinstall when switching.
* [PySpark Sandbox Restrictions](/concepts/pyspark-sandbox-restrictions.md) - Security restrictions applied to user code when Python and SQL table access control is enabled, including low-privilege users, filesystem restrictions, and network port limitations.
* [pyspark.ml.connect](/concepts/pysparkmlconnect.md) - A Spark ML module for distributed training and inference of Spark ML models, enabled by default in Databricks Runtime 17.0+ on Standard compute resources.
* [pysparklyr](/concepts/pysparklyr.md) - R package that manages the installation and configuration of Databricks Connect dependencies, including automatically determining the correct Databricks Runtime version from a cluster ID.
* [pysparklyr Package](/concepts/pysparklyr.md) - An R package that manages installation of Databricks Connect dependencies and Python environments, bridging R and PySpark for Databricks connectivity.
* [Python 3.11 Shared Memory Issues with StreamingDataset](/concepts/python-311-shared-memory-issues-with-streamingdataset.md) - Transient issues with StreamingDataset on Python 3.11 shared memory implementation in Databricks Runtime 15.4 LTS ML, resolved by upgrading to Databricks Runtime 16.4 LTS ML with Python 3.12.
* [Python and SQL Table Access Control](/concepts/python-and-sql-table-access-control.md) - A version of table access control that allows users to run SQL, Python, and PySpark DataFrame API commands while enforcing low-privilege user restrictions and network/firewall constraints.
* [Python Base Environment for UDF Execution](/concepts/python-base-environment-for-udf-execution.md) - UDFs execute in the Python environment of the Databricks compute (cluster or serverless), not on the client, with specific package sets defined by the Databricks Runtime version or serverless environment version.
* [Python Environment Compatibility for Databricks Connect](/concepts/python-environment-compatibility-for-databricks-connect.md) - The requirement that the minor version of Python on the development machine must match the minor Python version of the Databricks cluster, with recommended use of virtual environments.
* [Python UDF performance penalty in ABAC policies](/concepts/python-udf-performance-penalty-in-abac-policies.md) - Python UDFs used in ABAC policies must be wrapped in SQL UDFs and are generally slower than SQL UDFs because the optimizer cannot inline or optimize them, and the Python function executes for every row.
* [Python UDFs in Unity Catalog](/concepts/python-udfs-in-unity-catalog.md) - Python user-defined functions registered as first-class objects in Unity Catalog's three-level namespace, usable for on-demand feature computation.
* [Python Version Compatibility for UDFs](/concepts/python-version-compatibility-for-udfs.md) - The Python version on the Databricks Connect client must exactly match the Python version on the Databricks compute cluster for UDF serialization and deserialization to work correctly.
* [Python Version Compatibility in Databricks Connect](/concepts/python-version-compatibility-in-databricks-connect.md) - Databricks Connect requires the local Python version to match at least the same minor release as the cluster's Python version, configurable via the PYSPARK_PYTHON environment variable.
* [Python Virtual Environment for Databricks Connect](/concepts/python-virtual-environment-for-databricks-connect.md) - Databricks strongly recommends using a Python virtual environment (venv or Poetry) to isolate the correct versions of Python and Databricks Connect.
* [Python Wheel Files](/concepts/python-wheel-files.md) - A standard packaging format (.whl) for distributing Python applications, containing code, metadata, and dependencies.
* [Python Wheel Task](/concepts/python-wheel-task.md) - A Databricks Jobs task type that runs applications packaged as Python wheel files, enabling fast and reliable code execution.
* [Python/Scala UDF Wrapper Pattern for Column Masks](/concepts/pythonscala-udf-wrapper-pattern-for-column-masks.md) - The requirement to wrap Python or Scala UDFs in a SQL UDF before applying them as row filters or column masks, because only SQL UDFs can be directly assigned as filters or masks.
* [PyTorch DDP Implementation](/concepts/pytorch-ddp-implementation.md) - Using PyTorch's torch.nn.parallel.DistributedDataParallel module for distributed training of neural networks
* [PyTorch DDP on Databricks](/concepts/pytorch-ddp-on-databricks.md) - Using PyTorch's torch.nn.parallel.DDP module to train a multilayer perceptron (MLP) on Databricks with serverless GPU resources.
* [PyTorch Distributed Checkpoint (DCP)](/concepts/pytorch-distributed-checkpoint-dcp.md) - A PyTorch API for saving and loading model and optimizer state in distributed training contexts, supporting sharded state dicts and async checkpointing across multiple devices.
* [PyTorch Distributed Checkpoint (DCP) with AppState Wrapper](/concepts/pytorch-distributed-checkpoint-dcp-with-appstate-wrapper.md) - A pattern for checkpointing distributed model training using PyTorch's distributed checkpoint API combined with a Stateful AppState wrapper that automatically handles FSDP state dict management and checkpoint save/load operations.
* [PyTorch Distributed Checkpoint Stateful Protocol](/concepts/pytorch-distributed-checkpoint-stateful-protocol.md) - A pattern using AppState as a Stateful wrapper around model and optimizer to enable distributed checkpointing with dcp.save/dcp.load, automatically managing state_dict serialization.
* [PyTorch Distributed Training Error Troubleshooting](/concepts/pytorch-distributed-training-error-troubleshooting.md) - Common errors encountered with PyTorch DataParallel and DistributedDataParallel on Databricks and their solutions
* [PyTorch Fully Sharded Data Parallel (FSDP)](/concepts/fsdp-fully-sharded-data-parallel.md) - A distributed training strategy that shards model parameters, gradients, and optimizer states across multiple GPUs to reduce per-GPU memory and enable training of large models that don't fit on a single GPU.
* [PyTorch Installation on Databricks Runtimes](/concepts/pytorch-installation-on-databricks-runtimes.md) - Methods for installing PyTorch on Databricks Runtime vs Databricks Runtime for ML, including GPU and CPU cluster differences
* [PyTorch Lightning DDP integration](/concepts/pytorch-lightning-ddp-integration.md) - PyTorch Lightning provides a high-level interface that automatically handles DDP configuration for multi-GPU training.
* [PyTorch Lightning DDP multi-GPU training](/concepts/pytorch-lightning-ddp-multi-gpu-training.md) - Using PyTorch Lightning's high-level interface that automatically handles DDP configuration for multi-GPU training on Databricks AI Runtime.
* [PyTorch Lightning for DDP](/concepts/pytorch-lightning-for-ddp.md) - A high-level interface that automatically handles DDP configuration for multi-GPU training
* [PyTorch Lightning for Distributed Recommendation Training](/concepts/pytorch-lightning-for-distributed-recommendation-training.md) - Use of PyTorch Lightning to implement distributed training of recommendation models, enabling multi-GPU and multi-node scaling on Databricks AI Runtime.
* [PyTorch Lightning for Distributed Training](/concepts/pytorch-lightning-for-distributed-training.md) - A lightweight PyTorch wrapper used for scaling deep learning model training across multiple GPUs in a distributed setting, demonstrated here for recommendation models.
* [PyTorch Lightning Trainer for Distributed Training](/concepts/pytorch-lightning-trainer-for-distributed-training.md) - Using Lightning's Trainer API with DDP strategy to distribute two-tower model training across multiple GPUs.
* [PyTorch Lightning with TorchRec for Distributed Training](/concepts/pytorch-lightning-with-torchrec-for-distributed-training.md) - Combining PyTorch Lightning's Trainer API with TorchRec's embedding modules for scalable, multi-GPU training of recommendation models using DDP strategy.
* [PyTorch MNIST distributed training](/concepts/pytorch-mnist-distributed-training.md) - A reference notebook and example demonstrating how to train an MNIST model with PyTorch and scale it using HorovodRunner on Databricks.
* [PyTorch MNIST Distributed Training Example](/concepts/pytorch-mnist-distributed-training.md) - Reference notebook demonstrating how to train an MNIST model with PyTorch, first on a single node and then distributed with HorovodRunner on Databricks
* [PyTorch on Databricks](/concepts/pytorch-on-databricks.md) - Using PyTorch within the Databricks environment for GPU-accelerated tensor computation and building deep learning networks, supporting both single-node and distributed training.
* [QLoRA (Quantized Low-Rank Adaptation)](/concepts/qlora-quantized-low-rank-adaptation.md) - A parameter-efficient fine-tuning technique that combines 4-bit quantization with low-rank adapter matrices to enable efficient LLM fine-tuning on consumer-grade hardware
* [Qualitative and Quantitative Trace Selection](/concepts/quantitative-and-qualitative-trace-selection.md) - Methods for selecting representative traces for evaluation datasets, combining quantitative analysis (filtering by tags, latency, token usage) with qualitative review (examining inputs, outputs, and edge cases).
* [Quality and Evaluation Support in Genie Code](/concepts/quality-and-evaluation-support-in-genie-code.md) - Genie Code capability to review assessment scores from human feedback, LLM judges, and programmatic checks, inspect evaluation datasets and registered scorers, and get help setting up mlflow.genai.evaluate().
* [Quality Dimensions for GenAI Evaluation](/concepts/quality-dimensions-for-genai-evaluation.md) - The set of common quality axes evaluated by built-in judges including relevance, safety, groundedness, correctness, sufficiency, guideline adherence, and tool call correctness.
* [Quantitative and Qualitative Trace Selection](/concepts/quantitative-and-qualitative-trace-selection.md) - Methodologies for selecting representative traces for evaluation datasets using measurable filters (tags, latency, token usage) and human judgment (reviewing edge cases, low-quality outputs, patterns).
* [Query Route-Optimized Serving Endpoints](/concepts/query-route-optimized-serving-endpoints.md) - Route-optimized endpoints require a different URL and OAuth-based query pattern compared to non-optimized endpoints.
* [Query Timeout (3600 seconds)](/concepts/query-timeout-3600-seconds.md) - Long queries exceeding 3600 seconds are not supported in Databricks Connect for Databricks Runtime 13.3 LTS and above.
* [Querying Feature Serving endpoints](/concepts/querying-feature-serving-endpoints.md) - Methods to query endpoints using REST API, MLflow Deployments SDK, or the Serving UI, passing dataframe_records with lookup keys and dynamic inputs.
* [Querying Model Serving Endpoints with Databricks SDK](/concepts/querying-model-serving-endpoints-with-databricks-sdk.md) - Using the Databricks Python SDK's WorkspaceClient.serving_endpoints.query() method to send inference requests with dataframe_records payloads.
* [Querying Vision Models on Databricks Model Serving](/concepts/querying-vision-models-on-databricks-model-serving.md) - How to send query requests to foundation models optimized for vision tasks via Databricks Model Serving endpoints using a unified API.
* [Queuing and Autoscaling in Model Serving](/concepts/queuing-and-autoscaling-in-model-serving.md) - How traffic surges cause request queuing when autoscaling cannot provision capacity fast enough, leading to increased latency and HTTP 429 errors. Mitigation includes setting minimum provisioned concurrency, enabling route optimization, and client-side retry logic.
* [Queuing in Model Serving](/concepts/queuing-in-model-serving.md) - When incoming requests exceed provisioned concurrency, requests queue up, causing increased end-to-end latency due to additional wait time before processing begins.
* [Qwen2-0.5B Language Model](/concepts/qwen2-05b-language-model.md) - A 0.5 billion parameter large language model from the Qwen2 family, used as the base model for parameter-efficient fine-tuning with LoRA in this tutorial.
* [R .Renviron Databricks configuration pattern](/concepts/r-renviron-databricks-configuration-pattern.md) - Best practice pattern for storing Databricks workspace URL, personal access token, and cluster ID in a .Renviron file for use in R projects with Databricks Connect.
* [RAG Application Architecture on Databricks](/concepts/rag-application-architecture-on-databricks.md) - The four-step pipeline for building structured RAG apps on Databricks: create a feature serving endpoint, build a LangChain tool, use it in a LangChain agent, and host via Model Serving.
* [RAG Application Evaluation with RetrievalRelevance](/concepts/rag-application-evaluation-with-retrievalrelevance.md) - A pattern for evaluating Retrieval-Augmented Generation applications by annotating a retriever function with @mlflow.trace(span_type='RETRIEVER') and passing RetrievalRelevance as a scorer to mlflow.genai.evaluate.
* [RAG evaluation with MLflow](/concepts/rag-evaluation-with-mlflow.md) - The practice of evaluating Retrieval-Augmented Generation pipelines using MLflow's evaluation tools, including defining retriever functions, constructing prompts with context, and scoring responses.
* [RAG Ground Truth Evaluation with Expected Facts](/concepts/rag-ground-truth-evaluation-with-expected-facts.md) - A pattern for evaluating retrieval-augmented generation systems by providing expected_facts in the evaluation dataset to ground truth judge assessments.
* [RAG Pipeline (txtai)](/concepts/rag-pipeline-txtai.md) - Retrieval Augmented Generation pipeline in txtai that combines embeddings retrieval with LLM generation, supporting custom prompt templates and traceable via MLflow
* [RAG Pipeline Tracing with MLflow](/concepts/rag-pipeline-tracing-with-mlflow.md) - Instrumenting a Retrieval-Augmented Generation pipeline with MLflow traces using RETRIEVER, CHAT_MODEL, TOOL, and CHAIN span types, including document metadata, token usage, and tool definitions.
* [RAG Recommendation as Accuracy Mitigation](/concepts/rag-recommendation-as-accuracy-mitigation.md) - A consistent recommendation across all documented models to use Retrieval Augmented Generation (RAG) when accuracy is critical, to mitigate the risk of hallucinations and factual omissions.
* [RAG with Fine-tuned Open Source Model Architecture](/concepts/rag-with-fine-tuned-open-source-model-architecture.md) - Production architecture for retrieval-augmented generation applications that fine-tune an open source model on custom data
* [RAG with Third-Party LLM API Architecture](/concepts/rag-with-third-party-llm-api-architecture.md) - Production architecture for retrieval-augmented generation applications that connect to third-party LLM APIs via Databricks External Models
* [RAGAS (Retrieval Augmented Generation Assessment)](/concepts/ragas-retrieval-augmented-generation-assessment.md) - An evaluation framework for LLM applications that assesses retrieval quality, answer generation, agent behavior, and text similarity.
* [RAGAS Metric Categories](/concepts/ragas-metric-categories.md) - The five categories of RAGAS metrics: RAG metrics, Agent and tool use metrics, Natural language comparison metrics, General purpose metrics, and Other tasks.
* [RAGAS Scorers in MLflow](/concepts/ragas-scorers-in-mlflow.md) - MLflow integration that enables RAGAS metrics to be used as built-in scorers for evaluating LLM traces via mlflow.genai.evaluate() or direct invocation.
* [RAGAS Scorers Integration](/concepts/ragas-scorers-integration.md) - MLflow integration with RAGAS providing deep RAG evaluation with fine-grained context metrics and deterministic text comparison scores like BLEU and ROUGE
* [Rank-0 Only Logging Pattern for Distributed Workloads](/concepts/rank-0-only-logging-pattern-for-distributed-workloads.md) - On multi-node workloads, all nodes share the same MLflow run ID; logging should only be done from the rank-0 process to avoid duplicate metric recording.
* [Rate Limit Error Handling (429 Errors)](/concepts/rate-limit-error-handling-429-errors.md) - Error responses returned when rate limits are exceeded, including fields like limit_type, limit, current usage, and retry_after, along with strategies for handling them.
* [Rate limit error handling and 429 responses](/concepts/rate-limit-error-handling-429-errors.md) - When rate limits are exceeded, the API returns a 429 error with details including limit_type, limit value, current usage, and suggested retry_after time.
* [Rate Limit Error Handling Best Practices](/concepts/rate-limit-error-handling-best-practices.md) - Best practices for handling 429 rate limit errors including exponential backoff retry logic, token usage monitoring, and error response parsing.
* [Rate Limit Failure](/concepts/rate-limit-failure.md) - Request failures caused by exceeding the rate limits of a model serving endpoint, which can appear similar to timeouts.
* [Rate Limit Induced Request Failure](/concepts/rate-limit-induced-request-failure.md) - Request failures caused by exceeding the rate limit of a model serving endpoint, leading to additional request rejections.
* [Rate Limiting in Delta Streaming](/concepts/rate-limiting-in-delta-streaming.md) - Using maxFilesPerTrigger and maxBytesPerTrigger to control micro-batch input size and manage streaming processing performance
* [Rate Limiting in Evaluation](/concepts/rate-limiting-in-evaluation.md) - Technique for wrapping predict functions with rate-limiting logic when evaluating models subject to API rate limits, using libraries like ratelimit.
* [Rate Limiting in GenAI Evaluation](/concepts/rate-limiting-in-genai-evaluation.md) - Techniques for controlling API call rates during evaluation, including wrapping predict functions with rate-limit decorators and configuring parallel worker count via environment variables.
* [Rate Limits and Model Serving Endpoint Throttling](/concepts/rate-limits-and-model-serving-endpoint-throttling.md) - How exceeding rate limits on model serving endpoints can cause request failures and timeouts, including references to resource and payload limits.
* [Rate Limits and Timeouts in Model Serving](/concepts/rate-limits-and-timeouts-in-model-serving.md) - How exceeding rate limits on model serving endpoints can lead to request failures and timeouts, with references to resource and payload limits.
* [Ray and Spark Cluster Resource Configuration](/concepts/ray-spark-cluster-resource-configuration.md) - Best practices for configuring CPU and GPU resources when co-locating Ray and Spark worker nodes, including num_cpus_worker_node and num_gpus_worker_node settings.
* [Ray and Spark Integration on Databricks](/concepts/ray-spark-integration-on-databricks.md) - Running Ray and Spark in the same execution environment on Databricks, supported by Delta Lake and Unity Catalog for data management and lineage tracking.
* [Ray Client Remote Connection](/concepts/ray-client-remote-connection.md) - Connecting to a remote Ray cluster from a Databricks notebook using the Ray client with a connection string obtained from setup_ray_cluster
* [Ray Client Remote Connection on Databricks](/concepts/ray-client-remote-connection.md) - Using the Ray client (Ray 2.3.0+) to connect to a remote Ray cluster on Databricks via a connection string, with limitations on Ray Dataset API
* [Ray Cluster Bootstrap on Databricks Nodes](/concepts/ray-cluster-bootstrap-on-databricks-nodes.md) - Pattern for starting a Ray cluster on Databricks compute nodes using the workload command, including head node initialization and worker node joining logic.
* [Ray cluster bootstrap pattern for multi-node training](/concepts/ray-cluster-bootstrap-pattern-for-multi-node-training.md) - Pattern for starting a Ray cluster inline within a workload command, with a head node branch and worker node join logic, enabling single-node and multi-node scaling.
* [Ray cluster lifecycle on Databricks](/concepts/ray-cluster-lifecycle-on-databricks.md) - Creating, managing, and shutting down Ray clusters within Databricks compute using the setup_ray_cluster and shutdown_ray_cluster APIs, including auto-shutdown after inactivity
* [Ray cluster log collection on Databricks](/concepts/ray-log-collection-on-databricks.md) - Configuring log output destinations for Ray clusters using the collect_log_to_path argument, with recommendations for /dbfs/ or Unity Catalog Volume paths to preserve logs after cluster termination
* [Ray Cluster on Databricks](/concepts/ray-cluster-on-databricks.md) - Creating and running Ray compute clusters on top of Databricks Spark clusters using the ray.util.spark.setup_ray_cluster API
* [Ray cluster requirements and installation on Databricks](/concepts/ray-cluster-requirements-and-installation-on-databricks.md) - Prerequisites for creating Ray clusters on Databricks including Databricks Runtime 12.2 LTS ML+, access modes, and Ray installation methods for different runtime versions
* [Ray Cluster Resource Configuration for Spark](/concepts/ray-cluster-resource-configuration-for-spark.md) - Best practices for configuring CPU and GPU allocations when running Ray and Spark together on shared worker nodes, including num_cpus_worker_node and num_gpus_worker_node settings.
* [Ray Cluster Scaling Best Practices](/concepts/ray-cluster-scaling-best-practices.md) - Recommended configurations for Ray worker nodes include minimum 4 CPU cores and 10GB heap memory per node, and setting num_cpus_worker_node to match Spark worker node CPU count for optimal resource utilization.
* [Ray Core MLflow Integration](/concepts/ray-core-mlflow-integration.md) - Patterns for integrating MLflow tracking with Ray Core distributed applications, including driver-process logging and child-run approaches.
* [Ray Core to Spark Data Exchange Patterns](/concepts/ray-core-to-spark-integration-patterns.md) - Three patterns for exchanging data between Ray Core applications and Spark: persisting to temporary storage, using Spark Connect, and using third-party libraries like deltalake and deltaray.
* [Ray Core to Spark Integration Patterns](/concepts/ray-core-to-spark-integration-patterns.md) - Three patterns for writing data from Ray Core applications to Spark: persist to temporary location, connect via Spark Connect, or use third-party libraries like deltalake or deltaray.
* [Ray Data to Unity Catalog via write_databricks_table()](/concepts/ray-data-to-unity-catalog-via-write_databricks_table.md) - Write Ray datasets directly to Unity Catalog tables using ray.data.Dataset.write_databricks_table(), which stages data through Unity Catalog Volumes and requires the _RAY_UC_VOLUMES_FUSE_TEMP_DIR environment variable.
* [Ray Delta Sharing Read API](/concepts/ray-delta-sharing-read-api.md) - Reading data from Databricks Delta Sharing (OpenSharing) tables using ray.data.read_delta_sharing_tables as a more reliable alternative to SQL warehouse queries for external Ray applications
* [Ray GPU Cluster Configuration](/concepts/ray-gpu-cluster-configuration.md) - Configuring Ray clusters on Databricks to leverage GPU resources by specifying num_gpus_per_node and num_gpus_head_node parameters
* [Ray GPU cluster configuration on Databricks](/concepts/ray-gpu-cluster-configuration.md) - Configuring GPU resources for Ray clusters on Databricks, including head/worker node GPU allocation and best practices for mixed GPU/CPU workloads
* [Ray Head Node Resource Configuration](/concepts/ray-head-node-resource-configuration.md) - Databricks restricts Ray head node resources by default (0 CPU, 0 GPU, 128 MB memory) since it's used for coordination, but allows configuration of CPU, GPU, and memory for the head node via setup_ray_cluster API parameters.
* [Ray Job CLI Integration via Databricks Driver Proxy](/concepts/ray-job-cli-with-databricks-driver-proxy.md) - Connecting Ray Job CLI to a Databricks-hosted Ray cluster through the driver proxy endpoint
* [Ray Job CLI integration with Databricks](/concepts/ray-job-cli-integration-with-databricks.md) - Using the Ray Job CLI to submit jobs to Ray clusters running on Databricks via the driver proxy, alongside recommendations for using Lakeflow Jobs instead
* [Ray Job CLI with Databricks Driver Proxy](/concepts/ray-job-cli-with-databricks-driver-proxy.md) - Connecting the Ray Job CLI to a Databricks-hosted Ray cluster through the driver proxy endpoint for external job submission
* [Ray Log Collection on Databricks](/concepts/ray-log-collection-on-databricks.md) - Collecting and preserving Ray cluster logs using collect_log_to_path with DBFS or Unity Catalog Volume paths
* [Ray Memory Monitor Issues on Databricks](/concepts/ray-memory-monitor-issues-on-databricks.md) - Known issues in Ray 2.9.3 where the memory monitor can inadvertently kill Ray tasks, with workaround of disabling the monitor via RAY_memory_monitor_refresh_ms environment variable
* [Ray on Databricks](/concepts/ray-on-databricks.md) - An open-source framework for parallel compute processing that scales ML workflows and AI applications within the Databricks environment.
* [Ray on Databricks Autoscaling](/concepts/ray-on-databricks-autoscaling.md) - Ray clusters on Databricks support integration with Databricks autoscaling, allowing Ray worker nodes to scale up and down based on workload demand using min/max worker node parameters.
* [Ray on Databricks Best Practices](/concepts/ray-on-databricks-best-practices.md) - Collection of best practices for running Ray on Databricks including GPU allocation, transformers trainer MLflow integration, pickling errors, memory monitor, batch processing, and library scoping
* [Ray on Databricks Overview](/concepts/ray-on-databricks-overview.md) - Running Ray compute clusters on top of Databricks Spark clusters using Databricks Runtime ML
* [Ray on Spark Cluster Architecture](/concepts/ray-on-spark-cluster-architecture.md) - Ray on Spark creates a Ray cluster via a background Apache Spark job where each Spark task creates a Ray worker node, and the Ray head node runs on the Spark driver, with API version differences between Ray <2.10 and >=2.10.
* [Ray Serve on Databricks Limitations](/concepts/ray-serve-on-databricks-limitations.md) - Challenges of using Ray Serve for real-time inference on Databricks clusters due to network security constraints, with recommendation to use Databricks Model Serving instead.
* [Ray to Unity Catalog Table Writing](/concepts/ray-to-unity-catalog-table-writing.md) - Writing Ray datasets directly to Unity Catalog tables using ray.data.Dataset.write_databricks_table() via Unity Catalog Volumes as an intermediary
* [Ray Train distributed training on AI Runtime](/concepts/ray-train-distributed-training-on-ai-runtime.md) - Distributed data-parallel fine-tuning using Ray Train's TorchTrainer across multiple H100 GPUs on a single node
* [Ray Train Distributed Training on Databricks](/concepts/ray-train-distributed-training-on-databricks.md) - Pattern for distributed data-parallel fine-tuning using Ray Train's TorchTrainer across GPUs on a single node, with one worker per GPU, running on Databricks AI Runtime
* [Ray Train MLflow Logging](/concepts/ray-train-mlflow-logging.md) - Method for logging Ray Train models to MLflow by reloading from checkpoints and using MLflow model flavors such as PyTorch or TensorFlow.
* [Ray Train Resource Allocation](/concepts/ray-train-resource-allocation.md) - Configuration guidance for setting resources_per_worker in Ray Train to avoid resource contention on Databricks clusters.
* [Ray Train TorchTrainer](/concepts/ray-train-torchtrainer.md) - A Ray Train API that launches one worker per GPU, wraps models in DDP, and shards datasets across distributed workers for data-parallel training.
* [Ray Tune](/concepts/ray-tune.md) - A distributed hyperparameter tuning library that uses Ray as a backend for parallel compute, available in Databricks Runtime ML.
* [Ray Tune MLflow Integration](/concepts/ray-tune-mlflow-integration.md) - Integration of Ray Tune distributed hyperparameter tuning with MLflow experiment tracking using child runs and the MLflowLoggerCallback.
* [Ray Worker Node Memory Allocation](/concepts/ray-worker-node-memory-allocation.md) - Memory allocation for Ray worker nodes is determined by physical memory per Spark worker, number of local Ray nodes, and configurable object store memory, with heap and object store split ratios and OS shared memory caps.
* [Ray-Spark Cluster Resource Configuration](/concepts/ray-spark-cluster-resource-configuration.md) - Best practices for configuring CPU and GPU resource allocations when running Ray and Spark on the same Databricks cluster, including num_cpus_worker_node and num_gpus_worker_node settings.
* [Ray-Spark Integration on Databricks](/concepts/ray-spark-integration-on-databricks.md) - Architecture and patterns for running Ray and Spark in the same execution environment on Databricks, leveraging Delta Lake and Unity Catalog for data management.
* [Ray-Spark Resource Allocation Best Practices on Databricks](/concepts/ray-spark-resource-allocation-on-databricks.md) - Configure Ray worker nodes to match Spark worker node resources by setting num_cpus_worker_node and num_gpus_worker_node appropriately, ensuring one Ray worker per Spark worker for full resource utilization.
* [Ray-Spark Resource Allocation on Databricks](/concepts/ray-spark-resource-allocation-on-databricks.md) - Best practices for configuring CPU and GPU resources when running Ray and Spark workers on the same Databricks cluster, including num_cpus_worker_node settings
* [ray.data.from_spark() — In-Memory Spark-to-Ray Transfer](/concepts/raydatafrom_spark-in-memory-spark-to-ray-transfer.md) - Transfer Spark DataFrames to Ray datasets in-memory using ray.data.from_spark(), avoiding intermediate storage writes; available on Databricks Runtime ML 15.0+ with a required cluster config.
* [RayTune](/concepts/raytune.md) - Recommended replacement on Databricks for distributed hyperparameter tuning, similar to deprecated Hyperopt SparkTrials functionality
* [RayTune as distributed Hyperopt replacement on Databricks](/concepts/raytune.md) - Databricks recommends RayTune as a replacement for Hyperopt's distributed hyperparameter tuning functionality
* [RayTune for distributed hyperparameter tuning on Databricks](/concepts/raytune.md) - RayTune is recommended by Databricks as the replacement for Hyperopt for distributed hyperparameter tuning, providing a similar experience.
* [RDD to DataFrame API Migration in Databricks Connect](/concepts/rdd-to-dataframe-api-migration-in-databricks-connect.md) - The migration from RDD APIs to DataFrame APIs required when upgrading Databricks Connect, as RDD and SparkContext usage must be replaced.
* [RDD to DataFrame Migration Pattern](/concepts/rdd-to-dataframe-migration-pattern.md) - Replacing sc.parallelize with spark.read.json() pattern by using json.loads and spark.createDataFrame for Unity Catalog shared clusters.
* [Read Authentication for Feature Lookup with Served MLflow Models](/concepts/read-authentication-for-feature-lookup-with-served-models.md) - Methods and requirements for authenticating read operations when Databricks-hosted MLflow models look up features from third-party online stores.
* [Read Authentication for Feature Lookup with Served Models](/concepts/read-authentication-for-feature-lookup-with-served-models.md) - Authentication required when Databricks-hosted MLflow models look up feature values from third-party online stores during serving, supporting instance profiles or Databricks secrets with a read_secret_prefix.
* [Read Authentication for Served MLflow Models](/concepts/read-authentication-for-served-mlflow-models.md) - Required read credentials (via instance profile on serving endpoints or secrets with read_secret_prefix) for Databricks-hosted MLflow models to look up feature values from third-party online stores.
* [Read Predicates and Filters in Delta Lake Transactions](/concepts/read-predicates-and-filters-in-delta-lake-transactions.md) - The use of filter predicates and partition hints to narrow data scope for transaction reads, enabling row-level conflict detection and preventing full-table read conflicts.
* [Read Replicas for Online Feature Stores](/concepts/read-replicas-for-online-feature-stores.md) - Additional read-only compute instances that can be added to an online store to distribute read traffic, reduce latency, and improve availability by failing over if the primary instance fails.
* [Read Replicas for Online Stores](/concepts/read-replicas-for-online-stores.md) - A feature that distributes read traffic across multiple compute instances to reduce latency and improve scalability for high-concurrency workloads.
* [Read Replicas in Online Feature Stores](/concepts/read-replicas-in-online-feature-stores.md) - Read replicas distribute read traffic across multiple compute instances for high-concurrency workloads, supporting up to 3 replicas (4 total instances including primary), and are configured via the read_replica_count parameter during create or update.
* [Read vs Write Authentication Separation](/concepts/read-vs-write-authentication-separation.md) - Architectural pattern where read and write authentication for online stores use separate credentials, instance profiles, and secret scopes to enforce least-privilege access.
* [Read-Only Provider Data in Mounted Genie Spaces](/concepts/read-only-provider-data-in-mounted-genie-spaces.md) - The provider's tables in a mounted Genie Space are read-only — recipients can query them but cannot write to them.
* [Read/Write Secret Prefix Separation Pattern](/concepts/readwrite-secret-prefix-separation-pattern.md) - The pattern of using separate secret scopes (read-scope vs write-scope) and a shared prefix name to manage independent read-only and read-write credentials for the same online store.
* [Reading Databricks Tables via Databricks Connect](/concepts/reading-databricks-tables-via-databricks-connect.md) - The pattern of using spark.read.table() through Databricks Connect to query and display remote Databricks tables from a local Scala application.
* [Reading Databricks tables with Spark.read.table](/concepts/reading-databricks-tables-with-sparkreadtable.md) - The pattern of using SparkSession.read.table() via Databricks Connect to query existing tables on a Databricks cluster and display results.
* [Reading Delta Sharing Tables with the deltasharing Format Keyword](/concepts/reading-delta-sharing-tables-with-the-deltasharing-format-keyword.md) - Using .format('deltasharing') in Apache Spark DataFrames to read shared tables from an OpenSharing provider.
* [Reading Shared Tables and Streaming](/concepts/reading-shared-tables-and-streaming.md) - Technical operations for querying shared tables including batch reads, time-travel queries, change data feed (CDF), and Apache Spark Structured Streaming, with specific version and option requirements.
* [Reading tables with Databricks Connect](/concepts/reading-tables-with-databricks-connect.md) - Using spark.read.table() to query and display tables on a remote Databricks cluster from a local environment
* [Reasoning Models (Foundation Model API)](/concepts/reasoning-models-foundation-model-api.md) - Foundation models optimized for step-by-step reasoning tasks, accessed via Databricks' unified chat completions endpoint, with support for revealing internal thought processes before delivering final answers.
* [Reasoning Tokens](/concepts/reasoning-tokens.md) - Special tokens that let a model 'think' through a prompt internally via step-by-step reasoning before producing visible output tokens; some models expose these tokens to users while others discard them.
* [Reasoning-Only vs Hybrid Models](/concepts/reasoning-only-vs-hybrid-models.md) - Classification of reasoning models into reasoning-only (dedicated reasoning models) and hybrid (models that can toggle reasoning on/off), controlling how step-by-step thinking is enabled during inference.
* [Reasons for Collecting User Feedback in GenAI](/concepts/reasons-for-collecting-user-feedback-in-genai.md) - Five key motivations: real-world quality signals, continuous improvement, training data creation, quality monitoring, and model fine-tuning.
* [Recently Resolved Incidents](/concepts/recently-resolved-incidents.md) - A section in the Databricks data quality monitoring dashboard showing tables that auto-recovered from unhealthy to healthy status, helping distinguish transient issues from persistent problems.
* [Recipient (Delta Sharing)](/concepts/recipient-delta-sharing.md) - A named object in OpenSharing that represents the identity of a user or group of users who consume shared data, created with a specific authentication type.
* [Recipient and share concepts](/concepts/recipient-and-share-model.md) - Fundamental OpenSharing abstractions where a provider creates a 'recipient' to represent the data consumer and a 'share' representing the tables, volumes, and views to be shared.
* [Recipient and Share Model](/concepts/recipient-and-share-model.md) - The core abstraction in OpenSharing where a data provider creates a 'recipient' representing authorized users and a 'share' representing the tables, volumes, and views to be shared with that recipient.
* [Recipient Audit Log Events](/concepts/recipient-audit-logs.md) - Audit events recorded for data recipients in OpenSharing, covering share access attempts and management of provider objects.
* [Recipient Audit Logs](/concepts/recipient-audit-logs.md) - Audit logs recording events related to accessing shares and managing provider objects from the data consumer side, including recipient-specific error messages.
* [Recipient Creation for OIDC Federation](/concepts/recipient-creation-for-oidc-federation.md) - Step-by-step process in Databricks Catalog for creating an Open recipient with OIDC authentication method, configuring federation policy, and sharing the OIDC profile portal URL with the recipient.
* [Recipient Error Events in OpenSharing](/concepts/recipient-error-events-in-opensharing.md) - OpenSharing logs specific error messages for recipients, including permission denied on shares, missing shares, and missing tables within shares.
* [Recipient IdP App Registration for Delta Sharing](/concepts/recipient-idp-app-registration-for-delta-sharing.md) - The process of registering an OAuth application in a recipient's identity provider (e.g., Microsoft Entra ID) to enable OIDC-based authentication for Delta Sharing, including capturing issuer URL, subject claim, subject, and audience.
* [Recipient IP Access List Lifecycle](/concepts/recipient-ip-access-list-lifecycle.md) - The complete set of operations for managing IP access lists on recipients: assigning IPs/CIDRs at creation or to existing recipients, viewing the list, and removing entries, each with specific required permissions.
* [Recipient Lifecycle Management](/concepts/recipient-lifecycle-management.md) - Administration of OpenSharing recipients including viewing recipient lists (SHOW RECIPIENTS), viewing details (DESCRIBE RECIPIENT), viewing granted shares (SHOW GRANTS TO RECIPIENT), updating properties/owner/name/comments (ALTER RECIPIENT), and deleting recipients (DROP RECIPIENT), which invalidates tokens and revokes data access.
* [Recipient Object (Delta Sharing)](/concepts/recipient-object-delta-sharing.md) - A Unity Catalog object created by a data provider to represent a non-Databricks recipient, configured with authentication type TOKEN, and used to grant access to shared data shares.
* [Recipient Object (Unity Catalog)](/concepts/recipient-object-unity-catalog.md) - A Unity Catalog object that represents a data recipient in Open Sharing, configured with authentication type TOKEN, with an activation link and credential file generated upon creation.
* [Recipient Object in Unity Catalog](/concepts/recipient-object-unity-catalog.md) - A Unity Catalog object representing a data recipient in Delta Sharing, created with authentication type TOKEN for bearer token flows, which controls access to shared data shares.
* [Recipient Object with OIDC Federation](/concepts/recipient-object-with-oidc-federation.md) - A Databricks Unity Catalog recipient object configured for OIDC-based authentication, generated with a profile URL or downloadable profile file that contains no sensitive information.
* [Recipient Properties](/concepts/recipient-properties.md) - Custom key-value pairs (metadata) that can be attached to an OpenSharing recipient object for additional configuration or identification.
* [Recipient Properties (OpenSharing)](/concepts/recipient-properties.md) - Custom key-value metadata that can be attached to a recipient object in OpenSharing for organizing or annotating recipients.
* [Recipient Properties for Data Filtering](/concepts/recipient-properties-for-data-filtering.md) - Predefined (databricks.accountId, databricks.metastoreId, databricks.name) and custom properties (e.g., 'country') attached to recipient objects that enable partition filtering of shared tables and dynamic view row/column-level access control, allowing the same share to serve multiple recipients with different data boundaries.
* [Recipient Property-based Partition Filtering](/concepts/recipient-property-based-partition-filtering.md) - A technique that uses recipient properties (like databricks.accountId) to dynamically deliver only the relevant partition of a table to each recipient, enabling multi-tenant data sharing with a single share object.
* [Recipient Token Lifecycle Management](/concepts/recipient-token-lifecycle-management.md) - The process of creating, rotating, updating, and expiring bearer tokens for OpenSharing recipients, including token lifetime defaults, expiration policies, and mandatory rotation procedures.
* [Recipient Token Lifetime](/concepts/recipient-token-lifetime.md) - The configurable expiration period for bearer tokens assigned to open sharing recipients, with a maximum validity of one year from creation, configurable per recipient or at the metastore level.
* [Recipient Token Lifetime Configuration](/concepts/recipient-token-lifetime.md) - Configurable expiration settings for recipient bearer tokens, set at creation time or via metastore-level defaults, with a maximum validity of one year and the ability to update or rotate tokens.
* [Recipient Token Lifetime Management](/concepts/recipient-token-lifetime.md) - Configuration of expiration periods for Delta Sharing recipient bearer tokens, including per-recipient settings and metastore-wide defaults, with a maximum validity of one year.
* [Recipient Token Rotation](/concepts/recipient-token-rotation.md) - The process of expiring an existing recipient token and generating a new token and activation URL, used when tokens are about to expire, compromised, or when metastore token lifetime is modified.
* [Recipient Token Rotation (Delta Sharing)](/concepts/recipient-token-rotation.md) - The process of setting an existing bearer token to expire and replacing it with a new token and activation URL, used when tokens are compromised, about to expire, or when the metastore token lifetime changes.
* [Recipient Token Security Best Practices](/concepts/recipient-token-security-best-practices.md) - Security guidelines for managing bearer tokens including immediate expiration on compromise, secure channel communication, credential rotation, and responding to token exposure incidents.
* [Recipient-level SecureConnect toggle](/concepts/recipient-level-secureconnect-toggle.md) - Recipient owners and users with the USE_RECIPIENT privilege can toggle SecureConnect on or off for individual recipients, disabled by default unless the metastore setting is enabled.
* [Recipient-Local Views Pattern](/concepts/recipient-local-views-pattern.md) - A pattern where providers share only base tables (not views) and recipients create local views over the shared tables in a separate schema, allowing ABAC policies on the shared tables to be respected when accessed through those views.
* [Recipient-Side ABAC Enforcement for Shared Data](/concepts/recipient-side-abac-enforcement-for-shared-data.md) - Recipients of OpenSharing can apply their own ABAC policies (row filters, column masks) to shared tables to enforce access control on their side, independent of provider-side policies.
* [Recipient-side OpenSharing Audit Events](/concepts/recipient-side-opensharing-audit-events.md) - Audit events visible to data recipients in OpenSharing, focused on accessing shares and managing provider objects, with specific error messages for permission and existence failures.
* [Recipient-side resolution for Delta Sharing access errors](/concepts/provider-side-resolution-for-delta-sharing-access-errors.md) - Troubleshooting guidance where the data recipient must contact the data provider to resolve property misconfiguration issues.
* [Reconciliation queries in Databricks](/concepts/reconciliation-queries-in-databricks.md) - Queries that reconcile or validate external metadata against a source table, with restrictions on projections and aliasing.
* [Reconciliation Query Projections](/concepts/reconciliation-query-projections.md) - Query projections used in metadata reconciliation that have specific support limitations in Databricks
* [Recovery Window and UNDROP](/concepts/recovery-window-and-undrop.md) - A 7-day soft-delete window following deletion of a managed table or volume during which the object can be recovered using the UNDROP SQL command.
* [Redirect URL Destination](/concepts/redirect-url-destination.md) - A special destination type that redirects users to an external access request system, disabling the in-product request form and all other destinations for that object.
* [REDIRECT_READY_ALREADY_EXISTS](/concepts/redirect_ready_already_exists.md) - A sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED indicating the table already has a RedirectReady state, likely due to a concurrent ALTER TABLE SET MANAGED completing the migration first.
* [REDIRECT_READY_ALREADY_EXISTS error](/concepts/redirect_ready_already_exists.md) - A sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED indicating the table already has a RedirectReady state, possibly due to a concurrent migration command completing first.
* [REDIRECT_READY_ALREADY_EXISTS Sub-Error](/concepts/redirect_ready_already_exists.md) - Sub-error indicating another concurrent ALTER TABLE SET MANAGED command already migrated the table to managed state
* [RedirectReady state in Delta table migration](/concepts/redirectready-state-in-delta-table-migration.md) - An intermediate state during the ALTER TABLE SET MANAGED migration indicating the table is ready for redirect; duplicate RedirectReady state suggests concurrent migration already finished.
* [Refresh operation on Delta Uniform tables](/concepts/refresh-operation-on-delta-uniform-tables.md) - A metadata refresh operation for Delta Uniform tables, which may require specific arguments to function correctly
* [REFRESH SYNC UNIFORM](/concepts/refresh-sync-uniform.md) - A Databricks SQL identifier/command used to synchronize Uniform format metadata for Delta tables
* [REFRESH SYNC UNIFORM Command](/concepts/refresh-sync-uniform.md) - A Databricks SQL command to synchronize the UniForm (Iceberg/Hudi) metadata of a Delta table.
* [Regex masking performance on large text fields](/concepts/regex-masking-performance-on-large-text-fields.md) - Using regexp_replace inside column masks on serialized documents (XML/JSON stored as STRING) is expensive because it walks the full string per row; the recommended alternative is to materialize sensitive fields as typed scalar columns and mask those individually.
* [Regional metastore deployment](/concepts/regional-metastore-deployment.md) - The requirement that each Databricks region must have its own Unity Catalog metastore, which can be linked to multiple workspaces within that same region.
* [Regional Metastore Requirement](/concepts/regional-metastore-requirement.md) - Each AWS region in which an organization operates requires its own Unity Catalog metastore; workspaces can only link to a metastore in the same region.
* [Regional Model Availability (Databricks Foundation Model APIs)](/concepts/regional-model-availability-databricks-foundation-model-apis.md) - Restrictions on which foundation models are available in US-only vs. EU-and-US regions, with cross-Geo processing as an option for non-EU/non-US workspaces.
* [Regional Model Availability (US vs EU)](/concepts/regional-model-availability-us-vs-eu.md) - Certain foundation models have geographic restrictions: some models are US-only (e.g., Anthropic Claude Opus 4.1, Meta Llama 3.1 405B), while others are available in both EU and US regions (e.g., OpenAI GPT-5.x series, Anthropic Claude models).
* [Regional Model Availability Restrictions](/concepts/regional-model-availability-us-vs-eu.md) - Certain foundation models (e.g., Anthropic Claude Opus 4.1, Meta Llama 3.1 405B, BGE Large) are restricted to US regions only, while others are available in both EU and US regions.
* [Regional Tags](/concepts/regional-tags.md) - Classification tags that detect sensitive data specific to certain countries or regions, available only in Databricks workspaces deployed in the corresponding region.
* [Registering and Deploying AutoML Forecasting Models](/concepts/registering-and-deploying-automl-forecasting-models.md) - Process to register AutoML-trained forecasting models to Unity Catalog or Model Registry and deploy them to custom model serving endpoints on Databricks.
* [Registry URI Configuration in MLflow](/concepts/registry-uri-configuration-in-mlflow.md) - The mechanism of setting the MLflow registry URI to 'databricks' (workspace) or 'databricks-uc' (Unity Catalog) to target different model registries
* [Registry Webhook Events](/concepts/registry-webhook-events.md) - The set of MLflow Model Registry events that can trigger webhooks, including model version creation, stage transitions, transition requests, comments, tags, and registered model creation.
* [Regress](/concepts/regress.md) - The databricks.automl.regress method configures an AutoML run to train a regression model.
* [Relationship Between RPS, Latency, and Concurrency](/concepts/relationship-between-rps-latency-and-concurrency.md) - Throughput (requests per second) is inversely related to latency and directly related to concurrency; there is an optimal ratio of client-side to server-side concurrency that maximizes RPS while meeting latency targets.
* [Relevance Scorer](/concepts/relevance-scorer.md) - An Arize Phoenix scorer that measures how relevant an LLM's output is to the given query and context
* [Relevance scorer (Phoenix)](/concepts/relevance-scorer.md) - An Arize Phoenix scorer used within MLflow GenAI evaluation that measures how relevant an LLM's response is to the given query and context.
* [RelevanceToQuery](/concepts/relevancetoquery.md) - An MLflow LLM judge that evaluates whether a GenAI app's response directly addresses the user's input without deviating into unrelated topics
* [RelevanceToQuery Judge](/concepts/relevancetoquery.md) - An MLflow built-in LLM judge that evaluates whether a GenAI application's response directly addresses the user's input question without deviating into unrelated topics.
* [Remote Code Execution Model](/concepts/remote-code-execution-model.md) - The execution model in Databricks Connect where general Python/Scala code runs locally on the client, DataFrame APIs execute on remote Databricks compute, and UDFs are serialized and run on the cluster.
* [Remote connection in Databricks Connect](/concepts/remote-connection-in-databricks-connect.md) - Using the remote() method to connect to Databricks compute that is not attached to the current notebook or job, allowing connections to separate clusters via configuration kwargs or methods like host() and token().
* [Remote Connections from Databricks Notebooks](/concepts/remote-connections-from-databricks-notebooks.md) - Databricks Connect's remote() function allows notebooks to connect to Databricks compute not attached to the notebook or job, using configuration kwargs or methods like host() and token() to create a new session.
* [Remote Databricks compute connection via remote()](/concepts/remote-databricks-compute-connection-via-remote.md) - DatabricksSession.builder.remote() allows connecting to Databricks compute that is not attached to the current notebook or job, using keyword arguments or configuration methods like host(), token(), etc.
* [Remote DataFrame and table operations](/concepts/remote-dataframe-and-table-operations.md) - Patterns for creating, reading, writing, and querying DataFrames and tables on a remote Databricks cluster using Databricks Connect
* [Remote DataFrame Execution Model](/concepts/remote-dataframe-execution-model.md) - The execution model where DataFrame transformations run on remote Databricks compute while general Python/Scala code runs locally, with results materialized via commands like collect(), show(), and toPandas().
* [Remote DataFrame Operations](/concepts/remote-dataframe-operations.md) - The ability to create, manipulate, and persist DataFrames on a remote Databricks cluster using local Python code through Databricks Connect
* [Remote DataFrame Operations with Databricks Connect](/concepts/remote-dataframe-operations-with-databricks-connect.md) - Using Databricks Connect to read tables, create DataFrames, write to tables, and run SQL queries on a remote Databricks cluster from a local environment.
* [Remote DataFrame Testing Pattern](/concepts/remote-dataframe-testing-pattern.md) - A testing pattern where PySpark DataFrame operations (like count, show) execute on remote Databricks compute, and tests assert on results using standard pytest assertions.
* [Remote MLflow tracking server](/concepts/remote-mlflow-tracking-server.md) - Setting up a connection to a remote MLflow tracking server to log experiments from local development or across different Databricks workspaces.
* [Remote MLflow Tracking Server Configuration](/concepts/remote-mlflow-tracking-server-setup.md) - Process of setting up MLflow to log experiments to a remote tracking server by configuring both the tracking URI and experiment path.
* [Remote MLflow tracking server setup](/concepts/remote-mlflow-tracking-server-setup.md) - Configuration required to connect to a remote MLflow tracking server, including setting the tracking URI and experiment path to log experiments across different environments.
* [Remote Model Registry with Feature Store](/concepts/remote-model-registry-with-feature-store.md) - Ability to specify a remote model registry URI alongside a remote feature store URI, enabling training models using local or remote feature tables and registering them in local or remote model registries.
* [renv Environment Management for Databricks Connect](/concepts/renv-environment-management-for-databricks-connect.md) - Using the renv R package to create reproducible R environments when setting up Databricks Connect projects in RStudio.
* [renv for Databricks Connect projects](/concepts/renv-for-databricks-connect-projects.md) - Using renv (R package for reproducible environments) to manage project-specific R package dependencies when developing with Databricks Connect in RStudio.
* [renv Package Integration with Databricks Connect](/concepts/renv-package-integration-with-databricks-connect.md) - The use of the renv R package for creating reproducible R project environments when working with Databricks Connect, ensuring consistent package dependencies across sessions.
* [REORG for Upgrading Iceberg Read Support](/concepts/reorg-for-upgrading-iceberg-read-support.md) - A SQL command (REORG TABLE ... APPLY UPGRADE UNIFORM) used to enable Iceberg reads on tables with deletion vectors, upgrade from IcebergCompatV1, or rewrite data for compatibility with Hive-style Parquet readers.
* [REORG TABLE](/concepts/reorg-table.md) - A Databricks SQL command to reorganize Delta Lake tables by rewriting files to purge soft-deleted data, upgrade to Iceberg format, or perform checkpointing.
* [REORG TABLE APPLY (PURGE)](/concepts/reorg-table-apply-purge.md) - A Delta Lake SQL command to purge deletion vectors from a table, enabling compatibility with symlink manifest generation.
* [REORG TABLE APPLY (PURGE) Command](/concepts/reorg-table-apply-purge.md) - A Delta Lake command used to rewrite a table to remove deletion vectors, enabling operations like symlink manifest generation that are incompatible with deletion vectors.
* [REORG TABLE APPLY (UPGRADE UNIFORM)](/concepts/reorg-table-apply-upgrade-uniform.md) - A Databricks SQL command used to upgrade a Delta table to a specific IcebergCompatV version, rewrite data for Iceberg compatibility, and resolve several DELTA_ICEBERG_COMPAT_VIOLATION errors.
* [REORG TABLE APPLY command](/concepts/reorg-table.md) - A Databricks SQL command used to resolve IcebergCompatV violations by rewriting, purging deletion vectors, or upgrading the Uniform format version on Delta tables.
* [REORG TABLE command](/concepts/reorg-table.md) - A Databricks SQL command used to resolve IcebergCompat violations by rewriting data, upgrading uniform format, purging deletion vectors, or enabling IcebergCompat versions.
* [REORG TABLE for Iceberg compat upgrades](/concepts/reorg-table-for-iceberg-compat-upgrades.md) - The command pattern REORG TABLE APPLY (UPGRADE UNIFORM ('ICEBERG_COMPAT_VERSION = <version>')) used to rewrite a Delta table and enable a new IcebergCompat version, including PURGE sub-commands for deletion vectors.
* [REPLACE ON Complex-Match Overwrite](/concepts/replace-on-complex-match-overwrite.md) - Delta Lake overwrite option for user-defined matching conditions including NULL-safe equality, used when REPLACE USING's equality semantics are insufficient.
* [REPLACE USING Dynamic Data Overwrite](/concepts/replace-using-dynamic-data-overwrite.md) - Compute-independent, atomic overwrite that replaces rows when specified columns compare equal under equality, supporting partitioned, unpartitioned, and liquid-clustered tables.
* [REPLACE WHERE Selective Overwrite](/concepts/replace-where-selective-overwrite.md) - Delta Lake option to atomically replace data matching an arbitrary boolean expression, with optional constraint validation to ensure all written rows fall within the predicate.
* [REPLACE_TABLE_CHANGE_PARTITION_NAMES](/concepts/replace_table_change_partition_names.md) - Error sub-type indicating IcebergCompatV1 does not support replacing partitioned tables with a differently-named partition spec
* [REPLACE_TABLE_CHANGE_PARTITION_NAMES (Delta Iceberg error)](/concepts/replace_table_change_partition_names.md) - A sub-error of DELTA_ICEBERG_COMPAT_V1_VIOLATION that occurs when replacing a partitioned table with a differently-named partition spec, unsupported by IcebergCompatV1 due to Iceberg-Spark 1.1.0 limitations
* [REPLACE_TABLE_CHANGE_PARTITION_NAMES error](/concepts/replace_table_change_partition_names.md) - A sub-error of DELTA_ICEBERG_COMPAT_V1_VIOLATION triggered when replacing a partitioned table with a differently-named partition spec, mirroring a limitation in Iceberg-Spark 1.1.0.
* [Replacing spark.catalog API Calls for Unity Catalog](/concepts/replacing-sparkcatalog-api-calls-for-unity-catalog.md) - Workarounds for spark.catalog.tableExists, listTables, and setDefaultCatalog using SQL equivalents when running on shared clusters with Unity Catalog.
* [Representative Payload Design for Load Testing](/concepts/representative-payload-design-for-load-testing.md) - The principle that load test payloads must accurately reflect real production traffic characteristics, such as request size and data structure, to produce valid performance measurements. Example: a fraud-detection endpoint should send one transaction per request, not batches.
* [Representative Payload Design for Load Tests](/concepts/representative-payload-design-for-load-tests.md) - Ensuring load test validity by using payloads that accurately represent production traffic, matching the size and structure of real requests such as a single credit card transaction for fraud detection models.
* [Representative Payload Selection for Load Testing](/concepts/representative-payload-selection-for-load-testing.md) - The practice of choosing input payloads that accurately mirror real production traffic patterns to ensure valid and meaningful load test results.
* [RequestSource](/concepts/requestsource.md) - A data source that defines a schema for inference-time request payload data, used with ColumnSelection for pass-through features without pre-materialized tables
* [RequestSource and Inference-Time Features](/concepts/requestsource-and-inference-time-features.md) - A data source type for features provided at inference time via HTTP request payload (or extracted from labeled DataFrame during training); uses FieldDefinition with ScalarDataType and is combined with ColumnSelection only.
* [RequestSource Features](/concepts/requestsource-features.md) - Features passed directly in the request payload at inference time rather than looked up from an online store; these columns are added to the MLflow model signature during log_model.
* [Required table features and properties for Iceberg compat](/concepts/required-table-features-and-properties-for-iceberg-compat.md) - IcebergCompatV requires specific table features to be enabled and specific table properties to be set to particular values; disabling required features or misconfiguring properties triggers violation errors.
* [Requirements for Express Deployments](/concepts/requirements-for-express-deployments.md) - Specific conditions that must be met for express deployments: custom models only, serverless notebook registration, mlflow>=3.1, UC registration, CPU serving, and max 1GB environment size.
* [REQUIRES_WITH_HISTORY](/concepts/requires_with_history.md) - Sub-error indicating that the WITH HISTORY clause is required when performing a DEEP CLONE on a streaming table.
* [REQUIRES_WITH_HISTORY sub-error](/concepts/requires_with_history.md) - Sub-error stating that WITH HISTORY clause is mandatory when performing a deep clone of a streaming table.
* [Resource and Environment Management for Deep Learning](/concepts/resource-and-environment-management-for-deep-learning.md) - Techniques for customizing deep learning environments on Databricks at notebook, cluster, and job levels, including cluster policies to guide data scientists.
* [Resource Key Customization](/concepts/resource-key-customization.md) - The ability to specify a custom resource key when adding a resource to a Databricks App, which controls how the resource is referenced in app configuration and environment variable injection.
* [Resource Management in Combined Ray-Spark Environments](/concepts/resource-management-in-combined-ray-spark-environments.md) - When combining Ray and Spark, resource conflicts can be managed by configuring CPU/memory/GPU allocation, e.g., using setup_ray_cluster to split resources between the two frameworks.
* [Response-Only Training](/concepts/response-only-training.md) - A training technique that masks out user/instruction tokens during supervised fine-tuning so the model only learns from assistant responses, improving training efficiency.
* [Response-Only Training with Chat Templates](/concepts/response-only-training-with-chat-templates.md) - A training strategy that masks or ignores user/prompt tokens during supervised fine-tuning, so the model only learns to predict assistant responses, improving training efficiency and focus.
* [Responses API](/concepts/responses-api.md) - An OpenAI-compatible API on Databricks for multi-turn conversations using 'input' instead of 'messages', supporting custom tools, function calls, and grammar-based outputs for GPT-5 series models.
* [Responses API (Databricks)](/concepts/responses-api.md) - An OpenAI-compatible multi-turn conversation API that uses 'input' instead of 'messages', supporting custom tools, grammar-based outputs, and reasoning configs.
* [Responses API Limitations on Pay-Per-Token Models](/concepts/responses-api-limitations-on-pay-per-token-models.md) - Restrictions on pay-per-token foundation models when using the Responses API, including unsupported parameters (background, store, previous_response_id, service_tier) and a specific set of supported tool types.
* [REST API for Model Serving](/concepts/rest-api-for-model-serving.md) - Direct HTTP requests to Databricks model serving endpoints for querying chat models with structured JSON payloads.
* [RESTORE Operation Metrics](/concepts/restore-operation-metrics.md) - Output columns returned by a RESTORE operation showing table size, file counts, and files changed during the restore
* [RESTORE TABLE Command](/concepts/restore-table-command.md) - Delta Lake SQL command to restore a table to a previous state using timestamp or version
* [RESTORE TABLE TO TIMESTAMP AS OF](/concepts/restore-table-to-timestamp-as-of.md) - Restoring a Delta table to the state it was in at a specific point in time
* [RESTORE TABLE TO VERSION AS OF](/concepts/restore-table-to-version-as-of.md) - Restoring a Delta table to a specific version number obtained from DESCRIBE HISTORY
* [Restricted dbutils in Databricks Connect](/concepts/restricted-dbutils-in-databricks-connect.md) - Only 'fs' (file system) and 'secrets' Databricks Utilities are available through dbutils in Databricks Connect for Python; all other utilities are unavailable.
* [Restricted SQL Function Allowlist](/concepts/shared-view-function-allowlist.md) - A security mechanism in Databricks-to-Databricks view sharing that limits shared views to only use a predefined subset of built-in SQL functions and operators, preventing arbitrary code execution.
* [Restricting Workspace Admin Privileges](/concepts/restricting-workspace-admin-privileges.md) - Account admins can restrict workspace admin privileges using the RestrictWorkspaceAdmins setting to limit the scope of workspace-level administrative capabilities.
* [RestrictWorkspaceAdmins Setting](/concepts/restrictworkspaceadmins-setting.md) - A setting that account admins can use to restrict workspace admin privileges in Unity Catalog.
* [Retired Model Discovery Query](/concepts/retired-model-discovery-query.md) - A SQL query against Databricks system tables (system.serving) to find workloads using deprecated models and identify the owners for migration planning.
* [Retired Model Discovery via System Tables](/concepts/retired-model-discovery-via-system-tables.md) - SQL query pattern for identifying workloads using retired models by querying Databricks system tables endpoint_usage and served_entities.
* [Retrieval Augmented Generation (RAG)](/concepts/retrieval-augmented-generation-rag.md) - A technique recommended across all supported models to improve factual accuracy by retrieving relevant context from external sources and injecting it into the model prompt.
* [Retrieval Augmented Generation (RAG) on Databricks](/concepts/retrieval-augmented-generation-rag-on-databricks.md) - A recommended pattern for improving LLM accuracy by retrieving relevant context from documents before generating responses, consistently advised across all Databricks-hosted foundation models.
* [Retrieval Augmented Generation (RAG) Recommendation](/concepts/retrieval-augmented-generation-rag-recommendation.md) - Databricks recommends using retrieval augmented generation (RAG) in scenarios where accuracy is especially important to mitigate the risk of LLMs omitting facts or producing false information.
* [Retrieval Augmented Generation (RAG) with Foundation Models](/concepts/retrieval-augmented-generation-rag-with-foundation-models.md) - Databricks recommends using retrieval augmented generation (RAG) alongside foundation models to improve factual accuracy and reduce hallucination risks in scenarios where correctness is critical.
* [Retrieval Quality Judges](/concepts/retrieval-quality-judges.md) - A family of built-in LLM judges (RetrievalRelevance, RetrievalGroundedness, RetrievalSufficiency) that evaluate the quality of retrieved context in RAG applications.
* [Retrieval-based evaluation judges](/concepts/retrieval-focused-evaluation.md) - A family of built-in LLM judges (RetrievalRelevance, RetrievalGroundedness, RetrievalSufficiency) focused specifically on evaluating the quality of retrieval-augmented generation (RAG) systems.
* [Retrieval-Focused Evaluation](/concepts/retrieval-focused-evaluation.md) - A family of built-in judges (RetrievalRelevance, RetrievalGroundedness, RetrievalSufficiency) that specifically evaluate the quality of retrieved context in RAG-based applications.
* [RetrievalGroundedness judge](/concepts/retrievalgroundedness-judge.md) - An MLflow GenAI judge that evaluates whether a generated response is factually grounded in the retrieved context documents, detecting hallucination in RAG applications.
* [RetrievalRelevance](/concepts/retrievalrelevance.md) - An MLflow LLM judge that evaluates whether each document returned by a retriever is relevant to the user's input request, requiring RETRIEVER spans in traces
* [RetrievalRelevance Judge](/concepts/retrievalrelevance.md) - An MLflow built-in LLM judge that evaluates whether each document returned by a retriever in a RAG application is relevant to the user's input request.
* [RetrievalSufficiency Judge](/concepts/retrievalsufficiency-judge.md) - An MLflow GenAI scorer that evaluates whether retrieved documents contain enough context to sufficiently answer a user query.
* [RETRIEVER Spans](/concepts/retriever-spans.md) - Specialized MLflow span type for document retrieval operations that must output a list of Document objects with page_content, metadata, and optional ID fields.
* [Retrieving Labeling Session Feedback Responses](/concepts/retrieving-labeling-session-feedback-responses.md) - The mechanism for accessing reviewer assessments after a labeling session completes, available via the MLflow Experiments UI or programmatically through the MLflow API.
* [Retry Logic with Exponential Backoff](/concepts/retry-logic-with-exponential-backoff.md) - Best practice for retrying requests after rate limit errors using exponential backoff with jitter to handle 429 errors gracefully.
* [Retry with exponential backoff for rate limits](/concepts/retry-with-exponential-backoff-for-rate-limits.md) - Best practice pattern for handling 429 rate limit errors by retrying with exponential backoff, optionally adding jitter to avoid thundering herd problems.
* [Returning Complex Structured Types from Pandas UDFs](/concepts/returning-complex-structured-types-from-pandas-udfs.md) - Using array<struct<...>> return type annotations in Pandas UDFs to represent complex NLP pipeline outputs like entity recognition results
* [Reusable TraceAnalyzer class](/concepts/reusable-traceanalyzer-class.md) - Design pattern for a utility class that encapsulates advanced trace analysis including error summary, LLM usage aggregation, retrieval metrics, span hierarchy, and evaluation data export.
* [Reusing Stored Traces for Scorer Iteration](/concepts/reusing-stored-traces-for-scorer-iteration.md) - Technique of passing a Pandas DataFrame of precomputed traces directly to mlflow.genai.evaluate() as the input dataset, allowing scorer iteration without re-running the original application.
* [Review App Content Rendering](/concepts/review-app-content-rendering.md) - The automatic rendering of different trace content types in the Review App, including retrieved documents from RETRIEVER spans, OpenAI format chat messages, and dicts as pretty-printed JSON.
* [Review App for Human Feedback](/concepts/review-app-for-human-feedback.md) - Integrated application for collecting domain expert feedback on AI agent outputs, producing evaluation data for further iteration.
* [Review App Integration with Labeling Schemas](/concepts/review-app-integration-with-labeling-schemas.md) - Labeling schemas are associated with labeling sessions in the Review App; they control the question shown, input method, validation rules, and optional instructions, and are not used for vibe checks in the Chat UI.
* [Review App Trace Content Rendering](/concepts/review-app-content-rendering.md) - The automatic rendering behavior of the MLflow Review App for different trace content types, including RETRIEVER span documents, OpenAI-format messages, and dictionary inputs/outputs.
* [Review Apps for Expert Feedback](/concepts/review-apps-for-expert-feedback.md) - Feature in MLflow 3 for collecting expert-labeled datasets to align automated judges and scorers with human judgement
* [Revoking Share Access from Recipients](/concepts/revoking-share-access-from-recipients.md) - The process of removing a recipient's access to an OpenSharing share using Catalog Explorer, SQL, or CLI.
* [Risks of Direct File Deletion in Delta Lake on S3](/concepts/risks-of-direct-file-deletion-in-delta-lake-on-s3.md) - Using shell commands like rm -rf to drop Delta Lake tables on S3 can cause stale data and corruption; Delta Lake tables should be dropped using proper table operations.
* [Risks of using rm -rf to drop Delta Lake tables](/concepts/risks-of-using-rm-rf-to-drop-delta-lake-tables.md) - Using rm -rf to delete a Delta Lake table on S3 can lead to stale data issues; Databricks recommends using proper drop table operations instead.
* [Rolling Window Backtesting](/concepts/rolling-window-backtesting.md) - A time series evaluation technique using DateSplitter to create multiple train/test windows for robust model validation.
* [Rolling Window Backtesting for Time Series](/concepts/rolling-window-backtesting.md) - A model evaluation technique that creates multiple train/test splits using rolling windows over a time series, allowing assessment of forecast performance across different time periods.
* [Route Optimization (Databricks Model Serving)](/concepts/route-optimization-for-model-serving.md) - A feature that improves network path for inference requests on Databricks serving endpoints, enabling higher QPS and lower latency.
* [Route Optimization for Model Serving](/concepts/route-optimization-for-model-serving.md) - A Databricks-specific optimization that improves the network path for inference requests, resulting in faster, more direct communication between clients and models. Supports workloads above 200 QPS and sub-50ms overhead.
* [Route Optimization for Serving Endpoints](/concepts/route-optimization-for-serving-endpoints.md) - A performance feature recommended for endpoints with high QPS and throughput requirements that optimizes request routing.
* [Route Optimized Endpoints](/concepts/route-optimized-endpoints.md) - A model serving endpoint type that removes the workspace-level parallel requests limit (429 errors), recommended as a solution for high-concurrency workloads.
* [Route-Optimized Endpoint Configuration for Load Testing](/concepts/route-optimized-endpoint-configuration-for-load-testing.md) - Prerequisite configuration for Databricks Model Serving endpoints under test: start with a Small CPU endpoint (min/max concurrency=4), enable route optimization, and disable scale-to-zero to ensure consistent baseline measurements.
* [Route-optimized endpoint URL](/concepts/route-optimized-endpoint-url.md) - The dedicated URL format for route-optimized endpoints — https://<unique-id>.serving.cloud.databricks.com/<workspace-id>/serving-endpoints/<endpoint-name>/invocations — which provides the benefits of route optimization.
* [Route-Optimized Model Serving Endpoints](/concepts/route-optimized-model-serving-endpoints.md) - A Databricks Model Serving feature that enables OAuth-based authentication and requires specific endpoint configuration (fixed concurrency, disabled scale-to-zero) for effective load testing.
* [Route-optimized serving endpoints](/concepts/route-optimized-serving-endpoints.md) - Serving endpoints on Databricks with route optimization enabled for improved query performance, requiring dedicated URLs and OAuth authentication.
* [Row Filter and Column Mask Policies](/concepts/row-filter-policies.md) - ABAC-supported policy types for row and column-level security on tables, materialized views, and streaming tables in Unity Catalog.
* [Row Filter and Column Mask UDFs](/concepts/row-filter-and-column-mask-udfs.md) - User-defined functions used by row filter and column mask policies to implement filtering or masking logic, with SQL UDFs recommended for performance over Python UDFs.
* [Row Filter Policies](/concepts/row-filter-policies.md) - ABAC policy type that enforces row-level security on tables, materialized views, and streaming tables by filtering rows based on attributes
* [Row filter policies and column mask policies](/concepts/row-filter-policies.md) - ABAC policy types providing row-level and column-level security on tables, materialized views, and streaming tables in Unity Catalog.
* [Row filter policies in Unity Catalog](/concepts/row-filter-policies.md) - ABAC policy type that supports row-level security on tables, materialized views, and streaming tables.
* [Row filtering with column-only predicates](/concepts/row-filtering-with-column-only-predicates.md) - Filtering rows using simple boolean logic that references only table columns, enabling predicate pushdown for query optimization, including patterns for matching multiple country/region columns with a single policy.
* [Row filtering with region-based policies](/concepts/row-filtering-with-region-based-policies.md) - Applying row-level filters via Unity Catalog policies that restrict data visibility based on geographic region, using UDFs that compare a tagged column value against an allowed region constant.
* [Row filters (Unity Catalog)](/concepts/row-filters-unity-catalog.md) - SQL UDFs applied to tables that restrict which rows a user can see at query time based on per-row evaluation.
* [Row Filters and Column Masks](/concepts/row-filters-and-column-masks.md) - Per-table UDF-based row and column level filters that control what data users see at query time in Unity Catalog.
* [Row Filters in Unity Catalog](/concepts/row-filters-in-unity-catalog.md) - A row-level security feature in Databricks Unity Catalog that uses SQL UDFs to filter rows returned from a table based on user attributes or group membership.
* [Row Level Security in Delta Lake](/concepts/row-level-security-in-delta-lake.md) - A Delta Lake table feature for filtering rows based on user identity, which is incompatible with REFRESH SYNC UNIFORM.
* [Row-Level Conflict Detection in Delta Lake](/concepts/row-level-conflict-detection-in-delta-lake.md) - A mechanism in Delta Lake that detects conflicts at the row level when concurrent transactions modify the same data, with specific sub-conditions and limitations.
* [Row-Level Conflict Resolution](/concepts/row-level-conflict-resolution.md) - A mechanism in Delta Lake that detects and resolves conflicts at the individual row level during concurrent transactions
* [Row-Level Conflict Resolution in Delta Lake](/concepts/row-level-conflict-resolution-in-delta-lake.md) - A mechanism to resolve concurrent write conflicts at the row level rather than aborting the entire transaction, with specific failure modes like exceeded time or incompatible table features.
* [Row-Level Security (RLS) Policies](/concepts/row-level-security-rls-policies.md) - Databricks security feature for filtering rows that is incompatible with external metadata sources
* [Row-level Security (RLS) policies in Delta Lake](/concepts/row-level-security-rls-policies-in-delta-lake.md) - A Delta table security feature that filters rows at query time per user permissions, unsupported by external metadata sources.
* [Row-level security with Delta Uniform](/concepts/row-level-security-with-delta-uniform.md) - Row-level security features on Delta tables prevent REFRESH SYNC UNIFORM from succeeding
* [RStudio debugging with Databricks Connect](/concepts/rstudio-debugging-with-databricks-connect.md) - Databricks Connect enables interactive debugging of Spark code in RStudio Desktop, including setting breakpoints, inspecting variables in the Environment view, and stepping through code execution.
* [Rule-based LLM Evaluation without LLM Calls](/concepts/rule-based-llm-evaluation-without-llm-calls.md) - Guardrails AI scorers perform rule-based validation of LLM outputs without requiring additional LLM inference calls, offering a lightweight evaluation approach.
* [Run Evaluation From Dataset](/concepts/run-evaluation-from-dataset.md) - The workflow to trigger mlflow.genai.evaluate() from a stored evaluation dataset, generating a Python code snippet from the UI.
* [Run management in AI Runtime CLI](/concepts/run-management-in-ai-runtime-cli.md) - Commands for listing, inspecting status, and cancelling runs using air list runs and air cancel commands, with filtering options like --active and --limit.
* [Run Management in MLflow UI](/concepts/run-management-in-mlflow-ui.md) - Features for managing MLflow runs including selecting runs to display, deleting, comparing, adding/editing tags, filtering by parameter/metric values or tags, sorting by parameters, and grouping by parameter values.
* [Run Navigation and Log Retrieval Commands](/concepts/run-navigation-and-log-retrieval-commands.md) - The `air get run`, `air list runs`, and `air logs` commands provide navigation between Jobs, MLflow, and previous workloads, as well as streaming/downloading run logs.
* [Running Evaluations from Datasets](/concepts/run-evaluation-from-dataset.md) - The workflow and UI integration for triggering mlflow.genai.evaluate() directly from a saved evaluation dataset, including auto-generated Python code snippets.
* [Running MLflow Projects on Databricks](/concepts/running-mlflow-projects-on-databricks.md) - The process of executing MLflow Projects remotely on Databricks clusters using the MLflow CLI with 'mlflow run <uri> -b databricks --backend-config <cluster-spec>' and Databricks CLI authentication.
* [Runtime Requirements for Feature Store AutoML](/concepts/runtime-requirements-for-feature-store-automl.md) - Classification and regression experiments with feature store require Databricks Runtime 11.3 LTS ML+, forecasting requires 12.2 LTS ML+.
* [S3 Bucket Naming Restrictions for Unity Catalog](/concepts/s3-bucket-naming-restrictions-for-unity-catalog.md) - The requirement that S3 buckets used with Unity Catalog must not use dot notation in their names due to compatibility issues with features like OpenSharing.
* [S3 Bucket Versioning and Delta Lake](/concepts/s3-bucket-versioning-and-delta-lake.md) - Databricks recommends against enabling S3 bucket versioning for Delta Lake tables because Delta Lake implements its own versioning and garbage collection, and S3 versioning can cause performance degradation and storage bloat.
* [S3 Commit Service for Delta Lake](/concepts/s3-commit-service-for-delta-lake.md) - Databricks provides an S3 commit service to help manage multi-cluster writes and consistency for Delta Lake tables stored on S3.
* [S3 eventual consistency and Delta Lake](/concepts/s3-eventual-consistency-and-delta-lake.md) - The phenomenon where AWS S3's eventual consistency model can cause non-contiguous Delta log versions when a table is deleted and recreated at the same location.
* [S3 eventual consistency impact on Delta Lake](/concepts/s3-eventual-consistency-impact-on-delta-lake.md) - Amazon S3's eventually consistent model can cause data corruption or loss when multiple clusters or systems concurrently modify the same Delta Lake table stored on S3.
* [S3 Eventual Consistency Impact on Delta Tables](/concepts/s3-eventual-consistency-and-delta-lake.md) - A cause specific to AWS S3 where eventual consistency can lead to a non-contiguous delta log when a table is deleted and recreated at the same location.
* [S3 Eventually Consistent Model](/concepts/s3-eventually-consistent-model.md) - Amazon S3's eventual consistency model can cause data corruption and data loss when multiple systems or clusters modify the same Delta Lake table stored on S3 simultaneously.
* [Safe Span Management with Context Managers](/concepts/safe-span-management-with-context-managers.md) - Pattern for using Python context managers (contextlib.contextmanager) to ensure spans are always properly closed, even when exceptions occur.
* [Safety Judge (MLflow)](/concepts/safety-judge-mlflow.md) - A built-in MLflow judge that evaluates text content to identify potentially harmful, offensive, or inappropriate material, returning a pass/fail assessment with detailed rationale.
* [Safety Judge Scorer](/concepts/safety-judge-scorer.md) - A pre-built safety evaluation scorer in Databricks that can be registered, started, and configured with sampling to assess content safety of model outputs.
* [Safety Scorer (MLflow GenAI)](/concepts/safety-scorer-in-mlflow.md) - Pre-built safety evaluation judge from mlflow.genai.scorers that can be registered and configured with sampling rates to assess safety of model outputs.
* [Safety Scorer in MLflow](/concepts/safety-scorer-in-mlflow.md) - A built-in MLflow GenAI scorer that evaluates content safety, registered and started with a sampling configuration.
* [Safety Scorer in MLflow GenAI](/concepts/safety-scorer-in-mlflow.md) - A built-in MLflow GenAI scorer that evaluates the safety of model outputs, registered and started with optional sampling configuration before use.
* [Sampling Large Datasets for AutoML](/concepts/sampling-large-datasets-for-automl.md) - Automatic memory-based sampling of large datasets in AutoML, using stratified sampling for classification (sampleBy) and simple sampling for regression (sample).
* [Sampling Large Datasets in AutoML](/concepts/sampling-large-datasets-in-automl.md) - How AutoML handles large datasets by sampling, using stratified sampling for classification and simple sampling for regression to fit within single-node memory limits.
* [Sampling-based restoration detection](/concepts/sampling-based-restoration-detection.md) - Databricks samples files presumed to be archived to detect if they have been restored, and if so, assumes all files are restored for that query.
* [Sampling-based restored data detection](/concepts/sampling-based-restoration-detection.md) - Databricks samples files presumed archived to detect whether they have been restored; if sampling indicates restoration, it assumes all files are available.
* [SAP BDC as Authoritative Metadata Source](/concepts/sap-bdc-as-authoritative-metadata-source.md) - SAP Business Data Cloud is the source of truth for semantic metadata; metadata synced into Databricks is read-only and cannot be directly modified by users or OpenSharing recipients.
* [SAP BDC Connection](/concepts/sap-bdc-connection.md) - A named connection object that links a Databricks workspace to an SAP Business Data Cloud account for OpenSharing, enabling bidirectional data sharing.
* [SAP BDC Connection (Databricks)](/concepts/sap-bdc-connection.md) - A bidirectional connection between Databricks and SAP Business Data Cloud that enables Delta Sharing via the OpenSharing protocol, with automatic creation of provider and recipient objects.
* [SAP BDC Connection Deletion Prerequisites](/concepts/sap-bdc-connection-deletion-prerequisites.md) - Before deleting an SAP BDC connection, all mounted shares received from SAP BDC must be unmounted from Unity Catalog catalogs, or deletion is blocked.
* [SAP BDC Connection Deletion Workflow](/concepts/sap-bdc-connection-deletion-workflow.md) - A two-step process requiring first unmounting all shares from Unity Catalog catalogs, then deleting the connection; after deletion, both Databricks loses access to SAP data products and SAP loses access to shared assets.
* [SAP BDC Connection Lifecycle](/concepts/sap-bdc-connection-lifecycle.md) - The process of creating, updating ownership, and deleting an SAP BDC connection on Databricks including prerequisites and cleanup steps
* [SAP BDC Connection Lifecycle Management](/concepts/sap-bdc-connection-lifecycle.md) - The process of creating, transferring ownership, and deleting SAP BDC connections on Databricks, including prerequisite cleanup steps for deletion.
* [SAP BDC Connection Limitations](/concepts/sap-bdc-connection-limitations.md) - A hard limit of five SAP BDC connections per Databricks metastore, enforced at the metastore level.
* [SAP BDC Connection on Databricks](/concepts/sap-bdc-connection.md) - A configuration enabling bidirectional data sharing between SAP Business Data Cloud and Databricks via OpenSharing
* [SAP BDC Connection Ownership](/concepts/sap-bdc-connection-ownership.md) - The ownership model for SAP BDC connections, where the creator becomes owner, ownership can only be transferred to individual users (not groups or service principals), and the owner is required to delete connections.
* [SAP BDC Connector](/concepts/sap-bdc-connector.md) - The connection infrastructure between a Databricks workspace and an SAP Business Data Cloud account required before sharing data.
* [SAP BDC Metadata Consumption Interfaces](/concepts/sap-bdc-metadata-consumption-interfaces.md) - Multiple Databricks interfaces for consuming synced SAP BDC metadata: Catalog Explorer, SQL (DESCRIBE TABLE, INFORMATION_SCHEMA), Genie Spaces natural language queries, ABAC governance policies, and audit logs.
* [SAP BDC OpenSharing Connection](/concepts/sap-bdc-opensharing-connection.md) - A bidirectional Delta Sharing connection between Databricks and SAP Business Data Cloud that enables data exchange via the OpenSharing protocol.
* [SAP BDC OpenSharing Recipients](/concepts/sap-bdc-opensharing-recipients.md) - Recipient entities automatically created in Databricks when an SAP BDC connection is established, used to manage share access
* [SAP BDC Provider and Recipient Objects](/concepts/sap-bdc-provider-and-recipient-objects.md) - Two automatically created Unity Catalog objects (provider and recipient) that represent the SAP BDC account for sharing and receiving data.
* [SAP BDC Provider Object](/concepts/sap-bdc-provider-object.md) - An object automatically created upon SAP BDC connection establishment that represents the SAP BDC account as a data provider within Unity Catalog.
* [SAP BDC Recipient](/concepts/sap-bdc-recipient.md) - A Delta Sharing recipient automatically created when a Databricks-to-SAP BDC connection is established, used to grant data share access.
* [SAP BDC Recipient Object](/concepts/sap-bdc-recipient-object.md) - An object automatically created upon SAP BDC connection establishment that represents the SAP BDC account as a data recipient within Unity Catalog.
* [SAP BDC SDK](/concepts/sap-bdc-sdk.md) - A Python SDK (sap-bdc-connect-sdk) used to describe Delta Shares in Core Schema Notation for consumption by SAP BDC users.
* [SAP BDC Semantic Metadata Ingestion](/concepts/sap-bdc-semantic-metadata-ingestion.md) - Automatic syncing of SAP Business Data Cloud table and column semantic metadata (comments, constraints, tags) into Databricks Unity Catalog for mounted SAP BDC shares.
* [SAP BDC Semantic Metadata Sync](/concepts/sap-bdc-semantic-metadata-sync.md) - Automatic synchronization of SAP semantic metadata (table and column comments, primary keys, foreign keys, governance tags) into Unity Catalog when a share from SAP BDC is mounted to a catalog.
* [SAP BDC Semantic Metadata Synchronization](/concepts/sap-bdc-semantic-metadata-sync.md) - When an SAP BDC share is mounted to a Unity Catalog catalog, SAP semantic metadata—including table/column comments, primary keys, foreign keys, and governance tags—automatically syncs into Unity Catalog.
* [SAP Business Data Cloud (BDC) Connector](/concepts/sap-business-data-cloud-bdc-connector.md) - A Databricks connector that establishes a connection between a Databricks workspace and an SAP BDC account for data sharing
* [SAP PersonalData Governance Tags](/concepts/sap-personaldata-governance-tags.md) - System-reserved governance tags in the sap.PersonalData namespace that automatically sync from SAP BDC into Unity Catalog to classify tables containing personal or sensitive information.
* [SAP Semantic Metadata Synchronization](/concepts/sap-bdc-semantic-metadata-sync.md) - When an SAP BDC share is mounted to a Unity Catalog catalog, SAP semantic metadata (table/column comments, primary keys, foreign keys, governance tags) syncs automatically into Unity Catalog.
* [SavedModel Format Requirement for Keras Callbacks](/concepts/savedmodel-format-requirement-for-keras-callbacks.md) - When using horovod.spark with custom Keras callbacks, models must be saved in TensorFlow SavedModel format (.tf for TF 2.x, save_weights_only=True for TF 1.x).
* [Saving Spark DataFrames to TFRecords](/concepts/saving-spark-dataframes-to-tfrecords.md) - The process of writing Apache Spark DataFrames to the TFRecord binary format using the spark-tensorflow-connector library.
* [sbt-assembly Plugin for Building Databricks JARs](/concepts/sbt-assembly-plugin-for-building-databricks-jars.md) - Using sbt-assembly (version 2.3.1) to build a fat JAR for Databricks serverless compute; configured via project/plugins.sbt and built with sbt commands.
* [Scala Databricks Utilities limitation](/concepts/scala-databricks-utilities-limitation.md) - Only DBFS and secrets utilities are available through Databricks Utilities for Scala; no other Databricks Utilities functionality is supported.
* [Scalable Metadata Handling](/concepts/scalable-metadata-handling.md) - Delta Lake handles metadata at scale, allowing data lakes to manage billions of files and partitions without performance degradation.
* [Scalable metadata handling in Delta Lake](/concepts/scalable-metadata-handling-in-delta-lake.md) - Delta Lake's ability to efficiently manage metadata at scale for large data lake environments, preventing metadata bottlenecks common in traditional data lakes.
* [Scalar Iterator pandas UDFs](/concepts/scalar-iterator-pandas-udfs.md) - A newer variant of pandas UDFs that offers flexible APIs, support for any deep learning library, and high performance for featurization tasks in distributed Spark environments.
* [Scalar UDFs on Dedicated Access Mode](/concepts/scalar-udfs-on-dedicated-access-mode.md) - Scalar UDFs are unavailable on compute resources using dedicated access mode (formerly single user) in Databricks Connect for Scala.
* [Scale to Zero in Model Serving](/concepts/scale-to-zero-in-model-serving.md) - Configuration option allowing endpoints to scale down when not in use, introducing cold-start latency and not recommended for production
* [Scale-to-Zero and GPU Capacity Planning](/concepts/scale-to-zero-and-gpu-capacity-planning.md) - The scale-to-zero behavior and capacity planning considerations for custom LLM serving endpoints on Databricks, including fixed replica counts, cold start latency, and cloud provider GPU availability constraints.
* [Scale-to-Zero for Model Serving Endpoints](/concepts/scale-to-zero-in-model-serving.md) - An optional configuration that allows endpoints to scale down to zero when idle, with the trade-off of cold-start latency and no guaranteed capacity when scaling back up.
* [Scaled Dot Product Attention (SDPA)](/concepts/scaled-dot-product-attention-sdpa.md) - An attention implementation alternative to Flash Attention offering broader GPU compatibility in Axolotl configurations
* [Scheduled Jobs with AI Runtime](/concepts/scheduled-jobs-with-ai-runtime.md) - Recurring job execution on serverless GPU using AI Runtime, with restrictions on environment panel dependencies and auto-recovery.
* [Scheduled Jobs with Serverless GPU](/concepts/scheduled-jobs-with-serverless-gpu.md) - Recurring notebook jobs on serverless GPU compute, with limitations on environment panel usage and auto-recovery, plus a 7-day maximum runtime.
* [Scheduled Scorers (MLflow GenAI)](/concepts/scheduled-scorers-mlflow-genai.md) - A serverless workload in Databricks MLflow that periodically evaluates GenAI applications in production, subject to budget policy controls.
* [Scheduled Streaming Table Deep Clone Restriction](/concepts/scheduled-streaming-table-deep-clone-restriction.md) - Scheduled streaming tables cannot be used as the source of a deep clone operation
* [Scheduled Streaming Tables](/concepts/scheduled-streaming-tables.md) - Streaming tables configured to run on a schedule, which are not supported for deep clone operations in Databricks
* [SCHEDULED_TABLE_NOT_SUPPORTED](/concepts/scheduled_table_not_supported.md) - Sub-error indicating that scheduled streaming tables cannot be used as the source for a deep clone operation.
* [SCHEDULED_TABLE_NOT_SUPPORTED sub-error](/concepts/scheduled_table_not_supported.md) - Sub-error indicating that scheduled (automated refresh) streaming tables cannot be deep cloned.
* [Scheduling AI Runtime Jobs](/concepts/scheduling-ai-runtime-jobs.md) - Method for running recurring GPU-accelerated notebook jobs on serverless GPU with specific limitations on dependencies and auto-recovery
* [Schema and Data Type Compatibility with Apache Iceberg](/concepts/schema-and-data-type-compatibility-with-apache-iceberg.md) - IcebergCompatV imposes restrictions on supported data types and schema changes, including unsupported data types, partition data types, and type widening that are incompatible with Apache Iceberg.
* [Schema and data type constraints for Iceberg compat](/concepts/schema-and-data-type-constraints-for-iceberg-compat.md) - IcebergCompatV imposes restrictions on schema changes and data types, including unsupported data types, unsupported partition data types, type widening incompatibility, and partition spec rename restrictions.
* [Schema and Type Compatibility Between Delta and Iceberg](/concepts/schema-and-data-type-compatibility-with-apache-iceberg.md) - Constraints on schema changes and data types when enabling IcebergCompatV, including unsupported data types, unsupported partition data types, and incompatibility with type widening.
* [Schema Mismatch with Open Source Spark and Delta Sharing](/concepts/schema-mismatch-with-open-source-spark-and-delta-sharing.md) - Schema mismatch errors occur in OSS Spark when Delta table schema or partition columns change after DataFrame creation; fixable by setting spark.delta.sharing.client.useStructuralSchemaMatch to true (requires client 1.2.3+ and Spark 4.0+).
* [SCHEMA_MISMATCH sub-error](/concepts/schema_mismatch-sub-error.md) - A DELTA_METADATA_MISMATCH sub-error that occurs when the data schema does not match the Delta table schema, resolvable via mergeSchema or auto-merge configuration
* [Schema-Level Data Sharing](/concepts/schema-level-data-sharing.md) - Sharing an entire schema (database) in a share, which provides recipients access to all current and future tables, streaming tables, views, materialized views, models, and volumes in that schema. Tables shared this way always include full history. Has limitations: unsupported assets are filtered out, aliases/partitions are removed, and schema-level aliasing is not supported.
* [Schema-Level Sharing](/concepts/schema-level-sharing.md) - Sharing an entire schema in Unity Catalog so recipients automatically gain access to all current and future tables, views, volumes, and models within that schema.
* [Schema-level Sharing in OpenSharing](/concepts/schema-level-sharing-in-opensharing.md) - Sharing an entire schema (database) so recipients automatically access all current and future tables, views, models, and volumes within it, with specific limitations on unsupported asset types.
* [Schema-to-session integration](/concepts/schema-to-session-integration.md) - Labeling schemas are associated with labeling sessions in the Review App, and schemas defined in an experiment are automatically available when creating sessions.
* [Scoped Resource Access in Databricks Apps](/concepts/scoped-resource-access-in-databricks-apps.md) - The principle that adding a resource (like an MLflow experiment) to an app grants access only to that specific resource instance, not to all resources of that type in the workspace.
* [Scorer class](/concepts/scorer-class.md) - A Pydantic-based base class for defining scorers with internal state and custom logic, offering more flexibility than the @scorer decorator but not supported for production monitoring.
* [Scorer class (Pydantic base class)](/concepts/scorer-class.md) - An alternative, Pydantic-based base class for defining scorers with internal state and custom fields, supporting more flexibility than the @scorer decorator.
* [Scorer class for advanced customization](/concepts/scorer-class.md) - An alternative to the decorator pattern, using a Pydantic-based Scorer base class that supports internal state and custom fields, but not supported for production monitoring.
* [Scorer decorator pattern](/concepts/scorer-decorator-pattern.md) - The @scorer decorator for defining lightweight, stateless custom evaluation functions in MLflow's GenAI evaluation framework
* [Scorer error handling strategies](/concepts/scorer-error-handling-strategies.md) - Two approaches for handling errors in MLflow scorers: letting exceptions propagate naturally or handling them explicitly with AssessmentError.
* [Scorer Immutability](/concepts/scorer-immutability.md) - Scorers are immutable objects; every lifecycle operation returns a new scorer instance rather than modifying the original, preventing accidental modifications in production.
* [Scorer Immutable Update Pattern](/concepts/scorer-immutable-update-pattern.md) - Scorer instances are immutable — update() and stop() return new Scorer instances with modified configuration rather than mutating the original
* [Scorer inputs](/concepts/scorer-inputs.md) - The four optional input parameters available to scorers: inputs, outputs, expectations, and trace from MLflow traces
* [Scorer inputs and outputs](/concepts/scorer-inputs-and-outputs.md) - Scorers receive inputs (request), outputs (response), expectations (ground truth), and trace (complete MLflow trace), and return simple values or rich Feedback objects.
* [Scorer Lifecycle](/concepts/scorer-lifecycle.md) - The lifecycle states and transitions for production scorers in Databricks MLflow, centered around MLflow experiments.
* [Scorer Lifecycle Management](/concepts/scorer-lifecycle-management.md) - Lifecycle methods (register, start, update, stop) that control MLflow scorers running continuous quality assessment on production traces
* [Scorer Lifecycle Management (Register/Start Pattern)](/concepts/scorer-lifecycle-management-registerstart-pattern.md) - Two-step API pattern (.register() then .start()) for associating scorers with an MLflow experiment and activating continuous quality monitoring.
* [Scorer metric naming conventions](/concepts/scorer-naming-conventions.md) - Rules for how metric names are derived in MLflow evaluation runs from Feedback.name fields, function names, or Scorer.name fields.
* [Scorer Naming Conventions](/concepts/scorer-naming-conventions.md) - Rules for how metric names are derived from code-based scorers, based on function names, Feedback object names, or class names depending on return type.
* [Scorer Naming Conventions (MLflow)](/concepts/scorer-naming-conventions.md) - Rules determining how metric names are derived from decorator function names, Feedback object name fields, or Scorer class name fields.
* [Scorer Properties (sample_rate and filter_string)](/concepts/scorer-properties-sample_rate-and-filter_string.md) - Instance attributes sample_rate (0.0-1.0) and filter_string that reflect the current sampling configuration of an active or stopped scorer
* [Scorer Registration and Activation Lifecycle](/concepts/scorer-registration-and-activation-lifecycle.md) - The pattern of registering a scorer with a name and starting it with a sampling configuration before use, as seen with both built-in and custom scorers.
* [Scorer Registration and Lifecycle](/concepts/scorer-registration-and-lifecycle.md) - Two-phase lifecycle pattern for scorers: .register(name=...) to assign a name, then .start(sampling_config=...) to activate the scorer with a sampling configuration.
* [Scorer Registration and Registry](/concepts/scorer-registration-and-registry.md) - API functions to register, retrieve, list, and delete scorers via get_scorer(), list_scorers(), delete_scorer(), and Scorer.register()
* [Scorer Sampling Configuration](/concepts/scorer-sampling-configuration.md) - Mechanism to control what fraction of traces are evaluated by a scorer using ScorerSamplingConfig and sample_rate parameters.
* [Scorer State Management](/concepts/scorer-state-management.md) - Management of scorer states including registration, start (active monitoring), stop (sample rate set to 0, kept registered), update, and deletion from the server.
* [Scorers — Replacing Custom Metrics and Judges](/concepts/scorers-replacing-custom-metrics-and-judges.md) - Scorers are the unified evaluation building block in MLflow 3, replacing both @metric decorators and LLM judge functions, with predefined scorers (Correctness, Safety, Guidelines, etc.) and a @scorer decorator for custom logic.
* [Scorers (MLflow GenAI)](/concepts/scorers-mlflow-genai.md) - A unified interface within MLflow GenAI evaluation framework that defines evaluation criteria for models, agents, and applications, receiving traces and returning quality assessments as feedback.
* [Scorers and LLM Judges](/concepts/scorers-and-llm-judges.md) - Evaluation components (built-in, guidelines-based, custom-prompt, code-based) used to assess GenAI trace quality, supporting both single-turn and multi-turn judging.
* [Scorers in MLflow Evaluation](/concepts/scorers-in-mlflow-evaluation.md) - Quality metrics (built-in or custom) applied by mlflow.genai.evaluate() to assess GenAI app outputs, including LLM judges for relevance, safety, and more.
* [ScorerSamplingConfig](/concepts/scorersamplingconfig.md) - Data class holding sampling rate and optional filter string configuration for controlling which traces a scorer evaluates
* [Scoring Format Change](/concepts/scoring-format-change.md) - The difference in request/response format between Legacy MLflow Model Serving and new Model Serving, requiring application updates during migration.
* [SDK-Based Dataset Management in MLflow](/concepts/sdk-based-dataset-management-in-mlflow.md) - Programmatic creation, updating, and management of evaluation datasets using the MLflow Python SDK, including methods like create_dataset(), merge_records(), search_traces(), and to_df().
* [SDPA (Scaled Dot Product Attention)](/concepts/sdpa-scaled-dot-product-attention.md) - Flash Attention alternative used for broader GPU compatibility in transformer model training
* [Search by Certification Status](/concepts/search-by-certification-status.md) - A search capability in Databricks that allows users to filter and query Unity Catalog objects by their certification status using the certificationStatus keyword or filter menu.
* [Searching and Discovering Prompts](/concepts/searching-and-discovering-prompts.md) - Prompts can be discovered and filtered using mlflow.genai.search_prompts() with filter strings for catalog and schema, with optional max_results limiting.
* [Searching and Filtering Traces and Spans](/concepts/searching-and-filtering-traces-and-spans.md) - API methods (search_spans, search_assessments, search_traces) for finding specific spans and assessments by name, regex pattern, type, span ID, or combined criteria.
* [Searching by Certification Status in Databricks](/concepts/searching-by-certification-status-in-databricks.md) - Using the 'certificationStatus' keyword in Databricks search to filter for certified or deprecated data assets across supported object types.
* [Searching OTel Traces via MLflow SDK](/concepts/searching-otel-traces-via-mlflow-sdk.md) - Using the span.attributes.* prefix in mlflow.search_traces() filter_string to query ingested OTel traces by attribute values like session.id, user.id, or token counts.
* [Secret Naming Conventions for Online Stores](/concepts/secret-naming-conventions-for-online-stores.md) - Standardized naming pattern for Databricks secrets targeting online stores, using scope-prefix-key combinations with store-specific keys (e.g., access-key-id, secret-access-key, user, password).
* [Secret Scope Management for Feature Store](/concepts/secret-scope-management-for-feature-store.md) - Creating and managing Databricks secret scopes for accessing multiple online stores, including naming conventions (prefix-based) and avoiding scope limits by sharing scopes.
* [Secret Scope Separation for Read and Write Access](/concepts/secret-scope-separation-for-read-and-write-access.md) - The pattern of creating two separate Databricks secret scopes — one for read-only access and one for read-write access — to manage online store credentials with least-privilege principles.
* [Secrets-Based Environment Variables for Model Serving](/concepts/secrets-based-environment-variables-in-model-serving.md) - Using Databricks secrets to securely inject sensitive credentials like API keys into model serving endpoints at runtime
* [Secrets-Based Environment Variables in Databricks Model Serving](/concepts/secrets-based-environment-variables-in-databricks-model-serving.md) - Securely storing and referencing API keys, tokens, and other credentials as Databricks secrets, then injecting them into model serving endpoints via the {{secrets/scope/key}} syntax.
* [Secrets-based Environment Variables in Model Serving](/concepts/secrets-based-environment-variables-in-model-serving.md) - Using Databricks secrets to securely inject credentials like API keys into model serving endpoints at serving time
* [Securable AI Assets in Unity Catalog](/concepts/securable-ai-assets-in-unity-catalog.md) - ML models, Unity Catalog functions, HTTP connections, and hosted foundation models managed as securable objects with grant/revoke privileges.
* [Securable AI Objects in Unity Catalog](/concepts/securable-ai-objects-in-unity-catalog.md) - AI resources—including models, functions, HTTP connections, and hosted foundation models—that can have permissions granted and revoked using standard Unity Catalog privileges.
* [Securable Object Lifecycle in Delta Sharing](/concepts/securable-object-lifecycle-in-delta-sharing.md) - The constraint that a securable object (e.g., view, table) cannot be dropped while it is actively shared via OpenSharing, shares, or clean rooms, with ALTER VIEW recommended as an alternative.
* [Securable objects in Unity Catalog](/concepts/securable-objects-in-unity-catalog.md) - Objects in the Unity Catalog hierarchy (catalog, schema, table, etc.) that can have attributes and have policies attached to them for access control.
* [Securable Types in Databricks](/concepts/securable-types-in-databricks.md) - Objects such as tables, views, or schemas that can be shared via Delta Sharing and are protected from deletion while actively shared; the error message uses placeholders for securable type and full name.
* [Secure API Key Management for MLflow Tracing](/concepts/secure-api-key-management-for-mlflow-tracing.md) - Best practices for managing API keys in production GenAI tracing environments using AI Gateway or Databricks secrets instead of hardcoding credentials.
* [SecureConnect](/concepts/secureconnect.md) - Databricks feature enabling data sharing through firewalls by routing traffic through Databricks-internal IP addresses
* [SecureConnect and IP Access List Interaction](/concepts/secureconnect-and-ip-access-list-interaction.md) - When SecureConnect is enabled on the provider side, the IP access list also enforces access to shared data storage; without SecureConnect, storage URLs are reachable from any client IP regardless of the IP access list.
* [SecureConnect billing and usage attribution](/concepts/secureconnect-billing-and-usage-attribution.md) - Providers are billed for data transfer through SecureConnect, with per-recipient usage attributed via the recipient_id field in the billing system table for cost breakdown.
* [SecureConnect Billing for Data Transfer](/concepts/secureconnect-billing-for-data-transfer.md) - Databricks SecureConnect feature that bills data transfer through Databricks rather than the cloud vendor, providing an alternative billing model for egress costs.
* [SecureConnect IP access lists (ACLs)](/concepts/secureconnect-ip-access-lists-acls.md) - For open recipients, IP ACLs can restrict which client IP addresses are allowed to reach SecureConnect, covering both OpenSharing endpoint access and storage access.
* [SecureConnect Limitations](/concepts/secureconnect-limitations.md) - Known constraints of SecureConnect including lack of mTLS support for classic compute and OIDC recipients, and unsupported serverless-to-Open credential scenarios
* [SecureConnect limitations and supported scenarios](/concepts/secureconnect-limitations-and-supported-scenarios.md) - SecureConnect supports sharing to AWS, Azure, and GCP recipients but has limitations including incompatibility with Cloudflare R2 storage, unavailability on AWS GovCloud, and AWS PrivateLink-to-S3 incompatibility with FIPS endpoints in US regions.
* [SecureConnect managed proxy architecture](/concepts/secureconnect-managed-proxy-architecture.md) - Databricks routes recipient data requests through a managed proxy, eliminating the need for providers to update storage firewalls when adding new recipients.
* [SecureConnect PrivateLink and NCC configuration](/concepts/secureconnect-privatelink-and-ncc-configuration.md) - For shared storage behind private endpoints not reachable from the public network, account admins can configure a Network Connectivity Configuration (NCC) and attach it to the OpenSharing metastore, with a note that AWS PrivateLink to S3 is incompatible with FIPS endpoints in US regions.
* [SecureConnect Recipient Firewall Configuration](/concepts/secureconnect-recipient-firewall-configuration.md) - The network configuration steps a Databricks recipient must take to access shares from a provider using SecureConnect, including allowlisting inbound IPs
* [SecureView barrier and predicate pushdown on protected tables](/concepts/secureview-barrier-and-predicate-pushdown-on-protected-tables.md) - The SecureView barrier introduced by row filters and column masks blocks certain predicates from being pushed to the storage layer, preventing partition pruning and liquid clustering optimizations. Simple equality predicates can still be pushed down, but functions and implicit casts are blocked, potentially forcing full table scans.
* [Security and Data Management for Autologging](/concepts/security-and-data-management-for-autologging.md) - How model training information captured by Autologging is stored in MLflow Tracking and secured using MLflow Experiment permissions.
* [Selective Caching with Predicate Pushdown](/concepts/selective-caching-with-predicate-pushdown.md) - Technique of caching only a subset of columns and rows from a table using column lists and WHERE predicates, enabling subsequent queries to avoid scanning original files.
* [Selective Column and Row Caching](/concepts/selective-column-and-row-caching.md) - The ability to cache only a subset of columns (via column list) and rows (via WHERE predicate) in the disk cache, minimizing scan overhead.
* [Selective Feature Publishing](/concepts/selective-feature-publishing.md) - Publishing only a subset of features from an offline feature table to an online store using the 'features' argument in publish_table(), while primary keys and timestamp keys are always included.
* [Selective Time-Range Migration](/concepts/selective-time-range-migration.md) - The ability to migrate only traces created after a specific timestamp using the start_time_ms parameter in epoch milliseconds, with automatic deduplication of already-migrated traces.
* [Semantic Type Annotations](/concepts/semantic-type-annotations.md) - A mechanism to manually control semantic type assignment in Databricks AutoML using Python metadata annotations, supporting types: categorical, numeric, datetime, text, and the 'native' keyword to disable detection.
* [Semantic Type Annotations for AutoML](/concepts/semantic-type-annotations-for-automl.md) - Manual annotations (categorical, numeric, datetime, text, native) that users can apply to columns to override AutoML's automatic semantic type detection.
* [Semantic Type Annotations in AutoML](/concepts/semantic-type-annotations-in-automl.md) - How users can manually assign semantic types (categorical, numeric, datetime, text, or native) to columns using metadata annotations to override or disable automatic detection.
* [Semantic Type Detection](/concepts/semantic-type-detection.md) - AutoML's ability to infer semantic types (categorical, numeric, datetime, text) from raw Spark/pandas data types, with support for manual annotations.
* [Semantic Type Detection in AutoML](/concepts/semantic-type-detection.md) - AutoML's ability to detect semantic types (datetime, numeric, categorical, text) beyond raw Spark/pandas data types, including manual annotation support.
* [Separation of Duties for ABAC](/concepts/separation-of-duties-in-abac.md) - The organizational model distributing ABAC responsibilities across specialized roles: tag taxonomy creators, data stewards (tagging), governance admins (policy creation), data creators, and data consumers.
* [Separation of Duties in ABAC](/concepts/separation-of-duties-in-abac.md) - ABAC enables distributing governance tasks across specialized teams: taxonomy creators, data stewards (tagging), governance admins (policy creation), data creators, and data consumers.
* [Separation of Duties in ABAC Deployment](/concepts/separation-of-duties-in-abac-deployment.md) - A five-step workflow distributing ABAC responsibilities across specialized roles: tag taxonomy creation, data asset tagging, policy creation, data object creation, and governed data access — each with distinct permission requirements.
* [Sequential LLM Call Tracing in Workflows](/concepts/sequential-llm-call-tracing-in-workflows.md) - Pattern where automatic tracing captures multiple LLM calls within a single workflow (e.g., analyze-then-respond), producing a trace with a parent span for the overall function and child spans for each LLM invocation.
* [Sequential Multi-LLM Call Tracing in MLflow](/concepts/sequential-multi-llm-call-tracing-in-mlflow.md) - Tracing workflows that make multiple sequential LLM calls (e.g., analysis followed by response generation) within a single traced function.
* [Sequential Multi-LLM Call Tracing Pattern](/concepts/multi-llm-call-tracing-pattern.md) - A pattern where automatic tracing captures multiple sequential LLM calls in a single workflow, creating a trace with a parent span and child spans for each LLM invocation.
* [Served Entities Configuration](/concepts/served-entities-configuration.md) - The served_entities array in a serving endpoint config where each entity specifies a name, entity_name, entity_version, and throughput limits.
* [Served Entity Configuration](/concepts/served-entity-configuration.md) - Named configuration objects that define which model versions, workload sizes, and scaling settings are assigned to each model on a multi-model endpoint.
* [Served Entity Grants](/concepts/served-entity-grants.md) - Unity Catalog grants (READ, EXECUTE) that the endpoint creator must hold on each served model and its transitive function dependencies, validated at endpoint creation and update time.
* [Server-side Batch Pipeline Redaction](/concepts/server-side-batch-pipeline-redaction.md) - Recommended approach using Lakeflow Spark Declarative Pipelines to materialize redacted copies of OTel span, log, and annotation tables via incremental streaming.
* [Server-Side Request Timeout](/concepts/server-side-timeout.md) - Default timeout limits for requests sent to model serving endpoints, varying by endpoint type (CPU vs GPU).
* [Server-Side Request Timeouts in Model Serving](/concepts/server-side-timeout.md) - Timeouts caused by the serving endpoint taking too long to respond to a request, with default limits varying by endpoint type (CPU vs GPU).
* [Server-Side Timeout](/concepts/server-side-timeout.md) - Request processing time limits enforced by the model serving endpoint, varying by workload type (CPU/GPU small, medium, large).
* [Server-Side Timeouts](/concepts/server-side-timeout.md) - Default request timeout durations on the server side of Databricks Model Serving endpoints, varying by endpoint type (CPU vs GPU with different sizes).
* [Server-side Timeouts for Model Serving Endpoints](/concepts/server-side-timeout.md) - Default request timeout durations on the server side for CPU and GPU model serving endpoints, distinguishing them from client-side timeouts.
* [Serverless Autologging Restriction](/concepts/serverless-autologging-restriction.md) - On Databricks serverless compute clusters, autologging is not automatically enabled and requires explicit invocation
* [Serverless Budget Policy](/concepts/serverless-budget-policy.md) - A Databricks policy that controls cost and resource allocation for serverless workloads, which can be assigned to MLflow experiments to govern serverless compute usage.
* [Serverless Budget Policy (Databricks)](/concepts/serverless-budget-policy.md) - A policy that controls which budget policy is used for serverless workloads in Databricks, including those run by MLflow experiments.
* [Serverless Budget Policy for MLflow Experiments](/concepts/serverless-budget-policy.md) - A policy that controls which Databricks serverless budget policy MLflow uses for serverless workloads associated with an MLflow experiment.
* [Serverless Budget Policy Permissions](/concepts/serverless-budget-policy-permissions.md) - The requirement that users and service principals must be entitled to use a budget policy before they can assign it to an MLflow experiment.
* [Serverless Compute Autologging Requirement](/concepts/serverless-compute-autologging-requirement.md) - On Databricks serverless compute clusters, autologging for GenAI tracing frameworks is not automatically enabled and must be explicitly enabled by calling the appropriate mlflow.<library>.autolog() function.
* [Serverless Compute Cluster Autolog Requirements](/concepts/serverless-compute-autologging-requirement.md) - On Databricks serverless compute clusters, autologging for GenAI tracing frameworks is not automatically enabled and requires explicit calls to mlflow.<library>.autolog().
* [Serverless compute connection in Databricks Connect](/concepts/serverless-compute-connection-in-databricks-connect.md) - Configuration for connecting to serverless compute using DATABRICKS_SERVERLESS_COMPUTE_ID=auto, serverless_compute_id=auto in profiles, or builder.serverless()/remote(serverless=True).
* [Serverless Compute for Databricks Connect Testing](/concepts/serverless-compute-for-databricks-connect-testing.md) - Databricks recommends using serverless compute with Databricks Connect for all testing because it provides Databricks Runtime features, Unity Catalog integration, faster execution, and better end-to-end validation than local OSS PySpark testing.
* [Serverless Compute for Databricks JARs](/concepts/serverless-compute-for-databricks-jars.md) - How to build, configure, and run Java/Scala JARs as tasks on Databricks serverless compute using Lakeflow Jobs, including version requirements and limitations.
* [Serverless Compute for Model Serving](/concepts/serverless-compute-for-model-serving.md) - The underlying serverless compute architecture that powers Model Serving, enabling automatic scaling, cost savings, and high availability without manual infrastructure management.
* [Serverless Compute JAR Compatibility Requirements](/concepts/serverless-compute-jar-compatibility-requirements.md) - JARs targeting Databricks serverless compute must exactly match the runtime Scala and JDK versions, use Databricks Connect, and only use public Spark APIs.
* [Serverless Compute JAR Limitations](/concepts/serverless-compute-jar-limitations.md) - APIs and techniques unavailable on serverless compute: RDD API, Spark internal APIs, and native libraries
* [Serverless Compute Limitations for JAR Tasks](/concepts/serverless-compute-limitations-for-jar-tasks.md) - Restrictions when running JARs on Databricks serverless compute: no RDD API, no Spark internal APIs, no native libraries; code must use public Spark APIs only.
* [Serverless Compute Query Timeout](/concepts/serverless-compute-query-timeout.md) - A configurable timeout property (spark.databricks.execution.timeout) for Databricks notebooks running on serverless compute, which defaults to 9000 seconds.
* [Serverless compute query timeout configuration](/concepts/serverless-compute-query-timeout.md) - For notebooks on serverless compute, queries default to 9000 second timeout, configurable via spark.databricks.execution.timeout
* [Serverless compute query timeout in Databricks notebooks](/concepts/serverless-compute-query-timeout.md) - For notebooks running on serverless compute, queries time out after 9000 seconds by default, configurable via the Spark property spark.databricks.execution.timeout.
* [Serverless Compute Tracing Requirements](/concepts/serverless-compute-tracing-requirements.md) - On serverless compute clusters, GenAI tracing autologging is not automatically enabled and must be explicitly enabled by calling mlflow.<library>.autolog() functions.
* [Serverless Compute Trust Relationship (Databricks)](/concepts/serverless-compute-trust-relationship-databricks.md) - A required trust policy configuration on an AWS IAM role that allows Databricks serverless compute to assume the role when running model serving endpoints.
* [Serverless Compute Trust Relationship for IAM Roles](/concepts/serverless-compute-trust-relationship-for-iam-roles.md) - IAM roles used with Databricks model serving endpoints must have a trust relationship configured for serverless compute, or endpoints will fail with a specific error.
* [Serverless Compute Trust Relationship for Instance Profiles](/concepts/serverless-compute-trust-relationship-for-instance-profiles.md) - The requirement that an instance profile's IAM role must have a trust relationship configured for Databricks serverless compute; otherwise the endpoint creation fails with a specific error.
* [Serverless Compute with Databricks Connect](/concepts/serverless-compute-with-databricks-connect.md) - Support for connecting Databricks Connect to serverless compute resources, available starting from Databricks Connect version 15.1 with specific workspace requirements.
* [Serverless Environment Version Compatibility](/concepts/serverless-environment-version-compatibility.md) - JARs must match the serverless runtime's exact Scala, JDK, and Databricks Connect versions to avoid runtime failures
* [Serverless Environment Versioning](/concepts/serverless-environment-versioning.md) - Serverless compute uses versioned environments (e.g., version 4) that define exact Scala, JDK, and Databricks Connect versions your JAR must match to run without failures.
* [Serverless GPU @distributed Decorator](/concepts/serverless-gpu-distributed-decorator.md) - A Databricks serverless_gpu library decorator that launches PyTorch Lightning training across multiple GPUs on a single node
* [Serverless GPU API](/concepts/serverless-gpu-api.md) - Databricks API for scaling training across multiple GPUs and nodes without managing infrastructure.
* [Serverless GPU API for Distributed Training](/concepts/serverless-gpu-api-for-distributed-training.md) - A Databricks API for scaling machine learning training across multiple GPUs and nodes in a serverless manner, still in Beta.
* [Serverless GPU Cluster on Databricks](/concepts/serverless-gpu-cluster-on-databricks.md) - A Databricks compute configuration that provides on-demand GPU resources without managing clusters, suitable for running deep learning workloads like time series forecasting.
* [Serverless GPU Clusters for ML Workflows](/concepts/serverless-gpu-clusters-for-ml-workflows.md) - Databricks provides serverless GPU clusters for running machine learning workloads such as time-series forecasting with minimal infrastructure management.
* [Serverless GPU Compute](/concepts/serverless-gpu-compute.md) - On-demand, auto-scaling GPU compute in Databricks that powers AI Runtime workloads and auto-terminates after 60 minutes of inactivity.
* [Serverless GPU Compute (Databricks)](/concepts/serverless-gpu-compute-databricks.md) - Databricks managed compute that automatically provisions, scales, and orchestrates GPU resources (e.g., 8xH100) for distributed training without requiring users to manage infrastructure.
* [Serverless GPU Compute for Classic Machine Learning](/concepts/serverless-gpu-compute-for-classic-machine-learning.md) - Using serverless GPU clusters on Databricks to accelerate classic machine learning workloads such as time series forecasting with deep learning models.
* [Serverless GPU Compute on Databricks](/concepts/serverless-gpu-compute-on-databricks.md) - On-demand GPU compute resource that auto-terminates after 60 minutes of inactivity, configurable via accelerator selection like 8xH100
* [Serverless GPU Compute Specification](/concepts/serverless-gpu-compute-specification.md) - Configuration of GPU resources for AIR workloads, including accelerator type (e.g., GPU_1xA10), count of accelerators, and distributed multi-node support.
* [Serverless GPU Distributed Training API](/concepts/serverless_gpu-distributed-decorator.md) - A Databricks API using the @distributed decorator that handles provisioning GPUs, setting up the distributed environment, and managing the lifecycle of remote compute resources for multi-GPU training.
* [Serverless GPU Environment](/concepts/serverless-gpu-environment.md) - Managed compute environment for AI Runtime workloads specifying num_accelerators, accelerator_type, and environment version for running training jobs
* [Serverless GPU Environments in AI Runtime](/concepts/serverless-gpu-environments-in-ai-runtime.md) - Pre-configured serverless GPU compute environments (e.g., GPU_1xA10) with selectable environment versions that run training workloads without managing infrastructure.
* [Serverless GPU for Machine Learning](/concepts/serverless-gpu-for-machine-learning.md) - On-demand GPU compute resources on Databricks that enable training and fine-tuning of computer vision models without managing underlying infrastructure.
* [Serverless GPU Infrastructure](/concepts/serverless-gpu-infrastructure.md) - Fully managed, flexible access to GPUs on Databricks without cluster configuration, driver selection, or autoscaling policies.
* [Serverless GPU Python Library for Distributed Training](/concepts/serverless-gpu-python-library-for-distributed-training.md) - A serverless GPU Python library on Databricks that simplifies distributed training workflows across multiple GPUs.
* [Serverless GPU Training on Databricks](/concepts/serverless-gpu-training-on-databricks.md) - AI Runtime supports serverless GPU compute for single-node computer vision tasks (Public Preview), with a separate Beta API for distributed multi-GPU training.
* [Serverless JAR Jobs on Databricks](/concepts/serverless-jar-jobs-on-databricks.md) - Creating and running JAR tasks (Java/Scala Spark code) on serverless compute within Databricks Lakeflow Jobs.
* [Serverless Notebook Environment for Model Registration](/concepts/serverless-notebook-environment-for-model-registration.md) - The requirement that models must be logged and registered in Databricks Serverless Notebooks (client version 3 or 4) to use express deployments.
* [Serverless Notebook Environments](/concepts/serverless-notebook-environments.md) - Databricks compute environments (client versions 3 or 4) required for express model deployments, where model artifacts are packaged during registration.
* [Serverless Notebook Environments (Databricks)](/concepts/serverless-notebook-environments.md) - Databricks compute environments (client versions 3 or 4) used for model training and registration that enable express deployment features through environment consistency.
* [Serverless Usage Policies for Model Serving](/concepts/serverless-usage-policies-for-model-serving.md) - Custom tagging mechanism for attributing serverless usage costs to specific projects or teams via budget policies attached to endpoints.
* [Serverless vs Classic Compute for SecureConnect](/concepts/serverless-vs-classic-compute-for-secureconnect.md) - Serverless compute recipients do not need egress firewall configuration for SecureConnect, while classic compute and open recipients must allowlist inbound IPs
* [Serverless vs Classic Compute Forecasting](/concepts/serverless-vs-classic-compute-forecasting.md) - The distinction between Databricks' serverless forecasting (fully-managed compute) and classic compute forecasting (traditional compute-backed), with differences in setup, scaling, and management overhead.
* [Serverless Workload Types in MLflow](/concepts/serverless-workload-types-in-mlflow.md) - The three types of MLflow serverless workloads affected by budget policies: scheduled scorers, synthetic evaluation set generation, and agent evaluation.
* [serverless_gpu @distributed decorator](/concepts/serverless_gpu-distributed-decorator.md) - A Python decorator from Databricks' serverless_gpu library that handles provisioning GPUs, setting up the distributed training environment, and managing the lifecycle of remote compute resources for multi-GPU training.
* [serverless_gpu Distributed Training](/concepts/serverless_gpu-distributed-training.md) - A Databricks library that provides a @distributed decorator to run LLM training across multiple GPUs (e.g., 8xH100) with automatic rank management and device mapping.
* [serverless_gpu distributed training library](/concepts/serverless_gpu-distributed-training-library.md) - A Python library enabling distributed GPU training on Databricks Serverless GPU compute with decorators like @distributed to specify GPU count and type.
* [serverless_gpu Distributed Training on Databricks](/concepts/serverless_gpu-distributed-training.md) - A Python library providing a @distributed decorator pattern for scaling LLM finetuning across multiple GPUs (e.g., 8×H100) on Databricks serverless compute.
* [serverless_gpu Python Library](/concepts/serverless_gpu-python-library.md) - A first-party Databricks Python library that provides decorators and runtime utilities for executing distributed GPU workloads seamlessly from Databricks notebooks.
* [serverless_gpu.data.DataLoader](/concepts/serverless_gpudatadataloader.md) - A drop-in PyTorch DataLoader subclass tuned for serverless GPU I/O on Databricks, with higher default num_workers and prefetch_factor, plus MLflow timing logs.
* [serverless_gpu.runtime Module](/concepts/serverless_gpuruntime-module.md) - A runtime module within the serverless_gpu library that provides utilities like get_local_rank() and get_global_rank() for identifying GPU processes in a distributed setting.
* [Service Principal Authentication for Endpoint Testing](/concepts/service-principal-authentication-for-endpoint-testing.md) - Using Databricks service principals with OAuth tokens, secret scopes, and Can Query permissions to authenticate load tests against model serving endpoints.
* [Service Principal Authentication for Load Testing](/concepts/service-principal-authentication-for-load-testing.md) - Setting up a Databricks Service Principal with OAuth token generation and secret-scoped credentials to authenticate load test clients against route-optimized model serving endpoints.
* [Service Principal Authentication for ML Endpoint Load Tests](/concepts/service-principal-authentication-for-endpoint-testing.md) - Setting up a Databricks service principal with Can Query permissions and OAuth tokens stored in Databricks secrets to authenticate Locust load tests against model serving endpoints.
* [Service Principal Authentication for Model Serving Load Tests](/concepts/service-principal-authentication-for-endpoint-testing.md) - Setting up a Databricks service principal with Can Query permissions on the endpoint, storing its client ID and secret in a Databricks secret scope, and generating OAuth tokens for authenticating Locust load test traffic.
* [Service principal authentication for serving endpoints](/concepts/service-principal-authentication-for-serving-endpoints.md) - Using Databricks service principals to programmatically fetch OAuth tokens for production applications that query route-optimized serving endpoints.
* [Service Principal Authentication in Delta Sharing](/concepts/service-principal-authentication-in-delta-sharing.md) - Registering an OAuth application as a Service Principal (SP) in the recipient's IdP to authenticate M2M access to Delta Sharing shares without sharing long-lived secrets between parties.
* [Service Principal Authorization for Databricks App Resources](/concepts/service-principal-authorization-for-databricks-apps.md) - The authorization model where Databricks grants an app's associated service principal specific permissions on linked resources, scoping access to only those explicitly added resources.
* [Service Principal Authorization for Databricks Apps](/concepts/service-principal-authorization-for-databricks-apps.md) - The authorization model where Databricks Apps use a service principal, which is granted specific permissions to resources (like MLflow experiments) when those resources are added to the app.
* [Serving Endpoint ACLs](/concepts/serving-endpoint-acls.md) - Access control management for model serving endpoints, requiring CAN MANAGE permission to modify permissions on an endpoint.
* [Serving Endpoint Configuration for Custom LLMs](/concepts/serving-endpoint-configuration-for-custom-llms.md) - The configuration decisions for creating a custom LLM serving endpoint on Databricks, including workload_type (GPU selection), workload_size (Small/Medium/Large), and programmatic creation via Databricks SDK.
* [Serving Endpoint Deployment Container and Dependencies](/concepts/serving-endpoint-deployment-container-and-dependencies.md) - Production-grade container built during deployment that includes libraries captured in the MLflow model; application-level dependencies must be explicitly specified via pip_requirements, conda_env, or extra_pip_requirements.
* [Serving Endpoint Instance Profile for Feature Lookup](/concepts/serving-endpoint-instance-profile-for-feature-lookup.md) - Configuration of Databricks model serving endpoints with instance profiles to enable MLflow models to authenticate and read features from third-party online stores.
* [Serving Endpoint OpenAPI Schema](/concepts/serving-endpoint-openapi-schema.md) - A formal OpenAPI specification (JSON) that describes a serving endpoint's query interface, request/response format, and data types for reproducibility and client code generation.
* [Serving Endpoint REST API (Databricks)](/concepts/serving-endpoint-rest-api-databricks.md) - The Databricks REST API (endpoints /api/2.0/serving-endpoints) for creating and updating model serving endpoint configurations, including fields like instance_profile_arn.
* [Serving Endpoint REST API Instance Profile Configuration](/concepts/serving-endpoint-rest-api-instance-profile-configuration.md) - The instance_profile_arn field in the Databricks Serving Endpoint REST API (POST/PUT /api/2.0/serving-endpoints) used to attach an AWS instance profile during endpoint creation or update.
* [Serving Endpoint Update Semantics](/concepts/serving-endpoint-update-semantics.md) - How configuration updates to custom model endpoints work: old config keeps serving until new one is ready, concurrent updates are blocked, and updates can be cancelled from the UI.
* [Serving Endpoints (Databricks)](/concepts/serving-endpoints-databricks.md) - REST API endpoints for Databricks Model Serving that use a 'serving-endpoints' URL path, with full API support for creation, configuration, and lifecycle management.
* [Serving UI](/concepts/serving-ui.md) - The Databricks workspace interface for creating, viewing, and managing model serving endpoints, including canceling in-progress configuration updates.
* [Serving UI Advanced Configuration for Instance Profiles](/concepts/serving-ui-advanced-configuration-for-instance-profiles.md) - Instance profiles can be added to model serving endpoints through the Databricks Serving UI under Advanced configurations, with the endpoint creator's permission validated at creation time.
* [Serving UI token fetch](/concepts/serving-ui-token-fetch.md) - Fetching a short-lived OAuth token (valid for 1 hour) directly from the Databricks Serving UI for development and testing of route-optimized endpoints.
* [Session and User Association in MLflow Traces](/concepts/session-and-user-association-in-mlflow-traces.md) - Associating traces with sessions and users via the session.id and user.id OpenTelemetry span attributes for trace-level metadata and the session tab in MLflow UI.
* [Session ID Tracing](/concepts/session-id-tracing.md) - Tagging MLflow traces with session IDs via mlflow.update_current_trace to group individual turns into coherent conversations.
* [Session ID Tracking](/concepts/session-id-tracking.md) - The mechanism of tagging MLflow traces with session IDs to group individual turns into coherent conversation sessions for multi-turn evaluation.
* [Session temporary target restriction for clone with history](/concepts/session-temporary-target-restriction-for-clone-with-history.md) - Cloning a Delta table with history to a session temporary target table is not supported in Databricks.
* [Session temporary target tables in Delta Clone with History](/concepts/session-temporary-target-tables-in-delta-clone-with-history.md) - An error sub-condition (SESSION_TEMPORARY) where the target table for a Delta Clone with History operation is a session-scoped temporary table
* [Session Tracking with Trace Tags](/concepts/session-tracking-with-trace-tags.md) - Mechanism for instrumenting agents with session IDs on traces (mlflow.trace.session tag) to group individual turns into conversation sessions for evaluation.
* [Session User Identity Model for ABAC on Views](/concepts/session-user-identity-model-for-abac-on-views.md) - ABAC policies on underlying tables are evaluated using the session user's identity when querying through views and functions
* [SESSION_TEMPORARY target error](/concepts/session_temporary-target-error.md) - A subtype of DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_TARGET indicating the target table is a session temporary table (not supported for clone with history).
* [SESSION_TEMPORARY target table restriction](/concepts/session_temporary-target-table-restriction.md) - Delta clone with history does not support session-scoped temporary target tables.
* [Setting Initial Streaming Starting Point with Version or Timestamp](/concepts/setting-initial-streaming-starting-point-with-version-or-timestamp.md) - Using startingVersion or startingTimestamp options to begin a Delta Lake streaming read from a specific table version or point in time, skipping the full table snapshot.
* [Setting the active experiment in MLflow](/concepts/active-experiment-in-mlflow.md) - How to control where MLflow runs are logged by setting the active experiment using mlflow.set_experiment(), mlflow.start_run(), or environment variables.
* [SFTConfig and SFTTrainer](/concepts/sftconfig-and-sfttrainer.md) - Components from the TRL library that simplify supervised fine-tuning configuration and training, handling dataset loading, model initialization, checkpointing, and metric logging.
* [SFTTrainer](/concepts/sfttrainer.md) - A TRL trainer class for supervised fine-tuning that handles dataset formatting, tokenization, and training loop with optional DeepSpeed optimization
* [SFTTrainer (Supervised Fine-Tuning Trainer)](/concepts/sfttrainer-supervised-fine-tuning-trainer.md) - A TRL library component that simplifies supervised fine-tuning configuration by automatically applying optimizations like Liger kernels, mixed precision, gradient checkpointing, and chat format setup.
* [SFTTrainer (TRL)](/concepts/sfttrainer-trl.md) - A supervised fine-tuning trainer from the Transformer Reinforcement Learning (TRL) library that streamlines instruction tuning of language models with built-in support for LoRA, gradient checkpointing, and mixed precision.
* [SFTTrainer and TRL Library](/concepts/sfttrainer-and-trl-library.md) - Supervised Fine-Tuning Trainer from the Transformer Reinforcement Learning (TRL) library used to fine-tune causal language models with support for LoRA, gradient checkpointing, and mixed precision
* [SFTTrainer with Chat Template Processing](/concepts/sfttrainer-with-chat-template-processing.md) - The use of TRL's SFTTrainer with Unsloth's chat template utilities (get_chat_template, standardize_sharegpt, train_on_responses_only) for supervised finetuning on conversational datasets like mlabonne/FineTome-100k.
* [SFTTrainer with train_on_responses_only](/concepts/sfttrainer-with-train_on_responses_only.md) - A training approach using TRL's SFTTrainer combined with Unsloth's train_on_responses_only utility to mask instruction parts and train only on assistant responses.
* [SGC recommender system with PyTorch Lightning](/concepts/sgc-recommender-system-with-pytorch-lightning.md) - A tutorial pattern using Simplified Graph Convolution (SGC) for recommendations, implemented with PyTorch Lightning for distributed training on Databricks
* [Shallow Clone](/concepts/shallow-clone.md) - A cloning strategy that copies only the source table's metadata and references the source data files without copying the data.
* [Shallow Clone (Databricks)](/concepts/shallow-clone.md) - A metadata-only copy of a source Delta, Iceberg, or Parquet table that references the source data files without copying them, creating a lightweight pointer.
* [Shallow Clone (Delta Lake)](/concepts/shallow-clone.md) - A metadata-only copy of a source Delta, Parquet, or Foreign Iceberg table that references the source's data files without copying them.
* [Shallow Clone (Delta Table)](/concepts/shallow-clone.md) - A copy of a source table's definition that references the original data files without copying them, providing a lightweight metadata-only clone.
* [Share Deletion Implications](/concepts/share-deletion-implications.md) - Deleting an OpenSharing share immediately revokes recipient access to all shared data; only the share owner can perform deletion.
* [Share Details and Metadata](/concepts/share-details-and-metadata.md) - Information associated with a share including owner, creator, creation/update timestamps, comments, contained data assets, and associated recipients.
* [Share Owner Asset Permission Requirements](/concepts/share-owner-asset-permission-requirements.md) - The requirement that share owners must have sufficient permissions on all assets in the share for recipients to access them, and object creators must grant access to new assets.
* [Share Owner Exemption from ABAC Policies](/concepts/share-owner-exemption-from-abac-policies.md) - The share owner must be listed in the EXCEPT clause of ABAC policies on the provider side to share tables and views protected by row filters and column masks through OpenSharing.
* [Share Owner Permissions for Shared Assets](/concepts/share-owner-permissions-for-shared-assets.md) - A 'does not exist' error on a shared asset indicates the share owner lacks sufficient permissions on the asset; requires verifying permissions on all shared objects.
* [Share Ownership and ABAC Security](/concepts/share-ownership-and-abac-security.md) - The security implication that transferring share ownership to an over-privileged user may allow recipients over-privileged access when ABAC policies are in effect on underlying tables or schemas.
* [Share Update Operations](/concepts/share-update-operations.md) - Operations that can be performed on an existing OpenSharing share including renaming, removing assets, adding comments, renaming table aliases, enabling history access, managing partitions, and changing ownership.
* [Share Viewing Interfaces](/concepts/share-viewing-interfaces.md) - Three interfaces for viewing OpenSharing shares and their details: Catalog Explorer (UI), Databricks Unity Catalog CLI, and SQL commands in notebooks or SQL query editor.
* [Share-to-Catalog Mounting](/concepts/share-to-catalog-mounting.md) - The process of creating a new catalog from a share or mounting a share to an existing OpenSharing catalog, which makes shared data assets accessible in a recipient's Unity Catalog workspace.
* [Shareable Asset Types in OpenSharing](/concepts/shareable-asset-types-in-opensharing.md) - The broad range of data and AI assets that can be included in an OpenSharing share, including Delta tables, streaming tables, Iceberg tables, views, volumes, notebooks, Python UDFs, AI models, FeatureSpecs, and Genie Spaces.
* [Shared Cluster Compatibility in Unity Catalog](/concepts/shared-cluster-compatibility-in-unity-catalog.md) - Certain Spark APIs like spark.catalog, sc.parallelize, sc.emptyRDD, and input_file_name() are unsupported or behave differently on shared clusters when using Unity Catalog.
* [Shared Data Permission Model](/concepts/shared-data-permission-model.md) - A hierarchical privilege model for OpenSharing catalogs where catalog owners can grant read-only privileges (SELECT, READ VOLUME, USE CATALOG) while write/update privileges cannot be granted on shared assets.
* [Shared External Table Conversion Caveat](/concepts/shared-external-table-conversion-caveat.md) - The issue that multiple external tables sharing the same underlying Parquet directory require all such tables to be converted to Delta Lake, or they become inaccessible after the first conversion.
* [Shared Infrastructure Requirements for Cross-workspace Feature Store](/concepts/shared-infrastructure-requirements-for-cross-workspace-feature-store.md) - Prerequisite that both local and remote workspaces must share the same external Hive metastore and have access to the same DBFS storage for cross-workspace feature store operations.
* [Shared Mode Execution (Ray + Spark)](/concepts/shared-mode-execution-ray-spark.md) - Running both Ray and Spark in the same execution environment on Databricks allows teams to use Spark for data-intensive tasks and Ray for heavy computation stages within a single application.
* [Shared Platform Features for ML, Deep Learning, and GenAI](/concepts/shared-platform-features-for-ml-deep-learning-and-genai.md) - Databricks platform features — Model Serving, ai_query, AI Runtime, MLflow experiment tracking, and AI Search — that support classic ML, deep learning, and generative AI under a unified infrastructure.
* [Shared Storage for Model Checkpointing](/concepts/shared-storage-for-model-checkpointing.md) - Using shared storage for data loading and model checkpointing in distributed deep learning training on Databricks.
* [Shared View Function Allowlist](/concepts/shared-view-function-allowlist.md) - The security mechanism that restricts Databricks-to-Databricks shared views to only a specific subset of built-in SQL functions and operators, preventing potentially dangerous operations on shared data.
* [Shared View Materialization Failure (DS_MATERIALIZATION_QUERY_FAILED)](/concepts/shared-view-materialization-failure-ds_materialization_query_failed.md) - Error when sharing views, materialized views, or streaming tables because the provider lacks read-write access to the asset or the materialization workspace is misconfigured.
* [Shared Views in Databricks-to-Databricks Sharing](/concepts/shared-views-in-databricks-to-databricks-sharing.md) - Shared views have specific restrictions including limited built-in function support, a 20-view query limit across at most 5 provider-shares, and a 5-minute query timeout for on-the-fly materialization without direct access.
* [ShareGPT chat template standardization](/concepts/sharegpt-chat-template-standardization.md) - A data preprocessing step in Unsloth that converts conversational datasets into standardized chat templates (e.g., Llama 3.1 format) before training.
* [ShareGPT Dataset Format Standardization](/concepts/sharegpt-dataset-format-standardization.md) - A preprocessing step that normalizes conversational datasets (e.g., FineTome-100k) into a standardized multi-turn conversation format with 'from' and 'value' fields, making them compatible with chat-template-based tokenization.
* [Sharing Identifier](/concepts/sharing-identifier.md) - A unique string identifying a Unity Catalog metastore, formatted as 'cloud:region:uuid', used by data providers to set up Databricks-to-Databricks sharing.
* [Sharing Tables with Deletion Vectors and Column Mapping](/concepts/sharing-tables-with-deletion-vectors-and-column-mapping.md) - Sharing Delta tables that have storage optimizations like deletion vectors or column mapping enabled, requiring history sharing and specific recipient runtime versions.
* [Short-lived JWT Token Authentication](/concepts/short-lived-jwt-token-authentication.md) - Security model where JWTs are issued and lifetime-enforced by the recipient's IdP, not by Databricks, eliminating the need to manage shared credentials or long-lived bearer tokens.
* [Short-lived JWT Tokens for Delta Sharing](/concepts/short-lived-jwt-tokens-for-delta-sharing.md) - JWT tokens issued by the recipient's IdP with lifetimes enforced by that IdP, validated by Databricks against the federation policy, eliminating the need for recipients to manage long-lived shared credentials.
* [SHOW ARCHIVED FILES syntax](/concepts/show-archived-files-syntax.md) - A SQL command that returns URIs of archived files required to satisfy a given query, enabling targeted restoration.
* [SHOW EFFECTIVE POLICIES](/concepts/show-effective-policies.md) - A SQL command that lists all policies affecting a securable object, including inherited policies from parent scopes like catalog-level policies affecting a table.
* [Simple vs Complex Deployment Job Patterns](/concepts/simple-vs-complex-deployment-job-patterns.md) - Two common deployment job patterns: a simple 3-step pipeline (evaluation, approval, deployment) and a complex staged rollout with gradual deployment percentages, metrics collection, and rollback logic.
* [Single Node Clusters for PyTorch](/concepts/single-node-clusters-for-pytorch.md) - Using single-node Databricks clusters to test and migrate single-machine PyTorch workflows
* [Single Node GPU Cluster for Deep Learning Development](/concepts/single-node-gpu-clusters-for-deep-learning.md) - Best practice of using a single-node GPU cluster (driver only) for fast, cost-effective iterative deep learning model development before scaling to distributed training.
* [Single Node GPU Clusters for Deep Learning](/concepts/single-node-gpu-clusters-for-deep-learning.md) - Best practice recommendation to use Single Node (driver-only) GPU clusters for fast, cost-effective deep learning model development, as a single node with multiple GPUs avoids network communication overhead of distributed training.
* [Single Node GPU Clusters for Deep Learning Development](/concepts/single-node-gpu-clusters-for-deep-learning-development.md) - Using a single driver-only GPU cluster for fast, cost-effective iterative deep learning model development before scaling to distributed training
* [Single Node to Distributed Code Adaptation](/concepts/single-node-to-distributed-code-adaptation.md) - The process of modifying single-node PyTorch training code to work in a distributed environment using HorovodRunner, following Databricks' recommended development workflow
* [Single Node to Distributed Training Adaptation Pattern](/concepts/single-node-to-distributed-training-adaptation-pattern.md) - Recommended development workflow of first training a model on a single node, then adapting the code for distributed training using HorovodRunner
* [Single Run Details Dashboard](/concepts/single-run-details-dashboard.md) - A pre-built example dashboard that replicates the MLflow run details page using system table queries, filterable by experiment ID, run ID, and metric name
* [Single-Brace Format Compatibility](/concepts/single-brace-format-compatibility.md) - MLflow's conversion mechanism between its native double-brace template syntax and the single-brace Python f-string syntax used by LangChain, LlamaIndex, and other GenAI frameworks.
* [Single-GPU Fine-tuning](/concepts/single-gpu-fine-tuning.md) - Techniques and configurations for fine-tuning Hugging Face transformer models on a single GPU, including batch size reduction, FP16 training, and gradient accumulation.
* [Single-GPU Fine-Tuning with Hugging Face on Databricks](/concepts/single-gpu-fine-tuning-with-hugging-face-on-databricks.md) - The end-to-end workflow for fine-tuning Hugging Face transformer models on a single GPU node in Databricks, covering cluster setup, tokenization, training configuration, MLflow logging, and inference loading.
* [Single-Machine Preference for Neural Networks](/concepts/single-machine-preference-for-neural-networks.md) - Databricks recommendation to train neural networks on a single machine when possible, because distributed code is more complex and slower due to communication overhead
* [Single-Machine Training Preference on Databricks](/concepts/single-machine-training-preference-on-databricks.md) - Databricks recommends training neural networks on a single machine when possible, as distributed training adds complexity and communication overhead
* [Single-node AI Runtime](/concepts/single-node-ai-runtime.md) - Databricks AI Runtime configured for single-node ML tasks, currently in Public Preview
* [Single-node GPU Training on Databricks](/concepts/single-node-gpu-training-on-databricks.md) - AI Runtime supports single-node GPU tasks for model training and fine-tuning, currently in Public Preview.
* [Single-Node Multi-GPU Fine-tuning Pipeline](/concepts/single-node-multi-gpu-fine-tuning-pipeline.md) - A complete reference pattern for fine-tuning large language models (like Qwen2.5-3B) using Ray Train on a single node with 8 H100 GPUs, covering Ray cluster setup, DDP training, and metric logging.
* [Single-Node Ray Cluster Bootstrap](/concepts/single-node-ray-cluster-bootstrap.md) - A pattern where the workload command starts a Ray head on the local node with all GPUs, runs the training driver, then stops the cluster — with a fallback branch for multi-node scaling.
* [Single-Node vs Multi-GPU Workload Tiers](/concepts/single-node-vs-multi-gpu-workload-tiers.md) - Databricks distinguishes between single-node tasks (Public Preview) and multi-GPU distributed training (Beta), each with different stability and availability guarantees.
* [Single-node-to-distributed development workflow](/concepts/single-node-to-distributed-development-workflow.md) - A recommended development pattern for distributed training: first train and validate on a single node, then adapt the code for distributed execution using HorovodRunner.
* [Single-User vs Global Ray Clusters](/concepts/single-user-vs-global-ray-clusters.md) - Two modes for Ray clusters on Databricks: user-scoped clusters with auto-shutdown and shared global clusters without timeout
* [skipChangeCommits](/concepts/skipchangecommits.md) - Option to ignore data modification transactions (UPDATE, DELETE, MERGE, OVERWRITE) in a Delta Lake streaming source, processing only appends.
* [skipChangeCommits for handling source table modifications](/concepts/skipchangecommits.md) - Option to skip transactions that modify/delete records in a Delta source table during streaming, processing only appends; replaces legacy options ignoreChanges and ignoreDeletes.
* [skipChangeCommits for Source Table Changes](/concepts/skipchangecommits.md) - Option to ignore transactions that delete or modify existing records in a Delta source table, processing only appends during streaming reads
* [skipChangeCommits for Streaming Sources](/concepts/skipchangecommits.md) - An option to ignore transactions that delete or modify existing records when streaming from a Delta Lake source
* [Slicing Expressions in Data Profiling](/concepts/slicing-expressions-in-data-profiling.md) - Mechanism to compute metrics for data subsets defined by boolean expressions or column values, identified by slice_key and slice_value columns.
* [Small-world Networks](/concepts/small-world-networks.md) - A real-world network pattern characterized by closely linked subgroups and a short average path length between any two nodes; commonly observed in social and other real-world networks and can lead to data skew in distributed graph processing.
* [Smolagents](/concepts/smolagents.md) - A lightweight agent framework by HuggingFace emphasizing minimalism and composability
* [Snapshot Profile](/concepts/snapshot-profile.md) - A data profile type that profiles the full contents of a table over time, with a 4TB maximum table size limit
* [Snowflake Catalog Integration for Iceberg](/concepts/snowflake-catalog-integration-for-iceberg.md) - Snowflake SQL configuration to create a CATALOG INTEGRATION and linked database using the ICEBERG_REST catalog source with OAuth authentication and vended credentials access delegation.
* [Source Table Types for CLONE](/concepts/source-table-types-for-clone.md) - CLONE supports Delta, Parquet, Foreign Iceberg, and Managed Iceberg tables as source tables, each with different cloning capabilities.
* [Source Transformation and Filtering](/concepts/source-transformation-and-filtering.md) - Pre-processing source data before aggregation using filter_condition (SQL WHERE clause) and transformation_sql (SQL SELECT expression) with schema_json requirement, plus a convenience from_sql() factory method.
* [Sources of Human Feedback in MLflow](/concepts/sources-of-human-feedback-in-mlflow.md) - MLflow collects human feedback from three distinct sources — developers (during development via direct trace annotation), domain experts (via Review App with Chat UI or labeling existing traces), and end users (in production from live applications).
* [Span Attributes and Search](/concepts/span-attributes-and-search.md) - Spans carry typed attributes accessible via get_attribute() and SpanAttributeKey constants, and can be searched by name (exact or regex), span type, span ID, or combination of criteria using trace.search_spans().
* [Span customization parameters](/concepts/span-customization-parameters.md) - The @mlflow.trace decorator accepts name, span_type, and attributes parameters to customize span metadata, with optional dynamic updates via mlflow.get_current_active_span().
* [Span Customization via @mlflow.trace](/concepts/span-customization-via-mlflowtrace.md) - Customizing created spans through decorator parameters (name, span_type, attributes) and dynamically via mlflow.get_current_active_span().
* [Span Customization via Decorator Parameters](/concepts/span-customization-parameters.md) - The @mlflow.trace decorator accepts name, span_type, and attributes parameters to customize spans, and allows dynamic attribute updates via mlflow.get_current_active_span().
* [Span Hierarchy and Analysis](/concepts/span-hierarchy-and-analysis.md) - Spans form a parent-child hierarchy within a trace, and can be analyzed for relationships, critical paths, timing statistics, and categorized by span type (LLM, CHAT_MODEL, RETRIEVER, TOOL, etc.).
* [Span Hierarchy and Critical Path Analysis](/concepts/span-hierarchy-and-critical-path-analysis.md) - Technique for analyzing trace span trees including parent-child relationships, identifying root spans, computing duration statistics, and finding the critical path (longest duration path from root to leaf).
* [Span Search and Analysis](/concepts/span-search-and-analysis.md) - API for searching spans by name (exact or regex), span type, span ID, and combined criteria; supports building span hierarchy trees, computing timing statistics by type, and identifying critical paths.
* [Span Types and Lifecycle](/concepts/span-types-and-lifecycle.md) - Spans capture input arguments, return values, exception info, and latency; they can be typed (LLM, TOOL, etc.) and organized hierarchically to represent nested function calls.
* [Span-level Labeling in MLflow](/concepts/span-level-labeling-in-mlflow.md) - The ability to attach assessments to individual spans within a trace, with the root span representing the entire trace, enabling granular feedback on specific components.
* [SpanEvent Objects](/concepts/spanevent-objects.md) - MLflow SpanEvent objects that record specific occurrences during a span's lifetime, supporting timestamped events, custom attributes, and exception-derived events.
* [SpanStatus API](/concepts/spanstatus-api.md) - MLflow SpanStatus objects for defining span completion status (OK/ERROR), with caveats about context manager overwriting status upon exit.
* [Spark Barrier Execution Mode](/concepts/spark-barrier-execution-mode.md) - Spark execution mode that enables synchronous, long-running distributed training by providing task executors with topology awareness via BarrierTaskContext.
* [Spark Barrier Execution Mode Integration](/concepts/spark-barrier-execution-mode-integration.md) - HorovodRunner integrates Horovod with Spark's barrier execution mode, using BarrierTaskContext to collect executor IPs and trigger MPI jobs via mpirun for stable long-running deep learning training.
* [Spark Caching Anti-pattern with Delta Lake](/concepts/spark-caching-anti-pattern-with-delta-lake.md) - Databricks recommends against using Spark caching with Delta Lake because it loses data skipping benefits and can return stale data if the table is accessed via a different identifier
* [Spark Caching Incompatibility with Delta Lake](/concepts/spark-caching-anti-pattern-with-delta-lake.md) - Databricks recommends against using Spark caching with Delta Lake because it loses data skipping benefits and can become stale if the table is accessed via different identifiers.
* [Spark Cluster Integration for ML Training](/concepts/spark-cluster-integration-for-ml-training.md) - The pattern of leveraging Apache Spark clusters to orchestrate distributed machine learning training jobs with TensorFlow.
* [Spark Connect](/concepts/spark-connect.md) - An open-source gRPC-based protocol within Apache Spark that enables remote connectivity to Spark clusters using the DataFrame API with a decoupled client-server architecture.
* [Spark Connect API Limitations on Serverless](/concepts/spark-connect-api-limitations-on-serverless.md) - Serverless compute uses Spark Connect, which prohibits RDD APIs, Spark internal APIs, and native libraries; code must use only public Spark APIs (DataFrame/Dataset/SQL).
* [Spark Connect Architecture for JAR Execution](/concepts/spark-connect-architecture-for-jar-execution.md) - Serverless compute uses Spark Connect; JARs run against a thin client library exposing public Spark APIs while the Spark engine runs server-side; code bypassing public API cannot use Catalyst or Photon optimization.
* [Spark Connect Compatibility for JARs](/concepts/spark-connect-compatibility-for-jars.md) - When running JARs on serverless compute, the JAR must use only public Spark APIs (no RDDs, no Catalyst/Spark internal APIs, no native libraries) because serverless uses Spark Connect under the hood.
* [Spark Connect Connection String](/concepts/spark-connect-connection-string.md) - Advanced method to configure Databricks Connect by passing a Spark Connect connection string via the `remote` function or the `SPARK_REMOTE` environment variable.
* [Spark Connect Integration with Ray Tasks](/concepts/spark-connect-integration-with-ray-tasks.md) - Using DatabricksSession and Spark Connect within Ray remote tasks to directly interact with Spark DataFrames and tables from Ray workers
* [Spark Connect Limitations for JAR Tasks](/concepts/spark-connect-limitations-for-jar-tasks.md) - Serverless compute uses Spark Connect; RDD APIs, Spark internal APIs, and native libraries (.so, .dll, JNI) are unavailable. Code must use DataFrame/Dataset APIs via SparkSession.
* [Spark Connect Protocol](/concepts/spark-connect-protocol.md) - The communication protocol used by Databricks Connect to connect a local PySpark shell to a remote Spark server on a Databricks cluster, as seen in the connection URI scheme.
* [Spark Connect with Ray](/concepts/spark-connect-with-ray.md) - Using Spark Connect from within Ray remote tasks to allow Ray workers to interact directly with a Spark cluster running on the driver node.
* [Spark Connect with Ray Tasks](/concepts/spark-connect-with-ray.md) - Using Databricks Connect (Spark Connect) to allow Ray Core tasks to directly interact with a Spark cluster from within remote Ray tasks, with considerations for threading locks and resource allocation.
* [Spark Connect with Ray Tasks on Databricks](/concepts/spark-connect-with-ray.md) - Use DatabricksSession and Spark Connect to enable Ray Core tasks to directly interact with a Spark cluster from worker nodes, though tasks become sequential due to a threading lock on the single Spark driver.
* [Spark Context and RDD Limitations](/concepts/spark-context-and-rdd-limitations.md) - Spark Context and RDDs are not available when using Databricks Connect for Scala; users must use higher-level DataFrame APIs.
* [Spark DataFrame Evaluation Pattern](/concepts/spark-dataframe-evaluation-pattern.md) - Using Spark DataFrames for large-scale GenAI evaluation when data resides in Delta Lake or Unity Catalog.
* [Spark DataFrame to Hugging Face Dataset Conversion](/concepts/spark-dataframe-to-hugging-face-dataset-conversion.md) - Converting Apache Spark DataFrames to Hugging Face Dataset objects using the Dataset.from_spark method for model training
* [Spark DataFrame to MDS Conversion Workflow](/concepts/spark-dataframe-to-mds-conversion-workflow.md) - A recommended four-step workflow: load/preprocess data with Spark, convert to MDS format via dataframe_to_mds, load with StreamingDataset, and serve with StreamingDataLoader.
* [Spark DataFrame to ML Format Conversion](/concepts/spark-dataframe-to-ml-format-conversion.md) - Converting Apache Spark DataFrames into pandas DataFrames using toPandas() and optionally to NumPy format using to_numpy() for ML pipelines.
* [Spark Dataset Converter API](/concepts/spark-dataset-converter-api.md) - The Petastorm API method that materializes a Spark DataFrame in Parquet format and loads it as a tf.data.Dataset or torch.utils.data.DataLoader for deep learning frameworks.
* [Spark ML Distributed Training with pyspark.ml.connect](/concepts/spark-ml-distributed-training-with-pysparkmlconnect.md) - A module in Databricks Runtime 17.0+ for distributed training and inference of Spark ML models, enabled by default in Standard compute resources.
* [Spark ML Text Processing](/concepts/spark-ml-text-processing.md) - Using Spark ML's built-in text processing tools (tokenization, stop-word removal, word2vec, feature hashing) to create features from text columns within ML pipelines.
* [Spark MLlib Compatibility Limitations with Databricks Connect](/concepts/spark-mllib-limitations-with-databricks-connect.md) - Databricks Connect has limited compatibility with Apache Spark MLlib because MLlib uses RDDs while Databricks Connect only supports the DataFrame API, restricting certain sparklyr ML functions.
* [Spark MLlib limitations with Databricks Connect](/concepts/spark-mllib-limitations-with-databricks-connect.md) - Databricks Connect has limited compatibility with Apache Spark MLlib because MLlib uses RDDs while Databricks Connect only supports the DataFrame API; users needing full MLlib functions should use Databricks notebooks or the brickster package's db_repl function.
* [Spark NLP Library](/concepts/spark-nlp-library.md) - An open-source library for scaling deep learning NLP methods on Apache Spark, providing annotators and pre-trained transformers for tasks like tokenization, named entity recognition, summarization, and translation.
* [Spark NLP Transformers](/concepts/spark-nlp-transformers.md) - Pre-trained deep learning models available in Spark NLP (including BERT, T5, and Marian) for tasks such as text generation, summarization, named entity recognition, and machine translation.
* [Spark Shell and SBT with Databricks Connect](/concepts/spark-shell-and-sbt-with-databricks-connect.md) - Instructions for using Databricks Connect with the Spark shell (pyspark and spark-shell) and configuring SBT build files to link against Databricks Connect JARs.
* [spark_connect Function for Databricks Connect](/concepts/spark_connect-function-for-databricks-connect.md) - The sparklyr::spark_connect function with method='databricks_connect' parameter, which establishes a connection from an R environment to a remote Databricks cluster for distributed data operations.
* [SPARK_REMOTE Environment Variable](/concepts/spark_remote-environment-variable.md) - Environment variable used to point Databricks Connect or Spark Connect clients to a remote Spark server instance
* [SPARK_REMOTE environment variable authentication](/concepts/spark_remote-environment-variable-authentication.md) - A configuration method for Databricks Connect where the SPARK_REMOTE environment variable provides connection and authentication details without explicit code configuration.
* [SPARK_REMOTE Environment Variable Incompatibility](/concepts/spark_remote-environment-variable-limitation.md) - The Databricks SDK for Python does not recognize the SPARK_REMOTE environment variable used by Databricks Connect; authentication must use DATABRICKS_HOST and DATABRICKS_TOKEN instead.
* [SPARK_REMOTE Environment Variable Limitation](/concepts/spark_remote-environment-variable-limitation.md) - The Databricks SDK for Python does not recognize the SPARK_REMOTE environment variable for authentication, unlike the Databricks Connect client itself
* [Spark-TensorFlow Integration](/concepts/spark-tensorflow-integration.md) - The pattern of combining Apache Spark's distributed data processing with TensorFlow's deep learning capabilities for scalable ML workflows
* [spark-tensorflow-connector](/concepts/spark-tensorflow-connector.md) - A TensorFlow ecosystem library that enables conversion between Apache Spark DataFrames and TFRecord files.
* [spark-tensorflow-distributor](/concepts/spark-tensorflow-distributor.md) - An open-source native TensorFlow package for distributed training on Spark clusters
* [sparkdl.xgboost Deprecation and Migration to xgboost.spark](/concepts/sparkdlxgboost-deprecation-and-migration-to-xgboostspark.md) - The sparkdl.xgboost module is deprecated since Databricks Runtime 12.0 ML; users should migrate to the xgboost.spark module.
* [sparkdl.xgboost Module](/concepts/sparkdlxgboost-module.md) - A deprecated PySpark module providing XgboostRegressor and XgboostClassifier estimators for distributed XGBoost training within Spark ML pipelines, superseded by xgboost.spark in Databricks Runtime 12.0 ML.
* [sparkdl.xgboost Parameter Differences from Standard xgboost](/concepts/sparkdlxgboost-parameter-differences-from-standard-xgboost.md) - Key parameter discrepancies between sparkdl.xgboost and the standard xgboost package, including missing, baseMarginCol, weightCol, and validationIndicatorCol.
* [sparkdl.xgboost Parameter Differences from xgboost Package](/concepts/sparkdlxgboost-parameter-differences-from-xgboost-package.md) - Important semantic differences in parameters between sparkdl.xgboost and the standard xgboost package, including handling of missing values in sparse vectors and unsupported parameters like gpu_id, output_margin, and validate_features
* [sparkdl.xgboost PySpark Estimators](/concepts/sparkdlxgboost-pyspark-estimators.md) - PySpark ML pipeline estimators (XgboostRegressor and XgboostClassifier) from the sparkdl.xgboost module for distributed XGBoost training on Databricks
* [sparklyr](/concepts/sparklyr.md) - R interface to Apache Spark that serves as the primary integration point for Databricks Connect from R, enabling dplyr and dbplyr workflows against Databricks clusters.
* [sparklyr Databricks Connect Integration](/concepts/sparklyr-databricks-connect-integration.md) - The integration of sparklyr with Databricks Connect for Databricks Runtime 13.0+, enabling R users to run Spark DataFrame operations on remote Databricks clusters.
* [sparklyr Databricks Integration](/concepts/sparklyr-databricks-integration.md) - The sparklyr R package's method = 'databricks_connect' integration that provides an R interface to Databricks Connect for remote cluster operations.
* [SparkSession as DatabricksSession Fallback](/concepts/sparksession-as-databrickssession-fallback.md) - A pattern for using org.apache.spark.sql.SparkSession directly (instead of DatabricksSession) when the DatabricksSession class is unavailable, relying on SPARK_REMOTE for authentication.
* [SparkSession with SPARK_REMOTE](/concepts/sparksession-with-spark_remote.md) - Alternative pattern to connect to Databricks clusters using the standard SparkSession builder when DatabricksSession is unavailable, relying on the SPARK_REMOTE environment variable
* [SparkSession with SPARK_REMOTE authentication](/concepts/sparksession-with-spark_remote.md) - An alternative approach to connecting to Databricks clusters using the standard SparkSession class together with the SPARK_REMOTE environment variable for authentication, useful when DatabricksSession is unavailable.
* [SparkSession-based Databricks Connect](/concepts/sparksession-based-databricks-connect.md) - A fallback pattern for using Databricks Connect with the standard SparkSession class when DatabricksSession is unavailable, relying on SPARK_REMOTE for authentication.
* [SparkTrials](/concepts/sparktrials.md) - A Databricks API that distributes Hyperopt hyperparameter tuning trials across Spark workers for parallel execution.
* [SparkTrials for Distributed Hyperopt](/concepts/sparktrials-for-distributed-hyperopt.md) - A Trials subclass for Hyperopt that distributes hyperparameter tuning trials across a Spark cluster, accepting parallelism and timeout parameters for concurrent model fitting.
* [SparkTrials Incompatibility with Autoscaling Clusters](/concepts/sparktrials-incompatibility-with-autoscaling-clusters.md) - SparkTrials should not be used on autoscaling clusters because Hyperopt sets the parallelism value at execution start and cannot adapt if the cluster later rescales.
* [SparkTrials parallelism configuration](/concepts/sparktrials-parallelism-configuration.md) - SparkTrials accepts a parallelism parameter (default: number of Spark executors, max: 128) and a timeout parameter to control distributed tuning execution.
* [SparkTrials Parallelism for CPU vs GPU Clusters](/concepts/sparktrials-parallelism-for-cpu-vs-gpu-clusters.md) - SparkTrials parallelism configuration differs between CPU clusters (multiple executor threads per node) and GPU clusters (one executor thread per node to avoid GPU conflicts).
* [SparkTrials vs Trials in Hyperopt](/concepts/sparktrials-vs-trials-in-hyperopt.md) - Key distinction: SparkTrials distributes individual trials across workers for non-distributed algorithms, while Trials runs all trials on the driver for use with distributed training algorithms.
* [Sparse Data Optimization for XGBoost on Spark](/concepts/sparse-data-optimization-for-xgboost-on-spark.md) - Optimization for training on sparse feature datasets using enable_sparse_data_optim=True and missing=0.0 with pyspark.ml.linalg.SparseVector input.
* [Sparse Data Optimization in xgboost.spark](/concepts/sparse-data-optimization-in-xgboostspark.md) - Performance optimization for datasets with sparse features by setting enable_sparse_data_optim=True and missing=0.0, requiring SparseVector feature columns.
* [SQL Alerts on MLflow Metadata](/concepts/sql-alerts-on-mlflow-metadata.md) - Using Databricks SQL alerts with MLflow system tables to monitor experiment reliability and be notified when constraints are violated.
* [SQL Function Allowlist for Shared Views](/concepts/sql-function-allowlist-for-shared-views.md) - The restricted subset of Databricks built-in SQL functions and operators permitted in view definitions shared via Databricks-to-Databricks sharing
* [SQL Warehouse Integration for Trace Search](/concepts/sql-warehouse-integration-for-trace-search.md) - Performance optimization using a Databricks SQL warehouse to accelerate queries on large trace datasets in inference or Unity Catalog tables.
* [SQL-only Table Access Control](/concepts/sql-only-table-access-control.md) - A restrictive version of table access control that limits users to SQL commands only, configured via Spark conf spark.databricks.acl.sqlOnly true.
* [SQLSTATE 0AKDC](/concepts/sqlstate-0akdc.md) - The SQLSTATE code associated with the DELTA_CLONE_INCOMPATIBLE_SOURCE error, categorized under class 0A (Feature Not Supported).
* [SQLSTATE 0AKDC (Feature Not Supported)](/concepts/sqlstate-0akdc-feature-not-supported.md) - A Databricks SQLSTATE class indicating a feature is not supported by the source or operation context.
* [SQLSTATE 42601](/concepts/sqlstate-42601.md) - A SQL state code indicating a syntax error or access rule violation, used as the parent error class for the DELTA_UNIFORM_COMPATIBILITY_MISSING_OR_INVALID_LOCATION error.
* [SQLSTATE 42616](/concepts/sqlstate-42616.md) - A SQL standard error code in Class 42 (Syntax Error or Access Rule Violation) used by the DELTA_STREAMING_SKIP_OFFSETS_INVALID_OFFSET_RANGE error condition in Databricks.
* [SQLSTATE 42704](/concepts/sqlstate-42704.md) - A SQL standard error code (class 42) indicating a syntax error or access rule violation, associated with undefined objects
* [SQLSTATE 42711](/concepts/sqlstate-42711.md) - A SQL standard error code for duplicate column definition violations, referenced by the DELTA_DUPLICATE_COLUMNS_FOUND error class in Databricks.
* [SQLSTATE 42711 (Databricks Error Code)](/concepts/sqlstate-42711.md) - The SQL standard error state code (class 42, syntax error or access rule violation) associated with the DELTA_DUPLICATE_COLUMNS_FOUND error in Databricks.
* [SQLSTATE 42711 (duplicate column error)](/concepts/sqlstate-42711.md) - The SQL standard error code 42711, classified under class 42 (syntax error or access rule violation), which maps to the Delta Lake duplicate column error condition.
* [SQLSTATE 42809](/concepts/sqlstate-42809.md) - The SQL standard state code for syntax error or access rule violation, associated in Databricks with the DELTA_ALTER_TABLE_UNSET_MANAGED_FAILED error class.
* [SQLSTATE 428GU](/concepts/sqlstate-428gu.md) - The SQL standard error code assigned to the DELTA_CANNOT_WRITE_EMPTY_SCHEMA error, classified under Class 42 (Syntax Error or Access Rule Violation).
* [SQLSTATE 55006](/concepts/sqlstate-55006.md) - ANSI/ISO SQL standard state 'object not in prerequisite state', which this Databricks error maps to, indicating the object is in a state that prevents the requested operation.
* [SQLSTATE 55006 — Object Not in Prerequisite State](/concepts/sqlstate-55006-object-not-in-prerequisite-state.md) - A SQL standard error class (class 55) indicating the target object cannot be modified because it is not in the required prerequisite state, used here when a shared securable is blocked from deletion.
* [SQLSTATE 55019](/concepts/sqlstate-55019.md) - A SQL state code (class 55: Object not in prerequisite state) associated with table migration failures in Databricks.
* [SQLSTATE 55019 (Object Not in Prerequisite State)](/concepts/sqlstate-55019-object-not-in-prerequisite-state.md) - A SQL standard state code indicating that an object is not in the prerequisite state required for the requested operation.
* [SQLSTATE class 0A (Feature not supported)](/concepts/sqlstate-class-0a-feature-not-supported.md) - A SQLSTATE error class indicating a feature is not supported, which is the parent class of the DELTA_TABLE_CHANGES_VIEW_UNSUPPORTED error.
* [SQLSTATE KD00C](/concepts/sqlstate-kd00c.md) - The SQL state class KD00C used for Datasource-specific errors in Databricks, under which the DELTA_VERSIONS_NOT_CONTIGUOUS error falls.
* [SQLSTATE KD00E](/concepts/sqlstate-kd00e.md) - A Databricks-specific SQLSTATE error class for datasource-specific errors, used as the parent classification of DELTA_ICEBERG_WRITER_COMPAT_VIOLATION
* [SSL/TLS Certificate Configuration for Databricks Connect](/concepts/ssltls-certificates-for-databricks-connect.md) - Configuration of the GRPC_DEFAULT_SSL_ROOTS_FILE_PATH environment variable to support custom SSL/TLS certificates for Databricks workspace FQDN resolution.
* [SSL/TLS Certificates for Databricks Connect](/concepts/ssltls-certificates-for-databricks-connect.md) - Configuring custom SSL/TLS certificates for Databricks Connect by setting the `GRPC_DEFAULT_SSL_ROOTS_FILE_PATH` environment variable to the certificate path on the local development machine.
* [Stage-based Model Version Serving](/concepts/stage-based-model-version-serving.md) - A migration pattern that creates separate serving endpoints for Staging and Production model versions, aligned with the MLflow model registry stage concept.
* [Stage-Level Scheduling for GPU Parallelism](/concepts/stage-level-scheduling-for-gpu-parallelism.md) - Using Spark's ResourceProfile and stage-level scheduling to control the number of tasks running per GPU for increased parallelism
* [Staging and Testing ML Models](/concepts/staging-and-testing-ml-models.md) - The practice of testing model versions in staging with integration tests, A/B or shadow tests on production data, and stakeholder sign-off before promoting to production using model aliases.
* [Standard Access Mode](/concepts/standard-access-mode.md) - A Databricks cluster access mode that enables table access control by default.
* [Standard Access Mode Compute](/concepts/standard-access-mode-compute.md) - A Databricks compute access mode (formerly shared access mode) where the allowlist restricts what libraries and init scripts can run, using the library installer's identity for libraries and the cluster owner's identity for init scripts.
* [Standard access mode compute (Databricks)](/concepts/standard-access-mode-compute.md) - A cluster access mode where identity of the library installer or cluster owner is used for access control; allowlisted libraries and init scripts are required to run on this mode with Unity Catalog.
* [Standard Access Mode Compute in Unity Catalog](/concepts/standard-access-mode-compute-in-unity-catalog.md) - A compute access mode (formerly shared access mode) that enforces user isolation and requires allowlisted libraries and init scripts for non-default artifacts.
* [Standard Access Mode for Clusters](/concepts/standard-access-mode.md) - A Databricks cluster access mode that enables table access control by default, as opposed to legacy modes that require explicit configuration.
* [Standardized Metadata Fields for MLflow Traces](/concepts/standardized-metadata-fields-for-traces.md) - Predefined metadata keys like mlflow.trace.user, mlflow.trace.session, and mlflow.source.type that MLflow recognizes for filtering, grouping, and UI features in trace analysis.
* [Standardized Metadata Fields for Traces](/concepts/standardized-metadata-fields-for-traces.md) - Predefined metadata field names (like mlflow.trace.user, mlflow.trace.session, mlflow.source.type) that MLflow recognizes for automatic UI filtering, grouping, and environment detection.
* [Starting version and timestamp for Delta streaming sources](/concepts/starting-version-and-timestamp-for-streaming-reads.md) - Options (startingVersion, startingTimestamp) to control the starting point of Delta streaming reads without processing the entire table history.
* [Starting Version and Timestamp for Streaming Reads](/concepts/starting-version-and-timestamp-for-streaming-reads.md) - Options (startingVersion, startingTimestamp) to control the initial table version from which a Delta Lake streaming source begins processing changes.
* [State management in Scorer class](/concepts/state-management-in-scorer-class.md) - When using the Scorer class, state must be managed with instance attributes (not mutable class attributes) to avoid sharing state across scorer instances.
* [Stateful Scorers in Production](/concepts/stateful-scorers-in-production.md) - Technique for implementing stateful scorers by managing state inside the body of a @scorer-decorated function, since class-based Scorer subclasses are not supported.
* [Stateful scorers in production using @scorer](/concepts/stateful-scorers-in-production.md) - For stateful scoring logic in production, use the @scorer decorator and manage state inside the function body instead of class-based subclasses.
* [Stateful scorers with @scorer](/concepts/stateful-scorers-with-scorer.md) - To implement stateful scorers in production, use the @scorer decorator and manage state inside the function body, since class-based approaches are unsupported
* [Stateful scorers with @scorer decorator](/concepts/stateful-scorers-with-scorer-decorator.md) - For stateful scorers in production, use the @scorer decorator and manage state inside the function body
* [Static Dependency Capture Principle](/concepts/static-dependency-capture-principle.md) - The principle that real-time model serving workloads perform best when all file and artifact dependencies are statically packaged into the model at deployment time, rather than loaded dynamically from networks or external sources.
* [Statistical Parity](/concepts/statistical-parity.md) - A fairness metric that measures the difference in predicted outcomes (selection rates) between protected and unprotected groups
* [Statistics Collection during Delta Conversion](/concepts/statistics-collection-during-delta-conversion.md) - The process of collecting statistics during CONVERT TO DELTA to improve query performance, which can be bypassed with the NO STATISTICS option.
* [Stopping the PySpark Shell](/concepts/stopping-the-pyspark-shell.md) - Multiple methods to terminate the Databricks Connect PySpark shell including Ctrl+D, Ctrl+Z, quit(), or exit().
* [Storage Billing After Object Deletion](/concepts/storage-billing-after-object-deletion.md) - Databricks bills for default storage only during the 7-day recovery window; for customer-provided storage the cloud provider bills directly and charges may persist based on bucket policies.
* [Storage Credential (IAM Role) for Unity Catalog](/concepts/storage-credential-iam-role-for-unity-catalog.md) - An IAM role created in AWS that grants Unity Catalog access to an S3 bucket used for managed storage, requiring cross-account trust relationship policies.
* [Storage path conflict](/concepts/storage-path-conflict.md) - A conflict that occurs when an external location overlaps with the default Unity Catalog storage path of any workspace in a Databricks account, interfering with internal processing and data governance.
* [Streaming Data Approaches for Distributed Training](/concepts/streaming-data-approaches-for-distributed-training.md) - Methods for handling very large datasets that do not fit in memory when training distributed machine learning models, including PyTorch IterableDataset, Hugging Face datasets streaming, and Ray Data.
* [Streaming Data for Distributed Deep Learning](/concepts/streaming-data-for-distributed-deep-learning.md) - Streaming approaches for very large datasets that do not fit in memory, including PyTorch IterableDataset, Hugging Face Datasets streaming, and Ray Data.
* [Streaming Datasets on AI Runtime](/concepts/streaming-datasets-on-ai-runtime.md) - Strategies for handling very large datasets that don't fit in memory, including UCVolumeDataset, PyTorch IterableDataset, Hugging Face datasets streaming, and Ray Data.
* [Streaming Feature Computation Pipelines](/concepts/streaming-feature-computation-pipelines.md) - Feature tables can be updated incrementally using streaming DataFrames with `write_table` in merge mode, supporting real-time feature computation.
* [Streaming Feature Publishing](/concepts/streaming-feature-publishing.md) - Continuous streaming of features from Databricks Feature Store to an online store by setting the streaming=True parameter in publish_table().
* [Streaming from Delta Lake Tables with skipChangeCommits](/concepts/streaming-from-delta-lake-tables-with-skipchangecommits.md) - The skipChangeCommits option allows streaming queries to ignore upstream modifications (UPDATE, DELETE, MERGE, OVERWRITE) on a source Delta table, processing only append transactions.
* [Streaming Offset Event Time Restriction](/concepts/streaming-offset-event-time-restriction.md) - The constraint that streaming offsets used in Delta skip offset ranges must not contain event time (eventTimeMillis must be empty)
* [Streaming Output Tracing](/concepts/streaming-output-tracing.md) - Tracing of generator and iterator functions with @mlflow.trace, where spans start when the iterator is consumed and end when exhausted.
* [Streaming Response Feedback Collection](/concepts/streaming-response-feedback-collection.md) - Technique for collecting user feedback when using SSE/WebSocket streaming responses, where the trace ID is only available after stream completion
* [Streaming Response Feedback Handling](/concepts/streaming-response-feedback-handling.md) - Special approach for collecting user feedback on streaming responses where trace IDs are only available after stream completion, requiring deferred feedback UI design.
* [Streaming Table Default Publishing Mode Requirement](/concepts/streaming-table-default-publishing-mode-requirement.md) - Only streaming tables using the default publishing mode can be deep cloned; older architectures are not supported
* [Streaming table managed storage](/concepts/streaming-table-managed-storage.md) - Streaming tables in Databricks use managed storage, preventing specification of a custom LOCATION during deep clone operations.
* [Streaming Table Managed Storage Location Restriction](/concepts/streaming-table-managed-storage.md) - Specifying a custom LOCATION when deep cloning a streaming table is not supported because the cloned table uses managed storage
* [Streaming Table Publishing Modes](/concepts/streaming-table-publishing-modes.md) - Different publishing modes for streaming tables in Databricks, where only the default mode supports deep clone operations
* [Streaming Table Sharing](/concepts/streaming-table-sharing.md) - Sharing streaming tables (append-only Delta tables with incremental processing) with restrictions on partition filters, CDF support for open recipients, and predicate pushdown.
* [Streaming Tables and Materialized Views as Feature Tables](/concepts/streaming-tables-and-materialized-views-as-feature-tables.md) - Streaming tables and materialized views created by Lakeflow Spark Declarative Pipelines with primary key constraints can be used as feature tables.
* [Streaming tables in Databricks](/concepts/streaming-tables-in-databricks.md) - Incrementally updated Delta tables that ingest streaming data, one of the two table types supported for external metadata.
* [Streaming Trace Behavior in MLflow](/concepts/streaming-trace-behavior-in-mlflow.md) - MLflow tracing for streaming Bedrock calls (converse_stream, invoke_model_with_response_stream) creates a span only when chunks are consumed (e.g., iterated in a for-loop), not when the response object is first returned.
* [Streaming Trace Support for Foundation Models](/concepts/streaming-trace-support-for-foundation-models.md) - MLflow Tracing supports streaming API responses from Databricks Foundation Models, automatically capturing and rendering the concatenated streaming output in the span UI.
* [Streaming Upserts with foreachBatch and MERGE](/concepts/streaming-upserts-with-foreachbatch-and-merge.md) - Using MERGE operations inside foreachBatch to continuously apply streaming changes (upserts) to Delta Lake tables with idempotency guarantees
* [StreamingDataLoader](/concepts/streamingdataloader.md) - A PyTorch DataLoader variant that provides an additional checkpoint/resumption interface by tracking the number of samples seen by the model per rank.
* [StreamingDataset](/concepts/streamingdataset.md) - A PyTorch IterableDataset variant that features elastically deterministic shuffling, enabling fast mid-epoch resumption during distributed training.
* [STRUCT_ALL_VOID_FIELDS](/concepts/struct_all_void_fields.md) - A sub-reason for DELTA_CANNOT_WRITE_EMPTY_SCHEMA where a struct column has all fields of VOID type.
* [STRUCT_ALL_VOID_FIELDS error condition](/concepts/struct_all_void_fields.md) - A sub-error of DELTA_CANNOT_WRITE_EMPTY_SCHEMA where a struct column has all fields of VOID type
* [STRUCT_NO_FIELDS](/concepts/struct_no_fields.md) - A sub-reason for DELTA_CANNOT_WRITE_EMPTY_SCHEMA where a struct column has no fields defined.
* [STRUCT_NO_FIELDS error condition](/concepts/struct_no_fields.md) - A sub-error of DELTA_CANNOT_WRITE_EMPTY_SCHEMA where a struct column has zero fields defined
* [Structured Outputs in Foundation Model APIs](/concepts/structured-outputs-in-foundation-model-apis.md) - A feature allowing specification of output format via JSON Schema in Databricks Foundation Model API requests, supported across both Responses and Chat Completions APIs.
* [Structured RAG with Online Tables](/concepts/structured-rag-with-online-tables.md) - Using Databricks online tables to serve structured data (e.g., customer metadata, product catalogs) as real-time context for retrieval-augmented generation applications.
* [Structured Streaming on Shared Tables](/concepts/structured-streaming-on-shared-tables.md) - Using Spark Structured Streaming with OpenSharing shared tables as a source for incremental processing, requiring history sharing to be enabled.
* [STS Temporary Credentials for Data Access](/concepts/sts-temporary-credentials-for-data-access.md) - Model serving endpoints use AWS STS temporary security credentials to authenticate data access through the instance profile; these credentials cannot bypass network restrictions.
* [STS Temporary Credentials for Model Data Access](/concepts/sts-temporary-credentials-for-model-data-access.md) - AWS STS temporary security credentials are used to authenticate data access from model serving endpoints, but they cannot bypass network restrictions.
* [STS Temporary Credentials for Model Serving](/concepts/sts-temporary-credentials-for-data-access.md) - AWS Security Token Service (STS) temporary credentials are used by Databricks model serving endpoints to authenticate data access through an instance profile, but cannot bypass network restrictions.
* [STS Temporary Credentials for Model Serving Data Access](/concepts/sts-temporary-credentials-for-model-data-access.md) - AWS STS temporary security credentials are used to authenticate data access from model serving endpoints, but they cannot bypass network restrictions.
* [STS Token Sharing](/concepts/sts-token-sharing.md) - A Delta Sharing mechanism using scoped-down AWS STS tokens for temporary read access, logged as generateTemporaryTableCredentials and generateTemporaryVolumeCredentials events
* [STS Token Sharing Audit Details](/concepts/sts-token-sharing-audit-details.md) - Audit log fields for STS token-based sharing, including generateTemporaryTableCredentials and generateTemporaryVolumeCredentials events with credential and table metadata.
* [STS Token Sharing Audit Events](/concepts/sts-token-sharing-audit-events.md) - Audit events (generateTemporaryTableCredentials, generateTemporaryVolumeCredentials) logged when data recipients query shared data via scoped-down STS token-based sharing, with request_params details.
* [STS Token Sharing in OpenSharing](/concepts/sts-token-sharing.md) - A credential mechanism for OpenSharing that uses scoped-down STS tokens for temporary data access, logged through generateTemporaryTableCredentials and generateTemporaryVolumeCredentials events.
* [SUBQUERY constraint for table_changes](/concepts/subquery-constraint-for-table_changes.md) - A validation rule requiring that views used with table_changes must not contain subqueries
* [Subquery Restrictions in DELETE WHERE](/concepts/subquery-restrictions-in-delete-where.md) - Restrictions on subquery types within DELETE WHERE predicates, including nested subqueries and NOT IN within OR
* [Subquery Restrictions in DML Statements](/concepts/subquery-restrictions-in-dml-statements.md) - DELETE and other DML statements in Databricks SQL forbid nested subqueries and NOT IN subqueries inside OR expressions.
* [Subquery Restrictions in UPDATE WHERE Clause](/concepts/subquery-restrictions-in-update-where-clause.md) - Limitations on subqueries in UPDATE WHERE clauses: nested subqueries and NOT IN subqueries inside OR are not allowed; NOT IN can be rewritten as NOT EXISTS for better performance.
* [Subquery support in Databricks DELETE WHERE clause](/concepts/subquery-support-in-databricks-delete-where-clause.md) - The WHERE predicate in DELETE FROM supports subqueries including IN, NOT IN, EXISTS, NOT EXISTS, and scalar subqueries, with specific restrictions on nested subqueries and NOT IN inside OR expressions.
* [Subquery Support in DELETE WHERE Clause](/concepts/subquery-support-in-delete-where-clause.md) - Types of subqueries allowed (IN, NOT IN, EXISTS, NOT EXISTS, scalar) and not allowed (nested, NOT IN inside OR) in Delta Lake DELETE statements.
* [Subset Caching with CACHE SELECT](/concepts/subset-caching-with-cache-select.md) - Technique of caching only a subset of columns and/or rows from a table by specifying column names and a WHERE predicate in CACHE SELECT.
* [suggest_float with log scale](/concepts/suggest_float-with-log-scale.md) - Optuna method for suggesting continuous float hyperparameters, optionally on a logarithmic scale for wide-ranging values
* [Supervised Fine-Tuning (SFT)](/concepts/supervised-fine-tuning-sft.md) - A training technique that fine-tunes a pre-trained language model on labeled instruction-following data to improve task performance
* [Supervised Fine-Tuning (SFT) of Instruction-Tuned LLMs](/concepts/supervised-fine-tuning-sft-of-instruction-tuned-llms.md) - A training technique where a causal language model is fine-tuned on instruction-response pairs (e.g., Alpaca dataset) by tokenizing prompt-response sequences with EOS tokens and training with standard causal LM loss, often using FSDP for 8B+ parameter models.
* [Supervised Fine-Tuning (SFT) of LLMs](/concepts/supervised-fine-tuning-sft.md) - A training approach where a pre-trained language model is further trained on labeled conversational data to improve its performance on specific tasks, using techniques like bfloat16 precision, cosine learning rate scheduling, and gradient accumulation.
* [Supervised Fine-Tuning (SFT) with ChatML](/concepts/supervised-fine-tuning-sft-with-chatml.md) - The process of fine-tuning a causal language model on structured conversational datasets using the ChatML format for instruction-following behavior.
* [Supervised Fine-Tuning (SFT) with SFTTrainer](/concepts/supervised-fine-tuning-sft-with-trl.md) - Training paradigm using the TRL library's SFTTrainer to fine-tune language models on instruction-following datasets, with configurable hyperparameters and MLflow tracking.
* [Supervised Fine-Tuning (SFT) with TRL](/concepts/supervised-fine-tuning-sft-with-trl.md) - The training paradigm used to fine-tune language models on instruction-following datasets, implemented via the SFTTrainer class from the transformers reinforcement learning (TRL) library.
* [Supervised Fine-Tuning (SFT) with TRL Library](/concepts/supervised-fine-tuning-sft-with-trl.md) - A technique for fine-tuning large language models using the TRL SFTTrainer, which provides tools for supervised fine-tuning and reinforcement learning training of LLMs.
* [Supervised Fine-Tuning (SFT) with TRL SFTTrainer](/concepts/supervised-fine-tuning-sft-with-trl.md) - A supervised fine-tuning method for language models using the Transformer Reinforcement Learning (TRL) library's SFTTrainer, which streamlines the process of fine-tuning causal language models on instruction-following datasets.
* [Supervised Fine-Tuning (SFT) Workflow](/concepts/supervised-fine-tuning-sft-workflow.md) - End-to-end pipeline for fine-tuning a causal language model on a conversational dataset, including loading data, applying chat templates, configuring training hyperparameters, and saving artifacts.
* [Supported Bedrock APIs for MLflow Tracing](/concepts/supported-bedrock-apis-for-mlflow-tracing.md) - MLflow supports auto-tracing for four Bedrock APIs: converse, converse_stream, invoke_model, and invoke_model_with_response_stream.
* [Supported Data Feature Types for AutoML](/concepts/supported-feature-types-in-automl.md) - The data types that Databricks AutoML supports for classification, including numeric, boolean, string, timestamps, arrays, and decimal types.
* [Supported Data Feature Types for AutoML Forecasting](/concepts/supported-feature-types-in-automl.md) - The specific data types (numeric, boolean, string, timestamp, array, decimal) that AutoML accepts for forecasting tasks on Databricks.
* [Supported Data Feature Types in AutoML](/concepts/supported-data-feature-types-in-automl.md) - The data types supported by AutoML for training, including numeric, Boolean, string, timestamp, array, and decimal types, with exclusions like images.
* [Supported Data Types for Feature Lookup](/concepts/supported-data-types-for-feature-lookup.md) - Automatic feature lookup supports IntegerType, FloatType, BooleanType, StringType, DoubleType, LongType, TimestampType, DateType, ShortType, ArrayType, and MapType.
* [Supported Data Types for Feature Stores](/concepts/supported-data-types-for-feature-stores.md) - Databricks Feature Store supports PySpark data types including IntegerType, FloatType, ArrayType, MapType, StructType, and others, with specific version requirements for complex types.
* [Supported Feature Data Types](/concepts/supported-feature-data-types.md) - The set of PySpark data types supported by Databricks Feature Store for storing features such as vectors, tensors, embeddings, and text.
* [Supported Feature Types for AutoML](/concepts/supported-feature-types-in-automl.md) - The set of data types supported by Databricks AutoML including numeric types, Boolean, String, timestamps, ArrayType of Numeric, and DecimalType, with images explicitly unsupported.
* [Supported Feature Types for Forecasting](/concepts/supported-feature-types-in-automl.md) - The set of data types (numeric, boolean, string, timestamps, arrays, decimals) supported by Databricks AutoML for forecasting tasks.
* [Supported Feature Types in AutoML](/concepts/supported-feature-types-in-automl.md) - The set of data types (numeric, boolean, string, timestamp, array, decimal) that AutoML supports for classification feature columns.
* [Supported Feature Types in Databricks AutoML](/concepts/supported-feature-types-in-automl.md) - The exhaustive list of Spark data types (numeric, boolean, string, timestamp, array, decimal) that AutoML accepts as feature columns.
* [Supported Frameworks for Databricks Autologging](/concepts/supported-ml-frameworks-for-autologging.md) - The list of ML frameworks (scikit-learn, Spark MLlib, TensorFlow, Keras, PyTorch Lightning, XGBoost, LightGBM, Gluon, Fast.ai, statsmodels, PaddlePaddle, OpenAI, LangChain) for which Databricks Autologging natively captures training metrics.
* [Supported ML Frameworks for Autologging](/concepts/supported-ml-frameworks-for-autologging.md) - The list of machine learning frameworks compatible with Databricks Autologging, including scikit-learn, TensorFlow, PyTorch Lightning, XGBoost, and others.
* [Supported ML Frameworks for Databricks Autologging](/concepts/supported-ml-frameworks-for-autologging.md) - The comprehensive list of machine learning frameworks supported by Databricks Autologging including scikit-learn, Apache Spark MLlib, TensorFlow, Keras, PyTorch Lightning, XGBoost, LightGBM, Gluon, Fast.ai, statsmodels, PaddlePaddle, OpenAI, and LangChain.
* [Supported Model Providers and Architectures](/concepts/supported-model-providers-and-architectures.md) - The range of third-party model providers available through Databricks Foundation Model APIs including OpenAI, Google (Gemini), Anthropic (Claude), Meta (Llama), Alibaba Cloud (Qwen), and specialized embedding model providers.
* [Supported Models for Function Calling on Databricks](/concepts/supported-models-for-function-calling-on-databricks.md) - A curated list of models (Meta Llama, Mixtral, Claude, Gemini, and Databricks DBRX) available through Foundation Model APIs or External Models for function calling.
* [Supported Object Types for Unity Catalog Certification Tags](/concepts/supported-object-types-for-unity-catalog-certification-tags.md) - The list of Unity Catalog objects that support the certification status tag, including catalogs, schemas, tables, views, volumes, functions, registered models, dashboards, Genie Spaces, apps, and notebooks.
* [Supported PySpark Data Types for Features](/concepts/supported-pyspark-data-types-for-features.md) - The list of PySpark data types (IntegerType, FloatType, BooleanType, StringType, DoubleType, LongType, TimestampType, DateType, ShortType, ArrayType, BinaryType, DecimalType, MapType, StructType) supported by Databricks Feature Store for storing feature values.
* [Supported PySpark Data Types in Feature Store](/concepts/supported-data-types-for-feature-stores.md) - Feature Store supports PySpark types including IntegerType, FloatType, ArrayType, MapType, StructType, and others, with ArrayType and MapType stored as JSON in online stores.
* [Supported PySpark UDF Function Types](/concepts/supported-pyspark-udf-function-types.md) - Databricks Connect supports a wide range of PySpark UDF functions including scalar UDFs, pandas UDFs, UDTFs, mapInPandas/mapInArrow, grouped apply functions, and streaming foreach/foreachBatch.
* [Supported scenarios and limitations of Databricks Feature Engineering](/concepts/supported-scenarios-and-limitations-of-databricks-feature-engineering.md) - Scenarios where the Feature Engineering client can be used (Databricks Runtime) and where it cannot (local/external environments)
* [Supported SQL Functions for Shared Views](/concepts/supported-sql-functions-for-shared-views.md) - The comprehensive list of SQL operators, mathematical functions, string functions, date/time functions, array/map functions, aggregation functions, window functions, encryption functions, geospatial (H3) functions, and type conversion functions allowed in Databricks-to-Databricks view sharing.
* [Supported Table Types for External Metadata in Databricks](/concepts/supported-table-types-for-external-metadata-in-databricks.md) - Only streaming tables and materialized views are supported for external metadata operations; other table types trigger the TABLE_TYPE error.
* [Supported Tool Types for Pay-Per-Token Models](/concepts/supported-tool-types-for-pay-per-token-models.md) - The complete list of tool types supported by pay-per-token foundation models via the Responses API: function, custom, apply_patch, shell, image_generation, mcp, and web_search.
* [Suspended Custom Classifier](/concepts/suspended-custom-classifier.md) - A state where a custom classifier stops producing detections due to rule generation failure, invalid example columns, or invalid governed tags; requires editing or recreation to resolve.
* [Symlink format manifest](/concepts/symlink-format-manifest.md) - A manifest format that uses symlinks to allow Presto/Athena to read Delta tables by generating a list of Parquet files.
* [Symlink Manifest Generation](/concepts/symlink-manifest-generation.md) - A Delta Lake feature for generating symlink manifests for external query engines; mutually exclusive with deletion vectors in several configurations
* [Symlink Manifest Generation (Incremental)](/concepts/symlink-manifest-generation.md) - A mechanism for generating symlink manifests in Delta Lake, used for integrating with external query engines, which has constraints when combined with deletion vectors.
* [symlink_format_manifest mode](/concepts/symlink_format_manifest-mode.md) - A mode for the GENERATE command that produces manifest files enabling Presto and Athena to read Delta tables.
* [SYNC Command for Hive-to-Unity-Catalog Migration](/concepts/sync-command-for-hive-to-unity-catalog-migration.md) - SQL command (SYNC TABLE / SYNC SCHEMA) that copies Hive metastore external tables to Unity Catalog external tables, with support for ongoing synchronization when source tables change.
* [Syncing Expectations to Evaluation Datasets](/concepts/syncing-expectations-to-evaluation-datasets.md) - A workflow that synchronizes collected Expectation labels from a labeling session to an Evaluation Dataset via an intelligent upsert, using trace inputs as unique keys.
* [Synthetic Data Generation for Evaluation](/concepts/synthetic-data-generation-for-evaluation.md) - Databricks' capability to automatically generate a representative evaluation set from documents, enabling quick evaluation of agents with good coverage of test cases without requiring pre-existing labeled data.
* [Synthetic Data Generation for GenAI Evaluation](/concepts/synthetic-data-generation-for-evaluation.md) - Databricks can automatically generate representative evaluation datasets from documents, enabling quick evaluation of GenAI agents with broad test case coverage without manual curation.
* [Synthetic Evaluation Data Generation](/concepts/synthetic-evaluation-data-generation.md) - Databricks' ability to automatically generate a representative evaluation set from source documents, enabling quick coverage of test cases without manually curating examples.
* [Synthetic Evaluation Set Generation (Databricks)](/concepts/synthetic-evaluation-set-generation-databricks.md) - A serverless workload that generates synthetic evaluation datasets for GenAI agent evaluation, controlled via budget policies on experiments.
* [System Tables for Audit Logs](/concepts/system-tables-for-audit-logs.md) - The system.access.audit table where audit logs are stored when system tables are enabled, requiring account admin or granted permissions to read
* [System Tables for Unity Catalog Audit](/concepts/system-tables-for-unity-catalog-audit.md) - The system.access.audit system table requirement for accessing OpenSharing audit logs, requiring account admin enablement and appropriate permissions for non-admin users.
* [System Tags](/concepts/system-tags.md) - Predefined tags provided by Databricks (e.g., ai.model_creator) that can be referenced in GRANT policy conditions alongside user-created governed tags.
* [System Tags (Databricks)](/concepts/system-tags-databricks.md) - Tags automatically assigned by the Databricks system to columns when sensitive data is detected, such as those indicating PII or compliance-related data.
* [system.ai Schema](/concepts/systemai-schema.md) - The Unity Catalog schema under the system catalog where foundation model objects reside and EXECUTE permissions are managed to control model access.
* [System.ai Schema Default EXECUTE Permission](/concepts/systemai-schema-default-execute-permission.md) - The default permission model where all users have EXECUTE on the system.ai schema, granting access to all Databricks-hosted foundation models until an admin revokes it.
* [system.ai Schema Permission Model](/concepts/systemai-schema.md) - The Unity Catalog schema (system.ai) that governs access to all Databricks-hosted foundation models through EXECUTE permissions, applied consistently across pay-per-token, provisioned throughput, and batch inference workloads.
* [system.billing.usage system table](/concepts/systembillingusage-system-table.md) - A Databricks system table used to query billable usage records, including data quality monitoring costs.
* [system.data_classification.mask_value](/concepts/systemdata_classificationmask_value.md) - A built-in, type-aware masking UDF in Databricks that returns safe placeholder values based on column data types (e.g., 0 for integers, SHA-256 hash for strings, DATE '1970-01-01' for dates).
* [system.data_classification.results System Table](/concepts/data-classification-system-table.md) - A system table that stores all classification results across the entire metastore, including sample values from tables, accessible only to account admins by default and requiring serverless compute.
* [system.data_quality_monitoring.table_results](/concepts/systemdata_quality_monitoringtable_results.md) - A system table in Databricks that stores anomaly detection output results, including event time, table status, freshness metrics, completeness metrics, and downstream impact information.
* [system.data_quality_monitoring.table_results System Table](/concepts/systemdata_quality_monitoringtable_results.md) - The Databricks system table that stores anomaly detection output results, containing fields like event_time, catalog_name, schema_name, table_name, status, freshness metrics, completeness metrics, and downstream impact data.
* [system.mlflow.experiments_latest](/concepts/systemmlflowexperiments_latest.md) - A system table that records experiment names, soft-deletion events, and lifecycle information for MLflow experiments.
* [system.mlflow.run_metrics_history](/concepts/systemmlflowrun_metrics_history.md) - A system table that records the name, value, timestamp, and step of all metrics logged on runs, enabling detailed timeseries analysis.
* [system.mlflow.runs_latest](/concepts/systemmlflowruns_latest.md) - A system table that records run-lifecycle information, params, tags, and aggregated metric statistics (min, max, latest) for MLflow runs.
* [Table Access Control (Databricks)](/concepts/table-access-control-databricks.md) - A workspace-level and cluster-level setting that must be enabled to enforce Hive metastore privileges on Databricks clusters and SQL warehouses.
* [Table Access Control (TACL)](/concepts/table-access-control-tacl.md) - A workspace-level and cluster-level setting that must be enabled for Hive metastore privilege enforcement in Databricks.
* [Table Access Control Cluster Configuration](/concepts/table-access-control-cluster-configuration.md) - The per-cluster toggle that controls whether Hive metastore table access control is enforced, with default being open access to all data.
* [Table ACLs Blocking Delta Schema Evolution](/concepts/table-acls-blocking-delta-schema-evolution.md) - A restriction where enabling Table ACLs on a cluster disables automatic schema migration for Delta tables, requiring explicit ALTER TABLE commands.
* [Table alias in DELETE](/concepts/table-alias-in-delete.md) - DELETE FROM supports an optional table alias that must not include a column list, used to reference the target table in subqueries.
* [Table Alias in DELETE FROM](/concepts/table-alias-in-delete-from.md) - Using table aliases in DELETE FROM statements to simplify references in subqueries
* [Table aliases in Databricks DML statements](/concepts/table-aliases-in-delete-statements.md) - Databricks SQL allows defining a table alias in DELETE FROM statements, but the alias must not include a column list, enabling clearer references in subqueries and complex predicates.
* [Table Aliases in DELETE Statements](/concepts/table-aliases-in-delete-statements.md) - Databricks SQL allows optional table aliasing in DELETE FROM syntax but the alias must not include a column list.
* [Table deletion and recreation at same S3 location](/concepts/table-deletion-and-recreation-at-same-s3-location.md) - A specific operational pattern on AWS S3 that can lead to Delta log gaps due to S3's eventual consistency model when a table is dropped and re-created at the same path.
* [Table History Sharing](/concepts/table-history-sharing.md) - Sharing a table's history (Delta log) in a share to allow recipients to perform time travel queries, read with Spark Structured Streaming, run transactions, and improve performance for Databricks-to-Databricks shares. Requires Databricks Runtime 12.2 LTS or above. Required when sharing tables with deletion vectors or column mapping.
* [Table Metadata and Protocol Change Conflicts](/concepts/table-metadata-and-protocol-change-conflicts.md) - Conflict scenarios where concurrent operations change table metadata (schema, partitioning) or upgrade the table protocol, triggering the DELTA_CONCURRENT_APPEND error.
* [Table Metadata Change Transaction Conflicts](/concepts/table-metadata-change-transaction-conflicts.md) - A conflict type where concurrent operations that change table metadata (schema, partitioning, protocol) cause read transactions to fail, typically resolved by retrying the operation.
* [Table Metadata Conflicts in Delta Lake](/concepts/table-metadata-conflicts-in-delta-lake.md) - Conflicts arising from concurrent schema or partitioning changes to a Delta table while a transaction is in progress, requiring transaction retry.
* [Table Name Restrictions in DESCRIBE HISTORY](/concepts/table-name-restrictions-in-describe-history.md) - The table name in DESCRIBE HISTORY must not include a temporal specification or options specification.
* [Table Protocol Upgrade Conflicts](/concepts/table-protocol-upgrade-conflicts.md) - A conflict condition where a concurrent operation upgraded the table protocol, invalidating the current transaction's assumptions and requiring a retry.
* [Table provenance tracking](/concepts/table-provenance-tracking.md) - Delta Lake tracks metadata for each write operation including the operation type, user, and other provenance information.
* [Table provenance tracking in Delta Lake](/concepts/table-provenance-tracking.md) - Delta Lake automatically records operation type, user identity, and other metadata for every write to a table
* [Table Quality Details (Databricks)](/concepts/table-quality-details-databricks.md) - A detailed view showing trends and anomaly reasons for specific tables, including predicted vs observed values and upstream root cause analysis
* [Table Quality Details UI](/concepts/table-quality-details-ui.md) - A detailed view per table showing trend graphs of predicted vs observed values for each quality check, along with identified upstream root causes for failures.
* [Table Types in Unity Catalog](/concepts/table-types-in-unity-catalog.md) - Unity Catalog supports managed tables (storage managed by Unity Catalog), external tables (user-specified storage path), and foreign tables (from foreign catalogs), each with different lifecycle and governance characteristics.
* [TABLE_ALL_VOID_COLUMNS](/concepts/table_all_void_columns.md) - A sub-reason for DELTA_CANNOT_WRITE_EMPTY_SCHEMA where every column in the table schema has VOID type.
* [TABLE_ALL_VOID_COLUMNS error condition](/concepts/table_all_void_columns.md) - A sub-error of DELTA_CANNOT_WRITE_EMPTY_SCHEMA where every column in the table schema is of VOID type
* [table_changes function](/concepts/table_changes-function.md) - A Delta Lake function that reads Change Data Feed (CDF) records, but is unsupported on views.
* [table_changes function (Databricks)](/concepts/table_changes-function.md) - A Databricks function used to retrieve change data feed (CDF) records from Delta tables, which is unsupported on views.
* [table_name restrictions for DESCRIBE HISTORY](/concepts/table-name-restrictions-in-describe-history.md) - The table name used with DESCRIBE HISTORY must not include a temporal specification or options specification.
* [TABLE_NO_COLUMNS](/concepts/table_no_columns.md) - A sub-reason for DELTA_CANNOT_WRITE_EMPTY_SCHEMA where the table schema has no columns defined at all.
* [TABLE_NO_COLUMNS error condition](/concepts/table_no_columns.md) - A sub-error of DELTA_CANNOT_WRITE_EMPTY_SCHEMA where the table schema has no columns defined at all
* [Table-Level Filtering and Masking](/concepts/table-level-filtering-and-masking.md) - Fine-grained data access controls using row filters and column masks applied via UDFs to control what data users see within tables at query time.
* [Table-Level Row Filters and Column Masks](/concepts/row-filters-and-column-masks.md) - Per-table UDF-based filters and masks that control what data users see at query time within tables.
* [Table-prefix Format Benefits](/concepts/table-prefix-format-benefits.md) - The table-prefix UC trace format provides faster time-range queries, richer attribute types, a dedicated annotations table, and support for multiple trace destinations per schema compared to the older schema-linked format
* [Table-wide vs filtered Delta operations](/concepts/table-wide-vs-filtered-delta-operations.md) - A category of transaction conflicts caused by operations that modify an entire table (no filter predicates) rather than a subset of data, including WHOLE_TABLE_READ, WHOLE_TABLE_REPLACE, and EMPTY_READ_PREDICATES sub-errors.
* [Table-wide vs Filtered Operations in Delta Lake](/concepts/table-wide-vs-filtered-operations-in-delta-lake.md) - The distinction between operations that modify an entire table versus filtered subsets, where whole-table modifications are more prone to concurrent transaction conflicts.
* [Tag Inheritance](/concepts/tag-inheritance.md) - Unity Catalog securables inherit governed tags from parent catalogs and schemas by default, but column tags do not inherit from parent tables and must be applied directly.
* [Tag Mutability vs Metadata Immutability](/concepts/tag-mutability-vs-metadata-immutability.md) - Design distinction in MLflow tracing where tags are mutable (updateable after logging) while metadata is write-once and immutable once set.
* [Tag-based access gating](/concepts/tag-based-access-gating.md) - A governance pattern that blocks access to untagged or unclassified data using a default restrictive tag, then applies masking policies once classification tags are applied.
* [Tag-based access gating (unverified classification pattern)](/concepts/tag-based-access-gating.md) - A governance pattern that blocks access to untagged or unclassified data using a default restrictive tag, then applies column masking after classification is complete, using has_tag_value conditions in policies.
* [Tag-based access gating workflow](/concepts/tag-based-access-gating.md) - A governance pattern that uses a default restrictive tag (e.g. 'classification : unverified') to block access to unclassified data, then transitions to column masking once a data steward completes classification and updates the tag.
* [Tag-based metadata in Unity Catalog](/concepts/tag-based-metadata-in-unity-catalog.md) - Tags for labeling models and model versions in Unity Catalog, supporting search in Catalog Explorer (up to 50 tags per object), replacing stage labels from the Workspace Model Registry.
* [Tag-based Policy Conditions and Built-in Functions](/concepts/abac-policy-conditions-and-built-in-functions.md) - The has_tag() and has_tag_value() functions used in ABAC policy WHEN and MATCH COLUMNS clauses to match tables and columns based on governed tag attributes.
* [Task Parallelism (Ray)](/concepts/task-parallelism-ray.md) - Ray is designed for task parallelism, enabling multiple independent tasks to run concurrently, making it efficient for computation-focused workloads like reinforcement learning, hyperparameter search, and simulation modeling.
* [Task-Specific AI Functions](/concepts/task-specific-ai-functions.md) - Specialized built-in SQL functions in Databricks for specific AI tasks like ai_translate, designed for targeted batch inference workloads
* [Task-Specific AI Functions (Databricks)](/concepts/task-specific-ai-functions.md) - Specialized built-in AI functions such as ai_translate for targeted batch inference tasks without needing to specify a model.
* [Task-Specific AI Functions vs General-Purpose AI Functions](/concepts/task-specific-ai-functions-vs-general-purpose-ai-functions.md) - Databricks distinguishes between task-specific AI functions (e.g., ai_translate for translation) and the general-purpose ai_query function for applying AI models in SQL.
* [Telemetry Data Querying and Analysis](/concepts/telemetry-data-querying-and-analysis.md) - Patterns for querying persisted endpoint telemetry data using SQL, including filtering by severity, timestamp, trace IDs, and custom attributes for root cause analysis.
* [Template Variables for Judge Prompts](/concepts/template-variables-for-judge-prompts.md) - The four allowed template variables in make_judge prompts — inputs, outputs, expectations, and trace — that provide access to agent data during evaluation.
* [Temporal Specification Restriction on DESCRIBE HISTORY](/concepts/temporal-specification-restriction-on-describe-history.md) - The DESCRIBE HISTORY command does not accept temporal or options specifications in the table name parameter.
* [Tensor Input Scoring Format](/concepts/tensor-input-scoring-format.md) - A JSON format for scoring requests to Databricks Model Serving endpoints that accept tensor inputs, supporting both 'instances' (row format with same 0-th dimension) and 'inputs' (columnar format for heterogeneous tensors)
* [TensorBoard Known Issues and Workarounds on Databricks](/concepts/tensorboard-known-issues-and-workarounds-on-databricks.md) - Specific issues encountered when using TensorBoard on Databricks including iframe limitations, port availability, blank projector tabs, and DBFS logging errors.
* [TensorBoard Log Directory Best Practices](/concepts/tensorboard-log-directory-best-practices.md) - Recommendations for writing TensorBoard logs to cloud storage instead of ephemeral cluster storage, and organizing logs by experiment and run subdirectories.
* [TensorBoard Multi-Framework Support](/concepts/tensorboard-multi-framework-support.md) - TensorBoard supports visualization for TensorFlow, PyTorch, Hugging Face Transformers, and other ML programs through callbacks, functions, and file writers.
* [TensorBoard on Databricks](/concepts/tensorboard-on-databricks.md) - How to start and use TensorBoard visualization tools within Databricks notebooks using magic commands or the notebook module.
* [TensorBoard Process Lifecycle on Databricks](/concepts/tensorboard-process-lifecycle-on-databricks.md) - Managing TensorBoard server processes on Databricks clusters including starting, listing, and killing processes, and handling port range limitations.
* [TensorFlow 2 Distributed Training](/concepts/tensorflow-2-distributed-training.md) - The general distributed training capability introduced as a major feature in TensorFlow 2, including distribution strategies.
* [TensorFlow Deprecation from Top-Tier](/concepts/tensorflow-deprecation-from-top-tier.md) - Starting with Databricks Runtime 18.0 ML, TensorFlow and spark-tensorflow-connector are no longer top-tier libraries.
* [TensorFlow deprecation in Databricks Runtime ML](/concepts/tensorflow-deprecation-in-databricks-runtime-ml.md) - The removal of TensorFlow and spark-tensorflow-connector from the top-tier library list starting with Databricks Runtime 18.0 ML.
* [TensorFlow Ecosystem](/concepts/tensorflow-ecosystem.md) - A collection of libraries and tools extending TensorFlow, including the spark-tensorflow-connector for Spark integration.
* [TensorFlow on Databricks](/concepts/tensorflow-on-databricks.md) - Using TensorFlow and TensorBoard within Databricks Runtime ML for deep learning and general numerical computations on CPUs, GPUs, and GPU clusters.
* [TensorFlow on Spark clusters](/concepts/tensorflow-on-spark-clusters.md) - Pattern for integrating TensorFlow 2 distributed training with Apache Spark clusters via the TensorFlow ecosystem package
* [TensorFlow SavedModel Format Requirement](/concepts/tensorflow-savedmodel-format-requirement.md) - When using horovod.spark with custom Keras callbacks, models must be saved in the TensorFlow SavedModel format using the .tf suffix (TF 2.x) or save_weights_only=True (TF 1.x).
* [tensorflow.distribute.Strategy](/concepts/tensorflowdistributestrategy.md) - TensorFlow 2's native API for distributed training strategies, serving as the foundation for spark-tensorflow-distributor
* [Test Case Definition](/concepts/test-case-definition.md) - Scenarios for simulated conversations defined by a required goal and optional persona and context fields that guide the simulated user's behavior.
* [Test Case Definition for Conversation Simulation](/concepts/test-case-definition-for-conversation-simulation.md) - Structured specification of a conversation scenario using goal, persona, and context fields to guide the simulator in generating realistic dialogues.
* [Test Case Generation from Production Conversations](/concepts/test-case-generation-from-production-conversations.md) - Process of extracting test scenarios (goals and personas) from existing production conversation sessions using mlflow.search_sessions and generate_test_cases.
* [Test Cases for Conversation Simulation](/concepts/test-cases-for-conversation-simulation.md) - Structured specifications (goal, persona, context) defining each simulated conversation scenario, where goal describes the user's intent, persona shapes communication style, and context provides additional parameters to the agent function.
* [Testing ABAC policy UDF performance at scale](/concepts/testing-abac-policy-udf-performance-at-scale.md) - Guidelines for testing policy UDFs on at least 1 million rows before production deployment, including a template SQL query using range() and WITH test_data to isolate policy overhead.
* [Testing Private Repository Connectivity with Serverless Notebooks](/concepts/testing-private-repository-connectivity-with-serverless-notebooks.md) - Using serverless notebooks to validate connectivity, authentication, and package availability from private PyPI mirrors before deploying to Model Serving
* [Text Embedding Models](/concepts/text-embedding-models.md) - Compact models like GTE, BGE, and Qwen3-Embedding that map text to dense vector representations for semantic search, retrieval, clustering, and classification tasks, typically used alongside LLMs in RAG pipelines.
* [Text Embedding Models for RAG](/concepts/text-embedding-models.md) - Embedding models like GTE Large, BGE Large, and Qwen3-Embedding available via Databricks APIs that map text to dense vectors for semantic search, retrieval, clustering, and classification, often used with LLMs for RAG.
* [Text Embedding Models for Semantic Search](/concepts/text-embedding-models.md) - Models like Qwen3-Embedding, GTE, and BGE that map text to dense vector representations for retrieval, clustering, classification, and semantic similarity tasks, often used to power RAG pipelines.
* [Text Embedding Models on Databricks](/concepts/text-embedding-models.md) - A class of models (GTE, BGE, Qwen3-Embedding) available on Databricks for converting text into dense vector representations used in semantic search, clustering, and RAG pipelines.
* [Text vs Chat Prompt Types](/concepts/text-vs-chat-prompt-types.md) - MLflow Prompt Registry supports two prompt types: 'Text' for single completion-style templates and 'Chat' for role-based multi-message prompts targeting conversational models.
* [Textractor Pipeline (txtai)](/concepts/textractor-pipeline-txtai.md) - A txtai pipeline component for extracting text from documents and data sources, traceable via MLflow autolog
* [tf.data.TFRecordDataset](/concepts/tfdatatfrecorddataset.md) - A TensorFlow dataset class that reads records from TFRecord files, enabling efficient consumption of TFRecord data for training.
* [tf.distribute.Strategy](/concepts/tfdistributestrategy.md) - Core TensorFlow 2 API for distributing training across multiple devices and machines
* [tf.distribute.Strategy API](/concepts/tfdistributestrategy-api.md) - TensorFlow's native API for distributed training, recommended by Databricks as the replacement for HorovodRunner when using TensorFlow.
* [tf.distribute.Strategy for Distributed TensorFlow](/concepts/tfdistributestrategy-for-distributed-tensorflow.md) - Databricks-recommended replacement for Horovod when performing distributed training with TensorFlow.
* [TFRecord File Format](/concepts/tfrecord-file-format.md) - A simple record-oriented binary file format for storing ML training data, commonly used with TensorFlow.
* [TFRecord Format](/concepts/tfrecord-format.md) - A record-oriented binary format used by TensorFlow applications for training data, serving as a data source for distributed deep learning workflows.
* [Thinking Budget Tokens](/concepts/thinking-budget-tokens.md) - A configurable parameter (budget_tokens) that controls how many tokens the model can allocate to its internal reasoning/thinking process before generating the final answer.
* [Third-Party Client API Timeouts](/concepts/third-party-client-api-timeouts.md) - Timeouts caused by third-party client APIs (e.g., OpenAI Python client) used within custom PyFunc model pipelines, and how to configure their timeout parameters.
* [Third-Party Client API Timeouts in Model Pipelines](/concepts/third-party-client-api-timeouts.md) - Client-side timeouts caused by third-party API clients (e.g., OpenAI) used within custom PyFunc model pipelines, and configuration of their timeout parameters.
* [Third-Party Client API Timeouts in Model Serving Pipelines](/concepts/third-party-client-api-timeouts.md) - Client-side timeouts caused by third-party client APIs (e.g., OpenAI) used within custom PyFunc model pipelines, and how to configure their timeout parameters.
* [Third-Party Online Store Authentication for Feature Serving](/concepts/third-party-online-store-authentication-for-feature-serving.md) - Authentication methods (instance profiles and Databricks secrets) for publishing to and reading from third-party online stores in Databricks Feature Store on AWS.
* [Third-Party Online Stores for Feature Serving](/concepts/third-party-online-stores-for-feature-serving.md) - Databricks supports publishing feature tables to third-party online stores like Amazon DynamoDB (v0.3.8+), which must be configured with read-only credentials using Databricks secrets.
* [Third-Party OpenTelemetry Span Search](/concepts/third-party-opentelemetry-span-search.md) - Ability to search traces ingested from third-party OpenTelemetry tools (e.g., Langfuse) using the span.attributes.* prefix in filter strings.
* [Third-Party Scorers in MLflow GenAI](/concepts/third-party-scorers-in-mlflow-genai.md) - MLflow's extensibility mechanism allowing third-party evaluation scorers (like Arize Phoenix) to be used in GenAI evaluations
* [Three-Level Namespace](/concepts/three-level-namespace.md) - The catalog.schema.table naming convention used by Unity Catalog metastores to organize and reference data objects.
* [Three-level namespace (catalog.schema.table)](/concepts/three-level-namespace-catalogschematable.md) - The hierarchical naming convention in Unity Catalog that organizes data objects as catalog.schema.table, providing a structured way to manage and query data.
* [Three-Stage MLOps Environment Architecture](/concepts/three-stage-mlops-environment-architecture.md) - The recommended practice of creating separate execution environments (Development, Staging, Production) for ML code and model development with clearly defined transitions between stages.
* [Three-Step Alignment Workflow](/concepts/three-step-alignment-workflow.md) - The structured process for aligning judges: generate initial assessments, collect human feedback, then align and deploy using the align() method.
* [Throughput Plateau in Provisioned Throughput Systems](/concepts/throughput-plateau-in-provisioned-throughput-systems.md) - The phenomenon where endpoint throughput plateaus as concurrent requests increase beyond provisioned capacity, causing latency to grow unbounded while throughput stagnates.
* [Throughput vs Context Trade-off](/concepts/throughput-vs-context-trade-off.md) - The capacity trade-off where a fixed number of model units can handle either many small requests simultaneously or fewer long-context requests before exhausting capacity.
* [Time per Output Token (TPOT)](/concepts/time-per-output-token-tpot.md) - A performance metric measuring the time to generate each subsequent output token per user, directly influencing perceived model speed.
* [Time Series Aggregation](/concepts/time-series-aggregation.md) - Method for handling multiple values per timestamp in a time series, defaulting to averaging, with configurable options like sum.
* [Time Series Aggregation for Forecasting](/concepts/time-series-aggregation-for-forecasting.md) - The strategy used by Databricks AutoML to handle multiple values per timestamp in a time series, defaulting to averaging but configurable to summation via notebook source code edits.
* [Time Series Aggregation in AutoML Forecasting](/concepts/time-series-aggregation-in-automl-forecasting.md) - How AutoML handles multiple values at the same timestamp in a time series, defaulting to averaging and allowing customization to sum via notebook edits.
* [Time Series Cross-Validation](/concepts/time-series-cross-validation.md) - Chronological splitting method for forecasting data that incrementally extends the training set and validates on subsequent time points.
* [Time Series Cross-Validation for AutoML](/concepts/time-series-cross-validation-for-automl.md) - AutoML uses chronological time series cross-validation to split forecasting data into train, validation, and test sets, incrementally extending the training dataset over time.
* [Time Series Feature Engineering from Date Columns](/concepts/time-series-feature-engineering-from-date-columns.md) - Techniques for deriving predictive features such as weekend indicators from date/time columns to enrich forecasting models.
* [Time series feature publishing modes](/concepts/time-series-feature-publishing-modes.md) - Two strategies for publishing time series feature tables to online stores: snapshot mode (latest value per primary key) and window mode (all values within a time-to-live window, with automatic expiry).
* [Time series feature table](/concepts/time-series-feature-table.md) - A feature table that includes a TIMESTAMP primary key column, enabling point-in-time AS OF joins so that each training row uses the latest feature value prior to the label's timestamp.
* [Time Series Feature Table (Point-in-time Lookup)](/concepts/time-series-feature-table-point-in-time-lookup.md) - Feature tables that include a timestamp column to ensure training datasets use only feature values known up to the time of each target observation, preventing lookahead bias.
* [Time Series Feature Tables](/concepts/time-series-feature-tables.md) - Feature tables that include timestamp dimensions requiring timestamp lookup keys for point-in-time correct feature lookups in AutoML
* [Time Series Feature Tables in AutoML](/concepts/time-series-feature-tables.md) - For time series feature tables, AutoML requires a corresponding timestamp lookup key mapped to a timestamp column in the training dataset.
* [Time Series Identifier Columns](/concepts/time-series-identifier-columns.md) - Columns that identify individual time series in multi-series forecasting; Databricks groups data by these columns and trains a separate model per series.
* [Time Series Preprocessing in AutoML](/concepts/time-series-preprocessing-in-automl.md) - The automatic data preparation steps in Databricks Serverless Forecasting, including missing value imputation, data splitting, and one-hot encoding of categorical features.
* [Time Series Profile](/concepts/time-series-profile.md) - A data profiling analysis mode that computes metrics over time windows, limited to the last 30 days, useful for tracking data distribution changes across successive time periods.
* [Time Series vs Snapshot Profiling](/concepts/time-series-vs-snapshot-profiling.md) - Two analysis modes in data profiling: time series profiles compute metrics over rolling time windows (max 30 days) and support incremental processing, while snapshot profiles analyze a single point-in-state of a table (max 4TB).
* [Time to First Token (TTFT)](/concepts/time-to-first-token-ttft.md) - A latency metric measuring how quickly users see the first output token after submitting a query, driven by prompt processing and first token generation time.
* [Time Travel and Cloning Restrictions with ABAC](/concepts/time-travel-and-cloning-restrictions-with-abac.md) - Time travel queries and clone operations fail on tables with ABAC policies unless the principal is exempted via the EXCEPT clause
* [Time Travel on Streaming Tables](/concepts/time-travel-on-streaming-tables.md) - The ability to query historical versions of a streaming table, which is not supported when performing deep clone operations
* [Time Travel Restriction for Streaming Table Deep Clone](/concepts/time-travel-restriction-for-streaming-table-deep-clone.md) - Time travel (e.g., VERSION AS OF, TIMESTAMP AS OF) is not supported when deep cloning a streaming table
* [Time travel unsupported on streaming table deep clones](/concepts/time-travel-unsupported-on-streaming-table-deep-clones.md) - Time travel queries cannot be used in conjunction with deep clone operations on streaming tables in Databricks.
* [Time Window Aggregation in Data Profiling](/concepts/time-window-aggregation-in-data-profiling.md) - The mechanism by which Databricks profiling computes metrics over specified time intervals (windows), with Snapshot analysis using single points and TimeSeries/InferenceLog using granularity-based windows.
* [Time Window Aggregation Strategies](/concepts/time-window-aggregation-strategies.md) - Best practices and trade-offs for choosing window types (tumbling, sliding, rolling) and durations in feature engineering, balancing noise versus stability.
* [Time Window Aggregation Types](/concepts/time-window-aggregation-types.md) - Different window strategies (tumbling, sliding, rolling) for computing time-series aggregations with trade-offs between scalability and recency
* [Time Window Analysis Types](/concepts/time-window-analysis-types.md) - Three analysis modes (Snapshot, TimeSeries, InferenceLog) that determine how time intervals and granularities structure metric computation in data profiling.
* [Time Window for Unity Catalog Rollback](/concepts/time-window-for-unity-catalog-rollback.md) - The constraint that rolling back a managed table to external status is only supported within a limited number of days (numDays) after the migration to Unity Catalog managed table.
* [Time Window Granularity in Profiling](/concepts/time-window-granularity-in-profiling.md) - How profiling metrics are computed over configurable time intervals (windows) based on granularities specified in create_monitor, with Snapshot, TimeSeries, and InferenceLog analysis types.
* [Time Window Types](/concepts/time-window-types.md) - Three distinct lookback window semantics for time-series aggregations: Rolling (real-time, up-to-date), Tumbling (fixed non-overlapping partitions), and Sliding (overlapping windows with configurable slide interval).
* [Time Windows](/concepts/time-windows.md) - Three types of lookback windows (Rolling, Tumbling, Sliding) that control aggregation time ranges for feature computation in Databricks Feature Engineering
* [Time Windows for Aggregation](/concepts/time-windows.md) - Three types of time windows — rolling, tumbling, and sliding — that control lookback behavior for time-series aggregations in declarative features, each with distinct semantics for overlapping, non-overlapping, and real-time windows.
* [Time Windows for Feature Aggregation](/concepts/time-windows.md) - Three types of time windows (Rolling, Tumbling, Sliding) that control lookback behavior for time-window-based aggregations in declarative feature engineering.
* [TIME_TRAVEL_NOT_SUPPORTED](/concepts/time_travel_not_supported.md) - Sub-error indicating that time travel is not supported when performing a deep clone on a streaming table.
* [TIME_TRAVEL_NOT_SUPPORTED sub-error](/concepts/time_travel_not_supported-sub-error.md) - Sub-error indicating that time travel syntax (e.g., VERSION AS OF, TIMESTAMP AS OF) is incompatible with deep clone on streaming tables.
* [TIME_TRAVELLED_BY_TIMESTAMP error](/concepts/time_travelled_by_timestamp-error.md) - A sub-error of DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE indicating the source table used time travel by timestamp, which is incompatible with cloning with history.
* [TIME_TRAVELLED_BY_TIMESTAMP error condition](/concepts/time_travelled_by_timestamp-error.md) - A specific sub-error of DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE raised when the source table uses timestamp-based time travel, which is not supported for clone with history
* [TIME_TRAVELLED_BY_TIMESTAMP error sub-condition](/concepts/time_travelled_by_timestamp-error.md) - A specific error sub-condition under DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE, raised when the source table uses timestamp-based time travel.
* [TIME_TRAVELLED_BY_TIMESTAMP sub-error](/concepts/time_travelled_by_timestamp-error.md) - A specific sub-condition of DELTA_CLONE_WITH_HISTORY_UNSUPPORTED_SOURCE raised when the source table uses timestamp-based time travel, which is not supported for clone with history.
* [TIME_WINDOW_EXCEEDED](/concepts/time_window_exceeded.md) - Sub-error indicating the time window for rolling back a managed table to external has elapsed (beyond numDays)
* [TIME_WINDOW_EXCEEDED error](/concepts/time_window_exceeded-error.md) - A sub-error of DELTA_ALTER_TABLE_UNSET_MANAGED_FAILED indicating the rollback window (measured in days after migration to Unity Catalog) has expired.
* [Time-Based Feature Engineering for Forecasting](/concepts/time-based-feature-engineering-for-forecasting.md) - Creating derived temporal features such as is_weekend from date columns to improve forecasting model performance
* [Time-Series Feature Engineering for Covariates](/concepts/time-series-feature-engineering-for-covariates.md) - The process of deriving predictive features from time-series data, such as binary weekend indicators from date columns, to use as covariates.
* [Time-travelled source restriction for clone with history](/concepts/time-travelled-source-restriction-for-clone-with-history.md) - The error condition raised when cloning a Delta table with history from a source that was accessed via time travel by timestamp, which is unsupported.
* [Time-Windowed Aggregation Functions](/concepts/time-windowed-aggregation-functions.md) - Feature computation patterns using TumblingWindow, SlidingWindow, and RollingWindow for point-in-time aggregations over time-series data.
* [TimeSeries Profile](/concepts/timeseries-profile.md) - A data profile type that compares data distributions across time windows using a timestamp column and granularities
* [TimeSeries Profiling](/concepts/timeseries-profiling.md) - A data profiling type that partitions data into time-based windows using a timestamp column and metric granularities for trend analysis.
* [timestamp_lookup_key](/concepts/timestamp_lookup_key.md) - A parameter in FeatureLookup that specifies which DataFrame column contains the timestamps against which to perform point-in-time lookups on a time series feature table.
* [TO and EXCEPT Clauses for Principal Targeting](/concepts/to-and-except-clauses-for-principal-targeting.md) - Using TO and EXCEPT clauses in row filter and column mask policies to define which principals the policy applies to
* [TO and EXCEPT Clauses for Principal Targeting in ABAC](/concepts/to-and-except-clauses-for-principal-targeting-in-abac.md) - Using TO and EXCEPT clauses in row filter, column mask, and GRANT policies to define which users and groups policies apply to, keeping UDF logic simpler.
* [to_predict_fn for Deployed Endpoint Evaluation](/concepts/to_predict_fn-for-deployed-endpoint-evaluation.md) - Using mlflow.genai.to_predict_fn() to evaluate deployed Model Serving chat endpoints, Custom Agents, and custom endpoints.
* [TO/EXCEPT Principal Targeting in ABAC](/concepts/toexcept-principal-targeting-in-abac.md) - Using TO and EXCEPT clauses in row filter, column mask, and GRANT policies to define which users and groups a policy applies to
* [Token Credit-Back Mechanism](/concepts/token-credit-back-mechanism.md) - System that reserves output tokens against OTPM limits during pre-admission checking and credits back unused tokens from the actual response, making them immediately available for other requests.
* [Token Impact of Function Definitions](/concepts/token-impact-of-function-definitions.md) - The use of tool definitions increases input token consumption and billing because prompt injection and other techniques are applied to improve tool-call quality.
* [Token Management for OpenSharing Recipients](/concepts/token-management-for-opensharing-recipients.md) - For recipients using bearer token authentication (Databricks-to-Open sharing), providers can view/copy authentication links, rotate bearer tokens, and manage activation links. Tokens are invalidated when a recipient is deleted.
* [Token Processing Asymmetry](/concepts/token-processing-asymmetry.md) - The concept that generating output tokens is significantly more resource-intensive than processing input tokens during model inference.
* [Token Rotation and Security Management](/concepts/token-rotation-and-security-management.md) - The process of rotating a recipient's bearer token to generate a new token and activation URL, used when tokens are about to expire, compromised, or when security best practices require credential refresh.
* [Token Usage Aggregation at Trace Level in MLflow](/concepts/token-usage-aggregation-at-trace-level-in-mlflow.md) - Setting token counts (gen_ai.usage.input_tokens and gen_ai.usage.output_tokens) on the root span so MLflow aggregates them at the trace level for the UI summary.
* [Token Usage Impact of Function Calling](/concepts/token-usage-impact-of-function-calling.md) - Using function calling increases input and output token consumption due to prompt injection and tool definitions, directly affecting billing costs.
* [Token Usage in MLflow Traces](/concepts/token-usage-in-mlflow-traces.md) - Mechanism for tracking LLM token consumption (input, output, total tokens) within trace metadata and individual span attributes.
* [Token usage monitoring and optimization](/concepts/token-usage-monitoring-and-optimization.md) - Best practices for tracking input/output tokens, minimizing prompt length, controlling output via max_tokens, and logging token usage to avoid rate limits.
* [Token usage tracking across spans](/concepts/token-usage-tracking-across-spans.md) - Method for tracking and aggregating LLM token usage (input, output, total tokens) across individual spans and spans of type CHAT_MODEL in MLflow traces.
* [Token Usage Tracking in MLflow Traces](/concepts/token-usage-in-mlflow-traces.md) - Mechanism for tracking LLM token consumption via aggregated token counts (input, output, total) derived from provider APIs and span-level attributes.
* [Token Usage Tracking in MLflow Tracing](/concepts/token-usage-in-mlflow-traces.md) - Mechanism for tracking LLM token counts (input, output, total) within traces, with provider-specific methods for token tracking.
* [Token Usage Tracking in Tracing](/concepts/token-usage-tracking-in-tracing.md) - MLflow Tracing can track token usage from LLM provider APIs, exposing input_tokens, output_tokens, and total_tokens via trace metadata and span attributes.
* [Token-Based Rate Limiting (ITPM/OTPM)](/concepts/token-based-rate-limiting-itpmotpm.md) - Rate limits controlling the maximum number of input tokens per minute (ITPM) and output tokens per minute (OTPM) that can be processed within a 60-second window for pay-per-token endpoints.
* [Tokens Per Second as a Scaling Metric](/concepts/tokens-per-second-as-a-scaling-metric.md) - Using tokens per second (instead of concurrency) as the unit for configuring scale-out ranges for LLM serving endpoints, providing LLM-native scaling semantics.
* [Tool call evaluation judges](/concepts/tool-call-evaluation-judges.md) - Built-in LLM judges (ToolCallCorrectness, ToolCallEfficiency) that assess whether tool calls and arguments are correct for a user query, and whether they are made without redundancy.
* [Tool Choice Parameter](/concepts/tool-choice-parameter.md) - A configuration option ('auto', 'required', 'none', or a specific function name) that controls whether and which functions an LLM will invoke during a chat completion.
* [Tool Registration Pattern](/concepts/tool-registration-pattern.md) - The pattern of registering a tool with the LLM (for suggestion) and the execution agent (for execution) using register_for_llm and register_for_execution
* [Top-Tier Libraries](/concepts/top-tier-libraries.md) - A subset of Databricks Runtime ML libraries receiving faster updates, advanced support, testing, and embedded optimizations.
* [Top-tier Libraries in Databricks Runtime ML](/concepts/top-tier-libraries.md) - A subset of libraries designated for faster update cadence, advanced support, testing, and embedded optimizations, updated with each runtime release.
* [Top-Tier Library Removal Criteria](/concepts/top-tier-library-removal-criteria.md) - Specific conditions that trigger removal of a library from the top-tier list: no commits in 2 months and no releases in 6 months, significant usage drop, or replacement by newer packages.
* [Top-Tier Library Support in Databricks Runtime ML](/concepts/top-tier-libraries.md) - A subset of ML libraries in Databricks Runtime ML that receive faster update cadence, advanced support, testing, and embedded optimizations from Databricks.
* [Top-tier Library Support Policy](/concepts/top-tier-library-support-policy.md) - Databricks designates a subset of libraries as 'top-tier' with faster update cadence, advanced support, testing, and embedded optimizations, updated with each runtime release.
* [Top-Tier Removal and Re-addition Policy](/concepts/top-tier-removal-and-re-addition-policy.md) - Conditions under which Databricks may remove a library from top-tier status (2 months no commits, 6 months no releases, significant usage drop, or replacement by newer packages) and the possibility of reinstatement when active maintenance resumes.
* [TorchDistributor](/concepts/torchdistributor.md) - An open-source PySpark module that enables distributed PyTorch training on Spark clusters by initializing environment and communication channels via torch.distributed.run.
* [TorchDistributor (Databricks)](/concepts/deepspeed-distributor.md) - A base distributed training framework on Databricks for PyTorch models, upon which the DeepSpeed distributor is built.
* [TorchDistributor API Methods](/concepts/torchdistributor-api-methods.md) - The supported API methods for TorchDistributor, enabling launching PyTorch training as Spark jobs with parameters like num_processes, local_mode, and use_gpu
* [TorchDistributor Common Errors and Environment Variables](/concepts/torchdistributor-common-errors-and-environment-variables.md) - Troubleshooting guide for CUDA peer access, NCCL internal errors, and Gloo connection failures when using TorchDistributor
* [TorchDistributor Development Workflow](/concepts/torchdistributor-development-workflow.md) - A four-step workflow for converting single-node PyTorch training to distributed training using TorchDistributor: prepare single-node code, convert to distributed, move imports inside training function, and launch with distributor.run().
* [TorchDistributor for Distributed PyTorch](/concepts/torchdistributor-for-distributed-pytorch.md) - Databricks-recommended replacement for Horovod when performing distributed training with PyTorch.
* [TorchDistributor Foundation](/concepts/torchdistributor-foundation.md) - The DeepSpeed distributor is built on top of TorchDistributor, which is Databricks' native Spark-based PyTorch distributed training solution.
* [TorchDistributor Import Scoping for Pickling](/concepts/torchdistributor-import-scoping-for-pickling.md) - Pattern of placing import statements inside the training function to avoid pickling errors during distributed PyTorch training with TorchDistributor
* [TorchDistributor Migration from External Repositories](/concepts/torchdistributor-migration-from-external-repositories.md) - Process to migrate existing distributed PyTorch training code from external repositories to Databricks using Git folders and TorchDistributor.run(train_file).
* [TorchDistributor Requirements and Compatibility](/concepts/torchdistributor-requirements-and-compatibility.md) - TorchDistributor requires Spark 3.4 and Databricks Runtime 13.0 ML or above for compatibility.
* [TorchDistributor Troubleshooting: Gloo Connection Refused](/concepts/torchdistributor-troubleshooting-gloo-connection-refused.md) - Error when using Gloo backend for distributed CPU training, resolved by setting the GLOO_SOCKET_IFNAME=eth0 environment variable.
* [TorchDistributor Troubleshooting: Import Pickling Errors](/concepts/torchdistributor-troubleshooting-import-pickling-errors.md) - Common error when library imports are not distributed to executors; resolved by placing all import statements inside the training function passed to TorchDistributor.run().
* [TorchDistributor Troubleshooting: NCCL and GPU Communication Errors](/concepts/torchdistributor-troubleshooting-nccl-and-gpu-communication-errors.md) - CUDA peer access errors on G5 GPUs and NCCL internal errors in multi-node training, resolved by setting NCCL_P2P_DISABLE=1 and NCCL_SOCKET_IFNAME=eth0 environment variables.
* [TorchRec EmbeddingBagCollection](/concepts/torchrec-embeddingbagcollection.md) - TorchRec module that manages multiple EmbeddingBag tables for large-scale categorical features, pooling sparse features into dense embeddings for downstream neural networks.
* [Torchrun Multi-Node Launcher](/concepts/torchrun-multi-node-launcher.md) - A PyTorch distributed launcher that reads rendezvous environment variables (NUM_NODES, NODE_RANK, LOCAL_WORLD_SIZE, MASTER_ADDR, MASTER_PORT) to launch one training process per GPU across multiple nodes without a separate launcher script.
* [total_row_count and daily_row_count Structs](/concepts/total_row_count-and-daily_row_count-structs.md) - Structs in the anomaly detection results table that track overall and daily row count metrics per table.
* [ToxicLanguage Scorer](/concepts/toxiclanguage-scorer.md) - A Guardrails AI scorer for toxicity detection in LLM outputs, configurable with threshold and validation method parameters.
* [Trace (MLflow Evaluation)](/concepts/trace-mlflow-evaluation.md) - A per-input record within an evaluation run that captures the input prompt, model output, and any associated feedback from scorers.
* [Trace Addition to Labeling Sessions](/concepts/trace-addition-to-labeling-sessions.md) - The workflow of selecting MLflow Traces (via UI or API) and adding them to a labeling session so domain experts can review and label them.
* [Trace Analysis and Debugging with Genie Code](/concepts/trace-analysis-and-debugging-with-genie-code.md) - Genie Code capability to investigate failing traces, find errors, examine span trees, pinpoint root causes, analyze latency, and identify bottlenecks in an agent's execution flow.
* [Trace Append Semantics](/concepts/trace-append-semantics.md) - The behavior where archived traces are appended to an existing Delta table rather than overwriting or replacing previous entries.
* [Trace Archival](/concepts/trace-archival.md) - The process of saving MLflow traces and their assessments to a Unity Catalog Delta table for long-term storage and advanced analysis.
* [Trace Archival Permissions](/concepts/trace-archival-permissions.md) - The requirement for users to have the necessary write permissions on the target Unity Catalog Delta table to enable trace archiving.
* [Trace Archiving to Delta Tables](/concepts/trace-archiving-to-delta-tables.md) - The ability to save MLflow traces and their associated assessments to a Unity Catalog Delta table for long-term storage and advanced analysis.
* [Trace Assessments](/concepts/trace-assessments.md) - Evaluations and feedback attached to traces, searchable by name, type (feedback/expectation), span ID, and source, with details like value, rationale, and metadata.
* [Trace Assessments and Feedback](/concepts/trace-assessments-and-feedback.md) - Logging human feedback, LLM judge evaluations, ground truth expectations, and span-specific assessments to MLflow traces using log_feedback() and log_expectation().
* [Trace Assessments and Feedback Logging](/concepts/trace-assessments-and-feedback.md) - Pattern for attaching human feedback, LLM judge evaluations, ground truth expectations, and span-specific assessments to traces using mlflow.log_feedback() and mlflow.log_expectation().
* [Trace Context Enrichment (MLflow)](/concepts/trace-context-enrichment-mlflow.md) - The practice of adding metadata and tags to MLflow traces to track execution details, analyze user behavior, debug issues across environments, and monitor application performance.
* [Trace Context Metadata](/concepts/trace-context-metadata.md) - The practice of adding standard and custom metadata fields (tags and metadata) to MLflow traces to track execution details, user behavior, and environment information.
* [Trace Context Metadata in MLflow](/concepts/trace-context-metadata.md) - Adding standardized and custom metadata/tags to MLflow traces for tracking execution details, user behavior, and application performance
* [Trace Data Export and Conversion](/concepts/trace-data-export-and-conversion.md) - MLflow traces support export and conversion to multiple formats including Python dictionaries (to_dict), JSON strings (to_json), and pandas DataFrame rows (to_pandas_dataframe_row), with corresponding from_dict/from_json reconstruction methods.
* [Trace Data Export and Serialization](/concepts/trace-data-export-and-serialization.md) - Methods for converting MLflow Trace objects to dictionaries, JSON strings, and Pandas DataFrame rows for data portability and analysis.
* [Trace Data Serialization and Export](/concepts/trace-data-serialization-and-export.md) - Methods for converting Trace objects to dictionaries, JSON strings, or Pandas DataFrame rows, and reconstructing them from serialized formats.
* [Trace Debugging Workflows in MLflow](/concepts/trace-debugging-workflows-in-mlflow.md) - Common observability and debugging patterns using the MLflow Tracing UI, including identifying slow traces (sort by execution time), finding user-specific traces (filters/tags), locating error traces (filter by State=ERROR), and identifying negative feedback traces (assessment filters).
* [Trace Grouping by Users and Sessions](/concepts/trace-grouping-by-users-and-sessions.md) - MLflow supports grouping per-request traces by user sessions to maintain multi-turn interaction context and help understand end-user journeys across requests.
* [Trace Input/Output Tracking](/concepts/inputoutput-tracking.md) - The practice of capturing inputs, outputs, and metadata for each intermediate step (retrieval, tool calls, LLM interactions) in a GenAI application to enable debugging and analysis.
* [Trace Lifecycle (start/end span pattern)](/concepts/trace-lifecycle-startend-span-pattern.md) - Every trace follows a strict lifecycle where start_trace and start_span calls must have corresponding end_trace or end_span calls, managed in LIFO order.
* [Trace Logging Redirection](/concepts/trace-logging-redirection.md) - The step of stopping writes to the source experiment and redirecting trace logging to a new Unity Catalog-backed destination experiment before running migration, preventing data loss.
* [Trace Management for Labeling Sessions](/concepts/trace-management-for-labeling-sessions.md) - The process of selecting, adding, and retrieving MLflow Traces within labeling sessions for expert review, supporting both UI and API workflows.
* [Trace Metadata](/concepts/trace-metadata.md) - Immutable write-once key-value pairs attached to MLflow traces for fixed information such as model version, environment, or system configuration.
* [Trace Metadata (MLflow)](/concepts/trace-metadata.md) - Immutable, write-once key-value pairs attached to MLflow traces for fixed information like model version or environment; cannot be updated after logging.
* [Trace Metadata Filters](/concepts/trace-metadata-filters.md) - Structured search queries in the MLflow UI that allow filtering traces by metadata fields such as user, session, source type, and app version using a SQL-like syntax on metadata fields.
* [Trace Metadata in MLflow](/concepts/trace-metadata.md) - Write-once, immutable key-value pairs attached to MLflow traces during execution for fixed information such as model version or environment.
* [Trace Performance Profiling](/concepts/trace-performance-profiling.md) - Techniques for analyzing trace execution time distributions, computing percentile latencies (P50, P95, P99), and identifying performance outliers.
* [Trace Query Performance Optimization](/concepts/trace-query-performance-optimization.md) - Techniques for diagnosing and improving slow trace queries including materialized views, SQL warehouse sizing, and filtering strategies.
* [Trace Return Types](/concepts/trace-return-types.md) - mlflow.search_traces() returns results as either pandas DataFrames or lists of MLflow Trace objects, enabling further analysis or evaluation dataset creation.
* [Trace Reuse Pattern](/concepts/trace-reuse-pattern.md) - Technique of passing a Pandas DataFrame of previously stored traces directly to mlflow.genai.evaluate() to enable rapid iteration on scorers without re-running the full application.
* [Trace Search Query Syntax](/concepts/trace-search-query-syntax.md) - SQL-like filter language used with mlflow.search_traces() to filter traces by fields like status, execution time, tags, and metadata.
* [Trace Selection Strategies](/concepts/trace-selection-strategies.md) - Quantitative and qualitative methods for identifying representative traces from production logs to curate evaluation datasets, including filtering by quality scores, token usage, latency, and manual review of patterns.
* [Trace span hierarchy](/concepts/trace-span-hierarchy.md) - The parent-child structure of spans within an MLflow trace, where a root span (e.g., an application entry point function) can contain child spans (e.g., individual LLM API calls), providing hierarchical visibility into application execution.
* [Trace span hierarchy (root and child spans)](/concepts/trace-span-hierarchy.md) - The tree structure of MLflow traces where a root span (e.g., the application entry point) contains child spans (e.g., individual LLM calls), enabling multi-step visibility.
* [Trace Span Requirements for MLflow Judges](/concepts/mlflow-trace-requirements-for-llm-judges.md) - MLflow judges require specific trace span configurations: RelevanceToQuery needs inputs and outputs on the root span, while RetrievalRelevance requires at least one span with span_type set to RETRIEVER.
* [Trace Spans](/concepts/trace-spans.md) - Immutable building blocks representing individual operations within a trace, with properties including span_id, name, type, timing, status, inputs/outputs, and parent-child relationships.
* [Trace Spans (Root and Child)](/concepts/trace-spans-root-and-child.md) - The hierarchical span structure in MLflow Tracing where a root span captures the top-level function call (e.g., app entry point) and child spans capture sub-operations (e.g., individual LLM API calls), forming a tree of execution.
* [Trace State Management Across Application Components](/concepts/trace-state-management-across-application-components.md) - Pattern for managing active trace state using a stack-based approach, allowing nested traces and clean propagation across application components.
* [Trace Structure Control](/concepts/trace-structure-control.md) - The ability to define exactly which code segments are traced, create custom span hierarchies, and control span boundaries and relationships in GenAI applications.
* [Trace Tags](/concepts/trace-tags.md) - Mutable key-value pairs attached to MLflow traces for organization, search, and filtering; can be updated after logging.
* [Trace Tags (MLflow)](/concepts/trace-tags.md) - Mutable key-value pairs attached to MLflow traces for organization, search, and filtering; can be added or updated post-logging.
* [Trace Tags and Metadata](/concepts/trace-tags-and-metadata.md) - Two distinct annotation systems on traces: mutable tags (user-defined, updatable post-creation) and immutable trace_metadata (set at creation); used for environment, user ID, and other contextual information.
* [Trace Tags in MLflow](/concepts/trace-tags.md) - Mutable key-value pairs attached to MLflow traces for organization, search, and filtering, updatable during or after trace execution.
* [Trace Unified View Pattern](/concepts/trace-unified-view-pattern.md) - A join pattern that aggregates OTel spans, annotations, and tags into a single denormalized trace record, enabling unified querying across redacted trace data.
* [Trace-Associated Feedback Collection](/concepts/trace-associated-feedback-collection.md) - The pattern of linking user feedback to specific MLflow traces using either MLflow-generated trace IDs or custom client request IDs, enabling feedback to be permanently associated with interactions.
* [Trace-Based Dataset Creation](/concepts/trace-based-dataset-curation.md) - The practice of building evaluation datasets by selecting and merging existing MLflow traces (real historical application interactions) using the UI or SDK, enabling datasets grounded in production scenarios.
* [Trace-Based Dataset Curation](/concepts/trace-based-dataset-curation.md) - The practice of building evaluation datasets by selecting and curating traces from an application's historical interactions captured by MLflow Tracing, using either the UI or SDK.
* [Trace-based Evaluation Datasets](/concepts/trace-based-evaluation-datasets.md) - Using archived traces to build evaluation datasets for MLflow evaluation, enabling reuse of production trace data for model assessment.
* [Trace-based Evaluation Iteration](/concepts/trace-based-evaluation-iteration.md) - Technique of passing a Pandas DataFrame of pre-computed MLflow traces directly to mlflow.genai.evaluate() to iterate on scorers without rerunning the application.
* [Trace-Based Evaluation via Expert Review](/concepts/trace-based-evaluation-via-expert-review.md) - The practice of having domain experts label existing production traces to understand high-quality responses, align LLM judges with business requirements, and create evaluation datasets from real interactions.
* [Trace-based Feature Flag Analysis](/concepts/trace-based-feature-flag-analysis.md) - Comparing performance metrics of traces grouped by feature flag states (control vs treatment) by filtering traces on metadata tags with mlflow.search_traces().
* [Trace-Based Feedback Loop](/concepts/trace-based-feedback-loop.md) - The mechanism of using MLflow Traces to store both judge assessments and human feedback, which are then retrieved and used as training data for judge alignment.
* [Trace-based Judges](/concepts/trace-based-judges.md) - A type of custom judge that analyzes execution traces to validate tool calls, enabled by including {{ trace }} in judge instructions.
* [Trace-Based Performance and Cost Monitoring](/concepts/trace-based-performance-and-cost-monitoring.md) - Using MLflow Tracing to capture and monitor latency, cost, and resource utilization at each step of GenAI application execution to identify bottlenecks and optimize efficiency.
* [Trace-Driven Evaluation Datasets](/concepts/trace-driven-evaluation-datasets.md) - Practice of using archived trace data to construct MLflow evaluation datasets for further analysis and monitoring.
* [Trace-level Input/Output Rendering in MLflow](/concepts/trace-level-inputoutput-rendering-in-mlflow.md) - Using gen_ai.input.messages and gen_ai.output.messages attributes with JSON-serialized message arrays for rich UI rendering with role-labeled bubbles.
* [TraceAnalyzer Pattern](/concepts/traceanalyzer-pattern.md) - A reusable object-oriented utility class for advanced trace analysis, providing methods for error summaries, LLM usage aggregation, retrieval metrics, span hierarchy visualization, and evaluation data export.
* [TraceAnalyzer Reusable Utility Class](/concepts/traceanalyzer-reusable-utility-class.md) - Advanced reusable Python class for trace analysis providing error summary, LLM usage aggregation, retrieval metrics extraction, span hierarchy building, and evaluation data export.
* [Traced Gemini Interactions](/concepts/traced-gemini-interactions.md) - MLflow supports tracing multiple types of Gemini SDK interactions: synchronous text generation, multi-turn chat conversations, and embeddings API calls.
* [TraceData](/concepts/tracedata.md) - Execution data component of a trace containing spans, full request/response payloads, and intermediate outputs.
* [TraceData (MLflow)](/concepts/tracedata.md) - Execution data component of an MLflow Trace, containing spans, full request/response data, and intermediate outputs from non-root spans.
* [TraceData and Span Objects](/concepts/tracedata-and-span-objects.md) - TraceData holds the actual execution spans, each with properties like span_id, name, type, parent_id, timing, status, inputs/outputs, and attributes.
* [TraceData and Spans](/concepts/tracedata.md) - TraceData holds the execution data of a trace including spans (individual operations), full request/response data, and intermediate outputs from non-root spans.
* [TraceInfo](/concepts/traceinfo.md) - Metadata component of a trace containing identifiers, status, timing, tags, token usage, and assessments.
* [TraceInfo (MLflow)](/concepts/traceinfo.md) - Metadata component of an MLflow Trace, containing trace ID, status (OK/ERROR/IN_PROGRESS), request/response previews, storage location, timing, tags, token usage, and assessments.
* [TraceInfo Metadata](/concepts/traceinfo-metadata.md) - TraceInfo contains metadata about a trace including trace ID, request/response previews, timing information, tags, trace metadata (immutable), token usage, assessments, and storage location.
* [Traces](/concepts/traces.md) - Execution logs capturing inputs, outputs, and every intermediate step (LLM calls, retrievals, tool use) of a GenAI application, used for debugging, monitoring, and creating evaluation datasets.
* [Traces (MLflow GenAI)](/concepts/traces.md) - Execution logs that capture the full lifecycle of a GenAI application call, including inputs, outputs, and intermediate steps like LLM calls and retrievals.
* [Traces in MLflow Evaluation](/concepts/traces-in-mlflow-evaluation.md) - Per-input evaluation records within an evaluation run, each containing inputs, outputs, and associated feedback for one example in the evaluation dataset.
* [Traces-Based Evaluation Results](/concepts/traces-based-evaluation-results.md) - Evaluation results in MLflow 3 are stored as traces with assessments, accessed via mlflow.search_traces() instead of direct result tables like results.tables['eval_results'].
* [Tracing Approach Selection Guide](/concepts/tracing-approach-selection-guide.md) - Recommendation to start with automatic tracing and add manual tracing later when more control is needed
* [Tracing for Generative AI Applications](/concepts/tracing-for-generative-ai-applications.md) - The practice of adding trace instrumentation to generative AI applications for observability and debugging.
* [Tracing Limitations for Groq](/concepts/tracing-limitations-for-groq.md) - Only synchronous Groq SDK calls are traced; async and streaming methods are not supported
* [Tracking Additional Content with Autologging](/concepts/tracking-additional-content-with-autologging.md) - The pattern and best practice for adding extra metrics, parameters, files, and metadata to MLflow runs created by Databricks Autologging by calling mlflow.autolog() with exclusive=False and wrapping training in an mlflow.start_run() context.
* [Tracking URI in MLflow](/concepts/tracking-uri-in-mlflow.md) - Configuration parameter that specifies the location of the MLflow tracking server, used to direct experiment logging to a specific server instance.
* [Traffic Routing Across Multiple Models on a Single Endpoint](/concepts/traffic-routing-across-multiple-models-on-a-single-endpoint.md) - The ability to serve multiple custom models or model versions from a single serving endpoint, with configurable traffic-split percentages between served entities.
* [Traffic Splitting Between Models](/concepts/traffic-splitting-between-models.md) - Configuring percentage-based distribution of inference requests across different models hosted on the same serving endpoint.
* [train_on_responses_only in Unsloth](/concepts/train_on_responses_only-in-unsloth.md) - A technique that masks out user/instruction tokens during training so the model only learns from assistant responses, improving fine-tuning quality for chat models.
* [train.yaml Workload Configuration](/concepts/trainyaml-workload-configuration.md) - A YAML configuration file used with the AI Runtime CLI to define machine learning workloads with inline dependencies.
* [Training and Inference API](/concepts/training-and-inference-api.md) - Client methods for creating point-in-time correct training sets (create_training_set), logging models with feature lineage (log_model), and performing batch inference (score_batch)
* [Training and Inference API Methods](/concepts/training-and-inference-api-methods.md) - Core methods for ML lifecycle: create_training_set (point-in-time correct feature computation), log_model (model logging with feature lineage), and score_batch (offline batch inference with automatic feature lookup).
* [Training and Inference API with Point-in-Time Correctness](/concepts/training-and-inference-api-with-point-in-time-correctness.md) - API methods (create_training_set, log_model, score_batch) that ensure point-in-time correct feature computation, lineage tracking, and automatic feature lookup during batch inference.
* [Training and Inference Lifecycle](/concepts/training-and-inference-lifecycle.md) - End-to-end ML workflow using declarative features: create_training_set for point-in-time correct datasets, log_model for lineage tracking, and score_batch for offline batch inference with automatic feature lookup.
* [Training and Inference Pipeline API](/concepts/training-and-inference-pipeline-api.md) - End-to-end ML pipeline API with create_training_set() for point-in-time correct feature computation, log_model() for lineage tracking, and score_batch() for batch inference with automatic feature lookup.
* [Training Configuration with Hugging Face TrainingArguments](/concepts/training-configuration-with-hugging-face-trainingarguments.md) - Setting up TrainingArguments including batch size, precision, evaluation strategy, and gradient accumulation for single-GPU fine-tuning
* [Training Script Pattern with MLflow Tracking on Databricks](/concepts/training-script-pattern-with-mlflow-tracking-on-databricks.md) - Pattern for training a PyTorch model on Databricks that includes MLflow experiment logging, Unity Catalog checkpointing, and single-GPU training with SGD optimizer.
* [Training Set (Feature Store)](/concepts/training-set-feature-store.md) - A dataset composed of a list of features and a DataFrame containing raw training data, labels, and primary keys used to look up features during model training.
* [Training workflow with on-demand features](/concepts/training-workflow-with-on-demand-features.md) - End-to-end process for training ML models that incorporate on-demand features, including create_training_set, log_model, and Unity Catalog registration.
* [Training workload YAML format](/concepts/training-workload-yaml-format.md) - Configuration file format (train.yaml) that defines the distributed training job parameters for the AI Runtime CLI
* [Training/Serving Skew Elimination](/concepts/trainingserving-skew-elimination.md) - Automatic alignment of feature computations between model training and inference to prevent performance degradation caused by inconsistent feature engineering.
* [TrainingArguments Configuration](/concepts/trainingarguments-configuration.md) - Configuration of Hugging Face TrainingArguments for fine-tuning, including output directory, evaluation strategy, learning rate, and memory optimization settings.
* [TrainingArguments for Hugging Face](/concepts/trainingarguments-for-hugging-face.md) - A configuration class in the transformers library used to specify training parameters such as output directory, evaluation strategy, learning rate, batch size, and mixed precision training.
* [TrainValidationSplit](/concepts/trainvalidationsplit.md) - A Spark MLlib model tuning tool that, when used with MLlib automated MLflow tracking, automatically logs hyperparameter settings and evaluation metrics as nested MLflow runs.
* [Transfer Learning](/concepts/transfer-learning.md) - A machine learning technique that reuses knowledge from a pre-trained model on one problem domain to improve learning in a related domain, with featurization being one simple and powerful method for achieving it.
* [Transfer Learning Workflow on Databricks](/concepts/transfer-learning-workflow-on-databricks.md) - A four-step workflow: start with a pre-trained model, truncate its last layers, apply the modified model to a new dataset to compute features, then train a downstream model using those features.
* [Transformer Block Auto-Detection for FSDP Wrapping](/concepts/transformer-block-auto-detection-for-fsdp-wrapping.md) - A technique for automatically identifying transformer decoder layer class names in a model to configure FSDP's layer wrapping policy for sharding.
* [TrialInfo](/concepts/trialinfo.md) - Summary object for each individual trial in an AutoML run, with a method to load the model generated for that trial.
* [TRL (Transformer Reinforcement Learning)](/concepts/trl-transformer-reinforcement-learning.md) - A Hugging Face library for training language models using reinforcement learning and supervised fine-tuning, providing SFTTrainer and other abstractions.
* [TRL (Transformer Reinforcement Learning) Library](/concepts/trl-transformer-reinforcement-learning.md) - A Hugging Face library for training language models with reinforcement learning and supervised fine-tuning, providing SFTTrainer and SFTConfig for simplified training configuration.
* [TRL (Transformers Reinforcement Learning)](/concepts/trl-transformer-reinforcement-learning.md) - A Hugging Face library providing tools for training language models with reinforcement learning and supervised fine-tuning
* [TRL (Transformers Reinforcement Learning) SFTTrainer](/concepts/trl-transformer-reinforcement-learning.md) - A library and trainer class from Hugging Face for supervised fine-tuning of transformer models, supporting integration with LoRA, FSDP, and distributed training.
* [TRL SFTTrainer](/concepts/trl-sfttrainer.md) - A library component from Transformer Reinforcement Learning (TRL) that simplifies supervised fine-tuning configuration, integrates with LoRA via PEFT, and supports Liger kernel optimizations.
* [TRL SFTTrainer for Supervised Fine-Tuning](/concepts/trl-sfttrainer-for-supervised-fine-tuning.md) - A trainer from the Transformers Reinforcement Learning library that simplifies supervised fine-tuning of large language models with built-in support for LoRA, FSDP, and distributed training.
* [Troubleshooting ABAC Policy Conflicts](/concepts/troubleshooting-abac-policy-conflicts.md) - Diagnostic approach using SHOW EFFECTIVE POLICIES and INFORMATION_SCHEMA views to identify conflicts, with resolution strategies including refining conditions, adjusting principals, and restructuring policies.
* [Troubleshooting Ray on Databricks](/concepts/troubleshooting-ray-on-databricks.md) - Common issues and solutions when running Ray on Databricks, including pickling errors, OOM memory monitor bugs, transformers MLflow integration conflicts, and library scoping limitations.
* [TruLens Scorers Integration](/concepts/trulens-scorers-integration.md) - MLflow integration with TruLens for analyzing agent execution traces with goal-plan-action alignment metrics like logical consistency and plan adherence
* [Truncated Model Feature Extraction](/concepts/truncated-model-feature-extraction.md) - The technique of removing the final classification layers from a pre-trained deep learning model so the modified model outputs a feature tensor instead of a prediction.
* [Truncated Table History Affecting Rollback](/concepts/truncated-table-history-affecting-rollback.md) - A condition where the Delta table history has been truncated (e.g., via VACUUM or retention policies), making it impossible to find all necessary commits to rollback a table to its original state.
* [Truncated Table History and Rollback Failure in Delta Lake](/concepts/truncated-table-history-and-rollback-failure-in-delta-lake.md) - When Delta table history is truncated (e.g., VACUUM or log cleanup), the necessary commits to rollback a managed-to-external migration may be missing, causing TRUNCATED_HISTORY error.
* [TRUNCATED_HISTORY](/concepts/truncated_history.md) - Sub-error indicating the Delta table history has been truncated, making rollback from managed to external impossible
* [TRUNCATED_HISTORY rollback error](/concepts/truncated_history.md) - A sub-error of DELTA_ALTER_TABLE_UNSET_MANAGED_FAILED indicating the Delta table history has been truncated, preventing the system from finding all necessary commits to rollback.
* [Trust Relationship for Serverless Compute](/concepts/trust-relationship-for-serverless-compute.md) - The IAM role associated with an instance profile must have a trust relationship configured for Databricks serverless compute to avoid errors when attaching the profile to a serving endpoint.
* [Tuple-count vs size-on-disk balancing](/concepts/tuple-count-vs-size-on-disk-balancing.md) - The distinction between optimizations that balance files by number of records (Z-ordering) versus those that balance by byte size (bin-packing), and the implications for task skew.
* [TurnRelevancy Scorer](/concepts/turnrelevancy-scorer.md) - A conversational DeepEval scorer that evaluates relevance in multi-turn dialogue, supporting parameters like window_size and strict_mode.
* [Two-Step Connection Establishment](/concepts/two-step-connection-establishment.md) - Creating a Databricks–SAP BDC connection requires a two-party workflow: the Databricks admin shares a connection identifier with the SAP BDC admin, who generates a connection link that the Databricks admin uses to finalize the connection.
* [Two-Tower Models for Recommendation Systems](/concepts/two-tower-models-for-recommendation-systems.md) - Deep learning approach for building recommender systems using dual neural network architectures (two-tower models) to represent users and items.
* [Two-Tower Models for Recommender Systems](/concepts/two-tower-models-for-recommender-systems.md) - A deep learning architecture for recommendation systems that uses separate neural networks (towers) to encode users and items into a shared embedding space.
* [Two-Tower Recommendation Model](/concepts/two-tower-recommendation-model.md) - A deep learning architecture for recommendation systems that uses two separate neural network towers (one for queries/users, one for items/candidates) to learn embeddings for matching.
* [Two-Tower Recommender System](/concepts/two-tower-recommender-system.md) - A recommendation model architecture demonstrated as a use case for distributed training with DDP on Databricks
* [Two-tower recommender system training](/concepts/two-tower-recommender-system-with-ddp.md) - A recommendation model architecture trained using PyTorch Lightning and DDP on Databricks, with data prepared in Mosaic Streaming format.
* [Two-tower recommender system with DDP](/concepts/two-tower-recommender-system-with-ddp.md) - Training a two-tower recommendation model using PyTorch Lightning and DDP on Databricks, including data preparation with Mosaic Streaming (MDS) format.
* [Two-Tower Recommender Training with DDP](/concepts/two-tower-recommender-system-with-ddp.md) - Distributed training of a two-tower recommendation model using PyTorch Lightning DDP, including data preparation with Mosaic Streaming (MDS) format
* [txtai](/concepts/txtai.md) - An all-in-one embeddings database for semantic search, LLM orchestration, and language model workflows
* [txtai Agent](/concepts/txtai-agent.md) - An agent framework in txtai that uses tools (like semantic search) and LLMs to autonomously research and answer questions, traceable via MLflow
* [Type Widening inDelta-Iceberg Compatibility](/concepts/type-widening-indelta-iceberg-compatibility.md) - Type changes applied to a Delta table's schema (e.g., changing a column type) can be incompatible with IcebergCompatV and trigger a violation.
* [Typed Dataset APIs in Databricks Connect](/concepts/typed-dataset-apis-in-databricks-connect.md) - Support for running typed transformations like map(), filter(), mapPartitions() and aggregations via Databricks Connect for Scala, requiring compiled artifacts to be uploaded to the remote cluster.
* [U2M (User-to-Machine) Authentication Flow](/concepts/user-to-machine-u2m-authentication.md) - Authentication flow where a human user authenticates via their IdP (with optional MFA) and uses tools like Power BI or Tableau to access shared data, using the OAuth Authorization Code Grant flow.
* [UC-only workspaces](/concepts/uc-only-workspaces.md) - New Databricks workspaces provisioned without legacy features such as DBFS root, Hive metastore, no-isolation shared clusters, and old Databricks Runtime versions.
* [UCVolumeDataset](/concepts/ucvolumedataset.md) - A PyTorch IterableDataset that streams unstructured data (images, audio, text) from Unity Catalog volumes with automatic local caching and distributed partitioning.
* [UCVolumeDataset and UCVolumeWriter](/concepts/ucvolumedataset-and-ucvolumewriter.md) - Databricks utilities for streaming file-based data from Unity Catalog volumes and checkpointing distributed training across ranks.
* [UCX (Unity Catalog Migration Toolkit)](/concepts/ucx-unity-catalog-migration-toolkit.md) - A Databricks Labs open-source project providing tools and workflows to upgrade non-Unity-Catalog workspaces to Unity Catalog on Databricks.
* [UCX (Unity Catalog Upgrade Utilities)](/concepts/ucx-unity-catalog-upgrade-utilities.md) - A Databricks Labs open-source project providing automation tools to help upgrade non-Unity-Catalog workspaces to Unity Catalog at scale.
* [UCX Assessment Workflow](/concepts/ucx-assessment-workflow.md) - The first UCX migration workflow that assesses Unity Catalog compatibility of groups, storage, access controls, and tables, producing an assessment dashboard and Delta tables for migration planning.
* [UCX Group Migration Workflow](/concepts/ucx-group-migration-workflow.md) - A UCX workflow that upgrades workspace-local groups to account-level groups to support Unity Catalog, replicating permissions and removing unnecessary groups.
* [UCX Table Migration Workflow](/concepts/ucx-table-migration-workflow.md) - A UCX workflow that upgrades tables from the Hive metastore to the Unity Catalog metastore, using SYNC for external tables and DEEP CLONE for managed tables.
* [UDF Dependency Management on Databricks Connect](/concepts/dependency-management-in-databricks-connect.md) - The pattern where application dependencies are installed locally while UDF dependencies must be separately managed and installed on the Databricks cluster for remote execution.
* [UDF Dependency Sources](/concepts/udf-dependency-sources.md) - Dependencies for Databricks Connect UDFs can come from three sources: PyPI packages (PEP 508), Unity Catalog volumes (dbfs: paths), and local files (local: paths including .whl, .tar.gz, folders, and .py files).
* [UDF Execution in Databricks Connect](/concepts/udf-execution-in-databricks-connect.md) - User-defined functions (UDFs) defined locally are serialized and transmitted to the Databricks cluster for execution, requiring UDF dependencies to be installed on Databricks while application dependencies run locally.
* [UDF Serialization in Databricks Connect](/concepts/udf-serialization-in-databricks-connect.md) - User-defined functions are serialized by the Databricks Connect client and sent to the Databricks compute server for execution, requiring matching Python versions between client and server.
* [UDF-based Policy Logic](/concepts/udf-based-policy-logic.md) - ABAC row filter and column mask policies rely on user-defined functions (UDFs) in Unity Catalog that implement the filtering or masking logic, requiring EXECUTE permission on the UDF.
* [Unavailable Databricks Utilities Subset](/concepts/unavailable-databricks-utilities-subset.md) - Certain Databricks Utilities (credentials, library, notebook workflow, widgets) are not available in Databricks Connect for Scala.
* [Unified Development-Production Tracing](/concepts/unified-development-production-tracing.md) - A design principle where tracing instrumentation works consistently across development (IDE, notebooks) and production environments without reconfiguration.
* [Unified Development-to-Production Tracing](/concepts/unified-development-to-production-tracing.md) - MLflow Tracing offers a single instrumentation that works consistently across development (IDE, notebook) and production environments, eliminating the need to switch between multiple tools.
* [Unified ML Platform on Databricks](/concepts/unified-ml-platform-on-databricks.md) - Databricks provides a single platform for the entire ML workflow: data pipelines, feature engineering, model training, batch inference, model serving, and monitoring.
* [Unified OpenAI-compatible API (Databricks)](/concepts/unified-openai-compatible-api-databricks.md) - The standard, provider-agnostic API interface on Databricks Model Serving, contrasted with provider native APIs which offer access to provider-specific features.
* [Unified Streaming and Batch Processing](/concepts/unified-streaming-and-batch-processing.md) - Delta Lake enables a unified approach to stream and batch data processing, allowing users to treat streaming and batch data pipelines with the same table abstractions and semantics.
* [Unified streaming and batch processing in Delta Lake](/concepts/unified-streaming-and-batch-processing-in-delta-lake.md) - Delta Lake's capability to unify streaming and batch data processing under a single storage layer, simplifying data pipeline architectures.
* [Uniform (Apache Iceberg) Format](/concepts/uniform-apache-iceberg-format.md) - A Databricks feature that makes Delta tables readable as Apache Iceberg tables, requiring all files to be Iceberg-compatible and enforced via IcebergCompatV.
* [Uniform (Apache Iceberg) format in Databricks](/concepts/uniform-apache-iceberg-format.md) - A Databricks feature that makes Delta Lake tables readable as Apache Iceberg tables; enabling it requires all files to be Iceberg-compatible and triggers the DELTA_ICEBERG_COMPAT_VIOLATION if files lack compatibility tags.
* [Uniform (Delta-Uniform)](/concepts/delta-uniform.md) - A Databricks feature that makes Delta Lake tables readable by Apache Iceberg clients by storing Iceberg metadata alongside Delta metadata.
* [Uniform Apache Iceberg Ingress Table](/concepts/uniform-apache-iceberg-ingress-table.md) - A Databricks table type that allows ingesting Iceberg metadata into Delta format, supporting only CREATE and REFRESH operations.
* [Uniform Iceberg Ingress Table](/concepts/uniform-ingress-table.md) - A Delta Uniform table that ingests Apache Iceberg data, supporting only CREATE and REFRESH operations; violations of this constraint raise the OPERATION_NOT_SUPPORTED error.
* [Uniform Ingress Table](/concepts/uniform-ingress-table.md) - A Delta table configured to act as an ingress point for Apache Iceberg data, supporting only CREATE and REFRESH operations
* [Union of Direct Grants and GRANT Policies](/concepts/union-of-direct-grants-and-grant-policies.md) - Effective privileges on an object are the union of direct grants and any applicable GRANT policies, meaning a more selective GRANT policy does not override a broader direct grant.
* [Unit Testing with Databricks Feature Engineering Clients](/concepts/unit-testing-with-feature-engineering-clients.md) - Using mock frameworks (e.g., unittest.mock) to unit-test code that calls FeatureEngineeringClient or FeatureStoreClient APIs locally.
* [Unit Testing with Feature Engineering Clients](/concepts/unit-testing-with-feature-engineering-clients.md) - The ability to install databricks-feature-engineering or databricks-feature-store locally to unit test code using mock frameworks like unittest.mock for validating feature table operations.
* [Unity AI Gateway](/concepts/unity-ai-gateway.md) - An enterprise control plane for governing AI traffic across an organization, managing and monitoring LLM endpoints and MCP servers centrally.
* [Unity Catalog](/concepts/unity-catalog.md) - The unified governance layer for data and AI in Databricks, providing centralized access control, lineage, auditing, and data discovery across workspaces.
* [Unity Catalog ABAC (Attribute-Based Access Control)](/concepts/unity-catalog-attribute-based-access-control-abac.md) - Centralized, tag-driven policies that dynamically filter and mask data across the catalog, recommended by Databricks over per-table filters for scalability.
* [Unity Catalog ABAC Column Mask Policies](/concepts/unity-catalog-abac-column-mask-policies.md) - Attribute-based access control policies that mask column values by applying a UDF to matched columns, identified via governed tags or custom expressions using has_tag and has_tag_value.
* [Unity Catalog ABAC Row Filter Policies](/concepts/unity-catalog-abac-row-filter-policies.md) - Attribute-based access control policies that filter rows from query results based on a user-defined function (UDF) returning boolean values, excluding rows where the function returns FALSE.
* [Unity Catalog Access Control Complementary Models](/concepts/unity-catalog-access-control-complementary-models.md) - The four complementary access control models (privileges, ABAC, table-level filtering, workspace bindings) that work together for fine-grained security.
* [Unity Catalog Access Control for Feature Tables](/concepts/unity-catalog-access-control-for-feature-tables.md) - Governance of feature table access managed through Unity Catalog's privilege system, controlling who can read or write feature tables
* [Unity Catalog Access Control Models](/concepts/unity-catalog-access-control-models.md) - Four complementary access control models in Unity Catalog: privileges/ownership, ABAC, table-level filtering/masking, and workspace-level restrictions.
* [Unity Catalog Access Requests](/concepts/unity-catalog-access-requests.md) - Configurable destinations (email, Slack, Teams, webhooks) for receiving access requests on Unity Catalog securable objects.
* [Unity Catalog Admin Roles](/concepts/unity-catalog-admin-roles.md) - Account admin, workspace admin, and metastore admin roles with distinct scopes of administrative privilege in Unity Catalog.
* [Unity Catalog Admin Roles (Account, Workspace, Metastore)](/concepts/unity-catalog-admin-roles-account-workspace-metastore.md) - The three most important administrative roles in Unity Catalog — account admins, workspace admins, and metastore admins — each operating at different scopes within the Databricks architecture.
* [Unity Catalog AI Asset Governance](/concepts/unity-catalog-ai-asset-governance.md) - Managing AI assets—models, functions, connections, and hosted foundation models—as securable objects with standard Unity Catalog privileges.
* [Unity Catalog AI asset model](/concepts/unity-catalog-ai-asset-model.md) - AI assets (models, functions, connections, hosted foundation models) managed as securable objects in Unity Catalog with standard grant/revoke privileges.
* [Unity Catalog Allowlist](/concepts/unity-catalog-allowlist.md) - A security control in Unity Catalog that specifies which JARs, Maven coordinates, and init scripts are permitted to run on standard access mode compute in Databricks.
* [Unity Catalog Attribute-Based Access Control (ABAC)](/concepts/unity-catalog-attribute-based-access-control-abac.md) - Centralized, tag-driven policies that dynamically filter and mask data across the catalog, recommended over per-table row/column filters.
* [Unity Catalog Auto-enablement](/concepts/unity-catalog-auto-enablement.md) - Automatic enablement of Unity Catalog for all Databricks workspaces created after November 8, 2023
* [Unity Catalog Certification Status System Tag](/concepts/certification-status-system-tag.md) - A system-governed tag (system.certification_status) in Databricks Unity Catalog that marks data assets as certified or deprecated to indicate data quality and lifecycle status.
* [Unity Catalog Column Naming and Casing Rules](/concepts/unity-catalog-column-naming-and-casing-rules.md) - Column names in Unity Catalog can use special characters if escaped with backticks; column casing is preserved but queries are case-insensitive.
* [Unity Catalog Compute Requirements (Access Modes & Runtimes)](/concepts/unity-catalog-compute-requirements-access-modes-runtimes.md) - Clusters must run Databricks Runtime 11.3 LTS or above and be configured with standard or dedicated access mode to access Unity Catalog data.
* [Unity Catalog Data Classification](/concepts/unity-catalog-data-classification.md) - A Databricks feature that scans tables in the background and automatically applies system governed tags (class.*) to detected sensitive columns such as PII, financial data, and contact information.
* [Unity Catalog Data Lineage](/concepts/unity-catalog-data-lineage.md) - Automatically captured metadata showing where data in Databricks originated, how it was transformed, and which downstream assets consume it, down to the column level.
* [Unity Catalog Data Quality Monitoring](/concepts/unity-catalog-data-quality-monitoring.md) - A capability in Unity Catalog that helps ensure the quality of data assets through anomaly detection and data profiling, without modifying tables or adding job overhead.
* [Unity Catalog Data Quality Monitoring Alerts](/concepts/unity-catalog-data-quality-monitoring-alerts.md) - Automated notifications that trigger when anomaly detection identifies unhealthy tables in Unity Catalog, supporting both UI-based and SQL-based configuration.
* [Unity Catalog Dataset Storage](/concepts/unity-catalog-dataset-storage.md) - MLflow evaluation datasets are stored in Unity Catalog, which provides built-in versioning, lineage, sharing, and governance for the datasets.
* [Unity Catalog Delta Table](/concepts/unity-catalog-delta-table.md) - The target storage format and catalog system used for archiving traces, requiring the full three-level name (catalog.schema.table) for write access.
* [Unity Catalog Delta Table as Trace Sink](/concepts/unity-catalog-delta-table-as-trace-sink.md) - Using a Unity Catalog-managed Delta table (specified with catalog.schema.table notation) as the storage target for archived MLflow traces, with automatic table creation and append behavior.
* [Unity Catalog Delta Table for Trace Storage](/concepts/unity-catalog-delta-table-for-trace-storage.md) - The requirement and mechanism for storing archived traces in a Unity Catalog-managed Delta table, which is auto-created if it doesn't exist.
* [Unity Catalog Delta Tables as Trace Storage](/concepts/unity-catalog-delta-tables-as-trace-storage.md) - Unity Catalog Delta tables act as the durable storage target for archived MLflow traces, supporting automatic table creation, append-only writes, and cross-catalog/schema organization.
* [Unity Catalog Delta Tables for Trace Storage](/concepts/unity-catalog-delta-tables-for-trace-storage.md) - Target storage format for archived traces, requiring write permissions and supporting automatic table creation or append behavior.
* [Unity Catalog Enablement](/concepts/unity-catalog-enablement.md) - The process of enabling Unity Catalog in Databricks workspaces, which is automatic for workspaces created after November 8, 2023, and requires manual upgrade for older workspaces.
* [Unity Catalog External Table Conversion](/concepts/unity-catalog-external-table-conversion.md) - Converting Parquet or Iceberg tables stored in external locations managed by Unity Catalog to Delta Lake using CONVERT TO DELTA.
* [Unity Catalog Feature Lookup](/concepts/unity-catalog-feature-lookup.md) - Model serving endpoints use Unity Catalog to look up table-backed features from online stores at inference time, using entity keys from the request payload.
* [Unity Catalog Feature Store vs Workspace Feature Store](/concepts/unity-catalog-feature-store-vs-workspace-feature-store.md) - The two types of feature stores on Databricks: the newer Unity Catalog-based feature store and the legacy Workspace Feature Store
* [Unity Catalog Feature Table Access Control](/concepts/unity-catalog-feature-table-access-control.md) - Access control for Databricks Feature Store feature tables is managed through Unity Catalog privileges
* [Unity Catalog Feature Tables](/concepts/unity-catalog-feature-tables.md) - The modern feature store system in Databricks, managed through Unity Catalog, that AutoML can query to augment training data.
* [Unity Catalog File Format Support](/concepts/unity-catalog-file-format-support.md) - Managed tables in Unity Catalog require Delta or Iceberg format; external tables support Delta, CSV, JSON, Avro, Parquet, ORC, or text.
* [Unity Catalog for AI Governance](/concepts/unity-catalog-for-ai-governance.md) - Fine-grained access control, security policies, and governance for all data and AI assets across the Databricks platform.
* [Unity Catalog for Evaluation Data](/concepts/unity-catalog-for-evaluation-data.md) - Storage of MLflow evaluation datasets in Unity Catalog, providing built-in versioning, lineage, sharing, and governance capabilities for GenAI evaluation artifacts.
* [Unity Catalog for Machine Learning](/concepts/unity-catalog-for-machine-learning.md) - Unified governance of data, ML assets, and model endpoints, providing access management, audit logging, and lineage for the full ML lifecycle.
* [Unity Catalog for ML](/concepts/unity-catalog-for-ml.md) - Unified governance layer on Databricks that manages data, features, models, and functions with access control, lineage tracking, and discovery for ML assets.
* [Unity Catalog for ML Artifact Storage](/concepts/unity-catalog-for-ml-artifact-storage.md) - Using Databricks Unity Catalog volumes as a storage location for saving and versioning machine learning artifacts such as model checkpoints, enabling integration with catalog-level access control and governance.
* [Unity Catalog for ML Data Access](/concepts/unity-catalog-for-ml-data-access.md) - Using Unity Catalog as a unified governance solution to access data on clusters via both Spark and local file APIs for ML workloads.
* [Unity Catalog for ML Governance](/concepts/unity-catalog-for-ml-governance.md) - A unified governance layer for data, features, models, AI endpoints, and inference tables, built on open-source Unity Catalog APIs with fine-grained access controls.
* [Unity Catalog for ML Model Management](/concepts/unity-catalog-for-ml-models.md) - A Databricks feature for registering, versioning, and managing machine learning models as first-class assets within a governed catalog-schema hierarchy, enabling deployment to model serving endpoints.
* [Unity Catalog for ML Models](/concepts/unity-catalog-for-ml-models.md) - Databricks Unity Catalog provides a namespace (catalog/schema/model name) for organizing, registering, and versioning machine learning models, enabling model deployment to serving endpoints or batch inference.
* [Unity Catalog for Model Governance](/concepts/unity-catalog-for-model-governance.md) - Using Models in Unity Catalog to manage model versioning, governance, deployment status, access control, and catalog-per-environment isolation across the MLOps workflow.
* [Unity Catalog for Model Management](/concepts/unity-catalog-for-ml-models.md) - Databricks Unity Catalog namespace used to organize, store, register, and deploy fine-tuned models with MLflow
* [Unity Catalog for Model Registration](/concepts/unity-catalog-for-model-registration.md) - A Databricks feature that enables registering, versioning, and managing ML models in a unified catalog for deployment and inference
* [Unity Catalog Foundation Model Deployment](/concepts/unity-catalog-foundation-model-deployment.md) - The recommended workflow for deploying foundation models from the system.ai catalog in Unity Catalog using Databricks Catalog Explorer.
* [Unity Catalog governance](/concepts/unity-catalog-governance.md) - Central management of metadata, access control, auditing, and lineage for all registered securable objects
* [Unity Catalog Governance for Feature Tables](/concepts/unity-catalog-governance-for-feature-tables.md) - Feature tables inherit Unity Catalog's access control, permissions, and governance policies, including model permission inheritance from training data
* [Unity Catalog Governance for ML](/concepts/unity-catalog-governance-for-ml.md) - Unified governance of data, ML assets, model endpoints, and security across the ML lifecycle via Unity Catalog
* [Unity Catalog governance for ML assets](/concepts/unity-catalog-governance-for-ml-assets.md) - Centralized governance of ML models, feature tables, vector indexes, and tools through Unity Catalog, with fine-grained access control and authentication passthrough for agent, data, and tool access.
* [Unity Catalog Governed Tags for Data Classification](/concepts/unity-catalog-governed-tags-for-data-classification.md) - The system of governed tags (PII, GDPR, HIPAA, DPDPA, etc.) that are automatically applied by the classification engine to detected sensitive columns.
* [Unity Catalog Integration for Evaluation Datasets](/concepts/unity-catalog-integration-for-evaluation-datasets.md) - Infrastructure layer where MLflow evaluation datasets are stored in Unity Catalog tables, providing built-in versioning, lineage tracking, sharing across workspaces, and governance controls via Unity Catalog permissions.
* [Unity Catalog Integration for Feature Store](/concepts/unity-catalog-integration-for-features.md) - Built-in governance, cross-workspace sharing, and discovery of feature tables and models through Unity Catalog.
* [Unity Catalog Integration for Features](/concepts/unity-catalog-integration-for-features.md) - Feature tables and models registered in Unity Catalog enable built-in governance, point-in-time joins, and cross-workspace feature sharing and discovery.
* [Unity Catalog Integration for MLflow Models](/concepts/unity-catalog-integration-for-mlflow-models.md) - Saving, registering, and managing MLflow models within Unity Catalog volumes and the Unity Catalog model registry, including viewing parameters and metrics across experiments and workspaces.
* [Unity Catalog Integration for Model Storage](/concepts/unity-catalog-integration-for-model-storage.md) - Using Databricks Unity Catalog catalogs, schemas, and volumes to store model checkpoints and artifacts, requiring USE CATALOG and USE SCHEMA privileges.
* [Unity Catalog Integration for Prompts](/concepts/unity-catalog-integration-for-prompts.md) - Integration between the Prompt Registry and Unity Catalog for governance, access control, audit trails, and prompt-to-experiment lineage tracking.
* [Unity Catalog Limitations and Known Issues](/concepts/unity-catalog-limitations-and-known-issues.md) - Unity Catalog has various limitations including restrictions on workspace-level groups in GRANT statements, bucketing, R workload dynamic views, Python/Scala UDF support by runtime version, cloning constraints, and cross-region write reliability.
* [Unity Catalog managed functions as AI agent tools](/concepts/unity-catalog-managed-functions-as-ai-agent-tools.md) - Unity Catalog functions used as agent tools or for data transformations, enabling governed access to external APIs and data processing within AI workflows.
* [Unity Catalog Managed Table Migration Rollback Constraints](/concepts/unity-catalog-managed-table-migration-rollback-constraints.md) - Constraints governing the ability to roll back a table from managed to external in Unity Catalog, including time windows and history retention requirements.
* [Unity Catalog Managed Table Migration Rollback Time Window](/concepts/unity-catalog-managed-table-migration-rollback-time-window.md) - The time-limited window (in days) after migrating a table to Unity Catalog managed during which rollback via ALTER TABLE UNSET MANAGED is supported.
* [Unity Catalog Managed Tables](/concepts/unity-catalog-managed-tables.md) - Databricks recommendation to use Unity Catalog managed tables for Delta Lake workloads, with automatic optimization and governance.
* [Unity Catalog Metastore](/concepts/unity-catalog-metastore.md) - The top-level container for data in Unity Catalog that registers metadata about securable objects and permissions, exposing a three-level namespace.
* [Unity Catalog Metastore Auto-Assignment](/concepts/unity-catalog-metastore-auto-assignment.md) - An account admin setting that automatically assigns an existing Unity Catalog metastore to new workspaces created in the same AWS region, eliminating the need for manual workspace-to-metastore assignment.
* [Unity Catalog Metastore Deletion](/concepts/unity-catalog-metastore-deletion.md) - Permanent deletion of a Unity Catalog metastore that makes all managed objects inaccessible, auto-deletes managed table data and metadata after 30 days, but does not affect external table data in cloud storage.
* [Unity Catalog Metastore Recipient Token Lifetime Configuration](/concepts/metastore-recipient-token-lifetime-configuration.md) - The ability for account admins to set a default token lifetime for all new OpenSharing recipients at the metastore level, which does not automatically apply to existing recipients.
* [Unity Catalog Migration and Default Registry Behavior](/concepts/unity-catalog-migration-and-default-registry-behavior.md) - Guidance on when and how to use the Workspace Model Registry versus Unity Catalog for model management, including automatic routing based on workspace configuration and explicit registry URI targeting via mlflow.set_registry_uri().
* [Unity Catalog Migration Path](/concepts/unity-catalog-migration-path.md) - The recommended upgrade strategy to replace the legacy Hive metastore table access control with Unity Catalog for simplified, centralized data governance across workspaces.
* [Unity Catalog ML Governance](/concepts/unity-catalog-ml-governance.md) - Applying Unity Catalog's fine-grained access controls, lineage tracking, and governance to machine learning assets including data, features, models, and endpoints for a complete audit trail.
* [Unity Catalog ML Workflow Migration Strategy](/concepts/unity-catalog-ml-workflow-migration-strategy.md) - Recommended approach for migrating existing Databricks ML workflows to Unity Catalog by creating parallel pipelines instead of cut-over
* [Unity Catalog Model Registration](/concepts/unity-catalog-model-registration.md) - The process of registering machine learning models trained with MLflow into Databricks Unity Catalog for centralized governance, versioning, and deployment.
* [Unity Catalog Model Registration for Custom LLMs](/concepts/unity-catalog-model-registration-for-custom-llms.md) - The process of registering custom LLMs to Unity Catalog using MLflow with the databricks_model_serving environment pack to enable express deployments for custom LLM serving.
* [Unity Catalog model registration for LLMs](/concepts/unity-catalog-model-registration-for-llms.md) - The pattern of registering fine-tuned LLMs into Databricks Unity Catalog for governance, versioning, and deployment, including merging LoRA adapters into the base model.
* [Unity Catalog Model Registration for ML Models](/concepts/unity-catalog-model-registration.md) - The process of registering fine-tuned models in Databricks Unity Catalog for governance, version control, discoverability, and deployment via Model Serving.
* [Unity Catalog Model Registration with MLflow](/concepts/unity-catalog-model-registration-with-mlflow.md) - The workflow for registering finetuned LLMs into Databricks Unity Catalog via MLflow, including model versioning, metadata tagging, and merging LoRA adapters into the base model for deployment.
* [Unity Catalog Model Registry](/concepts/unity-catalog-model-registry.md) - A Databricks feature for storing model checkpoints, registering trained models, and managing model lifecycle using a catalog schema naming convention within Unity Catalog volumes.
* [Unity Catalog Model Registry for ML Models](/concepts/unity-catalog-model-registry-for-ml-models.md) - A Unity Catalog namespace for organizing, storing, registering, and deploying ML models, using catalog/schema/volume structure for model checkpoints and registered model serving.
* [Unity Catalog Object Hierarchy](/concepts/unity-catalog-object-hierarchy.md) - The hierarchical structure of securable objects in Unity Catalog, with the metastore at the top and a three-level namespace (catalog.schema.table) for data assets.
* [Unity Catalog Object Hierarchy and Privilege Inheritance](/concepts/unity-catalog-object-hierarchy-and-privilege-inheritance.md) - The hierarchical structure of securable objects in Unity Catalog where privileges flow from parent to child objects.
* [Unity Catalog Only Mode](/concepts/unity-catalog-only-mode.md) - A workspace admin setting that disables direct access to the legacy Hive metastore, forcing all queries to go through Unity Catalog.
* [Unity Catalog Only Mode Spark Configuration](/concepts/unity-catalog-only-mode-spark-configuration.md) - Spark settings spark.databricks.unityCatalogOnlyMode and spark.databricks.sql.initial.catalog.namespace to enforce Unity Catalog exclusively on a cluster.
* [Unity Catalog Overview](/concepts/unity-catalog.md) - The unified governance layer for data and AI in Databricks, providing centralized access control, lineage, auditing, and data discovery
* [Unity Catalog Permission Hierarchy](/concepts/unity-catalog-permissions-model.md) - Hierarchy of securable objects in Unity Catalog with privilege inheritance, where privileges on parent objects flow down to child objects.
* [Unity Catalog permission model for ML models](/concepts/unity-catalog-permission-model-for-ml-models.md) - Account-level Unity Catalog privileges replace workspace-level permissions for models, with a mapping from registry permissions (e.g., Can Read, Can Edit) to Unity Catalog privileges (e.g., EXECUTE, MANAGE), requiring USE CATALOG and USE SCHEMA for all actions.
* [Unity Catalog Permissions for OpenSharing](/concepts/unity-catalog-permissions-for-opensharing.md) - The required permissions (metastore admin, USE SHARE, SET SHARE PERMISSION, USE RECIPIENT, and ownership) needed to manage shares and recipients.
* [Unity Catalog Permissions for Prompts](/concepts/unity-catalog-permissions-for-prompts.md) - Storing and managing prompts in the MLflow Prompt Registry requires specific Unity Catalog schema privileges: CREATE FUNCTION, EXECUTE, and MANAGE.
* [Unity Catalog Permissions for Trace Archival](/concepts/unity-catalog-permissions-for-trace-archival.md) - The requirement that users must have necessary write permissions to the specified Unity Catalog Delta table when archiving traces; the table is created if it does not exist or appended to if it already exists.
* [Unity Catalog Permissions Hierarchy](/concepts/unity-catalog-permissions-model.md) - The object hierarchy and privilege inheritance model in Unity Catalog that governs how access flows from parent to child objects.
* [Unity Catalog Permissions Model](/concepts/unity-catalog-permissions-model.md) - The object hierarchy, privilege inheritance, and access flow from parent to child objects in Unity Catalog
* [Unity Catalog Policies](/concepts/unity-catalog-policies.md) - Policies attached at hierarchy levels (catalog, schema, table) that are evaluated dynamically to enforce access rules based on attributes.
* [Unity Catalog Policy Lifecycle Management](/concepts/unity-catalog-policy-lifecycle-management.md) - Create, edit, delete, show, and describe operations for ABAC policies using Catalog Explorer UI, SQL statements (CREATE POLICY, SHOW POLICIES, DESCRIBE POLICY), Python SDK, REST APIs, and Terraform.
* [Unity Catalog Privilege Management](/concepts/unity-catalog-privilege-management.md) - The overall system for granting, revoking, and inspecting permissions on Unity Catalog securable objects (catalogs, schemas, tables, views, volumes, etc.) using SQL commands, Catalog Explorer, CLI, or Terraform.
* [Unity Catalog Privileges and Ownership](/concepts/privileges-and-ownership.md) - Grant-based model controlling who can access what on securable objects using a hierarchical permission system with privilege inheritance.
* [Unity Catalog Privileges and Ownership Model](/concepts/privileges-and-ownership.md) - Grant-based model controlling who can access what securable objects, with privilege inheritance through the object hierarchy.
* [Unity Catalog Privileges for ML Workflows](/concepts/unity-catalog-privileges-for-ml-workflows.md) - Access control requirements including USE CATALOG, USE SCHEMA, CREATE MODEL, EXECUTE, and CREATE MODEL VERSION privileges needed for model operations in Unity Catalog
* [Unity Catalog Prompt Registry Constraints](/concepts/unity-catalog-prompt-registry-constraints.md) - Unity Catalog imposes specific constraints on prompt operations: catalog+schema required in searches, no pattern/tag filtering, and strict version-deletion ordering for prompt removal.
* [Unity Catalog Provider Import for OpenSharing](/concepts/unity-catalog-provider-import-for-opensharing.md) - A UI-based workflow in Databricks Catalog Explorer that allows recipients to import provider credentials, create catalogs from shares, and use Unity Catalog access controls without manually handling credential files.
* [Unity Catalog Region Support](/concepts/unity-catalog-region-support.md) - All regions support Unity Catalog, with further details available in Databricks' supported regions documentation.
* [Unity Catalog Regional Metastore Requirement](/concepts/unity-catalog-regional-metastore-requirement.md) - The architectural requirement that each AWS region must have its own Unity Catalog metastore, with workspaces linked only to metastores in their same region.
* [Unity Catalog Requirement for Databricks Connect](/concepts/unity-catalog-requirement-for-databricks-connect.md) - To use Databricks Connect, the Databricks account and workspace must have Unity Catalog enabled.
* [Unity Catalog Requirement for Delta Uniform](/concepts/unity-catalog-requirement-for-delta-uniform.md) - Unity Catalog must be enabled to read Apache Iceberg tables via Delta Uniform; otherwise a UNITY_CATALOG_NOT_ENABLED error is raised.
* [Unity Catalog requirement for Delta Uniform Iceberg reads](/concepts/unity-catalog-requirement-for-delta-uniform.md) - Unity Catalog must be enabled for reading Apache Iceberg tables through Delta Uniform; the UNITY_CATALOG_NOT_ENABLED error is raised when this prerequisite is not met.
* [Unity Catalog Resource Quotas](/concepts/unity-catalog-resource-quotas.md) - Unity Catalog enforces resource quotas on all securable objects; users can monitor usage via dedicated APIs and contact the Databricks account team if they expect to exceed limits.
* [Unity Catalog Schema as Prompt Registry Backend](/concepts/unity-catalog-schema-as-prompt-registry-backend.md) - Prompts are stored in Unity Catalog schemas, which require specific permissions (CREATE FUNCTION, EXECUTE, MANAGE) and are linked to MLflow experiments via a tag.
* [Unity Catalog Schema for Prompts](/concepts/unity-catalog-schema-for-prompts.md) - Prompts are stored in Unity Catalog schemas, requiring CREATE FUNCTION, EXECUTE, and MANAGE privileges for user access.
* [Unity Catalog Schema Permissions for Prompts](/concepts/unity-catalog-schema-permissions-for-prompts.md) - Storing prompts requires a Unity Catalog schema with CREATE FUNCTION, EXECUTE, and MANAGE privileges, and the schema acts as the storage location for registered prompts.
* [Unity Catalog Schema Requirements for Prompts](/concepts/unity-catalog-schema-requirements-for-prompts.md) - Prompts are stored in Unity Catalog schemas requiring specific permissions (CREATE FUNCTION, EXECUTE, MANAGE) and are linked to MLflow experiments via the mlflow.promptRegistryLocation tag.
* [Unity Catalog Securable Object Naming Constraints](/concepts/unity-catalog-securable-object-naming-constraints.md) - Object names in Unity Catalog are limited to 255 characters, stored as lowercase, and cannot contain periods, spaces, forward slashes, or ASCII control characters.
* [Unity Catalog Securable Objects](/concepts/unity-catalog-securable-objects.md) - The object hierarchy in Unity Catalog including catalogs, schemas, tables, external locations, and storage credentials that can have privileges assigned
* [Unity Catalog Shallow Clone](/concepts/unity-catalog-shallow-clone.md) - Shallow clone support for Unity Catalog managed tables in Databricks SQL and Databricks Runtime 13.3 LTS and above, with no support in Databricks Runtime 12.2 LTS and below.
* [Unity Catalog Shallow Clone Support](/concepts/unity-catalog-shallow-clone.md) - Shallow clone is supported for Unity Catalog managed tables in Databricks SQL and Databricks Runtime 13.3 LTS and above, but not in 12.2 LTS and below.
* [Unity Catalog Storage for Evaluation Datasets](/concepts/unity-catalog-storage-for-evaluation-datasets.md) - Evaluation datasets are stored as tables in Unity Catalog, providing built-in versioning, lineage tracking, cross-team sharing, and governance, requiring CREATE TABLE permissions on a schema.
* [Unity Catalog storage validation](/concepts/unity-catalog-storage-validation.md) - The process by which Unity Catalog validates external locations against every storage configuration registered to the Databricks account, not just the workspace where the location was created.
* [Unity Catalog Supported Objects for Tagging](/concepts/unity-catalog-supported-objects-for-tagging.md) - The list of Unity Catalog object types that support the certification status system tag, including catalogs, schemas, tables, views, volumes, functions, registered models, dashboards, Genie Spaces, Databricks Apps, and notebooks.
* [Unity Catalog System Governed Tags](/concepts/unity-catalog-system-governed-tags.md) - System-reserved tags (in the sap.* namespace) that are automatically assigned by the Databricks system and must not be manually created, modified, or deleted by users.
* [Unity Catalog System Tables for Data Quality](/concepts/unity-catalog-system-tables-for-data-quality.md) - The `system.data_quality_monitoring.table_results` system table that stores all anomaly detection scan results, usable for building dashboards or setting up SQL alerts across the metastore.
* [Unity Catalog Tagging Permissions](/concepts/unity-catalog-tagging-permissions.md) - The permission model required to assign governed tags in Unity Catalog, including the ASSIGN permission on the system tag and object-level privileges like APPLY TAG, USE SCHEMA, and USE CATALOG.
* [Unity Catalog Telemetry Table Structure](/concepts/unity-catalog-telemetry-table-structure.md) - Automatically generated Delta tables (_otel_logs, _otel_spans, _otel_metrics) in Unity Catalog that store endpoint telemetry with configurable table prefix.
* [Unity Catalog Three-Level Namespace](/concepts/unity-catalog-three-level-namespace.md) - The catalog.schema.table hierarchical naming convention used by Unity Catalog to organize and access data objects.
* [Unity Catalog Trace Format (Schema-linked vs Table-prefix)](/concepts/unity-catalog-trace-format-schema-linked-vs-table-prefix.md) - Two formats for storing MLflow traces in Unity Catalog: the older Beta schema-linked format (catalog.schema) and the newer Public Preview table-prefix format (catalog.schema.table_prefix)
* [Unity Catalog Trace Location](/concepts/unity-catalog-trace-location.md) - Mechanism to link an MLflow experiment to a Unity Catalog catalog and schema using set_experiment_trace_location(), which determines where ingested OTLP spans are stored as tables.
* [Unity Catalog Trace Storage](/concepts/unity-catalog-trace-storage.md) - Storing OpenTelemetry trace spans in Unity Catalog tables alongside MLflow traces, enabling SQL querying, access controls, lineage, and centralized governance.
* [Unity Catalog Trace Storage for OpenTelemetry](/concepts/unity-catalog-trace-storage.md) - Architecture and approach for storing OpenTelemetry trace data in Unity Catalog tables, enabling query via MLflow SDK and Databricks SQL.
* [Unity Catalog Upgrade Process](/concepts/unity-catalog-upgrade-process.md) - The structured step-by-step process for upgrading a Databricks workspace from legacy Hive metastore and DBFS to Unity Catalog governance.
* [Unity Catalog volume automation via DBUtils](/concepts/unity-catalog-volume-automation-via-dbutils.md) - Using DBUtils.getDBUtils with Databricks Connect for Scala to automate Unity Catalog volumes — creating, reading, and deleting files within volume paths.
* [Unity Catalog Volume Checkpointing](/concepts/unity-catalog-volume-checkpointing.md) - Save and load model checkpoints asynchronously to Unity Catalog volumes using UCVolumeWriter and UCVolumeReader with Torch Distributed Checkpoint API, optimized with NVMe-backed local staging.
* [Unity Catalog Volume file operations via Databricks Utilities Scala](/concepts/unity-catalog-volume-file-operations-via-databricks-utilities-scala.md) - Example pattern showing how to use dbutils.fs.put, dbutils.fs.head, and dbutils.fs.rm on Unity Catalog volume paths (e.g., /Volumes/main/default/my-volume/) in Databricks Connect for Scala.
* [Unity Catalog Volume File Operations via dbutils](/concepts/unity-catalog-volume-file-operations-via-dbutils.md) - Example workflow for creating, reading, and deleting files in Unity Catalog volumes using dbutils.fs methods like put, head, and rm
* [Unity Catalog Volume File Operations with dbutils.fs](/concepts/unity-catalog-volume-file-operations-with-dbutilsfs.md) - Through Databricks Connect, dbutils.fs can perform file operations like put, head, and rm on Unity Catalog volume paths
* [Unity Catalog Volume for Checkpoint Storage](/concepts/unity-catalog-volume-for-checkpoint-storage.md) - A Databricks pattern for storing distributed training checkpoints in Unity Catalog volumes, enabling versioned, governed storage of model artifacts that integrates with MLflow and the Databricks ecosystem.
* [Unity Catalog Volume Operations with DBUtils Scala](/concepts/unity-catalog-volume-operations-with-dbutils-scala.md) - Pattern for creating, reading, and deleting files in Unity Catalog volumes using DBUtils.fs in Scala
* [Unity Catalog Volumes for Checkpointing](/concepts/unity-catalog-volumes-for-checkpointing.md) - Using Unity Catalog volumes as persistent storage for model checkpoints during distributed training workloads on Databricks
* [Unity Catalog Volumes for ML Data](/concepts/unity-catalog-volumes-for-ml-data.md) - Using Unity Catalog volumes as persistent, cluster-independent storage for machine learning training data and Hugging Face dataset caches on Databricks
* [Unity Catalog Volumes for MLflow Dependencies](/concepts/unity-catalog-volumes-for-mlflow-dependencies.md) - Using Unity Catalog volumes as the recommended storage location for custom Python wheel files referenced in MLflow model logging
* [Unity Catalog Volumes for Model Artifacts](/concepts/unity-catalog-volumes-for-model-artifacts.md) - Using Unity Catalog volumes as a repository for model dependency files, with the requirement that these files be packaged into the MLflow model artifact for real-time serving on Databricks.
* [Unity Catalog Volumes for Model Checkpointing](/concepts/unity-catalog-volumes-for-model-checkpointing.md) - Using Unity Catalog Volumes to store and load PyTorch model checkpoints via paths like /Volumes/{catalog}/{schema}/{volume}/{model_name}, integrated with torch.distributed.checkpoint.
* [Unity Catalog vs Legacy Access Control for Feature Store](/concepts/unity-catalog-vs-legacy-access-control-for-feature-store.md) - Distinction between Unity Catalog-based privilege management and the older fine-grained permission model for workspaces not enabled for Unity Catalog
* [Unity Catalog Workspace Bindings](/concepts/unity-catalog-workspace-bindings.md) - Mechanism to restrict which workspaces can access specific catalogs, external locations, and storage credentials.
* [Unity Catalog Workspace-Catalog Binding](/concepts/unity-catalog-workspace-bindings.md) - Workspace-level restrictions that limit which workspaces can access specific catalogs, external locations, and storage credentials.
* [Unity Catalog Write Limitations for Non-Spark Writers](/concepts/unity-catalog-write-limitations-for-non-spark-writers.md) - The limitation that Unity Catalog does not share credentials for writing to tables from non-Spark writers, requiring data to be persisted and read with Spark or using Databricks Connect within Ray tasks.
* [Unity Catalog-Backed Experiment](/concepts/unity-catalog-backed-experiment.md) - An MLflow experiment bound to a Unity Catalog trace location, backed by four Delta tables (otel_spans, otel_annotations, otel_logs, otel_metrics) sharing a common table prefix.
* [Unresolved Logical Plan Size Limit (128 MB)](/concepts/unresolved-logical-plan-size-limit-128-mb.md) - Databricks Connect for Scala cannot create DataFrames with an unresolved logical plan larger than 128 MB, a limit on plan size not data size.
* [UNRESOLVED_COLUMN error subtype](/concepts/unresolved_column-error-subtype.md) - A specific error raised when a column referenced in a predicate cannot be resolved against the available clustering columns of a Liquid table.
* [Unsloth](/concepts/unsloth.md) - An optimization framework for efficiently fine-tuning large language models like Llama
* [Unsloth Chat Templates and ShareGPT Data Standardization](/concepts/unsloth-chat-templates-and-sharegpt-data-standardization.md) - Unsloth provides utilities like get_chat_template and standardize_sharegpt to convert conversational datasets into a standardized chat format (e.g., Llama 3.1) for training.
* [Unsloth Distributed Finetuning on Databricks](/concepts/unsloth-distributed-finetuning-on-databricks.md) - Methodology for fine-tuning large language models like Llama-3.2-3B across multiple GPUs using the Unsloth library within the Databricks environment on AWS.
* [Unsloth Fine-Tuning Library](/concepts/unsloth-fine-tuning-library.md) - An optimized library for faster LLM fine-tuning that supports LoRA/QLoRA, reduced memory usage, and integrated chat templates, included in Databricks AI v5 runtime.
* [Unsloth Library](/concepts/unsloth-library.md) - An optimized library for parameter-efficient fine-tuning (PEFT) of large language models, providing faster training and reduced memory usage through techniques like LoRA.
* [Unsloth LoRA Configuration](/concepts/unsloth-lora-configuration.md) - The specific pattern of applying LoRA adapters via `FastLanguageModel.get_peft_model()` in Unsloth, targeting q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj with rank 16 and zero dropout for optimized training.
* [Unsloth Optimized LLM Finetuning](/concepts/unsloth-optimized-llm-finetuning.md) - An optimization library that accelerates LLM fine-tuning through memory-efficient kernels, LoRA/QLoRA support, and reduced memory footprint.
* [Unsupported Delta Clone Targets](/concepts/unsupported-delta-clone-targets.md) - The three categories of target tables that cannot receive a Delta clone with history: non-Delta tables, path-based tables, and session temporary tables.
* [Unsupported features for Delta Lake on S3 with multi-cluster writes](/concepts/unsupported-features-for-delta-lake-on-s3-with-multi-cluster-writes.md) - Server-Side Encryption with Customer-Provided Keys (SSE-C) and S3 paths with credentials lacking STS access are not supported when multi-cluster writes are enabled on Delta Lake tables stored in S3.
* [Unsupported Operations on ABAC-Protected Tables](/concepts/unsupported-operations-on-abac-protected-tables.md) - Operations like time travel queries, deep/shallow cloning, OpenSharing, and AI Search index creation that fail on ABAC-secured tables unless the principal is exempt from all applicable policies.
* [Unsupported reader features for Delta Uniform refresh](/concepts/unsupported-reader-features-for-delta-uniform-refresh.md) - Some reader table features are incompatible with REFRESH SYNC UNIFORM and cause the UNSUPPORTED_READER_FEATURES sub-error
* [Unsupported source type for Delta Uniform refresh](/concepts/unsupported-source-type-for-delta-uniform-refresh.md) - REFRESH SYNC UNIFORM cannot be applied to certain source table types, raising UNSUPPORTED_TYPE or WRONG_TYPE sub-errors
* [UNSUPPORTED_DATA_TYPE (Delta Iceberg error)](/concepts/unsupported_data_type-in-icebergcompatv1.md) - A sub-error of DELTA_ICEBERG_COMPAT_V1_VIOLATION that occurs when a table schema contains MapType, ArrayType, or NullType, which are unsupported by IcebergCompatV1
* [UNSUPPORTED_DATA_TYPE in IcebergCompatV1](/concepts/unsupported_data_type-in-icebergcompatv1.md) - Error sub-type indicating IcebergCompatV1 does not support schemas containing MapType, ArrayType, or NullType columns
* [UNSUPPORTED_EXPRESSION error subtype](/concepts/unsupported_expression-error-subtype.md) - A specific error raised when an unsupported expression is used in a predicate for a command on a Liquid table.
* [UNSUPPORTED_OPERATOR constraint for table_changes](/concepts/unsupported_operator-constraint-for-table_changes.md) - A validation rule requiring that views used with table_changes must not contain operators that are not allowed for change data capture
* [UPDATE FROM JOIN (Unsupported in Databricks SQL)](/concepts/update-from-join-unsupported-in-databricks-sql.md) - Databricks SQL does not support the UPDATE...FROM...JOIN syntax used in other SQL dialects; users must use MERGE INTO instead.
* [Update Model Serving Endpoints](/concepts/update-model-serving-endpoints.md) - Process and constraints for updating model serving endpoint configurations, including compute configuration changes and the rule that external_model endpoints cannot be converted to or from non-external_model types.
* [UPDATE SET * and INSERT * with EXCEPT](/concepts/update-set-and-insert-with-except.md) - Shorthand syntax to update all target columns from corresponding source columns (or insert all source columns), with the ability to exclude specific columns using the EXCEPT clause (Databricks Runtime 12.2 LTS+).
* [UPDATE Statement (Delta Lake)](/concepts/update-statement-delta-lake.md) - SQL statement to update column values in Delta Lake tables, optionally filtered by a predicate using WHERE clause.
* [upgrade_workspace_table method](/concepts/upgrade_workspace_table-method.md) - API method that migrates metadata (primary keys, time series columns, comments, tags, lineage) from a workspace feature table to a target Unity Catalog table.
* [UpgradeClient (Databricks Feature Engineering)](/concepts/upgradeclient-databricks-feature-engineering.md) - A Python client class from databricks.feature_engineering used to upgrade workspace feature table metadata to Unity Catalog.
* [Upgrading Jobs to Unity Catalog](/concepts/upgrading-jobs-to-unity-catalog.md) - Migration patterns for updating existing Databricks job notebooks to work with Unity Catalog after upgrading from legacy workspaces.
* [Upsert from Streaming Queries using foreachBatch and MERGE](/concepts/upsert-from-streaming-queries-using-foreachbatch-and-merge.md) - Combining foreachBatch with Delta Lake MERGE operations to perform upserts (inserts and updates) from streaming queries, enabling SCD/CDC patterns and deduplication.
* [upstream_jobs Array](/concepts/upstream_jobs-array.md) - An array column in the anomaly detection results table that records information about upstream jobs feeding each scanned table.
* [USAGE Privilege Requirement](/concepts/usage-privilege-requirement.md) - The requirement that users must have the USAGE privilege on a schema (or meet one of several alternative conditions) in addition to any action-specific privilege to operate on objects within that schema.
* [Use Cases for Custom PyFunc Models](/concepts/use-cases-for-custom-pyfunc-models.md) - Scenarios where custom pyfunc is needed: unsupported frameworks, preprocessing/postprocessing, per-request branching logic, and fully custom code deployment.
* [Use cases for DeepSpeed distributor](/concepts/use-cases-for-deepspeed-distributor.md) - Scenarios where DeepSpeed distributor is beneficial: low GPU memory, large model training, and large input data such as batch inference.
* [User and Session Tracking in GenAI Traces](/concepts/user-and-session-tracking-in-llm-traces.md) - Using standardized metadata fields (mlflow.trace.user and mlflow.trace.session) to track user identity and conversation sessions for behavior analysis, personalization, and multi-turn conversation tracking.
* [User and Session Tracking in LLM Traces](/concepts/user-and-session-tracking-in-llm-traces.md) - Using standardized mlflow.trace.user and mlflow.trace.session metadata fields to associate traces with specific users and group them into multi-turn conversations for behavioral analytics.
* [User and Session Tracking in MLflow Traces](/concepts/user-and-session-tracking-in-llm-traces.md) - Using standardized mlflow.trace.user and mlflow.trace.session metadata fields to associate traces with specific users and group traces from multi-turn conversations for behavior analysis and personalization.
* [User and Session Tracking with Standard Metadata Fields](/concepts/user-and-session-tracking-with-standard-metadata-fields.md) - MLflow's standard metadata fields `mlflow.trace.user` and `mlflow.trace.session` for associating traces with users and grouping multi-turn conversations
* [User Assignment and Permissions for Labeling](/concepts/user-assignment-and-permissions-for-labeling.md) - The process of assigning domain experts to labeling sessions, including automatic WRITE permission grants on the underlying MLflow experiment and identity provisioning for external users.
* [User Feedback Collection in MLflow Tracing](/concepts/user-feedback-collection-in-mlflow-tracing.md) - MLflow provides a structured way to capture user feedback as assessments on traces, enabling quality tracking and evaluation dataset creation for GenAI applications.
* [User Proxy Agent](/concepts/user-proxy-agent.md) - An AutoGen agent that acts as an intermediary to execute tool calls and manage conversation termination, typically with human_input_mode='NEVER' for automated workflows
* [User-to-Machine (U2M) Authentication](/concepts/user-to-machine-u2m-authentication.md) - Authentication flow where a human user authenticates via their IdP (with MFA if configured) and accesses shared data through tools like Power BI or Tableau using the OAuth Authorization Code Grant flow.
* [User-to-Machine (U2M) Authentication Flow](/concepts/user-to-machine-u2m-authentication-flow.md) - OIDC-based authentication flow for interactive applications like Power BI and Tableau accessing Delta Sharing data
* [Using Columns Clause for Conditional Masking](/concepts/using-columns-clause-for-conditional-masking.md) - A pattern enabling column masks to reference additional columns or static parameters via the USING COLUMNS clause, allowing conditional logic based on values beyond the masked column itself.
* [Using existing Delta tables as feature tables](/concepts/using-existing-delta-tables-as-feature-tables.md) - Process of converting an existing Delta table in Unity Catalog into a feature table by adding NOT NULL constraints and a primary key constraint via ALTER TABLE DDL statements.
* [Using Views as Feature Tables](/concepts/using-views-as-feature-tables.md) - Simple SELECT views over Delta tables in Unity Catalog can serve as feature tables for offline training, but cannot be published to online stores or used for model serving.
* [V1ToV2SqlMigration](/concepts/v1tov2sqlmigration.md) - A Databricks migration utility class that copies trace spans and annotations from schema-linked UC trace tables to table-prefix format using Spark SQL; idempotent and safe to rerun
* [VACUUM after PURGE](/concepts/vacuum-after-purge.md) - After running APPLY (PURGE), old files containing soft-deleted data may still exist and VACUUM should be used to physically delete them.
* [VACUUM and Bloom filter index file cleanup](/concepts/vacuum-and-bloom-filter-index-file-cleanup.md) - Underlying Bloom filter index files are cleaned only when the Delta table is vacuumed after all indexes are dropped
* [VACUUM and File Visibility in Delta Lake](/concepts/vacuum-and-file-visibility-in-delta-lake.md) - The caveat that files not tracked by the Delta Lake transaction log are invisible to Delta Lake and can be deleted by VACUUM, meaning writes during conversion must go through Delta Lake.
* [VACUUM Command (Databricks)](/concepts/vacuum-command-databricks.md) - A Databricks SQL command that removes unused data files from Delta, Iceberg, and other table directories to free storage space.
* [VACUUM DRY RUN](/concepts/vacuum-dry-run.md) - A preview mode for VACUUM that returns up to 1000 files that would be deleted without actually removing them.
* [VACUUM FULL vs VACUUM LITE](/concepts/vacuum-full-vs-vacuum-lite.md) - Two modes of VACUUM operation on Delta tables: FULL scans all files in the table directory, while LITE uses the transaction log to identify deletable files more efficiently.
* [VACUUM Retention Threshold](/concepts/vacuum-retention-threshold.md) - The configurable time window (default 7 days) that determines which data files are safe to delete; files modified or logically removed within this window are retained.
* [VACUUM Safety Check](/concepts/vacuum-safety-check.md) - A Delta Lake safety mechanism that prevents dangerous VACUUM operations with too-short retention intervals, controllable via spark.databricks.delta.retentionDurationCheck.enabled.
* [Vacuumed Data File Errors in Delta Sharing](/concepts/vacuumed-data-file-errors-in-delta-sharing.md) - Querying a shared table fails with 404 'The specified key does not exist' when a pre-signed URL references a data file that has been vacuumed from a historical table version.
* [VALIDATE Clause](/concepts/validate-clause.md) - A validation clause in COPY INTO that checks data can be parsed, schema matches, and constraints are met without writing to the table
* [validate_serving_input](/concepts/validate_serving_input.md) - MLflow API to validate that model input conforms to the JSON format expected by Databricks Model Serving endpoints before deployment.
* [Validation of Judge Alignment](/concepts/validation-of-judge-alignment.md) - A method to quantitatively compare original versus aligned judge performance against human ground truth, measuring accuracy improvement.
* [VARIANT-based masking for multiple column types](/concepts/variant-based-masking-functions.md) - Using a single VARIANT-accepting and VARIANT-returning masking UDF to handle columns of different types (INT, DOUBLE, DECIMAL, DATE, struct) without needing separate policies per type.
* [VARIANT-based masking functions](/concepts/variant-based-masking-functions.md) - Using the VARIANT type to write a single masking UDF that handles multiple target column types (INT, DOUBLE, DECIMAL, DATE, STRUCT) by accepting and returning VARIANT, which Databricks automatically casts to the target column's type.
* [VARIANT-based masking functions for multiple column types](/concepts/variant-based-masking-functions.md) - Using a single VARIANT-accepting UDF to mask columns of different data types (INT, DOUBLE, DECIMAL, DATE, etc.) and even struct columns, reducing the number of UDFs and policies needed.
* [Vector Index for LLM Applications](/concepts/vector-index-for-llm-applications.md) - Using vector indexes and Databricks AI Search for fast similarity searches to provide context and domain knowledge in LLM queries
* [Vended Credentials Access Delegation](/concepts/vended-credentials-access-delegation.md) - An access delegation mode (ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS) used in Snowflake's Iceberg REST Catalog integration to delegate credential management for accessing shared data.
* [VERIFY ALL FILES mode](/concepts/verify-all-files-mode.md) - FSCK REPAIR TABLE mode that checks all files by reading their contents, including data files and deletion vectors
* [VERIFY ALL FILES verification](/concepts/verify-all-files-verification.md) - A comprehensive verification mode for FSCK REPAIR TABLE that checks both metadata and actual data file contents, including data file readability and deletion vector integrity.
* [Version Compatibility for Databricks Connect](/concepts/version-compatibility-for-databricks-connect.md) - Specific version requirements for Databricks Connect including supported Databricks Runtime versions (7.3 LTS to 12.2 LTS), Python minor version matching, and exact package version alignment.
* [Version compatibility for feature table upgrade](/concepts/version-compatibility-for-feature-table-upgrade.md) - Upgrading tags and time series columns is not supported in Databricks Runtime 13.2 ML and below; older client versions (0.1.2 and below) throw errors on comment mismatch instead of skipping.
* [Version Number Synchronization](/concepts/version-number-synchronization.md) - The process of ensuring Unity Catalog model version numbers match the corresponding workspace model version numbers by inserting placeholder versions for gaps
* [Version Pinning with Dot-Asterisk Notation](/concepts/version-pinning-with-dot-asterisk-notation.md) - Databricks recommends using 'X.Y.*' (pip) to pin the major.minor version while allowing the latest patch version of databricks-connect to be installed.
* [VERSION_MISMATCH](/concepts/version_mismatch.md) - Sub-error indicating the managed and external DeltaLog versions do not match, often due to a concurrent UNSET MANAGED command
* [VERSION_MISMATCH error](/concepts/version_mismatch.md) - A sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED raised when the managed DeltaLog version does not match the expected version, often due to a concurrent migration succeeding first.
* [VERSION_MISMATCH error (Delta managed migration)](/concepts/version_mismatch.md) - Sub-error of DELTA_ALTER_TABLE_SET_MANAGED_FAILED indicating the managed DeltaLog version doesn't match expectations, typically due to a concurrent migration
* [VERSION_MISMATCH in Delta Logs](/concepts/version_mismatch.md) - A sub-error of DELTA_ALTER_TABLE_UNSET_MANAGED_FAILED where the managed DeltaLog and external DeltaLog versions do not match, often due to concurrent rollback operations.
* [VERSION_MISMATCH Sub-Error](/concepts/version_mismatch-sub-error.md) - Sub-error raised when the managed DeltaLog version does not match the expected version, often due to concurrent migration
* [View constraints on table_changes (Databricks)](/concepts/view-constraints-on-table_changes-databricks.md) - Databricks enumerates specific view features that make a view incompatible with the table_changes function: multiple relations, non-deterministic expressions, non-Delta tables, non-shared views, subqueries, and unsupported operators.
* [View determinism requirement for table_changes](/concepts/view-determinism-requirement-for-table_changes.md) - Views used with table_changes must be deterministic; non-deterministic expressions cause a NON_DETERMINISTIC_EXPRESSIONS error.
* [View Restrictions in Delta Change Tracking](/concepts/view-restrictions-in-delta-change-tracking.md) - Constraints that prevent using table_changes on views, including multi-relation references, non-deterministic expressions, subqueries, unsupported operators, and non-Delta/non-shared views
* [View structural constraints for table_changes](/concepts/view-structural-constraints-for-table_changes.md) - Views used with table_changes must reference a single relation, avoid subqueries, and avoid unsupported operators.
* [View-based On-Read PII Redaction](/concepts/view-based-on-read-pii-redaction.md) - Alternative approach that applies ai_mask in a Unity Catalog view to redact PII at query time without duplicating storage.
* [Viewing and Searching MLflow Experiments](/concepts/viewing-and-searching-mlflow-experiments.md) - Experiments are viewed from the Experiments page (sidebar under AI/ML), from the workspace for workspace experiments, or from the notebook sidebar for notebook experiments. Filter experiments by name, location, or by searching tags like tags.mlflow.note.content for descriptions.
* [Viewing Share Grants and Recipients](/concepts/viewing-share-grants-and-recipients.md) - Inspecting which recipients have access to a share or which shares a recipient can access using Catalog Explorer, SQL, or CLI.
* [Views as Feature Tables](/concepts/views-as-feature-tables.md) - Simple SELECT views over a single Delta table can serve as feature tables for offline training/evaluation but cannot be published to online stores or served.
* [Views in Unity Catalog](/concepts/views-in-unity-catalog.md) - Views are read-only objects defined by stored SQL queries, including standard views, materialized views (pre-computed cached results with REFRESH privilege), and metric views (reusable metric definitions).
* [vLLM Engine Integration for Model Serving](/concepts/vllm-engine-integration-for-model-serving.md) - The use of vLLM as the inference engine powering custom LLM deployments on Databricks, including local testing, entrypoint configuration, and OpenAI-compatible API at /invocations.
* [VOID type in Delta Lake](/concepts/void-type-in-delta-lake.md) - An unsupported column type in Delta Lake schemas; when all columns or all struct fields are of VOID type, the table is considered unwritable.
* [VOID Type in Delta Lake Schemas](/concepts/void-type-in-delta-lake.md) - The VOID data type in Delta Lake represents columns or fields that have no valid type, making schemas containing it effectively empty and unwritable.
* [Volume Export for Large Delta Tables](/concepts/volume-export-for-large-delta-tables.md) - Technique to export oversized Delta tables to Parquet files in Unity Catalog volumes for direct loading via PyTorch or Hugging Face datasets, avoiding Spark overhead.
* [Volumes for Unstructured Data](/concepts/volumes-for-unstructured-data.md) - Volumes are securable objects in Unity Catalog for unstructured data in cloud storage, supporting file-level read/write access without SQL query operations, available as managed or external volumes.
* [VPC Gateway Endpoints for In-Region Access](/concepts/vpc-gateway-endpoints-for-in-region-access.md) - Best practice recommendation for recipients to use VPC gateway endpoints or interface endpoints for S3 instead of NAT gateways when accessing shared data in-region to reduce costs and enhance security.
* [Webhook Lifecycle Management](/concepts/webhook-lifecycle-management.md) - Management operations for webhooks: create (in TEST_MODE or ACTIVE), test with mock events, update (status and properties), list, and delete via REST API or Python client.
* [Webhook Payload Structure](/concepts/webhook-payload-structure.md) - The JSON payload sent to webhook endpoints, which includes event type, webhook ID, timestamp, model name, version, and event-specific fields. Sensitive info like artifact paths are excluded.
* [Webhook Security and Verification](/concepts/webhook-security-and-verification.md) - Security measures including HMAC-SHA256 signature (X-Databricks-Signature header), optional Authorization header, and client-side verification to validate webhook origin.
* [Weighted Edges](/concepts/weighted-edges.md) - Edges in a graph can carry weights representing properties such as capacity, distance, or strength of a relationship, enabling richer analysis like shortest-path with costs.
* [WHEN MATCHED Clause](/concepts/when-matched-clause.md) - Specifies actions (DELETE or UPDATE) to perform when a source row matches a target table row based on the merge condition, with optional conditional evaluation and ordering of multiple clauses.
* [WHEN NOT MATCHED (INSERT) Clause](/concepts/when-not-matched-insert-clause.md) - Inserts a new row into the target Delta table when a source row does not match any target row, supporting INSERT * (all columns), INSERT with specific columns, and optional EXCEPT clause.
* [WHEN NOT MATCHED BY SOURCE Clause](/concepts/when-not-matched-by-source-clause.md) - A merge clause that acts on target rows that have no match in the source table, supporting DELETE and UPDATE actions (Databricks Runtime 12.2 LTS+), with conditions limited to target table columns.
* [When to Avoid Delta Lake Conversion](/concepts/when-to-avoid-delta-lake-conversion.md) - Scenarios where maintaining Parquet format may be preferable, such as upstream/downstream system incompatibility with Delta Lake.
* [When to use DDP](/concepts/when-to-use-ddp.md) - Conditions for using DDP: model fits in single GPU memory, scaling via data throughput, and need for simplest distributed training approach.
* [When to use DDP vs FSDP/DeepSpeed](/concepts/when-to-use-ddp-vs-fsdpdeepspeed.md) - Decision criteria for choosing DDP over alternative distributed training techniques like FSDP or DeepSpeed based on model size and memory constraints.
* [When to use FSDP](/concepts/when-to-use-fsdp.md) - Decision criteria for adopting FSDP: when models are too large for single GPU memory, when training models in the 20B to 120B+ parameter range, and when more memory efficiency than DDP is needed.
* [When to use FSDP vs DDP vs DeepSpeed](/concepts/when-to-use-ddp-vs-fsdpdeepspeed.md) - Decision framework for choosing between FSDP, DDP, and DeepSpeed based on model size and memory efficiency needs
* [When to Use Third-Party vs Built-in Scorers](/concepts/when-to-use-third-party-vs-built-in-scorers.md) - Decision framework for choosing between built-in MLflow LLM judges and third-party scorers based on evaluation needs, domain specificity, and cost considerations
* [Whole Table Operation Conflicts in Delta Lake](/concepts/whole-table-conflicts-in-delta-lake.md) - Conflicts arising when a transaction reads or replaces the entire table while another concurrent operation modifies it, including sub-types WHOLE_TABLE_READ and WHOLE_TABLE_REPLACE.
* [Whole Table Read vs Filtered Read in Delta Lake](/concepts/whole-table-read-vs-filtered-read-in-delta-lake.md) - A pattern where reading an entire Delta table (without filters) increases the likelihood of transaction conflicts with concurrent delete operations, mitigated by adding filter predicates.
* [Whole Table Read/Replace Conflicts](/concepts/whole-table-readreplace-conflicts.md) - Conflict scenarios where a transaction reads or replaces the entire table while a concurrent modification occurs, with recommendations to narrow data scope or retry.
* [Whole Table Scan Conflicts in Delta Lake](/concepts/whole-table-scan-conflicts-in-delta-lake.md) - Transaction conflicts arising when a Delta Lake operation reads or replaces an entire table concurrently with other modifications, resolvable by narrowing query scope with filters or retrying.
* [Whole-Table Conflicts in Delta Lake](/concepts/whole-table-conflicts-in-delta-lake.md) - Transaction conflicts that arise when an operation attempts to read or replace an entire table while a concurrent modification is in progress.
* [Whole-Table Modification Conflicts in Delta Lake](/concepts/whole-table-conflicts-in-delta-lake.md) - A class of transaction conflicts in Delta Lake where either the current or concurrent transaction attempts to operate on the entire table without filters, causing a conflict.
* [Whole-Table Operation Pattern Conflicts](/concepts/whole-table-operation-pattern-conflicts.md) - A class of Delta Lake conflicts that arise when a transaction attempts to modify or read the entire table (without filters), conflicting with concurrent operations; resolution requires adding filters to narrow the data scope.
* [Whole-Table Read vs Filtered Read Conflict Patterns](/concepts/whole-table-versus-filtered-read-conflicts.md) - A class of Delta Lake transaction conflicts arising when a transaction reads an entire table (without filters) concurrently with a delete operation, resolvable by adding query filters to narrow the data scope.
* [Whole-table versus filtered read conflicts](/concepts/whole-table-versus-filtered-read-conflicts.md) - A class of Delta Lake concurrency conflicts where reading an entire table (without filters) collides with concurrent delete operations, with solutions involving narrowing the read scope via predicates or partition hints.
* [Whole-Table vs Filtered Delta Lake Operations](/concepts/whole-table-vs-filtered-delta-lake-operations.md) - Conflict patterns arising when one transaction modifies an entire table (no filters, full replace) while another attempts concurrent modifications, requiring narrowing data scope via filters.
* [Windows Path Space Issue in Databricks Connect](/concepts/windows-path-space-issue-in-databricks-connect.md) - On Windows, Databricks Connect fails with a filename syntax error if installed into a directory path containing spaces; workarounds include using a space-free path or the short name form.
* [WITH HISTORY clause for Databricks deep clone](/concepts/with-history-clause-for-deep-clone.md) - When deep cloning a streaming table in Databricks, the WITH HISTORY clause is required; a DEEP CLONE without it will fail with REQUIRES_WITH_HISTORY.
* [WITH HISTORY Clause for Deep Clone](/concepts/with-history-clause-for-deep-clone.md) - A required clause when performing a deep clone of a streaming table in Databricks to preserve the table's history
* [With History Requirement for Deep Clone of Streaming Tables](/concepts/with-history-requirement-for-deep-clone-of-streaming-tables.md) - The WITH HISTORY clause is required when performing a DEEP CLONE on a streaming table
* [WITH SCHEMA EVOLUTION for MERGE](/concepts/with-schema-evolution-for-merge.md) - A modifier that automatically updates the target Delta table schema to match the source table schema during a MERGE operation (Databricks Runtime 15.2+).
* [withEventTimeOrder for Initial Snapshot Processing](/concepts/witheventtimeorder-for-initial-snapshot-processing.md) - Feature to avoid data drops during initial snapshot processing by ordering data by event time instead of file modification time, using time bucket filtering.
* [withEventTimeOrder for Initial Snapshots](/concepts/witheventtimeorder-for-initial-snapshots.md) - An option to preserve event-time ordering during initial snapshot processing in stateful streaming queries on Delta tables
* [Workflow Architecture Patterns for Ray and Spark](/concepts/workflow-architecture-patterns-for-ray-and-spark.md) - Recommended patterns for integrating Ray and Spark pipelines include isolating ETL in a subtask, using Spark for data handling with Ray for computation, running Ray inside Spark functions (UDFs/foreachBatch), and concurrent Spark/Ray operations.
* [Workload Discovery for Deprecated Models](/concepts/workload-discovery-for-deprecated-models.md) - A SQL-based method to identify workloads, endpoints, and owners still using deprecated or retired models by querying Databricks system tables.
* [Workload YAML Configuration](/concepts/workload-yaml-configuration.md) - The YAML schema accepted by `air run --file` to define GPU training jobs, with fields for experiment name, environment, compute, code source, command, parameters, secrets, and reliability settings.
* [Workload YAML configuration (train.yaml)](/concepts/workload-yaml-configuration-trainyaml.md) - Pattern for defining distributed training workloads declaratively using a train.yaml file submitted via `air run -f train.yaml`.
* [Workload YAML configuration for distributed training](/concepts/workload-yaml-configuration-for-distributed-training.md) - YAML schema defining experiment name, environment dependencies, compute resources, code source, and inline command for launching distributed ML workloads on Databricks.
* [Workload YAML Configuration for Ray on Databricks](/concepts/workload-yaml-configuration-for-ray-on-databricks.md) - YAML configuration format for the Databricks AI Runtime CLI that defines compute resources, dependencies, code sources, and commands for distributed Ray training workloads.
* [Workload YAML for Distributed Training](/concepts/workload-yaml-for-distributed-training.md) - Declarative YAML configuration files (e.g., train.yaml) used with the air CLI to define and submit end-to-end distributed training workloads on Databricks
* [Workload YAML pattern for distributed training](/concepts/workload-yaml-pattern-for-distributed-training.md) - Configuration-driven approach where distributed training workloads are defined in YAML files (e.g., train.yaml) and submitted via `air run -f`, separating launcher scripts from training code.
* [Workspace Admin (Unity Catalog)](/concepts/workspace-admin-unity-catalog.md) - An admin role operating within a single Databricks workspace, managing workspace membership, jobs, and workspace objects, with default metastore privileges on auto-enabled workspaces.
* [Workspace Admin Privileges in Feature Store](/concepts/workspace-admin-privileges-in-feature-store.md) - Workspace administrators automatically have CAN MANAGE permission on all feature tables and can set permissions on the entire Feature Store
* [Workspace Admin Privileges in Unity Catalog](/concepts/workspace-admin-privileges-in-unity-catalog.md) - How workspace admin roles differ depending on whether Unity Catalog was auto-enabled or manually assigned, including the ability to create objects and grant access.
* [Workspace Admin Role (Unity Catalog)](/concepts/workspace-admin-unity-catalog.md) - A highly privileged role operating within a single Databricks workspace, managing workspace membership, jobs, and workspace objects.
* [Workspace admins (Unity Catalog)](/concepts/workspace-admin-unity-catalog.md) - Admin role operating within a single workspace, managing workspace membership, jobs, and workspace objects, with default privileges on automatically-provisioned metastores.
* [Workspace Admins in Unity Catalog](/concepts/workspace-admin-unity-catalog.md) - Workspace-scoped admin role managing workspace membership, jobs, and objects, with additional default privileges on the attached metastore when Unity Catalog is auto-enabled.
* [Workspace Built-in Hive Metastore](/concepts/built-in-hive-metastore.md) - A managed Hive metastore service that deploys per-Databricks-workspace and serves as a central metadata repository for all clusters in that workspace.
* [Workspace Catalog (Unity Catalog)](/concepts/workspace-catalog-unity-catalog.md) - A catalog provisioned for a workspace during automatic Unity Catalog enablement, where workspace admins are default owners and all workspace users receive USE CATALOG privilege.
* [Workspace Catalog Default Ownership and Privileges](/concepts/workspace-catalog-default-privileges.md) - When a workspace is auto-enabled for Unity Catalog, a workspace catalog is provisioned with workspace admins as default owners, granting them privilege management and ownership transfer capabilities over objects in that catalog.
* [Workspace Catalog Default Privileges](/concepts/workspace-catalog-default-privileges.md) - When a workspace is automatically enabled for Unity Catalog, a workspace catalog is created where workspace admins are default owners and all workspace users get USE CATALOG on the catalog plus several privileges on the default schema.
* [Workspace Catalog in Unity Catalog](/concepts/workspace-catalog-in-unity-catalog.md) - An automatically created catalog that is provisioned for each new workspace when metastore auto-assignment is enabled, giving all workspace users privileges to create objects within it.
* [Workspace Catalog Ownership](/concepts/workspace-catalog-ownership.md) - Workspace admins are default owners of the workspace catalog when one is provisioned, granting them the ability to manage privileges and transfer ownership within that catalog.
* [Workspace Default Serverless Budget Policy](/concepts/workspace-default-serverless-budget-policy.md) - The fallback budget policy applied to MLflow serverless workloads when no specific policy is assigned to an experiment; if disabled, workloads fail with a 403 error.
* [Workspace default storage path](/concepts/workspace-default-storage-path.md) - The default storage path configured for a Databricks workspace during deployment using a storage configuration object, used for internal Unity Catalog processing.
* [Workspace Deletion and Catalog Retention](/concepts/workspace-deletion-and-catalog-retention.md) - Deleting a Databricks workspace does not automatically delete its default Unity Catalog catalog; the catalog must be dropped manually from another workspace assigned to the same metastore.
* [Workspace Feature Store (Deprecated)](/concepts/workspace-feature-store-deprecated.md) - Legacy version of Databricks Feature Store for workspaces not enabled for Unity Catalog, now deprecated in favor of the Unity Catalog-backed Feature Store.
* [Workspace Feature Store UI](/concepts/workspace-feature-store-ui.md) - Databricks UI for managing, searching, and governing feature tables with lineage tracking and access control.
* [Workspace Model Registry](/concepts/workspace-model-registry.md) - A Databricks-hosted, legacy version of the MLflow Model Registry for managing the full lifecycle of ML models, providing model lineage, versioning, stage transitions, webhooks, and email notifications.
* [Workspace Model Registry Quota Limits](/concepts/workspace-model-registry-quota-limits.md) - Quota limits imposed on the total number of registered models and model versions per workspace in the Workspace Model Registry, effective May 2024, with recommendations for cleanup and retention strategies.
* [Workspace Model Registry Webhooks](/concepts/workspace-model-registry-webhooks.md) - Event-driven notifications that trigger actions (HTTP calls or job runs) when MLflow Model Registry events occur in a Databricks workspace.
* [Workspace-based and Git-based Code Workflows](/concepts/workspace-based-and-git-based-code-workflows.md) - Two code workflow modes supported by the AI Runtime CLI: sourcing code from Databricks workspaces or from git repositories.
* [Workspace-based code workflows](/concepts/workspace-based-code-workflows.md) - Approach for submitting training workloads using code stored in the Databricks workspace.
* [Workspace-based vs git-based code workflows](/concepts/workspace-based-vs-git-based-code-workflows.md) - AI Runtime CLI supports both Databricks workspace-based code and git-based code workflows for submitting training jobs.
* [Workspace-Catalog Binding](/concepts/workspace-catalog-binding.md) - Workspace-level restrictions that limit which workspaces can access specific catalogs, external locations, and storage credentials.
* [Workspace-Level Restrictions](/concepts/workspace-level-restrictions.md) - Control over where users can access data by binding securable objects like catalogs and external locations to specific workspaces.
* [Workspace-Metastore Assignment](/concepts/workspace-metastore-assignment.md) - The process of enabling a Databricks workspace for Unity Catalog by assigning it to a Unity Catalog metastore, either during workspace creation or later via the account console.
* [Workspace-Metastore Attachment](/concepts/workspace-metastore-linking.md) - The process of linking a Unity Catalog metastore to Databricks workspaces in the same region, giving all linked workspaces a unified view of metastore data.
* [Workspace-Metastore Linking](/concepts/workspace-metastore-linking.md) - The process of attaching a Databricks workspace to a Unity Catalog metastore, allowing multiple workspaces in the same region to share the same view of data with unified access control.
* [WorkspaceClient (Databricks SDK for Python)](/concepts/workspaceclient-databricks-sdk-for-python.md) - The class used to access Databricks Utilities (dbutils) from external Python environments via the Databricks SDK.
* [WorkspaceClient and dbutils](/concepts/workspaceclient-dbutils.md) - The WorkspaceClient class from the Databricks SDK for Python exposes a 'dbutils' variable that provides access to Databricks Utilities (fs and secrets) in Databricks Connect.
* [WorkspaceClient Authentication](/concepts/workspaceclient-authentication.md) - Methods for initializing the Databricks SDK WorkspaceClient including hard-coded credentials, configuration profiles, and environment variables
* [WorkspaceClient Authentication Methods](/concepts/workspaceclient-authentication.md) - Three supported authentication approaches for initializing the Databricks SDK WorkspaceClient: hard-coded credentials, configuration profiles, and environment variables (DATABRICKS_HOST and DATABRICKS_TOKEN).
* [WorkspaceClient dbutils](/concepts/workspaceclient-dbutils.md) - The WorkspaceClient class from the Databricks SDK for Python exposes a dbutils variable that provides access to limited Databricks Utilities through Databricks Connect
* [Wrapping built-in LLM judges in custom scorers](/concepts/wrapping-built-in-llm-judges-in-custom-scorers.md) - Technique of wrapping MLflow's predefined LLM judges (like is_context_relevant) inside a custom code-based scorer to preprocess trace data or post-process judge feedback before returning results.
* [Wrapping predefined LLM judges in custom scorers](/concepts/wrapping-built-in-llm-judges-in-custom-scorers.md) - The pattern of wrapping MLflow's built-in LLM judge functions inside custom code-based scorers to preprocess trace data or post-process judge feedback
* [Write Authentication for Feature Publishing](/concepts/write-authentication-for-feature-publishing.md) - Authentication required when publishing feature tables from the Databricks Feature Store to third-party online stores, supporting instance profiles or Databricks secrets with a write_secret_prefix.
* [Write Authentication for Publishing Feature Tables](/concepts/write-authentication-for-publishing-feature-tables.md) - Required write credentials (via instance profile or secrets with write_secret_prefix) for publishing feature tables to third-party online stores from Databricks Feature Store.
* [Write Authentication for Publishing Feature Tables to Third-Party Online Stores](/concepts/write-authentication-for-publishing-feature-tables-to-third-party-online-stores.md) - Methods and requirements for authenticating write operations when publishing Databricks feature tables to external online stores like DynamoDB or SQL databases.
* [Writing Ray Data to Unity Catalog](/concepts/writing-ray-data-to-unity-catalog.md) - Writing Ray datasets back to Unity Catalog tables using ray.data.Dataset.write_databricks_table(), which stages data in Unity Catalog Volumes before writing with Spark.
* [Writing Ray Data to Unity Catalog Tables](/concepts/writing-ray-data-to-unity-catalog.md) - The ray.data.Dataset.write_databricks_table function for writing Ray datasets to Unity Catalog tables via temporary storage in Unity Catalog Volumes.
* [WRONG_REQUIRED_TABLE_PROPERTY](/concepts/wrong_required_table_property.md) - Error sub-type indicating a required table property for IcebergCompatV1 is set to an incorrect value
* [WRONG_REQUIRED_TABLE_PROPERTY (Delta Iceberg error)](/concepts/wrong_required_table_property.md) - A sub-error of DELTA_ICEBERG_COMPAT_V1_VIOLATION that occurs when a required table property for IcebergCompatV1 is set to an incorrect value
* [XGBoost classification and regression on Databricks](/concepts/xgboost-classification-and-regression-on-databricks.md) - Using XGBoost for both classification (with ML pipeline) and regression (with cross-validation) tasks in Scala notebooks on Databricks.
* [XGBoost Classification on Databricks](/concepts/xgboost-classification-on-databricks.md) - Example notebook showing XGBoost classification embedded in an MLlib ML pipeline on Databricks
* [XGBoost Classification with ML Pipeline Notebook](/concepts/xgboost-classification-with-ml-pipeline-notebook.md) - An example Databricks notebook demonstrating XGBoost classification embedded in an MLlib ML pipeline
* [XGBoost hyperparameter tuning with MLlib cross-validation](/concepts/xgboost-hyperparameter-tuning-with-mllib-cross-validation.md) - Using MLlib's CrossValidator to tune XGBoost model hyperparameters (e.g., n_estimators, max_depth) in a distributed Scala environment.
* [XGBoost integration with MLlib ML pipelines](/concepts/xgboost-integration-with-mllib-ml-pipelines.md) - Embedding an XGBoost model as a stage within an MLlib Pipeline for end-to-end machine learning workflows on Databricks.
* [XGBoost on Databricks Runtime ML](/concepts/xgboost-on-databricks-runtime-ml.md) - Using XGBoost within Databricks Runtime ML which includes pre-installed ML libraries and optimizations
* [XGBoost Parameter Mapping for PySpark](/concepts/xgboost-parameter-mapping-for-pyspark.md) - Parameter differences between the standard xgboost package and sparkdl.xgboost, including unsupported parameters and alternative column-based parameters like weightCol, validationIndicatorCol, and baseMarginCol.
* [XGBoost Parameter Mapping: sparkdl vs Native xgboost](/concepts/xgboost-parameter-mapping-sparkdl-vs-native-xgboost.md) - The mapping and semantic differences between sparkdl.xgboost parameters and native Python xgboost package parameters, including missing, weightCol, baseMarginCol, and unsupported parameters like gpu_id and output_margin.
* [XGBoost Regression with Cross-Validation](/concepts/xgboost-regression-with-cross-validation.md) - Example notebook demonstrating XGBoost regression with MLlib cross-validation on Databricks
* [XGBoost Regression with Cross-Validation Notebook](/concepts/xgboost-regression-with-cross-validation.md) - An example Databricks notebook demonstrating XGBoost regression tuned via MLlib cross-validation
* [XGBoost with MLlib Integration](/concepts/xgboost-with-mllib-integration.md) - Using XGBoost models within Apache Spark MLlib pipelines for distributed training on Databricks
* [XGBoost with MLlib ML Pipeline](/concepts/xgboost-with-mllib-integration.md) - Embedding XGBoost models into Apache Spark MLlib ML pipelines for distributed training
* [XGBoost with MLlib Pipeline API](/concepts/xgboost-with-mllib-pipeline-api.md) - Embedding XGBoost models as stages within MLlib ML pipelines for structured workflows
* [xgboost.spark Module](/concepts/xgboostspark-module.md) - PySpark estimators (SparkXGBRegressor, SparkXGBClassifier, SparkXGBRanker) that support inclusion of XGBoost in SparkML Pipelines, available in xgboost>=1.7.
* [xgboost.spark Module (PySpark XGBoost Estimators)](/concepts/xgboostspark-module.md) - A Python module in xgboost>=1.7 providing SparkXGBRegressor, SparkXGBClassifier, and SparkXGBRanker for integrating XGBoost into SparkML Pipelines
* [xgboost.spark Module API](/concepts/xgboostspark-module.md) - New PySpark estimators (SparkXGBRegressor, SparkXGBClassifier, SparkXGBRanker) that integrate XGBoost into SparkML Pipelines
* [YAML Job Configuration for AI Runtime](/concepts/yaml-job-configuration-for-ai-runtime.md) - The declarative YAML-based job configuration format used by the AI Runtime CLI to define training jobs that can be checked into source control.
* [YAML-based job configuration](/concepts/yaml-based-job-configuration.md) - Declarative approach to defining training jobs using YAML configuration files that can be checked into source control.
* [YAML-based job configuration for AI Runtime](/concepts/yaml-based-job-configuration-for-ai-runtime.md) - Declarative YAML configuration format used by the AI Runtime CLI to define training jobs, enabling source-control integration.
* [YAML-based job configuration for training](/concepts/yaml-based-job-configuration-for-training.md) - Pattern of defining distributed training jobs declaratively in YAML files so they can be checked into source control, used by the AI Runtime CLI.
* [YAML-based LLM Fine-tuning Configuration](/concepts/yaml-based-llm-fine-tuning-configuration.md) - A structured YAML configuration pattern used by LLM Foundry to specify model architecture, optimizer, scheduler, FSDP settings, dataset loading, callbacks, and logging parameters for LLM fine-tuning.
* [Z-Ordering (Delta Lake)](/concepts/z-ordering-delta-lake.md) - A non-idempotent, incremental clustering technique that co-locates related data by number of tuples, improving data skipping performance.
* [ZeRO (Zero Redundancy Optimizer)](/concepts/zero-zero-redundancy-optimizer.md) - A memory optimization technique in DeepSpeed that shards optimizer states, gradients, and parameters across devices to enable training of large models
* [ZeRO Stages (Stage 1, 2, 3)](/concepts/zero-stages-stage-1-2-3.md) - Three stages of optimizer state sharding in DeepSpeed's ZeRO, providing increasing levels of memory savings by partitioning optimizer states, gradients, and model parameters respectively.
* [Zero-Downtime Endpoint Updates](/concepts/zero-downtime-endpoint-updates.md) - Databricks performs zero-downtime updates by keeping the existing endpoint configuration active until the new one is ready, with billing for both configurations during transition.
* [Zero-downtime endpoint updates and maintenance](/concepts/zero-downtime-endpoint-updates.md) - Databricks keeps existing endpoints serving while new configurations deploy, then performs zero-downtime transitions; during maintenance models are reloaded, and failures cause rollback ensuring no service interruption.
