---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: cc9c4148278c9be1c2cdc411e71f904a834f0e171eb56246df52a20265998c62
  pageDirectory: concepts
  sources:
    - use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md
  confidence: 1
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - compute-options-for-dbt-tasks-in-databricks
    - COFDTID
  citations:
    - file: use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md
title: Compute Options for dbt Tasks in Databricks
description: dbt tasks can run against serverless SQL warehouses, pro SQL warehouses, or Databricks all-purpose compute, but not job compute; SQL warehouse is recommended for most use cases.
tags:
  - dbt
  - databricks
  - compute
  - sql-warehouse
timestamp: "2026-06-19T23:21:46.908Z"
---

# Compute Options for dbt Tasks in Databricks

**Compute Options for dbt Tasks in Databricks** refers to the infrastructure choices available when running dbt Core projects as a task in a [Databricks Jobs](/concepts/dbt-task-in-databricks-jobs.md) workflow. The dbt task can target multiple compute types, each with distinct characteristics for development and production use. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Available Compute Options

You can run dbt transformations on the following compute types: ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

- A **serverless SQL warehouse** or **pro SQL warehouse**  
- **Databricks compute** (all-purpose compute or job clusters)  
- Any other dbt-supported warehouse (via a custom `profiles.yml`)

The dbt task runs the dbt Python process using Databricks compute, and the SQL generated by dbt is executed against the selected SQL warehouse. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Default Compute Behavior

When a workspace is Unity Catalog-enabled and Serverless Jobs is enabled, the job runs on Serverless compute by default. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Recommendations

- **Development** – Use a Databricks SQL warehouse (serverless or pro) to test the SQL generated by dbt and to leverage the SQL warehouse query history for debugging. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]
- **Production** – Use the dbt task in a Databricks job. The task runs dbt on Databricks compute (serverless or job clusters) while executing queries against the selected SQL warehouse. This separation allows you to automate, monitor, and notify on transformations. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

> **Important**  
> Using different Databricks Runtime versions for development (SQL warehouse) and production (Databricks compute) can lead to subtle differences in performance and SQL language support. Databricks recommends using the same runtime version for both. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Using a Custom Profiles.yml

To run dbt with a specific compute target (SQL warehouse or all-purpose compute), you can provide a custom `profiles.yml` file in your dbt project repository. This profile defines connection details such as host, HTTP path, and schema. The profile can use environment variables (e.g., `DBT_ACCESS_TOKEN`) to avoid hard‑coding secrets. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

The generated credentials are valid for the duration of the run (up to 30 days) and are automatically revoked after completion. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

When you configure the dbt task in the UI, you can point to the directory containing the `profiles.yml` file. If the field is left blank, the root of the repository is used by default. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Limitations

- **Job compute cannot be used as a target** – A dbt task can target only a SQL warehouse or all-purpose compute; job compute is not supported as a dbt target. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]
- **Python models require Databricks compute** – [dbt Python models](/concepts/dbt-python-models-on-databricks.md) (beta, dbt ≥1.3) cannot be run using a SQL warehouse. You must use Databricks compute for tasks that include Python models. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Accessing Task Output and Artifacts

After a dbt task runs, you can retrieve task output programmatically using the Databricks CLI or Jobs API. The response includes inline logs and a download link for archived artifacts (logs, results, manifests, configuration). For multi‑task jobs, use the individual task run ID, not the parent job run ID. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Related Concepts

- Databricks SQL Warehouse – The compute used to execute dbt‑generated SQL.
- Serverless SQL Warehouse – On‑demand SQL warehouse type.
- Pro SQL Warehouse – Classic pro‑tier SQL warehouse.
- Serverless Jobs – Default job compute when enabled and [Unity Catalog](/concepts/unity-catalog.md) is active.
- dbt Core – The open‑source transformation tool.
- [dbt-databricks](/concepts/dbt-databricks-adapter-package.md) – The recommended adapter for Databricks.
- [Databricks Jobs](/concepts/dbt-task-in-databricks-jobs.md) – Orchestration service for running dbt tasks.
- dbt profiles.yml – Configuration file for connecting to databases.

## Sources

- use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md

# Citations

1. [use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md](/references/use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws-9420061c.md)
