---
type: concept
x-llmwiki:
  schemaVersion: "0.1"
  contentHash: 3e76280676591e65b10e6b57dfac84e5cadae8524a6a2cdd0521c49e61d9ba11
  pageDirectory: concepts
  sources:
    - use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md
  confidence: 1
  provenanceState: extracted
  freshnessStatus: unverified
  aliases:
    - dbt-task-in-databricks-jobs
    - DTIDJ
    - Databricks Jobs
  citations:
    - file: use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md
title: dbt Task in Databricks Jobs
description: Running dbt Core projects as a task within Databricks Lakeflow Jobs, enabling automation, monitoring, multi-step workflows, and artifact archiving.
tags:
  - dbt
  - databricks-jobs
  - orchestration
timestamp: "2026-06-19T23:21:25.991Z"
---

# dbt Task in Databricks Jobs

The **dbt Task** is a job task type in [Databricks Jobs](/concepts/dbt-task-in-databricks-jobs.md) that allows you to run a dbt Core project as part of a workflow. By integrating dbt transformations directly into a job, you can automate, schedule, and monitor your data transformation pipelines alongside other tasks. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Overview

When you add a dbt task to a job, Databricks executes the specified dbt commands (e.g., `dbt deps`, `dbt seed`, `dbt run`) using a Python process on Databricks compute. The SQL generated by dbt is run against a selected Databricks SQL Warehouse (serverless or pro) or against a custom target defined in a `profiles.yml` file. The task can be part of a multi-step workflow, allowing you to chain data ingestion, transformation, and analysis in a single job. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Benefits

Running a dbt project as a job task provides several advantages:

- **Automation and scheduling** – you can schedule your dbt transformations to run on a recurring basis.
- **Monitoring and notifications** – you can track the status of dbt runs and receive alerts on success or failure.
- **Workflow integration** – you can combine dbt with other tasks such as Auto Loader ingestion, notebook analysis, or Python scripts.
- **Artifact archiving** – job runs automatically archive dbt artifacts including logs, results, manifests, and configuration files. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Requirements

To use a dbt task in a Databricks job, you must have:

- A dbt Core project stored in a [Databricks Git folder](/concepts/databricks-git-folders-for-cicd.md) (DBFS is not supported).
- A serverless or pro SQL warehouse enabled (for the recommended target configuration).
- The Databricks SQL entitlement.
- The `dbt-databricks` package installed (the Databricks-optimized fork of `dbt-spark`). ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Creating a dbt Job (UI)

To create a job with a dbt task using the Databricks UI:

1. Open the **Jobs & Pipelines** sidebar and click **Create > Job**.
2. Click the **dbt** tile to configure the first task.
3. Enter a task name and choose the source: **Git provider** (for a remote repository) or a workspace Git folder.
4. Specify the **Project directory** (e.g., a Git repository URL) and enter the dbt commands to run (e.g., `dbt deps`, `dbt seed`, `dbt run`). Each command must be prefixed with `dbt`.
5. Select a SQL warehouse to run the generated SQL. Optionally, specify a [Catalog and Schema](/concepts/catalog-and-schema.md) for the output.
6. (Optional) Change the compute configuration used to run the dbt CLI process by clicking **dbt CLI compute**.
7. Leave **Environment and Libraries** set to `dbt-default`.
8. Click **Create task** and then run the job. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Creating a dbt Job (API)

You can also create a dbt job programmatically using the Jobs API. The following example shows a JSON payload for a job with a single dbt task:

```json
{
  "name": "jaffle_shop dbt job",
  "git_source": {
    "git_url": "https://github.com/dbt-labs/jaffle_shop",
    "git_provider": "gitHub",
    "git_branch": "main",
    "sparse_checkout": { "patterns": ["models", "seeds"] }
  },
  "job_clusters": [ { "job_cluster_key": "dbt_CLI", "new_cluster": { ... } } ],
  "tasks": [{
    "task_key": "transform",
    "job_cluster_key": "dbt_CLI",
    "dbt_task": {
      "commands": ["dbt deps", "dbt seed", "dbt run"],
      "warehouse_id": "1a234b567c8de912"
    },
    "libraries": [ { "pypi": { "package": "dbt-databricks>=1.0.0,<2.0.0" } } ]
  }]
}
```

The `dbt_task` object contains the list of commands and the warehouse ID. The `libraries` array must include the `dbt-databricks` package. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Viewing Results

After the job completes, you can inspect the transformed data by running SQL queries from a notebook or the SQL editor. For example:

```sql
SELECT * FROM <schema>.customers LIMIT 10;
```

Replace `<schema>` with the schema configured in the task’s settings. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Accessing Task Output and Artifacts

You can retrieve the output of a completed dbt task using the Databricks CLI or the Jobs API. For multi-task jobs, you must use the individual dbt task’s run ID, not the parent job run ID.

- **CLI**: `databricks jobs get-run-output <task_run_id> --output JSON`
- **API**: `GET /api/2.0/jobs/runs/get-output?run_id=<task_run_id>`

The response includes inline logs and a download link to the archived artifacts (logs, results, manifests, configuration). ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Advanced: Using a Custom Profile

To run a dbt task with a target other than a SQL warehouse (e.g., all-purpose compute), you can provide a custom `profiles.yml` file in your repository. The file should define the target connection using environment variables for credentials. For example:

```yaml
jaffle_shop:
  target: databricks_job
  outputs:
    databricks_job:
      type: databricks
      method: http
      schema: '<schema>'
      host: '<http-host>'
      http_path: '<http-path>'
      token: "{{ env_var('DBT_ACCESS_TOKEN') }}"
```

The host and path values can reference a SQL warehouse or an all-purpose compute cluster. The credentials are generated automatically for the job run duration (up to 30 days) and are revoked after completion. In the job UI, set **SQL warehouse** to **None (Manual)** and specify the **Profiles Directory** (relative path to the folder containing `profiles.yml`). ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Using dbt Python Models

If your dbt project includes Python models (beta, requires dbt ≥1.3), you cannot run them against a SQL warehouse. Instead, you must use a custom profile that targets all-purpose compute. The dbt task can still be part of a larger workflow. ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Errors and Troubleshooting

### “Profile file does not exist” Error

**Error message**: `dbt looked for a profiles.yml file in /tmp/.../profiles.yml but did not find one.`

**Cause**: The `profiles.yml` file is missing from the specified directory or from the repository root. Ensure the file is present in the location indicated by the **Profiles Directory** setting (default is the repository root). ^[use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md]

## Related Concepts

- dbt Core – The open-source transformation tool that powers the task.
- [Databricks Jobs](/concepts/dbt-task-in-databricks-jobs.md) – The workflow orchestration platform.
- Databricks SQL Warehouse – The compute engine that runs the dbt-generated SQL.
- [Databricks Git folders](/concepts/databricks-git-folders-for-cicd.md) – Required storage for dbt project files.
- Jobs API – Programmatic interface for job management.
- [dbt-databricks](/concepts/dbt-databricks-adapter-package.md) – The recommended adapter package for Databricks.

## Sources

- use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md

# Citations

1. [use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws.md](/references/use-dbt-transformations-in-lakeflow-jobs-databricks-on-aws-9420061c.md)
