Our dbt runs are taking forever. How can I optimize our dbt project for speed?

Absolutely. The biggest dbt speedups usually come from building less data, scanning less data, and running fewer models—not from tweaking dbt itself.

1. Find the actual bottleneck first

Run your project and identify:

• The slowest models by execution time.
• Models that scan huge tables.
• Models that are rebuilt unnecessarily.
• Tests that take longer than the models themselves.
• Whether the bottleneck is dbt orchestration/compilation or your warehouse SQL execution.

dbt's docs emphasize state-aware approaches that can skip nodes whose code/data hasn't meaningfully changed.

A useful first pass is:

dbt run --select +slow_model+

Then inspect the generated SQL and your warehouse's query profile for that model.

2. Convert large fact models to incremental

If you're doing this:

{{ config(materialized='table') }}

select *
from {{ ref('events') }}

every run potentially rebuilds the entire table.

For a large, append-heavy fact table, consider:

{{ config(
    materialized='incremental',
    unique_key='event_id'
) }}

select
    event_id,
    user_id,
    event_timestamp,
    event_type
from {{ ref('events') }}

{% if is_incremental() %}
where event_timestamp >= (
    select max(event_timestamp)
    from {{ this }}
)
{% endif %}

The important part isn't merely setting materialized='incremental': your SQL must actually restrict the incremental workload.

For mutable source data, use an appropriate lookback window, e.g. the last 1–3 days, rather than assuming records are perfectly append-only.

3. Filter before expensive joins

This pattern can be disastrous:

select ...
from huge_events e
join huge_users u
  on e.user_id = u.user_id
where e.event_date >= '2026-08-01'

Prefer pushing the filter into a CTE/subquery:

with recent_events as (

    select ...
    from {{ ref('events') }}
    where event_date >= '2026-08-01'

)

select ...
from recent_events e
join {{ ref('users') }} u
  on e.user_id = u.user_id

Even better, make the date boundary dynamic for incremental models.

Also avoid select * in large transformations. Reading 50 columns when you need 8 can materially increase I/O and downstream processing.

4. Stop rebuilding unchanged models

If your CI currently does:

dbt build

on every PR, you're potentially doing far more work than necessary.

Use state-aware selection where your CI/deployment setup supports it, so you can build changed nodes and their relevant downstream dependencies rather than the entire project. dbt's current documentation describes dbt State specifically as a mechanism for reusing nodes when logic and data haven't changed.

A common CI strategy is conceptually:

dbt build --select state:modified+

with an appropriate previous-state manifest.

5. Reduce unnecessary model layers

A project can become slow because it has chains like:

raw
 ↓
staging_1
 ↓
staging_2
 ↓
intermediate_1
 ↓
intermediate_2
 ↓
fact

where each layer creates a physical table.

Ask of each model:

• Does this need to be persisted?
• Could it be a view?
• Could two transformations be safely combined?
• Is this intermediate table reused enough to justify its materialization?
• Is a downstream model forcing a huge intermediate result to be created?

Don't blindly turn everything into ephemeral models, though—repeated expensive transformations can simply move the cost downstream.

6. Optimize joins aggressively

Look especially for:

• Many-to-many joins.
• Joining two enormous tables before filtering either.
• Accidental Cartesian products.
• Joining on expressions such as date(timestamp) instead of appropriately structured keys.
• Repeated joins to the same enormous dimension.
• Deduplication happening after a huge join rather than before it.

For example, if a dimension isn't actually unique on user_id, this:

join users u
  on e.user_id = u.user_id

may multiply your fact rows dramatically.

7. Use warehouse-specific physical design

This is where your warehouse matters enormously.

Depending on whether you're on Snowflake, BigQuery, Databricks, Redshift, Postgres, etc., the best tactics differ:

• Partition large tables around common filtering columns.
• Cluster/sort where your warehouse benefits from it.
• Avoid repeatedly scanning columns you don't need.
• Take advantage of warehouse-native incremental/merge optimizations.
• Right-size compute for genuinely parallel workloads.
• Check query plans rather than assuming dbt is the culprit.

In many cases, a dbt model that takes 20 minutes is really a 20-minute warehouse query.

8. Don't let tests dominate runtime

A surprisingly common problem is a giant test suite repeatedly scanning massive tables.

For example, generic uniqueness tests against a 5-billion-row table can be expensive.

Consider:

• Testing the appropriate grain rather than every column.
• Testing recent partitions for particularly expensive checks where acceptable.
• Moving some expensive data-quality checks to dedicated jobs.
• Avoiding redundant tests at multiple layers.
• Ensuring relationships tests aren't scanning vastly more data than necessary.

9. Parallelize independent work

dbt can execute independent DAG branches concurrently.

If your project is effectively:

A → B → C → D → E

there isn't much parallelism.

But:

       → B →
A →             → E
       → C →
       → D →

can use substantially more warehouse concurrency.

So look at your DAG for artificial dependencies. Sometimes a model references another model simply because it was convenient, creating serialization that isn't logically necessary.

10. A practical optimization order

I'd tackle it in this order:

1. Identify the 10 slowest models.
2. Determine whether they're warehouse execution or dbt overhead.
3. Convert appropriate full-refresh tables to incremental models.
4. Make incremental predicates selective and correct.
5. Reduce source rows/columns before joins.
6. Fix large joins and accidental row multiplication.
7. Optimize partitioning/clustering/sorting for your warehouse.
8. Remove unnecessary persisted intermediate models.
9. Make CI state-aware so unchanged models aren't rebuilt.
10. Profile expensive tests separately.
11. Increase concurrency only after reducing unnecessary work.

If you tell me which warehouse you're using (Snowflake, BigQuery, Databricks, Redshift, etc.), roughly how many models you have, and an example of one of the slow models, I can give you a much more concrete optimization plan—including the dbt config and SQL I'd change.

For example, if a dimension isn't actually unique on user_id, this:

join users u
  on e.user_id = u.user_id

may multiply your fact rows dramatically.

7. Use warehouse-specific physical design

This is where your warehouse matters enormously.

Depending on whether you're on Snowflake, BigQuery, Databricks, Redshift, Postgres, etc., the best tactics differ:

• Partition large tables around common filtering columns.
• Cluster/sort where your warehouse benefits from it.
• Avoid repeatedly scanning columns you don't need.
• Take advantage of warehouse-native incremental/merge optimizations.
• Right-size compute for genuinely parallel workloads.
• Check query plans rather than assuming dbt is the culprit.

In many cases, a dbt model that takes 20 minutes is really a 20-minute warehouse query.

8. Don't let tests dominate runtime

A surprisingly common problem is a giant test suite repeatedly scanning massive tables.

For example, generic uniqueness tests against a 5-billion-row table can be expensive.

Consider:

• Testing the appropriate grain rather than every column.
• Testing recent partitions for particularly expensive checks where acceptable.
• Moving some expensive data-quality checks to dedicated jobs.
• Avoiding redundant tests at multiple layers.
• Ensuring relationships tests aren't scanning vastly more data than necessary.

9. Parallelize independent work

dbt can execute independent DAG branches concurrently.

If your project is effectively:

A → B → C → D → E

there isn't much parallelism.

But:

       → B →
A →             → E
       → C →
       → D →

can use substantially more warehouse concurrency.

So look at your DAG for artificial dependencies. Sometimes a model references another model simply because it was convenient, creating serialization that isn't logically necessary.

10. A practical optimization order

I'd tackle it in this order:

1. Identify the 10 slowest models.
2. Determine whether they're warehouse execution or dbt overhead.
3. Convert appropriate full-refresh tables to incremental models.
4. Make incremental predicates selective and correct.
5. Reduce source rows/columns before joins.
6. Fix large joins and accidental row multiplication.
7. Optimize partitioning/clustering/sorting for your warehouse.
8. Remove unnecessary persisted intermediate models.
9. Make CI state-aware so unchanged models aren't rebuilt.
10. Profile expensive tests separately.
11. Increase concurrency only after reducing unnecessary work.

If you tell me which warehouse you're using (Snowflake, BigQuery, Databricks, Redshift, etc.), roughly how many models you have, and an example of one of the slow models, I can give you a much more concrete optimization plan—including the dbt config and SQL I'd change.

When dbt runs start dragging, the bottleneck is usually a mix of inefficient materializations, database warehouse configurations, and unoptimized SQL. You can tackle this systematically across three key pillars: dbt configuration, materialization strategy , and warehouse/SQL tuning.

1. Tune dbt Project Configuration

• Increase Concurrency (threads): Check your profiles.yml . By default, threads might be set low (e.g., 1 or 4). Increase the number of threads (often 4 to 8+ depending on your data warehouse capacity) so dbt can run independent models in parallel.
• Prune Unused Models: Audit your DAG to find "orphan" models that no longer feed downstream dashboards or reports. Deleting dead code shrinks the run graph entirely.
• Leverage State-Aware Orchestration: If you use CI/CD or continuous runs, make use of --select state:modified+ combined with --defer . This ensures you only rebuild models that have actually changed or their direct downstream dependents, skipping redundant work.

2. Optimize Materialization Strategies

• Upgrade to Incremental Models: If large event, log, or append-only tables are running as full table materializations, switch them to incremental . This ensures dbt only processes new or updated rows rather than rewriting the entire historical table every run.
• Use the Microbatch Strategy: If you are on newer versions of dbt Core (v1.9+), look into the dbt Microbatch Strategy for time-series data. It breaks large incremental runs into manageable chunks (e.g., daily or hourly batches) without requiring complex custom is_incremental() boilerplate.
• Default to Views for Lightweight Transformations: Don't materialize everything as a table. Use view for intermediate, staging, or lightweight transformations that are queried infrequently or only pass data to a single downstream model.

3. Refactor SQL and Warehouse Settings

• **Avoid SELECT *** and unneeded JOIN s: Strip out heavy SELECT * statements, unnecessary subselects, and redundant distinct/joins that force the data warehouse to shuffle massive amounts of intermediate data.
• Implement Partitioning and Clustering: Ensure heavy tables are partitioned or clustered on commonly filtered columns (like date or id ) in your data warehouse (Snowflake, BigQuery, Redshift) so dbt scans only the data it needs.
• Right-Size Your Warehouse: Match your dbt target warehouse size to the workload. Sometimes scaling up a warehouse temporarily for a heavy run (and tuning auto-suspend times down so it doesn't idle and run up costs) dramatically cuts wall-clock time.

If you want to dive deeper, let me know:

• What data warehouse are you running on (Snowflake, BigQuery, Databricks, Postgres, etc.)?
• What is your current thread count in profiles.yml?

I can give you specific warehouse-level parameters and profile configurations to speed things up.

Slow dbt runs usually come from one (or more) of these areas:

1. Too much data being reprocessed
2. A DAG that has too much serial dependency
3. Warehouse queries that are inefficient
4. Too many models/tests running unnecessarily
5. Poor incremental model design

Here’s a practical tuning approach.

1. Find the actual bottlenecks first

Start by identifying your slowest models:

dbt run --select state:modified+ --defer --state path/to/previous/run

and inspect:

• Longest-running models
• Models blocking many downstream models
• Models that scan huge source tables
• Tests that consume most runtime

dbt’s run artifacts (run_results.json, logs) are useful for timing analysis.

A good first question:

“Which 5 models account for 80% of runtime?” Optimize those first.

---

2. Convert expensive tables to incremental models

If a model rebuilds billions of rows every run, incremental materialization is usually the biggest win.

Example:

{{ config(
    materialized='incremental',
    unique_key='order_id'
) }}

select *
from {{ ref('stg_orders') }}

{% if is_incremental() %}

where updated_at >= (
    select max(updated_at)
    from {{ this }}
)

{% endif %}

Instead of:

process all historical data
        ↓
rebuild table

you get:

process new/changed records
        ↓
merge into existing table

Incremental models are designed specifically to avoid full reprocessing.

For very large time-series datasets, newer dbt versions also support microbatch incremental strategies, which split processing into smaller time windows.

---

3. Fix incremental model anti-patterns

A common mistake:

{% if is_incremental() %}
where created_at > current_date - 7
{% endif %}

but then joining against a massive table:

select *
from events e
join customers c
  on e.customer_id = c.customer_id

The warehouse may still scan huge amounts of data.

Better:

• Filter early
• Reduce rows before joins
• Select only needed columns
• Avoid unnecessary distinct
• Avoid repeated expensive CTEs

Example:

with recent_events as (

    select
        event_id,
        customer_id,
        event_time

    from {{ ref('events') }}

    {% if is_incremental() %}
    where event_time >= dateadd(day, -3, current_date)
    {% endif %}

)

select *
from recent_events
join {{ ref('customers') }}
using (customer_id)

---

4. Increase dbt parallelism carefully

dbt can run independent models concurrently using threads.

Example:

# profiles.yml

target: prod

outputs:
  prod:
    type: snowflake
    threads: 8

But more threads are not always faster.

Too many threads can:

• overload your warehouse
• cause queueing
• increase contention

Tune based on warehouse capacity.

A useful test:

threads: 4
runtime: 55 min

threads: 8
runtime: 32 min

threads: 16
runtime: 38 min

Your sweet spot is probably around 8.

---

5. Reduce unnecessary DAG depth

A common slow pattern:

source
  ↓
staging
  ↓
intermediate_1
  ↓
intermediate_2
  ↓
intermediate_3
  ↓
fact_table

Every layer adds execution time.

Sometimes this is justified, but avoid creating models that only:

• rename columns
• reorder columns
• add trivial transformations

A flatter DAG often runs faster.

---

6. Use selectors instead of running everything

Instead of:

dbt run

use targeted execution:

dbt run --select tag:daily

or:

dbt run --select model_name+

Examples:

• CI:

dbt build --select state:modified+

• Daily jobs:

dbt build --select tag:daily

• Hourly jobs:

dbt build --select tag:hourly

---

7. Optimize your warehouse layer

Depending on your platform:

Snowflake

Look at:

• warehouse size
• clustering keys
• automatic clustering
• query profile

BigQuery

Focus on:

• partitioning
• clustering
• avoiding full table scans

Databricks

Focus on:

• Delta optimization
• partition strategy
• file compaction

Example:

{{ config(
    materialized='table',
    partition_by={
      "field": "event_date",
      "data_type": "date"
    }
) }}

---

8. Speed up tests

Tests can become a hidden bottleneck.

Common expensive tests:

- relationships
- unique
- not_null

on billion-row tables.

Options:

• Run full tests nightly
• Run lighter tests in CI
• Add filters where supported
• Test representative samples for huge tables

---

9. Remove unused models from production runs

Large projects often accumulate:

• deprecated models
• experimental models
• old snapshots
• unused marts

Use:

dbt ls

to audit what actually runs.

---

10. A good optimization order

I’d usually tackle dbt performance in this order:

1. Find top 5 slowest models
2. Convert biggest tables to incremental
3. Fix expensive joins/scans
4. Add partitioning/clustering
5. Increase threads
6. Remove unnecessary DAG layers
7. Optimize tests
8. Clean unused models

If you share your warehouse (Snowflake/BigQuery/Redshift/Databricks), approximate model count, and current runtime (for example “500 models, 3 hours”), I can suggest more specific optimizations.

{{ config(
    materialized='table',
    partition_by={
      "field": "event_date",
      "data_type": "date"
    }
) }}

---

8. Speed up tests

Tests can become a hidden bottleneck.

Common expensive tests:

- relationships
- unique
- not_null

on billion-row tables.

Options:

• Run full tests nightly
• Run lighter tests in CI
• Add filters where supported
• Test representative samples for huge tables

---

9. Remove unused models from production runs

Large projects often accumulate:

• deprecated models
• experimental models
• old snapshots
• unused marts

Use:

dbt ls

to audit what actually runs.

---

10. A good optimization order

I’d usually tackle dbt performance in this order:

1. Find top 5 slowest models
2. Convert biggest tables to incremental
3. Fix expensive joins/scans
4. Add partitioning/clustering
5. Increase threads
6. Remove unnecessary DAG layers
7. Optimize tests
8. Clean unused models

If you share your warehouse (Snowflake/BigQuery/Redshift/Databricks), approximate model count, and current runtime (for example “500 models, 3 hours”), I can suggest more specific optimizations.

Optimizing a slow dbt project usually comes down to eliminating unnecessary data scanning, reducing compilation overhead, and leveraging data warehouse performance features properly.

Here is a structured, scannable playbook to speed up your dbt runs:

1. Reduce Data Scanned & Transformed

• Use Incremental Models: Instead of rebuilding entire large fact and event tables on every run, switch them to incremental strategies. This ensures you only process new or updated rows using is_incremental().
• Filter Early in CTEs: Apply where clauses and partition/cluster filters as early as possible in your staging (stg_ ) models. Do not wait until the final select statement to filter massive datasets.
• Leverage ephemeral Materialization Sparingly: While ephemeral models keep your warehouse clean of unneeded physical views, chaining too many ephemeral CTEs can bloat the final compiled SQL query into an un-optimizable monolith for the query planner. Materialize as views or tables when complexity grows.

2. Optimize Project Configuration & Selection

• Target Specific Models in Development: Stop running the entire DAG during local development or quick testing. Use node selection syntax:
  • Run a model and its downstream dependents: dbt run --select +my_model
  • Run only modified models (in git): dbt run --select state:modified+ (make sure to configure --defer and --state against your production manifest).
• Tune Threads: Adjust the threads setting in your profiles.yml . Setting this closer to your warehouse's concurrency limits speeds up parallel execution, but setting it too high can cause queue contention or warehouse resource exhaustion.

3. Warehouse & Materialization Adjustments

• Add Incremental Predicates: For massive incremental models, use incremental_predicates in your config block to push partition pruning down into the merge/insert operations.
• Use Cluster / Partition Keys: Ensure your tables and incremental models are properly partitioned (e.g., by date) and clustered (e.g., by user_id) in your database configuration block (partition_by / cluster_by).
• Audit Heavy Macros and Hooks: Heavy pre-hook or post-hook operations or complex Jinja loops that run dynamic queries against information schemas during the parsing phase can heavily lag your startup time. Run dbt --debug to see where compilation hangs.

If you'd like to narrow this down, let me know:

• Which data warehouse are you using (Snowflake, BigQuery, Databricks, Redshift, etc.)?
• Are the bottlenecks happening during compilation/parsing or actual query execution in the warehouse?
• What percentage of your models are currently incremental versus table/view?

I can give you a more targeted optimization strategy!

When dbt runs start taking forever, the biggest wins usually come from finding where the time is going rather than blindly adding compute. A good tuning pass usually focuses on four areas: model materializations, SQL workload, DAG structure, and execution parallelism.

1. Find the slowest models first

Start with your run artifacts:

• Look at target/run_results.json
• Sort models by execution_time
• Identify: - slowest models
• models blocking many downstream models
• models that scan huge amounts of data repeatedly

A 30-minute model at the top of the DAG can hurt more than 20 smaller slow models because everything waits on it.

2. Convert expensive table rebuilds to incremental models

A common cause of slow dbt runs is rebuilding large tables every time:

{{ config(materialized='table') }}

select *
from raw_events

For large append-heavy datasets, switch to incremental:

{{ config(
    materialized='incremental',
    unique_key='event_id'
) }}

select *
from raw_events

{% if is_incremental() %}
where updated_at >= (
    select max(updated_at)
    from {{ this }}
)
{% endif %}

This avoids reprocessing historical data on every run. dbt incremental models are specifically designed to process only new or changed rows instead of rebuilding everything.

Common candidates:

• event tables
• transactions
• logs
• clickstream data
• daily snapshots

Be careful with:

• late-arriving data
• updates to old records
• incorrect unique_key choices

3. Optimize your incremental strategy

Incremental does not automatically mean fast.

Examples:

• Append-only data → append
• Mutable records → merge
• Partitioned time-series data → consider partition-aware strategies

For very large time-series tables, microbatch-style processing can reduce the amount processed per query by splitting work into time windows.

4. Reduce unnecessary model layers

A common pattern:

source
  ↓
staging view
  ↓
intermediate view
  ↓
another intermediate view
  ↓
fact table

Each view can force the warehouse to repeatedly execute upstream logic.

Consider:

• Keep lightweight staging models as views
• Make expensive reusable transformations tables/incrementals
• Use ephemeral models for tiny one-use transformations

Example:

{{ config(materialized='ephemeral') }}

select
    customer_id,
    lower(email) as email
from {{ ref('customers') }}

5. Increase parallelism carefully

dbt can execute independent models concurrently using threads.

Check your profiles.yml:

target: prod

outputs:
  prod:
    type: snowflake
    threads: 8

But more threads are not always faster.

Too many threads can:

• overload warehouse concurrency
• create slot contention
• slow every query down

Tune threads against your warehouse limits.

6. Fix DAG bottlenecks

Look for:

A
|
B
|
C
|
D
|
E

This runs mostly sequentially.

You want more shapes like:

        A
      / | \
     B  C  D
      \ | /
        E

Ways to improve:

• split huge monolithic models
• remove unnecessary dependencies
• avoid creating long chains of intermediate models
• make independent transformations independent

7. Optimize warehouse scans

Your SQL may be the real bottleneck.

Check:

• Are you doing select * everywhere?
• Are you joining billions of rows unnecessarily?
• Are filters applied before joins?
• Are partition columns being filtered?
• Are you scanning historical data repeatedly?

Example improvement:

Before:

select *
from events
join users using(user_id)

After:

select
    e.event_id,
    e.user_id,
    e.event_time,
    u.country
from (
    select *
    from events
    where event_date >= current_date - 7
) e
join users u using(user_id)

8. Use slim CI builds

If your CI pipeline runs the entire project for every PR, use state comparison:

dbt build --select state:modified+

This limits builds to changed models and their downstream dependencies.

9. Clean up unnecessary full refreshes

Audit your jobs:

Bad:

dbt run --full-refresh

every night.

Better:

• normal incremental runs daily
• scheduled full refreshes only when needed
• targeted rebuilds for affected models

10. Check your dbt version

Newer dbt versions include performance improvements in graph processing and incremental workflows. Keeping dbt Core and adapters current can help, especially for larger DAGs.

---

A practical tuning order I’d use:

1. Find top 10 slowest models.
2. Convert the biggest table rebuilds to incremental.
3. Fix the slowest SQL scans.
4. Increase threads until warehouse saturation.
5. Shorten the critical DAG path.
6. Add slim CI selection.

If you share:

• warehouse (Snowflake / BigQuery / Redshift / Databricks)
• current dbt runtime
• number of models
• slowest 3 model runtimes
• whether runs are full refreshes or incremental

I can suggest more targeted changes.

Before:

select *
from events
join users using(user_id)

After:

select
    e.event_id,
    e.user_id,
    e.event_time,
    u.country
from (
    select *
    from events
    where event_date >= current_date - 7
) e
join users u using(user_id)

8. Use slim CI builds

If your CI pipeline runs the entire project for every PR, use state comparison:

dbt build --select state:modified+

This limits builds to changed models and their downstream dependencies.

9. Clean up unnecessary full refreshes

Audit your jobs:

Bad:

dbt run --full-refresh

every night.

Better:

• normal incremental runs daily
• scheduled full refreshes only when needed
• targeted rebuilds for affected models

10. Check your dbt version

Newer dbt versions include performance improvements in graph processing and incremental workflows. Keeping dbt Core and adapters current can help, especially for larger DAGs.

---

A practical tuning order I’d use:

1. Find top 10 slowest models.
2. Convert the biggest table rebuilds to incremental.
3. Fix the slowest SQL scans.
4. Increase threads until warehouse saturation.
5. Shorten the critical DAG path.
6. Add slim CI selection.

If you share:

Optimizing a sluggish dbt project requires attacking bottlenecks at both the warehouse layer and the dbt configuration layer. Here is a structured approach to speeding up your runs:

• Materializations Strategy , Switch large, downstream, or infrequently updated models from view or table to incremental . This ensures you only process new or updated rows instead of full table scans on every run.
• Cluster and Partition , Implement partitioning and clustering (or sorting/distributing keys depending on your warehouse like Snowflake, BigQuery, or Redshift) on your large underlying tables to minimize data scanned during builds.
• Refactor CTEs and Avoid Select Star , Eliminate redundant Common Table Expressions (CTEs) that get materialized or processed repeatedly, and replace SELECT * with explicit column declarations to reduce memory overhead and data transfer.
• Leverage --select and State Comparison , Use --select state:modified+ via the --state flag in production CI/CD pipelines. This ensures you only build models that have actually changed or depend on changed upstream data, rather than rebuilding the entire DAG.
• Optimize Threads , Increase the number of threads in your profiles.yml file (matching your warehouse's concurrency limits) so independent models run in parallel rather than serially.
• Ephemerals Wisely , Use ephemeral materializations for small, single-use transformations that act as simple code clean-ups, but avoid them for complex or heavily-reused logic which can bloat the compiled SQL.
• Audit Heavy Macros and Hooks , Review pre- and post-hooks or heavy custom macros that execute expensive queries or information-schema lookups on every single model build.

If you want to dive deeper, let me know:

• Which data warehouse are you using (Snowflake, BigQuery, Databricks, Redshift, etc.)?
• Are the bottlenecks happening in local development or your production CI/CD pipeline?

Optimizing a dbt project for speed requires a mix of configuration tweaks, architectural adjustments, and database-level tuning. Here is a structured approach to speeding up your dbt runs:

• Use incremental models : Instead of rebuilding entire tables every run, configure heavy models as incremental using the is_incremental() macro so they only process new or updated rows.
• Leverage --select and state : Avoid running your entire DAG every time. Use state-based execution like dbt run --select state:modified+ to run only models that have changed and their downstream dependencies (compare against a previous manifest artifact).
• Refactor ephemeral and view strategies : Use materializations wisely. Heavy transformations shouldn't be ephemeral if they are referenced multiple times (which re-evaluates the CTE everywhere), and don't materialize everything as a table if a view or incremental model suffices.
• Optimize queries and joins : Audit your SQL code for slow operations. Avoid unnecessary SELECT * , reduce expensive cross-joins or heavy window functions on raw data, and pre-aggregate early in your DAG.
• Tune warehouse concurrency and cluster size : Check your data warehouse (e.g., Snowflake, BigQuery, Databricks). Scale up your warehouse size temporarily for heavy runs, or optimize cluster keys / partition strategies so queries scan less data.
• Disable docs generation on production runs : Running dbt docs generate adds overhead. Keep it out of your primary, high-frequency transformation pipelines unless explicitly needed.
• Use slim CI/CD : In your pull request pipelines, run only modified models (--select state:modified ) rather than a full refresh to drastically cut down CI/CD build times.

If you'd like to dive deeper, tell me:

• What data warehouse are you using (Snowflake, BigQuery, Databricks, Postgres, etc.)?
• Are your bottlenecks happening during full refreshes or incremental runs?

I can give you specific configuration snippets or warehouse-level tuning tips for your stack.

When dbt runs take forever, the biggest gains usually come from figuring out where the time is going first: model SQL execution, unnecessary model builds, warehouse contention, or dbt overhead. A good optimization pass usually looks like this:

1. Find the slowest models first

Start with run timing data:

dbt run

Look at the models with the longest execution times. In many projects, a handful of models account for most of the runtime. dbt recommends analyzing model timing and focusing optimization effort on the biggest bottlenecks rather than tuning everything.

Also inspect:

• target/run_results.json
• dbt Cloud run history (if applicable)
• warehouse query history (Snowflake Query Profile, BigQuery Job History, Databricks SQL history, etc.)

---

2. Convert expensive full-refresh tables to incremental models

The biggest win is often avoiding rebuilding large tables.

Before:

{{ config(materialized='table') }}

select *
from {{ source('app', 'events') }}

After:

{{ 
  config(
    materialized='incremental',
    unique_key='event_id'
  ) 
}}

select *
from {{ source('app', 'events') }}

{% if is_incremental() %}

where updated_at >= (
    select max(updated_at)
    from {{ this }}
)

{% endif %}

Incremental models process only new or changed data instead of recreating the entire dataset.

Common improvements:

• Use a reliable updated_at or event timestamp column
• Add a lookback window for late-arriving data
• Choose the right incremental strategy (merge, append, delete+insert, etc.)
• Avoid unnecessary full-refresh runs

---

3. Stop running the whole DAG during development

A common mistake is:

dbt run

for every change.

Instead:

dbt build --select my_model+

or:

dbt build --select state:modified+

State-based selection can avoid rebuilding unchanged parts of your project.

For CI pipelines, consider:

dbt build \
  --select state:modified+ \
  --defer \
  --state path/to/prod/artifacts

This lets unchanged models resolve against production artifacts rather than rebuilding everything.

---

4. Reduce expensive SQL patterns

Look for:

Huge unnecessary scans

Bad:

select *
from huge_table

Better:

select
    id,
    created_at,
    customer_id
from huge_table
where created_at >= '2026-01-01'

Late filtering

Bad:

select *
from (
    select *
    from events
    join users using(user_id)
)
where event_date >= current_date - 7

Better:

select *
from events e
join users u using(user_id)
where e.event_date >= current_date - 7

Push filters as close to the source as possible.

---

5. Review your materializations

Typical patterns:

Be careful with long chains of views. A warehouse may repeatedly expand and execute complex SQL when querying downstream models.

---

6. Tune warehouse-specific performance

Depending on your warehouse:

Snowflake

• Cluster large tables with selective filters
• Avoid repeated scans of massive tables
• Watch warehouse size vs concurrency

BigQuery

• Partition large tables
• Cluster frequently filtered columns
• Avoid select *
• Reduce unnecessary joins

Databricks

• Optimize Delta tables
• Use partitioning carefully
• Compact small files

---

7. Increase parallelism carefully

If your warehouse has capacity, increase dbt threads:

profiles.yml

target: prod

outputs:
  prod:
    type: snowflake
    threads: 8

More threads help only when:

• your DAG has parallelizable branches
• your warehouse can handle concurrency

More threads can make things slower if you overload the warehouse.

---

8. Enable faster parsing for large projects

If startup time is the issue (not SQL execution), enable partial parsing:

flags:
  partial_parse: true

Partial parsing avoids reparsing unchanged project files and can significantly reduce startup time in large projects.

---

9. Audit tests

Tests can quietly add hours.

Look for:

• relationships tests on huge tables
• duplicate scans of the same large datasets
• tests running unnecessarily in every environment

Consider:

• running full tests nightly
• running targeted tests in CI
• testing representative subsets in development

---

A practical optimization order

1. Find top 10 slowest models
2. Convert the biggest full-refresh tables to incremental
3. Remove unnecessary dbt run of the entire DAG
4. Optimize the slowest SQL queries in the warehouse
5. Tune materializations
6. Increase threads if warehouse capacity allows
7. Optimize tests and parsing

If you share:

• warehouse (Snowflake/BigQuery/Redshift/Databricks/etc.)
• current runtime
• number of models
• slowest 3 models and their runtimes
• whether runs are full-refresh or incremental

I can suggest a more targeted tuning plan.

Speeding up sluggish dbt (data build tool) runs requires a mix of configuration tweaks, architectural changes, and incremental processing. Here is a prioritized, scannable guide to optimizing your dbt project:

• Use Incremental Models : Instead of rebuilding entire tables every run, use config(materialized='incremental') . Filter for only new or updated records (e.g., using updated_at > (SELECT MAX(updated_at) FROM {this}})) on subsequent runs, and only do a full refresh when necessary via the --full-refresh flag.
• Leverage Ephemeral Materialization : For small, single-use transformations or CTEs that don't need to be queried downstream as physical database objects, use config(materialized='ephemeral') . This injects the code as a CTE directly into the downstream queries, saving storage and creation overhead.
• Optimize Queries and Indexes : Review your heavy SQL models for unnecessary SELECT * , redundant joins, or expensive window functions. On data warehouses like PostgreSQL, Snowflake, or BigQuery, ensure you are defining appropriate clustering keys, partition fields, or sorting keys in your model configs to minimize data scanning.
• Tune Threads : Increase the concurrency of your dbt run by bumping up the threads in your profiles.yml file or via the CLI flag -t 8 (or higher, depending on your data warehouse's concurrency limits and CPU capacity).
• Use Node Selection Safely : Avoid running the entire project every time if you are only working on a specific feature. Use dbt's powerful graph selection syntax—such as dbt run --select +my_model (upstream dependencies) or dbt run --select state:modified (using artifact comparison against a production manifest) to run only what changed.
• Refactor with Slim CI : Implement a "Slim CI" workflow in your CI/CD pipeline. By comparing your current branch's state against a production manifest.json using --select state:modified+ , you build only the models that were modified and their downstream children, drastically cutting down PR build times.
• Audit Heavy Tests : Heavy data tests (like unique/not_null combinations on massive tables) can slow things down. Consider using warn severities for non-critical checks, scoping tests with where clauses (e.g., testing only active records), or using custom generic tests that run more efficiently.

If you want to dive deeper, let me know:

• Which data warehouse are you using (Snowflake, BigQuery, Databricks, Postgres, etc.)?
• Are you currently using incremental models or a CI/CD tool (like GitHub Actions or GitLab CI)?

I can give you more targeted, warehouse-specific recommendations or help you set up a Slim CI workflow.

The biggest dbt speedups usually come from building less data, running fewer models, and making the warehouse do less work. I’d optimize in this order:

1. Find where the time actually goes

Start by separating dbt overhead from warehouse query time.

Look at:

• Slowest models by execution time
• Models scanning huge tables
• Models that rebuild every run despite rarely changing
• Long dependency chains that serialize work
• Tests that query enormous relations
• Seeds/snapshots or hooks taking significant time

If a model takes 20 minutes because the warehouse query takes 20 minutes, changing dbt configuration won't fix the underlying SQL.

2. Convert expensive full-refresh models to incremental

This is often the biggest win.

Instead of:

select *
from {{ ref('events') }}

use an incremental strategy that only processes new/changed records:

{{ config(
    materialized='incremental',
    unique_key='event_id'
) }}

select
    event_id,
    user_id,
    event_timestamp,
    event_type
from {{ ref('events') }}

{% if is_incremental() %}
where event_timestamp >= (
    select max(event_timestamp)
    from {{ this }}
)
{% endif %}

The important part isn't merely setting materialized='incremental': your SQL needs to avoid scanning/reprocessing the historical dataset.

Also consider your warehouse's best incremental strategy—e.g. merge, delete+insert, or insert-overwrite—and partition/cluster the underlying data appropriately.

3. Filter as early as possible

Avoid carrying billions of rows through intermediate CTEs only to filter them at the end.

Prefer:

with events as (
    select *
    from {{ ref('events') }}
    where event_date >= current_date - 30
),

users as (
    select *
    from {{ ref('users') }}
    where is_active = true
)

...

over joining massive unfiltered relations and applying predicates afterward.

Also avoid select * in production models when you only need a subset of columns.

4. Look for accidental repeated work

A common anti-pattern is having many downstream models independently perform the same expensive transformation.

For example:

raw_events
 ├── model_a → expensive deduplication
 ├── model_b → expensive deduplication
 └── model_c → expensive deduplication

Instead:

raw_events
      ↓
deduped_events
 ├── model_a
 ├── model_b
 └── model_c

Materializing an expensive shared transformation once can dramatically reduce total warehouse work.

The opposite can also happen: creating dozens of unnecessary intermediate tables can increase I/O. Benchmark rather than blindly materializing everything.

5. Exploit dbt's parallelism

dbt can execute independent DAG branches concurrently. If your project has:

A → B → C → D

there isn't much dbt can parallelize.

But:

        → B →
A →           → E
        → C →
        → D →

gives the runner more opportunity to execute work concurrently.

Look for unnecessary dependencies such as models referencing another model merely because it was convenient, rather than because the data dependency is real.

Also check your warehouse's concurrency limits—raising dbt's thread count isn't necessarily faster if it causes warehouse contention.

6. Use state-aware selection in CI

Don't rebuild the entire project for every pull request.

dbt's current documentation highlights dbt State as a way to skip unnecessary rebuilds by identifying nodes whose code/data haven't meaningfully changed.

A typical CI pattern is conceptually:

dbt build --select state:modified+

The + is important: it includes downstream dependencies of modified nodes.

For CI, combine this with a previous production manifest/state directory so a PR only builds the affected portion of the DAG.

7. Make development runs much smaller

Developers shouldn't routinely run the entire warehouse.

Useful patterns include:

dbt build --select my_model+

and limiting development data with a variable or environment-aware filter:

where event_date >= current_date - 7

You can also use --defer with a production manifest so unchanged upstream models can resolve to existing production relations instead of being rebuilt locally.

8. Optimize the SQL for your warehouse

This is often more important than dbt itself.

Check for:

• Large table scans
• Inefficient joins
• Exploding many-to-many joins
• Repeated window functions
• distinct over huge datasets
• Unnecessary order by
• Non-sargable predicates
• Excessive JSON/semi-structured-data parsing
• Poor partition pruning
• Poor clustering/sort/distribution choices

For example, this can be disastrous on a large table:

where cast(event_timestamp as date) = current_date - 1

whereas filtering directly on a partition-friendly timestamp/date column can allow the warehouse to prune most of the table.

9. Don't overlook tests

A surprisingly slow dbt project can actually be a testing problem.

Tests such as uniqueness and relationships can scan enormous tables. Consider:

• Testing appropriate subsets during development
• Avoiding redundant tests
• Ensuring tested columns are efficiently queryable
• Separating expensive comprehensive tests from every developer run

10. Reduce dbt parsing overhead

If your project has thousands of models/macros, local startup/parsing can become noticeable. dbt supports partial parsing, which avoids unnecessarily reparsing the entire project when only limited files have changed.

If parsing is a significant portion of your runtime, also inspect:

• Huge macro files
• Excessive generated YAML
• Large numbers of project files
• Environment-dependent parsing behavior
• Packages you don't actually use

dbt's current tooling also emphasizes faster parsing/development through its Fusion engine and VS Code integration.

---

What I'd do first

If you give me your warehouse (Snowflake, BigQuery, Databricks, Redshift, Postgres, etc.), roughly how many models you have, and a snippet of your slowest model, I can help you identify the highest-impact changes.

A good optimization exercise is:

The key principle: don't start by tweaking threads. First determine whether you're spending the time parsing dbt, waiting on the warehouse, rebuilding unnecessary data, or running an unnecessarily large DAG. Then optimize that bottleneck.