dbt source freshness: configure checks and handle stale data

A dashboard can refresh successfully while showing yesterday's source data. dbt source freshness helps detect that situation by checking how recent the source data is against thresholds you define. It gives your workflow a signal to investigate before it publishes another apparently current report.

Freshness checks and model rebuilds answer different questions. This guide covers late source data. If the source is available but a model needs to be rebuilt, use the dbt full-refresh guide.

Configure a source freshness check

This illustrative configuration uses the config layout documented for dbt Core 1.10 and later. Replace the schema, table, and timestamp field with your own source definitions:

version: 2
sources:
  - name: commerce
    schema: raw
    config:
      loaded_at_field: ingested_at
      freshness:
        warn_after: {count: 2, period: hour}
        error_after: {count: 4, period: hour}
    tables:
      - name: orders

warn_after marks data that needs attention; error_after makes the check fail when the data is too old. A table can override its source-level configuration. Older dbt versions place these properties differently. Use the freshness configuration reference for your version.

The two- and four-hour thresholds above are examples, not recommended defaults. Choose them from the delivery schedule and the consumer's tolerance for delay. A source loaded once each morning needs a different rule from an hourly feed.

Choose a timestamp that measures the delay you care about

An ingestion timestamp answers when records arrived in the warehouse. An event timestamp answers when the underlying event happened. Late-arriving events can make those two values very different.

Write down which delay you want to detect before choosing the field. Inspect its type, timezone, and how the loader populates it. A timestamp accidentally set in the future can make a freshness signal misleading.

Also keep completeness separate from recency. One newly loaded record does not establish that a whole batch arrived. Pair freshness with the row-count, key, and business checks needed for your dataset.

Run a targeted check

Run this in your configured dbt project:

dbt source freshness --select source:commerce.orders

Without selection, the command checks configured sources more broadly. dbt writes freshness results to target/sources.json; consult the artifact produced by your installed version when consuming it. The source command documentation covers selection and output.

Keep the result associated with its execution and configuration. If several jobs share a working directory, a later invocation can replace the artifact you intended to inspect. Store a run-specific copy before the next check.

Decide how the pipeline handles each result

The check supplies evidence. Your orchestration needs an explicit response to it:

Avoid a shell script that runs a check, ignores its exit status, and publishes anyway. Inspect both the command outcome and the expected source results. A missing artifact should not be interpreted as a passing result.

For a report that can tolerate a known delay, define the exception in terms of the consumer: which report, what data cutoff, and how readers will be informed. An exception for one morning's report should not silently authorize future stale releases.

Troubleshoot a result that looks wrong

First confirm that the intended source was selected and has a freshness configuration. Then inspect the underlying timestamp values and compare them with the warehouse clock.

If checks are consistently early, investigate whether the threshold ignores the loader's normal schedule. If checks pass while users report missing data, look for partial batches or a mismatch between ingestion time and event time. Increasing the threshold will not repair those issues.

A full refresh is not the default response to late data. Rebuilding a downstream table cannot restore records that have not reached its source. Recover the upstream delivery, recheck freshness, and then decide whether historical processing is needed. The Airflow backfill guide covers that separate operation.

Collect a decision for a bounded exception

Most freshness outcomes should follow an agreed policy. Add human review when someone must decide whether a specific downstream use is acceptable despite the delay.

An ActionBox request can present the affected source, last observed load, threshold, report destination, and proposed exception. ActionBox supplies the human decision record; dbt performs the measurement and your orchestration controls publication. Follow the integration architecture and callback verification guide when connecting that decision to a job.

Keep the failed check in the record even if the exception is approved. Approval explains why the team proceeded; it does not turn stale data into fresh data.

Before rollout, try a fresh source, an old timestamp, a query failure, and a missing expected result in a development project. Confirm that only the intended path reaches the downstream job.

Create a free Source · Plan the review step