bringyou opened a new issue, #70737:
URL: https://github.com/apache/doris/issues/70737
### Note before reading
Doris already exposes a per-table switch for this — `auto_analyze_policy`
(`enable` / `disable` / `base_on_catalog`), added in **#39532** and available
since 2.1.9 / 3.0.3. This request is **not** asking for a new server-side
capability; it is asking the adapter to use the existing one on relations it is
about to drop.
Related: #65993 (dbt-doris tracking issue), #68896 (failed auto-analyze jobs
restart every cycle).
### Environment
- dbt-core: 1.12.0
- dbt-doris: 1.0.0 (`extension/dbt-doris`)
- Apache Doris: 3.1.4
- Materializations in use: `table`, `incremental` (`append` and
`insert_overwrite`)
### Summary
Every `table` / `incremental` run creates a short-lived physical table
(`<model>__dbt_tmp`, `<model>__dbt_backup`) via CTAS and drops it seconds
later. Doris's auto-analyze daemon still holds that relation in its pending job
queue, so when it picks the job up the table is already gone and the FE logs a
full stack trace per relation, per run:
```text
WARN [StatisticsAutoCollector.collect():103] Failed to analyze table
ads_sss_series_10m__dbt_tmp
with columns [ads_sss_series_10m__dbt_tmp:value,
ads_sss_series_10m__dbt_tmp:metric, ...]
java.lang.RuntimeException: Table:
`internal.ds_zimmula_prod.ads_sss_series_10m__dbt_tmp` not exists
at
org.apache.doris.statistics.util.StatisticsUtil.findTable(StatisticsUtil.java:421)
at
org.apache.doris.statistics.StatisticsAutoCollector.collect(StatisticsAutoCollector.java:103)
at
org.apache.doris.statistics.StatisticsAutoCollector.runAfterCatalogReady(StatisticsAutoCollector.java:82)
```
The exception is caught per job inside `collect()` and the loop continues,
so nothing breaks — it is pure log noise. But a 40-model project running hourly
produces on the order of 1,000 stack traces per day, which drowns `fe.log`.
### Steps to reproduce
1. `dbt run --select some_incremental_model` — any run that takes the
`create_table_as(True, tmp_relation, sql)` path (`incremental.sql`, the
`append` and `insert_overwrite` branches), or the `table` materialization's
intermediate relation (`table.sql`).
2. Wait one auto-analyze cycle (`auto_check_statistics_in_minutes`, default
5 minutes).
3. `grep "Failed to analyze table .*__dbt_tmp" fe.log`.
### Root cause
The adapter creates the temp/intermediate relation as a **real, registered
table** in the target database:
```jinja
{% macro doris__create_table_as(temporary, relation, sql) -%}
...
create table {{ table }}
...
{{ doris__properties() }} as {{ doris__table_colume_type(sql) }};
{%- endmacro %}
```
Doris registers the new table for automatic statistics collection; dbt drops
it before the collector reaches it, and `StatisticsUtil.findTable()` throws.
The relation lives for a few seconds and is never queried, so collecting
statistics on it is meaningless in the first place.
### Proposed change
Apply the existing `auto_analyze_policy = disable` property to short-lived
relations:
```sql
create table db.tbl ( ... )
properties ("auto_analyze_policy" = "disable")
as select ...
```
Scope — relations created and dropped within the same run:
| call site | relation suffix | dropped after |
|---|---|---|
| `incremental.sql`, `create_table_as(True, tmp_relation, sql)` |
`__dbt_tmp` | insert finishes |
| `incremental.sql`, full-refresh / backup path | `__dbt_backup` | exchange
finishes |
| `table.sql`, intermediate relation | `__dbt_tmp` | exchange finishes |
Suggested implementation: a boolean project var, default `true`, read inside
`doris__create_table_as` and applied only to the disposable relations.
`doris__properties()` currently takes no arguments, so it would need an
optional `extra` dict:
```jinja
{% macro doris__properties(extra=none) -%}
{% set properties = config.get('properties',
validator=validation.any[dict]) %}
...
{% if extra %}{% do properties.update(extra) %}{% endif %}
{%- endmacro %}
```
### Caveat the maintainers should decide on
In the `table` materialization the intermediate relation **becomes** the
final table via `exchange_relation`, so any property set on it is inherited by
the model's real table — which would silently disable statistics collection for
every model in the project. Two ways out:
1. Set the property on the disposable relation and explicitly restore it
after the exchange (`alter table <target> set ("auto_analyze_policy" =
"enable")`).
2. Only apply it in the `incremental` path (`__dbt_tmp` / `__dbt_backup`)
and leave the `table` materialization alone.
Option 2 is smaller and fixes the bulk of the noise; option 1 is complete
but touches the exchange path. I'm fine with either.
### Alternative worth checking
If `enable_auto_analyze` is settable at **session** scope (not just `SET
GLOBAL`), the adapter could set it on the connection that runs the CTAS and
avoid DDL-level properties entirely. I have not verified that the enqueue path
honours the session value on 3.1 — please confirm against the FE source before
relying on it.
### Acceptance criteria
- `show create table <model>__dbt_tmp` (while it still exists) contains
`"auto_analyze_policy" = "disable"`.
- After a full `dbt run`, `fe.log` contains no `Failed to analyze table
.*__dbt_tmp`.
- With the var set to `false`, behaviour is unchanged.
- Model-level `properties` config still takes precedence on the final table.
### Workaround in the meantime
Server-side, either disable auto-analyze globally (`set global
enable_auto_analyze = false`, requires ADMIN) or drop the log line at
collection time (`Failed to analyze table .*__dbt_tmp`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]