yujun777 opened a new pull request, #69168:
URL: https://github.com/apache/doris/pull/69168
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
An IVM materialized view was maintained on exactly one table model: UNIQUE
KEY + merge-on-write, with a
hidden row-id, a delete sign, a sequence column and a plan signature,
whatever its query was. A query
that is append-only needs none of that, and such an MV is built as a plain
DUP table here.
**When the DUP (append-only detail) model applies.** The model is settled
once, while the MV is created,
from the query, its base tables and the MV's creation-time properties:
- every base table is an append-only detail table: a DUP table whose row
binlog is enabled. Such a table
produces APPEND events only -- the binlog op is derived from a row's
delete-sign column, which a DUP
table does not have -- and a table with row binlog refuses row-level
predicate DELETE, so nothing can
retract a row the MV already holds;
- the query holds no aggregate (DISTINCT and GROUPING SETS included) and no
outer join. Of the operators
the IVM normalize supports, exactly these two can correct a row already
materialized, which only the
unique-key model's row-id can carry. Everything else the normalize
supports is multiset-linear: scan,
project, filter, subquery alias, inner join, cross join and UNION ALL;
- `ivm_force_mow = true` is not set: a user can ask for the unique-key model
anyway, and it is fixed at
creation like `ivm_use_full_keys`.
**Why it is better than the merge-on-write model, for such a query:**
1. Better write performance. The MV is a DUP table, so a refresh appends
rows instead of upserting them:
no merge-on-write deduplication or delete-bitmap probing on the write
path, no per-row row-id
computation during the refresh, and no sequence column or delete sign to
carry through the delta.
2. Less space. Neither the MV's IVM hidden columns (row-id, dml factor, agg
count and aggregation state
columns) nor the merge-on-write hidden columns (delete sign, version,
skip bitmap, primary-key index)
exist on it, and the rows themselves carry no retraction bookkeeping.
3. No plan signature, so the layout the refresh reads is not pinned down.
The unique-key model pins its
layout by a signature: a plan that stops matching it -- after an upgrade
changes the normalize or
rewrite rules, say -- fails the refresh with `PLAN_SIGNATURE_MISMATCH`
and falls back to a full
refresh. A detail MV has no signature to mismatch, and its refresh reads
no layout.
What changes for such an MV, before -> after:
- Table model: UNIQUE KEY + MOW -> DUP. `DESC` reports DUP_KEYS, `SHOW
CREATE` prints DUPLICATE KEY, and
the MV chooses its own sort key and distribution instead of being pinned
to the hidden row-id.
- Refresh: the delta is the query rewritten to read each base table's
stream, with no dml factor, no
sequence and no row-id bookkeeping; `REFRESH ... INCREMENTAL WITH DRY RUN`
returns the MV's own columns
rather than columns plus a sequence column and a delete sign.
- A base table that stops being an append-only detail table (replaced with a
merge-on-write one, say)
fails the refresh as an `AnalysisException` rather than being retried as a
fallback, and the MV is left
marked as needing a rebuild.
### Release note
None (IVM is not released yet).
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- Behavior changed:
- [x] Yes. An IVM MV over an append-only query with no aggregate and no
outer join is now created as
a plain DUP table rather than UNIQUE KEY + merge-on-write, and its
refresh appends rather than
upserts.
- Does this need documentation?
- [x] No. IVM is not released yet; its user documentation is written
separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
