yujun777 commented on PR #69168:
URL: https://github.com/apache/doris/pull/69168#issuecomment-6207626351
Thanks! Answering the five points for this PR.
**1. What problem was fixed.**
An IVM materialized view was maintained on exactly one table model -- UNIQUE
KEY + merge-on-write, with a
hidden row-id, a delete sign, a sequence column and a plan signature --
whatever its query was. A query
that is append-only needs none of that: every base table is a DUP detail
table whose row binlog carries
APPEND events only (the binlog op is derived from a row's delete-sign
column, which a DUP table does not
have, and a table with row binlog refuses row-level predicate DELETE), and
the query holds no aggregate
and no outer join, so nothing can retract a row the MV already holds. Such
an MV holds the query's own
rows and its refresh only appends. Building it on the merge-on-write model
paid for maintenance it could
never need.
**2. Which behaviors were modified, and what changed.**
- Table model: an IVM MV over such a query is now DUP instead of UNIQUE KEY
+ merge-on-write. `DESC`
reports DUP_KEYS, `SHOW CREATE` prints DUPLICATE KEY, and the MV chooses
its own sort key and
distribution instead of being pinned to the hidden row-id.
- Refresh: the delta is the query rewritten to read each base table's
stream, with no dml factor, no
sequence and no row-id bookkeeping. `REFRESH ... INCREMENTAL WITH DRY RUN`
returns the MV's own columns
rather than those columns plus a sequence column and a delete sign.
- A base table that stops being an append-only detail table (replaced with a
merge-on-write one, say)
fails the refresh as an `AnalysisException` rather than being retried as a
fallback, and the MV is left
marked as needing a rebuild -- the previous behavior would have been a
fallback whose complete refresh
normalizes the same way and fails the same way.
- Impact: an IVM MV over an append-only query can no longer be upserted into
or deleted from row by row
through merge-on-write. A user who wants that model anyway sets
`ivm_force_mow = true` at creation.
- A query with an aggregate or an outer join is built on the unique-key
model exactly as before.
**3. What features were added.**
- The append-only detail model itself: which model an MV is built with is
settled once, while it is
created (from the query, its base tables and the MV's creation-time
properties), and read back from the
MV's own key model afterwards.
- `ivm_force_mow`: asks for the unique-key model on a query that would
qualify for the detail one. Fixed
at creation, like `ivm_use_full_keys`, and reported as unalterable by
`ALTER`.
- Chained MVs over a detail MV, both to another detail MV and to a
unique-key one.
**4. Which code was refactored, and why.**
- The normalize rule is split by the model it builds: `IvmNormalizeMTMV`
settles the model and dispatches,
`IvmNormalizeUniqKeysMTMV` lays out the unique-key model (unchanged
behavior) and
`IvmNormalizeDupKeysMTMV` checks that the query is one the detail model
can maintain.
- The incremental rewrite is split the same way: the MOW path
(`IvmDeltaRewriter` and the delta handlers)
is unchanged, and the detail path (`IvmDupDeltaRewriter`) shares the
pieces both need.
- Publishing a normalized plan happens in one place, and the
reserved-output-column check is shared by
both normalizers -- the detail model's MV carries hidden columns of its
own, so a user column named
after one has to be rejected there too (it was accepted before, and the
column then disappeared from
the schema the user reads).
**5. Which functions were optimized.**
For a query the detail model applies to, three things improve, all at the
same refresh cost model:
- Write: a refresh appends rows instead of upserting them -- no
merge-on-write deduplication or
delete-bitmap probing on the write path, no per-row row-id computation
during the refresh, and no
sequence column or delete sign to carry through the delta.
- Space: neither the MV's IVM hidden columns (row-id, dml factor, agg count,
aggregation state) nor the
merge-on-write hidden columns (delete sign, version, skip bitmap,
primary-key index) exist on it.
- Robustness: the unique-key model pins the layout its refresh reads by a
plan signature, so a plan that
stops matching it after an upgrade triggers `PLAN_SIGNATURE_MISMATCH` and
a full refresh. A detail MV
has no signature and no layout for a refresh to read.
No benchmark numbers are included here; the improvement is structural
(append instead of upsert) and the
cost model is unchanged. Tests: the full IVM FE unit-test set (582 cases)
and the whole `mtmv_p0/ivm`
regression directory (108 suites), both green, plus manual checks on a
cluster for the model, the
property's ALTER rejection and the changed dry-run shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
