yujun777 opened a new pull request, #68237:
URL: https://github.com/apache/doris/pull/68237
### What problem does this PR solve?
Since the refresh baseline check was added (#67267, picking #64041),
`MTMVTask#calculateNeedRefreshPartitions()` returns **every** MV partition as
soon as the refresh snapshot does not cover all current MV partitions.
Partition sync runs before that check and adds an MV partition for a base
partition that just appeared. The new partition legitimately has no snapshot
yet, so the check fails on it, and adding a single base-table partition turns
the refresh into a full rebuild of the MV. The partitions that were already in
sync are not skipped at all: the per-partition comparison sits below the early
return and never runs.
Observed on a pct materialized view: the base table gains one partition, and
`NeedRefreshPartitions` then covers every partition with
`RefreshMode=COMPLETE`, instead of only the new partition with `PARTIAL`. Every
MV whose base table gains partitions routinely is hit on each such refresh
(daily partitions, external catalog tables).
### What changed
- `MTMV#hasRefreshSnapshot()` (renamed from `hasCompleteRefreshSnapshot()`,
a name that no longer described it) reports whether a refresh baseline exists
at all - the snapshot map is not empty - and tolerates a null snapshot.
- `MTMVTask#calculateNeedRefreshPartitions()` keeps the early return only
for a baseline invalidated as a whole, which is what `ALTER ...
excluded_trigger_tables` leaves behind (it empties the snapshot map), so that
case still rebuilds. A partition that partition sync has just added is decided
by the per-partition comparison instead.
### Test
Unit tests:
- `MTMVTest.testHasRefreshSnapshotIgnoresPartitionsWithoutSnapshot`: a
snapshot for `p1` with partitions `{p1, p2}` counts as a baseline; an emptied
map and a null snapshot do not.
-
`MTMVTaskTest.testCalculateNeedRefreshPartitionsKeepsSyncedPartitionsWhenOneHasNoSnapshot`:
`REFRESH MATERIALIZED VIEW ... AUTO` with an existing baseline keeps the
per-partition path and plans only the unsynchronized partition.
Regression test `test_base_table_add_partition_mtmv`: an MV with two synced
partitions, then the base table adds a third one. Asserts
`NeedRefreshPartitions=["p_3"]`, `RefreshMode=PARTIAL`, that the new partition
carries its rows, and that a following refresh reports `NOT_REFRESH`.
`test_excluded_trigger_table_mtmv`, `test_multi_level_mtmv` and
`test_partition_refresh_mtmv` pass as well.
### Related PR
- apache/doris#67267 - the 4.1 pick of apache/doris#64041, which introduced
the refresh baseline check narrowed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
