yujun777 commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4372085417
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RefreshMTMVCommand.java:
##########
@@ -194,12 +208,55 @@ protected LogicalPlan createRefreshCommand(MTMV mtmv,
StatementContext statement
statementContext.setExcludedTriggerTables(mtmv.getExcludedTriggerTables());
return createIvmIncrRefreshManager().buildInsertCommand(mtmv);
case COMPLETE:
+
statementContext.setExcludedTriggerTables(mtmv.getExcludedTriggerTables());
+ Map<TableIf, String> tableWithPartKey =
getIncrementalTableMap(mtmv);
+ // Pinned before the context is built, the way a refresh task
pins its inputs before its
+ // own: the mapping, the read predicates and the scan binding
then all resolve one
+ // snapshot per input, instead of the mapping re-materializing
the latest one per MV
+ // partition and the scan possibly binding to a different one
than the mapping read.
+ Map<MvccTableInfo, MvccSnapshot> pinnedSnapshots =
+ pinBaseTableSnapshots(mtmv, statementContext);
+ // The refresh aligns before it reads anything -- the
partitions the base partitions no
+ // longer describe are dropped, the ones they now produce are
created, and only then is the
+ // set to rebuild taken from the MV -- so the preview takes
the same set. A dropped partition
+ // left in it would be read through a scope the refresh never
reads, and being undescribed
+ // there it would also cost the retained partitions their
scopes; see
+ // MTMVPartitionUtil#mappedBasePartitionsByMvPartition. The
partitions the alignment creates
+ // are rebuilt in the same round, so the preview covers their
read scopes as well, even
+ // though it cannot target them in the sink; see
MTMVPartitionUtil#pendingPartitions.
+ CompleteRefreshPreview preview =
+ completeRefreshPreview(mtmv, pinnedSnapshots,
tableWithPartKey);
+ MTMVRefreshContext refreshContext =
+ MTMVRefreshContext.buildContext(mtmv,
Maps.newHashMap(), pinnedSnapshots);
+ // The base partitions a retained MV partition is read from,
from the mapping the refresh
+ // records it with, and those of the partitions the alignment
creates, from the alignment
+ // itself. Paired per partition rather than unioned for the
batch; see
+ // UpdateMvByPartitionCommand#constructTableWithPredicates.
+ Map<String, Map<BaseTableInfo, Set<String>>> readScopes =
+
MTMVPartitionUtil.mappedBasePartitionsByMvPartition(tableWithPartKey,
refreshContext,
+ preview.getSinkPartitionNames());
+ readScopes.putAll(preview.readScopesByPendingName());
if (mtmv.isIvm()) {
-
statementContext.setIvmRewriteContext(Optional.of(IvmRewriteContext.fullExplain(mtmv)));
+ // The explained refresh covers every MV partition in one
plan, which is the refresh-all
+ // first batch a refresh would run: its PCT inputs are
reset to the base partitions the
+ // mapping names -- a table it names as empty is read as
no rows -- and a non-PCT input is
+ // read in RESET mode. Both are derived the way MTMVTask
derives them, from the same
+ // scopes, so the preview's reads are the reads execution
makes; see IvmFullRefreshMTMV.
+
statementContext.setIvmRewriteContext(Optional.of(IvmRewriteContext.fullExplain(mtmv,
Review Comment:
Good catch. Two problems here, and the second one was masked by the first:
1. `IvmFullRefreshMTMV` read the rewrite context from the thread-local
`ConnectContext`, while the preview plans the command's own statement context
(`RefreshMTMVCommand#getExplainConnectContext`). In the EXPLAIN path those
diverge, so the rule did not fire at all: the preview showed the plain
base-table plan of the MV query, with no reset/snapshot stream read and no
partition scoping. It now reads `ctx.statementContext`, like
`IvmIncrRefreshMTMV` does.
2. Once the rule was reached, a missing/disabled stream did fail the
preview. It now builds the read without a usable persisted stream: for EXPLAIN
only, when `IvmUtil.findIvmStream` finds none it returns the same partitions as
a plain base-table scan, which is what a RESET stream scan normalizes to at
execution, and it creates nothing, so the preview stays read-only. EXECUTE
still fails closed through `IvmUtil.getIvmStream`.
Coverage: `IvmFullRefreshMTMVTest` gains
`testExplainWithoutStreamReadsBaseTable`,
`testExplainWithoutStreamReadsResetPartitionsFromBaseTable`,
`testExplainWithDisabledStreamReadsBaseTable`,
`testExecuteWithoutStreamStillFailsClosed` and
`testFindIvmStreamReportsOnlyUsableStream`.
`test_ivm_partitions_fallback_stream_unusable` now drops the joined table's
stream, previews COMPLETE (succeeds, plan asserts the joined table's partitions
are read from the base table, and no stream is created), then runs the real
COMPLETE and asserts the stream is recovered and the MV is complete.
`test_ivm_absent_pct_input_read_scope` was failing for the same context-read
reason and passes now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
