yujun777 commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4225891120
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java:
##########
@@ -1619,6 +1621,65 @@ public Map<String, Map<MTMVRelatedTableIf, Set<String>>>
calculatePartitionMappi
return res;
}
+ /**
+ * The list partition each base table of this MV has that takes the rows
no other partition of it claims,
+ * by table, or none for a table that has no such partition.
+ *
+ * <p>Read once per mapping rather than per MV partition: the mapping
describes every MV partition and the
+ * answer is the table's, not the partition's. The table's partitions are
read under its read lock, so
+ * that a concurrent ADD or DROP PARTITION cannot be seen half applied --
its name list and the items the
+ * walk resolves against it have to come from one state of the table --
and so that this walk is not one
+ * more reader of a tree another thread is modifying.
+ */
+ private Map<MTMVRelatedTableIf, String> defaultListPartitionsOf() throws
AnalysisException {
+ Map<MTMVRelatedTableIf, String> res = Maps.newHashMap();
+ for (MTMVRelatedTableIf pctTable : mvPartitionInfo.getPctTables()) {
+ if (!(pctTable instanceof OlapTable)) {
+ continue;
+ }
+ OlapTable olapTable = (OlapTable) pctTable;
+ if (!(olapTable.getPartitionInfo() instanceof ListPartitionInfo)) {
+ continue;
+ }
+ olapTable.readLock();
+ try {
+ for (String partitionName : olapTable.getPartitionNames()) {
+ if
(olapTable.getPartitionItemOrAnalysisException(partitionName).isDefaultPartition())
{
+ res.put(pctTable, partitionName);
+ break;
+ }
+ }
+ } finally {
+ olapTable.readUnlock();
+ }
+ }
+ return res;
+ }
+
+ /**
+ * One MV partition's mapping, with every base table's default list
partition named in it.
+ *
+ * <p>Such a partition holds rows for every key its table can be read by,
so it belongs to every MV
+ * partition that reads the table -- not only to the one its own key, the
sentinel those rows were placed
+ * by, maps to. Naming it everywhere is what the read and the record have
to agree on: the refresh reads
+ * the rows of it that belong to the MV partition being refreshed, and the
partition is recorded among the
+ * ones that partition is read through, so an insert into it leaves that
MV partition out of sync instead
+ * of changing nothing the MV compares.
+ */
+ private Map<MTMVRelatedTableIf, Set<String>> withDefaultListPartitions(
+ Map<MTMVRelatedTableIf, Set<String>> mapping,
Map<MTMVRelatedTableIf, String> defaultListPartitions) {
+ if (defaultListPartitions.isEmpty()) {
+ return mapping;
+ }
+ Map<MTMVRelatedTableIf, Set<String>> res = Maps.newHashMap(mapping);
+ for (Entry<MTMVRelatedTableIf, String> entry :
defaultListPartitions.entrySet()) {
+ Set<String> partitions =
Sets.newHashSet(res.getOrDefault(entry.getKey(), Sets.newHashSet()));
+ partitions.add(entry.getValue());
+ res.put(entry.getKey(), partitions);
Review Comment:
Took your first direction in 67048fe1b4e: the rewrite is refused, the shape
is not.
Confirmed first, and it is worse than a narrow corner -- measured on
`LIST(k)` with `p1=(1)` and `p_default`, a committed `k=2` row in `p_default`,
and an MV partitioned by `k`: the MV holds key 1 alone, and a plain `select k,
sum(amount) from t group by k` came back with the k=1 row only before this
change. The rewrite answered from the view and lost committed data.
`getMTMVCanRewritePartitions` now answers with no partition when any PCT
table of the MV has a list partition's default partition
(`MTMVPartitionUtil#hasDefaultListPartition`, read under the table's read
lock). The same query now returns both rows and the plan is a base table scan;
the MV itself still builds, refreshes and answers direct queries.
Why the MV rather than the partitions that read that table: which keys the
default partition holds is not in the metadata, so "a query whose keys are all
covered by the MV's partitions" cannot be decided. Scoping it to the partitions
that read that table would not help either -- the k=2 row is read out of
`p_default` by any query over the table, and once an explicit partition is
added over that key the row stays where it is (the ADD-over-default case from
the thread above), so the MV still does not hold it.
The regression case in `test_mtmv_base_partition_read_scope` asserts both
sides: what the view holds (the k=1 row alone) and that the query over the base
table is answered whole with the rewrite on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
