github-actions[bot] commented on code in PR #69299: URL: https://github.com/apache/doris/pull/69299#discussion_r4317051884
########## fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionView.java: ########## @@ -0,0 +1,208 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.mvcc; + +import org.apache.doris.catalog.PartitionItem; +import org.apache.doris.catalog.PrimitiveType; +import org.apache.doris.catalog.ScalarType; +import org.apache.doris.catalog.Type; +import org.apache.doris.connector.spi.ConnectorPartitionInfo; +import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartition; +import org.apache.doris.nereids.rules.expression.rules.SortedPartitionRanges; +import org.apache.doris.qe.ConnectContext; +import org.apache.doris.thrift.TScalarType; +import org.apache.doris.thrift.TTypeDesc; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; + +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; + +/** One published partition generation: maps and pruning index share the same read-only partition items. */ +public final class PluginDrivenPartitionView { + private final ImmutableMap<String, PartitionItem> items; + private final ImmutableMap<String, Long> freshness; + private final SortedPartitionRanges<String> sortedRanges; + + PluginDrivenPartitionView(Map<String, PartitionItem> items, Map<String, Long> freshness) { + this.items = ImmutableMap.copyOf(items); + this.freshness = ImmutableMap.copyOf(freshness); + // Build before cache admission, like HivePartitionValues: the cache accounts for the index + // together with its maps, and readers never observe a partially initialized generation. + this.sortedRanges = SortedPartitionRanges.build(this.items); Review Comment: [P2] Skip sorted indexes for heterogeneous partition display keys This now sorts Iceberg's display/count partitions even when the connector marks the view UNPARTITIONED and disables internal pruning. For example, start with identity(a), write a=1, then add identity(b) and write (a=1,b=1). The current snapshot contains a=1 and a=1/b=1; buildListPartitionView uses each record's partitionValueTypes, producing one- and two-column keys. SortedPartitionRanges.build compares the longer key with the shorter key, and MultiColumnBound.compareTo indexes the missing second column and throws IndexOutOfBoundsException. An enabled partition-metadata block rule or enable_iceberg_internal_partition_pruning=true therefore makes even an unfiltered SELECT fail while materializing the view. Preserve these display/count maps without constructing a pruning index for heterogeneous, non-prunable keys. ########## fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java: ########## @@ -540,6 +552,10 @@ private static class DefaultExternalMetaCache extends ExternalCatalogMetaCache { MetaCacheEntryInvalidation.forTableIdentity( key -> key.getNameMapping().getLocalDbName(), key -> key.getNameMapping().getLocalTblName()))); + registerEntry(MetaCacheEntryDef.contextualOnly( Review Comment: [P2] Give partition views a scalable cache size estimator Registering this entry through the default cache path gives it MetaCacheSizeEstimators.reflective(), whose complete traversal stops after 10,000 distinct objects. When external_meta_cache_max_weight or a catalog/entry weight limit is enabled, a large partition view (for example, 120,000 partitions) necessarily exceeds that traversal budget, even if it fits the configured byte quota. ScopedMetaCache treats the incomplete estimate as an admission rejection, so the view is never cached. Every eager/internal-pruning or MV query then rebuilds the partition items and sorted index; getSortedPartitionRanges also bypasses the previous independent sorted cache. Supply a complete construction-time estimate for the view and key so the intended reuse works with memory governance enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
