github-actions[bot] commented on code in PR #66717:
URL: https://github.com/apache/doris/pull/66717#discussion_r3798340953
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:
##########
@@ -1751,18 +1766,29 @@ public static IcebergPartitionInfo
loadPartitionInfo(ExternalTable dorisTable, T
}
Map<String, IcebergPartition> nameToPartition = Maps.newHashMap();
Map<String, PartitionItem> nameToPartitionItem = Maps.newHashMap();
+ long retainedPayloadBytes = 0L;
- List<Column> partitionColumns =
IcebergUtils.getSchemaCacheValue(dorisTable, schemaId).getPartitionColumns();
+ List<Column> partitionColumns = IcebergUtils.getSchemaCacheValue(
+ dorisTable, schemaId, table).getPartitionColumns();
+ long partitionItemColumnBytes =
IcebergPartitionInfo.partitionItemColumnBytes(
+ partitionColumns.size());
for (IcebergPartition partition : icebergPartitions) {
nameToPartition.put(partition.getPartitionName(), partition);
+ retainedPayloadBytes = MetaCacheWeightUtils.saturatedAdd(
+ retainedPayloadBytes, partition.getRetainedPayloadBytes());
+ retainedPayloadBytes = MetaCacheWeightUtils.saturatedAdd(
Review Comment:
[P2] Charge range width only for surviving merged items
This surcharge is added for every physical partition before
`mergeOverlapPartitions()` mutates `nameToPartitionItem` and drops enclosed
ranges. It models the two `PartitionKey` endpoints of each
`RangePartitionItem`, so those deleted endpoint graphs are not retained; the
physical `IcebergPartition` values/transforms are already charged separately.
For a wide mixed-spec overlap group, the excess grows with every enclosed
day/hour partition and can reject a snapshot that fits or evict useful
metadata. Please apply the width surcharge after merging using
`nameToPartitionItem.size()`, and add a width-greater-than-one fixture that
actually removes enclosed ranges.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCache.java:
##########
@@ -86,41 +103,76 @@ public Table getPaimonTable(NameMapping nameMapping) {
public PaimonSnapshotCacheValue getSnapshotCache(ExternalTable dorisTable)
{
NameMapping nameMapping = dorisTable.getOrBuildNameMapping();
- return
tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getLatestSnapshotCacheValue();
+ PaimonTableCacheValue tableValue =
tableEntry.get(nameMapping.getCtlId()).get(nameMapping);
+ PaimonSnapshot fence = loadLatestSnapshotFence(nameMapping,
tableValue.getPaimonTable()).getSnapshot();
+ PaimonSnapshotEntryKey key = PaimonSnapshotEntryKey.of(
+ nameMapping, fence, tableValue.getGeneration());
+ MetaCacheEntry<PaimonSnapshotEntryKey, PaimonSnapshotCacheValue> entry
=
+ snapshotEntry.get(nameMapping.getCtlId());
+ PaimonSnapshotCacheValue snapshotValue = entry.get(key,
+ ignored -> executeAuthenticated(nameMapping,
+ () -> latestSnapshotProjectionLoader.loadAtFence(
+ nameMapping, fence,
tableValue.getGeneration())));
+ PaimonTableCacheValue currentTable =
tableEntry.get(nameMapping.getCtlId()).peekIfPresent(nameMapping);
Review Comment:
[P2] Retire children when base admission is rejected
`MetaCacheEntry` intentionally returns a loaded value even when its table
estimate is incomplete or exceeds `meta.cache.paimon.table.max-weight`, but
this guard invalidates the child only when another table is present. In the
rejection case `peekIfPresent()` remains null; each call receives a fresh
synthetic table generation and leaves its snapshot plus schema entries under
keys no future call can reuse. With only the table weight configured, those
children remain count-bounded and can accumulate until TTL/capacity eviction
while repeatedly reloading partitions. Please treat a null current table as
stale in both post-load guards (or prevent dependent publication without an
admitted base), and add a repeated-rejection regression that keeps
snapshot/schema cardinality stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
