github-actions[bot] commented on code in PR #66717:
URL: https://github.com/apache/doris/pull/66717#discussion_r3797975498
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCache.java:
##########
@@ -33,43 +35,57 @@
import java.util.Map;
import java.util.concurrent.ExecutorService;
+import javax.annotation.Nullable;
/**
* Paimon engine implementation of {@link AbstractExternalMetaCache}.
*
* <p>Registered entries:
* <ul>
* <li>{@code table}: loaded Paimon table handle per table mapping</li>
+ * <li>{@code snapshot}: immutable partition projection keyed by a captured
snapshot/schema fence</li>
* <li>{@code schema}: schema cache keyed by table identity + schema id</li>
* </ul>
*
- * <p>Latest snapshot metadata is modeled as a runtime projection memoized
inside the table cache
- * value instead of as an independent cache entry.
+ * <p>The latest main-branch snapshot is captured once as a fence and loaded
through an independent
+ * contextual entry. Branch/tag/options projections remain statement-local and
are not aliased to
+ * this main-snapshot key.
*
* <p>Invalidation behavior:
* <ul>
- * <li>db/table invalidation clears table and schema entries by matching
local names</li>
+ * <li>db/table invalidation clears table, snapshot and schema entries by
matching local names</li>
* <li>partition-level invalidation falls back to table-level
invalidation</li>
* </ul>
*/
public class PaimonExternalMetaCache extends AbstractExternalMetaCache {
public static final String ENGINE = "paimon";
public static final String ENTRY_TABLE = "table";
+ public static final String ENTRY_SNAPSHOT = "snapshot";
public static final String ENTRY_SCHEMA = "schema";
private final EntryHandle<NameMapping, PaimonTableCacheValue> tableEntry;
+ private final EntryHandle<PaimonSnapshotEntryKey,
PaimonSnapshotCacheValue> snapshotEntry;
private final EntryHandle<PaimonSchemaCacheKey, SchemaCacheValue>
schemaEntry;
private final PaimonTableLoader tableLoader;
private final PaimonLatestSnapshotProjectionLoader
latestSnapshotProjectionLoader;
public PaimonExternalMetaCache(ExecutorService refreshExecutor) {
- super(ENGINE, refreshExecutor);
+ this(refreshExecutor, new
ExternalMetaCacheBudgetManager(java.util.OptionalLong.empty()));
+ }
+
+ public PaimonExternalMetaCache(ExecutorService refreshExecutor,
ExternalMetaCacheBudgetManager budgetManager) {
+ super(ENGINE, refreshExecutor, budgetManager);
tableLoader = new PaimonTableLoader();
latestSnapshotProjectionLoader = new
PaimonLatestSnapshotProjectionLoader(
new PaimonPartitionInfoLoader(),
this::getPaimonSchemaCacheValue);
tableEntry = registerEntry(MetaCacheEntryDef.of(ENTRY_TABLE,
NameMapping.class, PaimonTableCacheValue.class,
this::loadTableCacheValue, defaultEntryCacheSpec(),
- MetaCacheEntryInvalidation.forNameMapping(nameMapping ->
nameMapping)));
+ MetaCacheEntryInvalidation.forNameMapping(nameMapping ->
nameMapping))
+ .withReplacementListener(this::retireTableGeneration));
Review Comment:
[P2] Put the base-table entry under the weight budget
`tableEntry` still has no size estimator, so `newMetaCacheEntry()` never
creates an entry/catalog/global reservation for its cached
`PaimonTableCacheValue`. This owner is independent of the weighted
`snapshotEntry`: every snapshot lookup loads it first, normal/table-only paths
return its `FileStoreTable` directly, and `fetchRowCount()` can call
`newReadBuilder().newScan()` on that same object, materializing the
`lazyStore`/`RowType` graph documented by the new calibration helper. Fixing
the snapshot estimator therefore still leaves every base table outside the
configured memory cap. Please add fail-closed publication sizing for
`tableEntry` (including its lazily materializable store graph), or couple
ownership so the base handle is reserved for its full lifetime, and cover a
base-only scan plus reservation release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
