924060929 commented on code in PR #66473:
URL: https://github.com/apache/doris/pull/66473#discussion_r3801153296
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java:
##########
@@ -119,6 +121,8 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
public static final String ERR_CACHE_INCONSISTENCY =
"ERR_CACHE_INCONSISTENCY: ";
private final ExecutorService fileListingExecutor;
+ private final Map<Long, AtomicLong> fileCacheInvalidationGenerations = new
ConcurrentHashMap<>();
Review Comment:
Fixed in fb29461e56c. `invalidateCatalog` (drop / property-driven full
rebuild) now removes both per-catalog generation counters after advancing them,
so dropped catalog ids (which are never reused) cannot accumulate;
`invalidateCatalogEntries` (REFRESH) intentionally keeps them.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java:
##########
@@ -136,6 +142,46 @@ public FileQueryScanNode(PlanNodeId id, TupleDescriptor
desc, String planNodeNam
StatisticalType statisticalType, ScanContext scanContext, boolean
needCheckColumnPriv, SessionVariable sv) {
super(id, desc, planNodeName, statisticalType, scanContext,
needCheckColumnPriv);
this.sessionVariable = sv;
+ ConnectContext context = ConnectContext.get();
+ StatementContext statementContext = context == null ? null :
context.getStatementContext();
+ this.externalScanTaskCache = statementContext == null
+ ? null : statementContext.getExternalScanTaskCache();
+ }
+
+ protected <T> List<T> getOrLoadExternalScanTasks(
+ ExternalScanTaskCacheKey<T> key, Callable<List<T>> loader) throws
Exception {
+ if (!sessionVariable.enableExternalScanTaskReuse ||
externalScanTaskCache == null) {
+ return loader.call();
+ }
+ return externalScanTaskCache.getOrLoad(key, loader);
+ }
+
+ protected <T> List<T> getOrLoadExternalScanTasks(
+ ExternalScanTaskCacheKey<T> key, Callable<List<T>> loader,
+ ToLongFunction<List<T>> weigher, long maxRetainedWeight) throws
Exception {
+ if (!sessionVariable.enableExternalScanTaskReuse ||
externalScanTaskCache == null) {
+ return loader.call();
+ }
+ return externalScanTaskCache.getOrLoad(key, loader, weigher,
maxRetainedWeight);
+ }
+
+ protected <T> List<T> getOrLoadExternalScanTasks(
+ ExternalScanTaskCacheKey<T> key,
+ StatementContext.ExternalScanTaskCache.WeightedLoader<T> loader,
+ ToLongFunction<List<T>> weigher,
+ StatementContext.ExternalScanTaskCache.WeightBudget weightBudget,
+ long maxEntryWeight, long maxRetainedWeight,
+ boolean reserveBeforeLoad) throws Exception {
+ if (!sessionVariable.enableExternalScanTaskReuse ||
externalScanTaskCache == null) {
Review Comment:
Fixed in 67079c6e38b. Paimon and Iceberg now check
`canReuseExternalScanTasks()` before entering the cache path and consume native
tasks directly when reuse is disabled or no statement cache exists, so an
opt-out no longer pays the serialization cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
