github-actions[bot] commented on code in PR #68238:
URL: https://github.com/apache/doris/pull/68238#discussion_r4058937722
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -140,13 +204,12 @@ public Table getPaimonTable(NameMapping nameMapping,
String branch, String query
* Doris REFRESH can repopulate its cache with the same stale Paimon table
handle.
*/
public Table reloadPaimonTable(NameMapping nameMapping) {
- makeSureInitialized();
Identifier identifier = tableIdentifier(nameMapping, null, null);
try {
- return executionAuthenticator.execute(() -> {
+ return withSdkCatalogCacheReadLock(() ->
executionAuthenticator.execute(() -> {
Review Comment:
[P2] Serialize this reload mutation against direct SDK misses. Explicit
refresh is now write-fenced, but an ordinary/TTL Doris cache miss still
invalidates and reloads Paimon under the shared read lock, while
TVF/existence/schema/branch misses use that same read lock. Paimon 1.4.2 loads
from the delegate and only then publishes with `tableCache.put`, so an older
direct miss can pause before its put, this reload can publish the fresh handle,
and the direct miss can resume and overwrite Paimon's cache with the old
handle. Doris's mutation epoch protects only `tableEntry`. Use the write fence
or an equivalent per-key generation for the entire invalidate/get sequence, and
add a paused direct-miss-versus-reload regression.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java:
##########
@@ -488,6 +488,12 @@ public void invalidateDb(long catalogId, String dbName) {
cache, catalogId, "invalidateDb", () ->
cache.invalidateDb(catalogId, dbName)));
}
+ public void invalidateDb(ExternalDatabase<?> database) {
+ long catalogId = database.getCatalog().getId();
+ routeCatalogEngines(catalogId, cache -> safeInvalidate(
+ cache, catalogId, "invalidateDb", () ->
cache.invalidateDb(database)));
Review Comment:
[P2] Make database-scope replay invalidation independent of the cached
`ExternalDatabase`. This typed route fixes the resolved-object case, but
database-cache eviction invokes its removal callback only once; a retained
TVF/branch `NameMapping` can republish an SDK-only handle afterward without
recreating the database value or `tableEntry`. `replayRefreshDb` then returns
on its cache-only miss, while `replayDropDb` falls back to name-only
invalidation that cannot enumerate that key. Thus both replayed REFRESH
DATABASE and DROP DATABASE can leave a follower serving the stale handle,
beyond the existing resolved-object thread. Persist/recover the remote database
identity or conservatively invalidate the SDK catalog when replay cannot
resolve it, and cover post-eviction late publication for both replays.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java:
##########
@@ -669,9 +676,9 @@ public ExternalRowCountCache getRowCountCache() {
}
public void invalidateTableCache(ExternalTable dorisTable) {
- invalidateTable(dorisTable.getCatalog().getId(),
- dorisTable.getDbName(),
- dorisTable.getName());
+ long catalogId = dorisTable.getCatalog().getId();
+ routeCatalogEngines(catalogId, cache -> safeInvalidate(
+ cache, catalogId, "invalidateTable", () ->
cache.invalidateTable(dorisTable)));
Review Comment:
[P1] Make table-scope replay invalidation independent of the cached
`ExternalTable`. This typed route works once called, but `replayRefreshTable`
and `unregisterTable` use cache-only `getTableForReplay` and return before
reaching it when the table object has expired or been capacity-evicted. That
table cache has no removal listener, and a TVF can populate only Paimon's SDK
cache, so the surviving names entry can rebuild the Doris object and a later
direct lookup can reuse the stale handle. A committed leader REFRESH or DROP
can therefore leave a follower stale. Persist/recover the remote mapping for
replay or conservatively invalidate the containing SDK scope, with SDK-only
eviction regressions for both operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
