morningman commented on code in PR #65867:
URL: https://github.com/apache/doris/pull/65867#discussion_r3654833953
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -190,6 +202,182 @@ protected void doInitialize() throws UserException {
}
}
+ /**
+ * Build the Paimon table object that is serialized to the BE.
+ *
+ * <p>Every table loaded from a metastore-backed Paimon catalog (HMS /
DLF) carries a Paimon
+ * {@code HiveCatalogLoader} in its {@link
org.apache.paimon.table.CatalogEnvironment}. The BE
+ * only reads — via FE-resolved splits and the object store — and never
needs the catalog, yet
+ * deserializing that loader forces the whole Hive metastore stack onto
the BE classpath:
+ * {@code HiveConf}, the metastore API, and, when a system table resolves
its latest snapshot,
+ * even the metastore client (DLF's {@code ProxyMetaStoreClient} and its
REST stack). So we
+ * serialize a catalog-less table to the BE:
+ * <ul>
+ * <li>data table: drop the catalog loader. A {@link FileStoreTable} is
fully defined by
+ * fileIO / location / schema / catalogEnvironment, and its dynamic
options (time travel,
+ * incremental) are merged into the schema by {@code copy(...)}, so
rebuilding from
+ * fileIO / location / schema preserves everything except the
catalog loader.</li>
+ * <li>system table (e.g. {@code $snapshots}): rebuild it over a
catalog-less data table so
+ * {@code SnapshotManager#latestSnapshotId} lists the snapshot
directory on the filesystem
+ * instead of calling the metastore. The base table is the one the
FE-side wrapper was
+ * built over, and for the system tables that pick their snapshot on
the BE ({@link
+ * #resolvesSnapshotOnBackend}) what the catalog would have done
there is done here
+ * instead: see {@link #authorizeDeferredScan} and {@link
#pinCatalogSnapshot}. Every
+ * other system table reads what the FE already planned, so it is
handed over
+ * untouched.</li>
+ * </ul>
+ */
+ private Table getPaimonTableForBackend() {
+ Table paimonTable = source.getPaimonTable();
+ if (paimonTable instanceof FileStoreTable) {
+ return dropCatalogLoader((FileStoreTable) paimonTable);
+ }
+ if (!(source.getExternalTable() instanceof PaimonSysExternalTable)) {
+ return paimonTable;
+ }
+ PaimonSysExternalTable sysTable = (PaimonSysExternalTable)
source.getExternalTable();
+ // The very same base table the FE-side wrapper was built over, so
that the BE never sees a
+ // different schema generation than the one this query was planned
with.
+ FileStoreTable dataTable = sysTable.getSysBaseTable();
+ if (dataTable == null) {
+ return paimonTable;
+ }
+ String sysTableType = sysTable.getSysTableType();
+ if (PAIMON_FILES_SYSTEM_TABLE_TYPE.equalsIgnoreCase(sysTableType)) {
+ authorizeDeferredScan(dataTable);
+ }
+ FileStoreTable baseForBackend = dropCatalogLoader(dataTable);
+ if (resolvesSnapshotOnBackend(sysTableType)) {
+ baseForBackend = pinCatalogSnapshot(baseForBackend, dataTable);
Review Comment:
The mechanism is real, but neither remedy is available through paimon's API,
and this is not a regression from the pin. Documented as a Known gap on
`pinCatalogSnapshot` in e63a97be480 instead of changed.
**Why the marker set cannot be bound to the pin.**
`FilesTable.FilesScan#innerPlan` (`FilesTable:189-217`) builds its markers from
`fileStoreTable.newSnapshotReader().partitions()`. That resolves as
`SnapshotReaderImpl#partitions:415` -> `scan.listPartitions()` ->
`AbstractFileStoreScan#readManifests:414` ->
`ManifestsReader#read(specifiedSnapshot = null, ..):114-122`, which falls
through to `snapshotManager.latestSnapshot()`. `SnapshotReaderImpl` never reads
`scan.snapshot-id` out of the `CoreOptions` it is handed - a snapshot is only
ever set by an explicit `withSnapshot(..)` (`:168-176`), and `FilesScan` does
not call it. So no table level option, on either the wrapper or the base table,
can move the FE marker set; "use the same captured bound for FE marker
generation" is not expressible here. The second suggestion is not either:
`FilesSplit` is a `private static` nested class of `FilesTable`, so Doris
cannot construct a snapshot independent marker.
**Why it is not introduced by the pin.** `$files` was already a two-phase
plan with two independent snapshot resolutions. Before this change the BE
re-planned each marker at read time against a catalog-ful table
(`FilesRead#createReader` -> `FilesSplit#splits` ->
`storeTable.newScan().plan()`), i.e. the marker set came from the FE at
split-generation time and the file list from a snapshot resolved strictly
later, across a wider window than init-to-finalize. The same hybrid was
therefore observable before, in the opposite direction; the pin only moves
which side of the window the mismatch falls on, and narrows it.
So the observable behaviour is unchanged in kind and improved in window
size, and closing it properly needs a change in paimon (`FilesScan` honoring
`scan.snapshot-id`, or exposing the marker type), not in Doris. Recorded next
to the existing `$snapshots` / `$buckets` and fallback-branch gaps, with the
exact call chain, so the next reader does not have to re-derive it.
For the record on scope: `$partitions` is not affected - `PartitionsScan`
emits a single stateless `SingletonSplit` and the BE reader plans the whole
table, so there is no FE-side marker set to skew.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
