OIiveirra commented on code in PR #68161:
URL: https://github.com/apache/doris/pull/68161#discussion_r4312497719
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -238,19 +274,97 @@ private void initReader() throws IOException {
hasReadTypeProjection = true;
readFields.add(tableField.newType(
variantProjection.readType().copy(tableField.type().isNullable())));
+ currentFields.add(readFields.get(readFields.size() - 1));
}
outputIndex++;
}
+ boolean repairPolicy = requiresDatetimeV2PrecisionRepair && authSplit
!= null
+ && authSplit.authResult() != null;
+ List<DataField> outputReadFields = new ArrayList<>(readFields);
+ RowType outputType = new RowType(currentFields);
+ if (repairPolicy) {
+ PaimonTimestampQueryAuth.addPolicyFields(
+ authSplit.authResult(), policyTableType, readFields,
currentFields);
+ split = PaimonTimestampQueryAuth.repairPolicy(split, new
RowType(currentFields));
+ hasReadTypeProjection = true;
+ }
if (hasReadTypeProjection) {
readBuilder.withReadType(new RowType(readFields));
} else {
readBuilder.withProjection(projected);
}
+ // FE removes only predicates that depend on timestamp values being
repaired. Safe
+ // non-timestamp predicates remain useful for Paimon file and row
pruning.
readBuilder.withFilter(getPredicates());
- reader =
newReadWithOptionalIOManager(readBuilder).executeFilter().createReader(getSplit());
+ reader =
newReadWithOptionalIOManager(readBuilder).executeFilter().createReader(split);
+ if (repairPolicy) {
+ reader = PaimonTimestampQueryAuth.projectOutput(reader, new
RowType(currentFields), outputType);
+ }
// Decode with the types that were REQUESTED, not the table's: a
pruned nested column comes back
// narrow, and taking table.rowType() here would misalign every one of
them.
- paimonDataTypeList =
readFields.stream().map(DataField::type).collect(Collectors.toList());
+ paimonDataTypeList =
outputReadFields.stream().map(DataField::type).collect(Collectors.toList());
+ }
+
+ private static Table dataSource(Table source) {
+ if (source instanceof FileStoreTable) {
+ return unwrapSystemPlanningSource((FileStoreTable) source);
+ }
+ if (!(source instanceof AuditLogTable || source instanceof
ReadOptimizedTable
+ || source instanceof RowTrackingTable)) {
+ return source;
+ }
+ // A row-backed system wrapper synthesizes its system fields outside
the data reader.
+ // Resolve the pinned data source, without reopening the catalog.
+ Optional<FileStoreTable> systemSource = hiddenSystemSource(source);
+ return systemSource.isPresent() ? dataSource(systemSource.get()) :
source;
+ }
+
+ private static RowType policyRowType(Table source, Split split) {
+ source = dataSource(source);
+ if (source instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable)
source;
+ boolean readFallback = split instanceof FallbackSplit &&
((FallbackSplit) split).isFallback();
+ FileStoreTable branch = isWrappedFirst(pair) != readFallback ?
pair.wrapped() : pair.other();
Review Comment:
Fixed in 1fff54b8214c71c6aa778f443225d9ba307f0515. Schema resolution now
handles ChainGroupReadTable before generic fallback dispatch and uses other(),
matching the SDK delta reader while preserving the original ChainSplit.
Added a real historical Parquet audit-log reader test through the outer
fallback and chain group, with independent snapshot/delta field IDs. Before the
fix, rowkind and id were correct but a stored whole-second timestamp became
NULL. After the fix, the test verifies non-NULL whole seconds, p6-to-p0
truncation and original NULLs.
Validation passed: 222 tests across six focused FE/JNI classes, Checkstyle,
quick checks, clang-tidy, the standard ASAN FE/BE build and verified personal
deployment, and branch-aligned paimon_timestamp_types External regression (1
suite, 0 failed/fatal/skipped). Full PR Standards and Spec reviews found no
remaining actionable issues after the fix. These are local results, not a claim
that the new community CI is already green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
