github-actions[bot] commented on code in PR #69175:
URL: https://github.com/apache/doris/pull/69175#discussion_r4296587534
##########
fe/be-java-extensions/java-common/src/main/java/org/apache/doris/common/jni/vec/VectorColumn.java:
##########
@@ -1535,6 +1542,100 @@ public HashMap<String, Object>[] getStructColumn(int
start, int end) {
return result;
}
+ /** Append FILE's canonical six children. Payload bytes are copied into
off-heap storage. */
+ public int appendFile(FileValue file) {
+ if (file == null) {
+ return appendNull(Type.FILE);
+ }
+ reserve(appendIndex + 1);
+ VectorColumn uriColumn = childColumns[FileValue.URI_FIELD_INDEX];
+ if (file.getUri() == null) {
+ uriColumn.appendNull(Type.STRING);
+ } else {
+ uriColumn.appendStringAndOffset(file.getUri());
+ }
+
+ VectorColumn offsetColumn = childColumns[FileValue.OFFSET_FIELD_INDEX];
+ if (file.getOffset() == null) {
+ offsetColumn.appendNull(Type.BIGINT);
+ } else {
+ offsetColumn.appendLong(file.getOffset());
+ }
+
+ VectorColumn sizeColumn = childColumns[FileValue.SIZE_FIELD_INDEX];
+ if (file.getSize() == null) {
+ sizeColumn.appendNull(Type.BIGINT);
+ } else {
+ sizeColumn.appendLong(file.getSize());
+ }
+
+ VectorColumn contentTypeColumn =
childColumns[FileValue.CONTENT_TYPE_FIELD_INDEX];
+ if (file.getContentType() == null) {
+ contentTypeColumn.appendNull(Type.STRING);
+ } else {
+ contentTypeColumn.appendStringAndOffset(file.getContentType());
+ }
+
+ VectorColumn checksumColumn =
childColumns[FileValue.CHECKSUM_FIELD_INDEX];
+ if (file.getChecksum() == null) {
+ checksumColumn.appendNull(Type.STRING);
+ } else {
+ checksumColumn.appendStringAndOffset(file.getChecksum());
+ }
+
+ VectorColumn inlineColumn = childColumns[FileValue.INLINE_FIELD_INDEX];
+ if (file.getInline() == null) {
+ inlineColumn.appendNull(Type.VARBINARY);
Review Comment:
[P2] Avoid native allocations for absent FILE payloads
Descriptor output calls `appendNull(VARBINARY)`, which delegates to
`appendVarbinary(new byte[0])`; SQL NULL FILEs and empty inline values reach
the same allocation. In production, `OffHeap.allocateMemory(0)` invokes native
`malloc(0)`, and [jemalloc
5.3.0](https://github.com/jemalloc/jemalloc/blob/5.3.0/src/jemalloc.c) returns
an owned allocation. `freeVarbinaryData()` releases only entries with length >
0, so every such row permanently leaks its payload pointer when the batch is
released. Freeing the entry array does not reclaim these separate allocations.
Avoid allocating null/empty payloads or free every owned nonzero pointer, and
cover native allocation/release.
##########
fe/be-java-extensions/paimon-connector/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -226,6 +231,18 @@ private void initReader() throws IOException {
projection.readType().copy(tableField.type().isNullable())));
}
}
+ Table readTable = table;
+ if (readFields.stream().anyMatch(field -> field.type() instanceof
BlobType)) {
+ // The SDK otherwise allocates each managed BLOB before returning
its row. Request
+ // references so PaimonColumnValue can enforce the size limit
before reading inline.
+ // Keep this copy scanner-local: cached table options still
describe the output mode.
+ Map<String, String> blobReadOptions = Collections.singletonMap(
+ CoreOptions.BLOB_AS_DESCRIPTOR.key(), "true");
+ readTable = table instanceof FileStoreTable
+ ? ((FileStoreTable)
table).copyWithoutTimeTravel(blobReadOptions)
+ : table.copy(blobReadOptions);
Review Comment:
[P2] Preserve the bound schema when copying system-table readers
For a BLOB table's `$audit_log`, this branch uses
[`AuditLogTable.copy()`](https://github.com/apache/paimon/blob/release-1.4.2/paimon-core/src/main/java/org/apache/paimon/table/system/AuditLogTable.java),
which delegates to `wrapped.copy()` and replays the inherited
`scan.snapshot-id`. After a schema-only ADD COLUMN, Doris pins the last data
snapshot while retaining the latest schema, so this copy drops the new column
after `projected` was computed. With an explicit `read.batch-size` (so the
earlier default helper leaves the wrapper unchanged), selecting the BLOB and
the new column reaches `withProjection(projected)` against the older row type
and fails instead of returning NULL for historical rows. Copy the hidden
FileStoreTable without time travel and rebuild the same wrapper; cover this
schema-only ALTER path.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -1318,6 +1364,174 @@ public List<Split> getSplits(int numBackends) throws
UserException {
return splits;
}
+ private boolean needsNativeBlobMaterialization(boolean hasProjectedFile) {
+ if (!hasProjectedFile || !sessionVariable.enableFileScannerV2
+ || !(processedTable instanceof FileStoreTable)
+ || processedTable instanceof FallbackReadFileStoreTable) {
+ return false;
+ }
+ FileStoreTable table = (FileStoreTable) processedTable;
+ if (!table.coreOptions().dataEvolutionEnabled() ||
table.coreOptions().queryAuthEnabled()) {
Review Comment:
[P2] Keep renewable REST-token BLOB scans on JNI
This new native-BLOB gate accepts a `RESTTokenFileIO` table with
`queryAuthEnabled()==false`. FE extracts only `validToken().token()` into
backend storage properties, dropping expiration and the REST refresh context;
native anchor/companion readers reuse that fixed map, and explicit AK/SK select
`SimpleAWSCredentialsProvider`. A queued or long scan that crosses the token
TTL therefore fails authentication, while the [JNI FileIO
refreshes](https://github.com/apache/paimon/blob/release-1.4.2/paimon-common/src/main/java/org/apache/paimon/rest/RESTTokenFileIO.java)
before opening subsequent files. Exclude renewable REST-token tables here
until native credential refresh is supported, as the existing Rust gate already
does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
