voonhous commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3542085557
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceReader.java:
##########
@@ -208,8 +208,17 @@ private ClosableIterator<UnsafeRow>
getUnsafeRowIterator(HoodieSchema requestedS
// Pinned to CONTENT: compaction/merge/log-replay need actual bytes to
rewrite.
// The user-facing `hoodie.read.blob.inline.mode` is honored by
SparkLanceReaderBase.
FileReadOptions readOpts =
FileReadOptions.builder().blobReadMode(BlobReadMode.CONTENT).build();
- ArrowReader arrowReader = lanceReader.readAll(columnNames, null,
DEFAULT_BATCH_SIZE, readOpts);
+ // BLOB reads must be chunked to dodge a lance-core FFI abort (see
LanceRecordIterator).
+ // Unwrap a nullable UNION wrapper before checking whether the
projection has a BLOB field.
+ HoodieSchema recordSchema = requestedSchema.getNonNullType();
+ boolean hasBlob = recordSchema.hasFields()
+ && recordSchema.getFields().stream().anyMatch(f ->
f.schema().isBlobField());
+ if (hasBlob) {
+ return LanceRecordIterator.chunkedBlobReader(allocator, lanceReader,
columnNames, readOpts,
Review Comment:
This branch is the internal CONTENT path (via
`HoodieSparkFileReaderFactory`), so it is only reached by
compaction/merge/log-replay, not datasource reads. Both batch-scale tests read
through `SparkLanceReaderBase`, so nothing exercises this chunked branch above
512 rows. A MOR blob table with >512 rows in a base file would run it during
compaction and cross the boundary untested. Can we add a >512-row MOR test that
forces inline compaction and reads back, so this path is covered at the
boundary too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
