voonhous commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3549713634
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/LanceRecordIterator.java:
##########
@@ -32,39 +32,54 @@
import org.apache.spark.sql.types.StructType;
import org.apache.spark.sql.vectorized.ColumnVector;
import org.apache.spark.sql.vectorized.ColumnarBatch;
+import org.lance.file.FileReadOptions;
import org.lance.file.LanceFileReader;
import org.lance.spark.vectorized.LanceArrowColumnVector;
+import org.lance.util.Range;
import java.io.IOException;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
/**
- * Iterator for reading Lance files and converting Arrow batches to Spark
{@link UnsafeRow}s.
- * Used by both Hudi's internal Lance reader and Spark datasource integration.
+ * Iterator over a Lance file that converts Arrow batches to Spark {@link
UnsafeRow}s, owning the
+ * {@link BufferAllocator}, {@link LanceFileReader}, {@link ArrowReader}(s)
and current
+ * {@link ColumnarBatch}. Used by both Hudi's internal Lance reader and the
Spark datasource. An
+ * optional {@link BlobDescriptorTransform} rewrites BLOB columns for
DESCRIPTOR-mode reads.
*
- * <p>The iterator manages the lifecycle of:
- * <ul>
- * <li>BufferAllocator - Arrow memory management</li>
- * <li>LanceFileReader - Lance file handle</li>
- * <li>ArrowReader - Arrow batch reader</li>
- * <li>ColumnarBatch - Current batch being iterated</li>
- * </ul>
- *
- * <p>An optional {@link BlobDescriptorTransform} can be composed in to
rewrite BLOB columns
- * in DESCRIPTOR mode.
+ * <p>BLOB chunked reading: lance-core 4.0.0 aborts the JVM in its Arrow
C-stream export whenever a
+ * single {@code readAll} stream crosses Lance's internal BLOB page boundary
(512 rows), and the
+ * requested {@code batchSize} does not change that. So BLOB reads issue one
{@code readAll} per
+ * row-range chunk of {@link #BLOB_READ_CHUNK_ROWS} rows; non-BLOB reads use
one streamed reader.
*/
public final class LanceRecordIterator implements ClosableIterator<UnsafeRow> {
+
+ /** Rows per {@code readAll} for BLOB reads. Must not exceed Lance's 512-row
BLOB page size. */
+ public static final int BLOB_READ_CHUNK_ROWS = 512;
Review Comment:
both addressed: blob detection now recurses (made containsBlobType() public
and used it in HoodieSparkLanceReader, recursive walk on the datasource side),
so a nested blob would still route through the chunked path. for the 512: lance
exposes no API for its blob page size, so pinned the constant javadoc to
lance-core 4.0.0 with a note to revalidate against the batch-scale tests on any
lance bump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
