q2788116586-tech opened a new issue, #20277: URL: https://github.com/apache/hudi/issues/20277
### Task Description ## Summary Track a focused improvement identified in #17128: support vectorized Parquet decoding for the base-file side of Spark MOR FileGroupReader (FGR), while retaining row-level merging and row output. The broader read-performance audit already identifies this limitation, and its triage discussion recommends splitting independently actionable items. This issue scopes that item to the explicit non-vectorized base reader used by the MOR merge path. It is not a claim of a newly discovered regression, a duplicate of the session-configuration fixes, or a request for a fully columnar merge engine. ## Source observations The following observations are pinned to upstream commit `be8a54c4ebdaf287993d24fefba60a58952a9c9e` (source inspection, not a benchmark of that commit), focusing on the Spark 3.4 implementation. 1. In [HoodieFileGroupReaderBasedFileFormat.scala, lines 274–283](https://github.com/apache/hudi/blob/be8a54c4ebdaf287993d24fefba60a58952a9c9e/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala#L274-L283), the ordinary base reader uses `supportVectorizedRead`. However, when `isMOR && supportVectorizedRead`, the separate FGR base reader is constructed with `enableVectorizedRead = false`. The nearby comment attributes this to row-level merging and leaves a TODO for vectorized reader support. 2. The FGR branch [passes that reader into SparkFileFormatInternalRowReaderContext](https://github.com/apache/hudi/blob/be8a54c4ebdaf287993d24fefba60a58952a9c9e/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala#L304-L307). 3. [HoodieMergeOnReadRDDV2.scala, lines 113–125](https://github.com/apache/hudi/blob/be8a54c4ebdaf287993d24fefba60a58952a9c9e/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala#L113-L125) also constructs its Parquet FGR base reader with `vectorized = false`, separately from `OPTION_RETURNING_BATCH = false`. 4. [Spark34ParquetReader.scala, lines 167–229](https://github.com/apache/hudi/blob/be8a54c4ebdaf287993d24fefba60a58952a9c9e/hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark34ParquetReader.scala#L167-L229) has distinct vectorized-decoding and row-based ParquetRecordReader branches. Its [builder](https://github.com/apache/hudi/blob/be8a54c4ebdaf287993d24fefba60a58952a9c9e/hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark34ParquetReader.scala#L252-L295) passes the supplied `vectorized` argument into `enableVectorizedReader`; enabling Spark's vectorized-reader SQL setting alone does not override the explicit false in these FGR paths. This concerns the reader supplied to FGR, not every MOR query or every base-file-only path. Likewise, `supportVectorizedRead: true` at the FileFormat level does not by itself demonstrate vectorized decoding inside FGR. ## Why investigate this? For wide projections over base-plus-log file slices, column pruning eliminates little base-file work. Non-vectorized base decoding could therefore become a meaningful cost even if FGR improves other parts of the read path. This is a performance hypothesis, not evidence that the setting alone explains a cross-version regression. No isolated upstream before/after measurement is claimed here. Vectorized decoding and returning ColumnarBatch to Spark are separate decisions. Could the base reader decode batches internally but expose compatible rows to the existing merge pipeline? This would not require vectorizing the merge algorithm itself. It is also not a proposal to simply flip the boolean without checking row representation, ownership and correctness. ## Proposed scope and validation - Identify the constraints behind the current restriction: row representation (including UnsafeRow assumptions), batch-buffer reuse/lifetime, projection/schema evolution, and merge semantics. - If feasible, support vectorized decoding for eligible Parquet base reads with row output, retaining a safe fallback for unsupported cases and covering both entry points above. - Validate inserts, updates, partial updates, deletes, nulls, schema evolution and timestamp handling, including records retained across batch boundaries. - Benchmark the current and candidate readers on the same upstream build baseline and frozen MOR snapshot, with identical resources and query expressions. Include narrow and full projections, base-plus-log slices, and base-only/log-only controls. Consume the projected values rather than relying on count(*) alone. Record scan/task time, input bytes, CPU/GC and spill, with repeated paired runs. These are proposed validation steps, not completed results. ## Questions for maintainers What correctness constraints must be addressed before vectorized base decoding can feed row-level FGR merging? In particular, which row-representation and buffer-lifetime contracts should an implementation preserve? If an existing implementation already covers this MOR-specific scope, please link it so this issue can be consolidated with that work. ### Task Type Performance optimization ### Related Issues - #17128 (HUDI-9674): umbrella Spark 1.X read-performance audit. It explicitly calls for exploring FGR vectorized reads; this issue proposes tracking that item separately. - #20089 / PR #20090 and #20140 / PR #20193: related per-scan reader decisions and session-configuration mutation. Fixing those decisions is distinct from enabling vectorized decoding in the MOR-specific reader explicitly constructed with `false` at the pinned commit. - #20182 / PR #20183: reports stale vectorization state and a `ColumnarBatchRow` / `UnsafeRow` mismatch, with a COW wide-nested-schema reproduction. Relevant to reader/output compatibility, but not evidence of the same failure in MOR merging. - #17026 (HUDI-9443): reflection on `batchIdx` in the vectorized reader's per-row access. Relevant when considering a vectorized-decoding/row-output implementation; it does not itself address the MOR reader's explicit disablement. - #20064: broader executor-overhead work in Hudi 1.x. Task serialization, configuration and schema work are separate performance factors, not attributed to vectorization by this issue. - PR #13127 (HUDI-9302): related work enabling vectorized reading for file slices without log files; useful background for extending support to the MOR merge path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
