[
https://issues.apache.org/jira/browse/SPARK-57415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated SPARK-57415:
---------------------------------
Description:
Umbrella tracking a series of performance improvements to the Parquet
vectorized reader in Spark SQL. The changes target allocation reduction,
bulk-read optimizations, and JIT-friendly code patterns across multiple
encoding paths.
All subtasks are independent and can be reviewed/merged in any order. Together
they yield significant throughput gains (1.2x to 9x depending on the encoding
and data shape) for Parquet reads with no user-facing behavioral changes.
Common themes:
- Allocation reduction: Replace per-value ByteBuffer.slice() /
ByteBuffer.wrap() with bulk reads into reusable buffers
- Bulk vectorized reads: Replace per-element virtual dispatch with single batch
calls backed by System.arraycopy
- JIT-friendly patterns: Per-class method overrides for monomorphic call sites;
avoiding megamorphic profile pollution
Benchmarks run on AMD EPYC 7763 with OpenJDK 17/21/25 via GHA workflow.
GitHub tracking issue: https://github.com/apache/spark/issues/56011
was:
Umbrella tracking a series of performance improvements to the Parquet
vectorized reader in Spark SQL. The changes target allocation reduction,
bulk-read optimizations, and JIT-friendly code patterns across multiple
encoding paths.
All subtasks are independent and can be reviewed/merged in any order. Together
they yield significant throughput gains (1.2x to 9x depending on the encoding
and data shape) for Parquet reads with no user-facing behavioral changes.
Subtasks:
- SPARK-56892: Bulk read optimization for Parquet DELTA_BINARY_PACKED decoding
(up to 9.0x)
- SPARK-56893: Optimize Parquet dictionary decoding with hasNull fast path and
per-class updater overrides (1.24x)
- SPARK-56894: Add vectorized Parquet BYTE_STREAM_SPLIT reader (2.8-4.5x)
- SPARK-56895: Batch ByteBuffer slice in RLE PACKED decode to reduce
allocation overhead (2.1-2.4x)
- SPARK-56896: Add bulk read paths for timestamp/date Parquet vector updaters
(2.0-2.6x)
- SPARK-56897: Reduce per-value allocations in DELTA_BYTE_ARRAY Parquet
decoder (1.1-1.9x)
- SPARK-56907: Reduce per-value allocation in DELTA_LENGTH_BYTE_ARRAY Parquet
vectorized reader (1.2-1.4x)
Common themes:
- Allocation reduction: Replace per-value ByteBuffer.slice() /
ByteBuffer.wrap() with bulk reads into reusable buffers
- Bulk vectorized reads: Replace per-element virtual dispatch with single
batch calls backed by System.arraycopy
- JIT-friendly patterns: Per-class method overrides for monomorphic call
sites; avoiding megamorphic profile pollution
Benchmarks run on AMD EPYC 7763 with OpenJDK 17/21/25 via GHA workflow.
GitHub tracking issue: [https://github.com/apache/spark/issues/56011]
> Parquet vectorized reader performance improvements
> --------------------------------------------------
>
> Key: SPARK-57415
> URL: https://issues.apache.org/jira/browse/SPARK-57415
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
>
> Umbrella tracking a series of performance improvements to the Parquet
> vectorized reader in Spark SQL. The changes target allocation reduction,
> bulk-read optimizations, and JIT-friendly code patterns across multiple
> encoding paths.
> All subtasks are independent and can be reviewed/merged in any order.
> Together they yield significant throughput gains (1.2x to 9x depending on the
> encoding and data shape) for Parquet reads with no user-facing behavioral
> changes.
> Common themes:
> - Allocation reduction: Replace per-value ByteBuffer.slice() /
> ByteBuffer.wrap() with bulk reads into reusable buffers
> - Bulk vectorized reads: Replace per-element virtual dispatch with single
> batch calls backed by System.arraycopy
> - JIT-friendly patterns: Per-class method overrides for monomorphic call
> sites; avoiding megamorphic profile pollution
> Benchmarks run on AMD EPYC 7763 with OpenJDK 17/21/25 via GHA workflow.
> GitHub tracking issue: https://github.com/apache/spark/issues/56011
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
