[
https://issues.apache.org/jira/browse/SPARK-56907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated SPARK-56907:
---------------------------------
Description:
Reduces object allocation in the DELTA_LENGTH_BYTE_ARRAY vectorized Parquet
reader ({{VectorizedDeltaLengthByteArrayReader}}) by applying three targeted
changes:
* *readBinary*: Replace per-value {{in.slice(length)}} (one ByteBuffer
allocation per value) with a single bulk {{in.slice(totalDataLen)}} that reads
the entire batch at once. Individual values are then written to the column
vector via {{putByteArray}} from the shared backing array.
* *skipBinary*: Replace the per-value skip loop (N separate {{in.skip()}}
calls) with a single bulk skip by summing all value lengths upfront.
* *readGeoData*: Remove the {{ByteBuffer.wrap()}} + {{ByteBufferOutputWriter}}
indirection per value and call {{putByteArray}} directly.
h3. Benchmark Results (GHA, AMD EPYC 7763, PR vs committed upstream baseline,
Best Time in ms, speedup = base/PR)
*skipBinary* (consistent win):
||Case||JDK 17||JDK 21||JDK 25||
|payloadLen=8|2.33x|1.75x|2.00x|
|payloadLen=32|2.33x|1.75x|2.00x|
|payloadLen=128|2.33x|1.75x|2.00x|
|payloadLen=512|2.33x|1.75x|2.00x|
*readBinary* (allocation-bound, wall-clock roughly neutral / within noise):
||Case||JDK 17||JDK 21||JDK 25||
|payloadLen=8|1.13x|0.94x|1.14x|
|payloadLen=32|1.13x|0.94x|1.07x|
|payloadLen=128|1.06x|0.79x|1.00x|
|payloadLen=512|1.05x|0.95x|1.00x|
{{skipBinary}} is the primary measurable win (1.75x-2.33x across all
sizes/JDKs) from replacing N per-value stream skips with a single bulk skip.
{{readBinary}} removes N-1 ByteBuffer allocations per batch (reduced GC
pressure), but Best Time is roughly neutral and within run-to-run variance at
these sizes (small gains on JDK 17/25, noise-level dips on JDK 21; 16-24 ms
measurements at 1 ms integer granularity).
Full committed results: JDK 17
https://github.com/iemejia/spark/actions/runs/31322679242 , JDK 21
https://github.com/iemejia/spark/actions/runs/31322682640 , JDK 25
https://github.com/iemejia/spark/actions/runs/31322686206
PR: https://github.com/apache/spark/pull/55932
Parent issue: https://github.com/apache/spark/issues/56011
was:
Reduces object allocation in the DELTA_LENGTH_BYTE_ARRAY vectorized Parquet
reader ({{VectorizedDeltaLengthByteArrayReader}}) by applying three targeted
changes:
* *readBinary*: Replace per-value {{in.slice(length)}} (one ByteBuffer
allocation per value) with a single bulk {{in.slice(totalDataLen)}} that reads
the entire batch at once. Individual values are then written to the column
vector via {{putByteArray}} from the shared backing array.
* *skipBinary*: Replace the per-value skip loop (N separate {{in.skip()}}
calls) with a single bulk skip by summing all value lengths upfront.
* *readGeoData*: Remove the {{ByteBuffer.wrap()}} + {{ByteBufferOutputWriter}}
indirection per value and call {{putByteArray}} directly.
h3. Benchmark Results (GHA, AMD EPYC 7763)
||Case||JDK 17||JDK 21||JDK 25||
|readBinary, payloadLen=8|1.14x|1.09x|1.18x|
|readBinary, payloadLen=32|1.10x|0.84x|1.17x|
|skipBinary, payloadLen=8|1.42x|1.69x|1.39x|
|skipBinary, payloadLen=32|1.40x|1.02x|1.34x|
{{readBinary}} speedup is larger for small payloads where allocation cost
dominates. {{skipBinary}} shows consistent improvement from eliminating
per-value stream operations.
PR: https://github.com/apache/spark/pull/55932
Parent issue: https://github.com/apache/spark/issues/56011
> Reduce per-value allocation in DELTA_LENGTH_BYTE_ARRAY Parquet vectorized
> reader
> --------------------------------------------------------------------------------
>
> Key: SPARK-56907
> URL: https://issues.apache.org/jira/browse/SPARK-56907
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
>
> Reduces object allocation in the DELTA_LENGTH_BYTE_ARRAY vectorized Parquet
> reader ({{VectorizedDeltaLengthByteArrayReader}}) by applying three targeted
> changes:
> * *readBinary*: Replace per-value {{in.slice(length)}} (one ByteBuffer
> allocation per value) with a single bulk {{in.slice(totalDataLen)}} that
> reads the entire batch at once. Individual values are then written to the
> column vector via {{putByteArray}} from the shared backing array.
> * *skipBinary*: Replace the per-value skip loop (N separate {{in.skip()}}
> calls) with a single bulk skip by summing all value lengths upfront.
> * *readGeoData*: Remove the {{ByteBuffer.wrap()}} +
> {{ByteBufferOutputWriter}} indirection per value and call {{putByteArray}}
> directly.
> h3. Benchmark Results (GHA, AMD EPYC 7763, PR vs committed upstream baseline,
> Best Time in ms, speedup = base/PR)
> *skipBinary* (consistent win):
> ||Case||JDK 17||JDK 21||JDK 25||
> |payloadLen=8|2.33x|1.75x|2.00x|
> |payloadLen=32|2.33x|1.75x|2.00x|
> |payloadLen=128|2.33x|1.75x|2.00x|
> |payloadLen=512|2.33x|1.75x|2.00x|
> *readBinary* (allocation-bound, wall-clock roughly neutral / within noise):
> ||Case||JDK 17||JDK 21||JDK 25||
> |payloadLen=8|1.13x|0.94x|1.14x|
> |payloadLen=32|1.13x|0.94x|1.07x|
> |payloadLen=128|1.06x|0.79x|1.00x|
> |payloadLen=512|1.05x|0.95x|1.00x|
> {{skipBinary}} is the primary measurable win (1.75x-2.33x across all
> sizes/JDKs) from replacing N per-value stream skips with a single bulk skip.
> {{readBinary}} removes N-1 ByteBuffer allocations per batch (reduced GC
> pressure), but Best Time is roughly neutral and within run-to-run variance at
> these sizes (small gains on JDK 17/25, noise-level dips on JDK 21; 16-24 ms
> measurements at 1 ms integer granularity).
> Full committed results: JDK 17
> https://github.com/iemejia/spark/actions/runs/31322679242 , JDK 21
> https://github.com/iemejia/spark/actions/runs/31322682640 , JDK 25
> https://github.com/iemejia/spark/actions/runs/31322686206
> PR: https://github.com/apache/spark/pull/55932
> Parent issue: https://github.com/apache/spark/issues/56011
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
