[
https://issues.apache.org/jira/browse/SPARK-59599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59599:
-----------------------------------
Labels: pull-request-available (was: )
> Pre-size the ArrowConverters.serializeBatch output buffer to avoid repeated
> reallocation
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-59599
> URL: https://issues.apache.org/jira/browse/SPARK-59599
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h3. Problem
> {{ArrowConverters.serializeBatch}} serializes a single {{ArrowRecordBatch}}
> into a {{byte[]}} through a {{ByteArrayOutputStream}} created with no initial
> capacity:
> {code:scala}
> val out = new ByteArrayOutputStream()
> val writeChannel = new WriteChannel(Channels.newChannel(out))
> MessageSerializer.serialize(writeChannel, batch)
> out.toByteArray
> {code}
> A no-arg {{ByteArrayOutputStream}} starts at the JDK default of 32 bytes and
> grows by repeatedly doubling its backing array ({{{}grow{}}} +
> {{{}Arrays.copyOf{}}}).
> The batch is written {*}incrementally{*}, not as one block:
> {{MessageSerializer.serialize}} writes the message metadata and then each of
> the batch's buffers (with alignment padding) individually, and the
> {{WritableByteChannel}} returned by {{Channels.newChannel}} further splits
> every write into chunks of at most 8 KB. So a 32-byte buffer reallocates and
> copies its contents on the order of {{log2(batchSize / 32)}} times per batch,
> producing that many short-lived intermediate arrays (GC pressure) plus the
> copies.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
