David Mollitor created SPARK-59599:
--------------------------------------
Summary: Pre-size the ArrowConverters.serializeBatch output buffer
to avoid repeated reallocation
Key: SPARK-59599
URL: https://issues.apache.org/jira/browse/SPARK-59599
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h3. Problem
{{ArrowConverters.serializeBatch}} serializes a single {{ArrowRecordBatch}}
into a {{byte[]}} through a {{ByteArrayOutputStream}} created with no initial
capacity:
{code:scala}
val out = new ByteArrayOutputStream()
val writeChannel = new WriteChannel(Channels.newChannel(out))
MessageSerializer.serialize(writeChannel, batch)
out.toByteArray
{code}
A no-arg {{ByteArrayOutputStream}} starts at the JDK default of 32 bytes and
grows by repeatedly doubling its backing array ({{{}grow{}}} +
{{{}Arrays.copyOf{}}}).
The batch is written {*}incrementally{*}, not as one block:
{{MessageSerializer.serialize}} writes the message metadata and then each of
the batch's buffers (with alignment padding) individually, and the
{{WritableByteChannel}} returned by {{Channels.newChannel}} further splits
every write into chunks of at most 8 KB. So a 32-byte buffer reallocates and
copies its contents on the order of {{log2(batchSize / 32)}} times per batch,
producing that many short-lived intermediate arrays (GC pressure) plus the
copies.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
