David Mollitor created SPARK-59619:
--------------------------------------
Summary: Pre-size the RequestMessage serialization buffer in
NettyRpcEnv
Key: SPARK-59619
URL: https://issues.apache.org/jira/browse/SPARK-59619
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.0
Reporter: David Mollitor
h3. Description
{{RequestMessage.serialize}} ({{{}core/.../rpc/netty/NettyRpcEnv.scala{}}}) is
called for every outgoing Netty RPC ({{{}send{}}} / {{{}ask{}}}). It allocates
its output buffer with the no-arg {{{}new ByteBufferOutputStream(){}}}, which
starts at the JDK {{ByteArrayOutputStream}} default of *32 bytes* and grows by
doubling (reallocate + {{{}Arrays.copyOf{}}}) as bytes are written.
Every RPC first writes a fixed preamble through an _unbuffered_
{{DataOutputStream}} – two {{{}RpcAddress{}}}'es (each {{{}boolean + UTF host +
int port{}}}) and the endpoint name – and then the Java-serialized message
body. Measuring the real serialized sizes (through the actual
{{{}JavaSerializer{}}}, byte-identical to {{{}serialize{}}}):
||Message||Bytes||
|preamble only (sender set / null)|60 / 43|
|{{HeartbeatResponse(false)}}|136|
|{{UpdateBlockInfo}}|205|
|{{ReviveOffers}}|247|
|{{GetLocations}}|263|
|{{BlockManagerHeartbeat}}|291|
|{{Heartbeat}} (empty accumulators)|637|
The preamble alone (43-60 bytes) already exceeds the 32-byte default, so
*every* RPC pays at least three reallocations (32 -> 64 -> 128 -> 256) before
the buffer holds even the smallest message.
Pre-size the buffer to {*}512 bytes{*}. 512 is the capacity the doubling growth
already reaches for the smallest/most-frequent messages, so for those it
replaces a multi-step grow-and-copy chain (and the dead intermediate arrays)
with a single right-sized allocation at effectively no change in retained
memory, and it removes the bulk of the reallocations for the mid-sized control
messages. This is an allocation-churn / GC reduction on a hot control-plane
path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
