David Mollitor created SPARK-59707:
--------------------------------------
Summary: Use ArrayDeque instead of LinkedList in
TransportFrameDecoder
Key: SPARK-59707
URL: https://issues.apache.org/jira/browse/SPARK-59707
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.0
Reporter: David Mollitor
h3. What changes were proposed in this pull request?
{{TransportFrameDecoder}} sits at the head of the Netty receive pipeline and
reassembles length-prefixed frames from incoming socket reads. It keeps the
input {{ByteBuf}}s not yet consumed into a frame in a {{buffers}} field,
accessed purely as a FIFO queue: appended at the tail on each {{channelRead}},
read/removed from the head as frames are decoded, plus iteration and {{clear}}
on cleanup.
This changes {{buffers}} from {{LinkedList<ByteBuf>}} to
{{ArrayDeque<ByteBuf>}}. Every operation used ({{add}}/{{addLast}},
{{getFirst}}, {{removeFirst}}, enhanced-for iteration, {{clear}}) has identical
FIFO semantics on {{ArrayDeque}}, so the swap is behavior-preserving. The queue
never holds nulls (the only insertion is the non-null {{ByteBuf}} from
{{channelRead}}), so {{ArrayDeque}}'s null-rejection is never triggered.
h3. Why are the changes needed?
{{LinkedList}} allocates a node object on every {{add}}.
{{TransportFrameDecoder.channelRead}} runs on every inbound read of every
connection (RPC and block transfer), so the old code produced a steady stream
of short-lived list-node allocations on a hot path. {{ArrayDeque}} supports the
same
add-at-tail / poll-at-head access pattern with a single reused backing array --
the queue is typically just one buffer -- eliminating the per-read node
allocation and improving cache locality.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
