[
https://issues.apache.org/jira/browse/SPARK-59707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118053#comment-18118053
]
Dongjoon Hyun commented on SPARK-59707:
---------------------------------------
Hi, [~davidmollitordb]. Please set a correct `Affected Version` when you create
a JIRA issue. For `Improvement`, it should be the version of next developer
branch, e.g., branch-4.x. According to the Apache Spark community backporting
policy, it's because the improvement patch cannot affect the release branches
like branch-4.1.
> Use ArrayDeque instead of LinkedList in TransportFrameDecoder
> -------------------------------------------------------------
>
> Key: SPARK-59707
> URL: https://issues.apache.org/jira/browse/SPARK-59707
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.4.0
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> h3. What changes were proposed in this pull request?
> {{TransportFrameDecoder}} sits at the head of the Netty receive pipeline and
> reassembles length-prefixed frames from incoming socket reads. It keeps the
> input {{ByteBuf}}s not yet consumed into a frame in a {{buffers}} field,
> accessed purely as a FIFO queue: appended at the tail on each
> {{channelRead}}, read/removed from the head as frames are decoded, plus
> iteration and {{clear}} on cleanup.
> This changes {{buffers}} from {{LinkedList<ByteBuf>}} to
> {{ArrayDeque<ByteBuf>}}. Every operation used ({{add}}/{{addLast}},
> {{getFirst}}, {{removeFirst}}, enhanced-for iteration, {{clear}}) has
> identical FIFO semantics on {{ArrayDeque}}, so the swap is
> behavior-preserving. The queue never holds nulls (the only insertion is the
> non-null {{ByteBuf}} from {{channelRead}}), so {{ArrayDeque}}'s
> null-rejection is never triggered.
> h3. Why are the changes needed?
> {{LinkedList}} allocates a node object on every {{add}}.
> {{TransportFrameDecoder.channelRead}} runs on every inbound read of every
> connection (RPC and block transfer), so the old code produced a steady stream
> of short-lived list-node allocations on a hot path. {{ArrayDeque}} supports
> the same add-at-tail / poll-at-head access pattern with a single reused
> backing array -- the queue is typically just one buffer -- eliminating the
> per-read node allocation and improving cache locality.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
