[
https://issues.apache.org/jira/browse/SPARK-47702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-47702.
---------------------------------
> Shuffle service endpoint is not removed from the locations list when RDD
> block is removed form a node.
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-47702
> URL: https://issues.apache.org/jira/browse/SPARK-47702
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.5.1
> Reporter: mahesh kumar behera
> Assignee: Attila Zsolt Piros
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.4, 4.0.0
>
>
> If SHUFFLE_SERVICE_FETCH_RDD_ENABLED is set to true, driver stores both
> executor end point and the external shuffle end points for a RDD block. When
> the RDD is migrated, the location info is updated to add the end point
> corresponds to new location and the old end point is removed. But currently,
> only the executor end point is removed. The shuffle service end point is not
> removed. This cause failure during RDD read if the shuffle service end point
> is chosen due to task locality.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
