[
https://issues.apache.org/jira/browse/SPARK-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-17547:
------------------------------------
Assignee: Josh Rosen (was: Apache Spark)
> Temporary shuffle data files may be leaked following exception in write
> -----------------------------------------------------------------------
>
> Key: SPARK-17547
> URL: https://issues.apache.org/jira/browse/SPARK-17547
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 1.5.3, 1.6.0, 2.0.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
>
> SPARK-8029 modified shuffle writers to first stage their data to a temporary
> file in the same directory as the final destination file and then to
> atomically rename the file at the end of the write job. However, this change
> introduced the potential for the temporary output file to be leaked if an
> exception occurs during the write because the shuffle writers' existing error
> cleanup code doesn't handle this new temp file.
> This is easy to fix: we just need to add a {{finally}} block to ensure that
> the temporary file is guaranteed to be either moved or deleted before
> existing the shuffle write method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
