Josh Rosen created SPARK-17547:
----------------------------------
Summary: Temporary shuffle data files may be leaked following
exception in write
Key: SPARK-17547
URL: https://issues.apache.org/jira/browse/SPARK-17547
Project: Spark
Issue Type: Bug
Components: Shuffle
Affects Versions: 2.0.0, 1.6.0, 1.5.3
Reporter: Josh Rosen
Assignee: Josh Rosen
SPARK-8029 modified shuffle writers to first stage their data to a temporary
file in the same directory as the final destination file and then to atomically
rename the file at the end of the write job. However, this change introduced
the potential for the temporary output file to be leaked if an exception occurs
during the write because the shuffle writers' existing error cleanup code
doesn't handle this new temp file.
This is easy to fix: we just need to add a {{finally}} block to ensure that the
temporary file is guaranteed to be either moved or deleted before existing the
shuffle write method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
