[
https://issues.apache.org/jira/browse/IMPALA-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yida Wu updated IMPALA-13677:
-----------------------------
Description:
Currently, when an executor spills data to a remote storage, scratch files
would remain in the remote storage if the executor exits abnormally or is
terminated after the graceful shutdown deadline.
Immediate removal may be challenging, and no concrete solution is currently
available. However, to clear leftover files on startup is possible.
was:
Currently, when an executor spills data to a remote storage, scratch files
would remain in the remote storage if the executor exits abnormally or is
terminated after the graceful shutdown deadline.
Immediate removal may be challenging, and no concrete solution is currently
available. However, we may consider adding an additional thread in the
coordinator to manage the cleanup of leftover scratch files in remote storage
or consider alternative methods to ensure their safe and complete removal.
Since s3 is the most common scenario, this task may specifically focus on
handling the leftover scratch files in s3.
> Cleanup of s3 scratch files on abnormal executor exit
> -----------------------------------------------------
>
> Key: IMPALA-13677
> URL: https://issues.apache.org/jira/browse/IMPALA-13677
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Critical
>
> Currently, when an executor spills data to a remote storage, scratch files
> would remain in the remote storage if the executor exits abnormally or is
> terminated after the graceful shutdown deadline.
> Immediate removal may be challenging, and no concrete solution is currently
> available. However, to clear leftover files on startup is possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
