[
https://issues.apache.org/jira/browse/IMPALA-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yida Wu updated IMPALA-13677:
-----------------------------
Description:
Currently, when an executor spills data to a remote storage, scratch files
would remain in the remote storage if the executor exits abnormally or is
terminated after the graceful shutdown deadline.
Immediate removal may be challenging, and no concrete solution is currently
available. However, we may consider adding an additional thread in the
coordinator to manage the cleanup of leftover scratch files in remote storage
or consider alternative methods to ensure their safe and complete removal.
Since s3 is the most common scenario, this task may specifically focus on
handling the leftover scratch files in s3.
was:
Currently, when an executor spills data to a remote storage, scratch files
would remain in the remote storage if the executor exits abnormally or is
terminated after the graceful shutdown deadline.
Immediate removal might be challenging, but we could consider adding an extra
thread in the coordinator to handle cleanup of leftover scratch files in remote
storage. Since s3 is the most common scenario, this task may specifically focus
on handling the leftover scratch files in s3.
> Cleanup of s3 scratch files on abnormal executor exit
> -----------------------------------------------------
>
> Key: IMPALA-13677
> URL: https://issues.apache.org/jira/browse/IMPALA-13677
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Major
>
> Currently, when an executor spills data to a remote storage, scratch files
> would remain in the remote storage if the executor exits abnormally or is
> terminated after the graceful shutdown deadline.
> Immediate removal may be challenging, and no concrete solution is currently
> available. However, we may consider adding an additional thread in the
> coordinator to manage the cleanup of leftover scratch files in remote storage
> or consider alternative methods to ensure their safe and complete removal.
> Since s3 is the most common scenario, this task may specifically focus on
> handling the leftover scratch files in s3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
