Yida Wu has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22549 )
Change subject: IMPALA-13798: Cleanup host-level remote scratch dir on shutdown ...................................................................... IMPALA-13798: Cleanup host-level remote scratch dir on shutdown IMPALA-13677 introduces cleanup of remote scratch files on startup, and also introduces a host-level directory to do easier cleanup. An empty host-level directory in remote storage does not use resources, but it can stay there forever if a host goes offline permanently or hostname changes. To improve this behavior, this patch adds support for removing the remote host-level temporary directory on shutdown. This helps prevent too many empty directories left in the remote scratch path, especially for hosts that never restart. Changed the flag remote_scratch_cleanup_on_startup to remote_scratch_cleanup_on_start_stop, so that this flag also controls cleanup on shutdown. Tests: Passed exhaustive tests. Added an e2e testcase test_scratch_dirs_remote_dir_removal_on_shutdown. Change-Id: Ic8f446894afdf975630aef80a9d964a9a78d3b46 Reviewed-on: http://gerrit.cloudera.org:8080/22549 Reviewed-by: Daniel Becker <[email protected]> Reviewed-by: Abhishek Rawat <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/service/impala-server.cc M tests/custom_cluster/test_scratch_disk.py 6 files changed, 87 insertions(+), 10 deletions(-) Approvals: Daniel Becker: Looks good to me, but someone else must approve Abhishek Rawat: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/22549 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic8f446894afdf975630aef80a9d964a9a78d3b46 Gerrit-Change-Number: 22549 Gerrit-PatchSet: 4 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]>
