Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20386
to look at the new patch set (#10).
Change subject: IMPALA-12389: Use -skipTrash for
HadoopFsCommandLineClient::delete_file_dir
......................................................................
IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir
The default behavior for deleting files on Hadoop is to
move them to a trash folder. The trash folder can be
aged out, but Impala's developer environment sets the
trash to live forever. This is a problem, because the
trash contents will continue to accumulate.
This changes HadoopFsCommandLineClient's delete_file_dir
to use -skipTrash to avoid accumulating the trash for
this case. It also modifies DelegatingHdfsClient to use
HadoopFsCommandLineClient for delete_file_dir. The WebHDFS
client doesn't have the option to skip trash.
TestHdfsEncryption and TestHdfsPermissions used WebHDFS-style
paths without the leading slash. This is incompatible with
using the HDFS commandline for delete_file_dir, so it switches
those tests to normal paths. This should be safe, because we
always use the delegating client which removes slashes when
it uses the WebHDFS client.
This relaxes the timing for TestRecursiveListing, because
deletes via the Hadoop commandline are slower than deletes
through the WebHDFS client.
This changes cleanup_database to check if the directory exists
before invoking delete_file_dir(). This may reduce the overhead
of this change.
Testing:
- Ran tests locally and examined the trash directory
Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8
---
M tests/conftest.py
M tests/metadata/test_hdfs_encryption.py
M tests/metadata/test_hdfs_permissions.py
M tests/metadata/test_recursive_listing.py
M tests/util/hdfs_util.py
5 files changed, 20 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20386/10
--
To view, visit http://gerrit.cloudera.org:8080/20386
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8
Gerrit-Change-Number: 20386
Gerrit-PatchSet: 10
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
