[
https://issues.apache.org/jira/browse/HIVE-25534?focusedWorklogId=655037&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-655037
]
ASF GitHub Bot logged work on HIVE-25534:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Sep/21 18:16
Start Date: 24/Sep/21 18:16
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #2650:
URL: https://github.com/apache/hive/pull/2650#discussion_r715805381
##########
File path:
shims/0.23/src/main/test/org/apache/hadoop/hive/shims/TestHadoop23Shims.java
##########
@@ -94,6 +96,96 @@ public void testConstructDistCpParams() {
}
+ @Test
+ public void testXAttrNotPreservedDueToDestFS() throws Exception {
+ FileSystem fsWithXAttrSupport = mock(FileSystem.class);
+ FileSystem fsWithoutXAttrSupport = mock(FileSystem.class);
+ when(fsWithXAttrSupport.getXAttrs(any())).thenReturn(new HashMap<>());
+ when(fsWithoutXAttrSupport.getXAttrs(any())).thenThrow(
+ new UnsupportedOperationException("XAttr not supported for file
system."));
+ Configuration conf = new Configuration();
+ Path copySrc = mock(Path.class);
+ when(copySrc.getFileSystem(any())).thenReturn(fsWithXAttrSupport);
+ Path copyDst = mock(Path.class);
+ when(copyDst.getFileSystem(any())).thenReturn(fsWithoutXAttrSupport);
+
+ Hadoop23Shims shims = new Hadoop23Shims();
+ List<String> paramsDefault =
shims.constructDistCpParams(Collections.singletonList(copySrc), copyDst, conf);
+
+ assertEquals(5, paramsDefault.size());
+ assertTrue("Distcp -pb set by default", paramsDefault.contains("-pb"));
+ assertTrue("Distcp -update set by default",
paramsDefault.contains("-update"));
+ assertTrue("Distcp -delete set by default",
paramsDefault.contains("-delete"));
+ assertEquals(copySrc.toString(), paramsDefault.get(3));
+ assertEquals(copyDst.toString(), paramsDefault.get(4));
+ }
+
+ @Test
+ public void testXAttrNotPreservedDueToSrcFS() throws Exception {
+ FileSystem fsWithXAttrSupport = mock(FileSystem.class);
+ FileSystem fsWithoutXAttrSupport = mock(FileSystem.class);
+ when(fsWithXAttrSupport.getXAttrs(any())).thenReturn(new HashMap<>());
+ when(fsWithoutXAttrSupport.getXAttrs(any())).thenThrow(
+ new UnsupportedOperationException("XAttr not supported for file
system."));
+ Configuration conf = new Configuration();
+ Path copySrc = mock(Path.class);
+ when(copySrc.getFileSystem(any())).thenReturn(fsWithoutXAttrSupport);
+ Path copyDst = mock(Path.class);
+ when(copyDst.getFileSystem(any())).thenReturn(fsWithXAttrSupport);
Review comment:
Whole mocking can be moved to a method and reused. We should avoid
duplication of code.
##########
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -1192,12 +1204,15 @@ public Boolean run() throws Exception {
@Override
public boolean runDistCp(List<Path> srcPaths, Path dst, Configuration conf)
throws IOException {
- DistCpOptions options = new DistCpOptions.Builder(srcPaths, dst)
- .withSyncFolder(true)
- .withDeleteMissing(true)
- .preserve(FileAttribute.BLOCKSIZE)
- .preserve(FileAttribute.XATTR)
- .build();
+ DistCpOptions.Builder builder = new DistCpOptions.Builder(srcPaths, dst)
+ .withSyncFolder(true)
+ .withDeleteMissing(true)
+ .preserve(FileAttribute.BLOCKSIZE);
+ if (checkFileSystemXAttrSupport(dst.getFileSystem(conf))
+ &&
checkFileSystemXAttrSupport(srcPaths.get(0).getFileSystem(conf))) {
+ builder = builder.preserve(FileAttribute.XATTR);
+ }
+ DistCpOptions options = builder.build();
Review comment:
What is the difference between option and params returned by
constructDistCpParams? Why are we passing preserve attribute in both. Can one
not suffice?
##########
File path:
shims/0.23/src/main/test/org/apache/hadoop/hive/shims/TestHadoop23Shims.java
##########
@@ -94,6 +100,70 @@ public void testConstructDistCpParams() {
}
+ @Test
+ public void testXAttrNotPreservedDueToDestFS() throws Exception {
+ Configuration conf = new Configuration();
+ Path copySrc = mock(Path.class);
+ FileSystem srcFs = new
MiniDFSCluster.Builder(conf).numDataNodes(1).build().getFileSystem();
+ assertTrue("srcFs supports XAttrs.", srcFs.getXAttrs(new
Path(Path.SEPARATOR)) != null);
Review comment:
What type of FS is this here in case of MiniHdfs? Why is this important.
Also, could n't we have used same interface checkFileSystemXAttrSupport()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 655037)
Time Spent: 1h 40m (was: 1.5h)
> Error when executing DistCp on file system not supporting XAttrs
> ----------------------------------------------------------------
>
> Key: HIVE-25534
> URL: https://issues.apache.org/jira/browse/HIVE-25534
> Project: Hive
> Issue Type: Bug
> Reporter: Haymant Mangla
> Assignee: Haymant Mangla
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Remove the preserve xattr while calling distcp.
> {code:java}
> 2021-08-23 10:06:18,485 ERROR org.apache.hadoop.tools.DistCp:
> [HiveServer2-Background-Pool: Thread-73]: XAttrs not supported on at least
> one file system:
> org.apache.hadoop.tools.CopyListing$XAttrsNotSupportedException: XAttrs not
> supported for file system: s3a://hmangla1-dev
> at
> org.apache.hadoop.tools.util.DistCpUtils.checkFileSystemXAttrSupport(DistCpUtils.java:513)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:337)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:304)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:214)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
> ~[hadoop-distcp-3.1.1.7.1.6.0-297.jar:?]{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
