yinsong created HDFS-14190:
------------------------------
Summary: Copying folders containing = - characters between hdfs
(using webhdfs) does not work in distcp
Key: HDFS-14190
URL: https://issues.apache.org/jira/browse/HDFS-14190
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Affects Versions: 3.1.1
Reporter: yinsong
Copying folders containing = - characters between hdfs (using webhdfs) does not
work in distcp
for example:
src:hadoop2.7 target:hadoop3.1.1
(1)
hadoop distcp \
-pugp \
-i \
webhdfs://1.1.1.1:50070/sudiyi_datawarehouse
webhdfs://2.2.2.2:50070/sudiyi_datawarehouse
ERROR tools.SimpleCopyListing: FileNotFoundException exception in listStatus:
File /sudiyi_datawarehouse/st_device_standard_ds/date_time%3D2018-10-10 does
not exist
(2)
hadoop distcp \
-Dmapreduce.framework.name=yarn \
-pugp \
-i \
webhdfs://1.1.1.1:50070/druid webhdfs://2.2.2.2:50070/druid
Error: java.io.IOException: File copy failed:
webhdfs://10.26.93.65:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800
-->
webhdfs://10.27.234.198:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:259)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:217)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying
webhdfs://10.26.93.65:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800
to
webhdfs://10.27.234.198:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:256)
... 10 more
Caused by: java.io.IOException: Failed to promote
tmp-file:webhdfs://10.27.234.198:50070/druid/.distcp.tmp.attempt_1545990837043_0016_m_000015_2
to:
webhdfs://10.27.234.198:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:250)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:140)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
