[jira] [Created] (HDFS-7994) Detect if resevered EC Block ID is already used
Tsz Wo Nicholas Sze created HDFS-7994: - Summary: Detect if resevered EC Block ID is already used Key: HDFS-7994 URL: https://issues.apache.org/jira/browse/HDFS-7994 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Since random block IDs were supported by some early version of HDFS, the block ID reserved for EC blocks could be already used by some existing blocks in a cluster. During NameNode startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap lookup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Hadoop-Hdfs-trunk - Build # 2076 - Still Failing
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2076/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 8437 lines...] main: [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/target/test-dir [INFO] Executed tasks [INFO] [INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-hdfs-project --- [INFO] Skipping javadoc generation [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-checkstyle-plugin:2.12.1:checkstyle (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop HDFS FAILURE [ 02:56 h] [INFO] Apache Hadoop HttpFS .. SKIPPED [INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED [INFO] Apache Hadoop HDFS-NFS SKIPPED [INFO] Apache Hadoop HDFS Project SUCCESS [ 2.194 s] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:56 h [INFO] Finished at: 2015-03-26T14:30:17+00:00 [INFO] Final Memory: 58M/620M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-hdfs: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results Updating HDFS-7928 Updating HADOOP-11719 Updating HADOOP-11524 Updating HADOOP-11724 Updating HADOOP-10670 Updating YARN-3397 Updating HDFS-6353 Updating YARN-2213 Sending e-mails to: hdfs-dev@hadoop.apache.org Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.hadoop.tracing.TestTracing.testWriteTraceHooks Error Message: timed out to get expected spans: Timed out waiting for condition. Thread diagnostics: Timestamp: 2015-03-26 02:24:08,706 "IPC Server handler 8 on 60098" daemon prio=5 tid=42 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2110) "881201469@qtp-296904196-1 - Acceptor0 SelectChannelConnector@localhost:33091" daemon prio=5 tid=54 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(A
Hadoop-Hdfs-trunk-Java8 - Build # 135 - Still Failing
See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/135/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 8917 lines...] main: [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/target/test-dir [INFO] Executed tasks [INFO] [INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-hdfs-project --- [INFO] Not executing Javadoc as the project is not a Java classpath-capable package [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-checkstyle-plugin:2.12.1:checkstyle (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop HDFS FAILURE [ 02:56 h] [INFO] Apache Hadoop HttpFS .. SKIPPED [INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED [INFO] Apache Hadoop HDFS-NFS SKIPPED [INFO] Apache Hadoop HDFS Project SUCCESS [ 1.789 s] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:56 h [INFO] Finished at: 2015-03-26T14:30:43+00:00 [INFO] Final Memory: 49M/238M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-hdfs: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results Updating HDFS-7928 Updating HADOOP-11719 Updating HADOOP-11524 Updating HADOOP-11724 Updating HADOOP-10670 Updating YARN-3397 Updating HDFS-6353 Updating YARN-2213 Sending e-mails to: hdfs-dev@hadoop.apache.org Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 2 tests failed. REGRESSION: org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline Error Message: After waiting the operation updatePipeline still has not taken effect on NN yet Stack Trace: java.lang.AssertionError: After waiting the operation updatePipeline still has not taken effect on NN yet at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1280) at org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline(TestRetryCacheWithHA.java:1178) FAILED: org.apache.hadoop.tracing.TestTracing.testWriteTraceHooks Error Message: timed out to get expected spans: Timed out waiting for condition. Thread diagnostics: Timestamp: 2015-03-26 02:30:35,576 "IPC Server handler 7 on 54648" daemon prio=5 tid=320 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2110) "refreshUsed-/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/had
[jira] [Resolved] (HDFS-7965) TestTracing#testWriteTraceHooks is flaky
[ https://issues.apache.org/jira/browse/HDFS-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HDFS-7965. Resolution: Duplicate > TestTracing#testWriteTraceHooks is flaky > > > Key: HDFS-7965 > URL: https://issues.apache.org/jira/browse/HDFS-7965 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.7.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > > TestTracing#testWriteTraceHooks is flaky. It seems to be timing out in trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7995) Implement chmod in the HDFS Web UI
Ravi Prakash created HDFS-7995: -- Summary: Implement chmod in the HDFS Web UI Key: HDFS-7995 URL: https://issues.apache.org/jira/browse/HDFS-7995 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ravi Prakash Assignee: Ravi Prakash We should let users change the permissions of files and directories using the HDFS Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7996) After swapping a volume, BlockReceiver reports ReplicaNotFoundException
Lei (Eddy) Xu created HDFS-7996:
---
Summary: After swapping a volume, BlockReceiver reports
ReplicaNotFoundException
Key: HDFS-7996
URL: https://issues.apache.org/jira/browse/HDFS-7996
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Critical
When removing a disk from an actively writing DataNode, the BlockReceiver
working on the disk throws {{ReplicaNotFoundException}} because the replicas
are removed from the memory:
{code}
2015-03-26 08:02:43,154 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removed
volume: /data/2/dfs/dn/current
2015-03-26 08:02:43,163 INFO org.apache.hadoop.hdfs.server.common.Storage:
Removing block level storage:
/data/2/dfs/dn/current/BP-51301509-10.20.202.114-1427296597742
2015-03-26 08:02:43,163 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
IOException in BlockReceiver.run():
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Cannot append
to a non-existent replica
BP-51301509-10.20.202.114-1427296597742:blk_1073742979_2160
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getReplicaInfo(FsDatasetImpl.java:615)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:1362)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.finalizeBlock(BlockReceiver.java:1281)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1241)
at java.lang.Thread.run(Thread.java:745)
{code}
{{FsVolumeList#removeVolume}} waits all threads release {{FsVolumeReference}}
on the volume to be removed, however, in {{PacketResponder#finalizeBlock()}},
it calls
{code}
private void finalizeBlock(long startTime) throws IOException {
BlockReceiver.this.close();
final long endTime = ClientTraceLog.isInfoEnabled() ? System.nanoTime()
: 0;
block.setNumBytes(replicaInfo.getNumBytes());
datanode.data.finalizeBlock(block);
{code}
The {{FsVolumeReference}} was released in {{BlockReceiver.this.close()}} before
calling {{datanode.data.finalizeBlock(block)}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-3668) XmlEditsVisitor should not use classes in com.sun.org.apache.xml.internal.serialize
[ https://issues.apache.org/jira/browse/HDFS-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai resolved HDFS-3668. -- Resolution: Not a Problem > XmlEditsVisitor should not use classes in > com.sun.org.apache.xml.internal.serialize > --- > > Key: HDFS-3668 > URL: https://issues.apache.org/jira/browse/HDFS-3668 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Tsz Wo Nicholas Sze > > These classes are not public library. It also generates a lot of "Access > restriction" javac warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
