[jira] [Created] (HDFS-10900) DiskBalancer: Complete the documents for the report command
Yiqun Lin created HDFS-10900:
Summary: DiskBalancer: Complete the documents for the report
command
Key: HDFS-10900
URL: https://issues.apache.org/jira/browse/HDFS-10900
Project: Hadoop HDFS
Issue Type: Sub-task
Components: documentation
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor
Now the documemnts of the command {{hdfs diskbalancer -report}} look not
completed. Two minors:
* The usage of {{hdfs diskbalancer -report}} is missing in {{HDFSCommands.md}}
* One subcommand of report command {{hdfs diskbalancer -report -top}} is
missing in {{HDFSDiskBalancer.md}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/ [Sep 24, 2016 5:51:03 PM] (naganarasimha_gr) YARN-5664. Fix Yarn documentation to link to correct versions. [Sep 25, 2016 5:32:46 AM] (brahma) HDFS-10869. Remove the unused method InodeId#checkId(). Contributed by -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/175/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/ [Sep 24, 2016 5:51:03 PM] (naganarasimha_gr) YARN-5664. Fix Yarn documentation to link to correct versions. [Sep 25, 2016 5:32:46 AM] (brahma) HDFS-10869. Remove the unused method InodeId#checkId(). Contributed by [Sep 25, 2016 12:06:30 PM] (naganarasimha_gr) YARN-3877. YarnClientImpl.submitApplication swallows exceptions. -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.ipc.TestRPCWaitForProxy hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.TestCrcCorruption hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestNMClient hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestMROpportunisticMaps compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/artifact/out/patch-compile-root.txt [308K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/artifact/out/patch-compile-root.txt [308K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/artifact/out/patch-compile-root.txt [308K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [120K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [440K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt [20K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/105/artifact/out/patch-unit-hadoop-yarn-project_hadoop-
[jira] [Created] (HDFS-10902) QJM should not consider stale/failed txn available in any one of JNs.
Vinayakumar B created HDFS-10902: Summary: QJM should not consider stale/failed txn available in any one of JNs. Key: HDFS-10902 URL: https://issues.apache.org/jira/browse/HDFS-10902 Project: Hadoop HDFS Issue Type: Bug Components: qjm Reporter: Vinayakumar B Assignee: Vinayakumar B Priority: Critical In one of our cluster faced an issue, where NameNode restart failed due to a stale/failed txn available in one JN but not others. Scenario is: 1. Full cluster restart 2. startLogSegment Txn(195222) synced in Only one JN but failed to others, because they were shutting down. Only editlog file was created but txn was not synced in others, so after restart they were marked as empty. 3. Cluster restarted. During failover, this new logSegment missed the recovery because this JN was slow in responding to this call. 4. Other JNs recover was successfull, as there was no in-progress files. 5. editlog.openForWrite() detected that (195222) was already available, and failed the failover. Same steps repeated until that stale editlog in JN was manually deleted. Since QJM is a quorum of JNs, txn is considered successfull, if its written min quorum. Otherwise it will be failed. So, same case should be applied while selecting streams for reading also. Stale/failed txns available in only less JNs should not be considered for reading. HDFS-10519, does similar work to consider 'durable' txns based on 'committedTxnId'. But updating 'committedTxnId' for every flush with one more RPC seems tobe problematic to performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10901) QJM should not consider stale/failed txn available in any one of JNs.
Vinayakumar B created HDFS-10901: Summary: QJM should not consider stale/failed txn available in any one of JNs. Key: HDFS-10901 URL: https://issues.apache.org/jira/browse/HDFS-10901 Project: Hadoop HDFS Issue Type: Bug Components: qjm Reporter: Vinayakumar B Assignee: Vinayakumar B Priority: Critical In one of our cluster faced an issue, where NameNode restart failed due to a stale/failed txn available in one JN but not others. Scenario is: 1. Full cluster restart 2. startLogSegment Txn(195222) synced in Only one JN but failed to others, because they were shutting down. Only editlog file was created but txn was not synced in others, so after restart they were marked as empty. 3. Cluster restarted. During failover, this new logSegment missed the recovery because this JN was slow in responding to this call. 4. Other JNs recover was successfull, as there was no in-progress files. 5. editlog.openForWrite() detected that (195222) was already available, and failed the failover. Same steps repeated until that stale editlog in JN was manually deleted. Since QJM is a quorum of JNs, txn is considered successfull, if its written min quorum. Otherwise it will be failed. So, same case should be applied while selecting streams for reading also. Stale/failed txns available in only less JNs should not be considered for reading. HDFS-10519, does similar work to consider 'durable' txns based on 'committedTxnId'. But updating 'committedTxnId' for every flush with one more RPC seems tobe problematic to performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
