[jira] [Created] (HDFS-12548) Jenkins build is unstable on branch-2
Rushabh S Shah created HDFS-12548:
-
Summary: Jenkins build is unstable on branch-2
Key: HDFS-12548
URL: https://issues.apache.org/jira/browse/HDFS-12548
Project: Hadoop HDFS
Issue Type: Bug
Components: build
Affects Versions: 2.9.0
Reporter: Rushabh S Shah
Priority: Critical
Feel free move the ticket to another project (e.g. infra).
Recently I attached branch-2 patch while working on one jira
[HDFS-12386|https://issues.apache.org/jira/browse/HDFS-12386?focusedCommentId=16180676&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16180676]
There were at-least 100 failed and timed out tests. I am sure they are not
related to my patch.
Also I came across another jira which was just a javadoc related change and
there were around 100 failed tests.
Below are the details for pre-commits that failed in branch-2
1 [HDFS-12386 attempt
1|https://issues.apache.org/jira/browse/HDFS-12386?focusedCommentId=16180069&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16180069]
{noformat}
Ran on slave: asf912.gq1.ygridcore.net/H12
Failed with following error message:
Build timed out (after 300 minutes). Marking the build as aborted.
Build was aborted
Performing Post build task...
{noformat}
2. [HDFS-12386 attempt
2|https://issues.apache.org/jira/browse/HDFS-12386?focusedCommentId=16180676&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16180676]
{noformat}
Ran on slave: asf900.gq1.ygridcore.net
Failed with following error message:
FATAL: command execution failed
Command close created at
at hudson.remoting.Command.(Command.java:60)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1123)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1121)
at hudson.remoting.Channel.close(Channel.java:1281)
at hudson.remoting.Channel.close(Channel.java:1263)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1128)
Caused: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1129)
at hudson.remoting.Channel$1.handle(Channel.java:527)
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused: java.io.IOException: Backing channel 'H0' is disconnected.
at
hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:192)
at
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
at com.sun.proxy.$Proxy125.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1043)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1035)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
at hudson.model.Run.execute(Run.java:1735)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:405)
{noformat}
3. [HDFS-12531 attempt
1|https://issues.apache.org/jira/browse/HDFS-12531?focusedCommentId=16176493&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16176493]
{noformat}
Ran on slave: asf911.gq1.ygridcore.net
Failed with following error message:
FATAL: command execution failed
Command close created at
at hudson.remoting.Command.(Command.java:60)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1123)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1121)
at hudson.remoting.Channel.close(Channel.java:1281)
at hudson.remoting.Channel.close(Channel.java:1263)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1128)
Caused: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1129)
at hudson.remoting.Channel$1.handle(Channel.java:527)
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused: java.io.IOException: Backing channel 'H11' is disconnected.
at
hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:192)
at
hudson.remoting.RemoteInvo
HDFS pre-commit builds are unstable on branch-2
Hi All, Recently hdfs precommit build for branch-2 is unstable, marking many (322 to be precise) unit tests as failed or timed out. I have created tracking ticket https://issues.apache.org/jira/browse/HDFS-12548 with additional details. Appreciate your inputs on that ticket.
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/ [Sep 25, 2017 12:12:25 PM] (templedf) HADOOP-14897. Loosen compatibility guidelines for native dependencies [Sep 25, 2017 4:25:07 PM] (aajisaka) HDFS-12304. Remove unused parameter from FsDatasetImpl#addVolume. [Sep 25, 2017 4:47:02 PM] (aajisaka) HADOOP-14892. MetricsSystemImpl should use Time.monotonicNow for [Sep 25, 2017 5:03:30 PM] (aajisaka) YARN-7153. Remove duplicated code in AMRMClientAsyncImpl.java. [Sep 25, 2017 8:35:44 PM] (jlowe) HADOOP-14881. LoadGenerator should use Time.monotonicNow() to measure [Sep 25, 2017 8:55:09 PM] (aengineer) HDFS-12529. Get source for config tags from file name. Contributed by [Sep 25, 2017 10:30:51 PM] (rchiang) HADOOP-14903. Add json-smart explicitly to pom.xml. (rchiang) [Sep 25, 2017 10:59:38 PM] (fabbri) HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity [Sep 25, 2017 11:02:55 PM] (Arun Suresh) YARN-7240. Add more states and transitions to stabilize the NM Container [Sep 26, 2017 12:12:01 AM] (fabbri) HADOOP-14890 Move up to AWS SDK 1.11.199. Contributed by Steve Loughran. -1 overall The following subsystems voted -1: unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation hadoop.yarn.client.api.impl.TestAMRMClient hadoop.mapreduce.TestMapreduceConfigFields hadoop.mapreduce.v2.hs.webapp.TestHSWebApp hadoop.yarn.sls.TestSLSRunner hadoop.yarn.sls.TestReservationSystemInvariants Timed out junit tests : org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-compile-javac-root.txt [292K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/whitespace-tabs.txt [1.2M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-javadoc-javadoc-root.txt [1.9M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [64K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [20K] Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12549) Ozone: OzoneClient: Support for REST protocol
Nandakumar created HDFS-12549: - Summary: Ozone: OzoneClient: Support for REST protocol Key: HDFS-12549 URL: https://issues.apache.org/jira/browse/HDFS-12549 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Nandakumar Assignee: Nandakumar Support for REST protocol in OzoneClient. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12550) Ozone: OzoneClient: Removal of old OzoneRestClient
Nandakumar created HDFS-12550: - Summary: Ozone: OzoneClient: Removal of old OzoneRestClient Key: HDFS-12550 URL: https://issues.apache.org/jira/browse/HDFS-12550 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Nandakumar Assignee: Nandakumar Once new REST based OzoneClient is read, the old OzoneRestClient can be removed. This jira is to track it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12551) Ozone: Documentation: Add Ozone overview documentation.
Anu Engineer created HDFS-12551: --- Summary: Ozone: Documentation: Add Ozone overview documentation. Key: HDFS-12551 URL: https://issues.apache.org/jira/browse/HDFS-12551 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Assignee: Anu Engineer Add an architectural overview doc for ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12552) Use slf4j instead of log4j in FSNamesystem
Ajay Kumar created HDFS-12552: - Summary: Use slf4j instead of log4j in FSNamesystem Key: HDFS-12552 URL: https://issues.apache.org/jira/browse/HDFS-12552 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ajay Kumar Assignee: Ajay Kumar FileNamesystem is still using log4j dependencies. We should move those to slf4j, as most of the methods using log4j are deprecated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12553) Add nameServiceId to QJournalProtocol
Bharat Viswanadham created HDFS-12553: - Summary: Add nameServiceId to QJournalProtocol Key: HDFS-12553 URL: https://issues.apache.org/jira/browse/HDFS-12553 Project: Hadoop HDFS Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Add namServiceId to QjournalProtocol. This is used during federated + HA setup to find journalnodes belonging to a nameservice. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/ [Sep 26, 2017 7:46:03 AM] (aajisaka) HADOOP-14893. WritableRpcEngine should use Time.monotonicNow. [Sep 26, 2017 9:04:09 AM] (rohithsharmaks) YARN-65. Reduce RM app memory footprint once app has completed. [Sep 26, 2017 9:21:42 AM] (yqlin) HDFS-12495. TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks [Sep 26, 2017 3:38:03 PM] (jlowe) Revert "YARN-6570. No logs were found for running application, running" -1 overall The following subsystems voted -1: unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.crypto.key.TestValueQueue hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem hadoop.crypto.key.kms.server.TestKMS hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.TestSafeModeWithStripedFile hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer hadoop.yarn.server.router.webapp.TestRouterWebServicesREST hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapreduce.TestMapreduceConfigFields hadoop.mapreduce.v2.hs.webapp.TestHSWebApp hadoop.yarn.sls.TestReservationSystemInvariants hadoop.yarn.sls.TestSLSRunner Timed out junit tests : org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-compile-javac-root.txt [292K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/whitespace-tabs.txt [1.2M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-javadoc-javadoc-root.txt [1.9M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [156K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [352K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [64K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt [604K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [24K] Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12554) Ozone: Fix TestDatanodeStateMachine#testDatanodeStateMachineWithInvalidConfiguration
Xiaoyu Yao created HDFS-12554:
-
Summary: Ozone: Fix
TestDatanodeStateMachine#testDatanodeStateMachineWithInvalidConfiguration
Key: HDFS-12554
URL: https://issues.apache.org/jira/browse/HDFS-12554
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Xiaoyu Yao
{{hadoop.ozone.container.common.TestDatanodeStateMachine#testDatanodeStateMachineWithInvalidConfiguration}}
failure is related to this patch.
Invalid ozone.scm.datanode.id like below in the failed test used to prevent
datanode from running and now it is allowed. Please update the unit test and
the OzoneGetStarted.md file correspondingly.
{code}
confList.add(Maps.immutableEntry(
ScmConfigKeys.OZONE_SCM_DATANODE_ID, ""));
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12555) HDFS federation should support configure secondary directory
luoge123 created HDFS-12555: --- Summary: HDFS federation should support configure secondary directory Key: HDFS-12555 URL: https://issues.apache.org/jira/browse/HDFS-12555 Project: Hadoop HDFS Issue Type: Improvement Components: federation Reporter: luoge123 HDFS federation support multiple namenodes horizontally scales the file system namespace. As the amount of data grows, using a single group of namenodes to manage a single directory, namenode still achieves performance bottlenecks. In order to reduce the pressure of namenode, we can split out the secondary directory, and manager it by a new namenode. This is transparent for users. For example, nn1 only manager the /user directory, when nn1 achieve performance bottlenecks, we can split out /user/hive directory, and ues nn2 to manager it. That means core-site.xml should support as follows configuration. fs.viewfs.mounttable.nsX.link./user hdfs://nn1:8020/user fs.viewfs.mounttable.nsX.link./user/hive hdfs://nn2:8020/user/hive -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
