[jira] [Created] (HDFS-14077) DFSAdmin Report datanode Count was not matched when datanode in Decommissioned state
Harshakiran Reddy created HDFS-14077: Summary: DFSAdmin Report datanode Count was not matched when datanode in Decommissioned state Key: HDFS-14077 URL: https://issues.apache.org/jira/browse/HDFS-14077 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.1.1 Reporter: Harshakiran Reddy {noformat} In DFSAdmin Reports showing the live datanodes are incorrect when some datanodes in Decommissioned State {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14078) Admin helper fails to prettify NullPointerExceptions
Elek, Marton created HDFS-14078: --- Summary: Admin helper fails to prettify NullPointerExceptions Key: HDFS-14078 URL: https://issues.apache.org/jira/browse/HDFS-14078 Project: Hadoop HDFS Issue Type: Bug Reporter: Elek, Marton Assignee: Elek, Marton org.apache.hadoop.hdfs.tools.AdminHelper has a method to prettifyExceptions: {code} static String prettifyException(Exception e) { return e.getClass().getSimpleName() + ": " + e.getLocalizedMessage().split("\n")[0]; } {code} But if e is a NPE the e.getLocalizedMessage() could be null. In that case NPE will be thrown and the original error message will be lost. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-837) Persist originNodeId as part of .container file in datanode
Nanda kumar created HDDS-837: Summary: Persist originNodeId as part of .container file in datanode Key: HDDS-837 URL: https://issues.apache.org/jira/browse/HDDS-837 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Datanode Reporter: Nanda kumar Assignee: Nanda kumar To differentiate the replica of QUASI_CLOSED containers we need {{originNodeId}} field. With this field, we can uniquely identify a QUASI_CLOSED container replica. This will be needed when we want to CLOSE a QUASI_CLOSED container. This field will be set by the node where the container is created and stored as part of {{.container}} file and will be sent as part of ContainerReport to SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14079) RBF : RouterAdmin should have failover concept for router
Surendra Singh Lilhore created HDFS-14079: - Summary: RBF : RouterAdmin should have failover concept for router Key: HDFS-14079 URL: https://issues.apache.org/jira/browse/HDFS-14079 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.1.1 Reporter: Surendra Singh Lilhore Assignee: Surendra Singh Lilhore Currenlty {{RouterAdmin}} connect with only one router for admin operation, if the configured router is down then router admin command is failing. It should allow to configure all the router admin address. {code} // Initialize RouterClient try { String address = getConf().getTrimmed( RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY, RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT); InetSocketAddress routerSocket = NetUtils.createSocketAddr(address); client = new RouterClient(routerSocket, getConf()); } catch (RPC.VersionMismatch v) { System.err.println( "Version mismatch between client and server... command aborted"); return exitCode; } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/ [Nov 13, 2018 2:38:27 AM] (yqlin) HDDS-831. TestOzoneShell in integration-test is flaky. Contributed by [Nov 13, 2018 4:40:43 AM] (aajisaka) HADOOP-15923. create-release script should set max-cache-ttl as well as [Nov 13, 2018 4:57:07 AM] (tasanuma) HADOOP-15912. start-build-env.sh still creates an invalid [Nov 13, 2018 7:15:44 AM] (brahma) HDFS-14070. Refactor NameNodeWebHdfsMethods to allow better [Nov 13, 2018 2:52:58 PM] (surendralilhore) HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in [Nov 13, 2018 6:09:14 PM] (shashikant) HDDS-675. Add blocking buffer and use watchApi for flush/close in [Nov 13, 2018 7:24:15 PM] (wangda) YARN-8918. [Submarine] Correct method usage of str.subString in [Nov 13, 2018 7:25:41 PM] (wangda) MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. [Nov 13, 2018 8:44:25 PM] (xiao) Revert "HDFS-13732. ECAdmin should print the policy name when an EC [Nov 13, 2018 9:13:27 PM] (wangda) YARN-9001. [Submarine] Use AppAdminClient instead of ServiceClient to [Nov 13, 2018 9:46:18 PM] (stevel) HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestReconstructStripedFile hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.balancer.TestBalancer hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.mapreduce.jobhistory.TestEvents cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-compile-javac-root.txt [324K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-patch-pylint.txt [40K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-patch-shellcheck.txt [68K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/whitespace-eol.txt [9.3M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/whitespace-tabs.txt [1.1M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_client.txt [24K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_common.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt [44K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs
[jira] [Created] (HDFS-14080) DFS usage metrics reported in incorrect prefix
Greg Phillips created HDFS-14080: Summary: DFS usage metrics reported in incorrect prefix Key: HDFS-14080 URL: https://issues.apache.org/jira/browse/HDFS-14080 Project: Hadoop HDFS Issue Type: Improvement Components: namenode, ui Reporter: Greg Phillips The NameNode webapp reports DFS usage metrics using standard SI prefixes (MB, GB, etc.). The number reported in the UI is calculated to be the binary size which should be noted using binary prefixes (MiB, GiB, etc.). The NameNode webapp should be modified to use the correct binary prefixes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[CANCELED] [VOTE] Release Apache Hadoop Ozone 0.3.0-alpha (RC0)
Unfortunately a memory issue is found with the default settings. Fixed in HDDS-834 (thanks Mukul and Shashikant) I cancel this vote and start a rc1 soon. Marton On 11/13/18 1:53 PM, Elek, Marton wrote: > Hi all, > > I've created the first release candidate (RC0) for Apache Hadoop Ozone > 0.3.0-alpha according to the plans shared here previously. > > This is the second release of Apache Hadoop Ozone. Notable changes since > the first release: > > * A new S3 compatible rest server is added. Ozone can be used from any > S3 compatible tools (HDDS-434) > * Ozone Hadoop file system URL prefix is renamed from o3:// to o3fs:// > (HDDS-651) > * Extensive testing and stability improvements of OzoneFs. > * Spark, YARN and Hive support and stability improvements. > * Improved Pipeline handling and recovery. > * Separated/dedicated classpath definitions for all the Ozone > components. (HDDS-447) > > The RC artifacts are available from: > https://home.apache.org/~elek/ozone-0.3.0-alpha-rc0/ > > The RC tag in git is: ozone-0.3.0-alpha-RC0 (dc661083683) > > Please try it out, vote, or just give us feedback. > > The vote will run for 5 days, ending on November 18, 2018 13:00 UTC. > > > Thank you very much, > Marton > > PS: > > The easiest way to try it out is: > > 1. Download the binary artifact > 2. Read the docs from ./docs/index.html > 3. TLDR; cd compose/ozone && docker-compose up -d > 4. open localhost:9874 or localhost:9876 > > > > The easiest way to try it out from the source: > > 1. mvn install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds > -DskipShade -am -pl :hadoop-ozone-dist > 2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha && docker-compose up -d > > > > The easiest way to test basic functionality (with acceptance tests): > > 1. mvn install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds > -DskipShade -am -pl :hadoop-ozone-dist > 2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha/smoketest > 3. ./test.sh > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop Ozone 0.3.0-alpha (RC1)
Hi all, I've created the second release candidate (RC1) for Apache Hadoop Ozone 0.3.0-alpha including one more fix on top of the previous RC0 (HDDS-854) This is the second release of Apache Hadoop Ozone. Notable changes since the first release: * A new S3 compatible rest server is added. Ozone can be used from any S3 compatible tools (HDDS-434) * Ozone Hadoop file system URL prefix is renamed from o3:// to o3fs:// (HDDS-651) * Extensive testing and stability improvements of OzoneFs. * Spark, YARN and Hive support and stability improvements. * Improved Pipeline handling and recovery. * Separated/dedicated classpath definitions for all the Ozone components. (HDDS-447) The RC artifacts are available from: https://home.apache.org/~elek/ozone-0.3.0-alpha-rc1/ The RC tag in git is: ozone-0.3.0-alpha-RC1 (ebbf459e6a6) Please try it out, vote, or just give us feedback. The vote will run for 5 days, ending on November 19, 2018 18:00 UTC. Thank you very much, Marton PS: The easiest way to try it out is: 1. Download the binary artifact 2. Read the docs from ./docs/index.html 3. TLDR; cd compose/ozone && docker-compose up -d 4. open localhost:9874 or localhost:9876 The easiest way to try it out from the source: 1. mvn install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds -DskipShade -am -pl :hadoop-ozone-dist 2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha && docker-compose up -d The easiest way to test basic functionality (with acceptance tests): 1. mvn install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds -DskipShade -am -pl :hadoop-ozone-dist 2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha/smoketest 3. ./test.sh - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14081) hdfs dfsadmin -metasave metasave_test results NPE
Shweta created HDFS-14081: - Summary: hdfs dfsadmin -metasave metasave_test results NPE Key: HDFS-14081 URL: https://issues.apache.org/jira/browse/HDFS-14081 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 3.2.1 Reporter: Shweta Assignee: Shweta Fix For: 3.2.1 Race condition is encountered while adding Block to postponedMisreplicatedBlocks which in turn tried to retrieve Block from BlockManager in which it may not be present. This happens in HA, metasave in first NN succeeded but failed in second NN, StackTrace showing NPE is as follows: 2018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server: IPC Server handler 24 on 8020, call Call#1 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from 172.26.9.163:602342018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server: IPC Server handler 24 on 8020, call Call#1 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from 172.26.9.163:60234java.lang.NullPointerException at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseSourceDatanodes(BlockManager.java:2175) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.dumpBlockMeta(BlockManager.java:830) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.metaSave(BlockManager.java:762) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1782) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1766) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.metaSave(NameNodeRpcServer.java:1320) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.metaSave(ClientNamenodeProtocolServerSideTranslatorPB.java:928) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-838) Basic operations like create volume, freon are not working
Dinesh Chitlangia created HDDS-838: -- Summary: Basic operations like create volume, freon are not working Key: HDDS-838 URL: https://issues.apache.org/jira/browse/HDDS-838 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client, Ozone Manager Reporter: Dinesh Chitlangia After pulling latest from trunk, running simple operations like create volume, freon rk are failing with following exception: {code:java} MYBOX:ozone-0.4.0-SNAPSHOT dchitlangia$ bin/ozone sh volume create /test 2018-11-14 15:30:59,918 [main] ERROR - Couldn't create protocol class org.apache.hadoop.ozone.client.rpc.RpcClient exception: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291) at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169) at org.apache.hadoop.ozone.web.ozShell.OzoneAddress.createClient(OzoneAddress.java:111) at org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:70) at org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:38) at picocli.CommandLine.execute(CommandLine.java:919) at picocli.CommandLine.access$700(CommandLine.java:104) at picocli.CommandLine$RunLast.handle(CommandLine.java:1083) at picocli.CommandLine$RunLast.handle(CommandLine.java:1051) at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959) at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242) at picocli.CommandLine.parseWithHandler(CommandLine.java:1181) at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61) at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52) at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:80) Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.ozone.om.OzoneManager.getServiceList(OzoneManager.java:1118) at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.getServiceList(OzoneManagerProtocolServerSideTranslatorPB.java:580) at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:39227) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) at org.apache.hadoop.ipc.Client.call(Client.java:1457) at org.apache.hadoop.ipc.Client.call(Client.java:1367) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy10.getServiceList(Unknown Source) at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceList(OzoneManagerProtocolClientSideTranslatorPB.java:766) at org.apache.hadoop.ozone.client.rpc.RpcClient.getScmAddressForClient(RpcClient.java:169) at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:130) ... 19 more java.lang.NullPointerException {code} Also verified using _jps_ that SCM, Datanode & OM are up and running. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-152) Support HA for Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDDS-152. Resolution: Fixed Resolving as dup of HDDS-505. This was filed first but there is a design doc on the other Jira so let's continue there. > Support HA for Ozone Manager > > > Key: HDDS-152 > URL: https://issues.apache.org/jira/browse/HDDS-152 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: DENG FEI >Priority: Major > > Ozone Manager(OM) provide the name services on top of HDDS(SCM). This ticket > is opened to add HA support for OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14082) RBF: Add option to fail operations when a subcluster is unavailable
Íñigo Goiri created HDFS-14082: -- Summary: RBF: Add option to fail operations when a subcluster is unavailable Key: HDFS-14082 URL: https://issues.apache.org/jira/browse/HDFS-14082 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri Assignee: Íñigo Goiri When a subcluster is unavailable, we succeed operations like {{getListing()}}. We should add an option to fail the operation if one of the subclusters is unavailable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-838) Basic operations like create volume, freon are not working
[ https://issues.apache.org/jira/browse/HDDS-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDDS-838. Resolution: Later > Basic operations like create volume, freon are not working > -- > > Key: HDDS-838 > URL: https://issues.apache.org/jira/browse/HDDS-838 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client, Ozone Manager > Environment: JDK11 >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > > After pulling latest from trunk, running simple operations like create > volume, freon rk are failing with following exception: > > {code:java} > MYBOX:ozone-0.4.0-SNAPSHOT dchitlangia$ bin/ozone sh volume create /test > 2018-11-14 15:30:59,918 [main] ERROR - Couldn't create protocol class > org.apache.hadoop.ozone.client.rpc.RpcClient exception: > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291) > at > org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169) > at > org.apache.hadoop.ozone.web.ozShell.OzoneAddress.createClient(OzoneAddress.java:111) > at > org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:70) > at > org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:38) > at picocli.CommandLine.execute(CommandLine.java:919) > at picocli.CommandLine.access$700(CommandLine.java:104) > at picocli.CommandLine$RunLast.handle(CommandLine.java:1083) > at picocli.CommandLine$RunLast.handle(CommandLine.java:1051) > at > picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959) > at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242) > at picocli.CommandLine.parseWithHandler(CommandLine.java:1181) > at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61) > at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52) > at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:80) > Caused by: > org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): > java.lang.NullPointerException > at > org.apache.hadoop.ozone.om.OzoneManager.getServiceList(OzoneManager.java:1118) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.getServiceList(OzoneManagerProtocolServerSideTranslatorPB.java:580) > at > org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:39227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.base/java.security.AccessController.doPrivileged(Native Method) > at java.base/javax.security.auth.Subject.doAs(Subject.java:423) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) > at org.apache.hadoop.ipc.Client.call(Client.java:1457) > at org.apache.hadoop.ipc.Client.call(Client.java:1367) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy10.getServiceList(Unknown Source) > at > org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceList(OzoneManagerProtocolClientSideTranslatorPB.java:766) > at > org.apache.hadoop.ozone.client.rpc.RpcClient.getScmAddressForClient(RpcClient.java:169) > at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:130) > ... 19 more > java.lang.NullPointerException > {code} > Also verified using _jps_ that SCM, Datanode & OM are up and running. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14072) DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace.
[ https://issues.apache.org/jira/browse/HDFS-14072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-14072. -- Resolution: Duplicate Resolving as a dup since you have posted a patch on HDFS-14074. > DataNode runs async disk checks maybe throws NullPointerException, and > DataNode failed to register to NameSpace. > -- > > Key: HDFS-14072 > URL: https://issues.apache.org/jira/browse/HDFS-14072 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.0, 3.0.0 > Environment: hadoop-2.7.3, hadoop-2.8.0 >Reporter: guangyi lu >Priority: Major > Labels: HDFS, HDFS-4 > Attachments: image-2018-11-13-20-23-22-138.png, > image-2018-11-13-20-23-26-813.png > > Original Estimate: 48h > Remaining Estimate: 48h > > In ThrottledAsyncChecker class,it members of the completedChecks is > WeakHashMap, its definition is as follows: > this.completedChecks = new WeakHashMap<>(); > and one of its uses is as follows in schedule method: > if (completedChecks.containsKey(target)) { > // here may be happen garbage collection,and result may be null. > final LastCheckResult result = completedChecks.get(target); > final long msSinceLastCheck = timer.monotonicNow() - > result.completedAt; > } > after "completedChecks.containsKey(target)", may be happen garbage > collection, and result may be null. > > !image-2018-11-13-20-23-26-813.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14073) DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace.
[ https://issues.apache.org/jira/browse/HDFS-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-14073. -- Resolution: Duplicate > DataNode runs async disk checks maybe throws NullPointerException, and > DataNode failed to register to NameSpace. > -- > > Key: HDFS-14073 > URL: https://issues.apache.org/jira/browse/HDFS-14073 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.0, 3.0.0 > Environment: hadoop-2.7.3, hadoop-2.8.0 >Reporter: guangyi lu >Priority: Major > Labels: HDFS, HDFS-4 > Original Estimate: 48h > Remaining Estimate: 48h > > In ThrottledAsyncChecker class,it members of the completedChecks is > WeakHashMap, its definition is as follows: > this.completedChecks = new WeakHashMap<>(); > and one of its uses is as follows in schedule method: > if (completedChecks.containsKey(target)) { > // here may be happen garbage collection,and result may be null. > final LastCheckResult result = completedChecks.get(target); > final long msSinceLastCheck = timer.monotonicNow() - > result.completedAt; > } > after "completedChecks.containsKey(target)", may be happen garbage > collection, and result may be null. > > !image-2018-11-13-20-23-26-813.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14071) DataNode runs async disk checks maybe throws NullPointerException In ThrottledAsyncChecker.java
[ https://issues.apache.org/jira/browse/HDFS-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-14071. -- Resolution: Duplicate > DataNode runs async disk checks maybe throws NullPointerException In > ThrottledAsyncChecker.java > --- > > Key: HDFS-14071 > URL: https://issues.apache.org/jira/browse/HDFS-14071 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: guangyi lu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-135) SCM CA: Update DelegationKey to retrieve private/public key
[ https://issues.apache.org/jira/browse/HDDS-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-135. - Resolution: Implemented This has been implemented with HDDS-836. > SCM CA: Update DelegationKey to retrieve private/public key > --- > > Key: HDDS-135 > URL: https://issues.apache.org/jira/browse/HDDS-135 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.9.2 (RC0)
Thanks for driving the release, Akira! +1 (non-binding) - verified checksums - succeeded in building the package - started hadoop cluster with 1 master and 5 slaves - ran TeraGen/TeraSort - verified Web UI (NN, RM, JobHistory, Timeline) - verified some operations of Router-based Federation Thanks, -Takanobu on 2018/11/14 10:02, "Akira Ajisaka" wrote: Hi folks, I have put together a release candidate (RC0) for Hadoop 2.9.2. It includes 204 bug fixes and improvements since 2.9.1. [1] The RC is available at http://home.apache.org/~aajisaka/hadoop-2.9.2-RC0/ Git signed tag is release-2.9.2-RC0 and the checksum is 826afbeae31ca687bc2f8471dc841b66ed2c6704 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1166/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Please try the release and vote. The vote will run for 5 days. [1] https://s.apache.org/2.9.2-fixed-jiras Thanks, Akira - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org