[jira] [Created] (HDFS-14077) DFSAdmin Report datanode Count was not matched when datanode in Decommissioned state

2018-11-14 Thread Harshakiran Reddy (JIRA)
Harshakiran Reddy created HDFS-14077:


 Summary: DFSAdmin  Report datanode Count was not matched when 
datanode in Decommissioned state
 Key: HDFS-14077
 URL: https://issues.apache.org/jira/browse/HDFS-14077
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Affects Versions: 3.1.1
Reporter: Harshakiran Reddy


{noformat}
In DFSAdmin Reports showing the live datanodes are incorrect when some 
datanodes in Decommissioned State
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-14078) Admin helper fails to prettify NullPointerExceptions

2018-11-14 Thread Elek, Marton (JIRA)
Elek, Marton created HDFS-14078:
---

 Summary: Admin helper fails to prettify NullPointerExceptions
 Key: HDFS-14078
 URL: https://issues.apache.org/jira/browse/HDFS-14078
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Elek, Marton
Assignee: Elek, Marton


org.apache.hadoop.hdfs.tools.AdminHelper has a method to prettifyExceptions:

{code}
  static String prettifyException(Exception e) {
return e.getClass().getSimpleName() + ": "
+ e.getLocalizedMessage().split("\n")[0];
  }
{code}

But if e is a NPE the e.getLocalizedMessage() could be null. In that case NPE 
will be thrown and the original error message will be lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDDS-837) Persist originNodeId as part of .container file in datanode

2018-11-14 Thread Nanda kumar (JIRA)
Nanda kumar created HDDS-837:


 Summary: Persist originNodeId as part of .container file in 
datanode
 Key: HDDS-837
 URL: https://issues.apache.org/jira/browse/HDDS-837
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Datanode
Reporter: Nanda kumar
Assignee: Nanda kumar


To differentiate the replica of QUASI_CLOSED containers we need 
{{originNodeId}} field. With this field, we can uniquely identify a 
QUASI_CLOSED container replica. This will be needed when we want to CLOSE a 
QUASI_CLOSED container.

This field will be set by the node where the container is created and stored as 
part of {{.container}} file and will be sent as part of ContainerReport to SCM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-14079) RBF : RouterAdmin should have failover concept for router

2018-11-14 Thread Surendra Singh Lilhore (JIRA)
Surendra Singh Lilhore created HDFS-14079:
-

 Summary: RBF : RouterAdmin should have failover concept for router
 Key: HDFS-14079
 URL: https://issues.apache.org/jira/browse/HDFS-14079
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Surendra Singh Lilhore
Assignee: Surendra Singh Lilhore


Currenlty {{RouterAdmin}} connect with only one router for admin operation, if 
the configured router is down then router admin command is failing. It should 
allow to configure all the router admin address.

{code}
// Initialize RouterClient
try {
  String address = getConf().getTrimmed(
  RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
  RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
  InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
  client = new RouterClient(routerSocket, getConf());
} catch (RPC.VersionMismatch v) {
  System.err.println(
  "Version mismatch between client and server... command aborted");
  return exitCode;
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-11-14 Thread Apache Jenkins Server
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/

[Nov 13, 2018 2:38:27 AM] (yqlin) HDDS-831. TestOzoneShell in integration-test 
is flaky. Contributed by
[Nov 13, 2018 4:40:43 AM] (aajisaka) HADOOP-15923. create-release script should 
set max-cache-ttl as well as
[Nov 13, 2018 4:57:07 AM] (tasanuma) HADOOP-15912. start-build-env.sh still 
creates an invalid
[Nov 13, 2018 7:15:44 AM] (brahma) HDFS-14070. Refactor NameNodeWebHdfsMethods 
to allow better
[Nov 13, 2018 2:52:58 PM] (surendralilhore) HADOOP-15869. 
BlockDecompressorStream#decompress should not return -1 in
[Nov 13, 2018 6:09:14 PM] (shashikant) HDDS-675. Add blocking buffer and use 
watchApi for flush/close in
[Nov 13, 2018 7:24:15 PM] (wangda) YARN-8918. [Submarine] Correct method usage 
of str.subString in
[Nov 13, 2018 7:25:41 PM] (wangda) MAPREDUCE-7158. Inefficient Flush Logic in 
JobHistory EventWriter.
[Nov 13, 2018 8:44:25 PM] (xiao) Revert "HDFS-13732. ECAdmin should print the 
policy name when an EC
[Nov 13, 2018 9:13:27 PM] (wangda) YARN-9001. [Submarine] Use AppAdminClient 
instead of ServiceClient to
[Nov 13, 2018 9:46:18 PM] (stevel) HADOOP-15876. Use keySet().removeAll() to 
remove multiple keys from Map




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.balancer.TestBalancer 
   
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   hadoop.mapreduce.jobhistory.TestEvents 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-compile-javac-root.txt
  [324K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-patch-pylint.txt
  [40K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-patch-shellcheck.txt
  [68K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/whitespace-eol.txt
  [9.3M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/whitespace-tabs.txt
  [1.1M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [24K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt
  [44K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/957/artifact/out/branch-findbugs

[jira] [Created] (HDFS-14080) DFS usage metrics reported in incorrect prefix

2018-11-14 Thread Greg Phillips (JIRA)
Greg Phillips created HDFS-14080:


 Summary: DFS usage metrics reported in incorrect prefix
 Key: HDFS-14080
 URL: https://issues.apache.org/jira/browse/HDFS-14080
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode, ui
Reporter: Greg Phillips


The NameNode webapp reports DFS usage metrics using standard SI prefixes (MB, 
GB, etc.). The number reported in the UI is calculated to be the binary size 
which should be noted using binary prefixes (MiB, GiB, etc.). The NameNode 
webapp should be modified to use the correct binary prefixes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[CANCELED] [VOTE] Release Apache Hadoop Ozone 0.3.0-alpha (RC0)

2018-11-14 Thread Elek, Marton
Unfortunately a memory issue is found with the default settings. Fixed
in HDDS-834 (thanks Mukul and Shashikant)

I cancel this vote and start a rc1 soon.

Marton

On 11/13/18 1:53 PM, Elek, Marton wrote:
> Hi all,
> 
> I've created the first release candidate (RC0) for Apache Hadoop Ozone
> 0.3.0-alpha according to the plans shared here previously.
> 
> This is the second release of Apache Hadoop Ozone. Notable changes since
> the first release:
> 
> * A new S3 compatible rest server is added. Ozone can be used from any
> S3 compatible tools (HDDS-434)
> * Ozone Hadoop file system URL prefix is renamed from o3:// to o3fs://
> (HDDS-651)
> * Extensive testing and stability improvements of OzoneFs.
> * Spark, YARN and Hive support and stability improvements.
> * Improved Pipeline handling and recovery.
> * Separated/dedicated classpath definitions for all the Ozone
> components. (HDDS-447)
> 
> The RC artifacts are available from:
> https://home.apache.org/~elek/ozone-0.3.0-alpha-rc0/
> 
> The RC tag in git is: ozone-0.3.0-alpha-RC0 (dc661083683)
> 
> Please try it out, vote, or just give us feedback.
> 
> The vote will run for 5 days, ending on November 18, 2018 13:00 UTC.
> 
> 
> Thank you very much,
> Marton
> 
> PS:
> 
> The easiest way to try it out is:
> 
> 1. Download the binary artifact
> 2. Read the docs from ./docs/index.html
> 3. TLDR; cd compose/ozone && docker-compose up -d
> 4. open localhost:9874 or localhost:9876
> 
> 
> 
> The easiest way to try it out from the source:
> 
> 1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
> -DskipShade -am -pl :hadoop-ozone-dist
> 2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha && docker-compose up -d
> 
> 
> 
> The easiest way to test basic functionality (with acceptance tests):
> 
> 1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
> -DskipShade -am -pl :hadoop-ozone-dist
> 2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha/smoketest
> 3. ./test.sh
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[VOTE] Release Apache Hadoop Ozone 0.3.0-alpha (RC1)

2018-11-14 Thread Elek, Marton
Hi all,

I've created the second release candidate (RC1) for Apache Hadoop Ozone
0.3.0-alpha including one more fix on top of the previous RC0 (HDDS-854)

This is the second release of Apache Hadoop Ozone. Notable changes since
the first release:

* A new S3 compatible rest server is added. Ozone can be used from any
S3 compatible tools (HDDS-434)
* Ozone Hadoop file system URL prefix is renamed from o3:// to o3fs://
(HDDS-651)
* Extensive testing and stability improvements of OzoneFs.
* Spark, YARN and Hive support and stability improvements.
* Improved Pipeline handling and recovery.
* Separated/dedicated classpath definitions for all the Ozone
components. (HDDS-447)

The RC artifacts are available from:
https://home.apache.org/~elek/ozone-0.3.0-alpha-rc1/

The RC tag in git is: ozone-0.3.0-alpha-RC1 (ebbf459e6a6)

Please try it out, vote, or just give us feedback.

The vote will run for 5 days, ending on November 19, 2018 18:00 UTC.


Thank you very much,
Marton

PS:

The easiest way to try it out is:

1. Download the binary artifact
2. Read the docs from ./docs/index.html
3. TLDR; cd compose/ozone && docker-compose up -d
4. open localhost:9874 or localhost:9876



The easiest way to try it out from the source:

1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
-DskipShade -am -pl :hadoop-ozone-dist
2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha && docker-compose up -d



The easiest way to test basic functionality (with acceptance tests):

1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
-DskipShade -am -pl :hadoop-ozone-dist
2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha/smoketest
3. ./test.sh

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-14081) hdfs dfsadmin -metasave metasave_test results NPE

2018-11-14 Thread Shweta (JIRA)
Shweta created HDFS-14081:
-

 Summary: hdfs dfsadmin -metasave metasave_test results NPE
 Key: HDFS-14081
 URL: https://issues.apache.org/jira/browse/HDFS-14081
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Affects Versions: 3.2.1
Reporter: Shweta
Assignee: Shweta
 Fix For: 3.2.1


Race condition is encountered while adding Block to 
postponedMisreplicatedBlocks which in turn tried to retrieve Block from 
BlockManager in which it may not be present. 

This happens in HA, metasave in first NN succeeded but failed in second NN, 
StackTrace showing NPE is as follows:

2018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
24 on 8020, call Call#1 Retry#0 
org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from 
172.26.9.163:602342018-07-12 21:39:09,783 WARN org.apache.hadoop.ipc.Server: 
IPC Server handler 24 on 8020, call Call#1 Retry#0 
org.apache.hadoop.hdfs.protocol.ClientProtocol.metaSave from 
172.26.9.163:60234java.lang.NullPointerException at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseSourceDatanodes(BlockManager.java:2175)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.dumpBlockMeta(BlockManager.java:830)
 at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.metaSave(BlockManager.java:762)
 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1782)
 at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.metaSave(FSNamesystem.java:1766)
 at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.metaSave(NameNodeRpcServer.java:1320)
 at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.metaSave(ClientNamenodeProtocolServerSideTranslatorPB.java:928)
 at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
 at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at 
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at 
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:422) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDDS-838) Basic operations like create volume, freon are not working

2018-11-14 Thread Dinesh Chitlangia (JIRA)
Dinesh Chitlangia created HDDS-838:
--

 Summary: Basic operations like create volume, freon are not working
 Key: HDDS-838
 URL: https://issues.apache.org/jira/browse/HDDS-838
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Client, Ozone Manager
Reporter: Dinesh Chitlangia


After pulling latest from trunk, running simple operations like create volume, 
freon rk are failing with following exception:

 
{code:java}
MYBOX:ozone-0.4.0-SNAPSHOT dchitlangia$ bin/ozone sh volume create /test
2018-11-14 15:30:59,918 [main] ERROR - Couldn't create protocol class 
org.apache.hadoop.ozone.client.rpc.RpcClient exception:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291)
at 
org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169)
at 
org.apache.hadoop.ozone.web.ozShell.OzoneAddress.createClient(OzoneAddress.java:111)
at 
org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:70)
at 
org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:38)
at picocli.CommandLine.execute(CommandLine.java:919)
at picocli.CommandLine.access$700(CommandLine.java:104)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
at 
picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:80)
Caused by: 
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): 
java.lang.NullPointerException
at 
org.apache.hadoop.ozone.om.OzoneManager.getServiceList(OzoneManager.java:1118)
at 
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.getServiceList(OzoneManagerProtocolServerSideTranslatorPB.java:580)
at 
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:39227)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511)
at org.apache.hadoop.ipc.Client.call(Client.java:1457)
at org.apache.hadoop.ipc.Client.call(Client.java:1367)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy10.getServiceList(Unknown Source)
at 
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceList(OzoneManagerProtocolClientSideTranslatorPB.java:766)
at 
org.apache.hadoop.ozone.client.rpc.RpcClient.getScmAddressForClient(RpcClient.java:169)
at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:130)
... 19 more
java.lang.NullPointerException
{code}
Also verified using _jps_ that SCM, Datanode & OM are up and running.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDDS-152) Support HA for Ozone Manager

2018-11-14 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDDS-152.

Resolution: Fixed

Resolving as dup of HDDS-505. This was filed first but there is a design doc on 
the other Jira so let's continue there.

> Support HA for Ozone Manager
> 
>
> Key: HDDS-152
> URL: https://issues.apache.org/jira/browse/HDDS-152
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: DENG FEI
>Priority: Major
>
> Ozone Manager(OM) provide the name services on top of HDDS(SCM). This ticket 
> is opened to add HA support for OM. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Created] (HDFS-14082) RBF: Add option to fail operations when a subcluster is unavailable

2018-11-14 Thread JIRA
Íñigo Goiri created HDFS-14082:
--

 Summary: RBF: Add option to fail operations when a subcluster is 
unavailable
 Key: HDFS-14082
 URL: https://issues.apache.org/jira/browse/HDFS-14082
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Íñigo Goiri
Assignee: Íñigo Goiri


When a subcluster is unavailable, we succeed operations like {{getListing()}}. 
We should add an option to fail the operation if one of the subclusters is 
unavailable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDDS-838) Basic operations like create volume, freon are not working

2018-11-14 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDDS-838.

Resolution: Later

> Basic operations like create volume, freon are not working
> --
>
> Key: HDDS-838
> URL: https://issues.apache.org/jira/browse/HDDS-838
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Client, Ozone Manager
> Environment: JDK11
>Reporter: Dinesh Chitlangia
>Assignee: Dinesh Chitlangia
>Priority: Major
>
> After pulling latest from trunk, running simple operations like create 
> volume, freon rk are failing with following exception:
>  
> {code:java}
> MYBOX:ozone-0.4.0-SNAPSHOT dchitlangia$ bin/ozone sh volume create /test
> 2018-11-14 15:30:59,918 [main] ERROR - Couldn't create protocol class 
> org.apache.hadoop.ozone.client.rpc.RpcClient exception:
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169)
> at 
> org.apache.hadoop.ozone.web.ozShell.OzoneAddress.createClient(OzoneAddress.java:111)
> at 
> org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:70)
> at 
> org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:38)
> at picocli.CommandLine.execute(CommandLine.java:919)
> at picocli.CommandLine.access$700(CommandLine.java:104)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> at 
> picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:80)
> Caused by: 
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.ozone.om.OzoneManager.getServiceList(OzoneManager.java:1118)
> at 
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.getServiceList(OzoneManagerProtocolServerSideTranslatorPB.java:580)
> at 
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:39227)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1367)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy10.getServiceList(Unknown Source)
> at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceList(OzoneManagerProtocolClientSideTranslatorPB.java:766)
> at 
> org.apache.hadoop.ozone.client.rpc.RpcClient.getScmAddressForClient(RpcClient.java:169)
> at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:130)
> ... 19 more
> java.lang.NullPointerException
> {code}
> Also verified using _jps_ that SCM, Datanode & OM are up and running.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-14072) DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace.

2018-11-14 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDFS-14072.
--
Resolution: Duplicate

Resolving as a dup since you have posted a patch on HDFS-14074.

> DataNode runs async disk checks  maybe  throws NullPointerException, and 
> DataNode failed to register to NameSpace.
> --
>
> Key: HDFS-14072
> URL: https://issues.apache.org/jira/browse/HDFS-14072
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 2.8.0, 3.0.0
> Environment: hadoop-2.7.3, hadoop-2.8.0
>Reporter: guangyi lu
>Priority: Major
>  Labels: HDFS, HDFS-4
> Attachments: image-2018-11-13-20-23-22-138.png, 
> image-2018-11-13-20-23-26-813.png
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> In ThrottledAsyncChecker class,it members of the completedChecks is 
> WeakHashMap, its definition is as follows:
>      this.completedChecks = new WeakHashMap<>();
> and one of its uses is as follows in schedule method:
>     if (completedChecks.containsKey(target)) {
>         // here may be happen garbage collection,and result may be null.
>          final LastCheckResult result = completedChecks.get(target);
>          final long msSinceLastCheck = timer.monotonicNow() - 
> result.completedAt;
>     }
> after  "completedChecks.containsKey(target)",  may be happen garbage 
> collection,  and result may be null.
>  
> !image-2018-11-13-20-23-26-813.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-14073) DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace.

2018-11-14 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDFS-14073.
--
Resolution: Duplicate

> DataNode runs async disk checks  maybe  throws NullPointerException, and 
> DataNode failed to register to NameSpace.
> --
>
> Key: HDFS-14073
> URL: https://issues.apache.org/jira/browse/HDFS-14073
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 2.8.0, 3.0.0
> Environment: hadoop-2.7.3, hadoop-2.8.0
>Reporter: guangyi lu
>Priority: Major
>  Labels: HDFS, HDFS-4
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> In ThrottledAsyncChecker class,it members of the completedChecks is 
> WeakHashMap, its definition is as follows:
>      this.completedChecks = new WeakHashMap<>();
> and one of its uses is as follows in schedule method:
>     if (completedChecks.containsKey(target)) {
>         // here may be happen garbage collection,and result may be null.
>          final LastCheckResult result = completedChecks.get(target);
>          final long msSinceLastCheck = timer.monotonicNow() - 
> result.completedAt;
>     }
> after  "completedChecks.containsKey(target)",  may be happen garbage 
> collection,  and result may be null.
>  
> !image-2018-11-13-20-23-26-813.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-14071) DataNode runs async disk checks maybe throws NullPointerException In ThrottledAsyncChecker.java

2018-11-14 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDFS-14071.
--
Resolution: Duplicate

> DataNode runs async disk checks  maybe  throws NullPointerException In 
> ThrottledAsyncChecker.java  
> ---
>
> Key: HDFS-14071
> URL: https://issues.apache.org/jira/browse/HDFS-14071
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: guangyi lu
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



[jira] [Resolved] (HDDS-135) SCM CA: Update DelegationKey to retrieve private/public key

2018-11-14 Thread Xiaoyu Yao (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao resolved HDDS-135.
-
Resolution: Implemented

This has been implemented with HDDS-836.

> SCM CA: Update DelegationKey to retrieve private/public key
> ---
>
> Key: HDDS-135
> URL: https://issues.apache.org/jira/browse/HDDS-135
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



Re: [VOTE] Release Apache Hadoop 2.9.2 (RC0)

2018-11-14 Thread Takanobu Asanuma
Thanks for driving the release, Akira!
 
+1 (non-binding)
   - verified checksums
   - succeeded in building the package
   - started hadoop cluster with 1 master and 5 slaves
   - ran TeraGen/TeraSort
   - verified Web UI (NN, RM, JobHistory, Timeline)
   - verified some operations of Router-based Federation

Thanks,
-Takanobu

on 2018/11/14 10:02, "Akira Ajisaka" wrote:

Hi folks,

I have put together a release candidate (RC0) for Hadoop 2.9.2. It
includes 204 bug fixes and improvements since 2.9.1. [1]

The RC is available at http://home.apache.org/~aajisaka/hadoop-2.9.2-RC0/
Git signed tag is release-2.9.2-RC0 and the checksum is
826afbeae31ca687bc2f8471dc841b66ed2c6704
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1166/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Please try the release and vote. The vote will run for 5 days.

[1] https://s.apache.org/2.9.2-fixed-jiras

Thanks,
Akira

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org