[jira] [Created] (HADOOP-12783) TestWebDelegationToken failure: login options not compatible with IBM JDK

2016-02-09 Thread Devendra Vishwakarma (JIRA) 
Devendra Vishwakarma created HADOOP-12783:
-

 Summary: TestWebDelegationToken failure: login options not 
compatible with IBM JDK
 Key: HADOOP-12783
 URL: https://issues.apache.org/jira/browse/HADOOP-12783
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Affects Versions: 2.7.1
 Environment: IBM JDK 1.8 + s390x architecture
Reporter: Devendra Vishwakarma
Assignee: Devendra Vishwakarma
 Fix For: 2.7.1


When running test with IBM JDK, the testcase in 
/hadoop-common-/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken
 failed due to incompatible login options for IBM Java.
The login options need to be updated considering the IBM Java.

Testcases which failed are - 
1. 
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken.testKerberosDelegationTokenAuthenticator
2. 
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken.testKerberosDelegationTokenAuthenticatorWithDoAs

Testcases failed with the following stack:
javax.security.auth.login.LoginException: Bad JAAS configuration: unrecognized 
option: isInitiator
at 
com.ibm.security.jgss.i18n.I18NException.throwLoginException(I18NException.java:27)
at com.ibm.security.auth.module.Krb5LoginModule.d(Krb5LoginModule.java:541)
at com.ibm.security.auth.module.Krb5LoginModule.a(Krb5LoginModule.java:169)
at com.ibm.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:507)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:788)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:196)
at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721)
at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719)
at java.security.AccessController.doPrivileged(AccessController.java:595)
at 
javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:719)
at javax.security.auth.login.LoginContext.login(LoginContext.java:593)
at 
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken.doAsKerberosUser(TestWebDelegationToken.java:710)
at 
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken.testKerberosDelegationTokenAuthenticator(TestWebDelegationToken.java:777)
at 
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken.testKerberosDelegationTokenAuthenticator(TestWebDelegationToken.java:729)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12784) TestKMS failure: login options not compatible with IBM JDK

2016-02-09 Thread Devendra Vishwakarma (JIRA) 
Devendra Vishwakarma created HADOOP-12784:
-

 Summary: TestKMS failure: login options not compatible with IBM JDK
 Key: HADOOP-12784
 URL: https://issues.apache.org/jira/browse/HADOOP-12784
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Affects Versions: 2.7.1
 Environment: IBM JDK 1.8 + s390x architecture
Reporter: Devendra Vishwakarma
Assignee: Devendra Vishwakarma
 Fix For: 2.7.1


When running test with IBM JDK, the testcase in 
/hadoop-common-/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken
 failed due to incompatible login options for IBM Java.
The login options need to be updated considering the IBM Java.

Testcases which failed are - 
 1.org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSRestartKerberosAuth
 2.org.apache.hadoop.crypto.key.kms.server.TestKMS.testDelegationTokenAccess
 3.org.apache.hadoop.crypto.key.kms.server.TestKMS.testServicePrincipalACLs
 4.org.apache.hadoop.crypto.key.kms.server.TestKMS.testKeyACLs
 5.org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSWithZKDTSM
 6.org.apache.hadoop.crypto.key.kms.server.TestKMS.testACLs
 7.org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSWithZKSignerAndDTSM
 8.org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSWithZKSigner
 9.org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSRestartSimpleAuth
 10.org.apache.hadoop.crypto.key.kms.server.TestKMS.testStartStopHttpKerberos
 11.org.apache.hadoop.crypto.key.kms.server.TestKMS.testWebHDFSProxyUserKerb
 12.org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSBlackList
 13.org.apache.hadoop.crypto.key.kms.server.TestKMS.testStartStopHttpsKerberos
 14.org.apache.hadoop.crypto.key.kms.server.TestKMS.testKMSAuthFailureRetry

Testcases failed with the following stack:
javax.security.auth.login.LoginException: Bad JAAS configuration: unrecognized 
option: isInitiator
at 
com.ibm.security.jgss.i18n.I18NException.throwLoginException(I18NException.java:27)
at com.ibm.security.auth.module.Krb5LoginModule.d(Krb5LoginModule.java:541)
at com.ibm.security.auth.module.Krb5LoginModule.a(Krb5LoginModule.java:169)
at com.ibm.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:507)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:788)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:196)
at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721)
at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719)
at java.security.AccessController.doPrivileged(AccessController.java:595)
at 
javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:719)
at javax.security.auth.login.LoginContext.login(LoginContext.java:593)
at org.apache.hadoop.crypto.key.kms.server.TestKMS.doAs(TestKMS.java:262)
at org.apache.hadoop.crypto.key.kms.server.TestKMS.access$100(TestKMS.java:75)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12785) [Handling exceptions] LdapGroupsMapping.getGroups() do not provide information about root cause

2016-02-09 Thread Mukhadin Buzdov (JIRA) 
Mukhadin Buzdov created HADOOP-12785:


 Summary: [Handling exceptions] LdapGroupsMapping.getGroups() do 
not provide information about root cause
 Key: HADOOP-12785
 URL: https://issues.apache.org/jira/browse/HADOOP-12785
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.7.1
 Environment: _Operating system_: CentOS Linux 7 
{color:gray}(7.1.1503){color}
_Platform_: HDP 2.3.4.0, Ambari 2.1.2
Reporter: Mukhadin Buzdov
Priority: Minor


_CommunicationException_ and _NamingException_ are not logged in 
_LdapGroupsMapping.getGroups()_.

{code:title=LdapGroupsMapping.java|borderStyle=solid}
  public synchronized List getGroups(String user) throws IOException {
List emptyResults = new ArrayList();
// ...
try {
  return doGetGroups(user);
} catch (CommunicationException e) {
  LOG.warn("Connection is closed, will try to reconnect");
} catch (NamingException e) {
  LOG.warn("Exception trying to get groups for user " + user + ": " + 
e.getMessage());
  return emptyResults;
}
//...
return emptyResults;
  }
{code}

{color:red}It is not possible to understand _LDAP_ level failures.{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

2016-02-09 Thread Wangda Tan 
Hi Eric,

replaceLabelsOnNode CLI syntax is different between 2.6 and 2.7.

As mentioned by Naga, In 2.6, you should use "node1<:port>,label
node2,label ..."
And in 2.7, you can use either "," or "=" to separate host and label.

So 2.7 is backward-compatible to 2.6, but it's not guaranteed that 2.7's
CLI works on 2.6.

Please let me know if you have any concern for this.

Thanks,
Wangda

On Tue, Feb 9, 2016 at 4:19 AM, Eric Payne  wrote:

> Naganarasimha Garla, thanks for the reply.
>
> Yes, I used the node ID. I did not include a port. Here are the steps I
> used, which work for me on 2.7:
>
> - yarn rmadmin -addToClusterNodeLabels abc
> - yarn rmadmin -replaceLabelsOnNode hostname.company.com=abc
> - configure queue properties as appropriate
> - yarn rmadmin -refreshQueues
>
> As I say, this works for me when I try it on 2.7 and later. It's probably
> something with my environment. I will continue to look into it.
>
> Thanks for your help
> -Eric
>
>
> 
> From: Naganarasimha Garla 
> To: mapreduce-...@hadoop.apache.org; Eric Payne 
> Cc: "common-dev@hadoop.apache.org" ; "
> hdfs-...@hadoop.apache.org" ; "
> yarn-...@hadoop.apache.org" 
> Sent: Monday, February 8, 2016 1:01 PM
> Subject: Re: [VOTE] Release Apache Hadoop 2.6.4 RC0
>
>
>
> +1 (non binding)
>
> * Downloaded hadoop-2.6.4-RC0-src.tar.gz- built from source both package,
> install, and verified the MD5 checksum
>
> * Did a Pseudo cluster and tested basic hdfs operations
> * Ran sleep job and Pi job
> * Added node label and ran job under the label by configuring
> default-node-label-expression and it ran fine
>
> Eric Payne,
> Hope you tried adding/replacing the labels using NodeId/Node Address and
> not the HTTP address!
> I executed the following command to configure the label and node
>   "./yarn rmadmin -replaceLabelsOnNode  "localhost:43795,test1"  "
> After this was able to submit the job for a label
>
> Regards,
> + Naga
>
>
> On Mon, Feb 8, 2016 at 11:06 PM, Eric Payne
>  wrote:
>
> Hi Junping Du. Thank you for your work preparing this release.
> >I did the following things to test release Hadoop 2.6.4 rc0:- Downloaded
> hadoop-2.6.4-RC0-src.tar.gz- built from source both package, install, and
> eclipse:eclipse- Set up a 3-node, unsecured cluster with 3 queues, one of
> which has preemption enabled- Ran a successful test to ensure that
> preemption would happen to containers on the preemptable queue if they were
> needed for an application on another queue.- Ran successful streaming and
> yarn shell tests
> >Junping, I did have a concern about labelled nodes and queues. Is full
> label support backported to 2.6.4? I see that the syntax for the rmadmin
> command lists label commands like -addToClusterNodeLabels and
> -replaceLabelsOnNode. I was able to add a label (using
> -addToClusterNodeLabels) and I was able to define a queue whose accessible
> node label was listed with my specified label. However, when I tried to set
> the node label to a specific node using -replaceLabelsOnNode, the label
> does not show up on the specified node in cluster nodes UI (
> http://RM:8088/cluster/nodes). I also confirmed that submitting a job to
> the labelled queue gets accepted but never runs, which is the behavior I
> would expect if no node had the specified label. I will also add that this
> procedure works fine in 2.7.
> >Thanks,-Eric Payne
> >
> >  From: Junping Du 
> > To: "hdfs-...@hadoop.apache.org" ; "
> yarn-...@hadoop.apache.org" ; "
> mapreduce-...@hadoop.apache.org" ; "
> common-dev@hadoop.apache.org" 
> > Sent: Wednesday, February 3, 2016 1:01 AM
> > Subject: [VOTE] Release Apache Hadoop 2.6.4 RC0
> >
> >
> >Hi community folks,
> >  I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
> maintenance release to follow up 2.6.3.) according to email thread of
> release plan 2.6.4 [1]. Below is details of this release candidate:
> >
> >The RC is available for validation at:
> >*http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
> >*
> >
> >The RC tag in git is: release-2.6.4-RC0
> >
> >The maven artifacts are staged via repository.apache.org at:
> >*https://repository.apache.org/content/repositories/orgapachehadoop-1028/
> ?
> > >*
> >
> >You can find my public key at:
> >http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
> >
> >Please try the release and vote. The vote will run for the usual 5 days.
> >
> >Thanks!
> >
> >
> >Cheers,
> >
> >Junping
> >
> >
> >[1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
> >
> >
> >
>

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

2016-02-09 Thread Wangda Tan 
+1 (binding)

- Deployed a local cluster.
- Configured node labels on queues/nodes.
* Run job with node labels successfully.

Regards,
Wangda

On Tue, Feb 9, 2016 at 10:36 PM, Wangda Tan  wrote:

> Hi Eric,
>
> replaceLabelsOnNode CLI syntax is different between 2.6 and 2.7.
>
> As mentioned by Naga, In 2.6, you should use "node1<:port>,label
> node2,label ..."
> And in 2.7, you can use either "," or "=" to separate host and label.
>
> So 2.7 is backward-compatible to 2.6, but it's not guaranteed that 2.7's
> CLI works on 2.6.
>
> Please let me know if you have any concern for this.
>
> Thanks,
> Wangda
>
> On Tue, Feb 9, 2016 at 4:19 AM, Eric Payne <
> eric.payne1...@yahoo.com.invalid> wrote:
>
>> Naganarasimha Garla, thanks for the reply.
>>
>> Yes, I used the node ID. I did not include a port. Here are the steps I
>> used, which work for me on 2.7:
>>
>> - yarn rmadmin -addToClusterNodeLabels abc
>> - yarn rmadmin -replaceLabelsOnNode hostname.company.com=abc
>> - configure queue properties as appropriate
>> - yarn rmadmin -refreshQueues
>>
>> As I say, this works for me when I try it on 2.7 and later. It's probably
>> something with my environment. I will continue to look into it.
>>
>> Thanks for your help
>> -Eric
>>
>>
>> 
>> From: Naganarasimha Garla 
>> To: mapreduce-...@hadoop.apache.org; Eric Payne > >
>> Cc: "common-dev@hadoop.apache.org" ; "
>> hdfs-...@hadoop.apache.org" ; "
>> yarn-...@hadoop.apache.org" 
>> Sent: Monday, February 8, 2016 1:01 PM
>> Subject: Re: [VOTE] Release Apache Hadoop 2.6.4 RC0
>>
>>
>>
>> +1 (non binding)
>>
>> * Downloaded hadoop-2.6.4-RC0-src.tar.gz- built from source both package,
>> install, and verified the MD5 checksum
>>
>> * Did a Pseudo cluster and tested basic hdfs operations
>> * Ran sleep job and Pi job
>> * Added node label and ran job under the label by configuring
>> default-node-label-expression and it ran fine
>>
>> Eric Payne,
>> Hope you tried adding/replacing the labels using NodeId/Node Address and
>> not the HTTP address!
>> I executed the following command to configure the label and node
>>   "./yarn rmadmin -replaceLabelsOnNode  "localhost:43795,test1"  "
>> After this was able to submit the job for a label
>>
>> Regards,
>> + Naga
>>
>>
>> On Mon, Feb 8, 2016 at 11:06 PM, Eric Payne
>>  wrote:
>>
>> Hi Junping Du. Thank you for your work preparing this release.
>> >I did the following things to test release Hadoop 2.6.4 rc0:- Downloaded
>> hadoop-2.6.4-RC0-src.tar.gz- built from source both package, install, and
>> eclipse:eclipse- Set up a 3-node, unsecured cluster with 3 queues, one of
>> which has preemption enabled- Ran a successful test to ensure that
>> preemption would happen to containers on the preemptable queue if they were
>> needed for an application on another queue.- Ran successful streaming and
>> yarn shell tests
>> >Junping, I did have a concern about labelled nodes and queues. Is full
>> label support backported to 2.6.4? I see that the syntax for the rmadmin
>> command lists label commands like -addToClusterNodeLabels and
>> -replaceLabelsOnNode. I was able to add a label (using
>> -addToClusterNodeLabels) and I was able to define a queue whose accessible
>> node label was listed with my specified label. However, when I tried to set
>> the node label to a specific node using -replaceLabelsOnNode, the label
>> does not show up on the specified node in cluster nodes UI (
>> http://RM:8088/cluster/nodes). I also confirmed that submitting a job to
>> the labelled queue gets accepted but never runs, which is the behavior I
>> would expect if no node had the specified label. I will also add that this
>> procedure works fine in 2.7.
>> >Thanks,-Eric Payne
>> >
>> >  From: Junping Du 
>> > To: "hdfs-...@hadoop.apache.org" ; "
>> yarn-...@hadoop.apache.org" ; "
>> mapreduce-...@hadoop.apache.org" ; "
>> common-dev@hadoop.apache.org" 
>> > Sent: Wednesday, February 3, 2016 1:01 AM
>> > Subject: [VOTE] Release Apache Hadoop 2.6.4 RC0
>> >
>> >
>> >Hi community folks,
>> >  I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
>> maintenance release to follow up 2.6.3.) according to email thread of
>> release plan 2.6.4 [1]. Below is details of this release candidate:
>> >
>> >The RC is available for validation at:
>> >*http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
>> >*
>> >
>> >The RC tag in git is: release-2.6.4-RC0
>> >
>> >The maven artifacts are staged via repository.apache.org at:
>> >*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
>> ><
>> https://repository.apache.org/content/repositories/orgapachehadoop-1028/
>> >*
>> >
>> >You can find my public key at:
>> >http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>> >
>> >Please try the release and vote. The vote will run for the usual 5 days.
>> >
>> >Thanks!
>> >
>> >
>> >Cheers,
>> >
>> >Junping
>> >
>> >
>> >[1]: 2.6.4 release

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

2016-02-09 Thread Kuhu Shukla 
+1 (non-binding)
- Built source- Deployed a single node cluster- Ran a sample streaming job- Ran 
basic hadoop fs commands.
Thanks Junping!Regards,Kuhu 

On Tuesday, February 9, 2016 8:46 AM, Wangda Tan  
wrote:
 

 +1 (binding)

- Deployed a local cluster.
- Configured node labels on queues/nodes.
* Run job with node labels successfully.

Regards,
Wangda

On Tue, Feb 9, 2016 at 10:36 PM, Wangda Tan  wrote:

> Hi Eric,
>
> replaceLabelsOnNode CLI syntax is different between 2.6 and 2.7.
>
> As mentioned by Naga, In 2.6, you should use "node1<:port>,label
> node2,label ..."
> And in 2.7, you can use either "," or "=" to separate host and label.
>
> So 2.7 is backward-compatible to 2.6, but it's not guaranteed that 2.7's
> CLI works on 2.6.
>
> Please let me know if you have any concern for this.
>
> Thanks,
> Wangda
>
> On Tue, Feb 9, 2016 at 4:19 AM, Eric Payne <
> eric.payne1...@yahoo.com.invalid> wrote:
>
>> Naganarasimha Garla, thanks for the reply.
>>
>> Yes, I used the node ID. I did not include a port. Here are the steps I
>> used, which work for me on 2.7:
>>
>> - yarn rmadmin -addToClusterNodeLabels abc
>> - yarn rmadmin -replaceLabelsOnNode hostname.company.com=abc
>> - configure queue properties as appropriate
>> - yarn rmadmin -refreshQueues
>>
>> As I say, this works for me when I try it on 2.7 and later. It's probably
>> something with my environment. I will continue to look into it.
>>
>> Thanks for your help
>> -Eric
>>
>>
>> 
>> From: Naganarasimha Garla 
>> To: mapreduce-...@hadoop.apache.org; Eric Payne > >
>> Cc: "common-dev@hadoop.apache.org" ; "
>> hdfs-...@hadoop.apache.org" ; "
>> yarn-...@hadoop.apache.org" 
>> Sent: Monday, February 8, 2016 1:01 PM
>> Subject: Re: [VOTE] Release Apache Hadoop 2.6.4 RC0
>>
>>
>>
>> +1 (non binding)
>>
>> * Downloaded hadoop-2.6.4-RC0-src.tar.gz- built from source both package,
>> install, and verified the MD5 checksum
>>
>> * Did a Pseudo cluster and tested basic hdfs operations
>> * Ran sleep job and Pi job
>> * Added node label and ran job under the label by configuring
>> default-node-label-expression and it ran fine
>>
>> Eric Payne,
>> Hope you tried adding/replacing the labels using NodeId/Node Address and
>> not the HTTP address!
>> I executed the following command to configure the label and node
>>  "./yarn rmadmin -replaceLabelsOnNode  "localhost:43795,test1"  "
>> After this was able to submit the job for a label
>>
>> Regards,
>> + Naga
>>
>>
>> On Mon, Feb 8, 2016 at 11:06 PM, Eric Payne
>>  wrote:
>>
>> Hi Junping Du. Thank you for your work preparing this release.
>> >I did the following things to test release Hadoop 2.6.4 rc0:- Downloaded
>> hadoop-2.6.4-RC0-src.tar.gz- built from source both package, install, and
>> eclipse:eclipse- Set up a 3-node, unsecured cluster with 3 queues, one of
>> which has preemption enabled- Ran a successful test to ensure that
>> preemption would happen to containers on the preemptable queue if they were
>> needed for an application on another queue.- Ran successful streaming and
>> yarn shell tests
>> >Junping, I did have a concern about labelled nodes and queues. Is full
>> label support backported to 2.6.4? I see that the syntax for the rmadmin
>> command lists label commands like -addToClusterNodeLabels and
>> -replaceLabelsOnNode. I was able to add a label (using
>> -addToClusterNodeLabels) and I was able to define a queue whose accessible
>> node label was listed with my specified label. However, when I tried to set
>> the node label to a specific node using -replaceLabelsOnNode, the label
>> does not show up on the specified node in cluster nodes UI (
>> http://RM:8088/cluster/nodes). I also confirmed that submitting a job to
>> the labelled queue gets accepted but never runs, which is the behavior I
>> would expect if no node had the specified label. I will also add that this
>> procedure works fine in 2.7.
>> >Thanks,-Eric Payne
>> >
>> >      From: Junping Du 
>> > To: "hdfs-...@hadoop.apache.org" ; "
>> yarn-...@hadoop.apache.org" ; "
>> mapreduce-...@hadoop.apache.org" ; "
>> common-dev@hadoop.apache.org" 
>> > Sent: Wednesday, February 3, 2016 1:01 AM
>> > Subject: [VOTE] Release Apache Hadoop 2.6.4 RC0
>> >
>> >
>> >Hi community folks,
>> >  I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
>> maintenance release to follow up 2.6.3.) according to email thread of
>> release plan 2.6.4 [1]. Below is details of this release candidate:
>> >
>> >The RC is available for validation at:
>> >*http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
>> >*
>> >
>> >The RC tag in git is: release-2.6.4-RC0
>> >
>> >The maven artifacts are staged via repository.apache.org at:
>> >*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
>> ><
>> https://repository.apache.org/content/repositories/orgapachehadoop-1028/
>> >*
>> >
>> >You can find my public key at:
>> >http://

Build failed in Jenkins: Hadoop-common-trunk-Java8 #1054

2016-02-09 Thread Apache Jenkins Server 
See 

Changes:

[jlowe] Update CHANGES.txt to reflect recent commits to branch-2.7 and

--
[...truncated 5588 lines...]
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.compress.TestCodecPool
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 sec - in 
org.apache.hadoop.io.compress.TestCodecPool
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.compress.snappy.TestSnappyCompressorDecompressor
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 sec - in 
org.apache.hadoop.io.compress.snappy.TestSnappyCompressorDecompressor
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.compress.TestBlockDecompressorStream
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 sec - in 
org.apache.hadoop.io.compress.TestBlockDecompressorStream
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.compress.TestCompressorStream
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.078 sec - in 
org.apache.hadoop.io.compress.TestCompressorStream
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestMD5Hash
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.223 sec - in 
org.apache.hadoop.io.TestMD5Hash
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSecureIOUtils
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.669 sec - in 
org.apache.hadoop.io.TestSecureIOUtils
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestMapFile
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.278 sec - in 
org.apache.hadoop.io.TestMapFile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestGenericWritable
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.437 sec - in 
org.apache.hadoop.io.TestGenericWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.nativeio.TestNativeIO
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.705 sec - in 
org.apache.hadoop.io.nativeio.TestNativeIO
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.nativeio.TestSharedFileDescriptorFactory
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.377 sec - in 
org.apache.hadoop.io.nativeio.TestSharedFileDescriptorFactory
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestDefaultStringifier
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.428 sec - in 
org.apache.hadoop.io.TestDefaultStringifier
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestArrayPrimitiveWritable
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.209 sec - in 
org.apache.hadoop.io.TestArrayPrimitiveWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestUTF8
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.572 sec - in 
org.apache.hadoop.io.TestUTF8
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestBloomMapFile
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.407 sec - in 
org.apache.hadoop.io.TestBloomMapFile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.retry.TestRetryProxy
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.241 sec - in 
org.apache.hadoop.io.retry.TestRetryProxy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.retry.TestDefaultRetryPolicy
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.366 sec - in 
org.apache.hadoop.io.retry.TestDefaultRetryPolicy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.retry.TestFailoverProxy
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

2016-02-09 Thread Sunil Govind 
+1 (non-binding) with one note,

- Installed tar ball from source and brought up the cluster
- Ran few MR jobs and those are working fine.
- UI and REST apis are also looks fine. Ran few rest queries for this test.
- Tested node label feature and works fine. (one observation during this
testing)

One note to add here. "-Dmapreduce.job.node-label-expression" support is
not there 2.6.4. (MAPREDUCE-6304). So we cannot specify label to
application while submitting.  I think this also can be ported to 2.6
branch so it will help to specify labels at submission time.

Thanks and Regards
Sunil

On Wed, Feb 3, 2016 at 12:31 PM Junping Du  wrote:

> Hi community folks,
>I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
> maintenance release to follow up 2.6.3.) according to email thread of
> release plan 2.6.4 [1]. Below is details of this release candidate:
>
> The RC is available for validation at:
> *http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
> *
>
> The RC tag in git is: release-2.6.4-RC0
>
> The maven artifacts are staged via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
>  >*
>
> You can find my public key at:
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the usual 5 days.
>
> Thanks!
>
>
> Cheers,
>
> Junping
>
>
> [1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>
>

Jenkins build is back to normal : Hadoop-Common-trunk #2352

2016-02-09 Thread Apache Jenkins Server 
See

[jira] [Created] (HADOOP-12786) "hadoop key" command usage is not documented

2016-02-09 Thread Akira AJISAKA (JIRA) 
Akira AJISAKA created HADOOP-12786:
--

 Summary: "hadoop key" command usage is not documented
 Key: HADOOP-12786
 URL: https://issues.apache.org/jira/browse/HADOOP-12786
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: Akira AJISAKA


I found "hadoop key" command usage is not documented when reviewing HDFS-9784.
In addition, we should document that uppercase is not allowed for key name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Release thread] 2.8.0 release activities

2016-02-09 Thread Vinod Kumar Vavilapalli 
Sure. The last time I checked, there were 20 odd blocker/critical tickets too 
that’ll need some of my time.

Given that, if you can get them in before a week, we should be good.

+Vinod

> On Feb 5, 2016, at 1:19 PM, Subramaniam V K  wrote:
> 
> Vinod,
> 
> Thanks for initiating the 2.8 release thread. We are in late review stages
> for YARN-4420 (Add REST API for listing reservations) and YARN-2575 (Adding
> ACLs for reservation system), hoping to get them by next week. Any chance
> you can put off cutting 2.8 by a week as we are planning to deploy
> ReservationSystem and these are critical for that?
> 
> Cheers,
> Subru
> 
> On Thu, Feb 4, 2016 at 3:17 PM, Chris Nauroth 
> wrote:
> 
>> FYI, I've just needed to raise HDFS-9761 to blocker status for the 2.8.0
>> release.
>> 
>> --Chris Nauroth
>> 
>> 
>> 
>> 
>> On 2/3/16, 6:19 PM, "Karthik Kambatla"  wrote:
>> 
>>> Thanks Vinod. Not labeling 2.8.0 stable sounds perfectly reasonable to me.
>>> Let us not call it alpha or beta though, it is quite confusing. :)
>>> 
>>> On Wed, Feb 3, 2016 at 8:17 PM, Gangumalla, Uma >> 
>>> wrote:
>>> 
 Thanks Vinod. +1 for 2.8 release start.
 
 Regards,
 Uma
 
 On 2/3/16, 3:53 PM, "Vinod Kumar Vavilapalli" 
 wrote:
 
> Seems like all the features listed in the Roadmap wiki are in. I¹m
 going
> to try cutting an RC this weekend for a first/non-stable release off of
> branch-2.8.
> 
> Let me know if anyone has any objections/concerns.
> 
> Thanks
> +Vinod
> 
>> On Nov 25, 2015, at 5:59 PM, Vinod Kumar Vavilapalli
>>  wrote:
>> 
>> Branch-2.8 is created.
>> 
>> As mentioned before, the goal on branch-2.8 is to put improvements /
>> fixes to existing features with a goal of converging on an alpha
 release
>> soon.
>> 
>> Thanks
>> +Vinod
>> 
>> 
>>> On Nov 25, 2015, at 5:30 PM, Vinod Kumar Vavilapalli
>>>  wrote:
>>> 
>>> Forking threads now in order to track all things related to the
>>> release.
>>> 
>>> Creating the branch now.
>>> 
>>> Thanks
>>> +Vinod
>>> 
>>> 
 On Nov 25, 2015, at 11:37 AM, Vinod Kumar Vavilapalli
  wrote:
 
 I think we¹ve converged at a high level w.r.t 2.8. And as I just
 sent
 out an email, I updated the Roadmap wiki reflecting the same:
 https://wiki.apache.org/hadoop/Roadmap
 
 
 I plan to create a 2.8 branch EOD today.
 
 The goal for all of us should be to restrict improvements & fixes
 to
 only (a) the feature-set documented under 2.8 in the RoadMap wiki
 and
 (b) other minor features that are already in 2.8.
 
 Thanks
 +Vinod
 
 
> On Nov 11, 2015, at 12:13 PM, Vinod Kumar Vavilapalli
> mailto:vino...@hortonworks.com>> wrote:
> 
> - Cut a branch about two weeks from now
> - Do an RC mid next month (leaving ~4weeks since branch-cut)
> - As with 2.7.x series, the first release will still be called as
> early / alpha release in the interest of
>  ‹ gaining downstream adoption
>  ‹ wider testing,
>  ‹ yet reserving our right to fix any inadvertent
 incompatibilities
> introduced.
 
>>> 
>> 
> 
 
 
>> 
>>

Jenkins build is back to normal : Hadoop-common-trunk-Java8 #1055

2016-02-09 Thread Apache Jenkins Server 
See

[jira] [Created] (HADOOP-12787) KMS SPNEGO sequence does not work with WEBHDFS

2016-02-09 Thread Xiaoyu Yao (JIRA) 
Xiaoyu Yao created HADOOP-12787:
---

 Summary: KMS SPNEGO sequence does not work with WEBHDFS
 Key: HADOOP-12787
 URL: https://issues.apache.org/jira/browse/HADOOP-12787
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This was a follow up of my 
[comments|https://issues.apache.org/jira/browse/HADOOP-12559?focusedCommentId=15059045&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15059045]
 for HADOOP-10698.

It blocks a delegation token based user (MR) using WEBHDFS to access KMS server 
for encrypted files. This might work in many cases before as JDK 7 has been 
aggressively do SPENGO implicitly. However, this is not the case in JDK 8 as we 
have seen many failures when using WEBHDFS with KMS and HDFS encryption zone.

 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12788) OpensslAesCtrCryptoCodec should log what random number generator is used.

2016-02-09 Thread Wei-Chiu Chuang (JIRA) 
Wei-Chiu Chuang created HADOOP-12788:


 Summary: OpensslAesCtrCryptoCodec should log what random number 
generator is used.
 Key: HADOOP-12788
 URL: https://issues.apache.org/jira/browse/HADOOP-12788
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang
Priority: Minor


{{OpensslAesCtrCryptoCodec}} uses random number generator, for example, 
{{OsSecureRandom}}, {{}OpensslSecureRandom} or {{SecureRandom}} but it's not 
clear which one would be loaded at runtime.

It would help debugging if we can print a debug message that states which one 
is loaded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

2016-02-09 Thread Tsuyoshi Ozawa 
+1(binding)

- Verified signatures.
- Ran tests for Apache Tez with the artifacts. All tests passed.
- Ran local cluster and ran some examples on it.

- Tsuyoshi

On Wed, Feb 10, 2016 at 2:18 AM, Sunil Govind  wrote:
> +1 (non-binding) with one note,
>
> - Installed tar ball from source and brought up the cluster
> - Ran few MR jobs and those are working fine.
> - UI and REST apis are also looks fine. Ran few rest queries for this test.
> - Tested node label feature and works fine. (one observation during this
> testing)
>
> One note to add here. "-Dmapreduce.job.node-label-expression" support is
> not there 2.6.4. (MAPREDUCE-6304). So we cannot specify label to
> application while submitting.  I think this also can be ported to 2.6
> branch so it will help to specify labels at submission time.
>
> Thanks and Regards
> Sunil
>
> On Wed, Feb 3, 2016 at 12:31 PM Junping Du  wrote:
>
>> Hi community folks,
>>I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
>> maintenance release to follow up 2.6.3.) according to email thread of
>> release plan 2.6.4 [1]. Below is details of this release candidate:
>>
>> The RC is available for validation at:
>> *http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
>> *
>>
>> The RC tag in git is: release-2.6.4-RC0
>>
>> The maven artifacts are staged via repository.apache.org at:
>> *https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
>> > >*
>>
>> You can find my public key at:
>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>>
>> Please try the release and vote. The vote will run for the usual 5 days.
>>
>> Thanks!
>>
>>
>> Cheers,
>>
>> Junping
>>
>>
>> [1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>>
>>

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

2016-02-09 Thread Junping Du 
Thanks Yongjun for identifying and proposing this change to 2.6.4. I think this 
is the right thing to do and check for following releases. For 2.6.4, it seems 
unnecessary to create another release candidate for this issue as we only 
kicking off a new RC build when last RC has serious problem in functionality. 
The vote progress is quite smoothly so far, so it seems unlikely that we will 
create a new RC. However, I think there are still two options here:
Option 1:  in final build, adopt change of HDFS-9629 that only updates the 
footer of Web UI to show year 2016.
Option 2: skip HDFS-9629 for 2.6.4 and adopt it later for 2.6.5.
I prefer Option 1 as this is a very low risky change without affecting any 
functionality, and we allow non-functional changes (like release date, etc.) 
happen on final build after RC passed. I would like to hear the voices in 
community here before acting for the next step. Thoughts?

Thanks,

Junping

From: Yongjun Zhang 
Sent: Wednesday, February 03, 2016 8:54 AM
To: Hadoop Common
Cc: hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

Hi Junping,

Thanks a lot for leading the effort on 2.6.4.

We were discussing in HDFS-9629, and have agreement on adding an additional
step in the release process to update the year in the webui footer. See

https://issues.apache.org/jira/browse/HDFS-9629?focusedCommentId=15130043&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15130043

I noticed that the current RC for 2.6.4 still has the year of 2014 (the
example steps in the above link is validated on 2.6.4 RC0). So the steps I
proposed in the link could be applied when we create new RCs.

Thanks Akira for +1 on this proposed additional release step. Once we have
an agreement on the details, we will update the wiki page accordingly.

Best,

--Yongjun


On Tue, Feb 2, 2016 at 11:01 PM, Junping Du  wrote:

> Hi community folks,
>I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
> maintenance release to follow up 2.6.3.) according to email thread of
> release plan 2.6.4 [1]. Below is details of this release candidate:
>
> The RC is available for validation at:
> *http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
> *
>
> The RC tag in git is: release-2.6.4-RC0
>
> The maven artifacts are staged via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
>  >*
>
> You can find my public key at:
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the usual 5 days.
>
> Thanks!
>
>
> Cheers,
>
> Junping
>
>
> [1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>
>

[GitHub] hadoop pull request: Merge pull request #1 from apache/trunk

2016-02-09 Thread 2899722744 
GitHub user 2899722744 opened a pull request:

https://github.com/apache/hadoop/pull/74

Merge pull request #1 from apache/trunk

sync

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/2899722744/hadoop trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/74.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #74


commit 14f060bd20b2aa31abd4e8a19cd204a3b6cce316
Author: 2899722744 <2899722...@qq.com>
Date:   2016-02-02T11:40:24Z

Merge pull request #1 from apache/trunk

sync




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hadoop pull request: Merge pull request #1 from apache/trunk

2016-02-09 Thread 2899722744 
Github user 2899722744 closed the pull request at:

https://github.com/apache/hadoop/pull/74


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hadoop pull request: Merge pull request #1 from apache/trunk

2016-02-09 Thread 2899722744 
Github user 2899722744 commented on the pull request:

https://github.com/apache/hadoop/pull/74#issuecomment-182214122
  
sync


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---