Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/46/

No changes




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-hdfs:22 
   hadoop-yarn-server-timelineservice:1 
   hadoop-yarn-client:4 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-mapreduce-client-jobclient:7 
   hadoop-distcp:3 
   hadoop-extras:1 
   hadoop-sls:1 

Failed junit tests :

   hadoop.hdfs.TestBlocksScheduledCounter 
   hadoop.hdfs.TestDatanodeRegistration 
   hadoop.hdfs.TestTrashWithEncryptionZones 
   hadoop.hdfs.TestListFilesInDFS 
   
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels 
   hadoop.mapreduce.lib.input.TestMultipleInputs 
   hadoop.mapreduce.lib.output.TestJobOutputCommitter 
   hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedComparator 
   hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities 
   hadoop.mapreduce.TestMapReduceLazyOutput 
   hadoop.mapreduce.lib.input.TestMRSequenceFileInputFilter 
   hadoop.mapreduce.lib.input.TestNLineInputFormat 
   hadoop.mapreduce.v2.TestNonExistentJob 
   hadoop.mapreduce.v2.TestMiniMRProxyUser 
   hadoop.mapreduce.lib.input.TestMRSequenceFileAsBinaryInputFormat 
   hadoop.mapreduce.v2.TestMRAppWithCombiner 
   hadoop.mapreduce.TestLocalRunner 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   hadoop.mapreduce.v2.TestMRJobs 
   hadoop.mapreduce.lib.output.TestMRMultipleOutputs 
   hadoop.mapreduce.TestMapperReducerCleanup 
   hadoop.mapreduce.v2.TestRMNMInfo 
   hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp 
   hadoop.mapreduce.v2.TestSpeculativeExecution 
   hadoop.mapreduce.v2.TestMROldApiJobs 
   hadoop.mapreduce.TestValueIterReset 
   hadoop.mapreduce.lib.input.TestLineRecordReaderJobs 
   hadoop.mapreduce.v2.TestMRJobsWithHistoryService 
   hadoop.mapreduce.lib.chain.TestChainErrors 
   hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat 
   hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection 
   hadoop.mapreduce.lib.output.TestMRSequenceFileAsBinaryOutputFormat 
   hadoop.mapreduce.TestLargeSort 
   hadoop.mapreduce.lib.input.TestMRSequenceFileAsTextInputFormat 
   hadoop.mapreduce.lib.chain.TestMapReduceChain 
   hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat 
   hadoop.mapreduce.lib.output.TestMRCJCFileOutputCommitter 
   hadoop.tools.TestIntegration 
   hadoop.tools.TestDistCpViewFs 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.TestReservationSystemInvariants 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 

Timed out junit tests :

   org.apache.hadoop.hdfs.TestLeaseRecovery2 
   org.apache.hadoop.hdfs.TestDFSClientFailover 
   org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream 
   org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade 
   org.apache.hadoop.hdfs.TestFileAppendRestart 
   org.apache.hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter 
   org.apache.hadoop.hdfs.TestDFSMkdirs 
   org.apache.hadoop.hdfs.TestDFSOutputStream 
   org.apache.hadoop.hdfs.TestDatanodeReport 
   org.apache.hadoop.hdfs.web.TestWebHDFS 
   org.apache.hadoop.hdfs.web.TestWebHDFSXAttr 
   org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes 
   org.apache.hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs 
   org.apache.hadoop.hdfs.TestMiniDFSCluster 
   org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs 
   org.apache.hadoop.hdfs.TestDistributedFileSystem 
   org.apache.hadoop.hdfs.web.TestWebHDFSForHA 
   org.apache.hadoop.hdfs.TestReplaceDatanodeFailureReplication 
   org.apache.hadoop.hdfs.TestDFSShell 
   org.apache.hadoop.hdfs.web.TestWebHDFSAcl 
   
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServices
 
   org.apache.hadoop.yarn.client.TestRMFailover 
   org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA 
   org.apache.hadoop.yarn.client.api.impl.TestYarnClient 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
   
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell 
   org.apache.hadoop.mapreduce.TestNewCombinerGrouping 
   org.apache.hadoop.mapreduce.TestMRJobClient 
   org.apache.hadoop.mapred.TestClusterMapReduceTestCase 
   org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge 
   org.apache.hadoop.mapreduce.Te

[jira] [Reopened] (HDFS-12820) Decommissioned datanode is counted in service cause datanode allcating failure

[Gang Xie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gang Xie reopened HDFS-12820:
-

After carefully check the issue reported in HDFS-9279, and found this issue is 
not a its dup.

this case is mainly about the param {color:#d04437}nodesInService{color}. When 
a datanode is decommissioned and then dead. nodesInService will not be 
subtracted.  
Then when allocation, the dead node will be counted in the maxload, which makes 
the maxload very low, in turns, causes any allocation failing.

if (considerLoad) {
{color:#d04437}  final double maxLoad = maxLoadRatio * 
stats.getInServiceXceiverAverage();{color}
  final int nodeLoad = node.getXceiverCount();
  if (nodeLoad > maxLoad) {
logNodeIsNotChosen(storage,
"the node is too busy (load:"+nodeLoad+" > "+maxLoad+") ");
stats.incrOverLoaded();
return false;
  }
}

> Decommissioned datanode is counted in service cause datanode allcating failure
> --
>
> Key: HDFS-12820
> URL: https://issues.apache.org/jira/browse/HDFS-12820
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: block placement
>Affects Versions: 2.4.0
>Reporter: Gang Xie
>
> When allocate a datanode when dfsclient write with considering the load, it 
> checks if the datanode is overloaded by calculating the average xceivers of 
> all the in service datanode. But if the datanode is decommissioned and become 
> dead, it's still treated as in service, which make the average load much more 
> than the real one especially when the number of the decommissioned datanode 
> is great. In our cluster, 180 datanode, and 100 of them decommissioned, and 
> the average load is 17. This failed all the datanode allocation. 
> private void subtract(final DatanodeDescriptor node) {
>   capacityUsed -= node.getDfsUsed();
>   blockPoolUsed -= node.getBlockPoolUsed();
>   xceiverCount -= node.getXceiverCount();
> {color:red}  if (!(node.isDecommissionInProgress() || 
> node.isDecommissioned())) {{color}
> nodesInService--;
> nodesInServiceXceiverCount -= node.getXceiverCount();
> capacityTotal -= node.getCapacity();
> capacityRemaining -= node.getRemaining();
>   } else {
> capacityTotal -= node.getDfsUsed();
>   }
>   cacheCapacity -= node.getCacheCapacity();
>   cacheUsed -= node.getCacheUsed();
> }



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Rushabh Shah]

+1 (non-binding).

Built from source and deployed it on pseudo cluster.
Ran sample fs shell commands.
Ran coupe of jobs: sleep and pi.
Tested on java: 1.8.0_131 version.

Thanks Andrew for all your efforts in getting us here. :)



On Tue, Nov 14, 2017 at 3:34 PM, Andrew Wang 
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
>
> Best,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Rakesh Radhakrishnan]

Thanks Andrew for getting this out !

+1 (non-binding)

* Built from source on CentOS 7.3.1611, jdk1.8.0_111
* Deployed non-ha cluster and tested few EC file operations.
* Ran basic shell commands(ls, mkdir, put, get, ec, dfsadmin).
* Ran some sample jobs.
* HDFS Namenode UI looks good.

Thanks,
Rakesh

On Wed, Nov 15, 2017 at 3:04 AM, Andrew Wang 
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
>
> Best,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

I'd definitely extend it for a few more days. I only see 3 binding +1s so far - 
not a great number to brag about on our first major release in years.

Also going to nudge folks into voting.

+Vinod

> On Nov 17, 2017, at 3:26 PM, Andrew Wang  wrote:
> 
> Hi Arpit,
> 
> I agree the timing is not great here, but extending it to meaningfully
> avoid the holidays would mean extending it an extra week (e.g. to the
> 29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
> really like to get the RC out before then.
> 
> In terms of downstream testing, we've done extensive integration testing
> with downstreams via the alphas and betas, and we have continuous
> integration running at Cloudera against branch-3.0. Because of this, I have
> more confidence in our integration for 3.0.0 than most Hadoop releases.
> 
> Is it meaningful to extend to say, the 21st, which provides for a full week
> of voting?
> 
> Best,
> Andrew
> 
> On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
> wrote:
> 
>> Hi Andrew,
>> 
>> Thank you for your hard work in getting us to this step. This is our first
>> major GA release in many years.
>> 
>> I feel a 5-day vote window ending over the weekend before thanksgiving may
>> not provide sufficient time to evaluate this RC especially for downstream
>> components.
>> 
>> Would you please consider extending the voting deadline until a few days
>> after the thanksgiving holiday? It would be a courtesy to our broader
>> community and I see no harm in giving everyone a few days to evaluate it
>> more thoroughly.
>> 
>> On a lighter note, your deadline is also 4 minutes short of the required 5
>> days. :)
>> 
>> Regards,
>> Arpit
>> 
>> 
>> 
>> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>> 
>>Hi folks,
>> 
>>Thanks as always to the many, many contributors who helped with this
>>release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>available here:
>> 
>>http://people.apache.org/~wang/3.0.0-RC0/
>> 
>>This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>> 
>>3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>>additions include the merge of YARN resource types, API-based
>> configuration
>>of the CapacityScheduler, and HDFS router-based federation.
>> 
>>I've done my traditional testing with a pseudo cluster and a Pi job.
>> My +1
>>to start.
>> 
>>Best,
>>Andrew
>> 
>> 
>> 


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Arpit Agarwal]

Thanks for that proposal Andrew, and for not wrapping up the vote yesterday.

> In terms of downstream testing, we've done extensive
> integration testing with downstreams via the alphas
> and betas, and we have continuous integration running
> at Cloudera against branch-3.0.

Could you please share what kind of downstream testing you have performed and 
with which downstream projects?

Regards,
Arpit


On 11/17/17, 3:27 PM, "Andrew Wang"  wrote:

Hi Arpit,

I agree the timing is not great here, but extending it to meaningfully
avoid the holidays would mean extending it an extra week (e.g. to the
29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
really like to get the RC out before then.

In terms of downstream testing, we've done extensive integration testing
with downstreams via the alphas and betas, and we have continuous
integration running at Cloudera against branch-3.0. Because of this, I have
more confidence in our integration for 3.0.0 than most Hadoop releases.

Is it meaningful to extend to say, the 21st, which provides for a full week
of voting?

Best,
Andrew

On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
wrote:

> Hi Andrew,
>
> Thank you for your hard work in getting us to this step. This is our first
> major GA release in many years.
>
> I feel a 5-day vote window ending over the weekend before thanksgiving may
> not provide sufficient time to evaluate this RC especially for downstream
> components.
>
> Would you please consider extending the voting deadline until a few days
> after the thanksgiving holiday? It would be a courtesy to our broader
> community and I see no harm in giving everyone a few days to evaluate it
> more thoroughly.
>
> On a lighter note, your deadline is also 4 minutes short of the required 5
> days. :)
>
> Regards,
> Arpit
>
>
>
> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>
> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based
> configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job.
> My +1
> to start.
>
> Best,
> Andrew
>
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Quick question.

I used to be able (in 2.x line) to create dist tarballs (mvn clean install 
-Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being voted 
on (hadoop-3.0.0-src.tar.gz).

The idea is to install HDFS, YARN, MR separately in separate root-directories 
from the generated individual dist tarballs.

But now I see that HDFS and common dist tarballs are empty
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39 
./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz - 
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40 
./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz

But YARN and MR are fine
-rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41 
./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
-rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41 
./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz

Is it just me? Or is this broken?

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Lei Xu]

+1 binding

Run the following steps:

* Check md5 of sources and package.
* Run a YARN + HDFS pseudo cluster.
* Run terasuite on YARN.
* Run HDFS CLIs (ls , rm , etc)


On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli
 wrote:
> Quick question.
>
> I used to be able (in 2.x line) to create dist tarballs (mvn clean install 
> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being 
> voted on (hadoop-3.0.0-src.tar.gz).
>
> The idea is to install HDFS, YARN, MR separately in separate root-directories 
> from the generated individual dist tarballs.
>
> But now I see that HDFS and common dist tarballs are empty
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39 
> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40 
> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>
> But YARN and MR are fine
> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41 
> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41 
> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>
> Is it just me? Or is this broken?
>
> Thanks
> +Vinod
>
>> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
>>
>> Hi folks,
>>
>> Thanks as always to the many, many contributors who helped with this
>> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>> available here:
>>
>> http://people.apache.org/~wang/3.0.0-RC0/
>>
>> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>
>> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>> additions include the merge of YARN resource types, API-based configuration
>> of the CapacityScheduler, and HDFS router-based federation.
>>
>> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
>> to start.
>>
>> Best,
>> Andrew
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12839) Refactor ratis-server tests to reduce the use DEFAULT_CALLID

[Tsz Wo Nicholas Sze (JIRA)]

Tsz Wo Nicholas Sze created HDFS-12839:
--

 Summary: Refactor ratis-server tests to reduce the use 
DEFAULT_CALLID
 Key: HDFS-12839
 URL: https://issues.apache.org/jira/browse/HDFS-12839
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Priority: Minor


This JIRA is to help reducing the patch size in RATIS-141.

We refactor the tests so that DEFAULT_CALLID is only used in MiniRaftCluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Shane Kumpf]

Thanks, Andrew!

+1 (non-binding)

- Verified checksums and signatures
- Deployed a single node cluster on CentOS 7.4 using the binary and source
release
- Ran hdfs commands
- Ran pi and distributed shell using the default and docker runtimes
- Verified the UIs
- Verified the change log

-Shane


On Tue, Nov 14, 2017 at 2:34 PM, Andrew Wang 
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
>
> Best,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Xiao Chen]

+1 (binding)

Thanks Andrew!


   - Verified md5 and built from source
   - Started a pseudo distributed cluster with KMS,
   - Performed basic hdfs operations plus encryption related operations
   - Verified logs and webui
   - Confidence from CDH testings (will let Andrew answer officially, but
   we have smokes and nightlies for various downstream projects)


-Xiao

On Mon, Nov 20, 2017 at 3:38 PM, Shane Kumpf 
wrote:

> Thanks, Andrew!
>
> +1 (non-binding)
>
> - Verified checksums and signatures
> - Deployed a single node cluster on CentOS 7.4 using the binary and source
> release
> - Ran hdfs commands
> - Ran pi and distributed shell using the default and docker runtimes
> - Verified the UIs
> - Verified the change log
>
> -Shane
>
>
> On Tue, Nov 14, 2017 at 2:34 PM, Andrew Wang 
> wrote:
>
> > Hi folks,
> >
> > Thanks as always to the many, many contributors who helped with this
> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> > available here:
> >
> > http://people.apache.org/~wang/3.0.0-RC0/
> >
> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> >
> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> > additions include the merge of YARN resource types, API-based
> configuration
> > of the CapacityScheduler, and HDFS router-based federation.
> >
> > I've done my traditional testing with a pseudo cluster and a Pi job. My
> +1
> > to start.
> >
> > Best,
> > Andrew
> >
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Wei-Chiu Chuang]

@vinod
I followed your command but I could not reproduce your problem.

[weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-common-project/hadoop-c
ommon/target/hadoop-common-3.0.0.tar.gz
-rw-rw-r-- 1 weichiu weichiu 37052439 Nov 20 21:59
hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz
[weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-hdfs-project/hadoop-hdf
s/target/hadoop-hdfs-3.0.0.tar.gz
-rw-rw-r-- 1 weichiu weichiu 29044067 Nov 20 22:00
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz

During compilation I found the following error with a Java 1.8.0_5 JDK:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven
-compiler-plugin:3.1:testCompile (default-testCompile) on project
hadoop-aws: Compilation failure: Compilation failure:
[ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[45,5]
reference to intercept is ambiguous
[ERROR]   both method intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
org.apache.hadoop.test.LambdaTestUtils and method
intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
in org.apache.hadoop.test.LambdaTestUtils match
[ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[69,5]
reference to intercept is ambiguous
[ERROR]   both method intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
org.apache.hadoop.test.LambdaTestUtils and method
intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
in org.apache.hadoop.test.LambdaTestUtils match
[ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[94,5]
reference to intercept is ambiguous
[ERROR]   both method intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
org.apache.hadoop.test.LambdaTestUtils and method
intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
in org.apache.hadoop.test.LambdaTestUtils match
[ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[120,5]
reference to intercept is ambiguous
[ERROR]   both method intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
org.apache.hadoop.test.LambdaTestUtils and method
intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
in org.apache.hadoop.test.LambdaTestUtils match

And then I realized Ray filed HADOOP-14900
 for the same
issue. This problem is not reproducible with a more recent JDK 8, such as
1.8.0_151
Maybe it would be a good idea to name a list of JDKs that are known to be
buggy. Can we get this documented somewhere? I don't consider it a blocker
so a release note in a later release or a wiki entry should be good enough.

On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli <
vino...@apache.org> wrote:

> Quick question.
>
> I used to be able (in 2.x line) to create dist tarballs (mvn clean install
> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being
> voted on (hadoop-3.0.0-src.tar.gz).
>
> The idea is to install HDFS, YARN, MR separately in separate
> root-directories from the generated individual dist tarballs.
>
> But now I see that HDFS and common dist tarballs are empty
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39
> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40
> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>
> But YARN and MR are fine
> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41
> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41
> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>
> Is it just me? Or is this broken?
>
> Thanks
> +Vinod
>
> > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
> wrote:
> >
> > Hi folks,
> >
> > Thanks as always to the many, many contributors who helped with this
> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> > available here:
> >
> > http://people.apache.org/~wang/3.0.0-RC0/
> >
> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> >
> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> > additions include the merge of YARN resource types, API-based
> configuration
> > of the CapacityScheduler, and HDFS router-based federation.
> >
> > I've done my traditional testing with a pseudo cluster and a Pi job. My
> +1
> > to start.
> >
> > Best,
> > Andrew
>
>
> -

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Compilation passed for me. Using jdk1.8.0_40.jdk.

+Vinod

> On Nov 20, 2017, at 4:16 PM, Wei-Chiu Chuang  wrote:
> 
> @vinod
> I followed your command but I could not reproduce your problem.
> 
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-common-project/hadoop-c
> ommon/target/hadoop-common-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 37052439 Nov 20 21:59
> hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-hdfs-project/hadoop-hdf
> s/target/hadoop-hdfs-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 29044067 Nov 20 22:00
> hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
> 
> During compilation I found the following error with a Java 1.8.0_5 JDK:
> 
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven
> -compiler-plugin:3.1:testCompile (default-testCompile) on project
> hadoop-aws: Compilation failure: Compilation failure:
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[45,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[69,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[94,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[120,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> 
> And then I realized Ray filed HADOOP-14900
>  for the same
> issue. This problem is not reproducible with a more recent JDK 8, such as
> 1.8.0_151
> Maybe it would be a good idea to name a list of JDKs that are known to be
> buggy. Can we get this documented somewhere? I don't consider it a blocker
> so a release note in a later release or a wiki entry should be good enough.
> 
> On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
> 
>> Quick question.
>> 
>> I used to be able (in 2.x line) to create dist tarballs (mvn clean install
>> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being
>> voted on (hadoop-3.0.0-src.tar.gz).
>> 
>> The idea is to install HDFS, YARN, MR separately in separate
>> root-directories from the generated individual dist tarballs.
>> 
>> But now I see that HDFS and common dist tarballs are empty
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39
>> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40
>> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>> 
>> But YARN and MR are fine
>> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41
>> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
>> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41
>> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>> 
>> Is it just me? Or is this broken?
>> 
>> Thanks
>> +Vinod
>> 
>>> On Nov 14, 2017, at 1:34 PM, Andrew Wang 
>> wrote:
>>> 
>>> Hi folks,
>>> 
>>> Thanks as always to the many, many contributors who helped with this
>>> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>> available here:
>>> 
>>> http://people.apache.org/~wang/3.0.0-RC0/
>>> 
>>> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>> 
>>> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>>> additions include the merge of YARN resource types, API-based
>

[jira] [Created] (HDFS-12840) Creating a replicated file in a EC zone does not correctly serialized in EditLogs

[Lei (Eddy) Xu (JIRA)]

Lei (Eddy) Xu created HDFS-12840:


 Summary: Creating a replicated file in a EC zone does not 
correctly serialized in EditLogs
 Key: HDFS-12840
 URL: https://issues.apache.org/jira/browse/HDFS-12840
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: erasure-coding
Affects Versions: 3.0.0-beta1
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Blocker


When create a replicated file in an existing EC zone, the edit logs does not 
differentiate it from an EC file. When {{FSEditLogLoader}} to replay edits, 
this file is treated as EC file, as a results, it crashes the NN because the 
blocks of this file are replicated, which does not match with {{INode}}.

{noformat}
ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered 
exception on operation AddBlockOp [path=/system/balancer.id, 
penultimateBlock=NULL, lastBlock=blk_1073743259_2455, RpcClientId=, 
RpcCallId=-2]
java.lang.IllegalArgumentException: reportedBlock is not striped
at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStriped.addStorage(BlockInfoStriped.java:118)
at 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.addBlock(DatanodeStorageInfo.java:256)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlock(BlockManager.java:3141)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlockUnderConstruction(BlockManager.java:3068)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processAndHandleReportedBlock(BlockManager.java:3864)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processQueuedMessages(BlockManager.java:2916)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processQueuedMessagesForBlock(BlockManager.java:2903)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.addNewBlock(FSEditLogLoader.java:1069)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:532)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:293)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:427)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:380)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:397)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

RE: Apache Hadoop 2.8.3 Release Plan

[Zheng, Kai]

Hi Junping,

Thank you for making 2.8.2 happen and now planning the 2.8.3 release. 

I have an ask, is it convenient to include the back port work for OSS connector 
module? We have some Hadoop users that wish to have it by default for 
convenience, though in the past they used it by back porting themselves. I have 
raised this and got thoughts from Chris and Steve. Looks like this is more 
wanted for 2.9 but I wanted to ask again here for broad feedback and thoughts 
by this chance. The back port patch is available for 2.8 and the one for 
branch-2 was already in. IMO, 2.8.x is promising as we can see some shift from 
2.7.x, hence it's worth more important features and efforts. How would you 
think? Thanks!

https://issues.apache.org/jira/browse/HADOOP-14964

Regards,
Kai

-Original Message-
From: Junping Du [mailto:j...@hortonworks.com] 
Sent: Tuesday, November 14, 2017 9:02 AM
To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Apache Hadoop 2.8.3 Release Plan

Hi,
We have several important fixes get landed on branch-2.8 and I would like 
to cut off branch-2.8.3 now to start 2.8.3 release work. 
So far, I don't see any pending blockers on 2.8.3, so my current plan is to 
cut off 1st RC of 2.8.3 in next several days: 
 -  For all coming commits to land on branch-2.8, please mark the fix 
version as 2.8.4.
 -  If there is a really important fix for 2.8.3 and getting closed, 
please notify me ahead before landing it on branch-2.8.3.
Please let me know if you have any thoughts or comments on the plan.

Thanks,

Junping

From: dujunp...@gmail.com  on behalf of 俊平堵 

Sent: Friday, October 27, 2017 3:33 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.

Hi all,

It gives me great pleasure to announce that the Apache Hadoop community has 
voted to release Apache Hadoop 2.8.2, which is now available for download from 
Apache mirrors[1]. For download instructions please refer to the Apache Hadoop 
Release page [2].

Apache Hadoop 2.8.2 is the first GA release of Apache Hadoop 2.8 line and our 
newest stable release for entire Apache Hadoop project. For major changes 
incuded in Hadoop 2.8 line, please refer Hadoop 2.8.2 main page[3].

This release has 315 resolved issues since previous 2.8.1 release with following
breakdown:
   - 91 in Hadoop Common
   - 99 in HDFS
   - 105 in YARN
   - 20 in MapReduce
Please read the log of CHANGES[4] and RELEASENOTES[5] for more details.

The release news is posted on the Hadoop website too, you can go to the 
downloads section directly [6].

Thank you all for contributing to the Apache Hadoop release!


Cheers,

Junping


[1] http://www.apache.org/dyn/closer.cgi/hadoop/common

[2] http://hadoop.apache.org/releases.html

[3] http://hadoop.apache.org/docs/r2.8.2/index.html

[4]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/CHANGES.2.8.2.html

[5]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/RELEASENOTES.2.8.2.html

[6] http://hadoop.apache.org/releases.html#Download


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Thanks for all the push, Andrew!

Looking at the RC. Went through my usual check-list. Here's my summary. Will 
cast my final vote after comparing and validating my findings with others.

Verification

 - [Check] Successful recompilation from source tar-ball
 - [Check] Signature verification
 - [Check] Generating dist tarballs from source tar-ball
 - [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs

Issues found during testing

Major
 - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty.
 - Cannot enable new UI in YARN because it is under a non-default compilation 
flag. It should be on by default.
 - One decommissioned node in YARN ResourceManager UI always appears to start 
with, even when there are no NodeManagers that are started yet:  Info :-1, 
DECOMMISSIONED, null rack. It shows up only in the UI though, not in the CLI 
node -list

Minor
 - resourcemanager-metrics.out is going into current directory instead of log 
directory
 - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even work. 
Not just deprecated in favor of timelineserver as was advertised.
 - Spurious warnings on CLI
17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml not 
found
17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find 
'resource-types.xml'.

Side notes

 - When did we stop putting CHANGES files into the source artifacts?
 - Even after "mvn install"ing once, shading is repeated again and again for 
every new 'mvn install' even though there are no source changes - we should see 
how this can be avoided.
 - Compatibility notes
-- NM's env list is curtailed unlike in 2.x (For e.g, HADOOP_MAPRED_HOME is 
not automatically inherited. Correct behavior)
-- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar into 
hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Sangjin Lee]

On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli  wrote:

> Thanks for all the push, Andrew!
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
> Will cast my final vote after comparing and validating my findings with
> others.
>
> Verification
>
>  - [Check] Successful recompilation from source tar-ball
>  - [Check] Signature verification
>  - [Check] Generating dist tarballs from source tar-ball
>  - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> Issues found during testing
>
> Major
>  - The previously supported way of being able to use different tar-balls
> for different sub-modules is completely broken - common and HDFS tar.gz are
> completely empty.
>  - Cannot enable new UI in YARN because it is under a non-default
> compilation flag. It should be on by default.
>  - One decommissioned node in YARN ResourceManager UI always appears to
> start with, even when there are no NodeManagers that are started yet:  Info
> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
> in the CLI node -list
>
> Minor
>  - resourcemanager-metrics.out is going into current directory instead of
> log directory
>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even
> work. Not just deprecated in favor of timelineserver as was advertised.
>  - Spurious warnings on CLI
> 17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml
> not found
> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
>
> Side notes
>
>  - When did we stop putting CHANGES files into the source artifacts?
>  - Even after "mvn install"ing once, shading is repeated again and again
> for every new 'mvn install' even though there are no source changes - we
> should see how this can be avoided.
>  - Compatibility notes
> -- NM's env list is curtailed unlike in 2.x (For e.g,
> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
> -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>

Sleep has always been in the jobclient test jar as long as I can remember,
so it's not new for 3.0.


>
> Thanks
> +Vinod
>
> > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
> wrote:
> >
> > Hi folks,
> >
> > Thanks as always to the many, many contributors who helped with this
> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> > available here:
> >
> > http://people.apache.org/~wang/3.0.0-RC0/
> >
> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> >
> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> > additions include the merge of YARN resource types, API-based
> configuration
> > of the CapacityScheduler, and HDFS router-based federation.
> >
> > I've done my traditional testing with a pseudo cluster and a Pi job. My
> +1
> > to start.
> >
> > Best,
> > Andrew
>
>

RE: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Chen, Sammi]

+1 (binding)

1) Compile with native from source code
2) Set up a one-node cluster, enabling short-circuit read
3) Tried some basic HDFS commands, ls, put etc. 
4) MapReduce workloads: TestDFSIO, TeraGen and TeraSort
5) Test storage policy and mover successfully

Sorry for the late response. 

-Original Message-
From: John Zhuge [mailto:john.zh...@gmail.com]
Sent: Friday, November 17, 2017 6:56 AM
To: Andrew Wang 
Cc: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

+1 (binding)

   - Verified checksums of all tarballs
   - Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
   - Passed all S3A and ADL integration tests
   - Deployed both binary and built source to a pseudo cluster, passed the
   following sanity tests in insecure, SSL, and SSL+Kerberos mode:
  - HDFS basic and ACL
  - DistCp basic
  - MapReduce wordcount (skipped in SSL+Kerberos mode)
  - KMS and HttpFS basic
  - Balancer start/stop


On Tue, Nov 14, 2017 at 1:34 PM, Andrew Wang 
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this 
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are 
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable 
> additions include the merge of YARN resource types, API-based 
> configuration of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. 
> My +1 to start.
>
> Best,
> Andrew
>



--
John

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Sangjin Lee]

I checked the client jars that are supposed to contain shaded dependencies,
and they don't look quite right:

$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
-rw-r--r--  0 andrew andrew44531 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
-rw-r--r--  0 andrew andrew45533 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.0.jar
-rw-r--r--  0 andrew andrew47015 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar

When I look at what's inside those jar, they only seem to include
pom-related files with no class files. Am I missing something?

When I build from the source with -Pdist, I do get much bigger jars:
total 113760
-rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
hadoop-client-api-3.0.0.jar
-rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
hadoop-client-minicluster-3.0.0.jar
-rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
hadoop-client-runtime-3.0.0.jar

Sangjin

On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee  wrote:

>
>
> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> Thanks for all the push, Andrew!
>>
>> Looking at the RC. Went through my usual check-list. Here's my summary.
>> Will cast my final vote after comparing and validating my findings with
>> others.
>>
>> Verification
>>
>>  - [Check] Successful recompilation from source tar-ball
>>  - [Check] Signature verification
>>  - [Check] Generating dist tarballs from source tar-ball
>>  - [Check] Testing
>> -- Start NN, DN, RM, NM, JHS, Timeline Service
>> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
>> grep, pi
>> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>>
>> Issues found during testing
>>
>> Major
>>  - The previously supported way of being able to use different tar-balls
>> for different sub-modules is completely broken - common and HDFS tar.gz are
>> completely empty.
>>  - Cannot enable new UI in YARN because it is under a non-default
>> compilation flag. It should be on by default.
>>  - One decommissioned node in YARN ResourceManager UI always appears to
>> start with, even when there are no NodeManagers that are started yet:  Info
>> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
>> in the CLI node -list
>>
>> Minor
>>  - resourcemanager-metrics.out is going into current directory instead of
>> log directory
>>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
>> even work. Not just deprecated in favor of timelineserver as was advertised.
>>  - Spurious warnings on CLI
>> 17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml
>> not found
>> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find
>> 'resource-types.xml'.
>>
>> Side notes
>>
>>  - When did we stop putting CHANGES files into the source artifacts?
>>  - Even after "mvn install"ing once, shading is repeated again and again
>> for every new 'mvn install' even though there are no source changes - we
>> should see how this can be avoided.
>>  - Compatibility notes
>> -- NM's env list is curtailed unlike in 2.x (For e.g,
>> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
>> -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
>> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>>
>
> Sleep has always been in the jobclient test jar as long as I can remember,
> so it's not new for 3.0.
>
>
>>
>> Thanks
>> +Vinod
>>
>> > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
>> wrote:
>> >
>> > Hi folks,
>> >
>> > Thanks as always to the many, many contributors who helped with this
>> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>> > available here:
>> >
>> > http://people.apache.org/~wang/3.0.0-RC0/
>> >
>> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>> >
>> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>> > additions include the merge of YARN resource types, API-based
>> configuration
>> > of the CapacityScheduler, and HDFS router-based federation.
>> >
>> > I've done my traditional testing with a pseudo cluster and a Pi job. My
>> +1
>> > to start.
>> >
>> > Best,
>> > Andrew
>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Akira Ajisaka]

>   - When did we stop putting CHANGES files into the source artifacts?

CHANGES files were removed by https://issues.apache.org/jira/browse/HADOOP-11792

>   - Even after "mvn install"ing once, shading is repeated again and again for 
every new 'mvn install' even though there are no source changes - we should see how this can 
be avoided.

Shading can be avoided by -DskipShade option and the option is documented in 
BUILDING.txt.

Regards,
Akira

On 2017/11/21 10:26, Vinod Kumar Vavilapalli wrote:

Thanks for all the push, Andrew!

Looking at the RC. Went through my usual check-list. Here's my summary. Will 
cast my final vote after comparing and validating my findings with others.

Verification

  - [Check] Successful recompilation from source tar-ball
  - [Check] Signature verification
  - [Check] Generating dist tarballs from source tar-ball
  - [Check] Testing
 -- Start NN, DN, RM, NM, JHS, Timeline Service
 -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, 
pi
 -- Tested CLIs to print nodes, apps etc and also navigated UIs

Issues found during testing

Major
  - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty.
  - Cannot enable new UI in YARN because it is under a non-default compilation 
flag. It should be on by default.
  - One decommissioned node in YARN ResourceManager UI always appears to start with, 
even when there are no NodeManagers that are started yet:  Info :-1, 
DECOMMISSIONED, null rack. It shows up only in the UI though, not in the CLI node 
-list

Minor
  - resourcemanager-metrics.out is going into current directory instead of log 
directory
  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
work. Not just deprecated in favor of timelineserver as was advertised.
  - Spurious warnings on CLI
 17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml not 
found
 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find 
'resource-types.xml'.

Side notes

  - When did we stop putting CHANGES files into the source artifacts?
  - Even after "mvn install"ing once, shading is repeated again and again for 
every new 'mvn install' even though there are no source changes - we should see how this 
can be avoided.
  - Compatibility notes
 -- NM's env list is curtailed unlike in 2.x (For e.g, HADOOP_MAPRED_HOME 
is not automatically inherited. Correct behavior)
 -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar into 
hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

Thanks
+Vinod


On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:

Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew





-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Thanks for the spot Sangjin. I think this bug introduced in create-release
by HADOOP-14835. The multi-pass maven build generates these dummy client
jars during the site build since skipShade is specified.

This might be enough to cancel the RC. Thoughts?

Best,
Andrew

On Mon, Nov 20, 2017 at 7:51 PM, Sangjin Lee  wrote:

> I checked the client jars that are supposed to contain shaded
> dependencies, and they don't look quite right:
>
> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
> -rw-r--r--  0 andrew andrew44531 Nov 14 11:53
> hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
> -rw-r--r--  0 andrew andrew45533 Nov 14 11:53
> hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.0.jar
> -rw-r--r--  0 andrew andrew47015 Nov 14 11:53
> hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar
>
> When I look at what's inside those jar, they only seem to include
> pom-related files with no class files. Am I missing something?
>
> When I build from the source with -Pdist, I do get much bigger jars:
> total 113760
> -rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
> hadoop-client-api-3.0.0.jar
> -rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
> hadoop-client-minicluster-3.0.0.jar
> -rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
> hadoop-client-runtime-3.0.0.jar
>
> Sangjin
>
> On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee  wrote:
>
>>
>>
>> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
>> vino...@apache.org> wrote:
>>
>>> Thanks for all the push, Andrew!
>>>
>>> Looking at the RC. Went through my usual check-list. Here's my summary.
>>> Will cast my final vote after comparing and validating my findings with
>>> others.
>>>
>>> Verification
>>>
>>>  - [Check] Successful recompilation from source tar-ball
>>>  - [Check] Signature verification
>>>  - [Check] Generating dist tarballs from source tar-ball
>>>  - [Check] Testing
>>> -- Start NN, DN, RM, NM, JHS, Timeline Service
>>> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
>>> grep, pi
>>> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>>>
>>> Issues found during testing
>>>
>>> Major
>>>  - The previously supported way of being able to use different tar-balls
>>> for different sub-modules is completely broken - common and HDFS tar.gz are
>>> completely empty.
>>>  - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>>  - One decommissioned node in YARN ResourceManager UI always appears to
>>> start with, even when there are no NodeManagers that are started yet:  Info
>>> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
>>> in the CLI node -list
>>>
>>> Minor
>>>  - resourcemanager-metrics.out is going into current directory instead
>>> of log directory
>>>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
>>> even work. Not just deprecated in favor of timelineserver as was advertised.
>>>  - Spurious warnings on CLI
>>> 17/11/20 17:07:34 INFO conf.Configuration:
>>> resource-types.xml not found
>>> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to
>>> find 'resource-types.xml'.
>>>
>>> Side notes
>>>
>>>  - When did we stop putting CHANGES files into the source artifacts?
>>>  - Even after "mvn install"ing once, shading is repeated again and again
>>> for every new 'mvn install' even though there are no source changes - we
>>> should see how this can be avoided.
>>>  - Compatibility notes
>>> -- NM's env list is curtailed unlike in 2.x (For e.g,
>>> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
>>> -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
>>> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>>>
>>
>> Sleep has always been in the jobclient test jar as long as I can
>> remember, so it's not new for 3.0.
>>
>>
>>>
>>> Thanks
>>> +Vinod
>>>
>>> > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
>>> wrote:
>>> >
>>> > Hi folks,
>>> >
>>> > Thanks as always to the many, many contributors who helped with this
>>> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>> > available here:
>>> >
>>> > http://people.apache.org/~wang/3.0.0-RC0/
>>> >
>>> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>> >
>>> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>>> > additions include the merge of YARN resource types, API-based
>>> configuration
>>> > of the CapacityScheduler, and HDFS router-based federation.
>>> >
>>> > I've done my traditional testing with a pseudo cluster and a Pi job.
>>> My +1
>>> > to start.
>>> >
>>> > Best,
>>> > Andrew
>>>
>>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Sangjin Lee]

On Mon, Nov 20, 2017 at 9:46 PM, Andrew Wang 
wrote:

> Thanks for the spot Sangjin. I think this bug introduced in create-release
> by HADOOP-14835. The multi-pass maven build generates these dummy client
> jars during the site build since skipShade is specified.
>
> This might be enough to cancel the RC. Thoughts?
>

IMO yes. This was one of the key features mentioned in the 3.0 release
notes. I appreciate your effort for the release Andrew!


>
> Best,
> Andrew
>
> On Mon, Nov 20, 2017 at 7:51 PM, Sangjin Lee  wrote:
>
>> I checked the client jars that are supposed to contain shaded
>> dependencies, and they don't look quite right:
>>
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
>> -rw-r--r--  0 andrew andrew44531 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
>> -rw-r--r--  0 andrew andrew45533 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.
>> 0.jar
>> -rw-r--r--  0 andrew andrew47015 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar
>>
>> When I look at what's inside those jar, they only seem to include
>> pom-related files with no class files. Am I missing something?
>>
>> When I build from the source with -Pdist, I do get much bigger jars:
>> total 113760
>> -rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
>> hadoop-client-api-3.0.0.jar
>> -rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
>> hadoop-client-minicluster-3.0.0.jar
>> -rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
>> hadoop-client-runtime-3.0.0.jar
>>
>> Sangjin
>>
>> On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee  wrote:
>>
>>>
>>>
>>> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
>>> vino...@apache.org> wrote:
>>>
 Thanks for all the push, Andrew!

 Looking at the RC. Went through my usual check-list. Here's my summary.
 Will cast my final vote after comparing and validating my findings with
 others.

 Verification

  - [Check] Successful recompilation from source tar-ball
  - [Check] Signature verification
  - [Check] Generating dist tarballs from source tar-ball
  - [Check] Testing
 -- Start NN, DN, RM, NM, JHS, Timeline Service
 -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
 grep, pi
 -- Tested CLIs to print nodes, apps etc and also navigated UIs

 Issues found during testing

 Major
  - The previously supported way of being able to use different
 tar-balls for different sub-modules is completely broken - common and HDFS
 tar.gz are completely empty.
  - Cannot enable new UI in YARN because it is under a non-default
 compilation flag. It should be on by default.
  - One decommissioned node in YARN ResourceManager UI always appears to
 start with, even when there are no NodeManagers that are started yet:  Info
 :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
 in the CLI node -list

 Minor
  - resourcemanager-metrics.out is going into current directory instead
 of log directory
  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
 even work. Not just deprecated in favor of timelineserver as was 
 advertised.
  - Spurious warnings on CLI
 17/11/20 17:07:34 INFO conf.Configuration:
 resource-types.xml not found
 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to
 find 'resource-types.xml'.

 Side notes

  - When did we stop putting CHANGES files into the source artifacts?
  - Even after "mvn install"ing once, shading is repeated again and
 again for every new 'mvn install' even though there are no source changes -
 we should see how this can be avoided.
  - Compatibility notes
 -- NM's env list is curtailed unlike in 2.x (For e.g,
 HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
 -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
 into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

>>>
>>> Sleep has always been in the jobclient test jar as long as I can
>>> remember, so it's not new for 3.0.
>>>
>>>

 Thanks
 +Vinod

 > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
 wrote:
 >
 > Hi folks,
 >
 > Thanks as always to the many, many contributors who helped with this
 > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
 > available here:
 >
 > http://people.apache.org/~wang/3.0.0-RC0/
 >
 > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
 >
 > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
 > additions include the merge of YARN resourc

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Thanks for the thorough review Vinod, some inline responses:

*Issues found during testing*
>
> Major
>  - The previously supported way of being able to use different tar-balls
> for different sub-modules is completely broken - common and HDFS tar.gz are
> completely empty.
>

Is this something people use? I figured that the sub-tarballs were a relic
from the project split, and nowadays Hadoop is one project with one release
tarball. I actually thought about getting rid of these extra tarballs since
they add extra overhead to a full build.


>  - Cannot enable new UI in YARN because it is under a non-default
> compilation flag. It should be on by default.
>

The yarn-ui profile has always been off by default, AFAIK. It's documented
to turn it on in BUILDING.txt for release builds, and we do it in
create-release.

IMO not a blocker. I think it's also more of a dev question (do we want to
do this on every YARN build?) than a release one.


>  - One decommissioned node in YARN ResourceManager UI always appears to
> start with, even when there are no NodeManagers that are started yet:
> Info :-1, DECOMMISSIONED, null rack. It shows up only in the UI though,
> not in the CLI node -list
>

Is this a blocker? Could we get a JIRA?

Thanks,
Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

On Mon, Nov 20, 2017 at 9:59 PM, Sangjin Lee  wrote:

>
> On Mon, Nov 20, 2017 at 9:46 PM, Andrew Wang 
> wrote:
>
>> Thanks for the spot Sangjin. I think this bug introduced in
>> create-release by HADOOP-14835. The multi-pass maven build generates these
>> dummy client jars during the site build since skipShade is specified.
>>
>> This might be enough to cancel the RC. Thoughts?
>>
>
> IMO yes. This was one of the key features mentioned in the 3.0 release
> notes. I appreciate your effort for the release Andrew!
>
>
Yea, I was leaning that way too. Let's cancel this RC. I hope to have a new
RC up tomorrow. With the upcoming holidays, we'll probably have to extend
the vote until mid-next week.

I'm also worried about the "mvn deploy" step since I thought it was safe to
specify skipShade there too. I'll check that as well.

Best,
Andrew

Re: Apache Hadoop 2.8.3 Release Plan

[Andrew Wang]

I'm against including new features in maintenance releases, since they're
meant to be bug-fix only.

If we're struggling with being able to deliver new features in a safe and
timely fashion, let's try to address that, not overload the meaning of
"maintenance release".

Best,
Andrew

On Mon, Nov 20, 2017 at 5:20 PM, Zheng, Kai  wrote:

> Hi Junping,
>
> Thank you for making 2.8.2 happen and now planning the 2.8.3 release.
>
> I have an ask, is it convenient to include the back port work for OSS
> connector module? We have some Hadoop users that wish to have it by default
> for convenience, though in the past they used it by back porting
> themselves. I have raised this and got thoughts from Chris and Steve. Looks
> like this is more wanted for 2.9 but I wanted to ask again here for broad
> feedback and thoughts by this chance. The back port patch is available for
> 2.8 and the one for branch-2 was already in. IMO, 2.8.x is promising as we
> can see some shift from 2.7.x, hence it's worth more important features and
> efforts. How would you think? Thanks!
>
> https://issues.apache.org/jira/browse/HADOOP-14964
>
> Regards,
> Kai
>
> -Original Message-
> From: Junping Du [mailto:j...@hortonworks.com]
> Sent: Tuesday, November 14, 2017 9:02 AM
> To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Apache Hadoop 2.8.3 Release Plan
>
> Hi,
> We have several important fixes get landed on branch-2.8 and I would
> like to cut off branch-2.8.3 now to start 2.8.3 release work.
> So far, I don't see any pending blockers on 2.8.3, so my current plan
> is to cut off 1st RC of 2.8.3 in next several days:
>  -  For all coming commits to land on branch-2.8, please mark the
> fix version as 2.8.4.
>  -  If there is a really important fix for 2.8.3 and getting
> closed, please notify me ahead before landing it on branch-2.8.3.
> Please let me know if you have any thoughts or comments on the plan.
>
> Thanks,
>
> Junping
> 
> From: dujunp...@gmail.com  on behalf of 俊平堵 <
> junping...@apache.org>
> Sent: Friday, October 27, 2017 3:33 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.
>
> Hi all,
>
> It gives me great pleasure to announce that the Apache Hadoop
> community has voted to release Apache Hadoop 2.8.2, which is now available
> for download from Apache mirrors[1]. For download instructions please refer
> to the Apache Hadoop Release page [2].
>
> Apache Hadoop 2.8.2 is the first GA release of Apache Hadoop 2.8 line and
> our newest stable release for entire Apache Hadoop project. For major
> changes incuded in Hadoop 2.8 line, please refer Hadoop 2.8.2 main page[3].
>
> This release has 315 resolved issues since previous 2.8.1 release with
> following
> breakdown:
>- 91 in Hadoop Common
>- 99 in HDFS
>- 105 in YARN
>- 20 in MapReduce
> Please read the log of CHANGES[4] and RELEASENOTES[5] for more details.
>
> The release news is posted on the Hadoop website too, you can go to the
> downloads section directly [6].
>
> Thank you all for contributing to the Apache Hadoop release!
>
>
> Cheers,
>
> Junping
>
>
> [1] http://www.apache.org/dyn/closer.cgi/hadoop/common
>
> [2] http://hadoop.apache.org/releases.html
>
> [3] http://hadoop.apache.org/docs/r2.8.2/index.html
>
> [4]
> http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/
> hadoop-common/release/2.8.2/CHANGES.2.8.2.html
>
> [5]
> http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/
> hadoop-common/release/2.8.2/RELEASENOTES.2.8.2.html
>
> [6] http://hadoop.apache.org/releases.html#Download
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

RE: Apache Hadoop 2.8.3 Release Plan

[Zheng, Kai]

Thanks Andrew for the comments.

Yes, if we're "strictly" following the "maintenance release" practice, that'd 
be great and it's never my intent to overload it and cause mess.

>> If we're struggling with being able to deliver new features in a safe and 
>> timely fashion, let's try to address that...

This is interesting. Do you aware any means to do that? Thanks!

Regards,
Kai

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: Tuesday, November 21, 2017 2:22 PM
To: Zheng, Kai 
Cc: Junping Du ; common-...@hadoop.apache.org; 
hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.3 Release Plan

I'm against including new features in maintenance releases, since they're meant 
to be bug-fix only.

If we're struggling with being able to deliver new features in a safe and 
timely fashion, let's try to address that, not overload the meaning of 
"maintenance release".

Best,
Andrew

On Mon, Nov 20, 2017 at 5:20 PM, Zheng, Kai  wrote:

> Hi Junping,
>
> Thank you for making 2.8.2 happen and now planning the 2.8.3 release.
>
> I have an ask, is it convenient to include the back port work for OSS 
> connector module? We have some Hadoop users that wish to have it by 
> default for convenience, though in the past they used it by back 
> porting themselves. I have raised this and got thoughts from Chris and 
> Steve. Looks like this is more wanted for 2.9 but I wanted to ask 
> again here for broad feedback and thoughts by this chance. The back 
> port patch is available for
> 2.8 and the one for branch-2 was already in. IMO, 2.8.x is promising 
> as we can see some shift from 2.7.x, hence it's worth more important 
> features and efforts. How would you think? Thanks!
>
> https://issues.apache.org/jira/browse/HADOOP-14964
>
> Regards,
> Kai
>
> -Original Message-
> From: Junping Du [mailto:j...@hortonworks.com]
> Sent: Tuesday, November 14, 2017 9:02 AM
> To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Apache Hadoop 2.8.3 Release Plan
>
> Hi,
> We have several important fixes get landed on branch-2.8 and I 
> would like to cut off branch-2.8.3 now to start 2.8.3 release work.
> So far, I don't see any pending blockers on 2.8.3, so my current 
> plan is to cut off 1st RC of 2.8.3 in next several days:
>  -  For all coming commits to land on branch-2.8, please mark 
> the fix version as 2.8.4.
>  -  If there is a really important fix for 2.8.3 and getting 
> closed, please notify me ahead before landing it on branch-2.8.3.
> Please let me know if you have any thoughts or comments on the plan.
>
> Thanks,
>
> Junping
> 
> From: dujunp...@gmail.com  on behalf of 俊平堵 < 
> junping...@apache.org>
> Sent: Friday, October 27, 2017 3:33 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.
>
> Hi all,
>
> It gives me great pleasure to announce that the Apache Hadoop 
> community has voted to release Apache Hadoop 2.8.2, which is now 
> available for download from Apache mirrors[1]. For download 
> instructions please refer to the Apache Hadoop Release page [2].
>
> Apache Hadoop 2.8.2 is the first GA release of Apache Hadoop 2.8 line 
> and our newest stable release for entire Apache Hadoop project. For 
> major changes incuded in Hadoop 2.8 line, please refer Hadoop 2.8.2 main 
> page[3].
>
> This release has 315 resolved issues since previous 2.8.1 release with 
> following
> breakdown:
>- 91 in Hadoop Common
>- 99 in HDFS
>- 105 in YARN
>- 20 in MapReduce
> Please read the log of CHANGES[4] and RELEASENOTES[5] for more details.
>
> The release news is posted on the Hadoop website too, you can go to 
> the downloads section directly [6].
>
> Thank you all for contributing to the Apache Hadoop release!
>
>
> Cheers,
>
> Junping
>
>
> [1] http://www.apache.org/dyn/closer.cgi/hadoop/common
>
> [2] http://hadoop.apache.org/releases.html
>
> [3] http://hadoop.apache.org/docs/r2.8.2/index.html
>
> [4]
> http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/
> hadoop-common/release/2.8.2/CHANGES.2.8.2.html
>
> [5]
> http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/
> hadoop-common/release/2.8.2/RELEASENOTES.2.8.2.html
>
> [6] http://hadoop.apache.org/releases.html#Download
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: Apache Hadoop 2.8.3 Release Plan

[Andrew Wang]

>
>
> >> If we're struggling with being able to deliver new features in a safe
> and timely fashion, let's try to address that...
>
> This is interesting. Do you aware any means to do that? Thanks!
>
> I've mentioned this a few times on the lists before, but our biggest gap
in keeping branches releasable is automated integration testing.

I think we try to put too much into our minor releases, and features arrive
before they're baked. Having automated integration testing helps with this.
When we were finally able to turn on CI for the 3.0.0 release branch, we
started finding bugs much sooner after they were introduced, which made it
easier to revert before too much other code was built on top. The early
alphas felt Sisyphean at times, with bugs being introduced faster than we
could uncover and fix them.

A smaller example would be release validation. I've long wanted a nightly
Jenkins job that makes an RC and runs some basic checks on it. We end up
rolling extra RCs for small stuff that could have been caught earlier.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Allen Wittenauer]

The original release script and instructions broke the build up into 
three or so steps. When I rewrote it, I kept that same model. It’s probably 
time to re-think that.  In particular, it should probably be one big step that 
even does the maven deploy.  There’s really no harm in doing that given that 
there is still a manual step to release the deployed jars into the production 
area.

We just need need to:

a) add an option to do deploy instead of just install.  if c-r is in asf mode, 
always activate deploy
b) pull the maven settings.xml file (and only the maven settings file… we don’t 
want the repo!) into the docker build environment
c) consolidate the mvn steps

This has the added benefit of greatly speeding up the build by removing 
several passes.

Probably not a small change, but I’d have to look at the code.  I’m on 
a plane tomorrow morning though.

Also:

>> 
>> Major
>> - The previously supported way of being able to use different tar-balls
>> for different sub-modules is completely broken - common and HDFS tar.gz are
>> completely empty.
>> 
> 
> Is this something people use? I figured that the sub-tarballs were a relic
> from the project split, and nowadays Hadoop is one project with one release
> tarball. I actually thought about getting rid of these extra tarballs since
> they add extra overhead to a full build.

I’m guessing no one noticed the tar errors when running mvn -Pdist.  
Not sure when they started happening.

> >   - When did we stop putting CHANGES files into the source artifacts?
> 
> CHANGES files were removed by 
> https://issues.apache.org/jira/browse/HADOOP-11792

To be a bit more specific about it, the maven assembly for source only 
includes things (more or less) that are part of the git repo.  When CHANGES.txt 
was removed from the source tree, it also went away from the tar ball.  This 
isn’t too much of an issue in practice though given the notes are put up on the 
web, part of the binary tar ball, and can be generated by following the 
directions in BUILDING.txt.  I don’t remember if Hadoop uploads them into the 
dist area, but if not probably should.

> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
> work. Not just deprecated in favor of timelineserver as was advertised.

This works for me in trunk and the bash code doesn’t appear to have 
changed in a very long time.  Probably something local to your install.  (I do 
notice that the deprecation message says “starting” which is awkward when the 
stop command is given though.)  Also: is the deprecation message even true at 
this point?

>> - Cannot enable new UI in YARN because it is under a non-default
>> compilation flag. It should be on by default.
>> 
> 
> The yarn-ui profile has always been off by default, AFAIK. It's documented
> to turn it on in BUILDING.txt for release builds, and we do it in
> create-release.
> 
> IMO not a blocker. I think it's also more of a dev question (do we want to
> do this on every YARN build?) than a release one.

-1 on making yarn-ui always build.

For what is effectively an optional component (the old UI is still 
there), it’s heavy dependency requirements make it a special burden outside of 
the Docker container.  If it can be changed such that it either always 
downloads the necessary bits (regardless of the OS/chipset!) and/or doesn’t 
kill the maven build if those bits can’t be found  (i.e., truly optional), then 
I’d be less opposed.  (and, actually, quite pleased because then the docker 
image build would be significantly faster.)



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org