Build failed in Jenkins: Hadoop-Common-trunk #909

2013-10-02 Thread Apache Jenkins Server 
See 

Changes:

[vinodkv] YARN-1260. Added webapp.http.address to yarn-default.xml so that 
default install with https enabled doesn't have broken link on NM UI. 
Contributed by Omkar Vinit Joshi.

[cnauroth] HDFS-5279. Guard against NullPointerException in NameNode JSP pages 
before initialization of FSNamesystem. Contributed by Chris Nauroth.

[suresh] HADOOP-10012. Secure Oozie jobs fail with delegation token renewal 
exception in Namenode HA setup. Contributed by Daryn Sharp and Suresh Srinivas.

[todd] HADOOP-8315. Support SASL-authenticated ZooKeeper in 
ActiveStandbyElector. Contributed by Todd Lipcon

[cmccabe] move HADOOP-9758 to the branch-2.1.2 section

[arp] HDFS-5255. Distcp job fails with hsftp when https is enabled in insecure 
cluster.

[suresh] HADOOP-10003. HarFileSystem.listLocatedStatus() fails. Contributed by 
Jason Dere and suresh.

[vinodkv] MAPREDUCE-5536. Fixed MR AM and JHS to respect 
mapreduce.jobhistory.webapp.https.address. Contributed by Omkar Vinit Joshi.

[sandy] YARN-1262. TestApplicationCleanup relies on all containers assigned in 
a single heartbeat (Karthik Kambatla via Sandy Ryza)

[cnauroth] YARN-1215. Correct CHANGES.txt.

[jlowe] MAPREDUCE-4421. Run MapReduce framework via the distributed cache. 
Contributed by Jason Lowe

[cnauroth] YARN-1215. Yarn URL should include userinfo. Contributed by Chuan 
Liu.

[sandy] YARN-1228. Clean up Fair Scheduler configuration loading. (Sandy Ryza)

[sandy] MAPREDUCE-5544. JobClient#getJob loads job conf twice. (Sandy Ryza)

[sandy] YARN-1010. FairScheduler: decouple container scheduling from 
nodemanager heartbeats. (Wei Yan via Sandy Ryza)

[kihwal] HDFS-4512. Cover package org.apache.hadoop.hdfs.server.common with 
tests. Contributed by Vadim Bondarev.

--
[...truncated 56651 lines...]
Adding reference: maven.local.repository
[DEBUG] Initialize Maven Ant Tasks
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.6/maven-antrun-plugin-1.6.jar!/org/apache/maven/ant/tasks/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.6/maven-antrun-plugin-1.6.jar!/org/apache/maven/ant/tasks/antlib.xml
 from a zip file
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.1/ant-1.8.1.jar!/org/apache/tools/ant/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.1/ant-1.8.1.jar!/org/apache/tools/ant/antlib.xml
 from a zip file
Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader 
(parentFirst)
 +Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask
Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent 
loader (parentFirst)
 +Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask
Setting project property: test.build.dir -> 

Setting project property: test.exclude.pattern -> _
Setting project property: hadoop.assemblies.version -> 3.0.0-SNAPSHOT
Setting project property: test.exclude -> _
Setting project property: distMgmtSnapshotsId -> apache.snapshots.https
Setting project property: project.build.sourceEncoding -> UTF-8
Setting project property: java.security.egd -> file:///dev/urandom
Setting project property: distMgmtSnapshotsUrl -> 
https://repository.apache.org/content/repositories/snapshots
Setting project property: distMgmtStagingUrl -> 
https://repository.apache.org/service/local/staging/deploy/maven2
Setting project property: avro.version -> 1.7.4
Setting project property: test.build.data -> 

Setting project property: commons-daemon.version -> 1.0.13
Setting project property: hadoop.common.build.dir -> 

Setting project property: testsThreadCount -> 4
Setting project property: maven.test.redirectTestOutputToFile -> true
Setting project property: jdiff.version -> 1.0.9
Setting project property: distMgmtStagingName -> Apache Release Distribution 
Repository
Setting project property: project.reporting.outputEncoding -> UTF-8
Setting project property: build.platform -> Linux-i386-32
Setting project property: protobuf.version -> 2.5.0
Setting project property: failIfNoTests -> false
Setting project property: protoc.path -> ${env.HADOOP_PROTOC_PATH}
Setting project property: jersey.version -> 1.9
Setting project property: distMgmtStagingId -> apache.staging.https
Setting project property: distMgmtSnapshotsName -> Apache Development Snapshot 
Repository
Setting project property: ant.file ->

Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)

2013-10-02 Thread Colin McCabe 
On Tue, Oct 1, 2013 at 8:59 PM, Arun C Murthy  wrote:
> Yes, sorry if it wasn't clear.
>
> As others seem to agree, I think we'll be better getting a protocol/api 
> stable GA done and then iterating on bugs etc.
>
> I'm not super worried about HADOOP-9984 since symlinks just made it to 
> branch-2.1 recently.
>
> Currently we only have 2 blockers: HADOOP-9984 & MAPREDUCE-5530. Both of 
> which are PA and I've reviewed MR-5530 and is good to go (thanks Robert). 
> Hopefully we can finish up HADOOP-9984 asap and we'll be good.

We've had several reviews for HADOOP-9984 and are currently just
waiting on a +1.

Sorry if this is a dumb question, but will the 2.2 release be made
from branch-2 or what is currently named branch-2.1-beta?  If it is
the latter, we have a few backports we'll need to do.

Colin


>
> thanks,
> Arun
>
> On Oct 1, 2013, at 4:53 PM, Alejandro Abdelnur  wrote:
>
>> Arun,
>>
>> Does this mean that you want to skip a beta release and go straight to GA
>> with the next release?
>>
>> thx
>>
>>
>> On Tue, Oct 1, 2013 at 4:15 PM, Arun C Murthy  wrote:
>>
>>> Guys,
>>>
>>> I took a look at the content in 2.1.2-beta so far, other than the
>>> critical fixes such as HADOOP-9984 (symlinks) and few others in YARN/MR,
>>> there is fairly little content (unit tests fixes etc.)
>>>
>>> Furthermore, it's standing up well in testing too. Plus, the protocols
>>> look good for now (I wrote a gohadoop to try convince myself), let's lock
>>> them in.
>>>
>>> Given that, I'm thinking we can just go ahead rename it 2.2.0 rather than
>>> make another 2.1.x release.
>>>
>>> This will drop a short-lived release (2.1.2) and help us move forward on
>>> 2.3 which has a fair bunch of content already...
>>>
>>> Thoughts?
>>>
>>> thanks,
>>> Arun
>>>
>>>
>>> On Sep 24, 2013, at 4:24 PM, Zhijie Shen  wrote:
>>>
 I've added MAPREDUCE-5531 to the blocker list. - Zhijie


 On Tue, Sep 24, 2013 at 3:41 PM, Arun C Murthy 
>>> wrote:

> With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out…
>>> I'll
> make it clear on the release page, that there are some known issues and
> that we will follow up very shortly with another release.
>
> Meanwhile, let's fix the remaining blockers (please mark them as such
>>> with
> Target Version 2.1.2-beta).
> The current blockers are here:
> http://s.apache.org/hadoop-2.1.2-beta-blockers
>
> thanks,
> Arun
>
> On Sep 16, 2013, at 11:38 PM, Arun C Murthy 
>>> wrote:
>
>> Folks,
>>
>> I've created a release candidate (rc0) for hadoop-2.1.1-beta that I
> would like to get released - this release fixes a number of bugs on top
>>> of
> hadoop-2.1.0-beta as a result of significant amounts of testing.
>>
>> If things go well, this might be the last of the *beta* releases of
> hadoop-2.x.
>>
>> The RC is available at:
> http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0
>> The RC tag in svn is here:
>
>>> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0
>>
>> The maven artifacts are available via repository.apache.org.
>>
>> Please try the release and vote; the vote will run for the usual 7
>>> days.
>>
>> thanks,
>> Arun
>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
>>> entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the
>>> reader
> of this message is not the intended recipient, you are hereby notified
>>> that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender
>>> immediately
> and delete it from your system. Thank You.
>



 --
 Zhijie Shen
 Hortonworks Inc.
 http://hortonworks.com/

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
>>> to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified
>>> that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
>>> immediately
 and delete it from your system. Thank You.
>>>
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
>>>
>>

Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)

2013-10-02 Thread Roman Shaposhnik 
On Tue, Oct 1, 2013 at 5:15 PM, Vinod Kumar Vavilapalli
 wrote:
> +1. We should get an RC as soon as possible so that we can get all the 
> downstream components to sign off.
> The earlier the better.

On this very note -- would there be any interest in joining efforts
with the Bigtop integration aimed at Hadoop 2.2.x based release
of all the Hadoop ecosystem projects?

Our current plan is to release Bigtop 0.7.0 within a couple of weeks.
That will be the last stable 2.0.x-based release. Bigtop 0.8.0 is supposed to
be based on Hadoop 2.x that gets us (Bigtop community) as close as possible
to the Hadoop's GA. Here's more on what we'll be doing with Bigtop 0.8.0:
 http://comments.gmane.org/gmane.comp.apache.incubator.bigtop.devel/10769

Of course, on the Bigtop side of things we're stuck with all the necessary
integration work anyway, but if there's anything at all that folks are willing
to help us and the bigger Hadoop community with that would be very
much appreciated. I think both communities will benefit from this type
of collaboration.

On a practical side of things, as soon as the branch for 2.2.0 gets cut
Bigtop can start publishing a complete set of Hadoop ecosystem
artifacts built against that particular version and easily install-able
on all of our supported systems. We can also start publishing VMs
so that folks on OSes other than Linux can help us with testing.

Thanks,
Roman.

Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)

2013-10-02 Thread Andrew Wang 
If we're serious about not breaking compatibility after GA, then we need to
slow down and make sure we get these new APIs right, or can add them in a
compatible fashion.

HADOOP-9984 ended up being a bigger change than initially expected, and we
need to break compatibility with out-of-tree FileSystems to do it properly.
I would like to see HADOOP-9972 in as well (globLinkStatus), and there are
open questions on HADOOP-9984 about changing PathFilter and
FileStatus.getPath() semantics (which would be incompatible). Yes, we could
just +1 HADOOP-9984 and stamp 2.2.0 on it, but I think it looks bad to then
immediately turn around and release an incompatible 2.3.

My preference is still for a 2.1.2 with the above API questions resolved,
then an actual API-stable 2.2.0 GA. This is already punting out all the
other related FS/tooling changes that we think can be done compatibly but
are still pretty crucial: shell, distcp, webhdfs, hftp; it'd be great to
get help on any of these.

Thanks,
Andrew


On Wed, Oct 2, 2013 at 2:56 PM, Roman Shaposhnik  wrote:

> On Tue, Oct 1, 2013 at 5:15 PM, Vinod Kumar Vavilapalli
>  wrote:
> > +1. We should get an RC as soon as possible so that we can get all the
> downstream components to sign off.
> > The earlier the better.
>
> On this very note -- would there be any interest in joining efforts
> with the Bigtop integration aimed at Hadoop 2.2.x based release
> of all the Hadoop ecosystem projects?
>
> Our current plan is to release Bigtop 0.7.0 within a couple of weeks.
> That will be the last stable 2.0.x-based release. Bigtop 0.8.0 is supposed
> to
> be based on Hadoop 2.x that gets us (Bigtop community) as close as possible
> to the Hadoop's GA. Here's more on what we'll be doing with Bigtop 0.8.0:
>
> http://comments.gmane.org/gmane.comp.apache.incubator.bigtop.devel/10769
>
> Of course, on the Bigtop side of things we're stuck with all the necessary
> integration work anyway, but if there's anything at all that folks are
> willing
> to help us and the bigger Hadoop community with that would be very
> much appreciated. I think both communities will benefit from this type
> of collaboration.
>
> On a practical side of things, as soon as the branch for 2.2.0 gets cut
> Bigtop can start publishing a complete set of Hadoop ecosystem
> artifacts built against that particular version and easily install-able
> on all of our supported systems. We can also start publishing VMs
> so that folks on OSes other than Linux can help us with testing.
>
> Thanks,
> Roman.
>

Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)

2013-10-02 Thread Colin McCabe 
I don't think HADOOP-9972 is a must-do for the next Apache release,
whatever version number it ends up having.  It's just adding a new
API, not changing any existing ones, and it can be done entirely in
generic code.  (The globber doesn't involve FileSystem or AFS
subclasses).

My understanding is that GA is about stabilizing APIs rather than
achieving feature completeness on symlinks.

Colin

On Wed, Oct 2, 2013 at 6:01 PM, Andrew Wang  wrote:
> If we're serious about not breaking compatibility after GA, then we need to
> slow down and make sure we get these new APIs right, or can add them in a
> compatible fashion.
>
> HADOOP-9984 ended up being a bigger change than initially expected, and we
> need to break compatibility with out-of-tree FileSystems to do it properly.
> I would like to see HADOOP-9972 in as well (globLinkStatus), and there are
> open questions on HADOOP-9984 about changing PathFilter and
> FileStatus.getPath() semantics (which would be incompatible). Yes, we could
> just +1 HADOOP-9984 and stamp 2.2.0 on it, but I think it looks bad to then
> immediately turn around and release an incompatible 2.3.
>
> My preference is still for a 2.1.2 with the above API questions resolved,
> then an actual API-stable 2.2.0 GA. This is already punting out all the
> other related FS/tooling changes that we think can be done compatibly but
> are still pretty crucial: shell, distcp, webhdfs, hftp; it'd be great to
> get help on any of these.
>
> Thanks,
> Andrew
>
>
> On Wed, Oct 2, 2013 at 2:56 PM, Roman Shaposhnik  wrote:
>
>> On Tue, Oct 1, 2013 at 5:15 PM, Vinod Kumar Vavilapalli
>>  wrote:
>> > +1. We should get an RC as soon as possible so that we can get all the
>> downstream components to sign off.
>> > The earlier the better.
>>
>> On this very note -- would there be any interest in joining efforts
>> with the Bigtop integration aimed at Hadoop 2.2.x based release
>> of all the Hadoop ecosystem projects?
>>
>> Our current plan is to release Bigtop 0.7.0 within a couple of weeks.
>> That will be the last stable 2.0.x-based release. Bigtop 0.8.0 is supposed
>> to
>> be based on Hadoop 2.x that gets us (Bigtop community) as close as possible
>> to the Hadoop's GA. Here's more on what we'll be doing with Bigtop 0.8.0:
>>
>> http://comments.gmane.org/gmane.comp.apache.incubator.bigtop.devel/10769
>>
>> Of course, on the Bigtop side of things we're stuck with all the necessary
>> integration work anyway, but if there's anything at all that folks are
>> willing
>> to help us and the bigger Hadoop community with that would be very
>> much appreciated. I think both communities will benefit from this type
>> of collaboration.
>>
>> On a practical side of things, as soon as the branch for 2.2.0 gets cut
>> Bigtop can start publishing a complete set of Hadoop ecosystem
>> artifacts built against that particular version and easily install-able
>> on all of our supported systems. We can also start publishing VMs
>> so that folks on OSes other than Linux can help us with testing.
>>
>> Thanks,
>> Roman.
>>