Re: [VOTE] Release Apache Hadoop 2.9.0 (RC2)

[Mukul Kumar Singh]

+1 (non-binding)

I built from source on Mac OS X 10.13.1 Java 1.8.0_111

- Deployed on a single node cluster.
- Deployed a ViewFS cluster with two hdfs mount points.
- Performed basic sanity checks.
- Performed basic DFS operations.

Thanks,
Mukul

> On 13-Nov-2017, at 3:01 AM, Subru Krishnan  wrote:
> 
> Hi Folks,
> 
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
> 
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
> New RC is available at: http://home.apache.org/~asuresh/hadoop-2.9.0-RC2/
> 
> 
> The RC tag in git is: release-2.9.0-RC2, and the latest commit id is:
> 1eb05c1dd48fbc9e4b375a76f2046a59103bbeb1.
> 
> The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1067/
> 
> 
> Please try the release and vote; the vote will run for the usual 5 days,
> ending on Friday 17th November 2017 2pm PT time.
> 
> We want to give a big shout out to Sunil, Varun, Rohith, Wangda, Vrushali
> and Inigo for the extensive testing/validation which helped prepare for
> RC2. Do report your results in this vote as it'll be very useful to the
> entire community.
> 
> Thanks,
> -Subru/Arun


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC2)

[Sunil G]

Hi Mukul

We have an RC3 release thread started as RC2 had an issue reported.
Kindly help to check the same

- Sunil

On Tue, Nov 14, 2017 at 3:00 PM Mukul Kumar Singh 
wrote:

> +1 (non-binding)
>
> I built from source on Mac OS X 10.13.1 Java 1.8.0_111
>
> - Deployed on a single node cluster.
> - Deployed a ViewFS cluster with two hdfs mount points.
> - Performed basic sanity checks.
> - Performed basic DFS operations.
>
> Thanks,
> Mukul
>
> > On 13-Nov-2017, at 3:01 AM, Subru Krishnan  wrote:
> >
> > Hi Folks,
> >
> > Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be
> the
> > starting release for Apache Hadoop 2.9.x line - it includes 30 New
> Features
> > with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues
> since
> > 2.8.2.
> >
> > More information about the 2.9.0 release plan can be found here:
> > *
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> > <
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> >*
> >
> > New RC is available at:
> http://home.apache.org/~asuresh/hadoop-2.9.0-RC2/
> > <
> http://www.google.com/url?q=http%3A%2F%2Fhome.apache.org%2F~asuresh%2Fhadoop-2.9.0-RC1%2F&sa=D&sntz=1&usg=AFQjCNE7BF35IDIMZID3hPqiNglWEVsTpg
> >
> >
> > The RC tag in git is: release-2.9.0-RC2, and the latest commit id is:
> > 1eb05c1dd48fbc9e4b375a76f2046a59103bbeb1.
> >
> > The maven artifacts are available via repository.apache.org at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1067/
> > <
> https://www.google.com/url?q=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066&sa=D&sntz=1&usg=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
> >
> >
> > Please try the release and vote; the vote will run for the usual 5 days,
> > ending on Friday 17th November 2017 2pm PT time.
> >
> > We want to give a big shout out to Sunil, Varun, Rohith, Wangda, Vrushali
> > and Inigo for the extensive testing/validation which helped prepare for
> > RC2. Do report your results in this vote as it'll be very useful to the
> > entire community.
> >
> > Thanks,
> > -Subru/Arun
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HADOOP-15039) move SemaphoredDelegatingExecutor to hadoop-common

[Genmao Yu (JIRA)]

Genmao Yu created HADOOP-15039:
--

 Summary: move SemaphoredDelegatingExecutor to hadoop-common
 Key: HADOOP-15039
 URL: https://issues.apache.org/jira/browse/HADOOP-15039
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs, fs/oss, fs/s3
Affects Versions: 3.0.0-beta1
Reporter: Genmao Yu
Assignee: Genmao Yu
Priority: Minor


Detailed discussions in HADOOP-14999 and HADOOP-15027.

share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.

cc [~ste...@apache.org] 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Heads up: branching branch-3.0.0 for GA

[Andrew Wang]

Hi folks,

We've resolved all the blockers for 3.0.0 and the release notes and
changelog look good, so I'm going to cut the branch and get started on the
RC.

* branch-3.0 will advance to 3.0.1-SNAPSHOT
* branch-3.0.0 will go to 3.0.0

Please keep this in mind when committing.

Cheers,
Andrew

Re: Heads up: branching branch-3.0.0 for GA

[Andrew Wang]

Branching is complete. Please use the 3.0.1 fix version for further commits
to branch-3.0. Ping me if you want something in branch-3.0.0 since I'm
rolling RC0 now.

On Tue, Nov 14, 2017 at 11:08 AM, Andrew Wang 
wrote:

> Hi folks,
>
> We've resolved all the blockers for 3.0.0 and the release notes and
> changelog look good, so I'm going to cut the branch and get started on the
> RC.
>
> * branch-3.0 will advance to 3.0.1-SNAPSHOT
> * branch-3.0.0 will go to 3.0.0
>
> Please keep this in mind when committing.
>
> Cheers,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Jonathan Hung]

Thanks Arun/Subru for working on this.

+1 (non-binding)
- Deployed RM HA on two nodes
- Tested manual failover
- Tested configuration mutation API with zk and leveldb backing store (also
ensuring configuration updates persisted on failover/restart), with queue
addition/removal/update
- Tested "yarn rmadmin -refreshQueues" enabled when configuration mutation
API disabled (and vice-versa)
- Tested queue admin configuration mutation policy



Jonathan Hung

On Mon, Nov 13, 2017 at 4:10 PM, Arun Suresh  wrote:

> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
>  Roadmap#Roadmap-Version2.9>*
>
> New RC is available at: *https://home.apache.org/~
> asuresh/hadoop-2.9.0-RC3/
> *
>
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
>
> The maven artifacts are available via repository.apache.org at:
>  apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066&sa=D&
> sntz=1&usg=AFQjCNFcern4uingMV_sEreko_zeLlgdlg>*https://
> repository.apache.org/content/repositories/orgapachehadoop-1068/
>  >*
>
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
>
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
>
> Thanks,
> -Arun/Subru
>

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Jian He]

+1 (binding)

Built from source code.
Successfully ran jobs and in the meantime tried to restart RM.

Jian

> On Nov 13, 2017, at 4:10 PM, Arun Suresh  wrote:
> 
> Hi Folks,
> 
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
> 
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
> New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> *
> 
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
> 
> The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1068/
> *
> 
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
> 
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
> 
> Thanks,
> -Arun/Subru


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew

Re: [DISCUSS] A final minor release off branch-2?

[Andrew Wang]

To follow up on my earlier email, I don't think there's need for a bridge
release given that we've successfully tested rolling upgrade from 2.x to
3.0.0. I expect we'll keep making improvements to smooth over any
additional incompatibilities found, but there isn't a requirement that a
user upgrade to a bridge release before upgrading to 3.0.

Otherwise, I don't have a strong opinion about when to discontinue branch-2
releases. Historically, a release line is maintained until interest in it
wanes. If the maintainers are taking care of the backports, it's not much
work for the rest of us to vote on the RCs.

Best,
Andrew

On Mon, Nov 13, 2017 at 4:19 PM, Wangda Tan  wrote:

> Thanks Vinod for staring this,
>
> I'm also leaning towards the plan (A):
>
>
>
>
> * (A)-- Make 2.9.x the last minor release off branch-2-- Have a
> maintenance release that bridges 2.9 to 3.x-- Continue to make more
> maintenance releases on 2.8 and 2.9 as necessary*
>
> The only part I'm not sure is having a separate bridge release other than
> 3.x.
>
> For the bridge release, Steve's suggestion sounds more doable:
>
> ** 3.1+ for new features*
> ** fixes to 3.0.x &, where appropriate, 2.9, esp feature stabilisation*
> ** whoever puts their hand up to do 2.x releases deserves support in
> testing &c*
> ** If someone makes a really strong case to backport a feature from 3.x to
> branch-2 and its backwards compatible, I'm not going to stop them. It's
> just once 3.0 is out and a 3.1 on the way, it's less compelling*
>
> This makes community can focus on 3.x releases and fill whatever gaps of
> migrating from 2.x to 3.x.
>
> Best,
> Wangda
>
>
> On Wed, Nov 8, 2017 at 3:57 AM, Steve Loughran 
> wrote:
>
>>
>> > On 7 Nov 2017, at 19:08, Vinod Kumar Vavilapalli 
>> wrote:
>> >
>> >
>> >
>> >
>> >> Frankly speaking, working on some bridging release not targeting any
>> feature isn't so attractive to me as a contributor. Overall, the final
>> minor release off branch-2 is good, we should also give 3.x more time to
>> evolve and mature, therefore it looks to me we would have to work on two
>> release lines meanwhile for some time. I'd like option C), and suggest we
>> focus on the recent releases.
>> >
>> >
>> >
>> > Answering this question is also one of the goals of my starting this
>> thread. Collectively we need to conclude if we are okay or not okay with no
>> longer putting any new feature work in general on the 2.x line after 2.9.0
>> release and move over our focus into 3.0.
>> >
>> >
>> > Thanks
>> > +Vinod
>> >
>>
>>
>> As a developer of new features (e.g the Hadoop S3A committers), I'm
>> mostly already committed to targeting 3.1; the code in there to deal with
>> failures and retries has unashamedly embraced java 8 lambda-expressions in
>> production code: backporting that is going to be traumatic in terms of
>> IDE-assisted code changes and the resultant diff in source between branch-2
>> & trunk. What's worse, its going to be traumatic to test as all my JVMs
>> start with an 8 at the moment, and I'm starting to worry about whether I
>> should bump a windows VM up to Java 9 to keep an eye on Akira's work there.
>> Currently the only testing I'm really doing on java 7 is yetus branch-2 &
>> internal test runs.
>>
>>
>> 3.0 will be out the door, and we can assume that CDH will ship with it
>> soon (*)  which will allow for a rapid round trip time on inevitable bugs:
>> 3.1 can be the release with compatibility tuned, those reported issues
>> addressed. It's certainly where I'd like to focus.
>>
>>
>> At the same time: 2.7.2-2.8.x are the broadly used versions, we can't
>> just say "move to 3.0" & expect everyone to do it, not given we have
>> explicitly got backwards-incompatible changes in. I don't seen people
>> rushing to do it until the layers above are all qualified (HBase, Hive,
>> Spark, ...). Which means big users of 2.7/2,8 won't be in a rush to move
>> and we are going to have to maintain 2.x for a while, including security
>> patches for old versions. One issue there: what if a patch (such as bumping
>> up a JAR version) is incompatible?
>>
>> For me then
>>
>> * 3.1+ for new features
>> * fixes to 3.0.x &, where appropriate, 2.9, esp feature stabilisation
>> * whoever puts their hand up to do 2.x releases deserves support in
>> testing &c
>> * If someone makes a really strong case to backport a feature from 3.x to
>> branch-2 and its backwards compatible, I'm not going to stop them. It's
>> just once 3.0 is out and a 3.1 on the way, it's less compelling
>>
>> -Steve
>>
>> Note: I'm implicitly assuming a timely 3.1 out the door with my work
>> included, all all issues arriving from 3,0 fixed. We can worry when 3.1
>> ships whether there's any benefit in maintaining a 3.0.x, or whether it's
>> best to say "move to 3.1"
>>
>>
>>
>> (*) just a guess based the effort & test reports of Andrew & others
>>
>>
>> -
>> To unsubscribe, e-mail: mapredu

[jira] [Created] (HADOOP-15040) AWS SDK NPE bug spams logs w/ Yarn Log Aggregation

[Aaron Fabbri (JIRA)]

Aaron Fabbri created HADOOP-15040:
-

 Summary: AWS SDK NPE bug spams logs w/ Yarn Log Aggregation
 Key: HADOOP-15040
 URL: https://issues.apache.org/jira/browse/HADOOP-15040
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.0.0-beta1
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri


My colleagues working with Yarn log aggregation found that they were getting 
this message spammed in their logs when they used an s3a:// URI for logs 
(yarn.nodemanager.remote-app-log-dir):

{noformat}
getting attribute Region of com.amazonaws.management:type=AwsSdkMetrics threw 
an exception
javax.management.RuntimeMBeanException: java.lang.NullPointerException
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
at 

Caused by: java.lang.NullPointerException
at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{noformat}

This happens even though the aws sdk cloudwatch metrics reporting was disabled 
(default), which is a bug. 

I filed a [github issue|https://github.com/aws/aws-sdk-java/issues/1375|] and 
it looks like a fix should be coming around SDK release 1.11.229 or so.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org