Re: [VOTE] Hadoop 3.1.x EOL

[Sean Busbey]

+1

> On Jun 3, 2021, at 1:14 AM, Akira Ajisaka  wrote:
> 
> Dear Hadoop developers,
> 
> Given the feedback from the discussion thread [1], I'd like to start
> an official vote
> thread for the community to vote and start the 3.1 EOL process.
> 
> What this entails:
> 
> (1) an official announcement that no further regular Hadoop 3.1.x releases
> will be made after 3.1.4.
> (2) resolve JIRAs that specifically target 3.1.5 as won't fix.
> 
> This vote will run for 7 days and conclude by June 10th, 16:00 JST [2].
> 
> Committers are eligible to cast binding votes. Non-committers are welcomed
> to cast non-binding votes.
> 
> Here is my vote, +1
> 
> [1] https://s.apache.org/w9ilb
> [2] 
> https://www.timeanddate.com/worldclock/fixedtime.html?msg=4&iso=20210610T16&p1=248
> 
> Regards,
> Akira
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

[Sean Busbey]

Sounds good to me. That would be until Thursday June 10th, right?

As a side note it’s concerning that a double-dot maintenance release is a big 
release, but I get that it’s the current state of the project.

> On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang  wrote:
> 
> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
> 
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
> 
> Looks like we may need some more time to test. How about extending it by a
> week?
> 
> On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang  wrote:
> 
>> Hi community,
>> 
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>> 
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>> 
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> 
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>> 
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> 
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> 
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> 
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> 
>> 
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>> 
>> Please try the release and vote. The vote will run for 5 days.
>> 
>> My +1 to start,
>> 
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> 
>> 
>> 



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker

[Sean Busbey]

Hi Brahma!

Thanks for organizing this. What’s the timezone for the 10p - midnight? Pacific 
Time?

> On Sep 8, 2021, at 1:17 AM, Brahma Reddy Battula  wrote:
> 
> Hi All,
> 
> Updated the meeting to record the session.. Please use the following link
> to attend the conference tomorrow.
> 
> 
> Uber's story on running Apache Hadoop deployment in Docker
> September 9, 2021, 10:00pm – September 10, 2021, 12:00am ·
> Google Meet joining info
> *Video call link: https://meet.google.com/few-wppc-xoa
> *
> Or dial: ‪(US) +1 443-424-3811‬ PIN: ‪384 713 518‬#
> More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915
> 
> On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula 
> wrote:
> 
>> Hi All,
>> 
>> Happy to announce the following conference. Block your calendar and make
>> yourself available.
>> Thanks Mithun and team for accepting.
>> 
>> 
>> *Topic:*Uber's story on running Apache Hadoop deployment in Docker
>> 
>> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*
>> 
>> *Meeting Link :   *Google Meet joining info
>>   Video call link:
>> https://meet.google.com/akk-qmzy-qsu
>> 
>> *  Note: *Created Teams meeting also as a backup
>> [2].
>> 
>> *High level Agenda: *
>> 
>>   - Importance of Hadoop Cluster Management (3-5 min)
>>   - Introduction about problem space
>>   - Challenges in this space and why it is important to solve them
>>   - Evolution of Uber Hadoop Cluster Management (10-15 min)
>>   -  How our strategy evolved over time since Hadoop was set up at Uber
>>   -  Key learnings from the evolution over the years
>>   - Our current approach today (15-20min)
>>   -  Key learnings from this major overhaul
>>   -  Current challenges that we are working on
>>   - Q&A (20 min)
>> 
>> 
>> Aiming for 20 min Q&A assuming that there will be more questions based on
>> the blog that was published(1) + presentation at the session.
>> 
>> 
>> 
>> 
>> ---
>> 1) https://eng.uber.com/hadoop-container-blog/
>> 
>> 2)
>> 
>> *Join on your computer or mobile app*
>> 
>> Click here to join the meeting
>> 
>> 
>> *Join with a video conferencing device*
>> 
>> 967904...@t.plcm.vc
>> 
>> Video Conference ID: 117 464 626 6
>> 
>> Alternate VTC instructions
>> 
>> 
>> *Or call in (audio only)*
>> 
>> +1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States,
>> Los Angeles
>> 
>> (833) 827-4491,,273569658# <8338274491,,273569658#>   United States
>> (Toll-free)
>> 
>> Phone Conference ID: 273 569 658#
>> 
>> Find a local number
>> 
>> | Reset PIN 
>> 
>> 
>> 
>> -- Brahma Reddy Battula
>> 
> 
> 
> -- 
> 
> 
> 
> --Brahma Reddy Battula



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Migrate to Yetus Interface classification annotations

[Sean Busbey]

I think consolidating on a common library and tooling for defining API 
expectations for Hadoop would be great.

Unfortunately, the Apache Yetus community recently started a discussion around 
dropping their maintenance of the audience annotations codebase[1] due to lack 
of community interest. In particular, there has been an outstanding problem 
with doclet support for filtering javadocs by annotation since JDK9 came out.

I think that means a necessary first step here would be to determine if we have 
contributors willing to show up over in that project to get things into a good 
state for future JDK adoption.



[1]:
https://s.apache.org/ybdl6
"[DISCUSS] Drop JDK8; audience-annotations" from d...@yetus.apache.org

> On Sep 27, 2021, at 2:46 AM, Viraj Jasani  wrote:
> 
> Since the early days, Hadoop has provided Interface classification
> annotations to represent the scope and stability for downstream
> applications to select Hadoop APIs carefully. After some time, these
> annotations (InterfaceAudience and InterfaceStability) have been migrated
> to Apache Yetus. As of today, with increasing number of Hadoop ecosystem
> applications using (or starting to use) Yetus stability annotations for
> their own downstreamers, we should also consider using IA/IS annotations
> provided by *org.apache.yetus.audience *directly in our codebase and retire
> our *org.apache.hadoop.classification* package for the better separation of
> concern and single source.
> 
> I believe we can go with this migration to maintain compatibility for
> Hadoop downstreamers:
> 
>   1. In Hadoop trunk (3.4.0+ releases), replace all usages of o.a.h.c
>   stability annotations with o.a.y.a annotations.
>   2. Deprecate o.a.h.c annotations, and provide deprecation warning that
>   we will remove o.a.h.c in 4.0.0 (or 5.0.0) release and the only source for
>   these annotations should be o.a.y.a.
> 
> Any thoughts?



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merging PRs vs. commit from CLI and keep committer field

[Sean Busbey]

If you add a line in the commit message that the commit closes a given PR # 
then GitHub will annotate the PR as related to the specific commit and close it 
for you.

i.e. you can add “closes #3454” to the commit message body and then PR 3454 
will close and link to that commit when it hits the default branch.

I believe these docs describing associating a GitHub issue via a commit message 
also apply to associating commits to pull requests, if you are interested in 
what specific phrases work:

https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword


Has anyone tried handling a squash merge via the GitHub CLI tooling rather than 
the web or plain git tooling? Does it similarly overwrite committer metadata?

e.g. the GitHub CLI example from the merging PRs docs?

https://docs.github.com/en/github/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/merging-a-pull-request#merging-a-pull-request
 




> On Oct 25, 2021, at 9:57 AM, Szilárd Németh  wrote:
> 
> Hi Ayush,
> Thanks a lot for your response.
> 
> 
>> Yahh, remember chasing Github support regarding this last year and they
>> reverted back that they are aware of this and have an internal jira for
>> this but can not comment upon how much time it would take. (As of now 1
>> year & counting)
>> 
> So if I get this right: Their internal jira is in the same untouched state
> and they wouldn't make any progress?
> 
> 
> Just in case you want to use the CLI & still make the PR marked as merged,
>> A dirty way to do this is:
>> Pull the changes to your local.
>> Rebase & Squash all commits, in the commit message in the end put actual
>> the PR number, eg. #1234,
>> force push to the PR, that is the contributors fork, this will update his
>> PR and then merge the Branch to trunk in your local and push. It marks
>> the PR as merged, at least last year it did when I tried. :-)
>> 
> 
> Thanks for this dirty hack :)
> Probably I will refrain from doing this, I don't like to force push if it's
> not "mandatory".
> Anyway, if the community is fine with it, I will continue to commit from
> the CLI and close the PR, even though it's not that convenient.
> 
> 
> Best,
> Szilard
> 
> 
> 
> On Wed, 20 Oct 2021 at 05:07, Ayush Saxena  wrote:
> 
>> Yahh, remember chasing Github support regarding this last year and they
>> reverted back that they are aware of this and have an internal jira for
>> this but can not comment upon how much time it would take. (As of now 1
>> year & counting)
>> 
>> As far as I remember this is with squash & commit only. If you do say a
>> rebase & merge. The commit email id is preserved. But other options we have
>> blocked.
>> 
>> I think most of the projects are using squash & merge and many people
>> won’t be ok to use CLI rather than UI
>> So, I don’t think we have an ALT here.
>> 
>> Just in case you want to use the CLI & still make the PR marked as merged,
>> A dirty way to do this is:
>> Pull the changes to your local.
>> Rebase & Squash all commits, in the commit message in the end put actual
>> the PR number, eg. #1234,
>> force push to the PR, that is the contributors fork, this will update his
>> PR and then merge the Branch to trunk in your local and push. It marks the
>> PR as merged, at least last year it did when I tried. :-)
>> 
>> -Ayush
>> 
>> 
>>> On 20-Oct-2021, at 3:27 AM, Szilárd Németh  wrote:
>>> 
>>> Hi,
>>> 
>>> I noticed something strange in our commits, in particular the committer
>>> field is not reflecting the user who committed the commit.
>>> 
>>> *1. First, I wanted to check Gergely's commits from the last month or so.
>>> This was getting to be suspicious as I expected to see a bunch of commits
>>> from Sept / Oct of this year. *
>>> 
>>> *git log CLI output:*
>>> ➜ git --no-pager log --format=fuller --committer=shuzirra
>>> commit 44bab51be44e31224dabbfa548eb27ea5fb2f916
>>> Author: Gergely Pollak 
>>> AuthorDate: Wed Aug 4 15:43:07 2021 +0200
>>> Commit: Gergely Pollak 
>>> CommitDate: Wed Aug 4 15:43:57 2021 +0200
>>> 
>>> 
>>>   YARN-10849 Clarify testcase documentation for
>>> TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by
>>> Szilard Nemeth
>>> 
>>> 
>>> commit e9339aa3761295fe65bb786e01938c7c177cd6e7
>>> Author: Gergely Pollak 
>>> AuthorDate: Tue Jun 1 15:57:22 2021 +0200
>>> Commit: Gergely Pollak 
>>> CommitDate: Tue Jun 1 15:57:22 2021 +0200
>>> 
>>> 
>>>   YARN-10797. Logging parameter issues in scheduler package. Contributed
>>> by Szilard Nemeth
>>> 
>>> 
>>> *2. Another example of a merged PR, here I was the author and Adam Antal
>>> was the committer:  *
>>> PR link: https://github.com/apache/hadoop/pull/3454
>>> 
>>> *git log CLI output:*
>>> ➜ git --no-pager log --format=fuller a9

Re: How should we do about dependency update?

[Sean Busbey]

speaking with my HBase hat on instead of my Hadoop hat, when the
Hadoop project publishes that there's a CVE but does not include a
maintenance release that mitigates it for a given minor release line,
we assume that means the Hadoop project is saying that release line is
EOM and should be abandoned.

I don't know if that's an accurate interpretation in all cases.

With my Hadoop hat on, I think downstream projects should use the
interfaces we say are safe to use and those interfaces should not
include dependencies where practical. I don't know how often a CVE
comes along for things like our logging API dependency, for example.
But downstream folks should definitely not rely on dependencies we use
for internal service, so I'm surprised that a version change for Jetty
would impact downstream.


On Mon, Oct 21, 2019 at 12:33 PM Wei-Chiu Chuang  wrote:
>
> Hi Hadoop developers,
>
> I've always had this question and I don't know the answer.
>
> For the last few months I finally spent time to deal with the vulnerability
> reports from our internal dependency check tools.
>
> Say in HADOOP-16152 
> we update Jetty from 9.3.27 to 9.4.20 because of CVE-2019-16869, should I
> cherrypick the fix into all lower releases?
> This is not a trivial change, and it breaks downstreams like Tez. On the
> other hand, it doesn't seem reasonable if I put this fix only in trunk, and
> left older releases vulnerable. What's the expectation of downstream
> applications w.r.t breaking compatibility vs fixing security issues?
>
> Thoughts?



--
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[DISCUSS] check style changes

[Sean Busbey]

Hi folks!

I’d like to start cleaning up our nightly tests. As a bit of low hanging fruit 
I’d like to alter some of our check style rules to match what I think we’ve 
been doing in the community. How would folks prefer I make sure we have 
consensus on such changes?

As an example, our last nightly run had ~81k check style violations (it’s a big 
number but it’s not that bad given the size of the repo) and roughly 16% of 
those were for line lengths in excess of 80 characters but <= 100 characters.

If I wanted to change our line length check to be 100 characters rather than 
the default of 80, would folks rather I have a DISCUSS thread first? Or would 
they rather a Jira + PR with the discussion of the merits happening there?

—
busbey



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[DISCUSS] Change project style guidelines to allow line length 100

[Sean Busbey]

Hello!

What do folks think about changing our line length guidelines to allow for 100 
character width?

Currently, we tell folks to follow the sun style guide with some exception 
unrelated to line length. That guide says width of 80 is the standard and our 
current check style rules act as enforcement.

Looking at the current trunk codebase our nightly build shows a total of ~15k 
line length violations; it’s about 18% of identified checkstyle issues.

The vast majority of those line length violations are <= 100 characters long. 
100 characters happens to be the length for the Google Java Style Guide, 
another commonly adopted style guide for java projects, so I suspect these 
longer lines leaking past the checkstyle precommit warning might be a 
reflection of committers working across multiple java codebases.

I don’t feel strongly about lines being longer, but I would like to move 
towards more consistent style enforcement as a project. Updating our project 
guidance to allow for 100 character lines would reduce the likelihood that 
folks bringing in new contributions need a precommit test cycle to get the 
formatting correct.

Does anyone feel strongly about keeping the line length limit at 80 characters?

Does anyone feel strongly about contributions coming in that clear up line 
length violations?


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Change project style guidelines to allow line length 100

[Sean Busbey]

Hi Bhavik!

What concerns do you have about back porting patches to earlier release 
branches?

If we change our style guidelines then presumably we can do that for all 
branches, so a backport from e.g. trunk to branch-3.3 won’t fail a style check 
on the destination branch unless something changed in the backporting.

If you are referring to patches for clearing up line length violations, my 
usual preference is to aim for my changes to be on all active release lines. So 
at least in the case of the patches coming from me or being committed by me, 
there’d be effort to make sure all branches end up as easy to backport to as 
they were prior to the clean up.



> On May 20, 2021, at 2:27 AM, Bhavik Patel  wrote:
> 
> I am just worried about the backporting of the Jira to child branch!! How
> we are planning to handle this?
> 
> On Thu, May 20, 2021, 11:09 AM Qi Zhu <821684...@qq.com 
> <mailto:821684...@qq.com>> wrote:
> 
>> +1 100 is reasonable.
>> 
>> 
>> 
>> ---Original---
>> From: "Xiaoqiao He"mailto:hexiaoq...@apache.org>>
>> Date: Thu, May 20, 2021 13:35 PM
>> To: "Masatake Iwasaki"> <mailto:iwasak...@oss.nttdata.co.jp>>;
>> Cc: "Akira Ajisaka"> <mailto:aajis...@apache.org>>;"Hadoop Common"<
>> common-...@hadoop.apache.org 
>> <mailto:common-...@hadoop.apache.org>>;"Hdfs-dev">  <mailto:hdfs-dev@hadoop.apache.org>
>> >;"yarn-dev"> <mailto:yarn-...@hadoop.apache.org>>;"mapreduce-dev"<
>> mapreduce-...@hadoop.apache.org <mailto:mapreduce-...@hadoop.apache.org>>;
>> Subject: Re: [DISCUSS] Change project style guidelines to allow line
>> length 100
>> 
>> 
>> +1 for <= 100 chars long per line length.
>> 
>> On Thu, May 20, 2021 at 10:28 AM Masatake Iwasaki <
>> iwasak...@oss.nttdata.co.jp> wrote:
>> 
>> > I'm +1 too.
>> > I feel 80 characters limit tends to degrade readability by introducing
>> > useless line breaks.
>> >
>> > >
>> >
>> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
>>  
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>
>> >
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>
>>  
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>>>
>> ;
>> > I have no inconvenience on 100 characters for using Emacs and
>> side-by-side
>> > diff even on 13-inch MBP.
>> >
>> > Masatake Iwasaki
>> >
>> > On 2021/05/20 11:00, Akira Ajisaka wrote:
>> > > I'm +1 to allow <= 100 chars.
>> > >
>> > > FYI: There were some discussions long before:
>> > > -
>> >
>> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
>>  
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>
>> >
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>
>>  
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>>>;
>> > -
>> >
>> https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E
>>  
>> <https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E>
>> >
>> <https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E>
>>  
>> <https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E>>>;
>> >
>> > > Thanks,
>> > > Akira
>> > >
>> > > On Thu, May 20, 2021 at 6:36 AM Sean Busbey
>> mailto:sbus...@apple.com.invalid>>
>> > wrote:
>> &

[DISCUSS] which release lines should we still consider actively maintained?

[Sean Busbey]

Hi folks!

Which release lines do we as a community still consider actively maintained?

I found an earlier discussion[1] where we had consensus to consider branches 
that don’t get maintenance releases on a regular basis end-of-life for 
practical purposes. The result of that discussion was written up in our wiki 
docs in the “EOL Release Branches” page, summarized here

>  If no volunteer to do a maintenance release in a short to mid-term (like 3 
> months to 1 or 1.5 year).

Looking at release lines that are still on our download page[3]:

* Hadoop 2.10.z - last release 8 months ago
* Hadoop 3.1.z - last release 9.5 months ago
* Hadoop 3.2.z - last release 4.5 months ago
* Hadoop 3.3.z - last release 10 months ago

And then trunk holds 3.4 which hasn’t had a release since the branch-3.3 fork 
~14 months ago.

I can see that Wei-Chiu has been actively working on getting the 3.3.1 release 
out[4] (thanks Wei-Chiu!) but I do not see anything similar for the other 
release lines.

We also have pages on the wiki for our project roadmap of release[5], but it 
seems out of date since it lists in progress releases that have happened or 
branches we have announced as end of life, i.e. 2.8.

We also have a group of pages (sorry, I’m not sure what the confluence jargon 
is for this) for “hadoop active release lines”[6] but this list has 2.8, 2.9, 
3.0, 3.1, and 3.3. So several declared end of life lines and no 2.10 or 3.2 
despite those being our release lines with the most recent releases.

Are there folks willing to go through being release managers to get more of 
these release lines on a steady cadence?

If I were to take up maintenance release for one of them which should it be?

Should we declare to our downstream users that some of these lines aren’t going 
to get more releases?

Is there downstream facing documentation somewhere that I missed for setting 
expectations about our release cadence and actively maintained branches?

Do we have a backlog of work written up that could make the release process 
easier for our release managers?


[1]: https://s.apache.org/7c8jt
[2]: https://s.apache.org/4no96
[3]: https://hadoop.apache.org/releases.html
[4]: https://s.apache.org/1bvwe
[5]: https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
[6]: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Change project style guidelines to allow line length 100

[Sean Busbey]

Hi folks!

The consensus seems pretty strongly in favor of increasing the line length 
limit. Do folks still want to see a formal VOTE thread?


> On May 19, 2021, at 4:22 PM, Sean Busbey  wrote:
> 
> Hello!
> 
> What do folks think about changing our line length guidelines to allow for 
> 100 character width?
> 
> Currently, we tell folks to follow the sun style guide with some exception 
> unrelated to line length. That guide says width of 80 is the standard and our 
> current check style rules act as enforcement.
> 
> Looking at the current trunk codebase our nightly build shows a total of ~15k 
> line length violations; it’s about 18% of identified checkstyle issues.
> 
> The vast majority of those line length violations are <= 100 characters long. 
> 100 characters happens to be the length for the Google Java Style Guide, 
> another commonly adopted style guide for java projects, so I suspect these 
> longer lines leaking past the checkstyle precommit warning might be a 
> reflection of committers working across multiple java codebases.
> 
> I don’t feel strongly about lines being longer, but I would like to move 
> towards more consistent style enforcement as a project. Updating our project 
> guidance to allow for 100 character lines would reduce the likelihood that 
> folks bringing in new contributions need a precommit test cycle to get the 
> formatting correct.
> 
> Does anyone feel strongly about keeping the line length limit at 80 
> characters?
> 
> Does anyone feel strongly about contributions coming in that clear up line 
> length violations?
> 

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: continuing releases on Apache Hadoop 2.6.x

[Sean Busbey]

Early december would be great, presuming the RC process doesn't take too
long. By then it'll already have over a month since the 2.6.2 release and
I'm sure the folks contributing the 18 patches we already have in would
like to see their work out there.

On Fri, Nov 20, 2015 at 7:51 AM, Junping Du  wrote:

> +1. Early Dec sounds too early for 2.6.3 release given we only have 18
> patches since recently release 2.6.2.
> We should nominate more fixes and wait a while for the feedback on 2.6.2.
>
> Thanks,
>
> Junping
> 
> From: Vinod Vavilapalli 
> Sent: Thursday, November 19, 2015 11:34 PM
> To: yarn-...@hadoop.apache.org
> Cc: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org
> Subject: Re: continuing releases on Apache Hadoop 2.6.x
>
> I see 18 JIRAs across the sub-projects as of now in 2.6.3. Seems like we
> will have a reasonable number of fixes if we start an RC early december.
>
> In the mean while, we should also review 2.7.3 and 2.8.0 blocker /
> critical list and see if it makes sense to backport any of those into 2.6.3.
>
> +Vinod
>
>
> On Nov 17, 2015, at 5:10 PM, Sangjin Lee  sj...@apache.org>> wrote:
>
> I'd like to pick up this email discussion again. It is time that we started
> thinking about the next release in the 2.6.x line. IMO we want to walk the
> balance between maintaining a reasonable release cadence and getting a good
> amount of high-quality fixes. The timeframe is a little tricky as the
> holidays are approaching. If we have enough fixes accumulated in
> branch-2.6, some time early December might be a good target for cutting the
> first release candidate. Once we miss that window, I think we are looking
> at next January. I'd like to hear your thoughts on this.
>
>


-- 
Sean

[DISCUSS] moving to Apache Yetus Audience Annotations

[Sean Busbey]

When Apache Yetus formed, it started with several key pieces of Hadoop that
looked reusable. In addition to our contribution testing infra, the project
also stood up a version of our audience annotations for delineating the
public facing API[1].

I recently got the Apache HBase community onto the Yetus version of those
annotations rather than their internal fork of the Hadoop ones[2]. It
wasn't pretty, mostly a lot of blind sed followed by spot checking and
reliance on automated tests.

What do folks think about making the jump ourselves? I'd be happy to work
through things, either as one unreviewable monster or per-module
transitions (though a piece-meal approach might complicate our javadoc
situation).


[1]: http://yetus.apache.org/documentation/0.5.0/interface-classification/
[2]: https://issues.apache.org/jira/browse/HBASE-17823

-- 
busbey

Re: [DISCUSS] moving to Apache Yetus Audience Annotations

[Sean Busbey]

I'd refer to it as an incompatible change; we expressly label the
annotations as IA.Public.

If you think it's too late to get in for 3.0, I can make a jira and put it
on the back burner for when trunk goes to 4.0?

On Fri, Sep 22, 2017 at 12:49 PM, Andrew Wang 
wrote:

> Is this itself an incompatible change? I imagine the bytecode will be
> different.
>
> I think we're too late to do this for beta1 given that I want to cut an
> RC0 today.
>
> On Fri, Sep 22, 2017 at 7:03 AM, Sean Busbey  wrote:
>
>> When Apache Yetus formed, it started with several key pieces of Hadoop
>> that
>> looked reusable. In addition to our contribution testing infra, the
>> project
>> also stood up a version of our audience annotations for delineating the
>> public facing API[1].
>>
>> I recently got the Apache HBase community onto the Yetus version of those
>> annotations rather than their internal fork of the Hadoop ones[2]. It
>> wasn't pretty, mostly a lot of blind sed followed by spot checking and
>> reliance on automated tests.
>>
>> What do folks think about making the jump ourselves? I'd be happy to work
>> through things, either as one unreviewable monster or per-module
>> transitions (though a piece-meal approach might complicate our javadoc
>> situation).
>>
>>
>> [1]: http://yetus.apache.org/documentation/0.5.0/interface-classi
>> fication/
>> [2]: https://issues.apache.org/jira/browse/HBASE-17823
>>
>> --
>> busbey
>>
>
>


-- 
busbey

Re: Do we still have nightly (or even weekly) unit test run for Hadoop projects?

[Sean Busbey]

Here's the email from last night to common-dev@hadoop:

https://s.apache.org/ARe1

On Wed, Oct 18, 2017 at 10:42 PM, Akira Ajisaka  wrote:

> Yes, qbt runs nightly and it sends e-mail to dev lists.
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
>
> Regards,
> Akira
>
>
> On 2017/10/19 7:54, Wangda Tan wrote:
>
>> Hi,
>>
>> Do we still have nightly (or even weekly) unit test run for Hadoop
>> projects? I couldn't find it on Jenkins dashboard and I haven't seen
>> reports set to dev lists for a while.
>>
>> Thanks,
>> Wangda
>>
>>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>


-- 
busbey

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Sean Busbey]

Just curious, Junping what would "solid evidence" look like? Is the
supposition here that the memory leak is within HDFS test code rather than
library runtime code? How would such a distinction be shown?

On Tue, Oct 24, 2017 at 4:06 PM, Junping Du  wrote:

> Allen,
>  Do we have any solid evidence to show the HDFS unit tests going
> through the roof are due to serious memory leak by HDFS? Normally, I don't
> expect memory leak are identified in our UTs - mostly, it (test jvm gone)
> is just because of test or deployment issues.
>  Unless there is concrete evidence, my concern on seriously memory
> leak for HDFS on 2.8 is relatively low given some companies (Yahoo,
> Alibaba, etc.) have deployed 2.8 on large production environment for
> months. Non-serious memory leak (like forgetting to close stream in
> non-critical path, etc.) and other non-critical bugs always happens here
> and there that we have to live with.
>
> Thanks,
>
> Junping
>
> 
> From: Allen Wittenauer 
> Sent: Tuesday, October 24, 2017 8:27 AM
> To: Hadoop Common
> Cc: Hdfs-dev; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
>
> > On Oct 23, 2017, at 12:50 PM, Allen Wittenauer 
> wrote:
> >
> >
> >
> > With no other information or access to go on, my current hunch is that
> one of the HDFS unit tests is ballooning in memory size.  The easiest way
> to kill a Linux machine is to eat all of the RAM, thanks to overcommit and
> that’s what this “feels” like.
> >
> > Someone should verify if 2.8.2 has the same issues before a release goes
> out …
>
>
> FWIW, I ran 2.8.2 last night and it has the same problems.
>
> Also: the node didn’t die!  Looking through the workspace (so the
> next run will destroy them), two sets of logs stand out:
>
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-
> linux-x86/ws/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
> and
>
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-
> linux-x86/ws/sourcedir/hadoop-hdfs-project/hadoop-hdfs/
>
> It looks like my hunch is correct:  RAM in the HDFS unit tests are
> going through the roof.  It’s also interesting how MANY log files there
> are.  Is surefire not picking up that jobs are dying?  Maybe not if memory
> is getting tight.
>
> Anyway, at the point, branch-2.8 and higher are probably fubar’d.
> Additionally, I’ve filed YETUS-561 so that Yetus-controlled Docker
> containers can have their RAM limits set in order to prevent more nodes
> going catatonic.
>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>


-- 
busbey

Re: HDFS Pre-commit taking more than 24 hours.

[Sean Busbey]

The ASF Jenkins farm has been under duress lately, at least in part due to
HDFS-12711.

I'd say wait for work on containing that issue to finish and then take a
look at if it's backlog or the test itself so you can determine if we need
to talk on builds@a.o or here.


On Fri, Oct 27, 2017 at 10:26 AM, Rushabh Shah 
wrote:

> Hi All,
> HDFS Pre-commit is taking more than 24 hours to run unit tests.
> Currently at (Fri Oct 27 15:17:18 UTC 2017), it is running patch submitted
> at Thursday Oct 26 07:43 UTC 2017 (HDFS-12582
> )
> Last 5-6 pre-commits ran on only 1 slave: asf903.gq1.ygridcore.net
> Is anyone else facing this issue ?
>
>
>
> Thanks,
> Rushabh Shah
>



-- 
busbey

Re: [DISCUSS]: securing ASF Hadoop releases out of the box

[Sean Busbey]

I really, really like the approach of defaulting to only non-routeable
IPs allowed. it seems like a good tradeoff for complexity of
implementation, pain to reconfigure, and level of protection.

On Thu, Jul 5, 2018 at 2:25 PM, Todd Lipcon  wrote:
> The approach we took in Apache Kudu is that, if Kerberos hasn't been
> enabled, we default to a whitelist of subnets. The default whitelist is
> 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16 which
> matches the IANA "non-routeable IP" subnet list.
>
> In other words, out-of-the-box, you get a deployment that works fine within
> a typical LAN environment, but won't allow some remote hacker to locate
> your cluster and access your data. We thought this was a nice balance
> between "works out of the box without lots of configuration" and "decent
> security". In my opinion a "localhost-only by default" would be be overly
> restrictive since I'd usually be deploying on some datacenter or EC2
> machine and then trying to access it from a client on my laptop.
>
> We released this first a bit over a year ago if my memory serves me, and
> we've had relatively few complaints or questions about it. We also made
> sure that the error message that comes back to clients is pretty
> reasonable, indicating the specific configuration that is disallowing
> access, so if people hit the issue on upgrade they had a clear idea what is
> going on.
>
> Of course it's not foolproof, since as Eric says, you're still likely open
> to the entirety of your corporation, and you may not want that, but as he
> also pointed out, that might be true even if you enable Kerberos
> authentication.
>
> -Todd
>
> On Thu, Jul 5, 2018 at 11:38 AM, Eric Yang  wrote:
>
>> Hadoop default configuration aimed for user friendliness to increase
>> adoption, and security can be enabled one by one.  This approach is most
>> problematic to security because system can be compromised before all
>> security features are turned on.
>> Larry's proposal will add some safety to remind system admin if security
>> is disabled.  However, reducing the number of knobs on security configs are
>> likely required to make the system secure for the banner idea to work
>> without writing too much guessing logic to determine if UI is secured.
>> Penetration test can provide better insights of what hasn't been secured to
>> improve the next release.  Thankfully most Hadoop vendors have done this
>> work periodically to help the community secure Hadoop.
>>
>> There are plenty of company advertised if you want security, use
>> Kerberos.  This statement is not entirely true.  Kerberos makes security
>> more difficult to crack for external parties, but it shouldn't be the only
>> method to secure Hadoop.  When the Kerberos environment is larger than
>> Hadoop cluster, anyone within Kerberos environment can access Hadoop
>> cluster freely without restriction.  In large scale enterprises or some
>> cloud vendors that sublet their resources, this might not be acceptable.
>>
>> From my point of view, a secure Hadoop release must default all settings
>> to localhost only and allow users to add more hosts through authorized
>> white list of servers.  This will keep security perimeter in check.  All
>> wild card ACLs will need to be removed or default to current user/current
>> host only.  Proxy user/host ACL list must be enforced on http channels.
>> This is basically realigning the default configuration to single node
>> cluster or firewalled configuration.
>>
>> Regards,
>> Eric
>>
>> On 7/5/18, 8:24 AM, "larry mccay"  wrote:
>>
>> Hi Steve -
>>
>> This is a long overdue DISCUSS thread!
>>
>> Perhaps the UIs can very visibly state (in red) "WARNING: UNSECURED UI
>> ACCESS - OPEN TO COMPROMISE" - maybe even force a click through the
>> warning
>> to get to the page like SSL exceptions in the browser do?
>> Similar tactic for UI access without SSL?
>> A new AuthenticationFilter can be added to the filter chains that
>> blocks
>> API calls unless explicitly configured to be open and obvious log a
>> similar
>> message?
>>
>> thanks,
>>
>> --larry
>>
>>
>>
>>
>> On Wed, Jul 4, 2018 at 11:58 AM, Steve Loughran <
>> ste...@hortonworks.com>
>> wrote:
>>
>> > Bitcoins are profitable enough to justify writing malware to run on
>> Hadoop
>> > clusters & schedule mining jobs: there have been a couple of
>> incidents of
>> > this in the wild, generally going in through no security, well known
>> > passwords, open ports.
>> >
>> > Vendors of Hadoop-related products get to deal with their lockdown
>> > themselves, which they often do by installing kerberos from the
>> outset,
>> > making users make up their own password for admin accounts, etc.
>> >
>> > The ASF releases though: we just provide something insecure out the
>> box
>> > and some docs saying "use kerberos if you want security"
>> >
>> > What we can do here?
>> >
>>

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Sean Busbey]

-1 (non-binding)

force pushes are extremely disruptive. there's no way to know who's
updated their local git repo to include these changes since the commit
went in. if a merge commit is so disruptive that we need to subject folks
to the inconvenience of a force push then we should have more tooling
in place to avoid them (like client side git hooks for all committers).

On Thu, Jul 5, 2018 at 4:37 PM, Subru Krishnan  wrote:
> Folks,
>
> There was a merge commit accidentally pushed to trunk, you can find the
> details in the mail thread [1].
>
> I have raised an INFRA ticket [2] to reset/force push to clean up trunk.
>
> Can we have a quick vote for INFRA sign-off to proceed as this is blocking
> all commits?
>
> Thanks,
> Subru
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E
> [2] https://issues.apache.org/jira/browse/INFRA-16727



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: 2.7.3 release plan

[Sean Busbey]

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
 wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



-- 
busbey

Re: 2.7.3 release plan

[Sean Busbey]

As of 2 days ago, there were already 135 jiras associated with 2.7.3,
if *any* of them end up introducing a regression the inclusion of
HDFS-8791 means that folks will have cluster downtime in order to back
things out. If that happens to any substantial number of downstream
folks, or any particularly vocal downstream folks, then it is very
likely we'll lose the remaining trust of operators for rolling out
maintenance releases. That's a pretty steep cost.

Please do not include HDFS-8791 in any 2.6.z release. Folks having to
be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
unreasonable burden.

I agree that this fix is important, I just think we should either cut
a version of 2.8 that includes it or find a way to do it that gives an
operational path for rolling downgrade.

On Thu, Mar 31, 2016 at 10:10 AM, Junping Du  wrote:
> Thanks for bringing up this topic, Sean.
> When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 
> haven't been committed in so that's why we didn't discuss this earlier.
> I remember in JIRA discussion, we treated this layout change as a Blocker bug 
> that fixing a significant performance regression before but not a normal 
> performance improvement. And I believe HDFS community already did their best 
> with careful and patient to deliver the fix and other related patches (like 
> upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ 
> rounds patch review back and forth by senior committers, not to mention the 
> outstanding performance test data in HDFS-8791.
> I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. 
> However, that needs Vinod's final confirmation who serves as RM for 
> branch-2.7. In addition, I didn't see any blocker issue to bring it into 
> 2.6.5 now.
> Just my 2 cents.
>
> Thanks,
>
> Junping
>
> 
> From: Sean Busbey 
> Sent: Thursday, March 31, 2016 2:57 PM
> To: hdfs-dev@hadoop.apache.org
> Cc: Hadoop Common; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> Subject: Re: 2.7.3 release plan
>
> A layout change in a maintenance release sounds very risky. I saw some
> discussion on the JIRA about those risks, but the consensus seemed to
> be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> we did RMs per release rather than per branch? No one claiming to be a
> release manager ever spoke up AFAICT.
>
> Should this change be included? Should it go into a special 2.8
> release as mentioned in the ticket?
>
> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>  wrote:
>> Thank you Vinod!
>>
>> FYI: 2.7.3 will be a bit special release.
>>
>> HDFS-8791 bumped up the datanode layout version,
>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> is impossible. We can rollback instead.
>>
>> https://issues.apache.org/jira/browse/HDFS-8791
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>
>> Regards,
>> Akira
>>
>>
>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>
>>> Hi all,
>>>
>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>>> did go out mid February). Got a little busy since.
>>>
>>> Following up the 2.7.2 maintenance release, we should work towards a
>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>>> features / improvements.
>>>
>>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>>> / critical issues. Will start moving out any tickets that are not blockers
>>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>>> outstanding as of now.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> [1] 2.7.3 release blockers:
>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>
>>
>
>
>
> --
> busbey



-- 
busbey

Re: [VOTE] Merge feature branch HADOOP-12930

[Sean Busbey]

+1 (non-binding)

reviewed everything, filed an additional subtask for a very trivial
typo in the docs. should be fine to make a full issue after close and
then fix.

tried merging locally, tried running through new shell tests (both
with and without bats installed), tried making an example custom
command (valid and malformed). everything looks great.

On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer  wrote:
>
> Hey gang!
>
> I’d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) 
> to merge the HADOOP-12930 feature branch into trunk. This branch was 
> developed exclusively by me as per the discussion two months ago as a way to 
> make what would be a rather large patch hopefully easier to review.  The vast 
> majority of the branch is code movement in the same file, additional license 
> headers, maven assembly hooks for distribution, and variable renames. Not a 
> whole lot of new code, but a big diff file none-the-less.
>
> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’ 
> commands to allow for subcommands to be added or modified at runtime.  This 
> allows for individual users or entire sites to tweak the execution 
> environment to suit their local needs.  For example, it has been a practice 
> for some locations to change the distcp jar out for a custom one.  Using this 
> functionality, it is possible that the ‘hadoop distcp’ command could run the 
> local version without overwriting the bundled jar and for existing 
> documentation (read: results from Internet searches) to work as written 
> without modification. This has the potential to be a huge win, especially for:
>
> * advanced end users looking to supplement the Apache Hadoop 
> experience
> * operations teams that may be able to leverage existing 
> documentation without having to remain local “exception” docs
> * development groups wanting an easy way to trial 
> experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the ‘hadoop’ command
> * Adds the infrastructure for hdfs script testing and the 
> first unit test for the ‘hdfs’ command
> * Modifies the hadoop-tools components to be dynamic rather 
> than hard coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be 
> consistent with other bundled profiles, including the ones introduced in this 
> branch
>
> Documentation, including a ‘hello world’-style example, is in the 
> UnixShellGuide markdown file.  (Of course!)
>
>  I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
> P.S.,
>
> There are still two open sub-tasks.  These are blocked by other 
> issues so that we may add unit testing to the shell code in those respective 
> areas.  I’ll covert to full issues after HADOOP-12930 is closed.
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: upstream jenkins build broken?

[Sean Busbey]

+dev@hbase

HBase has recently been cleaning up our precommit jenkins jobs to make them
more robust. From what I can tell our stuff started off as an earlier
version of what Hadoop uses for testing.

Folks on either side open to an experiment of combining our precommit check
tooling? In principle we should be looking for the same kinds of things.

Naturally we'll still need different jenkins jobs to handle different
resource needs and we'd need to figure out where stuff eventually lives,
but that could come later.

On Wed, Mar 11, 2015 at 4:34 PM, Chris Nauroth 
wrote:

> The only thing I'm aware of is the failOnError option:
>
> http://maven.apache.org/plugins/maven-clean-plugin/examples/ignoring-errors
> .html
>
>
> I prefer that we don't disable this, because ignoring different kinds of
> failures could leave our build directories in an indeterminate state.  For
> example, we could end up with an old class file on the classpath for test
> runs that was supposedly deleted.
>
> I think it's worth exploring Eddy's suggestion to try simulating failure
> by placing a file where the code expects to see a directory.  That might
> even let us enable some of these tests that are skipped on Windows,
> because Windows allows access for the owner even after permissions have
> been stripped.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
>
>
>
> On 3/11/15, 2:10 PM, "Colin McCabe"  wrote:
>
> >Is there a maven plugin or setting we can use to simply remove
> >directories that have no executable permissions on them?  Clearly we
> >have the permission to do this from a technical point of view (since
> >we created the directories as the jenkins user), it's simply that the
> >code refuses to do it.
> >
> >Otherwise I guess we can just fix those tests...
> >
> >Colin
> >
> >On Tue, Mar 10, 2015 at 2:43 PM, Lei Xu  wrote:
> >> Thanks a lot for looking into HDFS-7722, Chris.
> >>
> >> In HDFS-7722:
> >> TestDataNodeVolumeFailureXXX tests reset data dir permissions in
> >>TearDown().
> >> TestDataNodeHotSwapVolumes reset permissions in a finally clause.
> >>
> >> Also I ran mvn test several times on my machine and all tests passed.
> >>
> >> However, since in DiskChecker#checkDirAccess():
> >>
> >> private static void checkDirAccess(File dir) throws DiskErrorException {
> >>   if (!dir.isDirectory()) {
> >> throw new DiskErrorException("Not a directory: "
> >>  + dir.toString());
> >>   }
> >>
> >>   checkAccessByFileMethods(dir);
> >> }
> >>
> >> One potentially safer alternative is replacing data dir with a regular
> >> file to stimulate disk failures.
> >>
> >> On Tue, Mar 10, 2015 at 2:19 PM, Chris Nauroth
> >> wrote:
> >>> TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure,
> >>> TestDataNodeVolumeFailureReporting, and
> >>> TestDataNodeVolumeFailureToleration all remove executable permissions
> >>>from
> >>> directories like the one Colin mentioned to simulate disk failures at
> >>>data
> >>> nodes.  I reviewed the code for all of those, and they all appear to be
> >>> doing the necessary work to restore executable permissions at the end
> >>>of
> >>> the test.  The only recent uncommitted patch I¹ve seen that makes
> >>>changes
> >>> in these test suites is HDFS-7722.  That patch still looks fine
> >>>though.  I
> >>> don¹t know if there are other uncommitted patches that changed these
> >>>test
> >>> suites.
> >>>
> >>> I suppose it¹s also possible that the JUnit process unexpectedly died
> >>> after removing executable permissions but before restoring them.  That
> >>> always would have been a weakness of these test suites, regardless of
> >>>any
> >>> recent changes.
> >>>
> >>> Chris Nauroth
> >>> Hortonworks
> >>> http://hortonworks.com/
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 3/10/15, 1:47 PM, "Aaron T. Myers"  wrote:
> >>>
> Hey Colin,
> 
> I asked Andrew Bayer, who works with Apache Infra, what's going on with
> these boxes. He took a look and concluded that some perms are being
> set in
> those directories by our unit tests which are precluding those files
> from
> getting deleted. He's going to clean up the boxes for us, but we should
> expect this to keep happening until we can fix the test in question to
> properly clean up after itself.
> 
> To help narrow down which commit it was that started this, Andrew sent
> me
> this info:
> 
> "/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/
> has
> 500 perms, so I'm guessing that's the problem. Been that way since 9:32
> UTC
> on March 5th."
> 
> --
> Aaron T. Myers
> Software Engineer, Cloudera
> 
> On Tue, Mar 10, 2015 at 1:24 PM, Colin P. McCabe 
> wrote:
> 
> > Hi all,
> >
> > A very quick (and not thorough) survey shows that I can't find any
> > jenkins jobs that succeeded from the last 24 ho

Re: upstream jenkins build broken?

[Sean Busbey]

You could rely on a destructive git clean call instead of maven to do the
directory removal.

-- 
Sean
On Mar 11, 2015 4:11 PM, "Colin McCabe"  wrote:

> Is there a maven plugin or setting we can use to simply remove
> directories that have no executable permissions on them?  Clearly we
> have the permission to do this from a technical point of view (since
> we created the directories as the jenkins user), it's simply that the
> code refuses to do it.
>
> Otherwise I guess we can just fix those tests...
>
> Colin
>
> On Tue, Mar 10, 2015 at 2:43 PM, Lei Xu  wrote:
> > Thanks a lot for looking into HDFS-7722, Chris.
> >
> > In HDFS-7722:
> > TestDataNodeVolumeFailureXXX tests reset data dir permissions in
> TearDown().
> > TestDataNodeHotSwapVolumes reset permissions in a finally clause.
> >
> > Also I ran mvn test several times on my machine and all tests passed.
> >
> > However, since in DiskChecker#checkDirAccess():
> >
> > private static void checkDirAccess(File dir) throws DiskErrorException {
> >   if (!dir.isDirectory()) {
> > throw new DiskErrorException("Not a directory: "
> >  + dir.toString());
> >   }
> >
> >   checkAccessByFileMethods(dir);
> > }
> >
> > One potentially safer alternative is replacing data dir with a regular
> > file to stimulate disk failures.
> >
> > On Tue, Mar 10, 2015 at 2:19 PM, Chris Nauroth 
> wrote:
> >> TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure,
> >> TestDataNodeVolumeFailureReporting, and
> >> TestDataNodeVolumeFailureToleration all remove executable permissions
> from
> >> directories like the one Colin mentioned to simulate disk failures at
> data
> >> nodes.  I reviewed the code for all of those, and they all appear to be
> >> doing the necessary work to restore executable permissions at the end of
> >> the test.  The only recent uncommitted patch I¹ve seen that makes
> changes
> >> in these test suites is HDFS-7722.  That patch still looks fine
> though.  I
> >> don¹t know if there are other uncommitted patches that changed these
> test
> >> suites.
> >>
> >> I suppose it¹s also possible that the JUnit process unexpectedly died
> >> after removing executable permissions but before restoring them.  That
> >> always would have been a weakness of these test suites, regardless of
> any
> >> recent changes.
> >>
> >> Chris Nauroth
> >> Hortonworks
> >> http://hortonworks.com/
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 3/10/15, 1:47 PM, "Aaron T. Myers"  wrote:
> >>
> >>>Hey Colin,
> >>>
> >>>I asked Andrew Bayer, who works with Apache Infra, what's going on with
> >>>these boxes. He took a look and concluded that some perms are being set
> in
> >>>those directories by our unit tests which are precluding those files
> from
> >>>getting deleted. He's going to clean up the boxes for us, but we should
> >>>expect this to keep happening until we can fix the test in question to
> >>>properly clean up after itself.
> >>>
> >>>To help narrow down which commit it was that started this, Andrew sent
> me
> >>>this info:
> >>>
> >>>"/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> >>>Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/
> has
> >>>500 perms, so I'm guessing that's the problem. Been that way since 9:32
> >>>UTC
> >>>on March 5th."
> >>>
> >>>--
> >>>Aaron T. Myers
> >>>Software Engineer, Cloudera
> >>>
> >>>On Tue, Mar 10, 2015 at 1:24 PM, Colin P. McCabe 
> >>>wrote:
> >>>
>  Hi all,
> 
>  A very quick (and not thorough) survey shows that I can't find any
>  jenkins jobs that succeeded from the last 24 hours.  Most of them seem
>  to be failing with some variant of this message:
> 
>  [ERROR] Failed to execute goal
>  org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean)
>  on project hadoop-hdfs: Failed to clean project: Failed to delete
> 
> 
>
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-hdfs-pr
> oject/hadoop-hdfs/target/test/data/dfs/data/data3
>  -> [Help 1]
> 
>  Any ideas how this happened?  Bad disk, unit test setting wrong
>  permissions?
> 
>  Colin
> 
> >>
> >
> >
> >
> > --
> > Lei (Eddy) Xu
> > Software Engineer, Cloudera
>

Re: upstream jenkins build broken?

[Sean Busbey]

I'm on it. HADOOP-11721

On Mon, Mar 16, 2015 at 3:44 PM, Haohui Mai  wrote:

> +1 for git clean.
>
> Colin, can you please get it in ASAP? Currently due to the jenkins
> issues, we cannot close the 2.7 blockers.
>
> Thanks,
> Haohui
>
>
>
> On Mon, Mar 16, 2015 at 11:54 AM, Colin P. McCabe 
> wrote:
> > If all it takes is someone creating a test that makes a directory
> > without -x, this is going to happen over and over.
> >
> > Let's just fix the problem at the root by running "git clean -fqdx" in
> > our jenkins scripts.  If there's no objections I will add this in and
> > un-break the builds.
> >
> > best,
> > Colin
> >
> > On Fri, Mar 13, 2015 at 1:48 PM, Lei Xu  wrote:
> >> I filed HDFS-7917 to change the way to simulate disk failures.
> >>
> >> But I think we still need infrastructure folks to help with jenkins
> >> scripts to clean the dirs left today.
> >>
> >> On Fri, Mar 13, 2015 at 1:38 PM, Mai Haohui  wrote:
> >>> Any updates on this issues? It seems that all HDFS jenkins builds are
> >>> still failing.
> >>>
> >>> Regards,
> >>> Haohui
> >>>
> >>> On Thu, Mar 12, 2015 at 12:53 AM, Vinayakumar B <
> vinayakum...@apache.org> wrote:
> >>>> I think the problem started from here.
> >>>>
> >>>>
> https://builds.apache.org/job/PreCommit-HDFS-Build/9828/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailure/testUnderReplicationAfterVolFailure/
> >>>>
> >>>> As Chris mentioned TestDataNodeVolumeFailure is changing the
> permission.
> >>>> But in this patch, ReplicationMonitor got NPE and it got terminate
> signal,
> >>>> due to which MiniDFSCluster.shutdown() throwing Exception.
> >>>>
> >>>> But, TestDataNodeVolumeFailure#teardown() is restoring those
> permission
> >>>> after shutting down cluster. So in this case IMO, permissions were
> never
> >>>> restored.
> >>>>
> >>>>
> >>>>   @After
> >>>>   public void tearDown() throws Exception {
> >>>> if(data_fail != null) {
> >>>>   FileUtil.setWritable(data_fail, true);
> >>>> }
> >>>> if(failedDir != null) {
> >>>>   FileUtil.setWritable(failedDir, true);
> >>>> }
> >>>> if(cluster != null) {
> >>>>   cluster.shutdown();
> >>>> }
> >>>> for (int i = 0; i < 3; i++) {
> >>>>   FileUtil.setExecutable(new File(dataDir, "data"+(2*i+1)), true);
> >>>>   FileUtil.setExecutable(new File(dataDir, "data"+(2*i+2)), true);
> >>>> }
> >>>>   }
> >>>>
> >>>>
> >>>> Regards,
> >>>> Vinay
> >>>>
> >>>> On Thu, Mar 12, 2015 at 12:35 PM, Vinayakumar B <
> vinayakum...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> When I see the history of these kind of builds, All these are failed
> on
> >>>>> node H9.
> >>>>>
> >>>>> I think some or the other uncommitted patch would have created the
> problem
> >>>>> and left it there.
> >>>>>
> >>>>>
> >>>>> Regards,
> >>>>> Vinay
> >>>>>
> >>>>> On Thu, Mar 12, 2015 at 6:16 AM, Sean Busbey 
> wrote:
> >>>>>
> >>>>>> You could rely on a destructive git clean call instead of maven to
> do the
> >>>>>> directory removal.
> >>>>>>
> >>>>>> --
> >>>>>> Sean
> >>>>>> On Mar 11, 2015 4:11 PM, "Colin McCabe" 
> wrote:
> >>>>>>
> >>>>>> > Is there a maven plugin or setting we can use to simply remove
> >>>>>> > directories that have no executable permissions on them?  Clearly
> we
> >>>>>> > have the permission to do this from a technical point of view
> (since
> >>>>>> > we created the directories as the jenkins user), it's simply that
> the
> >>>>>> > code refuses to do it.
> >>>>>> >
> >>>>>> > Otherwise I guess we can just fix those tests...
&

Re: upstream jenkins build broken?

[Sean Busbey]

Can someone point me to an example build that is broken?

On Mon, Mar 16, 2015 at 3:52 PM, Sean Busbey  wrote:

> I'm on it. HADOOP-11721
>
> On Mon, Mar 16, 2015 at 3:44 PM, Haohui Mai  wrote:
>
>> +1 for git clean.
>>
>> Colin, can you please get it in ASAP? Currently due to the jenkins
>> issues, we cannot close the 2.7 blockers.
>>
>> Thanks,
>> Haohui
>>
>>
>>
>> On Mon, Mar 16, 2015 at 11:54 AM, Colin P. McCabe 
>> wrote:
>> > If all it takes is someone creating a test that makes a directory
>> > without -x, this is going to happen over and over.
>> >
>> > Let's just fix the problem at the root by running "git clean -fqdx" in
>> > our jenkins scripts.  If there's no objections I will add this in and
>> > un-break the builds.
>> >
>> > best,
>> > Colin
>> >
>> > On Fri, Mar 13, 2015 at 1:48 PM, Lei Xu  wrote:
>> >> I filed HDFS-7917 to change the way to simulate disk failures.
>> >>
>> >> But I think we still need infrastructure folks to help with jenkins
>> >> scripts to clean the dirs left today.
>> >>
>> >> On Fri, Mar 13, 2015 at 1:38 PM, Mai Haohui 
>> wrote:
>> >>> Any updates on this issues? It seems that all HDFS jenkins builds are
>> >>> still failing.
>> >>>
>> >>> Regards,
>> >>> Haohui
>> >>>
>> >>> On Thu, Mar 12, 2015 at 12:53 AM, Vinayakumar B <
>> vinayakum...@apache.org> wrote:
>> >>>> I think the problem started from here.
>> >>>>
>> >>>>
>> https://builds.apache.org/job/PreCommit-HDFS-Build/9828/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailure/testUnderReplicationAfterVolFailure/
>> >>>>
>> >>>> As Chris mentioned TestDataNodeVolumeFailure is changing the
>> permission.
>> >>>> But in this patch, ReplicationMonitor got NPE and it got terminate
>> signal,
>> >>>> due to which MiniDFSCluster.shutdown() throwing Exception.
>> >>>>
>> >>>> But, TestDataNodeVolumeFailure#teardown() is restoring those
>> permission
>> >>>> after shutting down cluster. So in this case IMO, permissions were
>> never
>> >>>> restored.
>> >>>>
>> >>>>
>> >>>>   @After
>> >>>>   public void tearDown() throws Exception {
>> >>>> if(data_fail != null) {
>> >>>>   FileUtil.setWritable(data_fail, true);
>> >>>> }
>> >>>> if(failedDir != null) {
>> >>>>   FileUtil.setWritable(failedDir, true);
>> >>>>     }
>> >>>> if(cluster != null) {
>> >>>>   cluster.shutdown();
>> >>>> }
>> >>>> for (int i = 0; i < 3; i++) {
>> >>>>   FileUtil.setExecutable(new File(dataDir, "data"+(2*i+1)),
>> true);
>> >>>>   FileUtil.setExecutable(new File(dataDir, "data"+(2*i+2)),
>> true);
>> >>>> }
>> >>>>   }
>> >>>>
>> >>>>
>> >>>> Regards,
>> >>>> Vinay
>> >>>>
>> >>>> On Thu, Mar 12, 2015 at 12:35 PM, Vinayakumar B <
>> vinayakum...@apache.org>
>> >>>> wrote:
>> >>>>
>> >>>>> When I see the history of these kind of builds, All these are
>> failed on
>> >>>>> node H9.
>> >>>>>
>> >>>>> I think some or the other uncommitted patch would have created the
>> problem
>> >>>>> and left it there.
>> >>>>>
>> >>>>>
>> >>>>> Regards,
>> >>>>> Vinay
>> >>>>>
>> >>>>> On Thu, Mar 12, 2015 at 6:16 AM, Sean Busbey 
>> wrote:
>> >>>>>
>> >>>>>> You could rely on a destructive git clean call instead of maven to
>> do the
>> >>>>>> directory removal.
>> >>>>>>
>> >>>>>> --
>> >>>>>> Sean
>> >>>>>> On Mar 11, 2015 4:11 PM, "Colin McCabe" 
>> wrote:
>> >>>>>>
>> >>>>&

Re: upstream jenkins build broken?

[Sean Busbey]

Is the simulation just removing the executable bit on the directory? I'd
like to get something I can reproduce locally.

On Tue, Mar 17, 2015 at 2:29 AM, Vinayakumar B 
wrote:

> I have simulated the problem in my env and verified that, both 'git clean
> -xdf' and 'mvn clean' will not remove the directory.
> mvn fails where as git simply ignores (not even display any warning) the
> problem.
>
>
>
> Regards,
> Vinay
>
> On Tue, Mar 17, 2015 at 2:32 AM, Sean Busbey  wrote:
>
> > Can someone point me to an example build that is broken?
> >
> > On Mon, Mar 16, 2015 at 3:52 PM, Sean Busbey 
> wrote:
> >
> > > I'm on it. HADOOP-11721
> > >
> > > On Mon, Mar 16, 2015 at 3:44 PM, Haohui Mai  wrote:
> > >
> > >> +1 for git clean.
> > >>
> > >> Colin, can you please get it in ASAP? Currently due to the jenkins
> > >> issues, we cannot close the 2.7 blockers.
> > >>
> > >> Thanks,
> > >> Haohui
> > >>
> > >>
> > >>
> > >> On Mon, Mar 16, 2015 at 11:54 AM, Colin P. McCabe  >
> > >> wrote:
> > >> > If all it takes is someone creating a test that makes a directory
> > >> > without -x, this is going to happen over and over.
> > >> >
> > >> > Let's just fix the problem at the root by running "git clean -fqdx"
> in
> > >> > our jenkins scripts.  If there's no objections I will add this in
> and
> > >> > un-break the builds.
> > >> >
> > >> > best,
> > >> > Colin
> > >> >
> > >> > On Fri, Mar 13, 2015 at 1:48 PM, Lei Xu  wrote:
> > >> >> I filed HDFS-7917 to change the way to simulate disk failures.
> > >> >>
> > >> >> But I think we still need infrastructure folks to help with jenkins
> > >> >> scripts to clean the dirs left today.
> > >> >>
> > >> >> On Fri, Mar 13, 2015 at 1:38 PM, Mai Haohui 
> > >> wrote:
> > >> >>> Any updates on this issues? It seems that all HDFS jenkins builds
> > are
> > >> >>> still failing.
> > >> >>>
> > >> >>> Regards,
> > >> >>> Haohui
> > >> >>>
> > >> >>> On Thu, Mar 12, 2015 at 12:53 AM, Vinayakumar B <
> > >> vinayakum...@apache.org> wrote:
> > >> >>>> I think the problem started from here.
> > >> >>>>
> > >> >>>>
> > >>
> >
> https://builds.apache.org/job/PreCommit-HDFS-Build/9828/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailure/testUnderReplicationAfterVolFailure/
> > >> >>>>
> > >> >>>> As Chris mentioned TestDataNodeVolumeFailure is changing the
> > >> permission.
> > >> >>>> But in this patch, ReplicationMonitor got NPE and it got
> terminate
> > >> signal,
> > >> >>>> due to which MiniDFSCluster.shutdown() throwing Exception.
> > >> >>>>
> > >> >>>> But, TestDataNodeVolumeFailure#teardown() is restoring those
> > >> permission
> > >> >>>> after shutting down cluster. So in this case IMO, permissions
> were
> > >> never
> > >> >>>> restored.
> > >> >>>>
> > >> >>>>
> > >> >>>>   @After
> > >> >>>>   public void tearDown() throws Exception {
> > >> >>>> if(data_fail != null) {
> > >> >>>>   FileUtil.setWritable(data_fail, true);
> > >> >>>> }
> > >> >>>> if(failedDir != null) {
> > >> >>>>   FileUtil.setWritable(failedDir, true);
> > >> >>>> }
> > >> >>>> if(cluster != null) {
> > >> >>>>   cluster.shutdown();
> > >> >>>> }
> > >> >>>> for (int i = 0; i < 3; i++) {
> > >> >>>>   FileUtil.setExecutable(new File(dataDir, "data"+(2*i+1)),
> > >> true);
> > >> >>>>   FileUtil.setExecutable(new File(dataDir, "data"+(2*i+2)),
> > >> true);
> > >> >>>> }
> &g

Re: upstream jenkins build broken?

[Sean Busbey]

Yeah I can do that now.

On Tue, Mar 17, 2015 at 2:53 AM, Vinayakumar B 
wrote:

> Seems like all builds of Precommit-HDFS-Build failing with below problem.
>
> FATAL: Command "git clean -fdx" returned status code 1:
> stdout:
> stderr: hudson.plugins.git.GitException
> <
> http://stacktrace.jenkins-ci.org/search?query=hudson.plugins.git.GitException
> >:
> Command "git clean -fdx" returned status code 1:
> stdout:
> stderr:
>
>
>
> Can someone remove "git clean -fdx" from build configurations of
> Precommit-HDFS-Build ?
>
>
> Regards,
> Vinay
>
> On Tue, Mar 17, 2015 at 12:59 PM, Vinayakumar B 
> wrote:
>
> > I have simulated the problem in my env and verified that, both 'git clean
> > -xdf' and 'mvn clean' will not remove the directory.
> > mvn fails where as git simply ignores (not even display any warning) the
> > problem.
> >
> >
> >
> > Regards,
> > Vinay
> >
> > On Tue, Mar 17, 2015 at 2:32 AM, Sean Busbey 
> wrote:
> >
> >> Can someone point me to an example build that is broken?
> >>
> >> On Mon, Mar 16, 2015 at 3:52 PM, Sean Busbey 
> wrote:
> >>
> >> > I'm on it. HADOOP-11721
> >> >
> >> > On Mon, Mar 16, 2015 at 3:44 PM, Haohui Mai 
> wrote:
> >> >
> >> >> +1 for git clean.
> >> >>
> >> >> Colin, can you please get it in ASAP? Currently due to the jenkins
> >> >> issues, we cannot close the 2.7 blockers.
> >> >>
> >> >> Thanks,
> >> >> Haohui
> >> >>
> >> >>
> >> >>
> >> >> On Mon, Mar 16, 2015 at 11:54 AM, Colin P. McCabe <
> cmcc...@apache.org>
> >> >> wrote:
> >> >> > If all it takes is someone creating a test that makes a directory
> >> >> > without -x, this is going to happen over and over.
> >> >> >
> >> >> > Let's just fix the problem at the root by running "git clean -fqdx"
> >> in
> >> >> > our jenkins scripts.  If there's no objections I will add this in
> and
> >> >> > un-break the builds.
> >> >> >
> >> >> > best,
> >> >> > Colin
> >> >> >
> >> >> > On Fri, Mar 13, 2015 at 1:48 PM, Lei Xu  wrote:
> >> >> >> I filed HDFS-7917 to change the way to simulate disk failures.
> >> >> >>
> >> >> >> But I think we still need infrastructure folks to help with
> jenkins
> >> >> >> scripts to clean the dirs left today.
> >> >> >>
> >> >> >> On Fri, Mar 13, 2015 at 1:38 PM, Mai Haohui 
> >> >> wrote:
> >> >> >>> Any updates on this issues? It seems that all HDFS jenkins builds
> >> are
> >> >> >>> still failing.
> >> >> >>>
> >> >> >>> Regards,
> >> >> >>> Haohui
> >> >> >>>
> >> >> >>> On Thu, Mar 12, 2015 at 12:53 AM, Vinayakumar B <
> >> >> vinayakum...@apache.org> wrote:
> >> >> >>>> I think the problem started from here.
> >> >> >>>>
> >> >> >>>>
> >> >>
> >>
> https://builds.apache.org/job/PreCommit-HDFS-Build/9828/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailure/testUnderReplicationAfterVolFailure/
> >> >> >>>>
> >> >> >>>> As Chris mentioned TestDataNodeVolumeFailure is changing the
> >> >> permission.
> >> >> >>>> But in this patch, ReplicationMonitor got NPE and it got
> terminate
> >> >> signal,
> >> >> >>>> due to which MiniDFSCluster.shutdown() throwing Exception.
> >> >> >>>>
> >> >> >>>> But, TestDataNodeVolumeFailure#teardown() is restoring those
> >> >> permission
> >> >> >>>> after shutting down cluster. So in this case IMO, permissions
> were
> >> >> never
> >> >> >>>> restored.
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>   @After
> >> >> >>>>   public void tearDown() throws Exception {
> >>

Re: Set minimum version of Hadoop 3 to JDK 8

[Sean Busbey]

A few options:

* Only change the builds for master to use jdk8
* build with both jdk7 and jdk8 by copying jobs
* build with both jdk7 and jdk8 using a jenkins matrix build

Robert, if you'd like help with any of these please send me a ping off-list.

On Tue, Apr 21, 2015 at 8:19 PM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com> wrote:

> We don't want JDK 8 only code going into branch-2 line. Moving Jenkins to
> 1.8 right-away will shield such code, how do we address that?
>
> Thanks,
> +Vinod
>
> On Apr 21, 2015, at 5:54 PM, Robert Kanter  wrote:
>
> > Sure, I'll try to change the Jenkins builds to 1.8 first.
> >
> > On Tue, Apr 21, 2015 at 3:31 PM, Andrew Wang 
> > wrote:
> >
> >> Hey Robert,
> >>
> >> As a first step, could we try switching all our precommit and nightly
> >> builds over to use 1.8? This is a prerequisite for HADOOP-11858, and
> safe
> >> to do in any case since it'll still target 1.7.
> >>
> >> I'll note that HADOOP-10530 details the pain Steve went through
> switching
> >> us to JDK7. Might be some lessons learned about how to do this
> transition
> >> more smoothly.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> On Tue, Apr 21, 2015 at 3:15 PM, Robert Kanter 
> >> wrote:
> >>
> >>> + yarn-dev, hdfs-dev, mapred-dev
> >>>
> >>> On Tue, Apr 21, 2015 at 3:14 PM, Robert Kanter 
> >>> wrote:
> >>>
>  Hi all,
> 
>  Moving forward on some of the discussions on Hadoop 3, I've created
>  HADOOP-11858 to set the minimum version of Hadoop 3 to JDK 8.  I just
>  wanted to let everyone know in case there's some reason we shouldn't
> go
>  ahead with this.
> 
>  thanks
>  - Robert
> 
> >>>
> >>
>
>


-- 
Sean

Re: committing HADOOP-11746 test-patch improvements

[Sean Busbey]

On Wed, Apr 22, 2015 at 2:10 AM, Allen Wittenauer  wrote:

>
>
> * There have been a few runs which seems to indicate that *something* is
> destroying the artifact directory in the middle of  runs…. which is very
> very odd and something I hadn’t seen in any of my testing.  In any case, I
> clearly need to add some safety code here to report back that something
> went awry and report back which node, console, etc this happened on.
> Someone more familiar with the Jenkins setup might be able to shed some
> light on why that might happen. All of these runs appear to be on H3, so
> might be related? Impacted issues with this have been:
>
> - HDFS-8200 (https://builds.apache.org/job/PreCommit-HDFS-Build/10335/)
> - HDFS-8147 (https://builds.apache.org/job/PreCommit-HDFS-Build/10338/)
> - YARN-3301 (https://builds.apache.org/job/PreCommit-YARN-Build/7441/)
>
>
>From the HDFS precommit build:

>
> PATCHPROCESS=${WORKSPACE}/../patchprocess
> mkdir -p ${PATCHPROCESS}
>

Working on directories outside of the workspace for the job is not good,
though I'm not sure if that's the source of the issue. Do I need to
coordinate fixing this with anyone?

-- 
Sean

Re: we need a fix: precommit failures correlate to hdfs patches

[Sean Busbey]

The patch artifact directory in the mainline hadoop jenkins jobs are
outside of the workspace. I'm not sure what, if anything, jenkins
guarantees about files out of the main workspace.

They all write to ${WORKSPACE}/../patchProcess, which will probably collide
if multiple runs happen on the same machine. They also all blindly move
that directory at the end of the run.

On Sun, May 3, 2015 at 3:02 PM, Allen Wittenauer  wrote:

>
> So, as some may have noticed, I slammed the Jenkins servers over
> the weekend to get some recent patch test runs in JIRA for the bug bash
> this week.  I've had a suspicion for a while now that either the long run
> times of the hadoop-hdfs module unit tests (typically 2+ hours) or the hdfs
> tests themselves were related to the patch process directory getting
> removed out from underneath test-patch.
>
> To test the hypothesis, I submitted all of the non-HDFS patches so
> that they were first in the queue.  Let them run for a very long time.
> Jenkins bounced back and forth between YARN, MR, and HADOOP.   No issues
> encounters.  Added HDFS patches into the mix. BOOM. The dreaded "The patch
> artifact directory has been removed! “ started to appear here and there.
> This seems to provide some evidence that, yes, hdfs unit tests are
> directory or indirectly related to the failures.
>
> IMO, I think we need to take a serious look at:
>
> * splitting up the hadoop-hdfs module into multiple modules to
> reduce unit test run times
> * checking to see if the pre commit hooks in hdfs are different
> than the rest (I do know that the YARN bits are different and appear to
> have some bugs as well)
> * increasing the timeout for jenkins job runs
>
> FWIW, I’ve also found some minor things here and there with the
> rewritten test-patch.sh.  JIRAs have been filed.  One critical, one major
> and a handful of minor things.




-- 
Sean

Re: upstream jenkins build broken?

[Sean Busbey]

Hi Folks!

After working on test-patch with other folks for the last few months, I
think we've reached the point where we can make the fastest progress
towards the goal of a general use pre-commit patch tester by spinning
things into a project focused on just that. I think we have a mature enough
code base and a sufficient fledgling community, so I'm going to put
together a tlp proposal.

Thanks for the feedback thus far from use within Hadoop. I hope we can
continue to make things more useful.

-Sean

On Wed, Mar 11, 2015 at 5:16 PM, Sean Busbey  wrote:

> HBase's dev-support folder is where the scripts and support files live.
> We've only recently started adding anything to the maven builds that's
> specific to jenkins[1]; so far it's diagnostic stuff, but that's where I'd
> add in more if we ran into the same permissions problems y'all are having.
>
> There's also our precommit job itself, though it isn't large[2]. AFAIK, we
> don't properly back this up anywhere, we just notify each other of changes
> on a particular mail thread[3].
>
> [1]: https://github.com/apache/hbase/blob/master/pom.xml#L1687
> [2]: https://builds.apache.org/job/PreCommit-HBASE-Build/ (they're all
> read because I just finished fixing "mvn site" running out of permgen)
> [3]: http://s.apache.org/NT0
>
>
> On Wed, Mar 11, 2015 at 4:51 PM, Chris Nauroth 
> wrote:
>
>> Sure, thanks Sean!  Do we just look in the dev-support folder in the HBase
>> repo?  Is there any additional context we need to be aware of?
>>
>> Chris Nauroth
>> Hortonworks
>> http://hortonworks.com/
>>
>>
>>
>>
>>
>>
>> On 3/11/15, 2:44 PM, "Sean Busbey"  wrote:
>>
>> >+dev@hbase
>> >
>> >HBase has recently been cleaning up our precommit jenkins jobs to make
>> >them
>> >more robust. From what I can tell our stuff started off as an earlier
>> >version of what Hadoop uses for testing.
>> >
>> >Folks on either side open to an experiment of combining our precommit
>> >check
>> >tooling? In principle we should be looking for the same kinds of things.
>> >
>> >Naturally we'll still need different jenkins jobs to handle different
>> >resource needs and we'd need to figure out where stuff eventually lives,
>> >but that could come later.
>> >
>> >On Wed, Mar 11, 2015 at 4:34 PM, Chris Nauroth > >
>> >wrote:
>> >
>> >> The only thing I'm aware of is the failOnError option:
>> >>
>> >>
>> >>
>> http://maven.apache.org/plugins/maven-clean-plugin/examples/ignoring-erro
>> >>rs
>> >> .html
>> >>
>> >>
>> >> I prefer that we don't disable this, because ignoring different kinds
>> of
>> >> failures could leave our build directories in an indeterminate state.
>> >>For
>> >> example, we could end up with an old class file on the classpath for
>> >>test
>> >> runs that was supposedly deleted.
>> >>
>> >> I think it's worth exploring Eddy's suggestion to try simulating
>> failure
>> >> by placing a file where the code expects to see a directory.  That
>> might
>> >> even let us enable some of these tests that are skipped on Windows,
>> >> because Windows allows access for the owner even after permissions have
>> >> been stripped.
>> >>
>> >> Chris Nauroth
>> >> Hortonworks
>> >> http://hortonworks.com/
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On 3/11/15, 2:10 PM, "Colin McCabe"  wrote:
>> >>
>> >> >Is there a maven plugin or setting we can use to simply remove
>> >> >directories that have no executable permissions on them?  Clearly we
>> >> >have the permission to do this from a technical point of view (since
>> >> >we created the directories as the jenkins user), it's simply that the
>> >> >code refuses to do it.
>> >> >
>> >> >Otherwise I guess we can just fix those tests...
>> >> >
>> >> >Colin
>> >> >
>> >> >On Tue, Mar 10, 2015 at 2:43 PM, Lei Xu  wrote:
>> >> >> Thanks a lot for looking into HDFS-7722, Chris.
>> >> >>
>> >> >> In HDFS-7722:
>> >> >> TestDataNodeVolumeFailureXXX tests reset data dir permissions in

Re: set up jenkins test for branch-2

[Sean Busbey]

pre-commit will already test on branch-2 provided you follow the patch
naming guidelines.

there is also a branch-2 specific jenkins job:
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-branch2/

I'd suggest starting by looking at that job and filing jiras to address
whatever the failures are. May 14th was the last time is wasn't marked as
failed and that build was unstable, so there's probably a good deal of work.

On Sun, Jun 14, 2015 at 3:00 PM, Yongjun Zhang  wrote:

> Hi,
>
> We touched this topic before but it was put on hold. I'd like to bring it
> to our attention again.
>
> From time to time we saw changes that work fine in trunk but not branch-2,
> and we don't catch the issue in a timely manner. The difference between
> trunk and branch-2 is sufficient to justify periodic jenkins test and even
> pre-commit test for branch-2.
>
> I created https://issues.apache.org/jira/browse/INFRA-9226 earlier but I'm
> not sure who are the right folks to take care of it.
>
> Any one could help follow-up?
>
> Thanks a lot and best regards,
>
> --Yongjun
>



-- 
Sean

Re: set up jenkins test for branch-2

[Sean Busbey]

On Mon, Jun 15, 2015 at 1:39 PM, Yongjun Zhang  wrote:

> Thanks Sean and Allen!
>
> I was not aware of that there is already a way to trigger branch-2 test.
> Good to know.
>
> There are multiple solutions here:
>
> 1. When posting patches, we can post two versions of patches, one for
> trunk, and one for branch-2, so to trigger two pre-commit jenkins test for
> both branches. This would help catching issue before committing. However,
> it's going to be more workload on the testing machines, so alternatively,
> we can probably defer the branch-2 testing until committing to trunk and
> before branch-2 testing.
>
> 2. Only post patch for trunk, we cherry-pick to branch-2 when committing.
> We can set up periodic jenkins test (such as nightly) for branch-2 to catch
> problem periodically. But that's basically being ignored by us as Allen
> pointed out.
>


I wouldn't worry about the load on the test machines. If you do, a third
option
is for committers to use test-patch on their local machine after making the
branch-2 version of the patch.

-- 
Sean

Re: F 6/19: Jenkins clogged up

[Sean Busbey]

Thanks for hte heads up.

On Fri, Jun 19, 2015 at 1:43 PM, Chris Nauroth 
wrote:

> Hi everyone,
>
> I was just in contact with Apache infrastructure.  Jenkins wasn't running
> jobs for a while, so there is a large backlog in the queue now (over 200
> jobs).  Infra has fixed the problems, so jobs are running now, but our
> test-patch runs might have to sit in the queue a long time today.
>
> --Chris Nauroth
>
>


-- 
Sean

Re: Planning Hadoop 2.6.1 release

[Sean Busbey]

Any update on a release plan for 2.6.1?

On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> HI vinod
>
> any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
> as stable give..?
>
>
> Thanks & Regards
>  Brahma Reddy Battula
>
> 
> From: Zhihai Xu [z...@cloudera.com]
> Sent: Wednesday, May 13, 2015 12:04 PM
> To: mapreduce-...@hadoop.apache.org
> Cc: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org
> Subject: Re: Planning Hadoop 2.6.1 release
>
> Hi Akira,
>
> Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
> bug.
> It will work better with YARN-2992.
>
> thanks
> zhihai
>
>
> On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <
> ajisa...@oss.nttdata.co.jp>
> wrote:
>
> > Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
> > to include the following:
> >
> > * HADOOP-11343. Overflow is not properly handled in calculating final iv
> > for AES CTR
> > * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
> > execute any further apps
> > * YARN-2992. ZKRMStateStore crashes due to session expiry
> > * YARN-3013. AMRMClientImpl does not update AMRM token properly
> > * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
> > die
> > * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
> to
> > a reducer
> >
> > All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
> >
> > Regards,
> > Akira
> >
> >
> > On 5/4/15 11:15, Brahma Reddy Battula wrote:
> >
> >> Hello Vinod,
> >>
> >> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
> >> will not be usable when cluster installed in HA mode and try to get
> >> filecontext like below..
> >>
> >>
> >> Path path = new
> >>
> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
> >> FileSystem fs = path.getFileSystem(new Configuration());
> >> path = fs.makeQualified(path);
> >> FileContext fc = FileContext.getFileContext(path.toUri(),new
> >> Configuration());
> >>
> >>
> >>
> >> Thanks & Regards
> >> Brahma Reddy Battula
> >> 
> >> From: Chris Nauroth [cnaur...@hortonworks.com]
> >> Sent: Friday, May 01, 2015 4:32 AM
> >> To: mapreduce-...@hadoop.apache.org; common-...@hadoop.apache.org;
> >> yarn-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> >> Subject: Re: Planning Hadoop 2.6.1 release
> >>
> >> Thank you, Arpit.  In addition, I suggest we include the following:
> >>
> >> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
> >> pipe is full
> >> HADOOP-11604. Prevent ConcurrentModificationException while closing
> domain
> >> sockets during shutdown of DomainSocketWatcher thread.
> >> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
> >> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
> there
> >> is an I/O error during requestShortCircuitShm
> >>
> >> HADOOP-11604 and 11648 are not critical by themselves, but they are
> >> pre-requisites to getting a clean cherry-pick of 11802, which we believe
> >> finally fixes the root cause of this issue.
> >>
> >>
> >> --Chris Nauroth
> >>
> >>
> >>
> >>
> >> On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:
> >>
> >>  HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
> >>> requested in [1].
> >>>
> >>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
> >>> should be non static
> >>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
> >>> synchronization
> >>>
> >>> HDFS-7009. Active NN and standby NN have different live nodes.
> >>> HDFS-7035. Make adding a new data directory to the DataNode an atomic
> and
> >>> improve error handling
> >>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
> >>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
> >>> block files are present in the same volume.
> >>> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang
> datanodes
> >>> HDFS-7503. Namenode restart after large deletions can cause slow
> >>> processReport.
> >>> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
> >>> HDFS-7579. Improve log reporting during block report rpc failure.
> >>> HDFS-7587. Edit log corruption can happen if append fails with a quota
> >>> violation.
> >>> HDFS-7596. NameNode should prune dead storages from storageMap.
> >>> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned
> blocks
> >>> in the blocksMap on NameNode restart.
> >>> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
> >>> DataNode to register successfully with only one NameNode.
> >>> HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
> >>> failure.
> >>> HDFS-7831. Fix the starting index and end condition of the loop in
> >>> Fil

Re: IMPORTANT: automatic changelog creation

[Sean Busbey]

On Jul 8, 2015 2:13 AM, "Tsuyoshi Ozawa"  wrote:
>
> +1, thanks Allen and Andrew for taking lots effort!
>
> > Is there any possibility that, we can restrict someone from editing the
> > issue in jira once its marked as "closed" after release?
>
> Vinay's comment looks considerable for us to me. What do you think?
>

Mistakes happen, even during the release process.

Presuming the set of folks who can edit closed tickets is already
restricted to contributors, why not assume any edits are the community
making things more accurate?

-- 
Sean

Re: Planning Hadoop 2.6.1 release

[Sean Busbey]

If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
currently only in 2.8+ and would benefit 2.6 and 2.7.

On Wed, Aug 5, 2015 at 2:56 PM, Junping Du  wrote:

> I would like to nominate YARN-3832 as 2.6.1 candidate which is critical
> and I also saw it happened recently on a 2.6.0 cluster. Hope this is not
> too late.
>
> Thanks,
>
> Junping
> 
> From: Rich Haase 
> Sent: Thursday, August 06, 2015 1:52 AM
> To: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Cc: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> Subject: Re: Planning Hadoop 2.6.1 release
>
> +1 to add those fixes.
>
>
> Rich Haase | Sr. Software Engineer | Pandora
> m 303.887.1146 | rha...@pandora.com
>
>
>
>
> On 8/5/15, 11:42 AM, "Wangda Tan"  wrote:
>
> >Can we add following two fixes to 2.6.1?
> >
> >https://issues.apache.org/jira/browse/YARN-2922 and
> >https://issues.apache.org/jira/browse/YARN-3487.
> >
> >They're not fatal issue, but they can cause lots of issue in a large
> >cluster.
> >
> >Thanks,
> >Wangda
> >
> >
> >On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee  wrote:
> >
> >> See my later update in the thread. HDFS-7704 is in the list.
> >>
> >> Thanks,
> >> Sangjin
> >>
> >> On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli <
> >> vino...@hortonworks.com> wrote:
> >>
> >> > Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and
> >>is
> >> > not part of the candidate list. Removed HDFS-7916 from the list.
> >> >
> >> > Thanks
> >> > +Vinod
> >> >
> >> > > On Jul 24, 2015, at 6:32 PM, Sangjin Lee  wrote:
> >> > >
> >> > > Out of the JIRAs we proposed, please remove HDFS-7916. I don't
> >>think it
> >> > > applies to 2.6.
> >> > >
> >> > > Thanks,
> >> > > Sangjin
> >> >
> >> >
> >>
>
>


-- 
Sean

Re: ASF OS X Build Infrastructure

[Sean Busbey]

Some talk about the MSDN-for-committers program recently passed by on a private
list. It's still active, it just changed homes within Microsoft. The
info should still be in the committer repo. If something is amiss
please let me know and I'll pipe up to the folks already plugged in to
confirming it's active.

On Fri, May 20, 2016 at 12:13 PM, Chris Nauroth
 wrote:
> It's very disappointing to see that vanish.  I'm following up to see if I
> can learn more about what happened or if I can do anything to help
> reinstate it.
>
> --Chris Nauroth
>
>
>
>
> On 5/20/16, 6:11 AM, "Steve Loughran"  wrote:
>
>>
>>> On 20 May 2016, at 10:40, Lars Francke  wrote:
>>>

 Regarding lack of personal access to anything but Linux, I'll take
this as
 an opportunity to remind everyone that ASF committers (not just
limited to
 Hadoop committers) are entitled to a free MSDN license, which can get
you
 a Windows VM for validating Windows issues and any patches that touch
 cross-platform concerns, like the native code.  Contributors who are
not
 committers still might struggle to get access to Windows, but all of us
 reviewing and committing patches do have access.

>>>
>>> Actually, from all I can tell this MSDN offer has been discontinued for
>>> now. All the information has been removed from the committers repo. Do
>>>you
>>> have any more up to date information on this?
>>>
>>
>>
>>That's interesting.
>>
>>I did an SVN update and it went away..looks like something happened on
>>April 26
>>
>>No idea, though the svn log has a bit of detail
>>
>>-
>>To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>>For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>>
>>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DICUSS] Upgrading Guice to 4.0(HADOOP-12064)

[Sean Busbey]

At the very least, I'm running through an updated shaded hadoop client
this week[1] (HBase is my test application and it wandered onto some
private things that broke in branch-2). And Sangjin has a good lead on
an lower-short-term-cost incremental improvement for runtime isolation
of apps built on yarn/mapreduce[2]. He's been patiently waiting for
more review feedback.


[1]: https://issues.apache.org/jira/browse/HADOOP-11804
[2]: https://issues.apache.org/jira/browse/HADOOP-13070

On Wed, Jun 29, 2016 at 12:33 PM, Vinod Kumar Vavilapalli
 wrote:
> My strong expectation is that we’ll have a version of classpath isolation in 
> our first release of 3.x. I’m planning to spending some cycles right away on 
> this.
>
> Assuming classpath isolation gets in, it is reasonable to bump up our 
> dependencies like Jetty / Guice to the latest stable versions.
>
> Thanks
> +Vinod
>
>> On Jun 27, 2016, at 6:01 AM, Tsuyoshi Ozawa  wrote:
>>
>> Hi developers,
>>
>> I will plan to upgrade Google Guice dependency on trunk. The change
>> also includes asm and cglib upgrade.
>> I checked following points:
>>
>> * Both HDFS and YARN UIs work well.
>> * All webIU-related tests pass as described on HADOOP-12064.
>> * Ran mapreduce job, and it works well.
>>
>> https://issues.apache.org/jira/browse/HADOOP-12064
>>
>> Do you have any concern or opinion?  I would like to merge it to trunk
>> on this Friday if you have no objections.
>>
>> Best,
>> - Tsuyoshi
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Official Docker Image at release time

[Sean Busbey]

FWIW, there is a "blessed" apache area on docker hub now, and it's
just an INFRA request to point out the needed Dockerfile in the repo.

PMCs can also request write access to bintray hosting of docker images
for PMC members.

Info on INFRA-8441, example on INFRA-12019.

A Docker image that starts up a pseudo distributed instance would be
great for new folks. I don't know if it's worth the investment to
build images that do more complex multi-instance deployments (though
there is some fledgling work in HBase for tooling that would do this
for us).

On Tue, Jul 19, 2016 at 2:46 AM, Tsuyoshi Ozawa  wrote:
> Hi developers,
>
> Klaus mentioned the availability of an official docker image of Apache
> Hadoop. Is it time that we start to distribute an official docker
> image at release time?
>
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201607.mbox/%3CSG2PR04MB162977CFE150444FA022510FB6370%40SG2PR04MB1629.apcprd04.prod.outlook.com%3E
>
> Thoughts?
>
> Thanks,
> - Tsuyoshi
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

[Sean Busbey]

thanks for bringing this up! big +1 on upgrading dependencies for 3.0.

I have an updated patch for HADOOP-11804 ready to post this week. I've
been updating HBase's master branch to try to make use of it, but
could use some other reviews.

On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa  wrote:
> Hi developers,
>
> I'd like to discuss how to make an advance towards dependency
> management in Apache Hadoop trunk code since there has been lots work
> about updating dependencies in parallel. Summarizing recent works and
> activities as follows:
>
> 0) Currently, we have merged minimum update dependencies for making
> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
> 1) After that, some people suggest that we should update the other
> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
> 2) In parallel, Sangjin and Sean are working on classpath isolation:
> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
>
> Main problems we try to solve in the activities above is as follows:
>
> * 1) tries to solve dependency hell between user-level jar and
> system(Hadoop)-level jar.
> * 2) tries to solve updating old libraries.
>
> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
> to separate class loader between client-side dependencies and
> server-side dependencies in Hadoop, so we can the change policy of
> updating libraries after doing 2). We can also decide which libraries
> can be shaded after 2).
>
> Hence, IMHO, a straight way we should go to is doing 2 at first.
> After that, we can update both client-side and server-side
> dependencies based on new policy(maybe we should discuss what kind of
> incompatibility is acceptable, and the others are not).
>
> Thoughts?
>
> Thanks,
> - Tsuyoshi
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Official Docker Image at release time

[Sean Busbey]

Folks might want to take a look at

https://issues.apache.org/jira/browse/HBASE-12721

and its associated review board posting of aggregate work. One of hte
HBase community members has been creating infra (geared towards test
ATM) for spinning up clusters of docker images. from what I
understand, right now that includes both hdfs, hbase, and yarn
clusters.

On Thu, Jul 21, 2016 at 5:07 AM, Tsuyoshi Ozawa  wrote:
> Hi Kai,
>
> Thanks for your positive feedback!
>
>> I think also providing Dockerfile in some flexible form (like template
> file or fetching configuration from env variables?)
> is useful because Docker image has fixed configuration and there is no room
> unless overriding with FROM.
>
> Sounds good idea. What do you think about the difference of roles
> between master and workers? Should we prepare separate Dockerfiles for each
> role?
>
> Thanks,
> - Tsuyoshi
>
> On Tuesday, 19 July 2016, Sasaki Kai  wrote:
>
>> Hi Tsuyoshi
>>
>> Official docker image of Hadoop sounds very good to me.
>> In my use case, we usually use Docker image of Hadoop as PoC or
>> development cluster because it is easy to deploy and modify.
>> Currently we use below Docker image or Dockerfile to launch our
>> development cluster.
>> https://hub.docker.com/r/sequenceiq/hadoop-docker/
>>
>> But it does not use the latest Hadoop package and dependencies. There is
>> often some time lag to catch up and updating.
>>
>> So there is a reason to provide Docker image and Dockerfile from
>> community. It enables developers to try Hadoop easily.
>> I think also providing Dockerfile in some flexible form (like template
>> file or fetching configuration from env variables?)
>> is useful because Docker image has fixed configuration and there is no
>> room unless overriding with FROM.
>> I assume developers who want to try new Hadoop also would like to modify
>> configuration or dependencies.
>> This can be achieved by flexible Dockerfile which uses configuration
>> passed from environment variables.
>>
>> Thanks
>>
>> Kai Sasaki
>>
>>
>> On Jul 19, 2016, at 4:46 PM, Tsuyoshi Ozawa > > wrote:
>>
>> Hi developers,
>>
>> Klaus mentioned the availability of an official docker image of Apache
>> Hadoop. Is it time that we start to distribute an official docker
>> image at release time?
>>
>>
>> http://mail-archives.apache.org/mod_mbox/hadoop-user/201607.mbox/%3CSG2PR04MB162977CFE150444FA022510FB6370%40SG2PR04MB1629.apcprd04.prod.outlook.com%3E
>>
>> Thoughts?
>>
>> Thanks,
>> - Tsuyoshi
>>
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> 
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> 
>>
>>
>>



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

[Sean Busbey]

My work on HADOOP-11804 *only* helps processes that sit outside of YARN. :)

On Fri, Jul 22, 2016 at 10:48 AM, Allen Wittenauer
 wrote:
>
> Does any of this work actually help processes that sit outside of YARN?
>
>> On Jul 21, 2016, at 12:29 PM, Sean Busbey  wrote:
>>
>> thanks for bringing this up! big +1 on upgrading dependencies for 3.0.
>>
>> I have an updated patch for HADOOP-11804 ready to post this week. I've
>> been updating HBase's master branch to try to make use of it, but
>> could use some other reviews.
>>
>> On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa  wrote:
>>> Hi developers,
>>>
>>> I'd like to discuss how to make an advance towards dependency
>>> management in Apache Hadoop trunk code since there has been lots work
>>> about updating dependencies in parallel. Summarizing recent works and
>>> activities as follows:
>>>
>>> 0) Currently, we have merged minimum update dependencies for making
>>> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
>>> 1) After that, some people suggest that we should update the other
>>> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
>>> 2) In parallel, Sangjin and Sean are working on classpath isolation:
>>> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
>>>
>>> Main problems we try to solve in the activities above is as follows:
>>>
>>> * 1) tries to solve dependency hell between user-level jar and
>>> system(Hadoop)-level jar.
>>> * 2) tries to solve updating old libraries.
>>>
>>> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
>>> to separate class loader between client-side dependencies and
>>> server-side dependencies in Hadoop, so we can the change policy of
>>> updating libraries after doing 2). We can also decide which libraries
>>> can be shaded after 2).
>>>
>>> Hence, IMHO, a straight way we should go to is doing 2 at first.
>>> After that, we can update both client-side and server-side
>>> dependencies based on new policy(maybe we should discuss what kind of
>>> incompatibility is acceptable, and the others are not).
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> - Tsuyoshi
>>>
>>> -
>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>>
>>
>>
>>
>> --
>> busbey
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

can someone add me to contributors for hte HDFS jira?

[Sean Busbey]

Hi!

I'm already in the contributor role on the HADOOP jira, could someone
add me as one on the HDFS jira? I'd like the ability to do some jira
gardening.

the relevant username is : busbey

-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Sean Busbey]

It's also the key Andrew has in the project's KEYS file:

http://www.apache.org/dist/hadoop/common/KEYS



On Tue, Aug 30, 2016 at 4:12 PM, Andrew Wang  wrote:
> Hi Eric, thanks for trying this out,
>
> I tried this gpg command to get my key, seemed to work:
>
> # gpg --keyserver pgp.mit.edu --recv-keys 7501105C
> gpg: requesting key 7501105C from hkp server pgp.mit.edu
> gpg: /root/.gnupg/trustdb.gpg: trustdb created
> gpg: key 7501105C: public key "Andrew Wang (CODE SIGNING KEY) <
> andrew.w...@cloudera.com>" imported
> gpg: no ultimately trusted keys found
> gpg: Total number processed: 1
> gpg:   imported: 1  (RSA: 1)
>
> Also found via search:
> http://pgp.mit.edu/pks/lookup?search=wang%40apache.org&op=index
>
>
> On Tue, Aug 30, 2016 at 2:06 PM, Eric Badger  wrote:
>
>> I don't know why my email client keeps getting rid of all of my spacing.
>> Resending the same email so that it is actually legible...
>>
>> All on OSX 10.11.6:
>> - Verified the hashes. However, Andrew, I don't know where to find your
>> public key, so I wasn't able to verify that they were signed by you.
>> - Built from source
>> - Deployed a pseudo-distributed clusterRan a few sample jobs
>> - Poked around the RM UI
>> - Poked around the attached website locally via the tarball
>>
>>
>> I did find one odd thing, though. It could be a misconfiguration on my
>> system, but I've never had this problem before with other releases (though
>> I deal almost exclusively in 2.x and so I imagine things might be
>> different). When I run a sleep job, I do not see any
>> diagnostics/logs/counters printed out by the client. Initially I ran the
>> job like I would on 2.7 and it failed (because I had not set
>> yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see
>> anything until I looked at the RM UI. There I was able to see all of the
>> logs for the failed job and diagnose the issue. Then, once I fixed my
>> parameters and ran the job again, I still didn't see any
>> diagnostics/logs/counters.
>>
>>
>> ebadger@foo: env | grep HADOOP
>> HADOOP_HOME=/Users/ebadger/Downloads/hadoop-3.0.0-alpha1-
>> src/hadoop-dist/target/hadoop-3.0.0-alpha1/
>> HADOOP_CONF_DIR=/Users/ebadger/conf
>> ebadger@foo: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/
>> mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep
>> -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
>> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1
>> -m 1 -r 1
>> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
>> ebadger@foo:
>>
>>
>> After running the above command, the RM UI showed a successful job, but as
>> you can see, I did not have anything printed onto the command line.
>> Hopefully this is just a misconfiguration on my part, but I figured that I
>> would point it out just in case.
>>
>>
>> Thanks,
>>
>>
>> Eric
>>
>>
>>
>> On Tuesday, August 30, 2016 4:00 PM, Eric Badger
>>  wrote:
>>
>>
>>
>> All on OSX 10.11.6:
>> Verified the hashes. However, Andrew, I don't know where to find your
>> public key, so I wasn't able to verify that they were signed by you.Built
>> from sourceDeployed a pseudo-distributed clusterRan a few sample jobsPoked
>> around the RM UIPoked around the attached website locally via the tarball
>> I did find one odd thing, though. It could be a misconfiguration on my
>> system, but I've never had this problem before with other releases (though
>> I deal almost exclusively in 2.x and so I imagine things might be
>> different). When I run a sleep job, I do not see any
>> diagnostics/logs/counters printed out by the client. Initially I ran the
>> job like I would on 2.7 and it failed (because I had not set
>> yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see
>> anything until I looked at the RM UI. There I was able to see all of the
>> logs for the failed job and diagnose the issue. Then, once I fixed my
>> parameters and ran the job again, I still didn't see any
>> diagnostics/logs/counters.
>> ebadger@foo: env | grep HADOOPHADOOP_HOME=/Users/
>> ebadger/Downloads/hadoop-3.0.0-alpha1-src/hadoop-dist/
>> target/hadoop-3.0.0-alpha1/HADOOP_CONF_DIR=/Users/ebadger/confebadger@foo:
>> $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/
>> mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep
>> -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
>> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1
>> -m 1 -r 1WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be
>> incomplete.ebadger@foo:
>> After running the above command, the RM UI showed a successful job, but as
>> you can see, I did not have anything printed onto the command line.
>> Hopefully this is just a misconfiguration on my part, but I figured that I
>> would point it out just in case.
>> Thanks,
>> Eric
>>
>>
>>
>> On Tuesday, August 30, 2016 12:58 PM, Andrew Wang <
>> andrew.w...@cloudera.com> wrote:

[NOTICE] breaking precommit checks

[Sean Busbey]

Hi folks!

a host of precommit checks are currently timing out due to an update
to our job configs (the timeout is currently set to 50 minutes).

I'm in the process of giving things more time based on our historic
usage, but if your check fails in the mean time and

1) the total run time is close to 50 minutes

2) the jenkins job when you click through shows status "aborted"

then please resubmit your patch.

-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [NOTICE] breaking precommit checks

[Sean Busbey]

Should be all set now.

On Tue, Nov 8, 2016 at 5:54 PM, Sean Busbey  wrote:
> Hi folks!
>
> a host of precommit checks are currently timing out due to an update
> to our job configs (the timeout is currently set to 50 minutes).
>
> I'm in the process of giving things more time based on our historic
> usage, but if your check fails in the mean time and
>
> 1) the total run time is close to 50 minutes
>
> 2) the jenkins job when you click through shows status "aborted"
>
> then please resubmit your patch.
>
> --
> busbey



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Sean Busbey]

disallowing force pushes to trunk was done back in:

* August 2014: INFRA-8195
* February 2016: INFRA-11136

On Mon, Apr 17, 2017 at 11:18 AM, Jason Lowe
 wrote:
> I found at least one commit that was dropped, MAPREDUCE-6673.  I was able to 
> cherry-pick the original commit hash since it was recorded in the commit 
> email.
> This begs the question of why we're allowing force pushes to trunk.  I 
> thought we asked to have that disabled the last time trunk was accidentally 
> clobbered?
> Jason
>
>
> On Monday, April 17, 2017 10:18 AM, Arun Suresh  
> wrote:
>
>
>  Hi
>
> I had the Apr-14 eve version of trunk on my local machine. I've pushed that.
> Don't know if anything was committed over the weekend though.
>
> Cheers
> -Arun
>
> On Mon, Apr 17, 2017 at 7:17 AM, Anu Engineer 
> wrote:
>
>> Hi Allen,
>>
>> https://issues.apache.org/jira/browse/INFRA-13902
>>
>> That happened with ozone branch too. It was an inadvertent force push.
>> Infra has advised us to force push the latest branch if you have it.
>>
>> Thanks
>> Anu
>>
>>
>> On 4/17/17, 7:10 AM, "Allen Wittenauer"  wrote:
>>
>> >Looks like someone reset HEAD back to Mar 31.
>> >
>> >Sent from my iPad
>> >
>> >> On Apr 16, 2017, at 12:08 AM, Apache Jenkins Server <
>> jenk...@builds.apache.org> wrote:
>> >>
>> >> For more details, see https://builds.apache.org/job/
>> hadoop-qbt-trunk-java8-linux-x86/378/
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -1 overall
>> >>
>> >>
>> >> The following subsystems voted -1:
>> >>docker
>> >>
>> >>
>> >> Powered by Apache Yetus 0.5.0-SNAPSHOT  http://yetus.apache.org
>> >>
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >
>> >
>> >-
>> >To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> >For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> >
>> >
>>
>>
>
>
>



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Pre-Commit build is failing

[Sean Busbey]

-dev@yetus to bcc, since I think this is a Hadoop issue and not a yetus
issue.

Please review/commit HADOOP-14686 (which I am providing as a
volunteer/contributor on the Hadoop project).

On Tue, Jul 25, 2017 at 7:54 PM, Allen Wittenauer 
wrote:

>
> Again: just grab the .gitignore file from trunk and update it in
> branch-2.7. It hasn't been touched (outside of one patch) in years.  The
> existing jobs should then work.
>
> The rest of this stuff, yes, I know and yes it's intentional.  The
> directory structure was inherited from the original jobs that Nigel set up
> with the old version of test-patch.  Maybe some day I'll fix it.  But
> that's a project for a different day.  In order to fix it, it means taking
> down the patch testing for Hadoop while I work it out.  You'll notice that
> all of the other Yetus jobs for Hadoop have a much different layout.
>
>
>
>
> > On Jul 25, 2017, at 7:24 PM, suraj acharya  wrote:
> >
> > Hi,
> >
> > Seems like the issue was incorrect/unclean checkout.
> > I made a few changes[1] to the directories the checkout happens to  and
> it is now running.
> > Of course, this build[2] will take some time to run, but at the moment,
> it is running maven install.
> >
> > I am not sure who sets up/ manages the jenkins job of HDFS and dont want
> to change that, but I will keep the dummy job around for a couple of days
> in case anyone wants to see.
> > Also, I see that you'll were using the master branch of Yetus. If there
> is no patch present there that is of importance, then I would recommend to
> use the latest stable release version 0.5.0
> >
> > If you have more questions, feel free to ping dev@yetus.
> > Hope this helps.
> >
> > [1]: https://builds.apache.org/job/PreCommit-HDFS-Build-Suraj-
> Copy/configure
> > [2]: https://builds.apache.org/job/PreCommit-HDFS-Build-Suraj-
> Copy/12/console
> >
> > -Suraj Acharya
> >
> > On Tue, Jul 25, 2017 at 6:57 PM, suraj acharya 
> wrote:
> > For anyone looking. I created another job here. [1].
> > Set it with debug to see the issue.
> > The error is being seen here[2].
> > From the looks of it, it looks like, the way the checkout is happening
> is not very clean.
> > I will continue to look at it, but in case anyone wants to jump in.
> >
> > [1] : https://builds.apache.org/job/PreCommit-HDFS-Build-Suraj-Copy/
> > [2] : https://builds.apache.org/job/PreCommit-HDFS-Build-Suraj-
> Copy/11/console
> >
> > -Suraj Acharya
> >
> > On Tue, Jul 25, 2017 at 6:28 PM, Konstantin Shvachko <
> shv.had...@gmail.com> wrote:
> > Hi Yetus developers,
> >
> > We cannot build Hadoop branch-2.7 anymore. Here is a recent example of a
> > failed build:
> > https://builds.apache.org/job/PreCommit-HDFS-Build/20409/console
> >
> > It seems the build is failing because Yetus cannot apply the patch from
> the
> > jira.
> >
> > ERROR: HDFS-11896 does not apply to branch-2.7.
> >
> > As far as I understand this is Yetus problem. Probably in 0.3.0.
> > I can apply this patch successfully, but Yetus test-patch.sh script
> clearly
> > failed to apply. Cannot say why because Yetus does not report it.
> > I also ran Hadoop's test-patch.sh script locally and it passed
> successfully
> > on branch-2.7.
> >
> > Could anybody please take a look and help fixing the build.
> > This would be very helpful for the release (2.7.4) process.
> >
> > Thanks,
> > --Konst
> >
> > On Mon, Jul 24, 2017 at 10:41 PM, Konstantin Shvachko <
> shv.had...@gmail.com>
> > wrote:
> >
> > > Or should we backport the entire HADOOP-11917
> > >  ?
> > >
> > > Thanks,
> > > --Konst
> > >
> > > On Mon, Jul 24, 2017 at 6:56 PM, Konstantin Shvachko <
> shv.had...@gmail.com
> > > > wrote:
> > >
> > >> Allen,
> > >>
> > >> Should we add "patchprocess/" to .gitignore, is that the problem for
> 2.7?
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> On Fri, Jul 21, 2017 at 6:24 PM, Konstantin Shvachko <
> > >> shv.had...@gmail.com> wrote:
> > >>
> > >>> What stuff? Is there a jira?
> > >>> It did work like a week ago. Is it a new Yetus requirement.
> > >>> Anyways I can commit a change to fix the build on our side.
> > >>> Just need to know what is missing.
> > >>>
> > >>> Thanks,
> > >>> --Konst
> > >>>
> > >>> On Fri, Jul 21, 2017 at 5:50 PM, Allen Wittenauer <
> > >>> a...@effectivemachines.com> wrote:
> > >>>
> > 
> >  > On Jul 21, 2017, at 5:46 PM, Konstantin Shvachko <
> >  shv.had...@gmail.com> wrote:
> >  >
> >  > + d...@yetus.apache.org
> >  >
> >  > Guys, could you please take a look. Seems like Yetus problem with
> >  > pre-commit build for branch-2.7.
> > 
> > 
> >  branch-2.7 is missing stuff in .gitignore.
> > >>>
> > >>>
> > >>>
> > >>
> > >
> >
> >
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>


-- 
busbey

Re: Reviving HADOOP-7435: Making Jenkins pre-commit build work with branches

[Sean Busbey]

+1

If we can make things look like HBase support for precommit testing on
branches (HBASE-12944), that would make it easier for new and occasional
contributors who might end up working in other ecosystem projects. AFAICT,
Jonathan's proposal for branch names in patch names does this.



On Wed, Mar 4, 2015 at 3:41 PM, Karthik Kambatla  wrote:

> Thanks for reviving this on email, Vinod. Newer folks like me might not be
> aware of this JIRA/effort.
>
> This would be wonderful to have so (1) we know the status of release
> branches (branch-2, etc.) and also (2) feature branches (YARN-2928).
> Jonathan's or Matt's proposal for including branch name looks reasonable to
> me.
>
> If none has any objections, I think we can continue on JIRA and get this
> in.
>
> On Wed, Mar 4, 2015 at 1:20 PM, Vinod Kumar Vavilapalli <
> vino...@hortonworks.com> wrote:
>
> > Hi all,
> >
> > I'd like us to revive the effort at
> > https://issues.apache.org/jira/browse/HADOOP-7435 to make precommit
> > builds being able to work with branches. Having the Jenkins verify
> patches
> > on branches is very useful even if there may be relaxed review oversight
> on
> > the said-branch.
> >
> > Unless there are objections, I'd request help from Giri who already has a
> > patch sitting there for more than a year before. This may need us to
> > collectively agree on some convention - the last comment says that the
> > branch patch name should be in some format for this to work.
> >
> > Thanks,
> > +Vinod
> >
>
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> 
> http://five.sentenc.es
>



-- 
Sean

Re: [DISCUSS] Prefer Github PR Integration over patch in JIRA

[Sean Busbey]

a word of caution. according to INFRA-18748, asf infra is going to be
cutting out blind PR building. So we'll need to be sure that precommit
integration works e.g. testing PRs because there's a JIRA that a
whitelisted contributor has associated and put in patch available
status.

On Mon, Jul 22, 2019 at 1:06 PM Wei-Chiu Chuang  wrote:
>
> Historically, Hadoop developers create patches and attache them to JIRA,
> andthen the Yetus bot runs precommit against the patch in the JIRA.
>
> The Github PR is more convenient for code review and less hassle for
> committers to merge a commit. I am proposing for the community to prefer
> Github PR over the traditional patch-in-jira. This doesn't mean we will
> reject the traditional way, but we can move gradually to the new way.
> Additionally, update the Hadoop "How to contribute" wiki, and advertise
> that Github PR is the preferred method.
>
> Thoughts?



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Make EOL branches to public?

[Sean Busbey]

For what it's worth, in HBase we've been approximating which Hadoop
lines are EOL by looking at release rates and specifically CVE
announcements that include an affected release line but do not include
a fix for that release line. Our current approximation[1] lists 2.6,
2.7, and 3.0 as dead. So that lines up well with your proposed list.


[1]: http://hbase.apache.org/book.html#hadoop

On Fri, Aug 16, 2019 at 5:14 AM Wangda Tan  wrote:
>
> Hi folks,
>
> Want to hear your thoughts about what we should do to make some branches
> EOL. We discussed a couple of times before in dev lists and PMC list.
> However, we couldn't get a formal process of EOL. According to the
> discussion. It is hard to decide it based on time like "After 1st release,
> EOL in 2 years". Because community members still want to maintain it and
> developers still want to get a newer version released.
>
> However, without a public place to figure out which release will be EOL, it
> is very hard for users to choose the right releases to upgrade and develop.
>
> So I want to propose to make an agreement about making a public EOL wiki
> page and create a process to EOL a release:
>
> The process I'm thinking is very simple: If no volunteer to do a
> maintenance release in a short to mid-term (like 3 months to 1 or 1.5
> year). We will claim a release is EOL. After EOL community can still choose
> to do a security-only release.
>
> Here's a list which I can think about:
>
> 1) 2.6.x (Or any release older than 2.6) (Last released at Oct 2016)
> 2) 2.7.x (Last released at Apr 2018)
> 4) 3.0.x (Last released at May 2018)
>
> Thoughts?
>
> Thanks,
> Wangda



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Sean Busbey]

+1

On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan  wrote:
>
> Hi all,
>
> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> and 3.0 EOL. This is based on discussions of [1]
>
> This discussion runs for 7 days and will conclude on Aug 28 Wed.
>
> Please feel free to share your thoughts.
>
> Thanks,
> Wangda
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> ,



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Sean Busbey]

+1 (non-binding)

On Fri, Aug 23, 2019 at 9:06 PM Wangda Tan  wrote:
>
> Hi devs,
>
> This is a voting thread to move Submarine source code, documentation from
> Hadoop repo to a separate Apache Git repo. Which is based on discussions of
> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
>
> Contributors who have permissions to push to Hadoop Git repository will
> have permissions to push to the new Submarine repository.
>
> This voting thread will run for 7 days and will end at Aug 30th.
>
> Please let me know if you have any questions.
>
> Thanks,
> Wangda Tan



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Sean Busbey]

On Mon, Aug 26, 2019 at 9:20 AM Eric Badger  wrote:
>
> - Stuff has been going into branch-2 sporadically but I don't who is
> actively
> using that code other than as part of a cherrypick backwards strategy.
>
> - Should we do a 2.10.x release? Or just say "time to upgrade?"
>
> We have talked at a few different Hadoop contributors meetups and community
> syncs and agreed that 2.10 should serve as a "bridge" release for 3.x so
> that 2.x clients can talk to 3.x servers and vice versa without
> compatibility issues. At the last meeting where we discussed this it seemed
> that there were 3 major compatibility issues. The biggest one was in the
> edit logs.
>


Just a quick friendly reminder to everyone that stuff has to make it
to the mailing lists for Hadoop for it to be considered a discussion
that happened "in the community". Especially decisions and the
discussions that lead to them have to happen on our mailing lists and
not at in person events.

-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] GitHub PRs without JIRA number

[Sean Busbey]

We should add a Pull Request Template that specifically calls out the
expectation that folks need to have a JIRA associated with their PR
for it to get reviewed. Expectations around time to response and how
to go about getting attention when things lag would also be good to
include. (e.g. are folks expected to ping on the jira? are folks
expected to email a relevant *-dev list?)

If anyone is interested in doing the work to make it so "test this" /
"retest this" / etc work, open a jira and I'll give you some pointers
of examples to go off of. We use a plugin to do this for yetus based
tests in some HBase repos.

On Wed, Sep 4, 2019 at 1:59 PM Wei-Chiu Chuang
 wrote:
>
> +general@
>
>
> On Wed, Aug 28, 2019 at 6:42 AM Wei-Chiu Chuang  wrote:
>
> > I don't think our GitHub integration supports those commands. Ozone has
> > its own github integration that can test individual PRs though.
> >
> >
> >
> > On Tue, Aug 27, 2019 at 12:40 PM Iñigo Goiri  wrote:
> >
> >> I wouldn't go for #3 and always require a JIRA for a PR.
> >>
> >> In general, I think we should state the best practices for using GitHub
> >> PRs.
> >> There were some guidelines but they were kind of open
> >> For example, adding always a link to the JIRA to the description.
> >> I think PRs can have a template as a start.
> >>
> >> The other thing I would do is to disable the automatic Jenkins trigger.
> >> I've seen the "retest this" and others:
> >> https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin
> >> https://github.com/jenkinsci/ghprb-plugin/blob/master/README.md
> >>
> >>
> >>
> >> On Tue, Aug 27, 2019 at 10:47 AM Wei-Chiu Chuang 
> >> wrote:
> >>
> >> > Hi,
> >> > There are hundreds of GitHub PRs pending review. Many of them just sit
> >> > there wasting Jenkins resources.
> >> >
> >> > I suggest:
> >> > (1) close PRs that went stale (i.e. doesn't compile). Or even close PRs
> >> > that hasn't been reviewed for more than a year.
> >> > (1) close PRs that doesn't have a JIRA number. No one is going to
> >> review a
> >> > big PR that doesn't have a JIRA anyway.
> >> > (2) For PRs without JIRA number, file JIRAs for the PR on behalf of the
> >> > reporter.
> >> > (3) For typo fixes, merge the PRs directly without a JIRA. IMO, this is
> >> the
> >> > best use of GitHub PR.
> >> >
> >> > Thoughts?
> >> >
> >>
> >



-- 
busbey

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12715) I didn't find the "-find" command on hadoop2.6

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HDFS-12715.

   Resolution: Invalid
Fix Version/s: (was: 2.6.0)

Please use the user@hadoop mailing list for these kinds of questions:

https://lists.apache.org/list.html?u...@hadoop.apache.org

> I didn't find the "-find" command on hadoop2.6
> --
>
> Key: HDFS-12715
> URL: https://issues.apache.org/jira/browse/HDFS-12715
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: auto-failover
>Affects Versions: 2.6.0
>Reporter: Djc
>Priority: Critical
>
> When I looked for files on HDFS, I found no "find" command. I didn't find the 
> "find" command by using "Hadoop FS -help".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-8101) DFSConfigKeys pulls in WebHDFS classes at runtime

[Sean Busbey (JIRA)]

Sean Busbey created HDFS-8101:
-

 Summary: DFSConfigKeys pulls in WebHDFS classes at runtime
 Key: HDFS-8101
 URL: https://issues.apache.org/jira/browse/HDFS-8101
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Affects Versions: 2.7.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Minor


Previously, all references to DFSConfigKeys in DFSClient were compile time 
constants which meant that normal users of DFSClient wouldn't resolve 
DFSConfigKeys at run time. As of HDFS-7718, DFSClient has a reference to a 
member of DFSConfigKeys that isn't compile time constant 
(DFS_CLIENT_KEY_PROVIDER_CACHE_EXPIRY_DEFAULT).

Since the class must be resolved now, this particular member

{code}
public static final String  DFS_WEBHDFS_AUTHENTICATION_FILTER_DEFAULT = 
AuthFilter.class.getName();
{code}

means that javax.servlet.Filter needs to be on the classpath.

javax-servlet-api is one of the properly listed dependencies for HDFS, however 
if we replace {{AuthFilter.class.getName()}} with the equivalent String literal 
then downstream folks can avoid including it while maintaining compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HDFS-8332) DFS client API calls should check filesystem closed

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reopened HDFS-8332:
---

According to git-bisect, this is the commit that started failing the following 
tests in hadoop-hdfs-httpfs on trunk:

* TestHttpFSFWithSWebhdfsFileSystem
* TestHttpFSWithHttpFSFileSystem
* TestHttpFSFWithWebhdfsFileSystem

{code}
$ git bisect start trunk b46c2bb51ae524e6640756620f70e5925cda7592
Bisecting: 272 revisions left to test after this (roughly 8 steps)
[baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6] HDFS-4448. Allow HA NN to start in 
secure mode with wildcard address configured (atm via asuresh)
$ git bisect run mvn -Dtest=TestHttpFSF*,TestHttpFSWithHttpFSFileSystem clean 
package
...SNIP...
e16f4b7f70b8675760cf5aaa471dfe29d48041e6 is the first bad commit
commit e16f4b7f70b8675760cf5aaa471dfe29d48041e6
Author: Uma Maheswara Rao G 
Date:   Fri May 8 12:26:47 2015 +0530

HDFS-8332. DFS client API calls should check filesystem closed. Contributed 
by Rakesh R.

:04 04 db7a6b4555c1bd18e8fe0a97a6689f7cf9ce15ec 
f9e0818f6198fbc0ac94b2d82bef7f065a90cc03 M  hadoop-hdfs-project
bisect run success
{code}

> DFS client API calls should check filesystem closed
> ---
>
> Key: HDFS-8332
> URL: https://issues.apache.org/jira/browse/HDFS-8332
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: 2.8.0
>
> Attachments: HDFS-8332-000.patch, HDFS-8332-001.patch, 
> HDFS-8332-002-Branch-2.patch, HDFS-8332-002.patch, 
> HDFS-8332.001.branch-2.patch
>
>
> I could see {{listCacheDirectives()}} and {{listCachePools()}} APIs can be 
> called even after the filesystem close. Instead these calls should do 
> {{checkOpen}} and throws:
> {code}
> java.io.IOException: Filesystem closed
>   at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:464)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HDFS-8135) Remove the deprecated FSConstants class

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reopened HDFS-8135:
---

This change breaks HBase.

The comment at the start of the removed class was 

{code}
- * @deprecated Please use {@link HdfsConstants}. This class
- * is left only for other ecosystem projects which depended on
- * it for SafemodeAction and DatanodeReport types.
- */
{code}

A few things

# please mark this change as breaking and include a release note, since hte 
javadocs expressly say it was there for ecosystem projecs (even though it does 
not carry a proper InterfaceAudience annotation)
# consider limiting this change to trunk and leave it out of branch-2
# HdfsConstants is labeled InterfaceAudience.Private, so what am I supposed to 
move HBase to?

> Remove the deprecated FSConstants class
> ---
>
> Key: HDFS-8135
> URL: https://issues.apache.org/jira/browse/HDFS-8135
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Li Lu
> Fix For: 2.8.0
>
> Attachments: HDFS-8135-041315.patch
>
>
> The {{FSConstants}} class has been marked as deprecated since 0.23. There is 
> no uses of this class in the current code base and it can be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-10767) downgrade from 2.7.2 to 2.5.0

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HDFS-10767.

Resolution: Invalid

> downgrade from 2.7.2 to 2.5.0
> -
>
> Key: HDFS-10767
> URL: https://issues.apache.org/jira/browse/HDFS-10767
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.5.0
> Environment: hdfs 2.5.0 2.7.2
>Reporter: jin xing
>
> I have already upgrade my cluster’s namenodes(with one stand by for HA) and 
> several datanodes from 2.5.0 folloing 
> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Downgrade_and_Rollback;
>  
> I take following steps:
> 1. hdfs dfsadmin -rollingUpgrade prepare;
> 2. hdfs dfsadmin -rollingUpgrade query;
> 3. hdfs dfsadmin -shutdownDatanode  upgrade
> 4. restart and upgrade datanode;
> However, I terminated the upgrade by mistake with command "hfs dfsadmin 
> -rollingUpgrade finalize"
> Currently, I have two 2.7.2 nematodes, and three 2.7.2 datanodes and 63 2.5.0 
> datanodes; Now I want to downgrade the nematodes and datanodes from 2.7.2 
> back to 2.5.0;
> But when I try to downgrade nematode and restart with “-rollingUpgrade 
> downgrade”, namenode cannot get started, I get rolling exception:
> 2016-08-16 20:37:08,642 WARN 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception 
> loading fsimage
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected 
> version of storage directory /home/maintain/hadoop/data/hdfs-namenode. 
> Reported: -63. Expecting = -57.
> at 
> org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
> at 
> org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
> at 
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:608)
> at 
> org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:323)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:751)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:735)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473)
> 2016-08-16 20:37:08,645 INFO org.mortbay.log: Stopped 
> HttpServer2$SelectChannelConnectorWithSafeStartup@dx-pipe-sata61-pm:50070
> 2016-08-16 20:37:08,745 INFO 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics 
> system...
> 2016-08-16 20:37:08,746 INFO 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system 
> stopped.
> 2016-08-16 20:37:08,746 INFO 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system 
> shutdown complete.
> 2016-08-16 20:37:08,746 FATAL 
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected 
> version of storage directory /home/maintain/hadoop/data/hdfs-namenode. 
> Reported: -63. Expecting = -57.
> at 
> org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
> at 
> org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
> at 
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:608)
> at 
> org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:323)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955)
> at 
> org