cmake

[Colin McCabe]

Hi all,

We'd like to use CMake instead of autotools to build native (C/C++) code in
Hadoop.  There are a lot of reasons to want to do this.  For one thing, it is
not feasible to use autotools on the Windows platform, because it depends on
UNIX shell scripts, the m4 macro processor, and some other pieces of
infrastructure which are not present on Windows.

For another thing, CMake builds are substantially simpler and faster, because
there is only one layer of generated code.  With autotools, you have automake
generating m4 code which autoconf reads, which it uses to generate a UNIX shell
script, which then generates another UNIX shell script, which eventually
generates Makefiles.  CMake simply generates Makefiles out of CMakeLists.txt
files-- much simpler to understand and debug, and much faster.
CMake is a lot easier to learn.

automake error messages can be very, very confusing.  This is because you are
essentially debugging a pile of shell scripts and macros, rather than a
coherent whole.  So you see error messages like "autoreconf: cannot empty
/tmp/ar0.4849: Is a directory" or "Can't locate object method "path" via
package "Autom4te... and so forth.  CMake error messages come from the CMake
application and they almost always immediately point you to the problem.

>From a build point of view, the net result of adopting CMake would be that you
would no longer need automake and related programs installed to build the
native parts of Hadoop.  Instead, you would need CMake installed.  CMake is
packaged by Red Hat, even in RHEL5, so it shouldn't be difficult to install
locally.  It's also available for Mac OS X and Windows, as I mentioned earlier.

The JIRA for this work is at https://issues.apache.org/jira/browse/HADOOP-8368
Thanks for reading.

sincerely,
Colin

Re: test-patch and native code compile

[Colin McCabe]

We could also call "uname" from test-patch.sh and skip running native
tests on Mac OS X.

I also think that HADOOP-7147 should be open rather than "won't fix,"
as Alejandro commented.  Allen Wittenauer closed it as "won't fix"
because he personally did not intend to fix it, but that doesn't mean
it's not a bug.

cheers.
Colin


On Thu, Sep 6, 2012 at 8:29 AM, Eli Collins  wrote:
> Yea we want jenkins to run with native. How about adding making native
> optional in test-patch via a flag and updating the jenkins jobs to use
> it?
>
> On Thu, Sep 6, 2012 at 7:25 AM, Alejandro Abdelnur  wrote:
>> Makes sense, though the Jenkins runs should continue to run w/ native, right?
>>
>> On Thu, Sep 6, 2012 at 12:49 AM, Hemanth Yamijala  wrote:
>>> Hi,
>>>
>>> The test-patch script in Hadoop source runs a native compile with the
>>> patch. On platforms like MAC, there are issues with the native
>>> compile. For e.g we run into HADOOP-7147 that has been resolved as
>>> Won't fix.
>>>
>>> Hence, should we have a switch in test-patch to not run native compile
>>> ? Could open a JIRA and fix, if that's OK ?
>>>
>>> Thanks
>>> hemanth
>>
>>
>>
>> --
>> Alejandro

Re: HADOOP-8887 port to branch-2

[Colin McCabe]

Hi Andrew,

It seems that your attachment did not appear on the mailing list.

I'm taking a look at the branch-2 build presently.

Colin


On Thu, Oct 18, 2012 at 6:20 PM, Andrew Purtell  wrote:
> The port of HADOOP-8887 to branch-2 fails the build. Please kindly see
> attached.
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: HADOOP-8887 port to branch-2

[Colin McCabe]

Hmm.  It did build for me (in the sense of completing successfully, etc).

You've definitely found an issue, though.  The version should be
2.0.3-SNAPSHOT for those projects, not 3.0.0-SNAPSHOT.

Do you want to file a JIRA and post this patch, or should I?

cheers.
Colin


On Thu, Oct 18, 2012 at 6:46 PM, Andrew Purtell  wrote:
> Sorry, I always forget that about the mailing list software.
>
> From a6efe4883f3f7694ec6e8da04de982fb21eed8a9 Mon Sep 17 00:00:00 2001
> From: Andrew Purtell 
> Date: Thu, 18 Oct 2012 17:22:17 -0700
> Subject: [PATCH] Fix incorrect versions and missing module parent introduced
>  by HADOOP-8887
>
> ---
>  dev-support/cmake-maven-ng-plugin/pom.xml |7 ++-
>  dev-support/pom.xml   |4 ++--
>  hadoop-project/pom.xml|2 +-
>  3 files changed, 9 insertions(+), 4 deletions(-)
>
> diff --git a/dev-support/cmake-maven-ng-plugin/pom.xml
> b/dev-support/cmake-maven-ng-plugin/pom.xml
> index e3d54bc..8652214 100644
> --- a/dev-support/cmake-maven-ng-plugin/pom.xml
> +++ b/dev-support/cmake-maven-ng-plugin/pom.xml
> @@ -17,10 +17,15 @@
>xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>http://maven.apache.org/maven-v4_0_0.xsd";>
>4.0.0
> +  
> +org.apache.hadoop
> +dev-support
> +2.0.3-SNAPSHOT
> +  
>org.apache.hadoop.cmake.maven.ng
>cmake-ng
>maven-plugin
> -  3.0.0-SNAPSHOT
> +  2.0.3-SNAPSHOT
>cmake-ng Maven Mojo
>http://maven.apache.org
>
> diff --git a/dev-support/pom.xml b/dev-support/pom.xml
> index 721e231..11a96cc 100644
> --- a/dev-support/pom.xml
> +++ b/dev-support/pom.xml
> @@ -20,12 +20,12 @@
>
>  org.apache.hadoop
>  hadoop-project
> -3.0.0-SNAPSHOT
> +2.0.3-SNAPSHOT
>  ../hadoop-project
>
>org.apache.hadoop
>dev-support
> -  3.0.0-SNAPSHOT
> +  2.0.3-SNAPSHOT
>Apache Hadoop Development Support
>Apache Hadoop Development Support
>pom
> diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
> index 568a59d..88d4676 100644
> --- a/hadoop-project/pom.xml
> +++ b/hadoop-project/pom.xml
> @@ -704,7 +704,7 @@
>  
>org.apache.hadoop.cmake.maven.ng
>cmake-ng
> -  3.0.0-SNAPSHOT
> +  2.0.3-SNAPSHOT
>  
>  
>org.apache.maven.plugins
> --
> 1.7.9.5
>
>
> On Thu, Oct 18, 2012 at 6:35 PM, Colin McCabe wrote:
>
>> Hi Andrew,
>>
>> It seems that your attachment did not appear on the mailing list.
>>
>> I'm taking a look at the branch-2 build presently.
>>
>> Colin
>>
>>
>> On Thu, Oct 18, 2012 at 6:20 PM, Andrew Purtell 
>> wrote:
>> > The port of HADOOP-8887 to branch-2 fails the build. Please kindly see
>> > attached.
>> >
>> > --
>> > Best regards,
>> >
>> >- Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>> >
>>
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Re: libhdfs on windows

[Colin McCabe]

Hi Peter,

This might be a good question for hdfs-dev?

As Harsh pointed out below, HDFS-573 was never committed.  I don't
even see a patch attached, although there is some discussion.

In the mean time, might I suggest using the webhdfs interface on
Windows?  webhdfs was intended as a stable REST interface that can be
accessed from any platform.

cheers,
Colin


On Thu, Oct 25, 2012 at 7:19 AM, Peter Marron
 wrote:
> Hi,
>
> I've been looking at using libhdfs and I would like to use it on windows.
> I have found HDFS-573 and the information on this page:
> http://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> which suggests that quite a lot of work was done on this way back in 2009.
> So is there some source from this effort retained somewhere? If so, where?
> Or do I have to start from scratch? Apologies if this has already been asked 
> recently.
>
> Any help appreciated.
>
> Peter Marron

Re: [VOTE] introduce Python as build-time and run-time dependency for Hadoop and throughout Hadoop stack

[Colin McCabe]

Nonbinding, but:

+1, +1, 0.

Also, let's please clearly define the versions of Python we support if
we do chooes to go this route.  Something like 2.4+ would be
reasonable.  The process launching APIs in particular changed a lot in
those early 2.x releases.

best,
Colin


On Sat, Nov 24, 2012 at 12:13 PM, Matt Foley  wrote:
> For discussion, please see previous thread "[PROPOSAL] introduce Python as
> build-time and run-time dependency for Hadoop and throughout Hadoop stack".
>
> This vote consists of three separate items:
>
> 1. Contributors shall be allowed to use Python as a platform-independent
> scripting language for build-time tasks, and add Python as a build-time
> dependency.
> Please vote +1, 0, -1.
>
> 2. Contributors shall be encouraged to use Maven tasks in combination with
> either plug-ins or Groovy scripts to do cross-platform build-time tasks,
> even under ant in Hadoop-1.
> Please vote +1, 0, -1.
>
> 3. Contributors shall be allowed to use Python as a platform-independent
> scripting language for run-time tasks, and add Python as a run-time
> dependency.
> Please vote +1, 0, -1.
>
> Note that voting -1 on #1 and +1 on #2 essentially REQUIRES contributors to
> use Maven plug-ins or Groovy as the only means of cross-platform build-time
> tasks, or to simply continue using platform-dependent scripts as is being
> done today.
>
> Vote closes at 12:30pm PST on Saturday 1 December.
> -
> Personally, my vote is +1, +1, +1.
> I think #2 is preferable to #1, but still has many unknowns in it, and
> until those are worked out I don't want to delay moving to cross-platform
> scripts for build-time tasks.
>
> Best regards,
> --Matt

Re: trailing whitespace

[Colin McCabe]

Some projects have a subversion pre or post-commit hook to check for
and/or correct these kinds of issues.

Inconsistent line endings (CRLF/LF), bogus tabs, and extra whitespace
are all things we could check for and/or correct this way.

Colin


On Mon, Nov 26, 2012 at 9:44 AM, Aaron T. Myers  wrote:
> On Sun, Nov 25, 2012 at 11:14 AM, Radim Kolar  wrote:
>
>> I propose addition to QA bot - check for trailing whitespace in patches.
>> Probably checking for tabs in Java files would be also good idea.
>>
>>
> Checking for tabs I could get behind; checking for trailing whitespace not
> so much.
>
> I've never understood why folks get worked up over a little trailing
> whitespace here and there, since you can't see it and it doesn't affect
> correctness. Spurious whitespace changes that make a review harder - those
> are annoying. Trailing whitespace inadvertently left on lines where
> legitimate changes were made in a patch - doesn't seem too harmful to me.
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera

Re: [VOTE] Hadoop Release 1.1.1

[Colin McCabe]

+1, nonbinding.

I installed it and ran TestDFSIO and a few other tests.  Looks good to me.

best,
Colin


On Tue, Nov 20, 2012 at 2:07 PM, Matt Foley  wrote:
> Hello,
> Hadoop-1.1.1-rc0 is now available for evaluation and vote:
> http://people.apache.org/~mattf/hadoop-1.1.1-rc0/
> or in the Nexus repository.
>
> The release notes are available at
> http://people.apache.org/~mattf/hadoop-1.1.1-rc0/releasenotes.html
>
> 20 bugs have been fixed, compared to release 1.1.0, with no backward
> incompatibilities.
> I took the opportunity to assure that all branch-1.0 changes are in 1.1.1,
> and all branch-1.1 changes are in branch-1.
> The jira database has been made consistent.
>
> Please vote.  Voting will end on Tuesday 27 Nov., at 2:05pm PST.
>
> Thank you,
> --Matt Foley
> release manager

Re: [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')

[Colin McCabe]

I think this is a good idea, for a few different reasons.

* We all know Java, so this code will be readable by all.
* It will benefit the wider community of Maven users, not just our project.
* It gets rid of the shell dependency.  The shell dependency is
problematic because Windows doesn't support most UNIX shell code, and
also because different Linux distributions have different default
shells.  The /bin/dash versus /bin/bash issue has caused problems in
the past.

Is there any way this code could be used in branch-1 to solve some of
the issues there?  I know the protoc code, at least, would be useful
there.

regards,
Colin


On Wed, Dec 5, 2012 at 9:24 AM, Alejandro Abdelnur  wrote:
> On the '[VOTE] introduce Python as build-time and run-time dependency for
> Hadoop and throughout Hadoop stack' thread the vote for using Maven plugins
> for the build had an overwhelming acceptance (several +1s, a few 0s and no
> -1s).
>
> Current tasks of the build for being 'pluginized' are:
>
> 1* cmake (HADOOP-8887)
> 2* protoc(HADOOP-9117)
> 3* saveVersion (HADOOP-8924) (but this one has been -1)
>
> I've tested #2 and #3 in Windows an the worked as expected. Regarding #1,
> if cmake is supported (and used for hadoop) in Window, i believe it should
> work with minimal changes.
>
> As I mentioned in HADOOP-8924, "Writing Maven plugins is more complex that
> writing scripts, I don't dispute this. The main motivation for using Maven
> plugins is to keep things in the POM declarative and to hide (if necessary)
> different handlings for different platforms."
>
> In order to enable the use of custom plugins for the Hadoop build, we need
> to have such plugins available in a Maven repo for Maven to download and
> use them. Note that they cannot be part of the same build that is building
> Hadoop itself.
>
> While we could argue this plugins are general purpose and they should be
> developed in a different standalone project, doing this would definitely
> slow down when we can use them in Hadoop.
>
> Because of this, I propose we create a Hadoop subproject 'hadoop-build' to
> host custom Maven plugins (and custom Ant tasks for branch-1) for Hadoop.
>
> Being a sub-project will allow us to do independent releases (from Hadoop
> releases) of plugins  we need for Hadoop build. Plus, use them immediately
> in the Hadoop build.
>
> Eventually, if the Maven community picks up some of these plugins we could
> simply remove them from our hadoop-build project and change Hadoop POMs to
> use the new ones.
>
> Looking forward to hear what others think.
>
> Thanks.
>
> --
> Alejandro

Re: [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')

[Colin McCabe]

On Fri, Dec 7, 2012 at 5:31 PM, Radim Kolar  wrote:
> 1. cmake and protoc maven plugins already exists. why you want to write a
> new ones?

This has already been discussed; see
https://groups.google.com/forum/?fromgroups=#!topic/cmake-maven-project-users/5FpfUHmg5Ho

Actually the situation is even worse than it might seem from that
thread, since it turns out that com.googlecode.cmake-maven-project has
no support for any platforms but Windows.  It also has no support for
running native unit tests, which is a big motivation behind
HADOOP-8887.

> 2. Groovy accepts java syntax. Just rewrite saveVersion.sh to java (its done
> already in JIRA) and put it in pom.xml - no overhaul of build infrastructure
> needed.

Part of the reason for this thread is so that we can come up with a
solution for both branch-1 and later branches.  This would not be
accomplished by putting all the logic into a pom.xml file, since
branch-1 doesn't use Maven.

best,
Colin

Re: [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')

[Colin McCabe]

On Mon, Dec 10, 2012 at 10:50 AM, Colin McCabe  wrote:
> On Fri, Dec 7, 2012 at 5:31 PM, Radim Kolar  wrote:
>> 1. cmake and protoc maven plugins already exists. why you want to write a
>> new ones?
>
> This has already been discussed; see
> https://groups.google.com/forum/?fromgroups=#!topic/cmake-maven-project-users/5FpfUHmg5Ho
>
> Actually the situation is even worse than it might seem from that
> thread, since it turns out that com.googlecode.cmake-maven-project has
> no support for any platforms but Windows.  It also has no support for
> running native unit tests, which is a big motivation behind
> HADOOP-8887.

To clarify this a little bit, some of the later versions of the
googlecode plugin add rudimentary support for gcc, but I wasn't able
to get it to work locally.  I spent a lot of time on this when I did
the cmake conversion, and it didn't really pay off.

As I commented, the plugin from HADOOP-8887 also does a lot more,
including add the ability to support things like "mvn test
-Dtest=my_native_test".

cheers,
Colin

>
>> 2. Groovy accepts java syntax. Just rewrite saveVersion.sh to java (its done
>> already in JIRA) and put it in pom.xml - no overhaul of build infrastructure
>> needed.
>
> Part of the reason for this thread is so that we can come up with a
> solution for both branch-1 and later branches.  This would not be
> accomplished by putting all the logic into a pom.xml file, since
> branch-1 doesn't use Maven.
>
> best,
> Colin

Re: [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')

[Colin McCabe]

Hi Radim,

In general, Maven plugins are built and deployed to a repository.
Then, Maven fetches the precompiled binaries from this repository
based on a specific version number in the pom.  This is how Maven
plugins work in general. not specific to this proposal.

I did experiment with bundling the Maven plugins with the main Hadoop
source in HADOOP-8887; however, this is not really feasible.  Hence
the proposal for a separate sub-project here.

So basically the summary is the Maven workflow wouldn't change.

Hope this helps.
Colin


On Tue, Dec 11, 2012 at 7:16 PM, Radim Kolar  wrote:
> what is proposed build workflow?
>
> Its enough to do mvn install to get custom plugins available to maven in
> build phase or you must setup own mvn repo and do mvn deploy?

Re: making a hadoop-common test run if a property is set

[Colin McCabe]

One approach we've taken in the past is making the junit test skip
itself when some precondition is not true.  Then, we often create a
property which people can use to cause the skipped tests to become a
hard error.

For example, all the tests that rely on libhadoop start with these lines:

> @Test
> public void myTest() {
>Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
>   ...
> }

This causes them to be silently skipped when libhadoop.so is not
available or loaded (perhaps because it hasn't been built.)

However, if you want to cause this to be a hard error, you simply run
> mvn test -Drequire.test.libhadoop

See TestHdfsNativeCodeLoader.java to see how this is implemented.

The main idea is that your Jenkins build slaves use all the -Drequire
lines, but people running tests locally are not inconvenienced by the
need to build libhadoop.so in every case.  This is especially good
because libhadoop.so isn't known to build on certain platforms like
AIX, etc.  It seems to be a good tradeoff so far.  I imagine that s3
could do something similar.

cheers,
Colin


On Fri, Dec 14, 2012 at 9:56 AM, Steve Loughran  wrote:
> The swiftfs tests need only to run if there's a target filesystem; copying
> the s3/s3n tests, something like
>
>   
> test.fs.swift.name
> swift://your-object-store-herel/
>   
>
> How does one actually go about making junit tests optional in mvn-land?
> Should the probe/skip logic be in the code -which can make people think the
> test passed when it didn't actually run? Or can I turn it on/off in maven?
>
> -steve

Re: making a hadoop-common test run if a property is set

[Colin McCabe]

On Mon, Dec 17, 2012 at 11:03 AM, Steve Loughran  wrote:
> On 17 December 2012 16:06, Tom White  wrote:
>
>> There are some tests like the S3 tests that end with "Test" (e.g.
>> Jets3tNativeS3FileSystemContractTest) - unlike normal tests which
>> start with "Test". Only those that start with "Test" are run
>> automatically (see the surefire configuration in
>> hadoop-project/pom.xml). You have to run the others manually with "mvn
>> test -Dtest=...".
>>
>> The mechanism that Colin describes is probably better though, since
>> the environment-specific tests can be run as a part of a full test run
>> by Jenkins if configured appropriately.
>>
>
> I'd like that -though one problem with the current system is that you need
> to get the s3 (and soon: openstack) credentials into
> src/test/resources/core-site.xml, which isn't the right approach. If we
> could get them into properties files things would be easier.
> That's overkill for adding a few more openstack tests -but I would like to
> make it easier to turn those and the rackspace ones without sticking my
> secrets into an XML file under SCM

I think the way to go is to have one XML file include another.



http://www.w3.org/2001/XInclude";>
  
 boring.config.1
 boring-value
 ... etc, etc...
  


That way, you can keep the boring configuration under version control,
and still have your password sitting in a small separate
non-version-controlled XML file.

We use this trick a bunch with the HA configuration stuff-- 99% of the
configuration is the same between the Active and Standby Namenodes,
but you can't give them the same dfs.ha.namenode.id or dfs.name.dir.
Includes help a lot here.

> another tactic could be to have specific test projects: test-s3,
> test-openstack, test-... which contain nothing but test cases. You'd set
> jenkins up those test projects too -the reason for having the separate
> names is to make it blatantly clear which tests you've not run

I dunno.  Every time a project puts unit or system tests into a
separate project, the developers never run them.  I've seen it happen
enough times that I think I can call it an anti-pattern by now.  I
like having tests alongside the code-- to the maximum extent that is
possible.

cheers,
Colin

>
>
>
>> Tom
>>
>> On Mon, Dec 17, 2012 at 10:06 AM, Steve Loughran 
>> wrote:
>> > thanks, I'l; have a look. I've always wanted to add the notion of skipped
>> > to test runs -all the way through to the XML and generated reports, but
>> > you'd have to do a new junit runner for this and tweak the reporting
>> code.
>> > Which, if it involved going near maven source, is not something I am
>> > prepared to do
>> >
>> > On 14 December 2012 18:57, Colin McCabe  wrote:
>> >
>> >> One approach we've taken in the past is making the junit test skip
>> >> itself when some precondition is not true.  Then, we often create a
>> >> property which people can use to cause the skipped tests to become a
>> >> hard error.
>> >>
>> >> For example, all the tests that rely on libhadoop start with these
>> lines:
>> >>
>> >> > @Test
>> >> > public void myTest() {
>> >> >Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
>> >> >   ...
>> >> > }
>> >>
>> >> This causes them to be silently skipped when libhadoop.so is not
>> >> available or loaded (perhaps because it hasn't been built.)
>> >>
>> >> However, if you want to cause this to be a hard error, you simply run
>> >> > mvn test -Drequire.test.libhadoop
>> >>
>> >> See TestHdfsNativeCodeLoader.java to see how this is implemented.
>> >>
>> >> The main idea is that your Jenkins build slaves use all the -Drequire
>> >> lines, but people running tests locally are not inconvenienced by the
>> >> need to build libhadoop.so in every case.  This is especially good
>> >> because libhadoop.so isn't known to build on certain platforms like
>> >> AIX, etc.  It seems to be a good tradeoff so far.  I imagine that s3
>> >> could do something similar.
>> >>
>> >> cheers,
>> >> Colin
>> >>
>> >>
>> >> On Fri, Dec 14, 2012 at 9:56 AM, Steve Loughran > >
>> >> wrote:
>> >> > The swiftfs tests need only to run if there's a target filesystem;
>> >> copying
>> >> > the s3/s3n tests, something like
>> >> >
>> >> >   
>> >> > test.fs.swift.name
>> >> > swift://your-object-store-herel/
>> >> >   
>> >> >
>> >> > How does one actually go about making junit tests optional in
>> mvn-land?
>> >> > Should the probe/skip logic be in the code -which can make people
>> think
>> >> the
>> >> > test passed when it didn't actually run? Or can I turn it on/off in
>> >> maven?
>> >> >
>> >> > -steve
>> >>
>>

Re: making a hadoop-common test run if a property is set

[Colin McCabe]

On Tue, Dec 18, 2012 at 1:05 AM, Colin McCabe  wrote:
> On Mon, Dec 17, 2012 at 11:03 AM, Steve Loughran  
> wrote:
>> On 17 December 2012 16:06, Tom White  wrote:
>>
>>> There are some tests like the S3 tests that end with "Test" (e.g.
>>> Jets3tNativeS3FileSystemContractTest) - unlike normal tests which
>>> start with "Test". Only those that start with "Test" are run
>>> automatically (see the surefire configuration in
>>> hadoop-project/pom.xml). You have to run the others manually with "mvn
>>> test -Dtest=...".
>>>
>>> The mechanism that Colin describes is probably better though, since
>>> the environment-specific tests can be run as a part of a full test run
>>> by Jenkins if configured appropriately.
>>>
>>
>> I'd like that -though one problem with the current system is that you need
>> to get the s3 (and soon: openstack) credentials into
>> src/test/resources/core-site.xml, which isn't the right approach. If we
>> could get them into properties files things would be easier.
>> That's overkill for adding a few more openstack tests -but I would like to
>> make it easier to turn those and the rackspace ones without sticking my
>> secrets into an XML file under SCM
>
> I think the way to go is to have one XML file include another.
>
> 
> 
> http://www.w3.org/2001/XInclude";>
>   
>  boring.config.1
>  boring-value
>  ... etc, etc...
>   
> 
>
> That way, you can keep the boring configuration under version control,
> and still have your password sitting in a small separate
> non-version-controlled XML file.
>
> We use this trick a bunch with the HA configuration stuff-- 99% of the
> configuration is the same between the Active and Standby Namenodes,
> but you can't give them the same dfs.ha.namenode.id or dfs.name.dir.
> Includes help a lot here.
>
>> another tactic could be to have specific test projects: test-s3,
>> test-openstack, test-... which contain nothing but test cases. You'd set
>> jenkins up those test projects too -the reason for having the separate
>> names is to make it blatantly clear which tests you've not run
>
> I dunno.  Every time a project puts unit or system tests into a
> separate project, the developers never run them.  I've seen it happen
> enough times that I think I can call it an anti-pattern by now.  I
> like having tests alongside the code-- to the maximum extent that is
> possible.

Just to be clear, I'm not referring to any Hadoop-related project
here, just certain other open source (and not) ones I've worked on.
System/unit tests belong with the rest of the code, otherwise they get
stale real fast.

It sometimes makes sense for integration tests to live in a separate
repo, since by their nature they're usually talking to stuff that
lives in multiple repos.

best,
Colin

>
> cheers,
> Colin
>
>>
>>
>>
>>> Tom
>>>
>>> On Mon, Dec 17, 2012 at 10:06 AM, Steve Loughran 
>>> wrote:
>>> > thanks, I'l; have a look. I've always wanted to add the notion of skipped
>>> > to test runs -all the way through to the XML and generated reports, but
>>> > you'd have to do a new junit runner for this and tweak the reporting
>>> code.
>>> > Which, if it involved going near maven source, is not something I am
>>> > prepared to do
>>> >
>>> > On 14 December 2012 18:57, Colin McCabe  wrote:
>>> >
>>> >> One approach we've taken in the past is making the junit test skip
>>> >> itself when some precondition is not true.  Then, we often create a
>>> >> property which people can use to cause the skipped tests to become a
>>> >> hard error.
>>> >>
>>> >> For example, all the tests that rely on libhadoop start with these
>>> lines:
>>> >>
>>> >> > @Test
>>> >> > public void myTest() {
>>> >> >Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
>>> >> >   ...
>>> >> > }
>>> >>
>>> >> This causes them to be silently skipped when libhadoop.so is not
>>> >> available or loaded (perhaps because it hasn't been built.)
>>> >>
>>> >> However, if you want to cause this to be a hard error, you simply run
>>> >> > mvn test -Drequire.test.libhadoop
>>> >>
>>> >> See TestHdfsNativeCodeLoader.java to see how th

Re: Hadoop build slaves software

[Colin McCabe]

In addition to protoc, can someone please also install a 32-bit C++ compiler?

The builds are all failing on this machine because of that.

regards,
Colin


On Fri, Jan 4, 2013 at 11:37 AM, Giridharan Kesavan
 wrote:
> When I configured the other machines I used the source to compile and
> install the protoc, as the 2.4.1 wasn't available in the ubuntu repo.
>
> BTW installed 2.4.1 on asf008.
> gkesavan@asf008:~$ protoc --version
> libprotoc 2.4.1
>
>
> -Giri
>
>
> On Thu, Jan 3, 2013 at 11:24 PM, Todd Lipcon  wrote:
>
>> Hey folks,
>>
>> It looks like hadoop8 has recently come back online as a build slave,
>> but is failing all the builds because it has an ancient version of
>> protobuf (2.2.0):
>> todd@asf008:~$ protoc  --version
>> libprotoc 2.2.0
>>
>> In contrast, other slaves have 2.4.1:
>> todd@asf001:~$ protoc --version
>> libprotoc 2.4.1
>>
>> asf001 has the newer protoc in /usr/local/bin but asf008 does not.
>> Does anyone know how software is meant to be deployed on these build
>> slaves? I'm happy to download and install protobuf 2.4.1 into
>> /usr/local on asf008 if manual installation is the name of the game,
>> but it seems like we should be doing something a little more
>> reproducible than one-off builds by rando developers to manage our
>> toolchain on the Jenkins slaves.
>>
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>

Re: maven setting of version of protocol buffer

[Colin McCabe]

Hi Yiyu,

Are you referring to com.google.protobuf?

We generally do depend on specific versions of jars in the pom.xml files,
to prevent exactly this sort of problem.  If you have a patch which adds
this, you should post it.  It might help someone else.

cheers,
Colin


On Thu, Jan 17, 2013 at 6:34 AM, yiyu jia  wrote:

> Hi,
>
> I noticed that I always get error message which says there is no
> Parser.class found in protocbuffer. Aftter checking, I found that is
> because there is no version info about protocbuff in POM file. So, Maven
> always try to download old version of protocbuffer though I build the
> lastest one and install it in local maven repository already. And the
> default version Maven downloaded has no Parser.class .
>
> I solve the issue by adding version info of protocbuffer in POM files.
> However, I think there should be a better way to solve this. anybody can
> show me how to get reqired version of protocbuff without changing POM
> files?
>
> thanks and reggards,
>
> Yiyu
>

Re: development environment for hadoop core

[Colin McCabe]

Hi Erik,

Eclipse can run junit tests very rapidly.  If you want a shorter test
cycle, that's one way to get it.

There is also Maven-shell, which reduces some of the overhead of starting
Maven.  But I haven't used it so I can't really comment.

cheers,
Colin


On Mon, Jan 21, 2013 at 8:36 AM, Erik Paulson  wrote:

> On Wed, Jan 16, 2013 at 7:31 AM, Glen Mazza  wrote:
>
> > On 01/15/2013 06:50 PM, Erik Paulson wrote:
> >
> >> Hello -
> >>
> >> I'm curious what Hadoop developers use for their day-to-day hacking on
> >> Hadoop. I'm talking changes to the Hadoop libraries and daemons, and not
> >> developing Map-Reduce jobs or using using the HDFS Client libraries to
> >> talk
> >> to a filesystem from an application.
> >>
> >> I've checked out Hadoop, made minor changes and built it with Maven, and
> >> tracked down the resulting artifacts in a target/ directory that I could
> >> deploy. Is this typically how a cloudera/hortonworks/mapr/etc dev works,
> >> or
> >> are the IDEs more common?
> >>
> > I haven't built Hadoop yet myself.  Your use of "a" in "a target/
> > directory" indicates you're also kind of new with Maven itself, as that's
> > the standard output folder for any Maven project.  One of many nice
> things
> > about Maven is once you learn how to build one project with it you pretty
> > much know how to build any project with it, as everything's standardized
> > with it.
> >
> > Probably best to stick with the command line for building and use Eclipse
> > for editing, to keep things simple, but don't forget the mvn
> > eclipse:eclipse command to set up Eclipse projects that you can
> > subsequently import into your Eclipse IDE:
> http://www.jroller.com/gmazza/*
> > *entry/web_service_tutorial#**EclipseSetup<
> http://www.jroller.com/gmazza/entry/web_service_tutorial#EclipseSetup>
> >
> >
> >
> >> I realize this sort of sounds like a dumb question, but I'm mostly
> curious
> >> what I might be missing out on if I stay away from anything other than
> >> vim,
> >> and not being entirely sure where maven might be caching jars that it
> uses
> >> to build,
> >>
> >
> > That will be your local Maven repository, in an .m2 hidden folder in your
> > user home directory.
> >
> >
> >
> >  and how careful I have to be to ensure that my changes wind up in
> >> the right places without having to do a clean build every time.
> >>
> >>
> > Maven can detect changes (using mvn install instead of mvn clean
> install),
> > but I prefer doing clean builds.  You can use the -Dmaven.test.skip
> setting
> > to speed up your "mvn clean installs" if you don't wish to run the tests
> > each time.
> >
>
> Thanks to everyone for their advice last week, it's been helpful.
>
> You're spot-on that I'm new to Maven, but I'm a little confused as to what
> the different targets/goals are best to use. Here's my scenario.
>
> What I'd like to get working is the DataNodeCluster, which lives in the
> tests.
>
> Running it from hadoop-hdfs-project/hadoop-hdfs/target as
> 'hadoop jar ./hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar
> org.apache.hadoop.hdfs.DataNodeCluster
> -n 2'
>
> blows up with a NPE inside of MiniDFSCluster - the offending line is
> 'dfsdir = conf.get(HDFS_MINIDFS_BASEDIR, null);' (line 2078 of
> MiniDFSCluster.java)
>
> I'm not worried about being able to figure out what's wrong (I'm pretty
> sure it's that conf is still null when this gets called) - what I'm trying
> to use this as is a way to understand what gets built when.
>
> Just to check, I added a System.out.println one line before 2078 of
> MiniDFSCluster, and recompiled from hadoop-common/hadoop-hdfs-project with
>
> mvn package -DskipTests
>
> Because I don't want to run all the tests.
>
> This certainly compiles the codes - if I leave the semicolon off of my
> change the compile fails, even with -DskipTests. However, it doesn't appear
> to rebuild
>
> target/hadoop-hdfs-3.0.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar
> - the timestamp is still the old version.
>
> It _does_ copy
>
> target/hadoop-hdfs-3.0.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar
> to target/hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar, or at least otherwise
> update the timestamp on target/hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar (unless
> it's copying or building it from somewhere else - but if it is, it's
> picking up old versions of my code)
>
> I only get an updated version if I ask for
> mvn package -Pdist -DskipTests
>
> Which is a 3 minute rebuild cycle, even for something as simple as changing
> the text in my System.out.println. (Even just a mvn package -DskipTests
> with no changes to any source code is a 40 second operation)
>
> I haven't sat around and waited for 'mvn package' to run and fire off the
> test suite. I don't know if that would result in an updated
> hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar
> being built.
>
> So, my question is:
>
> - Is there a better maven target to use if I just want to update code in
> MiniDFSCluster.java and run DataNo

Re: [VOTE] Plan to create release candidate Monday 3/18

[Colin McCabe]

+1 (non-binding)

best,
Colin


On Sun, Mar 10, 2013 at 8:38 PM, Matt Foley  wrote:

> Hi all,
> I have created branch-1.2 from branch-1, and propose to cut the first
> release candidate for 1.2.0 on Monday 3/18 (a week from tomorrow), or as
> soon thereafter as I can achieve a stable build.
>
> Between 1.1.2 and the current 1.2.0, there are 176 patches!!  Draft release
> notes are available at .../branch-1.2/src/docs/releasenotes.html in the
> sources.
>
> Any non-destabilizing patches committed to branch-1.2 during the coming
> week (and of course also committed to branch-1) will be included in the RC.
>  However, at this point I request that any big new developments not yet in
> branch-1.2 be targeted for 1.3.
>
> Release plans have to be voted on too, so please vote '+1' to approve this
> plan.  Voting will close on Sunday 3/17 at 8:30pm PDT.
>
> Thanks,
> --Matt
> (release manager)
>

Re: [DISCUSS] - Committing client code to 3rd Party FileSystems within Hadoop Common

[Colin McCabe]

You might try looking at what KosmoFS (KFS) did.  They have some code in
org/apache/hadoop/fs which calls their own Java shim.

This way, the shim code in hadoop-common gets updated whenever FileSystem
changes, but there is no requirement to install KFS before building Hadoop.

You might also try asking Steve Loughran, since he did some great work
recently to try to nail down the exact semantics of FileSystem and
FileContext and improve the related unit tests (see HADOOP-9258 and related
JIRAs.)

best,
Colin



On Thu, May 23, 2013 at 2:52 PM, Stephen Watt  wrote:

> Thanks for responding Harsh.
>
> I agree. Hadoop Common does do a good job of maintaining a stable and
> public FS and FS Context API. The pro for maintaining client libraries
> outside of Hadoop Common is that the release owner of the library has much
> more autonomy and agility in maintaining the library. From the glusterfs
> plugin perspective, I concur with this. In contrast, if my library was
> managed inside of Hadoop Common, I'd have to spend the time to earn
> committer status to have an equivalent amount of autonomy and agility,
> which is overkill for someone just wanting to maintain 400 lines of code.
>
> I ruminated a bit about one con which might be that because it doesn't get
> shipped with Hadoop Common it might make it harder for the Hadoop User
> community to find out about it and obtain it. However, if you consider the
> LZO codec, the fact that its not bundled certainly doesn't hamper its
> adoption.
>
> You mentioned testing. I don't think regression across Hadoop releases is
> as big of an issue as (based on my understanding) you really just have two
> FileSystem interfaces (abstract class) to worry about WRT to compliance,
> namely the FileSystem interface reflected for Hadoop 1.0 and the FileSystem
> interface reflected for Hadoop 2.0. However, this is a broader topic that I
> also want to discuss so I'll tee it up in a separate thread.
>
> Regards
> Steve Watt
>
>
> - Original Message -
> From: "Harsh J" 
> To: common-dev@hadoop.apache.org
> Sent: Thursday, May 23, 2013 1:37:30 PM
> Subject: Re: [DISCUSS] - Committing client code to 3rd Party FileSystems
> within Hadoop Common
>
> I think we do a fairly good work maintaining a stable and public FileSystem
> and FileContext API for third-party plugins to exist outside of Apache
> Hadoop but still be able to work well across versions.
>
> The question of test pops up though, specifically that of testing against
> trunk to catch regressions across various implementations, but it'd be much
> work for us to also maintain glusterfs dependencies and mechanisms as part
> of trunk.
>
> We do provide trunk build snapshot artifacts publicly for downstream
> projects to test against, which I think may help cover the continuous
> testing concerns, if there are those.
>
> Right now, I don't think the S3 FS we maintain really works all that well.
> I also recall, per recent conversations on the lists, that AMZN has started
> shipping their own library for a better implementation rather than
> perfecting the implementation we have here (correct me if am wrong but I
> think the changes were not all contributed back). I see some work going on
> for OpenStack's Swift, for which I think Steve also raised a similar
> discussion here: http://search-hadoop.com/m/W1S5h2SrxlG, but I don't
> recall
> if the conversation proceeded at the time.
>
> What's your perspective as the releaser though? Would you not find
> maintaining this outside easier, especially in terms of maintaining your
> code for quicker releases, for both bug fixes and features - also given
> that you can CI it against Apache Hadoop trunk at the same time?
>
>
> On Thu, May 23, 2013 at 11:47 PM, Stephen Watt  wrote:
>
> > (Resending - I think the first time I sent this out it got lost within
> all
> > the ByLaws voting)
> >
> > Hi Folks
> >
> > My name is Steve Watt and I am presently working on enabling glusterfs to
> > be used as a Hadoop FileSystem. Most of the work thus far has involved
> > developing a Hadoop FileSystem plugin for glusterfs. I'm getting to the
> > point where the plugin is becoming stable and I've been trying to
> > understand where the right place is to host/manage/version it.
> >
> > Steve Loughran was kind enough to point out a few past threads in the
> > community (such as
> >
> http://lucene.472066.n3.nabble.com/Need-to-add-fs-shim-to-use-QFS-td4012118.html
> )
> > that show a project disposition to move away from Hadoop Common
> containing
> > client code (plugins) for 3rd party FileSystems. This makes sense and
> > allows the filesystem plugin developer more autonomy as well as reduces
> > Hadoop Common's dependence on 3rd Party libraries.
> >
> > Before I embark down that path, can the PMC/Committers verify that the
> > preference is still to have client code for 3rd Party FileSystems hosted
> > and managed outside of Hadoop Common?
> >
> > Regards
> > Steve Watt
> >
>
>
>
> --
> Harsh J
>

Re: Branch policy question

[Colin McCabe]

If the underlying problem is lack of reviewers for these improvements,
how about a design doc giving some motivation for the improvements and
explaining how they'll be implemented?  Then we can decide if a branch
or a few JIRAs on trunk makes more sense.

The description for HADOOP-12857 is just "let's rework this to be
smarter" which could be the description of any JIRA in Hadoop (except
for the ones which make things dumber, of course).

best,
Colin


On Tue, Mar 22, 2016, at 11:23, Andrew Wang wrote:
> A branch sounds fine, but how are we going to get 3 +1's to merge it? If
> it's hard to find one reviewer, seems even harder to find two.
> 
> On Tue, Mar 22, 2016 at 10:56 AM, Allen Wittenauer <
> allenwittena...@yahoo.com.invalid> wrote:
> 
> >
> > > On Mar 22, 2016, at 10:49 AM, larry mccay  wrote:
> > >
> > > That sounds like a reasonable approach and valid use of branches to me.
> > >
> > > Perhaps a set of functional tests could be provided/identified that would
> > > help the review process by showing backward compatibility along with new
> > > extensions for things like dynamic commands?
> > >
> >
> > This is going into trunk, so no need for backward compatibility.
> >

Re: Branch policy question

[Colin McCabe]

> On 3/22/16, 11:03 PM, "Allen Wittenauer"
>  wrote:
> 
> >> On Mar 22, 2016, at 6:46 PM, Gangumalla, Uma 
> >>wrote:
> >> 
> >>> is it possible for me to setup a branch, self review+commit to that
> >>> branch, then request a branch merge?
> >> Basically this is something like Commit-Then-Review(here review later)
> >> process right. I have not seen we followed this approach here( not sure
> >>if
> >> I missed some branches followed that).
> >
> > At least in recent times, YARN-3368 attempted to do that with the merge
> >request found in YARN-4734. The first or second commit that went into
> >that branch clearly has never been reviewed by another committer.  Since
> >a PMC member did the commit and merge request, I thought I¹d ask since it
> >implies that the policy really is CTR for merge requests. (FWIW, I was
> >the lone -1 on the merge request, otherwise it likely would have gone in
> >given it already had 2 +1¹s despite a) breaking the build b) having
> >significant license issues and c) no design doc.)

Let me clarify that the policy is NOT CTR for merging branches back to
mainline.  It is RTC, with the added proviso that you need 3 +1 votes
from committers.  There is also a 5-day waiting period before you can
merge, so that people have time to chime in and possibly ask for changes
prior to the merge, or express a -1 vote.

I'm concerned that this discussion seems to be focusing on "how can I
use procedures and rules to avoid the need to get reviews from the
community" rather than "how can I get the community interested in giving
me reviews and feedback."

We still haven't spoken a word about what this feature or improvement
does, or why we would want it.  It is hard to say whether a feature
branch makes sense without that context.

best,
Colin

Re: Github integration for Hadoop

[Colin McCabe]

Yes, please.  Let's disable these mails.

C.

On Mon, Apr 4, 2016, at 06:21, Vinayakumar B wrote:
> bq. We don't spam common-dev about every time a new patch attachment
> gets posted
> to an existing JIRA.  We shouldn't do that for github either.
> 
> Is there any update on this. ?
> Any INFRA ticket filed to disable these mails?
> 
> Because, I am sure, people would have got frustrated by seeing mails
> generated by my recent PR submissions. And each time, when some comment
> gets added to PR.
> 
> -Vinay
> 
> On Tue, Nov 24, 2015 at 3:35 AM, Andrew Wang 
> wrote:
> 
> > Hi Eric,
> >
> > My comment wrt GH permissions is available in context earlier in the
> > thread, pasting again here:
> >
> > ===
> >
> > Great, glad to hear it. That wasn't mentioned in the email thread or on the
> > INFRA ticket, and the INFRA ticket mentions these integrations:
> >
> > Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull Request
> > Comment, and push, Push
> >
> > Are these the right set of permissions to disable integrating PRs? Namely,
> > the "push" permissions look unnecessary. We should also disable GH issues
> > since we don't want users filing issues there.
> >
> >
> > On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe 
> > wrote:
> >
> > > We don't spam common-dev about every time a new patch attachment gets
> > > posted to an existing JIRA.  We shouldn't do that for github either.
> > >
> > > +1 to Andrew and Steve's suggestion of disabling these emails (or at
> > > least sending them to a separate list that people can opt in to).
> > >
> > > best,
> > > Colin
> > >
> > > On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang  wrote:
> > > > Hi Andrew,
> > > >
> > > > Many Apache projects already have Github integration.  Chukwa also has
> > > > Github integration.  Pull request can be integrated into JIRA, i.e.
> > > > https://issues.apache.org/jira/browse/CHUKWA-783
> > > >
> > > > This allows code review process to happen at per line level on Github,
> > or
> > > > comment on JIRA for summary of the activities.  Micro comments are
> > squash
> > > > away.  The final commit is always happening on Apache git rather than
> > > > Github.  Therefore, there is no disabling required for pull request.
> > > > Primary activity is still on JIRA, and Github is only a supplement to
> > > make
> > > > line by line code review easy.  Overall user experience is better than
> > RB
> > > > or Gerrit in my opinion.
> > > >
> > > > regards,
> > > > Eric
> > > >
> > > > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang <
> > andrew.w...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Has there been any progress on locking down the Github permissions
> > like
> > > I
> > > >> asked above? It's been about 3 weeks.
> > > >>
> > > >> Steve also just asked on another thread to turn off the emails to
> > > >> common-dev. Since we're sticking with JIRA as the source-of-truth, the
> > > >> common-dev emails aren't useful.
> > > >>
> > > >> On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey 
> > > wrote:
> > > >>
> > > >> > Hi Colin!
> > > >> >
> > > >> > If Yetus is working on an issue and can't tell what the intended
> > > branch
> > > >> is
> > > >> > it points folks to project specific contribution guides.
> > > >> >
> > > >> > For Hadoop, the patch naming for specific branches should be covered
> > > in
> > > >> > this section of Hadoop's contribution guide:
> > > >> >
> > > >> > http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
> > > >> >
> > > >> > Yetus will actually support a little bit more than that guide
> > > suggests.
> > > >> If
> > > >> > a project doesn't define a URL to point people at for help in naming
> > > >> > patches we default to this guide:
> > > >> >
> > > >> > https://yetus.apache.org/documentation/latest/precommit-patchnames/
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Fri, Nov 13, 2015 at 8:05 PM, Colin P. McCabe <
> > cmcc...@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> > > Thanks, Allen, I wasn't aware that Yetus now supported testing for
> > > >> > > other branches.  Is there documentation about how to name the
> > branch
> > > >> > > so it gets tested?
> > > >> > >
> > > >> > > best,
> > > >> > > Colin
> > > >> > >
> > > >> > > On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer <
> > a...@altiscale.com
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > >> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe <
> > > cmcc...@apache.org>
> > > >> > > wrote:
> > > >> > > >>
> > > >> > > >> gerrit has a button on the UI to cherry-pick to different
> > > branches.
> > > >> > > >> The button creates separate "gerrit changes" which you can then
> > > >> > > >> commit.  Eventually we could hook those up to Jenkins--
> > something
> > > >> > > >> which we've never been able to do for different branches with
> > the
> > > >> > > >> patch-file-based workflow.
> > > >> > > >
> > > >> > > >
> > > >> > > > If you’re saying what I think you’re saying, people have
> > > been
> > > >> > > able to submi

Re: Github integration for Hadoop

[Colin McCabe]

Thanks, Vinayakumar (and Daniel Gruno.)

C.

On Tue, Apr 5, 2016, at 08:48, Vinayakumar B wrote:
> INFRA-11597 has been fixed. Now all github mails are diverted to
> common-iss...@hadoop.apache.org
> 
> -Vinay
> 
> On Tue, Apr 5, 2016 at 1:05 PM, Vinayakumar B 
> wrote:
> 
> > https://issues.apache.org/jira/browse/INFRA-11597 has been filed for this.
> >
> > -Vinay
> >
> > -Original Message-
> > From: Colin McCabe [mailto:co...@cmccabe.xyz]
> > Sent: 05 April 2016 08:07
> > To: common-dev@hadoop.apache.org
> > Subject: Re: Github integration for Hadoop
> >
> > Yes, please.  Let's disable these mails.
> >
> > C.
> >
> > On Mon, Apr 4, 2016, at 06:21, Vinayakumar B wrote:
> > > bq. We don't spam common-dev about every time a new patch attachment
> > > gets posted to an existing JIRA.  We shouldn't do that for github
> > > either.
> > >
> > > Is there any update on this. ?
> > > Any INFRA ticket filed to disable these mails?
> > >
> > > Because, I am sure, people would have got frustrated by seeing mails
> > > generated by my recent PR submissions. And each time, when some
> > > comment gets added to PR.
> > >
> > > -Vinay
> > >
> > > On Tue, Nov 24, 2015 at 3:35 AM, Andrew Wang
> > > 
> > > wrote:
> > >
> > > > Hi Eric,
> > > >
> > > > My comment wrt GH permissions is available in context earlier in the
> > > > thread, pasting again here:
> > > >
> > > > ===
> > > >
> > > > Great, glad to hear it. That wasn't mentioned in the email thread or
> > > > on the INFRA ticket, and the INFRA ticket mentions these integrations:
> > > >
> > > > Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull
> > > > Request Comment, and push, Push
> > > >
> > > > Are these the right set of permissions to disable integrating PRs?
> > > > Namely, the "push" permissions look unnecessary. We should also
> > > > disable GH issues since we don't want users filing issues there.
> > > >
> > > >
> > > > On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe
> > > > 
> > > > wrote:
> > > >
> > > > > We don't spam common-dev about every time a new patch attachment
> > > > > gets posted to an existing JIRA.  We shouldn't do that for github
> > either.
> > > > >
> > > > > +1 to Andrew and Steve's suggestion of disabling these emails (or
> > > > > +at
> > > > > least sending them to a separate list that people can opt in to).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang 
> > wrote:
> > > > > > Hi Andrew,
> > > > > >
> > > > > > Many Apache projects already have Github integration.  Chukwa
> > > > > > also has Github integration.  Pull request can be integrated into
> > JIRA, i.e.
> > > > > > https://issues.apache.org/jira/browse/CHUKWA-783
> > > > > >
> > > > > > This allows code review process to happen at per line level on
> > > > > > Github,
> > > > or
> > > > > > comment on JIRA for summary of the activities.  Micro comments
> > > > > > are
> > > > squash
> > > > > > away.  The final commit is always happening on Apache git rather
> > > > > > than Github.  Therefore, there is no disabling required for pull
> > request.
> > > > > > Primary activity is still on JIRA, and Github is only a
> > > > > > supplement to
> > > > > make
> > > > > > line by line code review easy.  Overall user experience is
> > > > > > better than
> > > > RB
> > > > > > or Gerrit in my opinion.
> > > > > >
> > > > > > regards,
> > > > > > Eric
> > > > > >
> > > > > > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang <
> > > > andrew.w...@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Has there been any progress on locking down the Github
> > > > > >> permissions
> > 

Re: 2.7.3 release plan

[Colin McCabe]

In general, the only bundled native component I can see is lz4.  I guess
debatably we should add tree.h to the NOTICE file as well, since it came
from BSD and is licensed under that license.

Please keep in mind bundling means "included in the source tree", NOT
"downloaded during the build process."  "mvn package" dumps a ton of
jars in the build directory, but these dependencies aren't considered
bundled since their source does not appear in our git repo.  Similarly,
linking against a library doesn't make it "bundled", nor does dlopening
symbols in that library.

The big omission is that we have a lot of Javascript source files in our
source tree that do not appear in LICENSE or NOTICE.  I agree that we
should address those before making a new release.

best,


On Wed, Apr 6, 2016, at 14:13, Allen Wittenauer wrote:
> 
>   This is probably a good time to remind/point folks to HADOOP-12893.  
> Apache Hadoop's binary artifacts (with or without native code) and source 
> artifacts are not complying with the licenses of bundled components.  I 
> fairly confident this means releases are off the table until someone audits 
> the entire codebase given how many have been added without any thought to 
> actually following the terms of the license….
> 
>

Re: 2.7.3 release plan

[Colin McCabe]

Sure.  We will continue this discussion on HADOOP-12893.  It now appears
that binary tarballs may need their own license and notice files-- see
the jira for details.

Best,
C.

On Thu, Apr 7, 2016, at 09:28, Vinod Kumar Vavilapalli wrote:
> Colin / Sean, appreciate moving your feedback (and copy-pasting your
> current comments) on this issue to the JIRA: HADOOP-12893
> 
> Thanks
> +Vinod
> 
> > On Apr 7, 2016, at 7:43 AM, Sean Busbey  wrote:
> > 
> > On Wed, Apr 6, 2016 at 6:26 PM, Colin McCabe  > <mailto:cmcc...@apache.org>> wrote:
> >> In general, the only bundled native component I can see is lz4.  I guess
> >> debatably we should add tree.h to the NOTICE file as well, since it came
> >> from BSD and is licensed under that license.
> >> 
> >> Please keep in mind bundling means "included in the source tree", NOT
> >> "downloaded during the build process."  "mvn package" dumps a ton of
> >> jars in the build directory, but these dependencies aren't considered
> >> bundled since their source does not appear in our git repo.  Similarly,
> >> linking against a library doesn't make it "bundled", nor does dlopening
> >> symbols in that library.
> >> 
> >> The big omission is that we have a lot of Javascript source files in our
> >> source tree that do not appear in LICENSE or NOTICE.  I agree that we
> >> should address those before making a new release.
> >> 
> >> best,
> >> 
> >> 
> > 
> > Each artifact that the PMC publishes must abide by the ASF licensing
> > policy. That includes
> > 
> > * Source release artifact
> > * any convenience binary artifacts places on dist.apache
> > * any convenience jars put into the ASF Nexus repository
> > 
> > That likely means that we bundle much more than just what's in the source 
> > tree.
> > 
> > (Though this sounds like we're getting off topic for the 2.7.3 release 
> > plan.)
> > 
> > -- 
> > busbey
> 

Re: [DISCUSS] Merge Jersey upgrade(HADOOP-9613) to trunk

[Colin McCabe]

+1 for updating this in trunk.  Thanks, Tsuyoshi Ozawa.

cheers,
Colin

On Mon, May 9, 2016, at 12:12, Tsuyoshi Ozawa wrote:
> Hi developers,
> 
> We’ve worked on upgrading jersey(HADOOP-9613) for a years. It's
> essential change to support complication with JDK8. It’s almost there.
> 
> One concern to merge this to trunk is incompatibility. After the
> release of Jersey 1.13, the root element whose content is empty
> collection is changed from null to empty object({}).  Because of this
> problem, I’ve marked HADOOP-9613 as incompatible change. Is it
> acceptable for us? If it’s acceptable change in trunk, I’d like to
> merge it into trunk.
> 
> Thanks,
> - Tsuyoshi
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: 7 build nodes are down

[Colin McCabe]

Did INFRA have any information on this?

best,


On Fri, May 6, 2016, at 15:14, Allen Wittenauer wrote:
> 
> Anyone know why?
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Treating LimitedPrivate({"MapReduce"}) as Public APIs for YARN applications

[Colin McCabe]

On Tue, May 10, 2016, at 11:34, Hitesh Shah wrote:
> There seems to be some incorrect assumptions on why the application had
> an issue. For rolling upgrade deployments, the application bundles the
> client-side jars that it was compiled against and uses them in its
> classpath and expects to be able to communicate with upgraded servers.
> Given that hadoop-common is a monolithic jar, it ends up being used on
> both client-side and server-side. The problem in this case was caused by
> the fact that the ResourceManager was generating the credentials file
> with a format understood only by hadoop-common from 3.x. For an
> application compiled against 2.x and has *only* hadoop-common from 2.x on
> its classpath, trying to read this file fails.  
> 
> This is not about whether internal implementations can change for
> non-public APIs. The file format for the Credential file in this scenario
> is *not* internal implementation especially when you can have different
> versions of the library trying to read the file. If an older client is
> talking to a newer versioned server, the general backward compat
> assumption is that the client should receive a response that it can parse
> and understand. In this scenario, the credentials file provided to the
> YARN app by the RM should have been written out with the older version or
> at the very least been readable by the older hadoop-common.jar.
> 
> In any case, does anyone have any specific concerns with changing
> LimitedPrivate({"MapReduce"}) to Public?

I think Allen's concern is valid.  These APIs were annotated as
LimitedPrivate so that we could change them in the future.  If they
become public, we lose that flexibility.

> 
> And sure, if we are saying that Hadoop-3.x requires all apps built
> against it to go through a full re-compile as well as downtime as
> existing apps may no longer work out of the box, lets call it out very
> explicitly in the Release notes. 

I think it's reasonable to require all applications to be re-compiled
for Hadoop 3.x.  One reason is because we make incompatible changes to
Hadoop's dependencies in major releases.  This dependency change also
means that "existing apps may no longer work out of the box," as you
stated, since they may need to be updated to use different versions of
our dependencies.

For the same reasons, it is reasonable to require downtime when
upgrading from 2.x to 3.x (or generally, from any major release to any
other major release).  For example, I believe the HDFS rolling upgrade
document explicitly calls out the fact that the mechanism is targetted
at going to between minor releases, not major ones.

If we need to document this better, then let's file a JIRA to do that.

best,
Colin


> 
> — Hitesh
> 
> > On May 10, 2016, at 9:24 AM, Allen Wittenauer  
> > wrote:
> > 
> > 
> >> On May 10, 2016, at 8:37 AM, Hitesh Shah  wrote:
> >> 
> >> There have been various discussions on various JIRAs where upstream 
> >> projects such as YARN apps ( Tez, Slider, etc ) are called out for using 
> >> the above so-called Private APIs. A lot of YARN applications that have 
> >> been built out have picked up various bits and pieces of implementation 
> >> from MapReduce and DistributedShell to get things to work.
> >> 
> >> A recent example is a backward incompatible change introduced ( where the 
> >> API is not even directly invoked ) in the Credentials class related to the 
> >> ability to read tokens/credentials from a file.
> > 
> > Let’s be careful here.  It should be noted that the problem happened 
> > primarily because the application jar appears to have included some hadoop 
> > jars in them.   So the API invocation isn’t the problem:  it’s the fact 
> > that the implementation under the hood changed.  If the application jar 
> > didn’t bundle hadoop jars —especially given that were already on the 
> > classpath--this problem should never have happened.
> > 
> >> This functionality is required by pretty much everyone as YARN provides 
> >> the credentials to the app by writing the credentials/tokens to a local 
> >> file which is read in when UserGroupInformation.getCurrentUser() is 
> >> invoked.
> > 
> > What you’re effectively arguing is that implementations should never 
> > change for public (and in this case LimitedPrivate) APIs.  I don’t think 
> > that’s reasonable.  Hadoop is filled with changes in major branches where 
> > the implementations have changed but the internals have been reworked to 
> > perform the work in a slightly different manner.
> > 
> >> This change breaks rolling upgrades for yarn applications from 2.x to 3.x 
> >> (whether we end up supporting rolling upgrades across 2.x to 3.x is a 
> >> separate discussion )
> > 
> > 
> > At least today, according to the document attached to YARN-666 (lol), 
> > rolling upgrades are only supported within the same major version.  
> > 
> >> 
> >> I would like to change our compatibility docs to state that any API that 
> >> is marked a

Re: [DISCUSS] Treating LimitedPrivate({"MapReduce"}) as Public APIs for YARN applications

[Colin McCabe]

Thanks for explaining, Chris.  I generally agree that
UserGroupInformation should be annotated as Public rather than
LimitedPrivate, although you guys have more context than I do.

However, I do think it's important that we clarify that we can break
public APIs across a major version transition such as 2.x -> 3.x.  It
would be particularly nice to remove a lot of the static or global state
in UGI, although I don't know if we'll get to that before 3.0 is
released.

best,
Colin

On Tue, May 10, 2016, at 14:46, Chris Nauroth wrote:
> Yes, I agree with you Andrew.
> 
> Sorry, I should clarify my prior response.  I didn't mean to imply a
> blind s/LimitedPrivate/Public/g across the whole codebase.  Instead, I'm
> +1 for the intent of HADOOP-10776: a transition to Public for
> UserGroupInformation, and by extension the related parts of its API like
> Credentials.
> 
> I'm in the camp that generally questions the usefulness of
> LimitedPrivate, but I agree that transitions to Public need case-by-case
> consideration.
> 
> --Chris Nauroth
> 
> From: Andrew Wang
> mailto:andrew.w...@cloudera.com>>
> Date: Tuesday, May 10, 2016 at 2:40 PM
> To: Chris Nauroth
> mailto:cnaur...@hortonworks.com>>
> Cc: Hitesh Shah mailto:hit...@apache.org>>,
> "yarn-...@hadoop.apache.org"
> mailto:yarn-...@hadoop.apache.org>>,
> "mapreduce-...@hadoop.apache.org"
> mailto:mapreduce-...@hadoop.apache.org>>,
> "common-dev@hadoop.apache.org"
> mailto:common-dev@hadoop.apache.org>>
> Subject: Re: [DISCUSS] Treating LimitedPrivate({"MapReduce"}) as Public
> APIs for YARN applications
> 
> Why don't we address these on a case-by-case basis, changing the
> annotations on these key classes to Public? LimitedPrivate{"YARN
> applications"} is the same thing as Public.
> 
> This way we don't need to add special exceptions to our compatibility
> policy. Keeps it simple.
> 
> Best,
> Andrew
> 
> On Tue, May 10, 2016 at 2:26 PM, Chris Nauroth
> mailto:cnaur...@hortonworks.com>> wrote:
> +1 for transitioning from LimitedPrivate to Public.
> 
> I view this as an extension of the need for UserGroupInformation and
> related APIs to be Public.  Regardless of the original intent behind
> LimitedPrivate, these are de facto public now, because there is no viable
> alternative for applications that want to integrate with a secured Hadoop
> cluster.
> 
> There is prior discussion of this topic on HADOOP-10776 and HADOOP-12913.
> HADOOP-10776 is a blocker for 2.8.0 to make the transition to Public.
> 
> --Chris Nauroth
> 
> 
> 
> 
> On 5/10/16, 11:34 AM, "Hitesh Shah"
> mailto:hit...@apache.org>> wrote:
> 
> >There seems to be some incorrect assumptions on why the application had
> >an issue. For rolling upgrade deployments, the application bundles the
> >client-side jars that it was compiled against and uses them in its
> >classpath and expects to be able to communicate with upgraded servers.
> >Given that hadoop-common is a monolithic jar, it ends up being used on
> >both client-side and server-side. The problem in this case was caused by
> >the fact that the ResourceManager was generating the credentials file
> >with a format understood only by hadoop-common from 3.x. For an
> >application compiled against 2.x and has *only* hadoop-common from 2.x on
> >its classpath, trying to read this file fails.
> >
> >This is not about whether internal implementations can change for
> >non-public APIs. The file format for the Credential file in this scenario
> >is *not* internal implementation especially when you can have different
> >versions of the library trying to read the file. If an older client is
> >talking to a newer versioned server, the general backward compat
> >assumption is that the client should receive a response that it can parse
> >and understand. In this scenario, the credentials file provided to the
> >YARN app by the RM should have been written out with the older version or
> >at the very least been readable by the older hadoop-common.jar.
> >
> >In any case, does anyone have any specific concerns with changing
> >LimitedPrivate({"MapReduce"}) to Public?
> >
> >And sure, if we are saying that Hadoop-3.x requires all apps built
> >against it to go through a full re-compile as well as downtime as
> >existing apps may no longer work out of the box, lets call it out very
> >explicitly in the Release notes.
> >
> >‹ Hitesh
> >
> >> On May 10, 2016, at 9:24 AM, Allen Wittenauer
> >>mailto:allenwittena...@yahoo.com>> wrote:
> >>
> >>
> >>> On May 10, 2016, at 8:37 AM, Hitesh Shah 
> >>> mailto:hit...@apache.org>> wrote:
> >>>
> >>> There have been various discussions on various JIRAs where upstream
> >>>projects such as YARN apps ( Tez, Slider, etc ) are called out for
> >>>using the above so-called Private APIs. A lot of YARN applications that
> >>>have been built out have picked up various bits and pieces of
> >>>implementation from MapReduce a

Compatibility guidelines for toString overrides

[Colin McCabe]

Hi all,

Recently a discussion came up on HADOOP-13028 about the wisdom of
overloading S3AInputStream#toString to output statistics information. 
It's a difficult judgement for me to make, since I'm not aware of any
compatibility guidelines for InputStream#toString.  Do we have
compatibility guidelines for toString functions?

It seems like the output of toString functions is usually used as a
debugging aid, rather than as a stable format suitable for UI display or
object serialization.  Clearly, there are a few cases where we might
want to specifically declare that a toString method is a stable API. 
However, I think if we attempt to treat the toString output of all
public classes as stable, we will have greatly increased the API
surface.  Should we formalize this and declare that toString functions
are @Unstable, Evolving unless declared otherwise?

best,
Colin

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] upgrade tomcat from 6.0.44 to 6.0.45 and use non apache.org mirror

[Colin McCabe]

I think the Tomcat situation is concerning in a lot of ways.

1. We are downloading without authentication, using http rather than
https.
2. We are downloading an obsolete release.
3. Our build process is violating the apache.archive.org guidelines by
downloading from the site directly, rather than from a mirror.
4. Clean builds take longer than necessary because this tar file needs
to be downloaded.

I'm not too familiar with how Tomcat works... does anyone have any ideas
for fixing this?

best,
Colin


On Tue, May 17, 2016, at 14:08, Wangda Tan wrote:
> Hi common-devs,
> 
> When I tried to build Hadoop trunk today, Maven build failed because kms
> failed to download:
> http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.44/bin/apache-tomcat-6.0.44.tar.gz
> .
> 
> When I open the link from browser, it shows bandwidth limit exceeded:
> 
> ===
> Bandwidth limit exceeded
> 
> The daily allowance of 5GB for this IP has been exceeded, and downloads
> disabled until midnight, UTC (circa 21 hours from now).
> If you have any questions about this, feel free to reach out to us at
> infrastruct...@apache.org.
> ===
> 
> Then I investigated a little more, there're two issues from what I can
> see
> 1) If you look at http://archive.apache.org/dist/, it clearly says:
> *> Please do not download from apache.org !*
> 
> 2) And also, http://archive.apache.org/dist/ says:
> > Older *non-recommended* releases can be found on our archive site
> .
> 
> We're using tomcat with version=6.0.44, on other Apache mirror site, only
> 6.0.45 tomcat is downloadable. Which essentially means 6.0.44 is a
> non-recommended release.
> 
> I would like to propose to make following changes:
> 1) Update tomcat download site from apache.org to another mirror.
> 2) Upgrade tomcat from 6.0.44 to 6.0.45.
> 
> But would like to hear your thoughts before filing JIRA.
> 
> Thanks,
> Wangda

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Why there are so many revert operations on trunk?

[Colin McCabe]

One anti-pattern that keeps coming up over and over again is people
trying to do big and complex features without feature branches.  This
happened with HDFS truncate as well.  This inevitably leads to
controversy because people see very big and invasive patches zooming
past and get alarmed.  Devops people see complicated new code going
directly into production branches and freak out.  Developers see new
APIs getting added without much discussion and start pushing back.

This all gets worse when it's combined with the lack of an up-to-date
design doc.  This leads to people asking the same questions over and
over, since there's no central place they can go to do background
reading about the new feature.  It also tends to lead to arguments,
since the design decisions that were made seem to come out of nowhere,
rather than being justified by careful consideration of alternatives.

When patches get committed directly to production branches, the
discussion also gets mixed up with release management discussions, which
are often contentious as well.  Release management discussions can go on
for a while at the best of times-- when you combine this with all the
above, it just becomes really messy.

My advice is, if 3 or more people ask you for a feature branch, just
create a feature branch already!  It makes things a lot simpler and less
controversial.  It decouples the feature development discussion from the
release management discussion.  It gives developers the ability to look
at the feature as a whole, rather than feeling like they have to
understand each commit in isolation.  Devops people feel like they can
relax because the decision about what stable branches to backport to
will be made after they have a chance to clearly understand the new
feature, rather than before.  The branch merge email functions as a kind
of overview of the feature which makes people more comfortable with it.

Feature branches actually speed up development for medium or large-sized
features.  You can have branch committers who are not full committers,
but can commit to the branch.  You can rewrite your API at any time
without worrying about compatibility.  You can rewrite the history of
the branch however you like to explore new approaches.  You don't need
to backport each of your patches to N stable branches.

Some people seem scared by the need for a merge vote to merge a feature
branch.  But if your feature is big, you are going to get scrutiny
anyway.  Committing stuff directly to trunk or branch-2 is a not a "get
out of jail free" card that bypasses community review.  If anything,
community review will probably be longer and more contentious because of
the reasons above.  People get frustrated when commits to production
branches continue even while big questions about the feature still
remain unanswered.

We already have rules constraining the use of the -1 vote.  Like any
other discussion, -1 votes need to be "constructive"-- that is, they
need to clearly spell out the way forward, rather than just saying no. 
In this particular case, the concerns we had were about the way the
feature was being developed, and the API.

I think that the discussion that is going on for HDFS async right now is
healthy, and will lead to a better feature.  We have people from
downstream projects such as HBase commenting on the kind of APIs they
would find useful.  We have discussions about the impact on the RPC
system, and the branches that it makes sense to target.  And now that we
have a feature branch, we have a lot more freedom to experiment.

best,
Colin


On Tue, Jun 7, 2016, at 11:02, larry mccay wrote:
> -1 needs not be a taken as a derogatory statement being a number should
> actually make it less emotional.
> It is dangerous to a community to become oversensitive to it.
> 
> I generally see language such as "I am -1 on this until this particular
> thing is fixed" or that it violates some common pattern or precedence set
> in the project. This is perfectly reasonable language and there is no
> reason to make the reviewer provide an alternative.
> 
> So, I am giving my -1 to any proposal for rule changes on -1 votes. :)
> 
> 
> On Tue, Jun 7, 2016 at 1:15 PM, Ravi Prakash 
> wrote:
> 
> > +1 on being more respectful. We seem to be having a lot of distasteful
> > discussions recently. If we fight each other, we are only helping our
> > competitors win (and trust me, its out there).
> >
> > I would also respectfully request people not to throw -1s around. I have
> > faced this a few times and its really frustrating. Every one has opinions
> > and some times different people can't fathom why someone else thinks the
> > way they do. I am pretty sure none of us is acting with malicious intent,
> > so perhaps a little more tolerance, faith and trust will help all of us
> > improve Hadoop and the ecosystem much faster. That's not to say that
> > sometimes -1s are not warranted, but we should look to it as an extreme
> > measure. Unfortunately there i

Re: bug in SequenceFile.sync()?

[Colin McCabe]

Hi Chris,

Thanks for the report.  I filed
https://issues.apache.org/jira/browse/HADOOP-9667 for this.

Colin
Software Engineer, Cloudera


On Mon, Jun 24, 2013 at 2:20 AM, Christopher Ng  wrote:
> cross-posting this from cdh-users group where it received little interest:
>
> is there a bug in SequenceFile.sync()?  This is from cdh4.3.0:
>
> /** Seek to the next sync mark past a given position.*/
> public synchronized void sync(long position) throws IOException {
>   if (position+SYNC_SIZE >= end) {
> seek(end);
> return;
>   }
>
>   if (position < headerEnd) {
> // seek directly to first record
> in.seek(headerEnd); <
> should this not call seek (ie this.seek) instead?
> // note the sync marker "seen" in the header
> syncSeen = true;
> return;
>   }
>
> the problem is that when you sync to the start of a compressed file, the
> noBufferedKeys and valuesDecompressed isn't reset so a block read isn't
> triggered.  When you subsequently call next() you're potentially getting
> keys from the buffer which still contains keys from the previous position
> of the file.

Re: git line endings trouble since recent trunk

[Colin McCabe]

I think the fix for this is to set svn:eol-style to "native" on this
file.  It's set on many other files, just not on this one:

cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
./hadoop-project-dist/README.txt
native
cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
cmccabe@keter:~/hadoopST/trunk>

It might also be a good idea to run dos2unix on it.

I thought that in general we wanted to have 'LF' everywhere, so
perhaps we should add this to the patch.sh script to prevent this from
re-occurring.

Colin


On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza  wrote:
> I haven't been able to find a solution.  Just filed
> https://issues.apache.org/jira/browse/HADOOP-9675.
>
>
> On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi wrote:
>
>> Sandy... did you fix this? any jira to track? me too facing same problem..
>>
>> Thanks,
>> Omkar Joshi
>> *Hortonworks Inc.* 
>>
>>
>> On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen 
>> wrote:
>>
>> > Have the same problem here, have to edit the patch manually to exclude
>> the
>> > changes in releasenotes.html
>> >
>> >
>> > On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza > > >wrote:
>> >
>> > > Has anybody else been having trouble with line endings since pulling
>> > trunk
>> > > recently?
>> > >
>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
>> > > shows up as modified even though I haven't touched it, and I can't
>> check
>> > it
>> > > out or reset to a previous version to make that go away.  The only
>> thing
>> > I
>> > > can do to neutralize it is to put it in a dummy commit, but I have to
>> do
>> > > this every time I switch branches or rebase.
>> > >
>> > > This appears to have began after the release notes commit
>> > >  (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line
>> > > endings change.
>> > >
>> > > -Sandy
>> > >
>> >
>> >
>> >
>> > --
>> > Zhijie Shen
>> > Hortonworks Inc.
>> > http://hortonworks.com/
>> >
>>

Re: git line endings trouble since recent trunk

[Colin McCabe]

Clarification: svn:eol-style = native causes the files to contain
whatever the native platform used to check out the code uses.  I think
just setting this property on all the HTML files should resolve this
and future problems.

patch posted.
C.

On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe  wrote:
> I think the fix for this is to set svn:eol-style to "native" on this
> file.  It's set on many other files, just not on this one:
>
> cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
> ./hadoop-project-dist/README.txt
> native
> cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
> ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
> cmccabe@keter:~/hadoopST/trunk>
>
> It might also be a good idea to run dos2unix on it.
>
> I thought that in general we wanted to have 'LF' everywhere, so
> perhaps we should add this to the patch.sh script to prevent this from
> re-occurring.
>
> Colin
>
>
> On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza  wrote:
>> I haven't been able to find a solution.  Just filed
>> https://issues.apache.org/jira/browse/HADOOP-9675.
>>
>>
>> On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi wrote:
>>
>>> Sandy... did you fix this? any jira to track? me too facing same problem..
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* <http://www.hortonworks.com>
>>>
>>>
>>> On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen 
>>> wrote:
>>>
>>> > Have the same problem here, have to edit the patch manually to exclude
>>> the
>>> > changes in releasenotes.html
>>> >
>>> >
>>> > On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza >> > >wrote:
>>> >
>>> > > Has anybody else been having trouble with line endings since pulling
>>> > trunk
>>> > > recently?
>>> > >
>>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
>>> > > shows up as modified even though I haven't touched it, and I can't
>>> check
>>> > it
>>> > > out or reset to a previous version to make that go away.  The only
>>> thing
>>> > I
>>> > > can do to neutralize it is to put it in a dummy commit, but I have to
>>> do
>>> > > this every time I switch branches or rebase.
>>> > >
>>> > > This appears to have began after the release notes commit
>>> > >  (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line
>>> > > endings change.
>>> > >
>>> > > -Sandy
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Zhijie Shen
>>> > Hortonworks Inc.
>>> > http://hortonworks.com/
>>> >
>>>

Re: git line endings trouble since recent trunk

[Colin McCabe]

On Sat, Jun 29, 2013 at 5:18 PM, Luke Lu  wrote:
> The problem is due to relnotes.py generating the html containing some CRLF
> (from JIRA) and the release manager not using git-svn, which caused the
> html with mixed eol getting checked in. The problem would then manifest for
> git users due to text=auto in .gitattributes (see HADOOP-8912) that auto
> converts CRLF to LF, hence the persistent modified status of the
> releasenotes.html for a fresh checkout.
>
> Adding svn:eol-style=native would only fix half the problem. We need to fix
> relnotes.py to avoid generating html with mixed eols (by normalizing
> everything to LF or native).

While I agree that it would be nice to fix relnotes.py, it seems to me
that setting svn:eol-style=native should fix the problem completely.
Files with this attribute set are stored internally by subversion with
all newlines as LF, and converted to CRLF as needed.  After all,
eol-style=native would not be very useful if it only applied on
checkout.  Windows users would be constantly checking in CRLF in that
case.

I'm not an svn expert, though, and I haven't tested the above.

Colin


>
>
> On Fri, Jun 28, 2013 at 1:03 PM, Colin McCabe wrote:
>
>> Clarification: svn:eol-style = native causes the files to contain
>> whatever the native platform used to check out the code uses.  I think
>> just setting this property on all the HTML files should resolve this
>> and future problems.
>>
>> patch posted.
>> C.
>>
>> On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe 
>> wrote:
>> > I think the fix for this is to set svn:eol-style to "native" on this
>> > file.  It's set on many other files, just not on this one:
>> >
>> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
>> > ./hadoop-project-dist/README.txt
>> > native
>> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
>> > ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
>> > cmccabe@keter:~/hadoopST/trunk>
>> >
>> > It might also be a good idea to run dos2unix on it.
>> >
>> > I thought that in general we wanted to have 'LF' everywhere, so
>> > perhaps we should add this to the patch.sh script to prevent this from
>> > re-occurring.
>> >
>> > Colin
>> >
>> >
>> > On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza 
>> wrote:
>> >> I haven't been able to find a solution.  Just filed
>> >> https://issues.apache.org/jira/browse/HADOOP-9675.
>> >>
>> >>
>> >> On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi > >wrote:
>> >>
>> >>> Sandy... did you fix this? any jira to track? me too facing same
>> problem..
>> >>>
>> >>> Thanks,
>> >>> Omkar Joshi
>> >>> *Hortonworks Inc.* <http://www.hortonworks.com>
>> >>>
>> >>>
>> >>> On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen 
>> >>> wrote:
>> >>>
>> >>> > Have the same problem here, have to edit the patch manually to
>> exclude
>> >>> the
>> >>> > changes in releasenotes.html
>> >>> >
>> >>> >
>> >>> > On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza <
>> sandy.r...@cloudera.com
>> >>> > >wrote:
>> >>> >
>> >>> > > Has anybody else been having trouble with line endings since
>> pulling
>> >>> > trunk
>> >>> > > recently?
>> >>> > >
>> >>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
>> >>> > > shows up as modified even though I haven't touched it, and I can't
>> >>> check
>> >>> > it
>> >>> > > out or reset to a previous version to make that go away.  The only
>> >>> thing
>> >>> > I
>> >>> > > can do to neutralize it is to put it in a dummy commit, but I have
>> to
>> >>> do
>> >>> > > this every time I switch branches or rebase.
>> >>> > >
>> >>> > > This appears to have began after the release notes commit
>> >>> > >  (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a
>> line
>> >>> > > endings change.
>> >>> > >
>> >>> > > -Sandy
>> >>> > >
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Zhijie Shen
>> >>> > Hortonworks Inc.
>> >>> > http://hortonworks.com/
>> >>> >
>> >>>
>>

Re: git line endings trouble since recent trunk

[Colin McCabe]

On Mon, Jul 1, 2013 at 11:09 AM, Raja Aluri  wrote:
> They are not that many that I can think of unless you are using notepad for
> editing, but some of the windows related files might require CRLF. This can
> be handled in .gitattributes
> I think it's a very good idea to add this check to test-patch script and
> reject the patch based on the CRLF check.
> -Raja

I don't see why you would need a hook.  Just set svn:eol-style=crlf on
the file that need CRLF.

>
>
> On Mon, Jul 1, 2013 at 11:00 AM, Alejandro Abdelnur wrote:
>
>> why not just add a precommit hook in svn to reject commits with CRLF?
>>
>>
>> On Mon, Jul 1, 2013 at 10:51 AM, Raja Aluri  wrote:
>>
>> > I added a couple of links that discusses 'line endings' when I added
>> > .gitattributes in this JIRA.
>> > HADOOP-8912<https://issues.apache.org/jira/browse/HADOOP-8912>
>> >
>> > I am just reproducing them here.
>> >
>> >1.
>> http://git-scm.com/docs/gitattributes#_checking_out_and_checking_in
>> >2.
>> >
>> >
>> http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git

Regardless of what we do or don't do in git, we should have the line
endings correct in subversion.

cheers.
Colin


>> >
>> > --
>> > Raja
>> >
>> >
>> > On Mon, Jul 1, 2013 at 10:42 AM, Colin McCabe > > >wrote:
>> >
>> > > On Sat, Jun 29, 2013 at 5:18 PM, Luke Lu  wrote:
>> > > > The problem is due to relnotes.py generating the html containing some
>> > > CRLF
>> > > > (from JIRA) and the release manager not using git-svn, which caused
>> the
>> > > > html with mixed eol getting checked in. The problem would then
>> manifest
>> > > for
>> > > > git users due to text=auto in .gitattributes (see HADOOP-8912) that
>> > auto
>> > > > converts CRLF to LF, hence the persistent modified status of the
>> > > > releasenotes.html for a fresh checkout.
>> > > >
>> > > > Adding svn:eol-style=native would only fix half the problem. We need
>> to
>> > > fix
>> > > > relnotes.py to avoid generating html with mixed eols (by normalizing
>> > > > everything to LF or native).
>> > >
>> > > While I agree that it would be nice to fix relnotes.py, it seems to me
>> > > that setting svn:eol-style=native should fix the problem completely.
>> > > Files with this attribute set are stored internally by subversion with
>> > > all newlines as LF, and converted to CRLF as needed.  After all,
>> > > eol-style=native would not be very useful if it only applied on
>> > > checkout.  Windows users would be constantly checking in CRLF in that
>> > > case.
>> > >
>> > > I'm not an svn expert, though, and I haven't tested the above.
>> > >
>> > > Colin
>> > >
>> > >
>> > > >
>> > > >
>> > > > On Fri, Jun 28, 2013 at 1:03 PM, Colin McCabe <
>> cmcc...@alumni.cmu.edu
>> > > >wrote:
>> > > >
>> > > >> Clarification: svn:eol-style = native causes the files to contain
>> > > >> whatever the native platform used to check out the code uses.  I
>> think
>> > > >> just setting this property on all the HTML files should resolve this
>> > > >> and future problems.
>> > > >>
>> > > >> patch posted.
>> > > >> C.
>> > > >>
>> > > >> On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe <
>> > cmcc...@alumni.cmu.edu>
>> > > >> wrote:
>> > > >> > I think the fix for this is to set svn:eol-style to "native" on
>> this
>> > > >> > file.  It's set on many other files, just not on this one:
>> > > >> >
>> > > >> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
>> > > >> > ./hadoop-project-dist/README.txt
>> > > >> > native
>> > > >> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
>> > > >> > ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
>> > > >> > cmccabe@keter:~/hadoopST/trunk>
>> > > >> >
>> > > >> > It might also be a goo

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Colin McCabe]

St.Ack wrote:

> + Once I figured where the logs were, found that JAVA_HOME was not being
> exported (don't need this in hadoop-2.0.5 for instance).  Adding an
> exported JAVA_HOME to my running shell which don't seem right but it took
> care of it (I gave up pretty quick on messing w/
> yarn.nodemanager.env-whitelist and yarn.nodemanager.admin-env -- I wasn't
> getting anywhere)

I thought that we were always supposed to have JAVA_HOME set when
running any of these commands.  At least, I do.  How else can the
system disambiguate between different Java installs?  I need 2
installs to test with JDK7.

> + This did not seem to work for me:
> hadoop.security.group.mapping
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback lue>.

We've seen this before.  I think your problem is that you have
java.library.path set correctly (what System.loadLibrary checks), but
your system library path does not include a necessary dependency of
libhadoop.so-- most likely, libjvm.so.  Probably, we should fix
NativeCodeLoader to actually make a function call in libhadoop.so
before it declares everything OK.

Colin


On Tue, Aug 20, 2013 at 5:35 PM, Stack  wrote:
> On Thu, Aug 15, 2013 at 2:15 PM, Arun C Murthy  wrote:
>
>> Folks,
>>
>> I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would
>> like to get released - this fixes the bugs we saw since the last go-around
>> (rc1).
>>
>> The RC is available at:
>> http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/
>> The RC tag in svn is here:
>> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2
>>
>> The maven artifacts are available via repository.apache.org.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>
> It basically works (in insecure mode), +1.
>
> + Checked signature.
> + Ran on small cluster w/ small load made using mapreduce interfaces.
> + Got the HBase full unit test suite to pass on top of it.
>
> I had the following issues getting it to all work. I don't know if they are
> known issues so will just list them here first.
>
> + I could not find documentation on how to go from tarball to running
> cluster (the bundled 'cluster' and 'standalone' doc are not about how to
> get this tarball off the ground).
> + I had a bit of a struggle putting this release in place under hbase unit
> tests.  The container would just exit w/ 127 errcode.  No logs in expected
> place.  Tripped over where minimrcluster was actually writing.  Tried to
> corral it so it played nicely w/o our general test setup but found that the
> new mini clusters have 'target' hardcoded as output dirs.
> + Once I figured where the logs were, found that JAVA_HOME was not being
> exported (don't need this in hadoop-2.0.5 for instance).  Adding an
> exported JAVA_HOME to my running shell which don't seem right but it took
> care of it (I gave up pretty quick on messing w/
> yarn.nodemanager.env-whitelist and yarn.nodemanager.admin-env -- I wasn't
> getting anywhere)
> + This did not seem to work for me:
> hadoop.security.group.mapping
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.
>  It just did this:
>
> Caused by: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V
> at org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative(Native
> Method)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:49)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.(JniBasedUnixGroupsMappingWithFallback.java:38)
>
> ..so I replaced it
> w/ org.apache.hadoop.security.ShellBasedUnixGroupsMapping on the hbase-side
> to get my cluster up and running.
>
> + Untarring the bin dir, it undoes as hadoop-X.Y.Z-beta.  Undoing the src
> dir it undoes as hadoop-X.Y.Z-beta-src.  I'd have thought they would undo
> into the one directory overlaying each other.
>
> St.Ack

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Colin McCabe]

On Wed, Aug 21, 2013 at 3:49 PM, Stack  wrote:
> On Wed, Aug 21, 2013 at 1:25 PM, Colin McCabe wrote:
>
>> St.Ack wrote:
>>
>> > + Once I figured where the logs were, found that JAVA_HOME was not being
>> > exported (don't need this in hadoop-2.0.5 for instance).  Adding an
>> > exported JAVA_HOME to my running shell which don't seem right but it took
>> > care of it (I gave up pretty quick on messing w/
>> > yarn.nodemanager.env-whitelist and yarn.nodemanager.admin-env -- I wasn't
>> > getting anywhere)
>>
>> I thought that we were always supposed to have JAVA_HOME set when
>> running any of these commands.  At least, I do.  How else can the
>> system disambiguate between different Java installs?  I need 2
>> installs to test with JDK7.
>>
>>
>
> That is fair enough but I did not need to define this explicitly previously
> (for hadoop-2.0.5-alpha for instance) or the JAVA_HOME that was figured in
> start scripts was propagated and now is not (I have not dug in).
>
>
>
>> > + This did not seem to work for me:
>> > hadoop.security.group.mapping
>> >
>> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback> > lue>.
>>
>> We've seen this before.  I think your problem is that you have
>> java.library.path set correctly (what System.loadLibrary checks), but
>> your system library path does not include a necessary dependency of
>> libhadoop.so-- most likely, libjvm.so.  Probably, we should fix
>> NativeCodeLoader to actually make a function call in libhadoop.so
>> before it declares everything OK.
>>
>
> My expectation was that if native group lookup fails, as it does here, then
> the 'Fallback' would kick in and we'd do the Shell query.  This mechanism
> does not seem to be working.

I filed https://issues.apache.org/jira/browse/HADOOP-9895 to address this issue.

best,
Colin


>
>
> St.Ack

Re: symlink support in Hadoop 2 GA

[Colin McCabe]

I think it makes sense to finish symlinks support in the Hadoop 2 GA release.

Colin

On Mon, Sep 16, 2013 at 6:49 PM, Andrew Wang  wrote:
> Hi all,
>
> I wanted to broadcast plans for putting the FileSystem symlinks work
> (HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I think
> it's pretty important we get it in since it's not a compatible change; if
> it misses the GA train, we're not going to have symlinks until the next
> major release.
>
> However, we're still dealing with ongoing issues revealed via testing.
> There's user-code out there that only handles files and directories and
> will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912
> for a nice example where globStatus returning symlinks broke Pig; some of
> us had a conference call to talk it through, and one definite conclusion
> was that this wasn't solvable in a generally compatible manner.
>
> There are also still some gaps in symlink support right now. For example,
> the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink
> resolution, and tooling like the FsShell and Distcp still need to be
> updated as well.
>
> So, there's definitely work to be done, but there are a lot of users
> interested in the feature, and symlinks really should be in GA. Would
> appreciate any thoughts/input on the matter.
>
> Thanks,
> Andrew

Re: symlink support in Hadoop 2 GA

[Colin McCabe]

The issue is not modifying existing APIs.  The issue is that code has
been written that makes assumptions that are incompatible with the
existence of things that are not files or directories.  For example,
there is a lot of code out there that looks at FileStatus#isFile, and
if it returns false, assumes that what it is looking at is a
directory.  In the case of a symlink, this assumption is incorrect.

Faced with this, we have considered making the default behavior of
listStatus and globStatus to be fully resolving symlinks, and simply
not listing dangling symlinks. Code which is prepared to deal symlinks
can use newer versions of the listStatus and globStatus functions
which do return symlinks as symlinks.

We might consider defaulting FileSystem#listStatus and
FileSystem#globStatus to "fully resolving symlinks by default" and
defaulting FileContext#listStatus and FileContext#Util#globStatus to
the opposite.  This seems like the maximally compatible solution that
we're going to get.  I think this makes sense.

The alternative is kicking the can down the road to Hadoop 3, and
letting vendors of alternative (including some proprietary
alternative) systems continue to claim that "Hadoop doesn't support
symlinks yet" (with some justice).

P.S.  I would be fine with putting this in 2.2 or 2.3 if that seems
more appropriate.

sincerely,
Colin

On Tue, Sep 17, 2013 at 8:23 AM, Suresh Srinivas  wrote:
> I agree that this is an important change. However, 2.2.0 GA is getting
> ready to rollout in weeks. I am concerned that these changes will add not
> only incompatible changes late in the game, but also possibly instability.
> Java API incompatibility is some thing we have avoided for the most part
> and I am concerned that this is adding such incompatibility in FileSystem
> APIs. We should find work arounds by adding possibly newer APIs and leaving
> existing APIs as is. If this can be done, my vote is to enable this feature
> in 2.3. Even if it cannot be done, I am concerned that this is coming quite
> late and we should see if could allow some incompatible changes into 2.3
> for this feature.
>
>
> On Mon, Sep 16, 2013 at 6:49 PM, Andrew Wang wrote:
>
>> Hi all,
>>
>> I wanted to broadcast plans for putting the FileSystem symlinks work
>> (HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I think
>> it's pretty important we get it in since it's not a compatible change; if
>> it misses the GA train, we're not going to have symlinks until the next
>> major release.
>>
>> However, we're still dealing with ongoing issues revealed via testing.
>> There's user-code out there that only handles files and directories and
>> will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912
>> for a nice example where globStatus returning symlinks broke Pig; some of
>> us had a conference call to talk it through, and one definite conclusion
>> was that this wasn't solvable in a generally compatible manner.
>>
>> There are also still some gaps in symlink support right now. For example,
>> the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink
>> resolution, and tooling like the FsShell and Distcp still need to be
>> updated as well.
>>
>> So, there's definitely work to be done, but there are a lot of users
>> interested in the feature, and symlinks really should be in GA. Would
>> appreciate any thoughts/input on the matter.
>>
>> Thanks,
>> Andrew
>>
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: symlink support in Hadoop 2 GA

[Colin McCabe]

What we're trying to get to here is a consensus on whether
FileSystem#listStatus and FileSystem#globStatus should return symlinks
__as_symlinks__.  If 2.1-beta goes out with these semantics, I think
we are not going to be able to change them later.  That is what will
happen in the "do nothing" scenario.

Also see Jason Lowe's comment here:
https://issues.apache.org/jira/browse/HADOOP-9912?focusedCommentId=13772002&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13772002

Colin


On Wed, Sep 18, 2013 at 5:11 PM, J. Rottinghuis  wrote:
> However painful protobuf version changes are at build time for Hadoop
> developers, at runtime with multiple clusters and many Hadoop users this is
> a total nightmare.
> Even upgrading clusters from one protobuf version to the next is going to
> be very difficult. The same users will run jobs on, and/or read&write to
> multiple clusters. That means that they will have to fork their code, run
> multiple instances? Or in the very least they have to do an update to their
> applications. All in sync with Hadoop cluster changes. And these are not
> doable in a rolling fashion.
> All Hadoop and HBase clusters will all upgrade at the same time, or we'll
> have to have our users fork / roll multiple versions ?
> My point is that these things are much harder than just fix the (Jenkins)
> build and we're done. These changes are massively disruptive.
>
> There is a similar situation with symlinks. Having an API that lets users
> create symlinks is very problematic. Some users create symlinks and as Eli
> pointed out, somebody else (or automated process) tries to copy to / from
> another (Hadoop 1.x?) cluster over hftp. What will happen ?
> Having an API that people should not use is also a nightmare. We
> experienced this with append. For a while it was there, but users were "not
> allowed to use it" (or else there were large #'s of corrupt blocks). If
> there is an API to create a symlink, then some of our users are going to
> use it and others are going to trip over those symlinks. We already know
> that Pig does not work with symlinks yet, and as Steve pointed out, there
> is tons of other code out there that assumes that !isDir() means isFile().
>
> I like symlink functionality, but in our migration to Hadoop 2.x this is a
> total distraction. If the APIs stay in 2.2 GA we'll have to choose to:
> a) Not uprev until symlink support is figured out up and down the stack,
> and we've been able to migrate all our 1.x (equivalent) clusters to 2.x
> (equivalent). Or
> b) rip out the API altogether. Or
> c) change the implementation to throw an UnsupportedOperationException
> I'm not sure yet which of these I like least.
>
> Thanks,
>
> Joep
>
>
>
>
> On Wed, Sep 18, 2013 at 9:48 AM, Arun C Murthy  wrote:
>
>>
>> On Sep 16, 2013, at 6:49 PM, Andrew Wang  wrote:
>>
>> > Hi all,
>> >
>> > I wanted to broadcast plans for putting the FileSystem symlinks work
>> > (HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I
>> think
>> > it's pretty important we get it in since it's not a compatible change; if
>> > it misses the GA train, we're not going to have symlinks until the next
>> > major release.
>>
>> Just catching up, is this an incompatible change, or not? The above reads
>> 'not an incompatible change'.
>>
>> Arun
>>
>> >
>> > However, we're still dealing with ongoing issues revealed via testing.
>> > There's user-code out there that only handles files and directories and
>> > will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912
>> > for a nice example where globStatus returning symlinks broke Pig; some of
>> > us had a conference call to talk it through, and one definite conclusion
>> > was that this wasn't solvable in a generally compatible manner.
>> >
>> > There are also still some gaps in symlink support right now. For example,
>> > the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink
>> > resolution, and tooling like the FsShell and Distcp still need to be
>> > updated as well.
>> >
>> > So, there's definitely work to be done, but there are a lot of users
>> > interested in the feature, and symlinks really should be in GA. Would
>> > appreciate any thoughts/input on the matter.
>> >
>> > Thanks,
>> > Andrew
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>

Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)

[Colin McCabe]

On Tue, Oct 1, 2013 at 8:59 PM, Arun C Murthy  wrote:
> Yes, sorry if it wasn't clear.
>
> As others seem to agree, I think we'll be better getting a protocol/api 
> stable GA done and then iterating on bugs etc.
>
> I'm not super worried about HADOOP-9984 since symlinks just made it to 
> branch-2.1 recently.
>
> Currently we only have 2 blockers: HADOOP-9984 & MAPREDUCE-5530. Both of 
> which are PA and I've reviewed MR-5530 and is good to go (thanks Robert). 
> Hopefully we can finish up HADOOP-9984 asap and we'll be good.

We've had several reviews for HADOOP-9984 and are currently just
waiting on a +1.

Sorry if this is a dumb question, but will the 2.2 release be made
from branch-2 or what is currently named branch-2.1-beta?  If it is
the latter, we have a few backports we'll need to do.

Colin


>
> thanks,
> Arun
>
> On Oct 1, 2013, at 4:53 PM, Alejandro Abdelnur  wrote:
>
>> Arun,
>>
>> Does this mean that you want to skip a beta release and go straight to GA
>> with the next release?
>>
>> thx
>>
>>
>> On Tue, Oct 1, 2013 at 4:15 PM, Arun C Murthy  wrote:
>>
>>> Guys,
>>>
>>> I took a look at the content in 2.1.2-beta so far, other than the
>>> critical fixes such as HADOOP-9984 (symlinks) and few others in YARN/MR,
>>> there is fairly little content (unit tests fixes etc.)
>>>
>>> Furthermore, it's standing up well in testing too. Plus, the protocols
>>> look good for now (I wrote a gohadoop to try convince myself), let's lock
>>> them in.
>>>
>>> Given that, I'm thinking we can just go ahead rename it 2.2.0 rather than
>>> make another 2.1.x release.
>>>
>>> This will drop a short-lived release (2.1.2) and help us move forward on
>>> 2.3 which has a fair bunch of content already...
>>>
>>> Thoughts?
>>>
>>> thanks,
>>> Arun
>>>
>>>
>>> On Sep 24, 2013, at 4:24 PM, Zhijie Shen  wrote:
>>>
 I've added MAPREDUCE-5531 to the blocker list. - Zhijie


 On Tue, Sep 24, 2013 at 3:41 PM, Arun C Murthy 
>>> wrote:

> With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out…
>>> I'll
> make it clear on the release page, that there are some known issues and
> that we will follow up very shortly with another release.
>
> Meanwhile, let's fix the remaining blockers (please mark them as such
>>> with
> Target Version 2.1.2-beta).
> The current blockers are here:
> http://s.apache.org/hadoop-2.1.2-beta-blockers
>
> thanks,
> Arun
>
> On Sep 16, 2013, at 11:38 PM, Arun C Murthy 
>>> wrote:
>
>> Folks,
>>
>> I've created a release candidate (rc0) for hadoop-2.1.1-beta that I
> would like to get released - this release fixes a number of bugs on top
>>> of
> hadoop-2.1.0-beta as a result of significant amounts of testing.
>>
>> If things go well, this might be the last of the *beta* releases of
> hadoop-2.x.
>>
>> The RC is available at:
> http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0
>> The RC tag in svn is here:
>
>>> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0
>>
>> The maven artifacts are available via repository.apache.org.
>>
>> Please try the release and vote; the vote will run for the usual 7
>>> days.
>>
>> thanks,
>> Arun
>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
>>> entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the
>>> reader
> of this message is not the intended recipient, you are hereby notified
>>> that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender
>>> immediately
> and delete it from your system. Thank You.
>



 --
 Zhijie Shen
 Hortonworks Inc.
 http://hortonworks.com/

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
>>> to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified
>>> that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
>>> immediately
 and delete it from your system. Thank You.
>>>
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
>>>
>>

Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)

[Colin McCabe]

I don't think HADOOP-9972 is a must-do for the next Apache release,
whatever version number it ends up having.  It's just adding a new
API, not changing any existing ones, and it can be done entirely in
generic code.  (The globber doesn't involve FileSystem or AFS
subclasses).

My understanding is that GA is about stabilizing APIs rather than
achieving feature completeness on symlinks.

Colin

On Wed, Oct 2, 2013 at 6:01 PM, Andrew Wang  wrote:
> If we're serious about not breaking compatibility after GA, then we need to
> slow down and make sure we get these new APIs right, or can add them in a
> compatible fashion.
>
> HADOOP-9984 ended up being a bigger change than initially expected, and we
> need to break compatibility with out-of-tree FileSystems to do it properly.
> I would like to see HADOOP-9972 in as well (globLinkStatus), and there are
> open questions on HADOOP-9984 about changing PathFilter and
> FileStatus.getPath() semantics (which would be incompatible). Yes, we could
> just +1 HADOOP-9984 and stamp 2.2.0 on it, but I think it looks bad to then
> immediately turn around and release an incompatible 2.3.
>
> My preference is still for a 2.1.2 with the above API questions resolved,
> then an actual API-stable 2.2.0 GA. This is already punting out all the
> other related FS/tooling changes that we think can be done compatibly but
> are still pretty crucial: shell, distcp, webhdfs, hftp; it'd be great to
> get help on any of these.
>
> Thanks,
> Andrew
>
>
> On Wed, Oct 2, 2013 at 2:56 PM, Roman Shaposhnik  wrote:
>
>> On Tue, Oct 1, 2013 at 5:15 PM, Vinod Kumar Vavilapalli
>>  wrote:
>> > +1. We should get an RC as soon as possible so that we can get all the
>> downstream components to sign off.
>> > The earlier the better.
>>
>> On this very note -- would there be any interest in joining efforts
>> with the Bigtop integration aimed at Hadoop 2.2.x based release
>> of all the Hadoop ecosystem projects?
>>
>> Our current plan is to release Bigtop 0.7.0 within a couple of weeks.
>> That will be the last stable 2.0.x-based release. Bigtop 0.8.0 is supposed
>> to
>> be based on Hadoop 2.x that gets us (Bigtop community) as close as possible
>> to the Hadoop's GA. Here's more on what we'll be doing with Bigtop 0.8.0:
>>
>> http://comments.gmane.org/gmane.comp.apache.incubator.bigtop.devel/10769
>>
>> Of course, on the Bigtop side of things we're stuck with all the necessary
>> integration work anyway, but if there's anything at all that folks are
>> willing
>> to help us and the bigger Hadoop community with that would be very
>> much appreciated. I think both communities will benefit from this type
>> of collaboration.
>>
>> On a practical side of things, as soon as the branch for 2.2.0 gets cut
>> Bigtop can start publishing a complete set of Hadoop ecosystem
>> artifacts built against that particular version and easily install-able
>> on all of our supported systems. We can also start publishing VMs
>> so that folks on OSes other than Linux can help us with testing.
>>
>> Thanks,
>> Roman.
>>

Re: how to compile with OpenIndiana or Solaris

[Colin McCabe]

Log4j configuration is a difficult topic, because there's so many ways
to do it.  I keep my log4j.properties file in the same directory as my
hadoop configuration XML files.  One of the ways log4j finds its
log4j.properties file is by CLASSPATH, and the configuration directory
is always in the CLASSPATH.

I don't think anyone is doing a regular bug scrub on Solaris /
OpenIndiana unit test issues.  This is an area where you could
definitely make a contribution.  You should try with and without
-Pnative in order to narrow down the range of issues.

Once Maven runs cmake and sets up the Makefiles, you can run them just
via "make".  This may be helpful if you want to iterate quickly.  Also
please check the compiler warnings-- they often hide very serious
issues.

Good luck.

best,
Colin

On Thu, Oct 10, 2013 at 12:33 AM, "Andreas Grüninger"
 wrote:
> Hi
>
> I compiled the native libraries of 2.1.1 for OpenIndiana.
> When I run the tests to check my changes I get a lot of errors.
> When I run the tests without my changes I get nearly the same errors.
>
> I checked the tests with the current stable 1.2.1 too with/without native 
> libraries.
> And I get there errors too, about 29 of 1522 (by heart, sitting at home).
>
> The inspection of the crash reports let me assume that there are errors the 
> usage of fstat and IPC errors.
> Maybe some of you compile the source with OI or Solaris and can give some 
> hints how to compile it. The compiler is gcc 4.6.2.
>
> I found the crash reports and the logs in build/test/
> But I did not understand which log4j.properties must be changed to get DEBUG 
> level information. And where to find it afterwards. I assume the same log 
> files which now contains the log messages from log4j with debug level INFO. I 
> found log4j.properties, changed to DEBUG, recompiled and repeated the tests, 
> but to no avail.
> I would really like to change the source code but I need the logging of the 
> DEBUG messages.
>
> TIA and Greetings
>
> Andreas

Re: [VOTE] Merge HDFS-4949 to trunk

[Colin McCabe]

+1.  Thanks, guys.

best,
Colin

On Thu, Oct 17, 2013 at 3:01 PM, Andrew Wang  wrote:
> Hello all,
>
> I'd like to call a vote to merge the HDFS-4949 branch (in-memory caching)
> to trunk. Colin McCabe and I have been hard at work the last 3.5 months
> implementing this feature, and feel that it's reached a level of stability
> and utility where it's ready for broader testing and integration.
>
> I'd also like to thank Chris Nauroth at Hortonworks for code reviews and
> bug fixes, and everyone who's reviewed the HDFS-4949 design doc and left
> comments.
>
> Obviously, I am +1 for the merge. The vote will run the standard 7 days,
> closing on October 24 at 11:59PM.
>
> Thanks,
> Andrew

Re: [VOTE] Merge HDFS-4949 to trunk

[Colin McCabe]

Hi Chris,

I think it's feasible to complete those tasks in the next 7 days.
Andrew is on HDFS-5386.

The test plan document is a great idea.  We'll try to get that up
early next week.  We have a lot of unit tests now, clearly, but some
manual testing is important too.

If we discover any issues during testing, then we can push out the
merge timeframe.  For example, one area that probably needs more
testing is caching+federation.

I would like to get HDFS-5378 and HDFS-5366 in as well.

The other subtasks are "nice to have" but not really critical, and I
think it would be just as easy to do them in trunk.  We're hoping that
having this in trunk will make it easier for us to collaborate on
HDFS-2832 and other ongoing work.

> Also, I want to confirm that this vote only covers trunk.
> I don't see branch-2 mentioned, so I assume that we're
> not voting on merge to branch-2 yet.

Yeah, this vote is only to merge to trunk.

cheers.
Colin

On Fri, Oct 18, 2013 at 10:48 AM, Chris Nauroth
 wrote:
> I agree that the code has reached a stable point.  Colin and Andrew, thank
> you for your contributions and collaboration.
>
> Throughout development, I've watched the feature grow by running daily
> builds in a pseudo-distributed deployment.  As of this week, the full
> feature set is working end-to-end.  I also think we've reached a point of
> API stability for clients who want to control caching programmatically.
>
> There are several things that I'd like to see completed before the merge as
> pre-requisites:
>
> - HDFS-5203: Concurrent clients that add a cache directive on the same path
> may prematurely uncache from each other.
> - HDFS-5385: Caching RPCs are AtMostOnce, but do not persist client ID and
> call ID to edit log.
> - HDFS-5386: Add feature documentation for datanode caching.
> - Standard clean-ups to satisfy Jenkins pre-commit on the merge patch.
>  (For example, I know we've introduced some Javadoc warnings.)
> - Full test suite run on Windows.  (The feature is not yet implemented on
> Windows.  This is just intended to catch regressions.)
> - Test plan posted to HDFS-4949, similar in scope to the snapshot test plan
> that was posted to HDFS-2802.  For my own part, I've run the new unit
> tests, and I've tested end-to-end in a pseudo-distributed deployment.  It's
> unlikely that I'll get a chance to test fully distributed before the vote
> closes, so I'm curious to hear if you've done this on your side yet.
>
> Also, I want to confirm that this vote only covers trunk.  I don't see
> branch-2 mentioned, so I assume that we're not voting on merge to branch-2
> yet.
>
> Before I cast my vote, can you please discuss whether or not it's feasible
> to complete all of the above in the next 7 days?  For the issues assigned
> to me, I do expect to complete them.
>
> Thanks again for all of your hard work!
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Thu, Oct 17, 2013 at 3:07 PM, Colin McCabe wrote:
>
>> +1.  Thanks, guys.
>>
>> best,
>> Colin
>>
>> On Thu, Oct 17, 2013 at 3:01 PM, Andrew Wang 
>> wrote:
>> > Hello all,
>> >
>> > I'd like to call a vote to merge the HDFS-4949 branch (in-memory caching)
>> > to trunk. Colin McCabe and I have been hard at work the last 3.5 months
>> > implementing this feature, and feel that it's reached a level of
>> stability
>> > and utility where it's ready for broader testing and integration.
>> >
>> > I'd also like to thank Chris Nauroth at Hortonworks for code reviews and
>> > bug fixes, and everyone who's reviewed the HDFS-4949 design doc and left
>> > comments.
>> >
>> > Obviously, I am +1 for the merge. The vote will run the standard 7 days,
>> > closing on October 24 at 11:59PM.
>> >
>> > Thanks,
>> > Andrew
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: [VOTE] Merge HDFS-4949 to trunk

[Colin McCabe]

On Thu, Oct 24, 2013 at 1:45 PM, Chris Nauroth  wrote:
> Hi Andrew,
>
> I've come to the conclusion that I'm very confused about merge votes.  :-)
>  It's not just about HDFS-4949.  I'm confused about all merge votes.
>  Rather than muddy the waters here, I've started a separate discussion on
> common-dev.
>
> I do agree with the general plan outlined here, and I will comment directly
> on the HDFS-4949 jira with a binding +1 when I see that we've completed
> that plan.

Thanks, Chris.  Andrew posted a merge patch to HDFS-4949.

We're happy that this code is getting closer to getting into trunk,
since it will make it easier to integrate with the other features in
trunk (like HDFS-2832).  There are still some follow-up tasks, but we
feel that it's easier to do those in trunk.

I'm going to update the design doc in just a moment so be sure to
check it out.  Are there any other things we should do today prior to
merging?

Colin


>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, Oct 23, 2013 at 2:18 PM, Andrew Wang wrote:
>
>> Hey Chris,
>>
>> Right now we're on track to have all of those things done by tomorrow.
>> Since the remaining issues are either not technical or do not involve major
>> changes, I was hoping we could +1 this merge vote in the spirit of "+1
>> pending jenkins". We've gotten clean unit test runs on upstream Jenkins as
>> well, so the only fixups we should need for test-patch.sh are findbugs and
>> javac (which are normally pretty trivial to clean up). Of course, all of
>> your listed prereqs and test-patch would be taken care of before actually
>> merging to trunk.
>>
>> So, we can reset the vote if you feel strongly about this, but it seems
>> like the only real result will be delaying the merge by a week.
>>
>> Thanks,
>> Andrew
>>
>>
>> On Wed, Oct 23, 2013 at 1:03 PM, Chris Nauroth > >wrote:
>>
>> > I've received some feedback that we haven't handled this merge vote the
>> > same as other comparable merge votes, and that the vote should be reset
>> > because of this.
>> >
>> > The recent custom is that we only call for the merge vote after all
>> > pre-requisites have been satisfied.  This would include committing to the
>> > feature branch all patches that the devs deem necessary before the code
>> > lands in trunk, posting a test plan, posting an updated design doc in
>> case
>> > implementation choices diverged from the original design doc, and
>> getting a
>> > good test-patch run from Jenkins on the merge patch.  This was the
>> process
>> > followed for other recent major features like HDFS-2802 (snapshots),
>> > HDFS-347 (short-circuit reads via sharing file descriptors), and
>> > HADOOP-8562 (Windows compatibility).  In this thread, we've diverged from
>> > that process by calling for a vote on a branch that hasn't yet completed
>> > the pre-requisites and stating a plan for work to be done before the
>> merge.
>> >
>> > I still support this work, but can we please restart the vote after the
>> > pre-requisites have landed in the branch?
>> >
>> > Chris Nauroth
>> > Hortonworks
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > On Fri, Oct 18, 2013 at 1:37 PM, Chris Nauroth > > >wrote:
>> >
>> > > +1
>> > >
>> > > Sounds great!
>> > >
>> > > Regarding testing caching+federation, this is another thing that I had
>> > > intended to pick up as part of HDFS-5149.  I'm not sure if I can get
>> this
>> > > done in the next 7 days, so I'll keep you posted.
>> > >
>> > > Chris Nauroth
>> > > Hortonworks
>> > > http://hortonworks.com/
>> > >
>> > >
>> > >
>> > > On Fri, Oct 18, 2013 at 11:15 AM, Colin McCabe > > >wrote:
>> > >
>> > >> Hi Chris,
>> > >>
>> > >> I think it's feasible to complete those tasks in the next 7 days.
>> > >> Andrew is on HDFS-5386.
>> > >>
>> > >> The test plan document is a great idea.  We'll try to get that up
>> > >> early next week.  We have a lot of unit tests now, clearly, but some
>> > >> manual testing is important too.
>> > >>
>> > >> If we discover any issues during testing, then we can push out the
>> > >> 

Re: Next releases

[Colin McCabe]

HADOOP-10020 is a JIRA that disables symlinks temporarily.  They will
be disabled in 2.2.1 as well, if the plan is to have only minor fixes
in that branch.

To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
there.  However, I have only been following the HDFS and common side
of things so I may not have the full picture.  Arun, can you give a
specific example of something you'd like to "blow away"?

Colin

On Mon, Nov 11, 2013 at 10:06 AM, Hari Mankude  wrote:
> Hi Arun,
>
> Another feature that would be relevant and got deferred was the symlink
> work (HADOOP-10020) that Colin and Andrew were working on. Can we include
> this in hadoop-2.3.0 also?
>
> thanks
> hari
>
>
> On Sun, Nov 10, 2013 at 2:07 PM, Alejandro Abdelnur wrote:
>
>> Arun, thanks for jumping on this.
>>
>> On hadoop branch-2.2. I've quickly scanned the commit logs starting from
>> the 2.2.0 release and I've found around 20 JIRAs that I like seeing in
>> 2.2.1. Not all of them are bugs but the don't shake anything and improve
>> usability.
>>
>> I presume others will have their own laundry lists as well and I wonder the
>> union of all of them how much adds up to the current 81 commits.
>>
>> How about splitting the JIRAs among a few contributors to assert there is
>> nothing risky in there? And if so get discuss getting rid of those commits
>> for 2.2.1. IMO doing that would be cheaper than selectively applying
>> commits on a fresh branch.
>>
>> Said this, I think we should get 2.2.1 out of the door before switching
>> main efforts to 2.3.0. I volunteer myself to drive 2.2.1 a  release if ASAP
>> if you don't have the bandwidth at the moment for it.
>>
>> Cheers.
>>
>> Alejandro
>>
>>
>> 
>> Commits in branch-2.2 that I'd like them to be in the 2.2.1 release:
>>
>> The ones prefixed with '*' technically are not bugs.
>>
>>  YARN-1284. LCE: Race condition leaves dangling cgroups entries for killed
>> containers. (Alejandro Abdelnur via Sandy Ryza)
>>  YARN-1265. Fair Scheduler chokes on unhealthy node reconnect (Sandy Ryza)
>>  YARN-1044. used/min/max resources do not display info in the scheduler
>> page (Sangjin Lee via Sandy Ryza)
>>  YARN-305. Fair scheduler logs too many "Node offered to app" messages.
>> (Lohit Vijayarenu via Sandy Ryza)
>> *MAPREDUCE-5463. Deprecate SLOTS_MILLIS counters. (Tzuyoshi Ozawa via Sandy
>> Ryza)
>>  YARN-1259. In Fair Scheduler web UI, queue num pending and num active apps
>> switched. (Robert Kanter via Sandy Ryza)
>>  YARN-1295. In UnixLocalWrapperScriptBuilder, using bash -c can cause Text
>> file busy errors. (Sandy Ryza)
>> *MAPREDUCE-5457. Add a KeyOnlyTextOutputReader to enable streaming to write
>> out text files without separators (Sandy Ryza)
>> *YARN-1258. Allow configuring the Fair Scheduler root queue (Sandy Ryza)
>> *YARN-1288. Make Fair Scheduler ACLs more user friendly (Sandy Ryza)
>>  YARN-1330. Fair Scheduler: defaultQueueSchedulingPolicy does not take
>> effect (Sandy Ryza)
>>  HDFS-5403. WebHdfs client cannot communicate with older WebHdfs servers
>> post HDFS-5306. Contributed by Aaron T. Myers.
>> *YARN-1335. Move duplicate code from FSSchedulerApp and FiCaSchedulerApp
>> into SchedulerApplication (Sandy Ryza)
>> *YARN-1333. Support blacklisting in the Fair Scheduler (Tsuyoshi Ozawa via
>> Sandy Ryza)
>> *MAPREDUCE-4680. Job history cleaner should only check timestamps of files
>> in old enough directories (Robert Kanter via Sandy Ryza)
>>  YARN-1109. Demote NodeManager "Sending out status for container" logs to
>> debug (haosdent via Sandy Ryza)
>> *YARN-1321. Changed NMTokenCache to support both singleton and an instance
>> usage. Contributed by Alejandro Abdelnur
>>  YARN-1343. NodeManagers additions/restarts are not reported as node
>> updates in AllocateResponse responses to AMs. (tucu)
>>  YARN-1381. Same relaxLocality appears twice in exception message of
>> AMRMClientImpl#checkLocalityRelaxationConflict() (Ted Yu via Sandy Ryza)
>>  HADOOP-9898. Set SO_KEEPALIVE on all our sockets. Contributed by Todd
>> Lipcon.
>>  YARN-1388. Fair Scheduler page always displays blank fair share (Liyin
>> Liang via Sandy Ryza)
>>
>>
>>
>> On Fri, Nov 8, 2013 at 10:35 PM, Chris Nauroth > >wrote:
>>
>> > Arun, what are your thoughts on test-only patches?  I know I've been
>> > merging a lot of Windows test stabilization patches down to branch-2.2.
>> >  These can't rightly be called blockers, but they do improve dev
>> > experience, and there is no risk to product code.
>> >
>> > Chris Nauroth
>> > Hortonworks
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > On Fri, Nov 8, 2013 at 1:30 AM, Steve Loughran > > >wrote:
>> >
>> > > On 8 November 2013 02:42, Arun C Murthy  wrote:
>> > >
>> > > > Gang,
>> > > >
>> > > >  Thinking through the next couple of releases here, appreciate f/b.
>> > > >
>> > > >  # hadoop-2.2.1
>> > > >
>> > > >  I was looking through commit logs and there is a *lot* of conte

Re: Next releases

[Colin McCabe]

On Wed, Nov 13, 2013 at 10:10 AM, Arun C Murthy  wrote:
>
> On Nov 12, 2013, at 1:54 PM, Todd Lipcon  wrote:
>
>> On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe wrote:
>>
>>> To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
>>> there.  However, I have only been following the HDFS and common side
>>> of things so I may not have the full picture.  Arun, can you give a
>>> specific example of something you'd like to "blow away"?
>
> There are bunch of issues in YARN/MapReduce which clearly aren't *critical*, 
> similarly in HDFS a cursory glance showed up some 
> *enhancements*/*improvements* in CHANGES.txt which aren't necessary for a 
> patch release, plus things like:
>
> HADOOP-9623
> Update jets3t dependency to 0.9.0

I'm fine with reverting HADOOP-9623 from branch-2.2 and pushing it out
to branch-2.3.  It does bring in httpcore, a dependency that wasn't
there before.

Colin

>
> Here is a straw man proposal:
>
> 
> A patch (third version) release should only include *blocker* bugs which are 
> critical from an operational, security or data-integrity issues.
>
> This way, we can ensure that a minor series release (2.2.x or 2.3.x or 2.4.x) 
> is always release-able, and more importantly, deploy-able at any point in 
> time.
>
> 
>
> Sandy did bring up a related point about timing of releases and the urge for 
> everyone to cram features/fixes into a dot release.
>
> So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 
> 2.4 etc.) and keep the patch releases limited to blocker bugs.
>
> Thoughts?
>
> thanks,
> Arun
>
>
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Next releases

[Colin McCabe]

If 2.4 is released in January, I think it's very unlikely to include
symlinks.  There is still a lot of work to be done before they're
usable.  You can look at the progress on HADOOP-10019.  For some of
the subtasks, it will require some community discussion before any
code can be written.

For better or worse, symlinks have not been requested by users as
often as features like NFS export, HDFS caching, ACLs, etc, so effort
has been focused on those instead.

For now, I think we should put the symlinks-disabling patches
(HADOOP-10020, etc) into branch-2, so that they will be part of the
next releases without additional effort.

I would like to see HDFS caching make it into 2.4.  The APIs and
implementation are beginning to stabilize, and around January it
should be ok to backport to a stable branch.

best,
Colin

On Thu, Dec 5, 2013 at 3:57 PM, Arun C Murthy  wrote:
> Ok, I've updated https://wiki.apache.org/hadoop/Roadmap with a initial 
> strawman list for hadoop-2.4 which I feel we can get out in Jan.
>
> What else would folks like to see? Please keep timeframe in mind.
>
> thanks,
> Arun
>
> On Dec 2, 2013, at 10:55 AM, Arun C Murthy  wrote:
>
>>
>> On Nov 13, 2013, at 1:55 PM, Jason Lowe  wrote:
>>>
>>>
>>> +1 to limiting checkins of patch releases to Blockers/Criticals.  If 
>>> necessary committers check into trunk/branch-2 only and defer to the patch 
>>> release manager for the patch release merge.  Then there should be fewer 
>>> surprises for everyone what ended up in a patch release and less likely the 
>>> patch release becomes destabilized from the sheer amount of code churn.  
>>> Maybe this won't be necessary if everyone understands that the patch 
>>> release isn't the only way to get a change out in timely manner.
>>
>> I've updated https://wiki.apache.org/hadoop/Roadmap to reflect that we only 
>> put in Blocker/Critical bugs into Point Releases.
>>
>> Committers, from now, please exercise extreme caution when committing to a 
>> point release: they should only be limited to Blocker bugs.
>>
>> thanks,
>> Arun
>>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Possible Length issue

[Colin McCabe]

There is a maximum length for message buffers that was introduced by
HADOOP-9676.  So messages with length 1752330339 should not be
accepted.

best,
Colin

On Sat, Dec 28, 2013 at 11:06 AM, Dhaivat Pandya
 wrote:
> Hi,
>
> I've been working a lot with the Hadoop NameNode IPC protocol (while
> building a cache layer on top of Hadoop). I've noticed that for request
> packets coming from the default DFS client that do not have a method name,
> the length field is often *completely *off.
>
> For example, I've been looking at packets with length 1752330339. I thought
> this might have been an issue with my cache layer, so I checked with
> Wireshark, and found packets with such absurd length parameters (obviously,
> the packets themselves weren't actually that long; the length field was
> misrepresented).
>
> Unfortunately, I haven't had the opportunity to test this issue on other
> machines and setups (the reproducing steps should be running an "ls /" with
> the default DFS client and sniffing the packets to find the length
> parameter, release 1.2.1).
>
> Is this normal behavior, a bug or something I'm missing?
>
> Thank you,
>
> Dhaivat

Re: [VOTE] Release Apache Hadoop 2.3.0

[Colin McCabe]

Looks good.

+1, also non-binding.

I downloaded the source tarball, checked md5, built, ran some unit
tests, ran an HDFS cluster.

cheers,
Colin

On Tue, Feb 11, 2014 at 6:53 PM, Andrew Wang  wrote:
> Thanks for putting this together Arun.
>
> +1 non-binding
>
> Downloaded source tarball
> Verified signature and digests
> Ran apache-rat:check
> Built with "mvn clean package install -Pdist -Dtar"
> Started a one-node cluster and ran wordcount
>
> Best,
> Andrew
>
> On Tue, Feb 11, 2014 at 5:40 PM, Ravi Prakash  wrote:
>
>> Thanks Arun for another release!
>>
>> +1 non-binding
>>
>> Verified signatures, deployed a single node cluster and ran sleep and
>> wordcount. Everything looks fine.
>>
>>
>> Regards
>> Ravi
>>
>>
>>
>>
>> On Tuesday, February 11, 2014 5:36 PM, Travis Thompson <
>> tthomp...@linkedin.com> wrote:
>>
>> Everything looks good so far, running on 100 nodes with security enabled.
>>
>> I've found two minor issues I've found with the new Namenode UI so far and
>> will work on them over the next few days:
>>
>> HDFS-5934
>> HDFS-5935
>>
>> Thanks,
>>
>> Travis
>>
>>
>> On Feb 11, 2014, at 4:53 PM, Mohammad Islam 
>> wrote:
>>
>> > Thanks Arun for the initiative.
>> >
>> > +1 non-binding.
>> >
>> >
>> > I tested the followings:
>> > 1. Build package from the source tar.
>> > 2. Verified with md5sum
>> > 3. Verified with gpg
>> > 4. Basic testing
>> >
>> > Overall, good to go.
>> >
>> > Regards,
>> > Mohammad
>> >
>> >
>> >
>> >
>> > On Tuesday, February 11, 2014 2:07 PM, Chen He 
>> wrote:
>> >
>> > +1, non-binding
>> > successful compiled on MacOS 10.7
>> > deployed to Fedora 7 and run test job without any problem.
>> >
>> >
>> >
>> > On Tue, Feb 11, 2014 at 8:49 AM, Arun C Murthy 
>> wrote:
>> >
>> >> Folks,
>> >>
>> >> I've created a release candidate (rc0) for hadoop-2.3.0 that I would
>> like
>> >> to get released.
>> >>
>> >> The RC is available at:
>> >> http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0
>> >> The RC tag in svn is here:
>> >> https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0
>> >>
>> >> The maven artifacts are available via repository.apache.org.
>> >>
>> >> Please try the release and vote; the vote will run for the usual 7 days.
>> >>
>> >> thanks,
>> >> Arun
>> >>
>> >> PS: Thanks to Andrew, Vinod & Alejandro for all their help in various
>> >> release activities.
>> >> --
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> entity to
>> >> which it is addressed and may contain information that is confidential,
>> >> privileged and exempt from disclosure under applicable law. If the
>> reader
>> >> of this message is not the intended recipient, you are hereby notified
>> that
>> >> any printing, copying, dissemination, distribution, disclosure or
>> >> forwarding of this communication is strictly prohibited. If you have
>> >> received this communication in error, please contact the sender
>> immediately
>> >> and delete it from your system. Thank You.
>>

Re: Porting to Solaris

[Colin McCabe]

There are a few existing portability JIRAs for libhadoop.  Check out
HADOOP-7147, HADOOP-9934, HADOOP-6767, HADOOP-7824, and HDFS-5642.

best,
Colin

On Tue, Feb 18, 2014 at 2:14 AM, Malcolm  wrote:
> I have started porting the native libraries of Hadoop 2.2.0 to Solaris and
> would like to eventually get my changes into the main Hadoop source tree.
>
> Are there any public coding guidelines/requirements that I should follow ?
>
> Is there a specific procedure to follow once I have completed the work i.e.
> who to send the changes to, for example.
>
> Thanks,
> Malcolm

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

[Colin McCabe]

+1 for making this guarantee explicit.

It also definitely seems like a good idea to test mixed versions in bigtop.

HDFS is not immune to "new client, old server" scenarios because the HDFS
client gets bundled into a lot of places.

Colin
On Mar 20, 2014 10:55 AM, "Chris Nauroth"  wrote:

> Our use of protobuf helps mitigate a lot of compatibility concerns, but
> there still can be situations that require careful coding on our part.
>  When adding a new field to a protobuf message, the client might need to do
> a null check, even if the server-side implementation in the new version
> always populates the field.  When adding a whole new RPC endpoint, the
> client might need to consider the possibility that the RPC endpoint isn't
> there on an old server, and degrade gracefully after the RPC fails.  The
> original issue in MAPREDUCE-4052 concerned the script commands passed in a
> YARN container submission, where protobuf doesn't provide any validation
> beyond the fact that they're strings.
>
> Forward compatibility is harder than backward compatibility, and testing is
> a big challenge.  Our test suites in the Hadoop repo don't cover this.
>  Does anyone know if anything in Bigtop tries to run with mixed versions?
>
> I agree that we need to make it clear in the language that upgrading client
> alone is insufficient to get access to new server-side features, including
> new YARN APIs.  Thanks for the suggestions, Steve.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran  >wrote:
>
> > I'm clearly supportive of this, though of course the testing costs needed
> > to back up the assertion make it more expensive than just a statement.
> >
> > Two issues
> >
> > -we'd need to make clear that new cluster features that a client can
> invoke
> > won't be available. You can't expect snapshot or symlink support running
> > against a -2.2.0 cluster, even if the client supports it.
> >
> > -in YARN, there are no guarantees that an app compiled against later YARN
> > APIs will work in old clusters. Because YARN apps upload themselves to
> the
> > server, and run with their hadoop, hdfs & yarn libraries. We have to do a
> > bit of introspection in our code already to support this situation. The
> > compatibility doc would need to be clear on that too: "YARN apps that use
> > new APIs (including new fields in datastructures) can expect link
> > exceptions"
> >
> >
> >
> >
> >
> > On 20 March 2014 04:25, Vinayakumar B  wrote:
> >
> > > +1, I agree with your point Chris. It depends on the client application
> > > how they using the hdfs jars in their classpath.
> > >
> > > As implementation already supports the compatibility (through
> protobuf),
> > > No extra code changes required to support new Client + old server.
> > >
> > > I feel it will be good to explicitly mention about the compatibility of
> > > existing APIs in both versions.
> > >
> > > Anyway this is not applicable for the new APIs in latest client and
> this
> > > is understood. We can make it explicit in the document though.
> > >
> > >
> > > Regards,
> > > Vinayakumar B
> > >
> > > -Original Message-
> > > From: Chris Nauroth [mailto:cnaur...@hortonworks.com]
> > > Sent: 20 March 2014 05:36
> > > To: common-dev@hadoop.apache.org
> > > Cc: mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > > yarn-...@hadoop.apache.org
> > > Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> > > Client + Old Server
> > >
> > > I think this kind of compatibility issue still could surface for HDFS,
> > > particularly for custom applications (i.e. something not executed via
> > > "hadoop jar" on a cluster node, where the client classes ought to be
> > > injected into the classpath automatically).  Running DistCP between 2
> > > clusters of different versions could result in a 2.4.0 client calling a
> > > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> > > client as a dependency and try to use it to make HTTP calls to a 2.3.0
> > HDFS
> > > cluster.
> > >
> > > Chris Nauroth
> > > Hortonworks
> > > http://hortonworks.com/
> > >
> > >
> > >
> > > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> > > vino...@apache.org
> > > > wrote:
> > >
> > > > It makes sense only for YARN today where we separated out the
> clients.
> > > > HDFS is still a monolithic jar so this compatibility issue is kind of
> > > > invalid there.
> > > >
> > > > +vinod
> > > >
> > > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth  >
> > > > wrote:
> > > >
> > > > > I'd like to discuss clarification of part of our compatibility
> > policy.
> > > > > Here is a link to the compatibility documentation for release
> 2.3.0:
> > > > >
> > > > >
> > > >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > > > /Compatibility.html#Wire_compatibility
> > > > >
> > > > > For convenience, here are the specific lines in question:
> > > > >
> > > > > Client-Server

Re: very old dependencies

[Colin McCabe]

I think we need some way of isolating YARN, MR, and HDFS clients from
the Hadoop dependencies.  Anything else just isn't sane... whatever we
may say, there will always be clients that rely on the dependencies
that we pull in, if we make those visible.  I can't really blame
clients for this.  It's sort of unclear what happens when you pull in
multiple versions of a dependency through Maven.

The easiest way to solve this would be to use something like Maven-shade.

best,
Colin

On Fri, Mar 28, 2014 at 10:30 AM, Steve Loughran  wrote:
> I don't disagree about version age -I've just been downgrading a project to
> use some of the versions. The issue with YARN apps is that you get these on
> your classpath.
>
>
>1. There's a JIRA: https://issues.apache.org/jira/browse/HADOOP-9991
>2. Nobody is working full time on these, I sporadically apply the
>patches -I get to feel the pain downstream
>
> commons lang is at 2.6 now, which should keep you happy.
>
> jetty? Now that the shuffle is on netty we could think about this and jersey
>
> Guava is trouble. Leave it: new code doesn't work. Remove it and old code
> stops working.
> https://issues.apache.org/jira/browse/HADOOP-10101
>
> I think for branch-3 we should go ahead an apply the patch. For branch 2,
> it's too destructuve.
>
> At some point we also have to commit being java7 + only -this would
> actually help us move up to some of the later dependencies. That's clearly
> not a branch-2
>
>
>
>
> On 28 March 2014 14:59, Sangjin Lee  wrote:
>
>> Hi folks,
>>
>> Even as 2.3 was released, several dependencies of hadoop are quite dated.
>> And more and more of them are causing friction for hadoop-related libraries
>> and hadoop users in general, as these dependency versions are often
>> incompatibly different than the versions most people use these days. So
>> this is becoming a very real problem, if not one already.
>>
>> Some key ones include
>> - guava: 11.0.2 (current 16.0.1)
>> - jetty: 6.1.26 (current 9.1.3)
>> - commons-lang: 2.6 (current 3.3.1)
>>
>> In particular, guava is already causing a lot of issues as many developers
>> have moved onto a newer version and are using APIs that do not exist in
>> 11.0.2, and get NoSuchMethodErrors and such.
>>
>> Also, for jetty, 6.1.26 has been EOFed for some time now.
>>
>> Do we have a plan to review some of these dependencies and upgrade them to
>> reasonably up-to-date versions? I remember seeing some JIRAs on specific
>> dependencies, but has a review been done to determine a good set of
>> versions to upgrade to?
>>
>> It would be great if we could upgrade some of the more common ones at least
>> to modern versions.
>>
>> Thanks,
>> Sangjin
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Plans of moving towards JDK7 in trunk

[Colin McCabe]

I've been using JDK7 for Hadoop development for a while now, and I
know a lot of other folks have as well.  Correct me if I'm wrong, but
what we're talking about here is not "moving towards JDK7" but
"breaking compatibility with JDK6."

There are a lot of good reasons to ditch JDK6.  It would let us use
new APIs in JDK7, especially the new file APIs.  It would let us
update a few dependencies to newer versions.

I don't like the idea of breaking compatibility with JDK6 in trunk,
but not in branch-2.  The traditional reason for putting something in
trunk but not in branch-2 is that it is new code that needs some time
to prove itself.  This doesn't really apply to incrementing min.jdk--
we could do that easily whenever we like.  Meanwhile, if trunk starts
accumulating jdk7-only code and dependencies, backports from trunk to
branch-2 will become harder and harder over time.

Since we make stable releases off of branch-2, and not trunk, I don't
see any upside to this.  To be honest, I see only negatives here.
More time backporting, more issues that show up only in production
(branch-2) and not on dev machines (trunk).

Maybe it's time to start thinking about what version of branch-2 will
drop jdk6 support.  But until there is such a version, I don't think
trunk should do it.

best,
Colin


On Fri, Apr 4, 2014 at 3:15 PM, Haohui Mai  wrote:
> I'm referring to the later case. Indeed migrating JDK7 for branch-2 is more
> difficult.
>
> I think one reasonable approach is to put the hdfs / yarn clients into
> separate jars. The client-side jars can only use JDK6 APIs, so that
> downstream projects running on top of JDK6 continue to work.
>
> The HDFS/YARN/MR servers need to be run on top of JDK7, and we're free to
> use JDK7 APIs inside them. Given the fact that there're way more code in
> the server-side compared to the client-side, having the ability to use JDK7
> in the server-side only might still be a win.
>
> The downside I can think of is that it might complicate the effort of
> publishing maven jars, but this should be an one-time issue.
>
> ~Haohui
>
>
> On Fri, Apr 4, 2014 at 2:37 PM, Alejandro Abdelnur wrote:
>
>> Haohui,
>>
>> Is the idea to compile/test with JDK7 and recommend it for runtime and stop
>> there? Or to start using JDK7 API stuff as well? If the later is the case,
>> then backporting stuff to branch-2 may break and patches may have to be
>> refactored for JDK6. Given that branch-2 got GA status not so long ago, I
>> assume it will be active for a while.
>>
>> What are your thoughts on this regard?
>>
>> Thanks
>>
>>
>> On Fri, Apr 4, 2014 at 2:29 PM, Haohui Mai  wrote:
>>
>> > Hi,
>> >
>> > There have been multiple discussions on deprecating supports of JDK6 and
>> > moving towards JDK7. It looks to me that the consensus is that now hadoop
>> > is ready to drop the support of JDK6 and to move towards JDK7. Based on
>> the
>> > consensus, I wonder whether it is a good time to start the migration.
>> >
>> > Here are my understandings of the current status:
>> >
>> > 1. There is no more public updates of JDK6 since Feb 2013. Users no
>> longer
>> > get fixes of security vulnerabilities through official public updates.
>> > 2. Hadoop core is stuck with out-of-date dependency unless moving towards
>> > JDK7. (see
>> > http://hadoop.6.n7.nabble.com/very-old-dependencies-td71486.html)
>> > The implementation can also benefit from it thanks to the new
>> > functionalities in JDK7.
>> > 3. The code is ready for JDK7. Cloudera and Hortonworks have successful
>> > stories of supporting Hadoop on JDK7.
>> >
>> >
>> > It seems that the real work of moving to JDK7 is minimal. We only need to
>> > (1) make sure the jenkins are running on top of JDK7, and (2) to update
>> the
>> > minimum required Java version from 6 to 7. Therefore I propose that let's
>> > move towards JDK7 in trunk in the short term.
>> >
>> > Your feedbacks are appreciated.
>> >
>> > Regards,
>> > Haohui
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>>
>>
>> --
>> Alejandro
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemina

Re: [Important] Confirmation related to OpenSsl security issue

[Colin McCabe]

I took a quick glance at the build output, and I don't think openssl
is getting linked statically into libhadooppipes.a.

I see the following lines:

Linking CXX static library libhadooppipes.a
/usr/bin/cmake -P CMakeFiles/hadooppipes.dir/cmake_clean_target.cmake
/usr/bin/cmake -E cmake_link_script
CMakeFiles/hadooppipes.dir/link.txt --verbose=1
/usr/bin/ar cr libhadooppipes.a
CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o
/usr/bin/ranlib libhadooppipes.a

later on there are lines like this:

/usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_GNU_SOURCE
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
CMakeFiles/pipes-sort.dir/main/native/examples/impl/sort.cc.o  -o
examples/pipes-sort -rdynamic libhadooppipes.a libhadooputils.a -lssl
-lcrypto -lpthread

So when using libhadooppipes.a, you must supply your own copy of
libssl.so.  If you supply a vulnerable copy, you will be vulnerable.
If you supply a non-vulnerable copy, you won't be.  So I don't think
there is an impact on our build (unless I missed something here).

Just to make sure, it would be good if someone who uses
libhadooppipes.a to look at one of the versions in our release tarball
and verify that it works with the fixed openssl.

Colin

On Fri, Apr 11, 2014 at 2:27 AM, Vinayakumar B  wrote:
> Hi,
>
> Recently one security issue has been found in OpenSSL which has impacted so 
> many customers of different vendors.
>
> http://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD+Reference=720951&SearchOrder=4
>
> I want to ask, whether is there in impact of this on the Hadoop Release?
>
> Currently Hadoop-pipes are uses openssl-devel packages for building native 
> support.
>
> Can someone familiar with Hadoop-pipes can confirm whether is there any 
> impact of this security issue on builds of Hadoop built with defective 
> openssl?
>
> Regards,
>Vinay
>
> 
> This e-mail and attachments contain confidential information from HUAWEI,
> which is intended only for the person or entity whose address is listed
> above. Any use of the information contained herein in any way (including,
> but not limited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient's) is
> prohibited. If you receive this e-mail in error, please notify the sender by
> phone or email immediately and delete it!
>

Re: Plans of moving towards JDK7 in trunk

[Colin McCabe]

I think the bottom line here is that as long as our stable release
uses JDK6, there is going to be a very, very strong disincentive to
put any code which can't run on JDK6 into trunk.

Like I said earlier, the traditional reason for putting something in
trunk but not the stable release is that it needs more testing.  If a
stable release that drops support for JDK6 is more than a year away,
does it make sense to put anything in trunk like that?  What might
need more than a year of testing?  Certainly not changes to
LocalFileSystem to use the new APIs.  I also don't think an upgrade to
various libraries qualifies.

It might be best to shelve this for now, like we've done in the past,
until we're ready to talk about a stable release that requires JDK7+.
At least that's my feeling.

If we're really desperate for the new file APIs JDK7 provides, we
could consider using loadable modules for it in branch-2.  This is
similar to how we provide JNI versions of certain things on certain
platforms, without dropping support for the other platforms.

best,
Colin

On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata  wrote:
> There's an outstanding question addressed to me: "Are there particular
> features or new dependencies that you would like to contribute (or see
> contributed) that require using the Java 1.7 APIs?"  The question
> misses the point: We'd figure out how to write something we wanted to
> contribute to Hadoop against the APIs of Java4 if that's what it took
> to get them into a stable release.  And at current course and speed,
> that's how ridiculous things could get.
>
> To summarize, it seems like there's a vague consensus that it might be
> okay to eventually allow the use of Java7 in trunk, but there's no
> decision.  And there's been no answer to the concern that even if such
> dependencies were allowed in Java7, the only people using them would
> be people who uninterested in getting their patches into a stable
> release of Hadoop on any knowable timeframe, which doesn't bode well
> for the ability to stabilize that Java7 code when it comes time to
> attempt to.
>
> I don't have more to add, so I'll go back to lurking.  It'll be
> interesting to see where we'll be standing a year from now.
>
> On Sun, Apr 13, 2014 at 2:09 AM, Tsuyoshi OZAWA
>  wrote:
>> Hi,
>>
>> +1 for Karthik's idea(non-binding).
>>
>> IMO, we should keep the compatibility between JDK 6 and JDK 7 on both 
>> branch-1
>> and branch-2, because users can be using them. For future releases that we 
>> can
>> declare breaking compatibility(e.g. 3.0.0 release), we can use JDK 7
>> features if we
>> can get benefits. However, it can increase maintenance costs and distributes 
>> the
>> efforts of contributions to maintain branches. Then, I think it is
>> reasonable approach
>> that we use limited and minimum JDK-7 APIs when we have reasons we need to 
>> use
>> the features.
>> By the way, if we start to use JDK 7 APIs, we should declare the basis
>> when to use
>> JDK 7 APIs on Wiki not to confuse contributors.
>>
>> Thanks,
>> - Tsuyoshi
>>
>> On Wed, Apr 9, 2014 at 11:44 AM, Raymie Stata  wrote:
 It might make sense to try to enumerate the benefits of switching to
 Java7 APIs and dependencies.
>>>
>>>   - Java7 introduced a huge number of language, byte-code, API, and
>>> tooling enhancements!  Just to name a few: try-with-resources, newer
>>> and stronger encyrption methods, more scalable concurrency primitives.
>>>  See http://www.slideshare.net/boulderjug/55-things-in-java-7
>>>
>>>   - We can't update current dependencies, and we can't add cool new ones.
>>>
>>>   - Putting language/APIs aside, don't forget that a huge amount of effort
>>> goes into qualifying for Java6 (at least, I hope the folks claiming to
>>> support Java6 are putting in such an effort :-).  Wouldn't Hadoop
>>> users/customers be better served if qualification effort went into
>>> Java7/8 versus Java6/7?
>>>
>>> Getting to Java7 as a development env (and Java8 as a runtime env)
>>> seems like a no-brainer.  Question is: How?
>>>
>>> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza  wrote:
 It might make sense to try to enumerate the benefits of switching to Java7
 APIs and dependencies.  IMO, the ones listed so far on this thread don't
 make a compelling enough case to drop Java6 in branch-2 on any time frame,
 even if this means supporting Java6 through 2015.  For example, the change
 in RawLocalFileSystem semantics might be an incompatible change for
 branch-2 any way.


 On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla 
 wrote:

> +1 to NOT breaking compatibility in branch-2.
>
> I think it is reasonable to require JDK7 for trunk, if we limit use of
> JDK7-only API to security fixes etc. If we make other optimizations (like
> IO), it would be a pain to backport things to branch-2. I guess this all
> depends on when we see ourselves shipping Hadoop-3. Any ideas on that?
>
>
> On 

Re: HADOOP_ROOT_LOGGER

[Colin McCabe]

It's not always practical to edit the log4j.properties file.  For one
thing, if you're using a management system, there may be many log4j
properties sprinkled around the system, and it could be difficult to figure
out which is the one you need to edit.  For another, you may not (should
not?) have permission to do this on a production cluster.

Doing something like "HADOOP_ROOT_LOGGER="DEBUG,console" hadoop fs -cat
/foo" has helped me diagnose problems in the past.

best,
Colin


On Thu, May 22, 2014 at 6:34 AM, Robert Rati  wrote:

> In my experience the default HADOOP_ROOT_LOGGER definition will override
> any root logger defined in log4j.properties, which is where the problems
> have arisen.  If the HADOOP_ROOT_LOGGER definition in hadoop-config.sh were
> removed, wouldn't the root logger defined in the log4j.properties file be
> used?  Or do the client commands not read that configuration file?
>
> I'm trying to understand why the root logger should be defined outside of
> the log4j.properties file.
>
> Rob
>
>
> On 05/22/2014 12:53 AM, Vinayakumar B wrote:
>
>> Hi Robert,
>>
>> I understand your confusion.
>>
>> HADOOP_ROOT_LOGGER is set to default value "INFO,console" if it hasn't set
>> for anything and logs will be displayed on the console itself.
>> This will be true for any client commands you run. For ex: "hdfs dfs -ls
>> /"
>>
>> But for the server scripts (hadoop-daemon.sh, yarn-daemon.sh, etc)
>>   HADOOP_ROOT_LOGGER will be set to "INFO, RFA" if HADOOP_ROOT_LOGGER env
>> variable is not defined.
>> So that all the log messages of the server daemons goto some log files and
>> this will be maintained by RollingFileAppender. If you want to override
>> all
>> these default and set your own loglevel then define that as env variable
>> HADOOP_ROOT_LOGGER.
>>
>> For ex:
>> export HADOOP_ROOT_LOGGER="DEBUG,RFA"
>>export above env variable and then start server scripts or execute
>> client
>> commands, all logs goto files and will be maintained by
>> RollingFileAppender.
>>
>>
>> Regards,
>> Vinay
>>
>>
>> On Wed, May 21, 2014 at 6:42 PM, Robert Rati  wrote:
>>
>>  I noticed in hadoop-config.sh there is this line:
>>>
>>> HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.root.logger=${HADOOP_
>>> ROOT_LOGGER:-INFO,console}"
>>>
>>> which is setting a root logger if HADOOP_ROOT_LOGGER isn't set.  Why is
>>> this here.needed?  There is a log4j.properties file provided that
>>> defines a
>>> default logger.  I believe the line above will result in overriding
>>> whatever is set for the root logger in the log4j.properties file.  This
>>> has
>>> caused some confusion and hacks to work around this.
>>>
>>> Is there a reason not to remove the above code and just have all the
>>> logger definitions in the log4j.properties file?  Is there maybe a
>>> compatibility concern?
>>>
>>> Rob
>>>
>>>
>>
>>
>>

Re: Plans of moving towards JDK7 in trunk

[Colin McCabe]

I think we should come up with a plan for when the next Hadoop release
will drop support for JDK6.  We all know that day needs to come... the
only question is when.  I agree that writing the JDK7-only code
doesn't seem very productive unless we have a plan for when it will be
released and usable.

best,
Colin

On Tue, Jun 17, 2014 at 10:08 PM, Andrew Wang  wrote:
> Reviving this thread, I noticed there's been a patch and +1 on
> HADOOP-10530, and I don't think we actually reached a conclusion.
>
> I (and others) have expressed concerns about moving to JDK7 for trunk.
> Summarizing a few points:
>
> - We can't move to JDK7 in branch-2 because of compatibility
> - branch-2 is currently the only Hadoop release vehicle, there are no plans
> for a trunk-based Hadoop 3
> - Introducing JDK7-only APIs in trunk will increase divergence with
> branch-2 and make backports harder
> - Almost all developers care only about branch-2, since it is the only
> release vehicle
>
> With this in mind, I struggle to see any upsides to introducing JDK7-only
> APIs to trunk. Please let's not do anything on HADOOP-10530 or related
> until we agree on this.
>
> Thanks,
> Andrew
>
>
> On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran 
> wrote:
>
>> On 14 April 2014 17:46, Andrew Purtell  wrote:
>>
>> > How well is trunk tested? Does anyone deploy it with real applications
>> > running on top? When will the trunk codebase next be the basis for a
>> > production release? An impromptu diff of hadoop-common trunk against
>> > branch-2 as of today is 38,625 lines. Can they be said to be the same
>> > animal? I ask because any disincentive toward putting code in trunk is
>> > beside the point, if the only target worth pursuing today is branch-2
>> > unless one doesn't care if the code is released for production use.
>> > Questions on whither JDK6 or JDK7+ (or JRE6 versus JRE7+) only matter for
>> > the vast majority of Hadoopers if talking about branch-2.
>> >
>> >
>> I think its partly a timescale issue; its also because the 1-2 transition
>> was so significant, especially at the YARN layer, that it's still taking
>> time to trickle through.
>>
>> If you do want code to ship this year, branch-2 is where you are going to
>> try and get it in -and like you say, that's where things get tried in the
>> field. At the same time, the constraints of stability are holding us back
>> -already-.
>>
>> I don't see why we should have such another major 1-2 transition in future;
>> the rate that Arun is pushing out 2.x releases its almost back to the 0.1x
>> timescale -though at that point most people were fending for themselves and
>> expectations of stability were less. We do want smaller version increments
>> in future, which branch-2 is -mostly- delivering.
>>
>> While Java 7 doesn't have some must-have features, Java 8 is a significant
>> improvement in the language, and we should be looking ahead to that, maybe
>> even doing some leading-edge work on the side, so the same discussion
>> doesn't come up in two years time when java 7 goes EOL.
>>
>>
>> -steve
>>
>> (personal opinions only, etc, )
>>
>>
>> >
>> > On Mon, Apr 14, 2014 at 9:22 AM, Colin McCabe > > >wrote:
>> >
>> > > I think the bottom line here is that as long as our stable release
>> > > uses JDK6, there is going to be a very, very strong disincentive to
>> > > put any code which can't run on JDK6 into trunk.
>> > >
>> > > Like I said earlier, the traditional reason for putting something in
>> > > trunk but not the stable release is that it needs more testing.  If a
>> > > stable release that drops support for JDK6 is more than a year away,
>> > > does it make sense to put anything in trunk like that?  What might
>> > > need more than a year of testing?  Certainly not changes to
>> > > LocalFileSystem to use the new APIs.  I also don't think an upgrade to
>> > > various libraries qualifies.
>> > >
>> > > It might be best to shelve this for now, like we've done in the past,
>> > > until we're ready to talk about a stable release that requires JDK7+.
>> > > At least that's my feeling.
>> > >
>> > > If we're really desperate for the new file APIs JDK7 provides, we
>> > > could consider using loadable modules for it in branch-2.  This is
>> > > similar to how we provide JNI versions of certain things on c

Re: Plans of moving towards JDK7 in trunk

[Colin McCabe]

f code that have moved can't be backported.
>>
>> > >
>>
>> > > -Steve
>>
>> > >
>>
>> > >
>>
>> > >
>>
>> > >
>>
>> > >
>>
>> > > On 17 June 2014 22:08, Andrew Wang > andrew.w...@cloudera.com>>
>>
>> > wrote:
>>
>> > >
>>
>> > > > Reviving this thread, I noticed there's been a patch and +1 on
>>
>> > > > HADOOP-10530, and I don't think we actually reached a conclusion.
>>
>> > > >
>>
>> > > > I (and others) have expressed concerns about moving to JDK7 for
>> trunk.
>>
>> > > > Summarizing a few points:
>>
>> > > >
>>
>> > > > - We can't move to JDK7 in branch-2 because of compatibility
>>
>> > > > - branch-2 is currently the only Hadoop release vehicle, there are
>>
>> > > > no
>>
>> > > plans
>>
>> > > > for a trunk-based Hadoop 3
>>
>> > > > - Introducing JDK7-only APIs in trunk will increase divergence with
>>
>> > > > branch-2 and make backports harder
>>
>> > > > - Almost all developers care only about branch-2, since it is the
>>
>> > > > only release vehicle
>>
>> > > >
>>
>> > > > With this in mind, I struggle to see any upsides to introducing
>>
>> > > > JDK7-only APIs to trunk. Please let's not do anything on
>>
>> > > > HADOOP-10530 or related until we agree on this.
>>
>> > > >
>>
>> > > > Thanks,
>>
>> > > > Andrew
>>
>> > > >
>>
>> > > >
>>
>> > > > On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran
>>
>> > > > mailto:ste...@hortonworks.com>>
>>
>> > > > wrote:
>>
>> > > >
>>
>> > > > > On 14 April 2014 17:46, Andrew Purtell > <mailto:apurt...@apache.org>> wrote:
>>
>> > > > >
>>
>> > > > > > How well is trunk tested? Does anyone deploy it with real
>>
>> > > applications
>>
>> > > > > > running on top? When will the trunk codebase next be the basis
>>
>> > > > > > for a production release? An impromptu diff of hadoop-common
>>
>> > > > > > trunk against
>>
>> > > > > > branch-2 as of today is 38,625 lines. Can they be said to be the
>>
>> > > > > > same animal? I ask because any disincentive toward putting code
>>
>> > > > > > in trunk
>>
>> > > is
>>
>> > > > > > beside the point, if the only target worth pursuing today is
>>
>> > > > > > branch-2 unless one doesn't care if the code is released for
>>
>> > production use.
>>
>> > > > > > Questions on whither JDK6 or JDK7+ (or JRE6 versus JRE7+) only
>>
>> > > > > > matter
>>
>> > > > for
>>
>> > > > > > the vast majority of Hadoopers if talking about branch-2.
>>
>> > > > > >
>>
>> > > > > >
>>
>> > > > > I think its partly a timescale issue; its also because the 1-2
>>
>> > > transition
>>
>> > > > > was so significant, especially at the YARN layer, that it's still
>>
>> > > taking
>>
>> > > > > time to trickle through.
>>
>> > > > >
>>
>> > > > > If you do want code to ship this year, branch-2 is where you are
>>
>> > > > > going
>>
>> > > to
>>
>> > > > > try and get it in -and like you say, that's where things get tried
>>
>> > > > > in
>>
>> > > the
>>
>> > > > > field. At the same time, the constraints of stability are holding
>>
>> > > > > us
>>
>> > > back
>>
>> > > > > -already-.
>>
>> > > > >
>>
>> > > > > I don't see why we should have such another major 1-2 transition
>>
>> > > > > in
>>
>> > > > future;
>>
>> > > > > the rate that Arun is pushing out 2.x releases its 

Re: Plans of moving towards JDK7 in trunk

[Colin McCabe]

I think the important thing to do right now is to ensure our code
works with jdk8.  This is similar to the work we did last year to fix
issues that cropped up with jdk7.  There were just a few minor things
like the error in which it ran unit tests that caused issues.

I don't think it's out of the question to bump minVersion directly
from 1.6 to 1.8.  Java 7 is EOL in less than a year, after all... at
least according to the current roadmap here:
http://www.oracle.com/technetwork/java/eol-135779.html

We should try to do things in a way that causes the least disruption
for users upgrading clusters, if possible...

cheers,
Colin

On Thu, Jun 19, 2014 at 3:31 PM, Andrew Purtell  wrote:
> There are a number of security (algorithm, not vulnerability) and
> performance improvements that landed in 8, not 7. As a runtime for the
> performance conscious, it might be recommendable. I've come across GC
> issues in 6 or 7 where, talking with some Java platform folks, the first
> suggested course of action is try again with 8. Would be be more of the
> current moment if this discussion was about setting guidelines that
> prescribe when and when not to use 8+ language features, or concurrency
> library improvements that rely on intrinsics only available in the 8
> runtime? Has the Java 6 ship sailed? Just set the minimum supported JDK and
> runtime at 7 at next release? Use of the <> operator or multicatch wouldn't
> and shouldn't need be debated, they are quite minor. On the other hand, I
> would imagine discussion and debate on what 8+ language features might be
> useful to use at some future time could be a lively one.
>
>
>
> On Wed, Jun 18, 2014 at 3:03 PM, Colin McCabe 
> wrote:
>
>> In CDH5, Cloudera encourages people to use JDK7.  JDK6 has been EOL
>> for a while now and is not something we recommend.
>>
>> As we discussed before, everyone is in favor of upgrading to JDK7.
>> Every cluster operator of a reasonably modern Hadoop should do it
>> whatever distro or release you run.  As developers, we run JDK7 as
>> well.
>>
>> I'd just like to see a plan for when branch-2 (or some other branch)
>> will create a stable release that drops support for JDK1.6.  If we
>> don't have such a plan, I feel like it's too early to talk about this
>> stuff.
>>
>> If we drop support for 1.6 in trunk but not in branch-2, we are
>> fragmenting the project.  People will start writing unreleaseable code
>> (because it doesn't work on branch-2) and we'll be back to the bad old
>> days of Hadoop version fragmentation that branch-2 was intended to
>> solve.  Backports will become harder.  The biggest problem is that
>> trunk will start to depend on libraries or Maven plugins that branch-2
>> can't even use, because they're JDK7+-only.
>>
>> Steve wrote: "if someone actually did file a bug on something on
>> branch-2 which didn't work on Java 6 but went away on Java7+, we'd
>> probably close it as a WORKSFORME".
>>
>> Steve, if this is true, we should just bump the minimum supported
>> version for branch-2 to 1.7 today and resolve this.  If we truly
>> believe that there are no issues here, then let's just decide to drop
>> 1.6 in a specific future release of Hadoop 2.  If there are issues
>> with releasing JDK1.7+ only code, then let's figure out what they are
>> before proceeding.
>>
>> best,
>> Colin
>>
>>
>> On Wed, Jun 18, 2014 at 1:41 PM, Sandy Ryza 
>> wrote:
>> > We do release warnings when we are aware of vulnerabilities in our
>> > dependencies.
>> >
>> > However, unless I'm grossly misunderstanding, the vulnerability that you
>> > point out is not a vulnerability within the context of our software.
>> >  Hadoop doesn't try to sandbox within JVMs.  In a secure setup, any JVM
>> > running non-trusted user code is running as that user, so "breaking out"
>> > doesn't offer the ability to do anything malicious.
>> >
>> > -Sandy
>> >
>> > On Wed, Jun 18, 2014 at 1:30 PM, Ottenheimer, Davi <
>> davi.ottenhei...@emc.com
>> >> wrote:
>> >
>> >> Andrew,
>> >>
>> >>
>> >>
>> >> “I don't see any point to switching” is an interesting perspective,
>> given
>> >> the well-known risks of running unsafe software. Clearly customer best
>> >> interest is stability. JDK6 is in a known unsafe state. The longer
>> anyone
>> >> delays the necessary transition to safety the longer the door is left

Re: Plans of moving towards JDK7 in trunk

[Colin McCabe]

Er, that should read "order in which it ran unit tests."

C.

On Fri, Jun 20, 2014 at 11:02 AM, Colin McCabe  wrote:
> I think the important thing to do right now is to ensure our code
> works with jdk8.  This is similar to the work we did last year to fix
> issues that cropped up with jdk7.  There were just a few minor things
> like the error in which it ran unit tests that caused issues.
>
> I don't think it's out of the question to bump minVersion directly
> from 1.6 to 1.8.  Java 7 is EOL in less than a year, after all... at
> least according to the current roadmap here:
> http://www.oracle.com/technetwork/java/eol-135779.html
>
> We should try to do things in a way that causes the least disruption
> for users upgrading clusters, if possible...
>
> cheers,
> Colin
>
> On Thu, Jun 19, 2014 at 3:31 PM, Andrew Purtell  wrote:
>> There are a number of security (algorithm, not vulnerability) and
>> performance improvements that landed in 8, not 7. As a runtime for the
>> performance conscious, it might be recommendable. I've come across GC
>> issues in 6 or 7 where, talking with some Java platform folks, the first
>> suggested course of action is try again with 8. Would be be more of the
>> current moment if this discussion was about setting guidelines that
>> prescribe when and when not to use 8+ language features, or concurrency
>> library improvements that rely on intrinsics only available in the 8
>> runtime? Has the Java 6 ship sailed? Just set the minimum supported JDK and
>> runtime at 7 at next release? Use of the <> operator or multicatch wouldn't
>> and shouldn't need be debated, they are quite minor. On the other hand, I
>> would imagine discussion and debate on what 8+ language features might be
>> useful to use at some future time could be a lively one.
>>
>>
>>
>> On Wed, Jun 18, 2014 at 3:03 PM, Colin McCabe 
>> wrote:
>>
>>> In CDH5, Cloudera encourages people to use JDK7.  JDK6 has been EOL
>>> for a while now and is not something we recommend.
>>>
>>> As we discussed before, everyone is in favor of upgrading to JDK7.
>>> Every cluster operator of a reasonably modern Hadoop should do it
>>> whatever distro or release you run.  As developers, we run JDK7 as
>>> well.
>>>
>>> I'd just like to see a plan for when branch-2 (or some other branch)
>>> will create a stable release that drops support for JDK1.6.  If we
>>> don't have such a plan, I feel like it's too early to talk about this
>>> stuff.
>>>
>>> If we drop support for 1.6 in trunk but not in branch-2, we are
>>> fragmenting the project.  People will start writing unreleaseable code
>>> (because it doesn't work on branch-2) and we'll be back to the bad old
>>> days of Hadoop version fragmentation that branch-2 was intended to
>>> solve.  Backports will become harder.  The biggest problem is that
>>> trunk will start to depend on libraries or Maven plugins that branch-2
>>> can't even use, because they're JDK7+-only.
>>>
>>> Steve wrote: "if someone actually did file a bug on something on
>>> branch-2 which didn't work on Java 6 but went away on Java7+, we'd
>>> probably close it as a WORKSFORME".
>>>
>>> Steve, if this is true, we should just bump the minimum supported
>>> version for branch-2 to 1.7 today and resolve this.  If we truly
>>> believe that there are no issues here, then let's just decide to drop
>>> 1.6 in a specific future release of Hadoop 2.  If there are issues
>>> with releasing JDK1.7+ only code, then let's figure out what they are
>>> before proceeding.
>>>
>>> best,
>>> Colin
>>>
>>>
>>> On Wed, Jun 18, 2014 at 1:41 PM, Sandy Ryza 
>>> wrote:
>>> > We do release warnings when we are aware of vulnerabilities in our
>>> > dependencies.
>>> >
>>> > However, unless I'm grossly misunderstanding, the vulnerability that you
>>> > point out is not a vulnerability within the context of our software.
>>> >  Hadoop doesn't try to sandbox within JVMs.  In a secure setup, any JVM
>>> > running non-trusted user code is running as that user, so "breaking out"
>>> > doesn't offer the ability to do anything malicious.
>>> >
>>> > -Sandy
>>> >
>>> > On Wed, Jun 18, 2014 at 1:30 PM, Ottenheimer, Davi <
>>> davi.ottenhei...@emc.com
>>> >

Re: Porting Hadoop to FreeBSD

[Colin McCabe]

Thanks for working on this, Dmitry.  It's good to see Hadoop support
another platform.  The code changes look pretty minor, too.

best,
Colin

On Tue, Jul 8, 2014 at 7:08 AM, Dmitry Sivachenko  wrote:
> Hello,
>
> I am trying to make hadoop usable on FreeBSD OS.  Following Steve Loughran's 
> suggestion, I created HADOOP-10796 to accumulate links to other issues 
> related to this subject.
> I added links to (almost) all patches needed to achieve this goal.  Right now 
> I have a small cluster running Hadoop-2.4.1 on FreeBSD and passing basic 
> tests (TeraSort mainly).
>
> I am looking forward for cooperation with you to include all relevant changes 
> into hadoop distribution.
> Please do not hesitate to contact me if you are not satisfied with the 
> proposed solutions.
>
> Thanks in advance!

Re: [jira] [Created] (HADOOP-10795) unale to build hadoop 2.4.1(redhat5.8 x64)

[Colin McCabe]

A unit test failed.  See my response on JIRA.

best,
Colin

On Tue, Jul 8, 2014 at 3:11 PM, Jay Vyas  wrote:
> these appear to be java errors related use to your jdk?
> maybe your JDK doesnt match up well with your OS.
> Consider trying red hat 6+ or Fedora 20?
>
>
> On Jul 8, 2014, at 5:45 AM, "moses.wang (JIRA)"  wrote:
>
>> moses.wang created HADOOP-10795:
>> ---
>>
>> Summary: unale to build hadoop 2.4.1(redhat5.8 x64)
>> Key: HADOOP-10795
>> URL: https://issues.apache.org/jira/browse/HADOOP-10795
>> Project: Hadoop Common
>>  Issue Type: Bug
>>  Components: build
>> Environment: OS version rehat 5.8 x64
>> maven version 3.3.1
>> java version jdk 1.7_15 for x64
>>Reporter: moses.wang
>>
>>
>> unale to build hadoop 2.4.1(redhat5.8 x64)
>> WARNING] Some problems were encountered while building the effective model 
>> for org.apache.hadoop:hadoop-project:pom:2.4.1
>> [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but 
>> found duplicate declaration of plugin 
>> org.apache.maven.plugins:maven-enforcer-plugin @ line 1015, column 15
>>
>> [WARNING] Some problems were encountered while building the effective model 
>> for org.apache.hadoop:hadoop-common:jar:2.4.1
>> [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but 
>> found duplicate declaration of plugin 
>> org.apache.maven.plugins:maven-surefire-plugin @ line 479, column 15
>> [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but 
>> found duplicate declaration of plugin 
>> org.apache.maven.plugins:maven-enforcer-plugin @ 
>> org.apache.hadoop:hadoop-project:2.4.1, 
>> /home/software/Server/hadoop-2.4.1-src/hadoop-project/pom.xml, line 1015, 
>> column 15
>>
>> WARNING] 
>> /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/FastByteComparisons.java:[25,15]
>>  Unsafe is internal proprietary API and may be removed in a future release
>> [WARNING] 
>> /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:[42,15]
>>  Unsafe is internal proprietary API and may be removed in a future release
>> [WARNING] 
>> /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SignalLogger.java:[21,15]
>>  Signal is internal proprietary API and may be removed in a future release
>> [WARNING] 
>> /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SignalLogger.java:[22,15]
>>  SignalHandler is internal proprietary API and may be removed in a future 
>> release
>>
>> [WARNING] 
>> /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:[22,24]
>>  AlgorithmId is internal proprietary API and may be removed in a future 
>> release
>> [WARNING] 
>> /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:[23,24]
>>  CertificateAlgorithmId is internal proprietary API and may be removed in a 
>> future release
>>
>> testROBufferDirAndRWBufferDir[1](org.apache.hadoop.fs.TestLocalDirAllocator) 
>>  Time elapsed: 0.014 sec  <<< FAILURE!
>>
>> [ERROR] Failed to execute goal 
>> org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
>> project hadoop-common: There are test failures.
>> [ERROR]
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.2#6252)
>

Re: [DISCUSS] Assume Private-Unstable for classes that are not annotated

[Colin McCabe]

+1.

Colin


On Tue, Jul 22, 2014 at 2:54 PM, Karthik Kambatla 
wrote:

> Hi devs
>
> As you might have noticed, we have several classes and methods in them that
> are not annotated at all. This is seldom intentional. Avoiding incompatible
> changes to all these classes can be considerable baggage.
>
> I was wondering if we should add an explicit disclaimer in our
> compatibility guide that says, "Classes without annotations are to
> considered @Private"
>
> For methods, is it reasonable to say - "Class members without specific
> annotations inherit the annotations of the class"?
>
> Thanks
> Karthik
>

Re: Code guidelines and bash

[Colin McCabe]

On Tue, Jul 29, 2014 at 2:45 AM, 俊平堵  wrote:
> Sun's java code convention (published in year of 97) suggest 80 column per
> line for old-style terminals. It sounds pretty old, However, I saw some
> developers (not me :)) like to open multiple terminals in one screen for
> coding/debugging so 80-colum could be just fit. Google's java convention (
> https://google-styleguide.googlecode.com/svn/trunk/javaguide.html#s4.4-column-limit)
> shows some flexibility here with 80 or 100 column (and some exception
> cases).
> Like Chris mentioned early, I think this 80-column should just be a general
> guideline but not a strict limit - we can break it if it hurts legibility
> of code reading.
> btw, some research work found that CPL (characters per line) only had small
> effects on readability for news, including factors of speed and
> comprehension (
> http://psychology.wichita.edu/surl/usabilitynews/72/LineLength.asp). Not
> sure if reading code is the same (assume break lines properly).

There is a lot of contradictory research in this area.  For example,
http://eric.ed.gov/?id=EJ749012 talks about 70 characters per line as
"ideal."

I think a lot of these studies don't really translate very well to
code.  (A lot of them are college students seeing how quickly they can
read a news article.)  Code with extremely long line lengths tends to
have super-deep nesting, which makes it hard to keep track of what is
going on (the so-called "arrow anti-pattern").  This is especially
true when there are break and continue statements involved.
Super-long lines make diffs very difficult to do.  And it's just
unpleasant to read, forcing users to choose between horizontal
scrolling or tiny text...

Maybe it makes sense to extend the bash line length, though, if it's
tough to fit in 80 chars.  Bash is whitespace sensitive and doing the
line continuation thing is a pain.  Another option might be renaming
some variables, or using temp variables with shorter names...

best,
Colin


>
>
> 2014-07-29 15:24 GMT+08:00 Andrew Purtell :
>
>> On Mon, Jul 28, 2014 at 12:05 PM, Doug Cutting  wrote:
>>
>> > On Sun, Jul 27, 2014 at 9:28 PM, Ted Dunning 
>> > wrote:
>> > > I don't know of any dev environments in common use today that can't
>> > display >100 characters.
>> >
>> > I edit in an 80-column Emacs window that just fits beside an 80-column
>> > shell window on a portrait-rotated 24" monitor.
>> >
>>
>> You win the Internet today, Old School category! (smile)
>>
>>
>> --
>> Best regards,
>>
>>- Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>

Re: [VOTE] Migration from subversion to git for version control

[Colin McCabe]

+1.

best,
Colin

On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla  wrote:
> I have put together this proposal based on recent discussion on this topic.
>
> Please vote on the proposal. The vote runs for 7 days.
>
>1. Migrate from subversion to git for version control.
>2. Force-push to be disabled on trunk and branch-* branches. Applying
>changes from any of trunk/branch-* to any of branch-* should be through
>"git cherry-pick -x".
>3. Force-push on feature-branches is allowed. Before pulling in a
>feature, the feature-branch should be rebased on latest trunk and the
>changes applied to trunk through "git rebase --onto" or "git cherry-pick
>".
>4. Every time a feature branch is rebased on trunk, a tag that
>identifies the state before the rebase needs to be created (e.g.
>tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once
>the feature is pulled into trunk and the tags are no longer useful.
>5. The relevance/use of tags stay the same after the migration.
>
> Thanks
> Karthik
>
> PS: Per Andrew Wang, this should be a "Adoption of New Codebase" kind of
> vote and will be Lazy 2/3 majority of PMC members.

Re: Hortonworks scripting ...

[Colin McCabe]

This mailing list is for questions about Apache Hadoop, not commercial
Hadoop distributions.  Try asking a Hortonworks-specific mailing list.

best,
Colin

On Thu, Aug 14, 2014 at 3:23 PM, Niels Basjes  wrote:
> Hi,
>
> In the core Hadoop you can on your (desktop) client have multiple clusters
> available simply by having multiple directories with setting files (i.e.
> core-site.xml etc.) and select the one you want by changing the environment
> settings (i.e. HADOOP_CONF_DIR and such) around.
>
> This doesn't work when I run under the Hortonworks 2.1.2 distribution.
>
> There I find that in all of the scripts placed in /usr/bin/ there is
> "mucking about" with the environment settings. Things from /etc/default are
> sourced and they override my settings.
> Now I can control part of it by directing the BIGTOP_DEFAULTS_DIR into a
> blank directory.
> But in /usr/bin/pig sourcing /etc/default/hadoop hardcoded into the script.
>
> Why is this done this way?
>
> P.S. Where is the git(?) repo located where this (apperently HW specific)
> scripting is maintained?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes

Re: [DISCUSS] Switch to log4j 2

[Colin McCabe]

On Fri, Aug 15, 2014 at 8:50 AM, Aaron T. Myers  wrote:
> Not necessarily opposed to switching logging frameworks, but I believe we
> can actually support async logging with today's logging system if we wanted
> to, e.g. as was done for the HDFS audit logger in this JIRA:
>
> https://issues.apache.org/jira/browse/HDFS-5241

Yes, this is a great example of making something async without
switching logging frameworks.  +1 for doing that where it is
appropriate.

>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>
>
> On Fri, Aug 15, 2014 at 5:44 AM, Steve Loughran 
> wrote:
>
>> moving to SLF4J as an API is independent —it's just a better API for
>> logging than commons-logging, was already a dependency and doesn't force
>> anyone to switch to a new log back end.

Interesting idea.  Did anyone do a performance comparison and/or API
comparison with SLF4j on Hadoop?

>>
>>
>> On 15 August 2014 03:34, Tsuyoshi OZAWA  wrote:
>>
>> > Hi,
>> >
>> > Steve has started discussion titled "use SLF4J APIs in new modules?"
>> > as a related topic.
>> >
>> >
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201404.mbox/%3cca+4kjvv_9cmmtdqzcgzy-chslyb1wkgdunxs7wrheslwbuh...@mail.gmail.com%3E
>> >
>> > It sounds good to me to use asynchronous logging when we log INFO. One

-1.  Async logging for everything will make a lot of failures
un-debuggable.  Just to give one example, what if you get a JVM out of
memory crash?  You'll lose the last few log messages which could have
told you what was going on.  Even if the JVM doesn't terminate, log
messages will be out of order, which is annoying, and will make
debugging harder.

The kernel already buffers the log files in memory.  Not every log
message generates a disk seek.  But on the other hand, if the JVM
process crashes, you've got everything.  In other words, we've already
got as much buffering and asynchronicity as we need!

If the problem is that the noisy logs are overloading the disk
bandwidth, that problem can't be solved by adding Java-level async.
You need more bandwidth.  A simple way of doing this is putting the
log partition on /dev/shm.  We could also look into stripping some of
the boilerplate from log messages-- there are a lot of super-long log
messages that could be much more concise.  Other Java logging
frameworks might have less overhead (I'm not an expert on this, but
maybe someone could post some numbers?)

best,
Colin


>> > concern is that asynchronous logging makes debugging difficult - I
>> > don't know log4j 2 well, but I suspect that ordering of logging can be
>> > changed even if WARN or  FATAL are logged with synchronous logger.
>> >
>> > Thanks,
>> > - Tsuyoshi
>> >
>> > On Thu, Aug 14, 2014 at 6:44 AM, Arpit Agarwal > >
>> > wrote:
>> > > I don't recall whether this was discussed before.
>> > >
>> > > I often find our INFO logging to be too sparse for useful diagnosis. A
>> > high
>> > > performance logging framework will encourage us to log more.
>> > Specifically,
>> > > Asynchronous Loggers look interesting.
>> > > https://logging.apache.org/log4j/2.x/manual/async.html#Performance
>> > >
>> > > What does the community think of switching to log4j 2 in a Hadoop 2.x
>> > > release?
>> > >
>> > > --
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the individual or
>> entity
>> > to
>> > > which it is addressed and may contain information that is confidential,
>> > > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > > of this message is not the intended recipient, you are hereby notified
>> > that
>> > > any printing, copying, dissemination, distribution, disclosure or
>> > > forwarding of this communication is strictly prohibited. If you have
>> > > received this communication in error, please contact the sender
>> > immediately
>> > > and delete it from your system. Thank You.
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>

Re: Updates on migration to git

[Colin McCabe]

Thanks for making this happen, Karthik and Daniel.  Great job.

best,
Colin

On Tue, Aug 26, 2014 at 5:59 PM, Karthik Kambatla  wrote:
> Yes, we have requested for force-push disabled on trunk and branch-*
> branches. I didn't test it though :P, it is not writable yet.
>
>
> On Tue, Aug 26, 2014 at 5:48 PM, Todd Lipcon  wrote:
>
>> Hey Karthik,
>>
>> Just to confirm, have we disabled force-push support on the repo?
>>
>> In my experience, especially when a project has committers new to git,
>> force-push support causes more trouble than it's worth.
>>
>> -Todd
>>
>>
>> On Tue, Aug 26, 2014 at 4:39 PM, Karthik Kambatla 
>> wrote:
>>
>> > Looks like our git repo is good to go.
>> >
>> > On INFRA-8195, I am asking Daniel to enable writing to it. In case you
>> find
>> > any issues, please comment on the JIRA.
>> >
>> > Thanks
>> > Karthik
>> >
>> >
>> > On Tue, Aug 26, 2014 at 3:28 PM, Arpit Agarwal > >
>> > wrote:
>> >
>> > > I cloned the new repo, built trunk and branch-2, verified all the
>> > branches
>> > > are present. Also checked a few branches and the recent commit history
>> > > matches our existing repo. Everything looks good so far.
>> > >
>> > >
>> > > On Tue, Aug 26, 2014 at 1:19 PM, Karthik Kambatla 
>> > > wrote:
>> > >
>> > > > The git repository is now ready for inspection. I ll take a look
>> > shortly,
>> > > > but it would be great if a few others could too.
>> > > >
>> > > > Once we are okay with it, we can ask it to be writable.
>> > > >
>> > > > On Tuesday, August 26, 2014, Karthik Kambatla 
>> > > wrote:
>> > > >
>> > > > > Hi Suresh
>> > > > >
>> > > > > There was one vote thread on whether to migrate to git, and the
>> > > > > implications to the commit process for individual patches and
>> feature
>> > > > > branches -
>> > > > >
>> > >
>> https://www.mail-archive.com/common-dev@hadoop.apache.org/msg13447.html
>> > > > .
>> > > > > Prior to that, there was a discuss thread on the same topic.
>> > > > >
>> > > > > As INFRA handles the actual migration from subversion to git, the
>> > vote
>> > > > > didn't include those specifics. The migration is going on as we
>> speak
>> > > > (See
>> > > > > INFRA-8195). The initial expectation was that the migration would
>> be
>> > > done
>> > > > > in a few hours, but it has been several hours and the last I heard
>> > the
>> > > > > import was still running.
>> > > > >
>> > > > > I have elaborated on the points in the vote thread and drafted up a
>> > > wiki
>> > > > > page on how-to-commit -
>> > > > https://wiki.apache.org/hadoop/HowToCommitWithGit
>> > > > > . We can work on improving this further and call a vote thread on
>> > those
>> > > > > items if need be.
>> > > > >
>> > > > > Thanks
>> > > > > Karthik
>> > > > >
>> > > > >
>> > > > > On Tue, Aug 26, 2014 at 11:41 AM, Suresh Srinivas <
>> > > > sur...@hortonworks.com
>> > > > > > wrote:
>> > > > >
>> > > > >> Karthik,
>> > > > >>
>> > > > >> I would like to see detailed information on how this migration
>> will
>> > be
>> > > > >> done, how it will affect the existing project and commit process.
>> > This
>> > > > >> should be done in a document that can be reviewed instead of in an
>> > > email
>> > > > >> thread on an ad-hoc basis. Was there any voting on this in PMC and
>> > > > should
>> > > > >> we have a vote to ensure everyone is one the same page on doing
>> this
>> > > and
>> > > > >> how to go about it?
>> > > > >>
>> > > > >> Regards,
>> > > > >> Suresh
>> > > > >>
>> > > > >>
>> > > > >> On Tue, Aug 26, 2014 at 9:17 AM, Karthik Kambatla <
>> > ka...@cloudera.com
>> > > > >> >
>> > > > >> wrote:
>> > > > >>
>> > > > >> > Last I heard, the import is still going on and appears closer to
>> > > > getting
>> > > > >> > done. Thanks for your patience with the migration.
>> > > > >> >
>> > > > >> > I ll update you as and when there is something. Eventually, the
>> > git
>> > > > repo
>> > > > >> > should be at the location in the wiki.
>> > > > >> >
>> > > > >> >
>> > > > >> > On Mon, Aug 25, 2014 at 3:45 PM, Karthik Kambatla <
>> > > ka...@cloudera.com
>> > > > >> >
>> > > > >> > wrote:
>> > > > >> >
>> > > > >> > > Thanks for bringing these points up, Zhijie.
>> > > > >> > >
>> > > > >> > > By the way, a revised How-to-commit wiki is at:
>> > > > >> > > https://wiki.apache.org/hadoop/HowToCommitWithGit . Please
>> feel
>> > > > free
>> > > > >> to
>> > > > >> > > make changes and improve it.
>> > > > >> > >
>> > > > >> > > On Mon, Aug 25, 2014 at 11:00 AM, Zhijie Shen <
>> > > > zs...@hortonworks.com
>> > > > >> >
>> > > > >> > > wrote:
>> > > > >> > >
>> > > > >> > >> Do we have any convention about "user.name" and
>> "user.email"?
>> > > For
>> > > > >> > >> example,
>> > > > >> > >> we'd like to use @apache.org for the email.
>> > > > >> > >>
>> > > > >> > >
>> > > > >> > > May be, we can ask people to use project-specific configs here
>> > and
>> > > > use
>> > > > >> > > their real name and @apache.org address.
>> > > > >> > >
>> > > > >> > > Is there any downside to le

Re: migrating private branches to the new git repo

[Colin McCabe]

+1 for using git log instead of CHANGES.txt.

Colin

On Wed, Sep 3, 2014 at 11:07 AM, Chris Douglas  wrote:
> On Tue, Sep 2, 2014 at 2:38 PM, Andrew Wang  wrote:
>> Not to derail the conversation, but if CHANGES.txt is making backports more
>> annoying, why don't we get rid of it? It seems like we should be able to
>> generate it via a JIRA query, and "git log" can also be used for a quick
>> check (way faster than svn log).
>
> +1
>
> As long as release notes and incompatible changes are recorded in each
> branch, we gain no accuracy by maintaining this manually. Commit
> messages that record the merge revisions instead of the change are
> similarly hateful. -C

Re: Git repo ready to use

[Colin McCabe]

It's an issue with test-patch.sh.  See
https://issues.apache.org/jira/browse/HADOOP-11084

best,
Colin

On Mon, Sep 8, 2014 at 3:38 PM, Andrew Wang  wrote:
> We're still not seeing findbugs results show up on precommit runs. I see
> that we're archiving "../patchprocess/*", and Ted thinks that since it's
> not in $WORKSPACE it's not getting picked up. Can we get confirmation of
> this issue? If so, we could just add "patchprocess" to the toplevel
> .gitignore.
>
> On Thu, Sep 4, 2014 at 8:54 AM, Sangjin Lee  wrote:
>
>> That's good to know. Thanks.
>>
>>
>> On Wed, Sep 3, 2014 at 11:15 PM, Vinayakumar B 
>> wrote:
>>
>> > I think its still pointing to old svn repository which is just read only
>> > now.
>> >
>> > You can use latest mirror:
>> > https://github.com/apache/hadoop
>> >
>> > Regards,
>> > Vinay
>> > On Sep 4, 2014 11:37 AM, "Sangjin Lee"  wrote:
>> >
>> > > It seems like the github mirror at
>> > https://github.com/apache/hadoop-common
>> > > has stopped getting updates as of 8/22. Could this mirror have been
>> > broken
>> > > by the git transition?
>> > >
>> > > Thanks,
>> > > Sangjin
>> > >
>> > >
>> > > On Fri, Aug 29, 2014 at 11:51 AM, Ted Yu  wrote:
>> > >
>> > > > From https://builds.apache.org/job/Hadoop-hdfs-trunk/1854/console :
>> > > >
>> > > > ERROR: No artifacts found that match the file pattern
>> > > > "trunk/hadoop-hdfs-project/*/target/*.tar.gz". Configuration
>> > > > error?ERROR :
>> > > > ?trunk/hadoop-hdfs-project/*/target/*.tar.gz? doesn?t match anything,
>> > > > but ?hadoop-hdfs-project/*/target/*.tar.gz? does. Perhaps that?s what
>> > > > you mean?
>> > > >
>> > > >
>> > > > I corrected the path to hdfs tar ball.
>> > > >
>> > > >
>> > > > FYI
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Aug 29, 2014 at 8:48 AM, Alejandro Abdelnur <
>> t...@cloudera.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > it seems we missed updating the HADOOP precommit job to use Git, it
>> > was
>> > > > > still using SVN. I've just updated it.
>> > > > >
>> > > > > thx
>> > > > >
>> > > > >
>> > > > > On Thu, Aug 28, 2014 at 9:26 PM, Ted Yu 
>> wrote:
>> > > > >
>> > > > > > Currently patchprocess/ (contents shown below) is one level
>> higher
>> > > than
>> > > > > > ${WORKSPACE}
>> > > > > >
>> > > > > > diffJavadocWarnings.txt
>> > > >  newPatchFindbugsWarningshadoop-hdfs.html
>> > > > > >  patchFindBugsOutputhadoop-hdfs.txt
>> patchReleaseAuditOutput.txt
>> > > > > >  trunkJavadocWarnings.txt
>> > > > > > filteredPatchJavacWarnings.txt
>> > > newPatchFindbugsWarningshadoop-hdfs.xml
>> > > > > > patchFindbugsWarningshadoop-hdfs.xml
>> patchReleaseAuditWarnings.txt
>> > > > > > filteredTrunkJavacWarnings.txt  patch
>> > > > > > patchJavacWarnings.txttestrun_hadoop-hdfs.txt
>> > > > > > jirapatchEclipseOutput.txt
>> > > > > >  patchJavadocWarnings.txt  trunkJavacWarnings.txt
>> > > > > >
>> > > > > > Under Files to archive input box of
>> > PreCommit-HDFS-Build/configure, I
>> > > > > saw:
>> > > > > >
>> > > > > > '../patchprocess/*' doesn't match anything, but '*' does. Perhaps
>> > > > that's
>> > > > > > what you mean?
>> > > > > >
>> > > > > > I guess once patchprocess is moved back under ${WORKSPACE}, a lot
>> > of
>> > > > > things
>> > > > > > would be back to normal.
>> > > > > >
>> > > > > > Cheers
>> > > > > >
>> > > > > > On Thu, Aug 28, 2014 at 9:16 PM, Alejandro Abdelnur <
>> > > t...@cloudera.com
>> > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > i'm also seeing broken links for javadocs warnings.
>> > > > > > >
>> > > > > > > Alejandro
>> > > > > > > (phone typing)
>> > > > > > >
>> > > > > > > > On Aug 28, 2014, at 20:00, Andrew Wang <
>> > andrew.w...@cloudera.com
>> > > >
>> > > > > > wrote:
>> > > > > > > >
>> > > > > > > > I noticed that the JUnit test results aren't getting picked
>> up
>> > > > > > anymore. I
>> > > > > > > > suspect we just need to update the path to the surefire
>> output,
>> > > but
>> > > > > > based
>> > > > > > > > on a quick examination I'm not sure what that is.
>> > > > > > > >
>> > > > > > > > Does someone mind taking another look?
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Thu, Aug 28, 2014 at 4:21 PM, Karthik Kambatla <
>> > > > > ka...@cloudera.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >> Thanks Giri and Ted for fixing the builds.
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >>> On Thu, Aug 28, 2014 at 9:49 AM, Ted Yu <
>> yuzhih...@gmail.com
>> > >
>> > > > > wrote:
>> > > > > > > >>>
>> > > > > > > >>> Charles:
>> > > > > > > >>> QA build is running for your JIRA:
>> > > > > > > >>>
>> > > > >
>> https://builds.apache.org/job/PreCommit-hdfs-Build/7828/parameters/
>> > > > > > > >>>
>> > > > > > > >>> Cheers
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > >  On Thu, Aug 28, 2014 at 9:41 AM, Charles Lamb <
>> > > > cl...@cloudera.com
>> > > > > >
>> > > > > > > >>> wrote:
>> > > > > >

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

[Colin McCabe]

On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer  wrote:
>
> It’s now September.  With the passage of time, I have a lot of doubts 
> about this plan and where that trajectory takes us.
>
> * The list of changes that are already in branch-2 scare the crap out of any 
> risk adverse person (Hello to my fellow operations people!). Not only are the 
> number of changes extremely high, but in addition there are a lot of major, 
> blockbuster features in what is supposed to be a minor release.  Combined 
> with the fact that we’ve had to do some micro releases, it seems to hint that 
> branch-2 is getting less stable over time.

I don't see what is so scary about 2.6, can you be more concrete?  It
seems like a pretty normal release to me and most of the new features
are optional.

I also don't see why you think that "branch-2 is getting less stable
over time."  Actually, I think that branch-2 has gotten more stable
over time as people have finally gotten around to upgrading from 1.x
or earlier, and contributed their efforts to addressing regressions in
branch-2.

> *  One of the plans talked about was rolling a 2.7 release that drops JDK6 
> and makes JDK7 the standard.  If 2.7 comes after 2.6 in October, date wise  
> makes it somewhere around January 2015.  JDK7 EOL’s in April 2015.  So we’ll 
> have a viable JDK7 release for exactly 3 months.  Frankly, it is too late for 
> us to talk about JDK7 and we need to start thinking about JDK8.
>
> * trunk is currently sitting at 3 years old.  There is a lot of stuff that 
> has been hanging around that really needs to get into people hands so that we 
> can start stabilizing it for a “real” release.

We have been pretty careful about minimizing trunk's divergence from
branch-2.  I can't think of an example of anything in trunk that
"really needs to get into people's hands"-- did I forget something?

>
>
> To me this all says one thing:
>
> Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha 
> with JDK8 as the minimum.  2.5.1 becomes the base for all sustaining work.  
> This gives the rest of the community time to move to JDK8 if they haven’t 
> already.  For downstream vendors, it gives a roadmap for their customers who 
> will be asking about JDK8 sooner rather than later.  By the time 3.0 
> stabilizes, we’re probably looking at April, which is perfect timing.
>
> One of the issues I’ve heard mention is that 3.0 doesn’t have 
> anything “compelling” in it.  Well, dropping 2.6 makes the feature list the 
> carrot, JDK8 support is obviously the stick.
>
> Thoughts?

As we've discussed before, supporting JDK8 is very different from
forcing people to use JDK8.  branch-2 and Hadoop 2.6 most certainly
should support JDK8, and most certainly NOT force people to use JDK8.
Cloudera has been using JDK7 internally for a long time, and
recommending it to customers too.  Some developers are using JDK8 as
well.  It works fine (although I'm sure there will be bugs and
workarounds that get reported and fixed as more people migrate).  I
don't see this particular issue as a reason to change the schedule.

best,
Colin


>
>
>
>
> On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan  wrote:
>
>> Thanks for initiating the thread Arun.
>>
>> Can we add YARN-1051  to
>> the list? We have most of the patches for the sub-JIRAs under review and
>> have committed a couple.
>>
>> -Subru
>>
>> -- Forwarded message --
>>
>> From: Arun C Murthy 
>>
>> Date: Tue, Aug 12, 2014 at 1:34 PM
>>
>> Subject: Thinking ahead to hadoop-2.6
>>
>> To: "common-dev@hadoop.apache.org" , "
>> hdfs-...@hadoop.apache.org" , "
>> mapreduce-...@hadoop.apache.org" ,
>>
>> "yarn-...@hadoop.apache.org" 
>>
>>
>>
>>
>>
>> Folks,
>>
>>
>>
>> With hadoop-2.5 nearly done, it's time to start thinking ahead to
>> hadoop-2.6.
>>
>>
>>
>> Currently, here is the Roadmap per the wiki:
>>
>>
>>
>>• HADOOP
>>
>>• Credential provider HADOOP-10607
>>
>>• HDFS
>>
>>• Heterogeneous storage (Phase 2) - Support APIs for using
>> storage tiers by the applications HDFS-5682
>>
>>• Memory as storage tier HDFS-5851
>>
>>• YARN
>>
>>• Dynamic Resource Configuration YARN-291
>>
>>• NodeManager Restart YARN-1336
>>
>>• ResourceManager HA Phase 2 YARN-556
>>
>>• Support for admin-specified labels in YARN YARN-796
>>
>>• Support for automatic, shared cache for YARN application
>> artifacts YARN-1492
>>
>>• Support NodeGroup layer topology on YARN YARN-18
>>
>>• Support for Docker containers in YARN YARN-1964
>>
>>• YARN service registry YARN-913
>>
>>
>>
>> My suspicion is, as is normal, some will make the cut and some won't.
>>
>> Please do add/subtract from the list as appropriate. Ideally, it would be
>> good to ship hadoop-2.6 in

Re: Cannot access Jenkins generated test results, findbug warnings and javac warnings

[Colin McCabe]

Thanks, Steve.

Should we just put everything in patchprocess/ like before?  It seems
like renaming this directory to PreCommit-HADOOP-Build-patchprocess/
or PreCommit-YARN-Build-patchprocess/ in various builds has created
problems, and not made things any more clear.  What do you guys think?

Colin

On Fri, Sep 19, 2014 at 5:42 AM, Steve Loughran  wrote:
> well, I have those permissions, but not sure where things are going right
> now
>
> 1. jenkins is set to archive patchprocess/*.*
> 1 & publish tests from trunk/**/target/surefire-reports/*.xml
>
>
> I've set it to use the path
> PreCommit-HADOOP-Build-patchprocess/*.*
>
> & will do some manual triggering to see if it takes
>
> On 19 September 2014 10:54, Wangda Tan  wrote:
>
>> Hi Steve,
>> I guess this problem should be also caused by wrong URL, if anybody have
>> admin access to Jenkins, correct URL should be easily found.
>>
>> Thanks,
>> Wangda
>>
>> On Fri, Sep 19, 2014 at 4:32 PM, Steve Loughran 
>> wrote:
>>
>> > Looks like HADOOP-11084 isn't complete —the patch to the build to get it
>> > working post-git
>> >
>> > before that patch the builds weren't working at all ... now its just
>> > getting the URLs wrong.
>> >
>> > If you can work out the right URLs we can fix this easily enough
>> >
>> > On 19 September 2014 09:24, Wangda Tan  wrote:
>> >
>> > > Hi Hadoop developers,
>> > > I found recently, I cannot access Jenkins generated results, like:
>> > >
>> > >
>> > >
>> > > *Test
>> > > results:
>> > > https://builds.apache.org/job/PreCommit-YARN-Build/5039//testReport/
>> > > > > > >Findbugs
>> > > warnings:
>> > >
>> >
>> https://builds.apache.org/job/PreCommit-YARN-Build/5039//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
>> > > <
>> > >
>> >
>> https://builds.apache.org/job/PreCommit-YARN-Build/5039//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
>> > > >Javac
>> > > warnings:
>> > >
>> >
>> https://builds.apache.org/job/PreCommit-YARN-Build/5039//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
>> > > <
>> > >
>> >
>> https://builds.apache.org/job/PreCommit-YARN-Build/5039//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
>> > > >*
>> > >
>> > > It will report 404 when trying to access findbugs/javac warnings and it
>> > > will redirect to info page of build when trying to access test report.
>> > >
>> > > I'm not sure if there's any recent changes on Jenkins configuration.
>> Did
>> > > you hit the problem like this?
>> > >
>> > > Thanks,
>> > > Wangda
>> > >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: [DISCUSS] Branches and versions for Hadoop 3

[Colin McCabe]

On Mon, Aug 28, 2017, at 09:58, Allen Wittenauer wrote:
> 
> > On Aug 25, 2017, at 1:23 PM, Jason Lowe  wrote:
> > 
> > Allen Wittenauer wrote:
> >  
> > > Doesn't this place an undue burden on the contributor with the first 
> > > incompatible patch to prove worthiness?  What happens if it is decided 
> > > that it's not good enough?
> > 
> > It is a burden for that first, "this can't go anywhere else but 4.x" 
> > change, but arguably that should not be a change done lightly anyway.  (Or 
> > any other backwards-incompatible change for that matter.)  If it's worth 
> > committing then I think it's perfectly reasonable to send out the dev 
> > announce that there's reason for trunk to diverge from 3.x, cut branch-3, 
> > and move on.  This is no different than Andrew's recent announcement that 
> > there's now a need for separating trunk and the 3.0 line based on what's 
> > about to go in.
> 
>   So, by this definition as soon as a patch comes in to remove deprecated 
> bits there will be no issue with a branch-3 getting created, correct?

Jason wrote that making backwards incompatible changes should not be
"done lightly."  By that definition, "a patch... to remove deprecated
bits" would not cause us to create a whole branch for it.  It should be
something where someone could reasonably make the case that breaking
backwards compatibility was worth it.

> 
> >  Otherwise if past trunk behavior is any indication, it ends up mostly 
> > enabling people to commit to just trunk, forgetting that the thing they are 
> > committing is perfectly valid for branch-3. 
> 
>   I'm not sure there was any "forgetting" involved.  We likely wouldn't 
> be talking about 3.x at all if it wasn't for the code diverging enough.
> 
> > > Given the number of committers that openly ignore discussions like this, 
> > > who is going to verify that incompatible changes don't get in?
> >  
> > The same entities who are verifying other bugs don't get in, i.e.: the 
> > committers and the Hadoop QA bot running the tests.
> >  Yes, I know that means it's inevitable that compatibility breakages will 
> > happen, and we can and should improve the automation around compatibility 
> > testing when possible.
> 
>   The automation only goes so far.  At least while investigating Yetus 
> bugs, I've seen more than enough blatant and purposeful ignored errors and 
> warnings that I'm not convinced it will be effective. ("That javadoc compile 
> failure didn't come from my patch!"  Um, yes, yes it did.) PR for features 
> has greatly trumped code correctness for a few years now.
> 
>   In any case, specifically thinking of the folks that commit maybe one 
> or two patches a year.  They generally don't pay attention to *any* of this 
> stuff and it doesn't seem like many people are actually paying attention to 
> what gets committed until it breaks their universe.
> 
> >  But I don't think there's a magic bullet for preventing all compatibility 
> > bugs from being introduced, just like there isn't one for preventing 
> > general bugs.  Does having a trunk branch separate but essentially similar 
> > to branch-3 make this any better?
> 
>   Yes: it's been the process for over a decade now.  Unless there is some 
> outreach done, it is almost a guarantee that someone will commit something to 
> trunk they shouldn't because they simply won't know (or care?) the process 
> has changed.  

This is no different than any other type of bug.  If someone commits
something that is buggy, we should revert it.  If there are too many of
these issues, then we need more review, more testing, or both.

> 
> > > Longer term:  what is the PMC doing to make sure we start doing major 
> > > releases in a timely fashion again?  In other words, is this really an 
> > > issue if we shoot for another major in (throws dart) 2 years?
> > 
> > If we're trying to do semantic versioning
> 
>   FWIW: Hadoop has *never* done semantic versioning. A large percentage 
> of our minors should really have been majors. 
> 
> > then we shouldn't have a regular cadence for major releases unless we have 
> > a regular cadence of changes that break compatibility.  
> 
>   But given that we don't follow semantic versioning

In case someone new to the community is reading this thread, Hadoop does
have compatibility guidelines for major and minor releases, and has had
them for a very long time. 
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Compatibility.html
Of course, there have been bugs in the past, and they have been
frustrating.

> 
> > I'd hope that's not something we would strive towards.  I do agree that we 
> > should try to be better about shipping releases, major or minor, in a more 
> > timely manner, but I don't agree that we should cut 4.0 simply based on a 
> > duration since the last major release.
> 
>   ... the only thing we're really left with is (technically) time, either 
> in the form of a volunteer saying "

Re: [DISCUSS] Branches and versions for Hadoop 3

[Colin McCabe]

On Mon, Aug 28, 2017, at 14:22, Allen Wittenauer wrote:
> 
> > On Aug 28, 2017, at 12:41 PM, Jason Lowe  wrote:
> > 
> > I think this gets back to the "if it's worth committing" part.
> 
>   This brings us back to my original question:
> 
>   "Doesn't this place an undue burden on the contributor with the first 
> incompatible patch to prove worthiness?  What happens if it is decided that 
> it's not good enough?"

I feel like this line of argument is flawed by definition.  "What
happens if the patch isn't worth breaking compatibility over"?  Then we
shouldn't break compatibility over it.  We all know that most
compatibility breaks are avoidable with enough effort.  And it's an
effort we should make, for the good of our users.

Most useful features can be implemented without compatibility breaks. 
And for the few that truly can't, the community should surely agree that
it's worth breaking compatibility before we do it.  If it's a really
cool feature, that approval will surely not be hard to get (I'm tempted
to quote your earlier email about how much we love features...)

> 
>   The answer, if I understand your position, is then at least a maybe 
> leaning towards yes: a patch that prior to this branching policy change that  
> would have gone in without any notice now has a higher burden (i.e., major 
> feature) to prove worthiness ... and in the process eliminates a whole class 
> of contributors and empowers others. Thus my concern ...
> 
> > As you mentioned, people are already breaking compatibility left and right 
> > as it is, which is why I wondered if it was really any better in practice.  
> > Personally I'd rather find out about a major breakage sooner than later, 
> > since if trunk remains an active area of development at all times it's more 
> > likely the community will sit up and take notice when something crazy goes 
> > in.  In the past, trunk was not really an actively deployed area for over 5 
> > years, and all sorts of stuff went in without people really being aware of 
> > it.
> 
>   Given the general acknowledgement that the compatibility guidelines are 
> mostly useless in reality, maybe the answer is really that we're doing 
> releases all wrong.  Would it necessarily be a bad thing if we moved to a 
> model where incompatible changes gradually released instead of one big one 
> every seven?

I haven't seen anyone "acknowledge that... compatibility guidelines are
mostly useless"... even you.  Reading your posts from the past, I don't
get that impression.  On the contrary, you are often upset about
compatibility breakages.

What would be positive about allowing compatibility breaks in minor
releases?  Can you give a specific example of what would be improved?

> 
>   Yes, I lived through the "walking on glass" days at Yahoo! and realize 
> what I'm saying.  But I also think the rate of incompatible changes has 
> slowed tremendously.  Entire groups of APIs aren't getting tossed out every 
> week anymore.
> 
> > It sounds like we agree on that part but disagree on the specifics of how 
> > to help trunk remain active.
> 
>   Yup, and there is nothing wrong with that. ;)
> 
> >  Given that historically trunk has languished for years I was hoping this 
> > proposal would help reduce the likelihood of it happening again.  If we 
> > eventually decide that cutting branch-3 now makes more sense then I'll do 
> > what I can to make that work well, but it would be good to see concrete 
> > proposals on how to avoid the problems we had with it over the last 6 years.
> 
> 
>   Yup, agree. But proposals rarely seem to get much actual traction. 
> (It's kind of fun reading the Hadoop bylaws and compatibility guidelines and 
> old [VOTE] threads to realize how much stuff doesn't actually happen despite 
> everyone generally agree that abc is a good idea.)  To circle back a bit, I 
> do also agree that automation has a role to play
> 
>Before anyone can accuse or imply me of being a hypocrite (and I'm 
> sure someone eventually will privately if not publicly), I'm sure some folks 
> don't realize I've been working on this set of problems from a different 
> angle for the past few years.
> 
>   There are a handful of people that know I was going to attempt to do a 
> 3.x release a few years ago. [Andrew basically beat me to it. :) ] But I ran 
> into the release process.  What a mess.  Way too much manual work, lots of 
> undocumented bits, violation of ASF rules(!) , etc, etc.  We've all heard the 
> complaints.
> 
>   My hypothesis:  if the release process itself is easier, then getting a 
> release based on trunk is easier too. The more we automate, the more 
> non-vendors ("non traditional release managers"?) will be willing to roll 
> releases.  The more people that feel comfortable rolling a release, the more 
> likelihood releases will happen.  The more likelihood of releases happening, 
> the greater chance trunk had of getting out the door.

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Colin McCabe]

The "git" way of doing things would be to rebase the feature branch on
master (trunk) and then commit the patch stack.

Squashing the entire feature into a 10 MB megapatch is the "svn" way of
doing things.

The svn workflow evolved because merging feature branches back to trunk
was really painful in svn.  So people preferred just to basically do an
rsync from a checkout of the feature branch, edit that up a bit to make
sure they weren't overwriting something that happened in trunk, and then
do an "svn commit" that did not tie back to the feature branch.

P.S. Merges were so painful in SVN that some of the organizations I
worked for maintained spreadsheets of patches which had been merged to
each branch, since svn was of so little help with merging

P.P.S. svn eventually got slightly smarter about merges -- I think in
svn 1.6 or something?

Colin


On Wed, Aug 30, 2017, at 14:37, Sangjin Lee wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> 
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran 
> wrote:
> 
> > I'd have assumed it would have gone in as one single patch, rather than a
> > full history. I don't see why the trunk needs all the evolutionary history
> > of a build.
> >
> > What should our policy/process be here?
> >
> > I do currently plan to merge the s3guard in as one single squashed patch;
> > just getting HADOOP-14809 sorted first.
> >
> >
> > > On 30 Aug 2017, at 07:09, Vrushali C  wrote:
> > >
> > > I'm adding my +1 (binding) to conclude the vote.
> > >
> > > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > > the merge to trunk shortly. Thanks everyone!
> > >
> > > Regards
> > > Vrushali
> > >
> > >
> > > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> > > varun.saxena.apa...@gmail.com> wrote:
> > >
> > >> +1 (binding).
> > >>
> > >> Kudos to all the team members for their great work!
> > >>
> > >> Being part of the ATSv2 team, I have been involved with either
> > development
> > >> or review of most of the JIRAs'.
> > >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> > there
> > >> is no impact when ATSv2 is turned off.
> > >>
> > >> Regards,
> > >> Varun Saxena.
> > >>
> > >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >> vrushalic2...@gmail.com> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> > >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > >>> will
> > >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>>
> > >>> We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > Timeline
> > >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> > >>> confident that the state of the feature meets the criteria to be merged
> > >>> onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > >>> and provide valuable feedback so that we can make it production-ready.
> > >>>
> > >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > >>> usability improvements based on a new architecture. What we would like
> > to
> > >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > >>> complete end-to-end read/write flow with security and read level
> > >>> authorization via whitelists. You should be able to start setting it up
> > >>> and
> > >>> testing it.
> > >>>
> > >>> At a high level, the following are the key features that have been
> > >>> implemented since alpha1:
> > >>> - Security via Kerberos Authentication and delegation tokens
> > >>> - Read side simple authorization via whitelist
> > >>> - Client configurable entity sort ordering
> > >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics by
> > >>> timerange, pagination, sub-app entities
> > >>> - Support for storing sub-application entities (entities that exist
> > >>> outside
> > >>> the scope of an application)
> > >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> prefixes,
> > >>> configurable hbase cluster
> > >>> - Flow level aggregations done as dynamic (table level) coprocessors
> > >>> - Uses latest stable HBase release 1.2.6
> > >>>
> > >>> There are a total of 82 subtasks that were completed as part of this
> > >>> effort.
> > >>>
> > >>> We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > >>> does not impact existing functionality when disabled (by default).
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > >>> Saxena, Haib

Re: submissions with bsd code?

[Colin McCabe]

On Wed, Oct 1, 2014 at 4:30 PM, John Smith  wrote:
> hi developers.
>
> i have some native code working on solaris, but my changes use getgrouplist
> from openssh.  is that ok?  do i need to do anything special?  is the
> license in file enough?

Is the license in which file enough?

Colin

>
> thank you for your help.

builds failing on H9 with "cannot access java.lang.Runnable"

[Colin McCabe]

It looks like builds are failing on the H9 host with "cannot access
java.lang.Runnable"

Example from 
https://builds.apache.org/job/PreCommit-HDFS-Build/8313/artifact/patchprocess/trunkJavacWarnings.txt
:

[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 03:13 min
[INFO] Finished at: 2014-10-03T18:04:35+00:00
[INFO] Final Memory: 57M/839M
[INFO] 
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hadoop-mapreduce-client-app:
Compilation failure
[ERROR] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/commit/TestCommitterEventHandler.java:[189,-1]
cannot access java.lang.Runnable
[ERROR] bad class file: java/lang/Runnable.class(java/lang:Runnable.class)

I don't have shell access to this, does anyone know what's going on on H9?

best,
Colin

Re: builds failing on H9 with "cannot access java.lang.Runnable"

[Colin McCabe]

Thanks, Andrew and Giridharan.

Colin

On Fri, Oct 3, 2014 at 1:20 PM, Andrew Bayer  wrote:
> Yeah, the other possibility is that an ansible run borks running
> slaves. If this happens again, let us know.
>
> A.
>
> On Fri, Oct 3, 2014 at 1:15 PM, Giridharan Kesavan
>  wrote:
>> all the slaves are getting re-booted give it some more time
>>
>> -giri
>>
>> On Fri, Oct 3, 2014 at 1:13 PM, Ted Yu  wrote:
>>
>>> Adding builds@
>>>
>>> On Fri, Oct 3, 2014 at 1:07 PM, Colin McCabe 
>>> wrote:
>>>
>>> > It looks like builds are failing on the H9 host with "cannot access
>>> > java.lang.Runnable"
>>> >
>>> > Example from
>>> >
>>> https://builds.apache.org/job/PreCommit-HDFS-Build/8313/artifact/patchprocess/trunkJavacWarnings.txt
>>> > :
>>> >
>>> > [INFO]
>>> > 
>>> > [INFO] BUILD FAILURE
>>> > [INFO]
>>> > 
>>> > [INFO] Total time: 03:13 min
>>> > [INFO] Finished at: 2014-10-03T18:04:35+00:00
>>> > [INFO] Final Memory: 57M/839M
>>> > [INFO]
>>> > 
>>> > [ERROR] Failed to execute goal
>>> > org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
>>> > (default-testCompile) on project hadoop-mapreduce-client-app:
>>> > Compilation failure
>>> > [ERROR]
>>> >
>>> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/commit/TestCommitterEventHandler.java:[189,-1]
>>> > cannot access java.lang.Runnable
>>> > [ERROR] bad class file:
>>> java/lang/Runnable.class(java/lang:Runnable.class)
>>> >
>>> > I don't have shell access to this, does anyone know what's going on on
>>> H9?
>>> >
>>> > best,
>>> > Colin
>>> >
>>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.

Re: Build failed in Jenkins: Hadoop-Common-trunk due to missing header file

[Colin McCabe]

On Thu, Oct 2, 2014 at 1:15 PM, Ted Yu  wrote:
> On my Mac and on Linux, I was able to
> find /usr/include/openssl/opensslconf.h
>
> However the file is absent on Jenkins machine(s).
>
> Just want to make sure that the file is needed for native build before
> filing INFRA ticket.

opensslconf.h is part of the openssl-devel package (at least on my
machine) and if it is missing, I would suspect that openssl is either
not installed or incorrectly installed.

We need it for the native build to have coverage for the
openssl-related things (like random number generation and encryption).

Colin

>
> Cheers
>
> On Thu, Oct 2, 2014 at 9:09 AM, Tsuyoshi OZAWA 
> wrote:
>
>> Hi Ted,
>>
>> On my local, the build of trunk with "mvn package -Pnative,dist" works
>> well. I'm not certain whether this problem is related, but some build
>> fails on YARN(e.g. YARN-2562, YARN-2615, YARN-2640).
>>
>> The version information of OS and libssl-dev on my local environment
>> is as follows:
>>
>> $ uname -a
>> Linux ip-172-31-4-83 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10
>> 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>>
>> $ apt-cache show libssl-dev
>> Package: libssl-dev
>> Priority: optional
>> Section: libdevel
>> Installed-Size: 6162
>> Maintainer: Ubuntu Developers 
>> Original-Maintainer: Debian OpenSSL Team
>> 
>> Architecture: amd64
>> Source: openssl
>> Version: 1.0.1f-1ubuntu2
>> Depends: libssl1.0.0 (= 1.0.1f-1ubuntu2), zlib1g-dev
>> Recommends: libssl-doc
>> Filename: pool/main/o/openssl/libssl-dev_1.0.1f-1ubuntu2_amd64.deb
>> Size: 1066212
>> MD5sum: 321724885048f9a78d0e93887a7eb296
>> SHA1: e337538bed6e5765a0a85c4ca2af1d0deefd6ce0
>> SHA256: ed199dc9131923fa3c911202f165402b1310f50dcdfab987f6f5c2669fc698cc
>>
>> Cheers,
>> - Tsuyoshi
>>
>> On Thu, Oct 2, 2014 at 11:43 PM, Ted Yu  wrote:
>> > Hadoop-Common-trunk build failed due to missing opensslconf.h
>> >
>> > Is this environment issue or due to recent commits ?
>> >
>> > Cheers
>> >
>> > On Thu, Oct 2, 2014 at 7:31 AM, Apache Jenkins Server <
>> > jenk...@builds.apache.org> wrote:
>> >
>> >> See 
>> >>
>> >> --
>> >>  [exec] /usr/bin/cmake -E cmake_progress_report <
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/target/native/CMakeFiles
>> >
>> >> 8
>> >>  [exec] [ 16%] Building C object
>> >>
>> CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o
>> >>  [exec] /usr/bin/cc  -Dhadoop_EXPORTS -m32 -g -Wall -O2 -D_REENTRANT
>> >> -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -fPIC -I<
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/target/native/javah
>> >
>> >> -I<
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/native/src
>> >
>> >> -I<
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src
>> >
>> >> -I<
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/src
>> >
>> >> -I<
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/target/native
>> >
>> >> -I/home/jenkins/tools/java/latest/include
>> >> -I/home/jenkins/tools/java/latest/include/linux -I<
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util
>> >
>> >>   -o
>> >>
>> CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o
>> >>  -c <
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
>> >> >
>> >>  [exec] In file included from <
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/nat
>> >> >
>> >>  [exec]
>> >> ive/src/org/apache/hadoop/crypto/org_apache_hadoop_crypto.h:33:0,
>> >>  [exec]  from <
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
>> >> >:19:
>> >>  [exec] /usr/incl
>> >>  [exec] ude/openssl/aes.h:55:33: fatal error: openssl/opensslconf.h:
>> >> No such file or directory
>> >>  [exec]  #include 
>> >>  [exec]  ^
>> >>  [exec] compilation terminated.
>> >>  [exec] make[2]: ***
>> >>
>> [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o]
>> >> Error 1
>> >>  [exec] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
>> >>  [exec] make: *** [all] Error 2
>> >>  [exec] make[2]: Leaving directory `<
>> >>
>> https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/target/native
>> >> '>
>> >>  [exec] make[1]: Le

Re: Guava

[Colin McCabe]

I'm usually an advocate for getting rid of unnecessary dependencies
(cough, jetty, cough), but a lot of the things in Guava are really
useful.

Immutable collections, BiMap, Multisets, Arrays#asList, the stuff for
writing hashCode() and equals(), String#Joiner, the list goes on.  We
particularly use the Cache/CacheBuilder stuff a lot in HDFS to get
maps with LRU eviction without writing a lot of boilerplate.  The QJM
stuff uses ListenableFuture a lot, although perhaps we could come up
with our own equivalent for that.

On Mon, Nov 10, 2014 at 9:26 AM, Alejandro Abdelnur  wrote:
> IMO we should:
>
> 1* have a clean and thin client API JAR (which does not drag any 3rd party
> dependencies, or a well defined small set -i.e. slf4j & log4j-)
> 2* have a client implementation that uses a classloader to isolate client
> impl 3rd party deps from app dependencies.
>
> #2 can be done using a stock URLClassLoader (i would just subclass it to
> forbid packages in the API JAR and exposed 3rd parties to be loaded from
> the app JAR)
>
> #1 is the tricky thing as our current API modules don't have a clean
> API/impl separation.
>
> thx
> PS: If folks are interested in pursing this, I can put together a prototype
> of how  #2 would work (I don't think it will be more than 200 lines of code)

Absolutely, I agree that we should not be using Guava types in public
APIs.  Guava has not been very responsible with backwards
compatibility, that much is clear.

A client / server jar separation is an interesting idea.  But then we
still have to get rid of Guava and other library deps in the client
jars.  I think it would be more work than it seems.  For example, the
HDFS client uses Guava Cache a lot, so we'd have to write our own
version of this.

Can't we just shade this stuff?  Has anyone tried shading Hadoop's Guava?

best,
Colin


>
>
> On Mon, Nov 10, 2014 at 5:18 AM, Steve Loughran 
> wrote:
>
>> Yes, Guava is a constant pain; there's lots of open JIRAs related to it, as
>> its the one we can't seamlessly upgrade. Not unless we do our own fork and
>> reinsert the missing classes.
>>
>> The most common uses in the code are
>>
>> @VisibleForTesting (easily replicated)
>> and the Precondition.check() operations
>>
>> The latter is also easily swapped out, and we could even add the check they
>> forgot:
>> Preconditions.checkArgNotNull(argname, arg)
>>
>>
>> These are easy; its the more complex data structures that matter more.
>>
>> I think for Hadoop 2.7 & java 7 we need to look at this problem and do
>> something. Even if we continue to ship Guava 11 so that the HBase team
>> don't send any (more) death threats, we can/should rework Hadoop to build
>> and run against Guava 16+ too. That's needed to fix some of the recent java
>> 7/8+ changes.
>>
>> -Everything in v11 dropped from v16 MUST  to be implemented with our own
>> versions.
>> -anything tagged as deprecated in 11+ SHOULD be replaced by newer stuff,
>> wherever possible.
>>
>> I think for 2.7+ we should add some new profiles to the POM, for Java 8 and
>> 9 alongside the new baseline java 7. For those later versions we could
>> perhaps mandate Guava 16.
>>
>>
>>
>> On 10 November 2014 00:42, Arun C Murthy  wrote:
>>
>> > … has been a constant pain w.r.t compatibility etc.
>> >
>> > Should we consider adopting a policy to not use guava in
>> Common/HDFS/YARN?
>> >
>> > MR doesn't matter too much since it's application-side issue, it does
>> hurt
>> > end-users though since they still might want a newer guava-version, but
>> at
>> > least they can modify MR.
>> >
>> > Thoughts?
>> >
>> > thanks,
>> > Arun
>> >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Colin McCabe]

Multi-subproject patches used to work.  If they don't work now, it is
probably a bug in test-patch.sh that we should fix.  The code there is
written expecting multi-project changes, but maybe it doesn't get much of a
workout normally.

Conceptually, I think it's important to support patches that modify
multiple sub-projects.  Otherwise refactoring things in common becomes a
multi-step process.  You can't very well change an API in hadoop-common
that multiple projects are using with patches that touch only one
subproject, unless you use a cumbersome "deprecate and then remove"
process.  This just slows down development and doesn't really seem to have
any upside.  I also think it makes sense to have discussion about a change
in Hadoop-common all in one place, rather than splitting it into multiple
discussions on multiple per-subproject JIRAs.

best,
Colin

On Tue, Nov 18, 2014 at 2:24 PM, Chris Nauroth 
wrote:

> That's an interesting find.  This has not always been the case.  In the
> past, we've provided consolidated patches on a HADOOP jira to get
> pre-commit coverage when changes span multiple sub-projects.  Yongjun, if
> you haven't already done so, I suggest filing a jira for this.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
> On Sat, Nov 15, 2014 at 8:19 AM, Yongjun Zhang 
> wrote:
>
> > For all's info, confirmed what Ray stated, indeed it's the reason!
> >
> > Thanks.
> >
> > --Yongjun
> >
> >
> > On Fri, Nov 14, 2014 at 2:40 PM, Yongjun Zhang 
> > wrote:
> >
> > > Many thanks Ray, good to know!
> > >
> > > --Yongjun
> > >
> > > On Fri, Nov 14, 2014 at 2:03 PM, Ray Chiang 
> > wrote:
> > >
> > >> Your patch covers multiple projects (I see 81 files across
> > >> hadoop-common-project, hadoop-hdfs-project, hadoop-mapreduce-project,
> > >> hadoop-tools, hadoop-yarn-project).
> > >>
> > >> Jenkins had problems with that for me a few months back.  If you break
> > it
> > >> up per-project, it should work just fine.
> > >>
> > >> -Ray
> > >>
> > >>
> > >> On Fri, Nov 14, 2014 at 1:49 PM, Ted Yu  wrote:
> > >>
> > >> > Adding builds@apache
> > >> >
> > >> > On Fri, Nov 14, 2014 at 1:34 PM, Yongjun Zhang  >
> > >> > wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > One issue to report here, any help would be greatly appreciated!
> > >> > >
> > >> > > I noticed that multiple patch submissions to
> > >> > > https://issues.apache.org/jira/browse/HADOOP-11293
> > >> > > did not trigger jenkins test run.
> > >> > >
> > >> > > Thanks Chris Nauroth for the help to trigger one manually for me:
> > >> > >
> > >> > > https://builds.apache.org/job/PreCommit-HADOOP-Build/5079/
> > >> > >
> > >> > > and it turned out the manually triggered one did not report result
> > >> back
> > >> > to
> > >> > > the jira upon finishing.
> > >> > >
> > >> > > I submitted the same patch to another jira (HADOOP-11045) that I
> was
> > >> > > working on, which triggered jenkins test before, and it seems not
> > >> > > triggering this time too.
> > >> > >
> > >> > > So it looks like something is broken. This might be just a HADOOP
> > jira
> > >> > > issue (HDFS jira seems to be fine).
> > >> > >
> > >> > > Thanks a lot for looking into.
> > >> > >
> > >> > > --Yongjun
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Colin McCabe]

+1 for increasing the test timeout for tests spanning multiple sub-projects.

I can see the value in what Steve L. suggested... if you make a major
change that touches a particular subproject, you should try to get the
approval of a committer who knows that subproject.  But I don't think that
forcing artificial patch splits is the way to do this...  There are also
some patches that are completely mechanical and don't really require the
involvement of YARN / HDFS committer, even if they change that project.
For example, fixing a misspelling in the name of a hadoop-common API.

Colin

On Tue, Nov 25, 2014 at 8:45 AM, Yongjun Zhang  wrote:

> Thanks all for the feedback. To summarize (and I have a suggestion at the
> end of this email), there are two scenarios:
>
>1. A change that span multiple *bigger* projects. r.g. hadoop, hbase.
>2. A change that span multiple *sub* projects* within hadoop, e.g.,
>common, hdfs, yarn
>
> For 1, it's required for the change to be backward compatible, thus
> splitting change for multiple *bigger* projects is a must.
>
> For 2, there are two sub types,
>
>- 2.1 those changes that can be made within hadoop sub-projects, and
>there is no external impact
>- 2.2 those changes that have external impact, that is, the changes
>involve adding new APIs and marking old API deprecated, and
> corresponding
>changes in other *bigger* projects will have to be made independently.
> *But
>the changes within hadoop subjects can still be done altogether.*
>
> I think (Please correct me if I'm wrong):
>
>- What Colin referred to is 2.1 and changes within hadoop sub-subjects
>for 2.2;
>- Steve's "not for changes across hadoop-common and hdfs, or
>hadoop-common and yarn" means 2.1, Steve's  "changes that only
>span hdfs-and-yarn would be fairly doubtful too." implies his doubt of
>existence of 2.1.
>
> For changes of 2.1 (if any) and *hadoop* changes of 2.2, we do have an
> option of making the change across all hadoop sub-projects altogether, to
> save the multiple steps Colin referred to.
>
> If this option is feasible, should we consider increasing the jenkins
> timeout for this kind of changes (I mean making the timeout adjustable, if
> it's for single sub-project, use the old timeout; otherwise, increase
> accordingly)  so that we have at least this option when needed?
>
> Thanks.
>
> --Yongjun
>
>
> On Tue, Nov 25, 2014 at 2:28 AM, Steve Loughran 
> wrote:
>
> > On 25 November 2014 at 00:58, Bernd Eckenfels 
> > wrote:
> >
> > > Hello,
> > >
> > > Am Mon, 24 Nov 2014 16:16:00 -0800
> > > schrieb Colin McCabe :
> > >
> > > > Conceptually, I think it's important to support patches that modify
> > > > multiple sub-projects.  Otherwise refactoring things in common
> > > > becomes a multi-step process.
> > >
> > > This might be rather philosophical (and I dont want to argue the need
> > > to have the patch infrastructure work for the multi-project case),
> > > howevere if a multi-project change cannot be applied in multiple steps
> > > it is probably also not safe at runtime (unless the multiple projects
> > > belong to a single instance/artifact). And then beeing forced to
> > > commit/compile/test in multiple steps actually increases the
> > > dependencies topology.
> > >
> >
> > +1 for changes that span, say hadoop and hbase. but not for changes
> across
> > hadoop-common and hdfs, or hadoop-common and yarn. changes that only span
> > hdfs-and-yarn would be fairly doubtful too.
> >
> > there is a dependency graph in hadoop's own jars —and cross module (not
> > cross project) changes do need to happen.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

Re: a friendly suggestion for developers when uploading patches

[Colin McCabe]

On Wed, Nov 26, 2014 at 2:58 PM, Karthik Kambatla 
wrote:

> Yongjun, thanks for starting this thread. I personally like Steve's
> suggestions, but think two digits should be enough.
>
> I propose we limit the restrictions to versioning the patches with version
> numbers and .patch extension. People have their own preferences for the
> rest of the name (e.g. MAPREDUCE, MapReduce, MR, mr, mapred) and I don't
> see a gain in forcing everyone to use one.
>
> Putting the suggestions (tight and loose) on the wiki would help new
> contributors as well.
>
>
+1

best,
Colin


> On Wed, Nov 26, 2014 at 2:43 PM, Eric Payne  >
> wrote:
>
> > +1.The "different color for newest patch" doesn't work very well if you
> > are color blind, so I do appreciate a revision number in the name.
> >
> >   From: Yongjun Zhang 
> >  To: common-dev@hadoop.apache.org
> >  Sent: Tuesday, November 25, 2014 11:37 PM
> >  Subject: Re: a friendly suggestion for developers when uploading patches
> >
> > Thanks Harsh for the info and Andrew for sharing the script. It looks
> that
> > the script is intelligent enough to pick the latest attachment even if
> all
> > attachments have the same name.
> >
> > Yet, I hope we use the following as the guideline for patch names:
> >
> > <*projectName*>-<*jiraNum*>-<*revNum*>.patch
> >
> >
> > So we can easily identify individual patch revs.
> >
> > Thanks.
> >
> > --Yongjun
> >
> > On Tue, Nov 25, 2014 at 5:54 PM, Andrew Wang 
> > wrote:
> >
> > > This might be a good time to mention my fetch-patch script, I use it to
> > > easily download the latest attachment on a jira:
> > >
> > > https://github.com/umbrant/dotfiles/blob/master/bin/fetch-patch
> > >
> > > On Tue, Nov 25, 2014 at 5:44 PM, Harsh J  wrote:
> > >
> > > > For the same filename, you can observe also that the JIRA colors the
> > > > latest one to be different than the older ones automatically - this
> is
> > > > what I rely on.
> > > >
> > > > On Sat, Nov 22, 2014 at 12:36 AM, Yongjun Zhang  >
> > > > wrote:
> > > > > Hi,
> > > > >
> > > > > When I look at patches uploaded to jiras, from time to time I
> notice
> > > that
> > > > > different revisions of the patch is uploaded with the same patch
> file
> > > > name,
> > > > > some time for quite a few times. It's confusing which is which.
> > > > >
> > > > > I'd suggest that as a guideline, we do the following when
> uploading a
> > > > patch:
> > > > >
> > > > >- include a revision number in the patch file name.A
> > > > >- include a comment, stating that a new patch is uploaded,
> > including
> > > > the
> > > > >revision number of the patch in the comment.
> > > > >
> > > > > This way, it's easier to refer to a specific version of a patch,
> and
> > to
> > > > > know which patch a comment is made about.
> > > > >
> > > > > Hope that makes sense to you.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > --Yongjun
> > > >
> > > >
> > > >
> > > > --
> > > > Harsh J
> > > >
> > >
> >
> >
> >
> >
>

Re: Thinking ahead to hadoop-2.7

[Colin McCabe]

On Fri, Dec 5, 2014 at 11:15 AM, Karthik Kambatla  wrote:
> It would be nice to cut the branch for the next "feature" release (not just
> Java 7) in the first week of January, so we can get the RC out by the end
> of the month?
>
> Yesterday, this came up in an offline discussion on ATS. Given people can
> run 2.6 on Java 7, is there merit to doing 2.7 with the exact same bits
> targeting Java 7? I am okay with going through with it, as long as it
> doesn't delay the next feature release.
>
> Thoughts?

That's a good point.  I think it's important to point out that most of
our users are already on JDK7.  We shouldn't think of this decision as
adding support for something new, we should think about it as taking
something away (JDK6 support).  I think it's good that we are finally
moving away from supporting JDK6, but I'm not completely sure that we
need to devote a whole release to that.  Are there a lot of open JDK7
issues that would require a release to straighten out?

best,
Colin


>
> On Wed, Dec 3, 2014 at 8:59 AM, Sangjin Lee  wrote:
>
>> Late January sounds fine to me. I think we should be able to wrap it up
>> much earlier than that (hopefully).
>>
>> Thanks,
>> Sangjin
>>
>> On Tue, Dec 2, 2014 at 5:19 PM, Arun C Murthy  wrote:
>>
>> > Sangjin/Karthik,
>> >
>> >  How about planning on hadoop-2.8 by late Jan? Thoughts?
>> >
>> > thanks,
>> > Arun
>> >
>> > On Dec 2, 2014, at 11:09 AM, Sangjin Lee  wrote:
>> >
>> > > If 2.7 is being positioned as the JDK7-only release, then it would be
>> > good
>> > > to know how 2.8 lines up in terms of timing. Our interest is landing
>> the
>> > > shared cache feature (YARN-1492)... Thanks.
>> > >
>> > > Sangjin
>> > >
>> > > On Mon, Dec 1, 2014 at 2:55 PM, Karthik Kambatla 
>> > wrote:
>> > >
>> > >> Thanks for starting this thread, Arun.
>> > >>
>> > >> Your proposal seems reasonable to me. I suppose we would like new
>> > features
>> > >> and improvements to go into 2.8 then? If yes, what time frame are we
>> > >> looking at for 2.8? Looking at YARN, it would be nice to get a release
>> > with
>> > >> shared-cache and a stable version of reservation work. I believe they
>> > are
>> > >> well under way and should be ready in a few weeks.
>> > >>
>> > >> Regarding 2.7 release specifics, do you plan to create a branch off of
>> > >> current branch-2.6 and update all issues marked fixed for 2.7 to be
>> > fixed
>> > >> for 2.8?
>> > >>
>> > >> Thanks
>> > >> Karthik
>> > >>
>> > >> On Mon, Dec 1, 2014 at 2:42 PM, Arun Murthy 
>> > wrote:
>> > >>
>> > >>> Folks,
>> > >>>
>> > >>> With hadoop-2.6 out it's time to think ahead.
>> > >>>
>> > >>> As we've discussed in the past, 2.6 was the last release which
>> supports
>> > >>> JDK6.
>> > >>>
>> > >>> I'm thinking it's best to try get 2.7 out in a few weeks (maybe by
>> the
>> > >>> holidays) with just the switch to JDK7 (HADOOP-10530) and possibly
>> > >>> support for JDK-1.8 (as a runtime) via HADOOP-11090.
>> > >>>
>> > >>> This way we can start with the stable base of 2.6 and switch over to
>> > >>> JDK7 to allow our downstream projects to use either for a short time
>> > >>> (hadoop-2.6 or hadoop-2.7).
>> > >>>
>> > >>> I'll update the Roadmap wiki accordingly.
>> > >>>
>> > >>> Thoughts?
>> > >>>
>> > >>> thanks,
>> > >>> Arun
>> > >>>
>> > >>> --
>> > >>> CONFIDENTIALITY NOTICE
>> > >>> NOTICE: This message is intended for the use of the individual or
>> > entity
>> > >> to
>> > >>> which it is addressed and may contain information that is
>> confidential,
>> > >>> privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > >>> of this message is not the intended recipient, you are hereby
>> notified
>> > >> that
>> > >>> any printing, copying, dissemination, distribution, disclosure or
>> > >>> forwarding of this communication is strictly prohibited. If you have
>> > >>> received this communication in error, please contact the sender
>> > >> immediately
>> > >>> and delete it from your system. Thank You.
>> > >>>
>> > >>
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/hdp/
>> >
>> >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>
>
> --
> -- Karthik Kambatla, Software Engineer, Cloudera
> 
> Q: Why is this email five sentences or less?
> A: http://five.sentenc.es

Re: Solaris Port

[Colin McCabe]

Hi Malcolm,

It's great that you are going to contribute!  Please make your patches
against trunk.

2.2 is fairly old at this point.  It hasn't been the focus of
development in more than a year.

We don't use github or pull requests.

Check the section on http://wiki.apache.org/hadoop/HowToContribute
that talks about "Contributing your work".  Excerpt:
"Finally, patches should be attached to an issue report in Jira via
the Attach File link on the issue's Jira. Please add a comment that
asks for a code review following our code review checklist. Please
note that the attachment should be granted license to ASF for
inclusion in ASF works (as per the Apache License §5)."

As this says, you attach the patch file to a JIRA that you have
created, and then hit "submit patch."

I don't think a branch is required for this work since it is just
build fixes, right?

best,
Colin


On Mon, Dec 8, 2014 at 3:30 AM, malcolm  wrote:
> I have ported Hadoop  native libraries to Solaris 11 (both Sparc and Intel )
> Oracle have agreed to release my changes to the community so that Solaris
> platforms can benefit.
> Reading the HowToContribute and GitandHadoop documents, I am not 100% clear
> on how to get my changes into the main tree. I am also a Git(hub) newbie,
> and was using svn previously.
>
> Please let me know if I am going the correct path:
>
> 1. I forked Hadoop on Github and downloaded a clone to my development
> machine.
>
> 2. The changes I made were to 2.2.0, can I still add changes to this branch,
> and hopefully get them accepted or must I migrate my changes to 2.6 ? (On
> the main Hadoop download page, 2.2 is still listed as the GA version )
>
> 3. I understand that I should create a new branch for my changes, and then
> generate pull requests after uploading them to Github.
>
> 4. I also registered  at Jira in the understanding that I need to generate a
> Jira number for my changes, and to name my branch accordingly ?
>
> Does all this make sense ?
>
> Thanks,
> Malcolm
>
>

Re: Switching to Java 7

[Colin McCabe]

On Mon, Dec 8, 2014 at 7:46 AM, Steve Loughran  wrote:
> On 8 December 2014 at 14:58, Ted Yu  wrote:
>
>> Looks like there was still OutOfMemoryError :
>>
>>
>> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameDirAcrossSnapshottableDirs/
>>
>
> Well, I'm going to ignore that for now as it's a java 8 problem, surfacing
> this weekend once the builds were actually switched to Java 8. memory size
> tuning can continue.
>
> I have now committed the Java 7+ only patch to branch-2 and up: new code
> does not have to worry about java 6 compatibility unless they plan to
> backport to Java 2.6 or earlier. Having written some Java 7 code, the <>
> constructor for typed classes are a convenience, the multiple-catch entries
> more useful, as they eliminate duplicate code in exception handling.
>
> Getting this patch in has revealed that the Jenkins builds of hadoop are
> (a) a bit of a mess and (b) prone to race conditions related to the m2
> repository if >1 project builds simultaneously. The way the nightly builds
> are staggered means this doesn't usually surface, but it may show up during
> precommit/postcommit builds.

It would be nice if we could have a separate .m2 directory per test executor.

It seems like that would eliminate these race conditions once and for
all, at the cost of storing a few extra jars (proportional to the # of
simultaneous executors)

best,
Colin


>
> The switch to Java 7 as the underlying JDK appears to be triggering
> failures, these are things that the projects themselves are going to have
> to look at.
>
>
> This then, is where we are with builds right now. This is not a consequence
> of the changes to the POM; this list predates that patch. This is Jenkins
> running Hadoop builds and tests with Java 7u55
>
>
> *Working: *
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-branch2/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk/
>
> *failing tests*
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-2-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/
>
> *failing tests on Java 8 (may include OOM)*
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-common-trunk-Java8/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Java8/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/
>
>
> *failing with maven internal dependency problems*
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/
>
>
> *failing even though it appears to work in the logs*
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-trunk/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Solaris Port

[Colin McCabe]

Hi Malcolm,

In general we file JIRAs for particular issues.  So if one issue is
handling errlist on Solaris, that might be one JIRA.  Another issue
might be handling socket write timeouts on Solaris.  And so on.  Most
of these should probably be HADOOP tickets since they sound like they
are mostly in the generic hadoop-common code.

"solaris does not have errno" seems like a bold statement.  errno is
part of POSIX, and Solaris is a POSIX os, right?  Am I way off base on
this?
I googled around and one of the first results I found talked about
errno values on Solaris.
http://www.pixelstech.net/article/1413273556-A-trick-of-building-multithreaded-application-on-Solaris
 Perhaps I misunderstood what you meant by this statement.

Anyway, please file JIRAs for any portability improvements you can think of!

best,
Colin

On Mon, Dec 8, 2014 at 9:09 PM, malcolm  wrote:
> Hi Colin,
>
> A short summary of my changes are as follows:
>
> - Native C source files: added 5,  modified 6, requiring also changes to
> CMakeLists.txt. Of course, all changes are "ifdeffed" for Solaris
> appropriately and new files, are prefixed with solaris_ as well.
>
> For example, Solaris does not have errno, or errlist any more which are used
> quite a lot in hadoop native code. I could have replaced all calls to use
> strerror instead which would be compatible with Linux, however in the
> interests of making minimal changes, I recreated and added these files from
> a running Solaris machine instead.
>
> Another issue is that Solaris doesn't have the timeout option for sockets,
> so I had to write my own solaris_read routine with timeout and added it to
> DomainSocket.c . A few issues with lz4 on Sparc needed modification, and
> some other OS specific issues: getgrouplist, container-executer (from yarn).
>
> - Some very minor changes were made to some Java source files (mainly tests
> to get them to pass on Solaris)
>
> The above changes were made to 2.2, I will recheck everything against the
> latest trunk, maybe some fixes aren't needed any more.
>
> I have generated a single patch file with all changes. Perhaps it would be
> better to file multiple JIRAs for each change, perhaps grouped, one per
> issue ? Or should I file a JIRA for each modified source file ?
>
> Thank you,
> Malcolm
>
>
> On 12/08/2014 09:53 PM, Colin McCabe wrote:
>>
>> Hi Malcolm,
>>
>> It's great that you are going to contribute!  Please make your patches
>> against trunk.
>>
>> 2.2 is fairly old at this point.  It hasn't been the focus of
>> development in more than a year.
>>
>> We don't use github or pull requests.
>>
>> Check the section on http://wiki.apache.org/hadoop/HowToContribute
>> that talks about "Contributing your work".  Excerpt:
>> "Finally, patches should be attached to an issue report in Jira via
>> the Attach File link on the issue's Jira. Please add a comment that
>> asks for a code review following our code review checklist. Please
>> note that the attachment should be granted license to ASF for
>> inclusion in ASF works (as per the Apache License §5)."
>>
>> As this says, you attach the patch file to a JIRA that you have
>> created, and then hit "submit patch."
>>
>> I don't think a branch is required for this work since it is just
>> build fixes, right?
>>
>> best,
>> Colin
>>
>>
>> On Mon, Dec 8, 2014 at 3:30 AM, malcolm 
>> wrote:
>>>
>>> I have ported Hadoop  native libraries to Solaris 11 (both Sparc and
>>> Intel )
>>> Oracle have agreed to release my changes to the community so that Solaris
>>> platforms can benefit.
>>> Reading the HowToContribute and GitandHadoop documents, I am not 100%
>>> clear
>>> on how to get my changes into the main tree. I am also a Git(hub) newbie,
>>> and was using svn previously.
>>>
>>> Please let me know if I am going the correct path:
>>>
>>> 1. I forked Hadoop on Github and downloaded a clone to my development
>>> machine.
>>>
>>> 2. The changes I made were to 2.2.0, can I still add changes to this
>>> branch,
>>> and hopefully get them accepted or must I migrate my changes to 2.6 ? (On
>>> the main Hadoop download page, 2.2 is still listed as the GA version )
>>>
>>> 3. I understand that I should create a new branch for my changes, and
>>> then
>>> generate pull requests after uploading them to Github.
>>>
>>> 4. I also registered  at Jira in the understanding that I need to
>>> generate a
>>> Jira number for my changes, and to name my branch accordingly ?
>>>
>>> Does all this make sense ?
>>>
>>> Thanks,
>>> Malcolm
>>>
>>>
>

Re: Solaris Port

[Colin McCabe]

On Wed, Dec 10, 2014 at 2:31 AM, malcolm  wrote:
> Hi Colin,
>
> Thanks for the hints around JIRAs.
>
> You are correct errno still exists, however sys_errlist does not.
>
> Hadoop uses a function terror (defined in exception.c) which indexes
> sys_errlist by errno to return the error message from the array. This
> function is called 26 times in various places (in 2.2)
>
> Originally, I thought to replace all calls to terror with strerror, but
> there can be issues with multi-threading (it returns a buffer which can be
> overwritten), so it seemed simpler just to recreate the sys_errlist message
> array.
>
> There is also a multi-threaded version strerror_r where you pass the buffer
> as a parameter, but this would necessitate changing every call to terror
> with mutiple lines of code.

Why don't you just use strerror_r inside terror()?

I wrote that code originally.  The reason I didn't want to use
strerror_r there is because GNU libc provides a non-POSIX definition
of strerror_r, and forcing it to use the POSIX one is a pain.  But you
can do it.  You also will require a thread-local buffer to hold the
return from strerror_r, since it is not guaranteed to be static
(although in practice it is 99% of the time-- another annoyance with
the API).

>
> Sorry, I wasn't clear.
>
> Also, I have been requested to ensure my port is available on 2.4, perceived
> as a more stable release. If I make changes to this branch are they
> automatically available for 2.6, or will I need multiple JIRAs ?

As Steve commented, 2.4 is pretty much "done."  I don't think this
kind of thing would get backported there.  Even getting it in a 2.6
release seems unlikely to me (but I haven't thought about it that
hard).  I would just get the work done, and let it show up in the
release it's ready in.

cheers,
Colin

>
> Thanks,
> Malcolm
>
>
> On 12/10/2014 10:45 AM, Colin McCabe wrote:
>>
>> Hi Malcolm,
>>
>> In general we file JIRAs for particular issues.  So if one issue is
>> handling errlist on Solaris, that might be one JIRA.  Another issue
>> might be handling socket write timeouts on Solaris.  And so on.  Most
>> of these should probably be HADOOP tickets since they sound like they
>> are mostly in the generic hadoop-common code.
>>
>> "solaris does not have errno" seems like a bold statement.  errno is
>> part of POSIX, and Solaris is a POSIX os, right?  Am I way off base on
>> this?
>> I googled around and one of the first results I found talked about
>> errno values on Solaris.
>>
>> http://www.pixelstech.net/article/1413273556-A-trick-of-building-multithreaded-application-on-Solaris
>>   Perhaps I misunderstood what you meant by this statement.
>>
>> Anyway, please file JIRAs for any portability improvements you can think
>> of!
>>
>> best,
>> Colin
>>
>> On Mon, Dec 8, 2014 at 9:09 PM, malcolm 
>> wrote:
>>>
>>> Hi Colin,
>>>
>>> A short summary of my changes are as follows:
>>>
>>> - Native C source files: added 5,  modified 6, requiring also changes to
>>> CMakeLists.txt. Of course, all changes are "ifdeffed" for Solaris
>>> appropriately and new files, are prefixed with solaris_ as well.
>>>
>>> For example, Solaris does not have errno, or errlist any more which are
>>> used
>>> quite a lot in hadoop native code. I could have replaced all calls to use
>>> strerror instead which would be compatible with Linux, however in the
>>> interests of making minimal changes, I recreated and added these files
>>> from
>>> a running Solaris machine instead.
>>>
>>> Another issue is that Solaris doesn't have the timeout option for
>>> sockets,
>>> so I had to write my own solaris_read routine with timeout and added it
>>> to
>>> DomainSocket.c . A few issues with lz4 on Sparc needed modification, and
>>> some other OS specific issues: getgrouplist, container-executer (from
>>> yarn).
>>>
>>> - Some very minor changes were made to some Java source files (mainly
>>> tests
>>> to get them to pass on Solaris)
>>>
>>> The above changes were made to 2.2, I will recheck everything against the
>>> latest trunk, maybe some fixes aren't needed any more.
>>>
>>> I have generated a single patch file with all changes. Perhaps it would
>>> be
>>> better to file multiple JIRAs for each change, perhaps grouped, one per
>>> issue ? Or should I file a JIRA for each modified source file ?
>>>
>>> Thank you,

Re: Solaris Port

[Colin McCabe]

;>   {
>>>   snprintf(buf, buf_len, "%s %d",
>>>   "Can't get system error message for code", code);
>>>   }
>>> #endif
>>> #endif
>>> }
>>>
>>> Note that HPUX does not have strerror_r() since strerror() itself is
>>> thread-safe.  Also Windows does not have snprintf().  The equivalent
>>> function _snprintf() has a subtle difference in its interface.
>>>
>>> -- Asokan
>>> 
>>> From: malcolm [malcolm.kaval...@oracle.com]
>>> Sent: Thursday, December 11, 2014 11:02 AM
>>> To: common-dev@hadoop.apache.org
>>> Subject: Re: Solaris Port
>>>
>>> Fine with me, I volunteer to do this, if accepted.
>>>
>>> On 12/11/2014 05:48 PM, Allen Wittenauer wrote:
>>>>
>>>> sys_errlist was removed for a reason.  Creating a fake sys_errlist on
>>>> Solaris will mean the libhadoop.so will need to be tied a specific build
>>>> (kernel/include pairing) and therefore limits upward 
>>>> mobility/compatibility.
>>>> That doesn’t seem like a very good idea.
>>>>
>>>> IMO, switching to strerror_r is much preferred, since other than the
>>>> brain-dead GNU libc version, is highly portable and should work regardless
>>>> of the kernel or OS in place.
>>>>
>>>> On Dec 11, 2014, at 5:20 AM, malcolm 
>>>> wrote:
>>>>
>>>>> FYI, there are a couple more files that reference sys_errlist directly
>>>>> (not just terror within exception.c) , but also hdfs_http_client.c and
>>>>> NativeiO.c
>>>>>
>>>>> On 12/11/2014 07:38 AM, malcolm wrote:
>>>>>>
>>>>>> Hi Colin,
>>>>>>
>>>>>> Exactly, as you noticed, the problem is the thread-local buffer needed
>>>>>> to return from terror.
>>>>>> Currently, terror just returns a static string from an array, this is
>>>>>> fast, simple and error-proof.
>>>>>>
>>>>>> In order to use strerror_r inside terror,  would require allocating a
>>>>>> buffer inside terror  and depend on the caller to free the buffer after
>>>>>> using it, or to pass a buffer to terrror (which is basically the same as
>>>>>> strerror_r, rendering terror redundant).
>>>>>> Both cases require modification outside terror itself, as far as I can
>>>>>> tell, no simple fix. Unless you have an alternative which I haven't 
>>>>>> thought
>>>>>> of ?
>>>>>>
>>>>>> As far as I can tell, we have two choices:
>>>>>>
>>>>>> 1. Remove terror and replace calls with strerror_r, passing a buffer
>>>>>> from the callee.
>>>>>>   Advantage: a more modern portable interface.
>>>>>>   Disadvantage: All calls to terror need to be modified, though
>>>>>> all seem to be in a few files as far as I can tell.
>>>>>>
>>>>>> 2. Adding a sys_errlist array (ifdeffed for Solaris)
>>>>>>   Advantage: no change to any calls to terror
>>>>>>   Disadvantage: 2 additional files added to source tree (.c and
>>>>>> .h) and some minor ifdefs only used for Solaris.
>>>>>>
>>>>>> I think it is more a question of style than anything else, so I leave
>>>>>> you to make the call.
>>>>>>
>>>>>> Thanks for your patience,
>>>>>> Malcolm
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12/10/2014 09:54 PM, Colin McCabe wrote:
>>>>>>>
>>>>>>> On Wed, Dec 10, 2014 at 2:31 AM, malcolm
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> Hi Colin,
>>>>>>>>
>>>>>>>> Thanks for the hints around JIRAs.
>>>>>>>>
>>>>>>>> You are correct errno still exists, however sys_errlist does not.
>>>>>>>>
>>>>>>>> Hadoop uses a function terror (defined in exception.c) which indexes
>>>>>>>> sys_errlist by errno to return the error message from the array.
>>>>>>>> This
>>>>>>>> function is called 26 times in various places (in 2.2)
>>>>>>>>
>>>>>>>> Originally, I thought to replace all calls to terror with strerror,
>>>>>>>> but
>>>>>>>> there can be issues with multi-threading (it returns a buffer which
>>>>>>>> can be
>>>>>>>> overwritten), so it seemed simpler just to recreate the sys_errlist
>>>>>>>> message
>>>>>>>> array.
>>>>>>>>
>>>>>>>> There is also a multi-threaded version strerror_r where you pass the
>>>>>>>> buffer
>>>>>>>> as a parameter, but this would necessitate changing every call to
>>>>>>>> terror
>>>>>>>> with mutiple lines of code.
>>>>>>>
>>>>>>> Why don't you just use strerror_r inside terror()?
>>>>>>>
>>>>>>> I wrote that code originally.  The reason I didn't want to use
>>>>>>> strerror_r there is because GNU libc provides a non-POSIX definition
>>>>>>> of strerror_r, and forcing it to use the POSIX one is a pain. But you
>>>>>>> can do it.  You also will require a thread-local buffer to hold the
>>>>>>> return from strerror_r, since it is not guaranteed to be static
>>>>>>> (although in practice it is 99% of the time-- another annoyance with
>>>>>>> the API).
>>>>>>>
>>>>>>>
>>> 
>>>
>>>
>>>
>>> ATTENTION: -
>>>
>>> The information contained in this message (including any files
>>> transmitted with this message) may contain proprietary, trade secret or
>>> other confidential and/or legally privileged information. Any pricing
>>> information contained in this message or in any files transmitted with this
>>> message is always confidential and cannot be shared with any third parties
>>> without prior written approval from Syncsort. This message is intended to be
>>> read only by the individual or entity to whom it is addressed or by their
>>> designee. If the reader of this message is not the intended recipient, you
>>> are on notice that any use, disclosure, copying or distribution of this
>>> message, in any form, is strictly prohibited. If you have received this
>>> message in error, please immediately notify the sender and/or Syncsort and
>>> destroy all copies of this message in your possession, custody or control.
>
>

Re: Solaris Port SOLVED!

[Colin McCabe]

c/main/winutils/libwinutils.c
>>>>
>>>>
>>>>
>>>> $ find ~/work/hadoop/hadoop-trunk/ -name '*.c'|xargs grep terror|awk
>>>> -F: '{print $1}'|sort -u
>>>>
>>>> /home/asokan/work/hadoop/hadoop-trunk/hadoop-common-project/hadoop-common/src/main/native/src/exception.c
>>>>
>>>>
>>>> /home/asokan/work/hadoop/hadoop-trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/SharedFileDescriptorFactory.c
>>>>
>>>>
>>>> /home/asokan/work/hadoop/hadoop-trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/net/unix/DomainSocket.c
>>>>
>>>>
>>>> /home/asokan/work/hadoop/hadoop-trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/net/unix/DomainSocketWatcher.c
>>>>
>>>>
>>>> /home/asokan/work/hadoop/hadoop-trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c
>>>>
>>>>
>>>>
>>>> This means you need not worry about the Windows version of terror().
>>>> You need to change five files that contain UNIX specific native code.
>>>>
>>>> I have a question on your suggested implementation:
>>>>
>>>> How do you initialize the static errlist array in a thread-safe manner?
>>>>
>>>> 
>>>> Here is another thread-safe implementation that I could come up with:
>>>>
>>>> #include 
>>>> #include 
>>>> #include 
>>>> #include 
>>>>
>>>> #define MESSAGE_BUFFER_SIZE 256
>>>>
>>>> char * getSystemErrorMessage(char * buf, int buf_len, int code) {
>>>> #if defined(_HPUX_SOURCE)
>>>> char * msg;
>>>> errno = 0;
>>>> msg = strerror(code);
>>>> if (errno == 0) {
>>>>   strncpy(buf, msg, buf_len-1);
>>>>   buf[buf_len-1] = '\0';
>>>> } else {
>>>>   snprintf(buf, buf_len, "%s %d",
>>>>   "Can't get system error message for code", code);
>>>> }
>>>> #else
>>>> if (strerror_r(code, buf, buf_len) != 0) {
>>>>   snprintf(buf, buf_len, "%s %d",
>>>>   "Can't get system error message for code", code);
>>>> }
>>>> #endif
>>>> return buf;
>>>> }
>>>>
>>>> #define TERROR(code) \
>>>> getSystemErrorMessage(messageBuffer, sizeof(messageBuffer), code)
>>>>
>>>> int main(int argc, char ** argv) {
>>>> if (argc > 1) {
>>>>   char messageBuffer[MESSAGE_BUFFER_SIZE];
>>>>   int code = atoi(argv[1]);
>>>>
>>>>   fprintf(stderr, "System error for code %s: %s\n", argv[1],
>>>> TERROR(code));
>>>> }
>>>> return 0;
>>>> }
>>>>
>>>>
>>>> This changes terror to a macro TERROR and requires all functions that
>>>> call TERROR macro to declare the local variable messageBuffer. Since
>>>> there are only five files to modify, I think it is not a big effort.
>>>> What do you think?
>>>>
>>>> -- Asokan
>>>>
>>>> On 12/13/2014 04:29 AM, malcolm wrote:
>>>> Colin,
>>>>
>>>> I am not sure what you mean by a thread-local buffer (in native
>>>> code). In Java this is pretty standard, but I couldn't find any
>>>> implementation for C code.
>>>>
>>>> Here is the terror function:
>>>>
>>>>   const char* terror(int errnum)
>>>>   {
>>>> if ((errnum < 0) || (errnum >= sys_nerr)) {
>>>>   return "unknown error.";
>>>> }
>>>> return sys_errlist[errnum];
>>>>   }
>>>>
>>>>
>>>> The interface is identical to strerror, but the implementation is
>>>> actually re-entrant since it returns a pointer to a static string.
>>>>
>>>> If I understand your suggestion, the new function would look like this:
>>>>
>>>>  const char* terror(int errnum)
>>>>  {
>>>> static char result[65];
>>>>
>>&g

Re: NFSv3 Filesystem Connector

[Colin McCabe]

Why not just use LocalFileSystem with an NFS mount (or several)?  I read
through the README but I didn't see that question answered anywhere.

best,
Colin

On Tue, Jan 13, 2015 at 1:35 PM, Gokul Soundararajan  wrote:

> Hi,
>
> We (Jingxin Feng, Xing Lin, and I) have been working on providing a
> FileSystem implementation that allows Hadoop to utilize a NFSv3 storage
> server as a filesystem. It leverages code from hadoop-nfs project for all
> the request/response handling. We would like your help to add it as part of
> hadoop tools (similar to the way hadoop-aws and hadoop-azure).
>
> In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+) and
> Apache Spark (1.2+) to use a NFSv3 storage server as a storage endpoint.
> The NFS Connector can be run in two modes: (1) secondary filesystem - where
> Hadoop/Spark runs using HDFS as its primary storage and can use NFS as a
> second storage endpoint, and (2) primary filesystem - where Hadoop/Spark
> runs entirely on a NFSv3 storage server.
>
> The code is written in a way such that existing applications do not have to
> change. All one has to do is to copy the connector jar into the lib/
> directory of Hadoop/Spark. Then, modify core-site.xml to provide the
> necessary details.
>
> The current version can be seen at:
> https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
>
> It is my first time contributing to the Hadoop codebase. It would be great
> if someone on the Hadoop team can guide us through this process. I'm
> willing to make the necessary changes to integrate the code. What are the
> next steps? Should I create a JIRA entry?
>
> Thanks,
>
> Gokul
>

Re: AARCH64 build broken

[Colin McCabe]

Good find.  I filed HADOOP-11505 to fix the incorrect usage of
unoptimized code on x86 and the incorrect bswap on alternative
architectures.

Let's address the fmemcmp stuff in a separate jira.

best,
Colin


On Thu, Jan 22, 2015 at 11:34 AM, Edward Nevill
 wrote:
> On 21 January 2015 at 11:42, Edward Nevill  wrote:
>
>> Hi,
>>
>> Hadoop currently does not build on ARM AARCH64. I have raised a JIRA issue
>> with a patch.
>>
>> https://issues.apache.org/jira/browse/HADOOP-11484
>>
>> I have submitted the patch and it builds OK and passes all the core tests.
>>
>
> Hi Colin,
>
> Thanks for pushing this patch.  Steve Loughran raised the issue in the card
> that although this patch fixes the ARM issue it does nothing for other
> archs.
>
> I would be happy to prepare a patch which makes it downgrade to C code on
> other CPU families if this would be useful.
>
> The general format would be
>
> #idef __aarch64__
>__asm__("ARM Asm")
> #elif defined(??X86??)
>__asm__("X86 Asm")
> #else
>C Implementation
> #endif
>
> My question is what to put for the defined(??X86??)
>
> According to the following page
>
> http://nadeausoftware.com/articles/2012/02/c_c_tip_how_detect_processor_type_using_compiler_predefined_macros
>
> the only way to fully detect all x86 variants is to write
>
> #if defined(__x86_64__) || defined(_M_X64) || defined(__i386) ||
> defined(_M_IX86)
>
> will detect all variants of 32 and 64 bit x86 across gcc and windows.
>
> Interestingly the bswap64 inline function in primitives.h has the following
>
> #ifdef __X64
>   __asm__("rev ");
> #else
>   C implementation
> #endif
>
> However if I compile Hadoop on my 64 bit Red Hat Enterprise Linux system it
> actually compiles the C implementation (I have verified this by putting a
> #error at the start of the C implementation. This is because the correct
> macro to detect 64 bit x86 on gcc is __x86_64__ I had also thought that the
> macro for windows was _M_X64 not __X64 but maybe __X64 works just as well
> on windows? Perhaps someone with access to windows development platform
> could do some tests and tell us what macros actually work.
>
> Another question is whether we actually care about 32 bit platforms, or can
> they just all downgrade to C code. Does anyone actually build Hadoop on a
> 32 bit platform?
>
> Another thing to be aware of is that there are endian dependncies in
> primitives.h, for example in fmemcmp() just a bit further down is the line
>
> return (int64_t)bswap(*(uint32_t*)src) -
> (int64_t)bswap(*(uint32_t*)dest);
>
> This is little endian dependant so will work on the likes of X86 and ARM
> but will fail on Sparc. Note, I haven't trawled looking for endian
> dependancies but this was one I just spotted while looking at the aarch64
> non compilation issue.
>
> All the best,
> Ed.

Re: Native style guide

[Colin McCabe]

There are two kinds of native code.

We've been roughly following the Linux Kernel Coding style in the C
code.  Details here:
https://www.kernel.org/doc/Documentation/CodingStyle
The main exception is that we use 4 spaces for indentation, not hard tabs.

For C++, there was a thread a while back about adopting the Google C++
Coding Style.  goto would not be acceptable in C++, since you would
use the "resource ownership is acquisition" (RAII) idiom.  There are
various kinds of smart pointer to help with that, although some of the
flavors of smart pointer create incompatibilities with pre-C++11 code.

best,
Colin

On Tue, Feb 3, 2015 at 3:19 PM, Chris Nauroth  wrote:
> Hi Steve,
>
> I don't believe we have an official statement of the native code style, so
> that's something that would be beneficial.
>
> To address your immediate question, yes, it is acceptable to use goto in
> error checking logic to jump to a labeled cleanup routine.  This is an
> established pattern throughout the native codebase.  One example file is
> NativeIO.c.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
>
>
>
> On 2/3/15, 1:39 PM, "Steve Loughran"  wrote:
>
>>
>>Do we have a native code style guide?
>>
>>Specifically, it is OK to use goto as a way of bailing out of things and
>>into some memory release logic as part of some JNI exception generation
>>routine?
>>
>>Don't worry, I'm asking on behalf of someone else
>>
>>
>>-Steve
>>
>>ps: what I have been intermittently doing is writing a proper style guide
>>for the Java side of things.
>>
>>https://github.com/steveloughran/formality/blob/master/styleguide/stylegui
>>de.md
>

Re: 2.7 status

[Colin McCabe]

+1 for starting thinking about releasing 2.7 soon.

Re: building Windows binaries.  Do we release binaries for all the
Linux and UNIX architectures?  I thought we didn't.  It seems a little
inconsistent to release binaries just for Windows, but not for those
other architectures and OSes.  I wonder if we can improve this
situation?

best,
Colin

On Fri, Feb 13, 2015 at 4:36 PM, Karthik Kambatla  wrote:
> 2 weeks from now (end of Feb) sounds reasonable. The one feature I would
> like for to be included is shared-cache: we are pretty close - two more
> main items to take care of.
>
> In an offline conversation, Steve mentioned building Windows binaries for
> our releases. Do we want to do that for 2.7? If so, can anyone with Windows
> expertise setup a Jenkins job to build these artifacts, and may be hook it
> up to https://builds.apache.org/job/HADOOP2_Release_Artifacts_Builder/
>
>
>
> On Fri, Feb 13, 2015 at 11:07 AM, Arun Murthy  wrote:
>
>> My bad, been sorted distracted.
>>
>> I agree, we should just roll fwd a 2.7 ASAP with all the goodies.
>>
>> What sort of timing makes sense? 2 week hence?
>>
>> thanks,
>> Arun
>>
>> 
>> From: Jason Lowe 
>> Sent: Friday, February 13, 2015 8:11 AM
>> To: common-dev@hadoop.apache.org
>> Subject: Re: 2.7 status
>>
>> I'd like to see a 2.7 release sooner than later.  It has been almost 3
>> months since Hadoop 2.6 was released, and there have already been 634 JIRAs
>> committed to 2.7.  That's a lot of changes waiting for an official release.
>>
>> https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2Chdfs%2Cyarn%2Cmapreduce%29%20AND%20fixversion%3D2.7.0%20AND%20resolution%3DFixed
>> Jason
>>
>>   From: Sangjin Lee 
>>  To: "common-dev@hadoop.apache.org" 
>>  Sent: Tuesday, February 10, 2015 1:30 PM
>>  Subject: 2.7 status
>>
>> Folks,
>>
>> What is the current status of the 2.7 release? I know initially it started
>> out as a "java-7" only release, but looking at the JIRAs that is very much
>> not the case.
>>
>> Do we have a certain timeframe for 2.7 or is it time to discuss it?
>>
>> Thanks,
>> Sangjin
>>
>>
>>
>
>
> --
> Karthik Kambatla
> Software Engineer, Cloudera Inc.
> 
> http://five.sentenc.es

Re: upstream jenkins build broken?

[Colin McCabe]

Is there a maven plugin or setting we can use to simply remove
directories that have no executable permissions on them?  Clearly we
have the permission to do this from a technical point of view (since
we created the directories as the jenkins user), it's simply that the
code refuses to do it.

Otherwise I guess we can just fix those tests...

Colin

On Tue, Mar 10, 2015 at 2:43 PM, Lei Xu  wrote:
> Thanks a lot for looking into HDFS-7722, Chris.
>
> In HDFS-7722:
> TestDataNodeVolumeFailureXXX tests reset data dir permissions in TearDown().
> TestDataNodeHotSwapVolumes reset permissions in a finally clause.
>
> Also I ran mvn test several times on my machine and all tests passed.
>
> However, since in DiskChecker#checkDirAccess():
>
> private static void checkDirAccess(File dir) throws DiskErrorException {
>   if (!dir.isDirectory()) {
> throw new DiskErrorException("Not a directory: "
>  + dir.toString());
>   }
>
>   checkAccessByFileMethods(dir);
> }
>
> One potentially safer alternative is replacing data dir with a regular
> file to stimulate disk failures.
>
> On Tue, Mar 10, 2015 at 2:19 PM, Chris Nauroth  
> wrote:
>> TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure,
>> TestDataNodeVolumeFailureReporting, and
>> TestDataNodeVolumeFailureToleration all remove executable permissions from
>> directories like the one Colin mentioned to simulate disk failures at data
>> nodes.  I reviewed the code for all of those, and they all appear to be
>> doing the necessary work to restore executable permissions at the end of
>> the test.  The only recent uncommitted patch I¹ve seen that makes changes
>> in these test suites is HDFS-7722.  That patch still looks fine though.  I
>> don¹t know if there are other uncommitted patches that changed these test
>> suites.
>>
>> I suppose it¹s also possible that the JUnit process unexpectedly died
>> after removing executable permissions but before restoring them.  That
>> always would have been a weakness of these test suites, regardless of any
>> recent changes.
>>
>> Chris Nauroth
>> Hortonworks
>> http://hortonworks.com/
>>
>>
>>
>>
>>
>>
>> On 3/10/15, 1:47 PM, "Aaron T. Myers"  wrote:
>>
>>>Hey Colin,
>>>
>>>I asked Andrew Bayer, who works with Apache Infra, what's going on with
>>>these boxes. He took a look and concluded that some perms are being set in
>>>those directories by our unit tests which are precluding those files from
>>>getting deleted. He's going to clean up the boxes for us, but we should
>>>expect this to keep happening until we can fix the test in question to
>>>properly clean up after itself.
>>>
>>>To help narrow down which commit it was that started this, Andrew sent me
>>>this info:
>>>
>>>"/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
>>>Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/ has
>>>500 perms, so I'm guessing that's the problem. Been that way since 9:32
>>>UTC
>>>on March 5th."
>>>
>>>--
>>>Aaron T. Myers
>>>Software Engineer, Cloudera
>>>
>>>On Tue, Mar 10, 2015 at 1:24 PM, Colin P. McCabe 
>>>wrote:
>>>
 Hi all,

 A very quick (and not thorough) survey shows that I can't find any
 jenkins jobs that succeeded from the last 24 hours.  Most of them seem
 to be failing with some variant of this message:

 [ERROR] Failed to execute goal
 org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean)
 on project hadoop-hdfs: Failed to clean project: Failed to delete


/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-hdfs-pr
oject/hadoop-hdfs/target/test/data/dfs/data/data3
 -> [Help 1]

 Any ideas how this happened?  Bad disk, unit test setting wrong
 permissions?

 Colin

>>
>
>
>
> --
> Lei (Eddy) Xu
> Software Engineer, Cloudera