Re: Developing cross-component patches post-split

2009-07-01 Thread Nigel Daley
On Jul 1, 2009, at 10:16 PM, Todd Lipcon wrote: On Wed, Jul 1, 2009 at 10:10 PM, Raghu Angadi inc.com> wrote: -1 for committing the jar. Most of the various options proposed sound certainly better. Can build.xml be updated such that Ivy fetches recent (nightly) build? +1. Using ant co

Re: [VOTE] Release Hadoop 0.19.2 (candidate 0)

2009-07-01 Thread Owen O'Malley
On Wed, Jul 1, 2009 at 8:09 PM, jason hadoop wrote: > It is a bug fix, It removes an undocumented limitation. *sigh* If I remember right, that patch breaks compatibility. If so, I'd vote against putting it into any of the release branches. -- Owen

Re: Developing cross-component patches post-split

2009-07-01 Thread Owen O'Malley
On Wed, Jul 1, 2009 at 6:45 PM, Todd Lipcon wrote: > Agree with Phillip here. Requiring a new jar to be checked in anywhere after > every common commit seems unscalable and nonperformant. For git users this > will make the repository size baloon like crazy (the jar is 400KB and we > have around 530

Re: Developing cross-component patches post-split

2009-07-01 Thread Philip Zeyliger
> > For example, > ant test > does the same as it does now, but > ant -Dhadoop.common.jar=/home/dhruba/common/hadoop-common.jar test will > pick up the common jar from my home directory. +1; we'll also need the same sort of thing for the daemons. Regardless of whether or not jars are checked i

Re: Developing cross-component patches post-split

2009-07-01 Thread Sharad Agarwal
Raghu Angadi wrote: Can build.xml be updated such that Ivy fetches recent (nightly) build? HDFS could have a build target that builds common jar from a specified source location for common. +1 This is the standard way to go. Ivy can fetch the latest nightly build or from the last successful

Re: Developing cross-component patches post-split

2009-07-01 Thread Todd Lipcon
On Wed, Jul 1, 2009 at 10:10 PM, Raghu Angadi wrote: > > -1 for committing the jar. > > Most of the various options proposed sound certainly better. > > Can build.xml be updated such that Ivy fetches recent (nightly) build? > This seems slightly better than actually committing the jars. However,

Re: Developing cross-component patches post-split

2009-07-01 Thread Todd Lipcon
On Wed, Jul 1, 2009 at 10:11 PM, Dhruba Borthakur wrote: > Hi Todd, > > Another option (one that is used by Hive) is to have an ant macro that can > be overridden from the ant command line. This macro points to the location > of the common.jar. By default, it is set to the same value as it is now

Re: Developing cross-component patches post-split

2009-07-01 Thread Raghu Angadi
-1 for committing the jar. Most of the various options proposed sound certainly better. Can build.xml be updated such that Ivy fetches recent (nightly) build? HDFS could have a build target that builds common jar from a specified source location for common. Raghu. Todd Lipcon wrote: On Wed

Re: Developing cross-component patches post-split

2009-07-01 Thread Dhruba Borthakur
Hi Todd, Another option (one that is used by Hive) is to have an ant macro that can be overridden from the ant command line. This macro points to the location of the common.jar. By default, it is set to the same value as it is now. If a developer has a common jar that is built in his/her directory

Re: [VOTE] Release Hadoop 0.19.2 (candidate 0)

2009-07-01 Thread jason hadoop
It is a bug fix, it fixes the mapside join to work as documented instead of having an undocumented failure case when you have more than 32 tables in your join statement. On Wed, Jul 1, 2009 at 5:40 PM, Nigel Daley wrote: > HADOOP-5589 is not a bug fix so it shouldn't go into branch-0.19. > > Nig

Re: Developing cross-component patches post-split

2009-07-01 Thread Todd Lipcon
On Wed, Jul 1, 2009 at 2:10 PM, Philip Zeyliger wrote: > -1 to checking in jars. It's quite a bit of bloat in the repository (which > admittedly affects the git.apache folks more than the svn folks), but it's > also cumbersome to develop. > > It'd be nice to have a one-liner that builds the equi

Re: [VOTE] Release Hadoop 0.19.2 (candidate 0)

2009-07-01 Thread Nigel Daley
HADOOP-5589 is not a bug fix so it shouldn't go into branch-0.19. Nige On Jul 1, 2009, at 7:49 AM, jason hadoop wrote: Can you put http://issues.apache.org/jira/browse/HADOOP-5589 in please? On Wed, Jul 1, 2009 at 2:44 AM, Tom White wrote: I have created a candidate build for Hadoop 0.

[jira] Created: (HADOOP-6125) extend DistributedCache to work locally (LocalJobRunner) (common half)

2009-07-01 Thread Philip Zeyliger (JIRA)
extend DistributedCache to work locally (LocalJobRunner) (common half) -- Key: HADOOP-6125 URL: https://issues.apache.org/jira/browse/HADOOP-6125 Project: Hadoop Common Issu

Re: Developing cross-component patches post-split

2009-07-01 Thread Marcus Herou
Hi. My 5 cents about svn:externals. I could not live without it but...I always tend to forget to update our svn:externals and about once a month I wonder why I accidentally released bleeding edge code in our production environent *smile* (should've written that auto-branching script waaay back ago

Re: Developing cross-component patches post-split

2009-07-01 Thread Philip Zeyliger
> > - When we have one of these cross-project changes, how are we supposed to >> do >> builds? >> > > I talked with Owen & Nigel about this, and we thought that, for now, it > might be reasonable to have the mapreduce and hdfs trunk each contain an svn > external link to the current common jar. T

Re: Developing cross-component patches post-split

2009-07-01 Thread Jakob Homan
Todd Lipcon wrote: >> If someone could write up some kind of "post-split transition guide" >> on the Wiki I think that would be generally appreciated. Here's something I wrote up for re-setting up Eclipse after the split in a way that gives relatively seamless access to the various projects' so

Re: Developing cross-component patches post-split

2009-07-01 Thread Doug Cutting
Todd Lipcon wrote: - Whenever a task will need to touch both Common and one of the components (Mapred/HDFS) should there be two JIRAs or is it sufficient to have just one "HADOOP" JIRA with separate patches uploaded for the two repositories? Two Jiras, I think. In the long run, such issues sho

Developing cross-component patches post-split

2009-07-01 Thread Todd Lipcon
Hey all, I think there's general confusion right now about how the development process is supposed to work now that the project split has taken place. Or, at least, I am generally confused :) Here are a couple questions I have lingering: - Whenever a task will need to touch both Common and one of

Re: [jira] Commented: (HADOOP-6120) Add support for Avro types in hadoop

2009-07-01 Thread Konstantin Boudnik
Sorry, please disregard this message - I've sent it to the wrong lists. On 7/1/09 1:04 PM, Konstantin Boudnik wrote: Thunderbird and Mutt are at least two which does thread JIRA's threads properly. The way they do so is by using Message-ID and In-Reply-To fields of a RFC-822 header. Cos On 6/2

Re: [jira] Commented: (HADOOP-6120) Add support for Avro types in hadoop

2009-07-01 Thread Konstantin Boudnik
Thunderbird and Mutt are at least two which does thread JIRA's threads properly. The way they do so is by using Message-ID and In-Reply-To fields of a RFC-822 header. Cos On 6/29/09 11:31 PM, Sharad Agarwal (JIRA) wrote: [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlas

Task tracker shutdown given inactivity

2009-07-01 Thread Faisal Khan
Hi everyone, Is it possible to configure Hadoop's TaskTracker in a way that it shutdown itself once being idle (no task to run) for a configurable amount of time? I guess it is not possible in current implementation. This feature is very useful in situations (like of ours) where a MapReduce jobs

Re: [VOTE] Release Hadoop 0.19.2 (candidate 0)

2009-07-01 Thread Chris K Wensel
+1 cascading unit/regression tests pass thanks Tom! ckw On Jul 1, 2009, at 2:44 AM, Tom White wrote: I have created a candidate build for Hadoop 0.19.2. This fixes 42 issues in 0.19.1 (http://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=DESC&sorter/fiel

Re: [VOTE] Release Hadoop 0.19.2 (candidate 0)

2009-07-01 Thread jason hadoop
Can you put http://issues.apache.org/jira/browse/HADOOP-5589 in please? On Wed, Jul 1, 2009 at 2:44 AM, Tom White wrote: > I have created a candidate build for Hadoop 0.19.2. This fixes 42 > issues in 0.19.1 > ( > http://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&so

[VOTE] Release Hadoop 0.19.2 (candidate 0)

2009-07-01 Thread Tom White
I have created a candidate build for Hadoop 0.19.2. This fixes 42 issues in 0.19.1 (http://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=DESC&sorter/field=priority&pid=12310240&fixfor=12313650). *** Please download, test and vote before the *** vote closes on

Build failed in Hudson: Hadoop-Patch-vesta.apache.org #540

2009-07-01 Thread Apache Hudson Server
See http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/540/ -- started Building remotely on vesta.apache.org (Ubuntu) Location 'http://svn.apache.org/repos/asf/hadoop/core/trunk' does not exist Location could be expanded on build 'Hadoop