Re: Developing cross-component patches post-split

2009-07-17 Thread Doug Cutting
Giridharan Kesavan wrote: I agree with you on this; We have some common places on people server people.apache.org/repo and people.apache.org/repository but I've not seen any ivy artifacts being published there. Writing things there causes them to be published to repo{1,2}.maven.org. http://ww

RE: Developing cross-component patches post-split

2009-07-17 Thread Giridharan Kesavan
ache.org] > Sent: Friday, July 17, 2009 11:15 PM > To: common-dev@hadoop.apache.org > Subject: Re: Developing cross-component patches post-split > > > On Jul 16, 2009, at 3:54 AM, Giridharan Kesavan wrote: > > > 2) Publishing artifacts to the people.apache.org > &g

Re: Developing cross-component patches post-split

2009-07-17 Thread Owen O'Malley
On Jul 16, 2009, at 3:54 AM, Giridharan Kesavan wrote: 2) Publishing artifacts to the people.apache.org ssh resolver is configured which publishes common/hdfs/mapred artifacts to my home folder /home/gkesavan/ivyrepo I think that publishing to your home directory is a mistake. We need so

RE: Developing cross-component patches post-split

2009-07-16 Thread Giridharan Kesavan
sage- > From: Scott Carey [mailto:sc...@richrelevance.com] > Sent: Thursday, July 02, 2009 10:32 PM > To: common-dev@hadoop.apache.org > Subject: Re: Developing cross-component patches post-split > > > On 7/1/09 11:58 PM, "Nigel Daley" wrote: > > > >

Re: Developing cross-component patches post-split

2009-07-03 Thread Steve Loughran
Owen O'Malley wrote: On Wed, Jul 1, 2009 at 6:45 PM, Todd Lipcon wrote: Agree with Phillip here. Requiring a new jar to be checked in anywhere after every common commit seems unscalable and nonperformant. For git users this will make the repository size baloon like crazy (the jar is 400KB and we

Re: Developing cross-component patches post-split

2009-07-03 Thread Steve Loughran
Todd Lipcon wrote: On Wed, Jul 1, 2009 at 10:10 PM, Raghu Angadi wrote: -1 for committing the jar. Most of the various options proposed sound certainly better. Can build.xml be updated such that Ivy fetches recent (nightly) build? This seems slightly better than actually committing the ja

Re: Developing cross-component patches post-split

2009-07-03 Thread Steve Loughran
Todd Lipcon wrote: On Wed, Jul 1, 2009 at 2:10 PM, Philip Zeyliger wrote: -1 to checking in jars. It's quite a bit of bloat in the repository (which admittedly affects the git.apache folks more than the svn folks), but it's also cumbersome to develop. It'd be nice to have a one-liner that bu

Re: Developing cross-component patches post-split

2009-07-03 Thread Steve Loughran
Marcus Herou wrote: Hi. My 5 cents about svn:externals. I could not live without it but...I always tend to forget to update our svn:externals and about once a month I wonder why I accidentally released bleeding edge code in our production environent *smile* (should've written that auto-branching

Re: Developing cross-component patches post-split

2009-07-03 Thread Enis Soztutar
I have been using ivy for a multi-module project which have similar independent modules built separately. I think the way to got is like : 1. Let the projects be checked out independently and in arbitrary directory structure 2. Define ivy repositories in the following order : local, maven 3. A

Re: Developing cross-component patches post-split

2009-07-02 Thread Scott Carey
On 7/1/09 11:58 PM, "Nigel Daley" wrote: > > > On Jul 1, 2009, at 10:16 PM, Todd Lipcon wrote: > >> On Wed, Jul 1, 2009 at 10:10 PM, Raghu Angadi > inc.com> wrote: >> >>> >>> -1 for committing the jar. >>> >>> Most of the various options proposed sound certainly better. >>> >>> Can build.

Re: Developing cross-component patches post-split

2009-07-01 Thread Nigel Daley
On Jul 1, 2009, at 10:16 PM, Todd Lipcon wrote: On Wed, Jul 1, 2009 at 10:10 PM, Raghu Angadi inc.com> wrote: -1 for committing the jar. Most of the various options proposed sound certainly better. Can build.xml be updated such that Ivy fetches recent (nightly) build? +1. Using ant co

Re: Developing cross-component patches post-split

2009-07-01 Thread Owen O'Malley
On Wed, Jul 1, 2009 at 6:45 PM, Todd Lipcon wrote: > Agree with Phillip here. Requiring a new jar to be checked in anywhere after > every common commit seems unscalable and nonperformant. For git users this > will make the repository size baloon like crazy (the jar is 400KB and we > have around 530

Re: Developing cross-component patches post-split

2009-07-01 Thread Philip Zeyliger
> > For example, > ant test > does the same as it does now, but > ant -Dhadoop.common.jar=/home/dhruba/common/hadoop-common.jar test will > pick up the common jar from my home directory. +1; we'll also need the same sort of thing for the daemons. Regardless of whether or not jars are checked i

Re: Developing cross-component patches post-split

2009-07-01 Thread Sharad Agarwal
Raghu Angadi wrote: Can build.xml be updated such that Ivy fetches recent (nightly) build? HDFS could have a build target that builds common jar from a specified source location for common. +1 This is the standard way to go. Ivy can fetch the latest nightly build or from the last successful

Re: Developing cross-component patches post-split

2009-07-01 Thread Todd Lipcon
On Wed, Jul 1, 2009 at 10:10 PM, Raghu Angadi wrote: > > -1 for committing the jar. > > Most of the various options proposed sound certainly better. > > Can build.xml be updated such that Ivy fetches recent (nightly) build? > This seems slightly better than actually committing the jars. However,

Re: Developing cross-component patches post-split

2009-07-01 Thread Todd Lipcon
On Wed, Jul 1, 2009 at 10:11 PM, Dhruba Borthakur wrote: > Hi Todd, > > Another option (one that is used by Hive) is to have an ant macro that can > be overridden from the ant command line. This macro points to the location > of the common.jar. By default, it is set to the same value as it is now

Re: Developing cross-component patches post-split

2009-07-01 Thread Raghu Angadi
-1 for committing the jar. Most of the various options proposed sound certainly better. Can build.xml be updated such that Ivy fetches recent (nightly) build? HDFS could have a build target that builds common jar from a specified source location for common. Raghu. Todd Lipcon wrote: On Wed

Re: Developing cross-component patches post-split

2009-07-01 Thread Dhruba Borthakur
Hi Todd, Another option (one that is used by Hive) is to have an ant macro that can be overridden from the ant command line. This macro points to the location of the common.jar. By default, it is set to the same value as it is now. If a developer has a common jar that is built in his/her directory

Re: Developing cross-component patches post-split

2009-07-01 Thread Todd Lipcon
On Wed, Jul 1, 2009 at 2:10 PM, Philip Zeyliger wrote: > -1 to checking in jars. It's quite a bit of bloat in the repository (which > admittedly affects the git.apache folks more than the svn folks), but it's > also cumbersome to develop. > > It'd be nice to have a one-liner that builds the equi

Re: Developing cross-component patches post-split

2009-07-01 Thread Marcus Herou
Hi. My 5 cents about svn:externals. I could not live without it but...I always tend to forget to update our svn:externals and about once a month I wonder why I accidentally released bleeding edge code in our production environent *smile* (should've written that auto-branching script waaay back ago

Re: Developing cross-component patches post-split

2009-07-01 Thread Philip Zeyliger
> > - When we have one of these cross-project changes, how are we supposed to >> do >> builds? >> > > I talked with Owen & Nigel about this, and we thought that, for now, it > might be reasonable to have the mapreduce and hdfs trunk each contain an svn > external link to the current common jar. T

Re: Developing cross-component patches post-split

2009-07-01 Thread Jakob Homan
Todd Lipcon wrote: >> If someone could write up some kind of "post-split transition guide" >> on the Wiki I think that would be generally appreciated. Here's something I wrote up for re-setting up Eclipse after the split in a way that gives relatively seamless access to the various projects' so

Re: Developing cross-component patches post-split

2009-07-01 Thread Doug Cutting
Todd Lipcon wrote: - Whenever a task will need to touch both Common and one of the components (Mapred/HDFS) should there be two JIRAs or is it sufficient to have just one "HADOOP" JIRA with separate patches uploaded for the two repositories? Two Jiras, I think. In the long run, such issues sho

Developing cross-component patches post-split

2009-07-01 Thread Todd Lipcon
Hey all, I think there's general confusion right now about how the development process is supposed to work now that the project split has taken place. Or, at least, I am generally confused :) Here are a couple questions I have lingering: - Whenever a task will need to touch both Common and one of