Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-28 Thread Dhruba Borthakur
I think we should not require Job Q compatibility for 1.0 release. thanks, dhruba On Mon, Sep 28, 2009 at 11:06 AM, Sanjay Radia wrote: > > On Sep 28, 2009, at 3:15 AM, Steve Loughran wrote: > > Dhruba Borthakur wrote: >> > It is really nice to have wire-compatibility between clients and serv

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-28 Thread Sanjay Radia
On Sep 28, 2009, at 3:15 AM, Steve Loughran wrote: Dhruba Borthakur wrote: > It is really nice to have wire-compatibility between clients and servers > running different versions of hadoop. The reason we would like this is > because we can allow the same client (Hive, etc) submit jobs to t

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-28 Thread Andrew Purtell
: Stronger API Compatibility from 0.21 onwards Dhruba Borthakur wrote: > It is really nice to have wire-compatibility between clients and servers > running different versions of hadoop. The reason we would like this is > because we can allow the same client (Hive, etc) submit jobs to two &g

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-28 Thread Steve Loughran
Dhruba Borthakur wrote: It is really nice to have wire-compatibility between clients and servers running different versions of hadoop. The reason we would like this is because we can allow the same client (Hive, etc) submit jobs to two different clusters running different versions of hadoop. But

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Allen Wittenauer
On 9/25/09 2:40 PM, "Doug Cutting" wrote: > Would it be materially better for you if we waited longer before calling > a release 1.0, assuming that the same features are released in the same > order and on the same schedule regardless of the release name? Yes. There is something magic to manager

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Doug Cutting
Allen Wittenauer wrote: Oh, I completely understand. I'm just throwing in a non-developer's opinion... because I'm sure I'm not the only one expecting/assuming that 1.0 == completely stable. If we have to live up to that expectation then we might never release 1.0! Frankly, I fear the long

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Allen Wittenauer
On 9/25/09 1:18 PM, "Doug Cutting" wrote: > The question is not whether wire compatibility is a good thing. The > question is whether API compatibility is useless without wire > compatibility and, vice versa, whether wire compatibility is useless > without API compatibility. They're both valu

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Doug Cutting
Allen Wittenauer wrote: This is just so disappointing and, quite frankly, makes 1.0 less than useful for Real Work. Great, the APIs don't change but you still have the same problems of getting data on/off the grid without upgrading your clients every time. To me, without wire compatibility,

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Allen Wittenauer
On 9/25/09 12:44 PM, "Sanjay Radia" wrote: > > On Sep 25, 2009, at 12:03 PM, Allen Wittenauer wrote: > >> On 9/25/09 10:13 AM, "Dhruba Borthakur" wrote: >>> It is really nice to have wire-compatibility between clients and >> servers >>> running different versions of hadoop. The reason we wo

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Sanjay Radia
On Sep 25, 2009, at 12:03 PM, Allen Wittenauer wrote: On 9/25/09 10:13 AM, "Dhruba Borthakur" wrote: > It is really nice to have wire-compatibility between clients and servers > running different versions of hadoop. The reason we would like this is > because we can allow the same client (

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Allen Wittenauer
On 9/25/09 10:13 AM, "Dhruba Borthakur" wrote: > It is really nice to have wire-compatibility between clients and servers > running different versions of hadoop. The reason we would like this is > because we can allow the same client (Hive, etc) submit jobs to two > different clusters running diff

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Dhruba Borthakur
It is really nice to have wire-compatibility between clients and servers running different versions of hadoop. The reason we would like this is because we can allow the same client (Hive, etc) submit jobs to two different clusters running different versions of hadoop. But I am not stuck up on the n

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Doug Cutting
Sanjay Radia wrote: Both Facebook (Dhruba tells me) and Yahoo are suffering badly from the lack of wire compatibility - a major motivaiton for Yahoo to develop Avro. Indeed. Wire compatibility is a crucial feature that we should release as soon as possible. Perhaps before 1.0 if 1.0 slips,

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-09-25 Thread Sanjay Radia
On Aug 28, 2009, at 3:15 PM, Doug Cutting wrote: Sanjay Radia wrote: > No. The 1.0 proposal was that it included both API and wire compatibility. The proposal includes a lot of things, but it's so far just a proposal. There's been no vote to formally define what 1.0 will mean. In eve

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-08-28 Thread Doug Cutting
Sanjay Radia wrote: No. The 1.0 proposal was that it included both API and wire compatibility. The proposal includes a lot of things, but it's so far just a proposal. There's been no vote to formally define what 1.0 will mean. In every discussion I've heard, from the very beginning of the p

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-08-28 Thread Sanjay Radia
On Aug 28, 2009, at 10:35 AM, Doug Cutting wrote: Sanjay Radia wrote: > I propose that we honor the compatibility of stable interfaces from > release 0.21 onwards; > i.e. apply the same post 1.0 rules to pre-1.0 releases. I think that makes 0.21 effectively 1.0. The hallmark of 1.0 will be o

Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-08-28 Thread Doug Cutting
Sanjay Radia wrote: I propose that we honor the compatibility of stable interfaces from release 0.21 onwards; i.e. apply the same post 1.0 rules to pre-1.0 releases. I think that makes 0.21 effectively 1.0. The hallmark of 1.0 will be our promise not to change APIs incompatibly until 2.0.

Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards

2009-08-28 Thread Sanjay Radia
Hadoop 1.0's goal was compatibility on several fronts. (See https://issues.apache.org/jira/browse/HADOOP-5071) for details. Due to the amount of work involved, it has been necessary to split this work across several releases prior to 1.0. Turns out that release 0.21 has a number of Jiras ta