Re: Compatibility in Apache Hadoop

2013-04-23 Thread Karthik Kambatla
On Tue, Apr 23, 2013 at 9:00 AM, Steve Loughran wrote: > On 22 April 2013 18:32, Eli Collins wrote: > > > On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran > > wrote: > > > > > > > > There's a separate issue that says "we make some guarantee that the > > > behaviour of a interface remains consiste

Re: Compatibility in Apache Hadoop

2013-04-23 Thread Andrew Purtell
API jars only containing interfaces would work. They would be small and lightweight, and not carry dependencies other than other API modules presumably. Some additional details of factoring out the codecs might be debatable (e.g. native code loading) but I do feel it is a useful motivating example

Re: Compatibility in Apache Hadoop

2013-04-23 Thread Alejandro Abdelnur
Andrew, Or with a twist, why not break/consolidate things as follows? common API common IMPL hdfs CLIENT IMPL hdfs SERVER IMPL hdfs TOOLS CLIENT yarn API yarn CLIENT IMPL yarn SERVER IMPL yarn TOOLS mapred API mapred IMPL mapred TOOLS IMO, this would help significantly to reduce dependency hell

Re: Compatibility in Apache Hadoop

2013-04-23 Thread Steve Loughran
On 23 April 2013 11:32, Andrew Purtell wrote: > At the risk of hijacking this conversation a bit, what do you think of the > notion of moving interfaces like Seekable and PositionedReadable into a new > foundational Maven module, perhaps just for such interfaces that define and > tag support for

Re: Compatibility in Apache Hadoop

2013-04-23 Thread Andrew Purtell
At the risk of hijacking this conversation a bit, what do you think of the notion of moving interfaces like Seekable and PositionedReadable into a new foundational Maven module, perhaps just for such interfaces that define and tag support for core semantics, as their details are better defined and

Re: Compatibility in Apache Hadoop

2013-04-23 Thread Steve Loughran
On 23 April 2013 09:00, Steve Loughran wrote: > > > On 22 April 2013 18:32, Eli Collins wrote: > >> >> > >> However if a change made FileSystem#close three times slower, this >> perhaps a smaller semantic change (eg doesn't change what exceptions >> get thrown) but probably much less tolerable f

Re: Compatibility in Apache Hadoop

2013-04-23 Thread Steve Loughran
On 22 April 2013 18:32, Eli Collins wrote: > On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran > wrote: > > > > > There's a separate issue that says "we make some guarantee that the > > behaviour of a interface remains consistent over versions", which is hard > > to do without some rigorous defini

Fwd: Compatibility in Apache Hadoop

2013-04-22 Thread Eli Collins
On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran wrote: > > On 22 April 2013 14:00, Karthik Kambatla wrote: > > > Hadoop devs, > > > > > > This doc does not intend to propose new policies. The idea is to have one > > document that outlines the various compatibility concerns (lots of areas > > beyo

Re: Compatibility in Apache Hadoop

2013-04-22 Thread Steve Loughran
On 22 April 2013 14:00, Karthik Kambatla wrote: > Hadoop devs, > > > This doc does not intend to propose new policies. The idea is to have one > document that outlines the various compatibility concerns (lots of areas > beyond API compatibility), captures the respective policies that exist, and >

Compatibility in Apache Hadoop

2013-04-22 Thread Karthik Kambatla
Hadoop devs, Eli and I have been working on a writeup on compatibility for developers, downstream projects and end users. The wiki page - http://wiki.apache.org/hadoop/Compatibility - is a strawman effort towards consolidating prior discussions and decisions; including the API classification effor