The current Avro MR integration is done trough composition and I think that's the better way to go indeed, so I haven't created a patch from my solution. Also, my solution suffers from copy-paste reuse of code already present in the current Avro MR integration. I wouldn't suggest making it part of the Avro source. It just works for me for now, so I thought I'd share it.
Problem with inheritance would be that anything that changes on the Hadoop side of things, automatically leaks into the Avro side of things, which may not be the preference. That said, the new mapreduce API looks to be designed for inheritance (with Mapper and Reducer being classes instead of interfaces). So, in my approach, the Avro job is just a regular mapreduce job that knows about the fact that the input, output and shuffle are done using Avro records. To go the composition way, you would nee Avro counterparts of these: - Mapper - Reducer - MapContext extends TaskInputOutputContext extends TaskAttemptContext extends JobContext implements Progressable - ReduceContext extends TaskInputOutputContext extends TaskAttemptContext extends JobContext implements Progressable Problem are the last two. To give Avro jobs the same kind of control as regular jobs, you'd need to implement anything that's present in that inheritance tree. If the API were super stable and nothing would ever change in these five levels of inheritance, that would be OK. It would be a lot nicer, though, if there was a Context interface that defines the contract for a context object passed to a mapper or reducer. So when I encountered this, I didn't bother to do the complete composition and went for the easy (inheritance) way out. Friso On 13 nov. 2011, at 21:23, Scott Carey wrote: I have heard some suggestions that it would be useful if we could somehow model Avro's interaction with mapreduce using composition rather than inheritance. Has anyone tried that? Or would it be too clumsy? A good relationship with the mapreduce/mapred api via composition might require changes on the hadoop side however. On 11/13/11 5:04 AM, "Friso van Vollenhoven" <[email protected]<mailto:[email protected]>> wrote: Hi, I use my own set of classes for this. I mostly copied from / modeled after the Avro mapred support for the old API. My approach is slightly different, though. The existing MR support fully abstracts / wraps away the Hadoop MR API and only exposes the Avro one. The only Hadoop API that the Avro classes see is the Configuration object. Problem is that in the new API, the Configuration object is kept within a context instance and you'd need to wrap the whole context thing and give the wrapper to the Avro mapper and reducer. This felt a bit overkill so I chose to just make mapper and reducer subclasses that handle the Avro work and then call a protected method to do the actual mapping or reducing. Problem is that you lose the property of a bare mapper or reducer being the identity function, but you could reintroduce this in a generic way, I think. I just don't use the identity functions a lot in practice, so I didn't bother. I pushed the code here: https://github.com/friso/avro-mapreduce. There is a unit test with some usage examples. Cheers, Friso On 11 nov. 2011, at 20:43, Doug Cutting wrote: On 11/10/2011 12:38 AM, Andrew Kenworthy wrote: Are there plans to extend it to work with org.apache.hadoop.mapreduce as well? There's an issue in Jira for this: https://issues.apache.org/jira/browse/AVRO-593 I don't know of anyone actively working on this at present. It would be a great addition to Avro and I am hopeful someone will resume work on it soon. Doug
