Hi Aljoscha, avro-mapred jar contains different M/R output formats for avro, and their writers it's primary used in M/R jobs that produce avro output see some details here : https://avro.apache.org/docs/1.7.6/mr.html
I have extracted(kind of copy-pasted+adjustments) some of the classes from there to remove this dependency and it's seems to work with basic scenario I still want to write it as if it was created with M/R job to be compatible with this library(e.g. key-value pairs are wrapped into AvroKeyValue object) so that it's not important if Flink or regular M/R created this files, still their consumer can read them in a same way WDYT? On 27 April 2016 at 11:27, Aljoscha Krettek <aljos...@apache.org> wrote: > Hi, > which code did you reuse from there? I asked Robert and I think it is > somewhat problematic to add these somewhat bigger dependencies. > > Cheers, > Aljoscha > > On Mon, 25 Apr 2016 at 21:24 Igor Berman <igor.ber...@gmail.com> wrote: > >> Hi, >> it's not a problem, I'll find time to change it(I understand the >> refactoring is in master and not released yet). >> Wanted to ask if it's acceptable to add following dependency to flink? >> I mean my code reused code in this jar(pay attention it's not present >> currently in flink classpath) >> <dependency> >> <groupId>org.apache.avro</groupId> >> <artifactId>avro-mapred</artifactId> >> <version>1.7.6</version> >> <classifier>hadoop2</classifier> >> </dependency> >> >> On 25 April 2016 at 16:20, Aljoscha Krettek <aljos...@apache.org> wrote: >> >>> Hi, >>> the code looks very good! Do you think it can be adapted to the slightly >>> modified interface introduced here: >>> https://issues.apache.org/jira/browse/FLINK-3637 >>> >>> It basically requires the writer to know the write position, so that we >>> can truncate to a valid position in case of failure. >>> >>> Cheers, >>> Aljoscha >>> >>> On Thu, 21 Apr 2016 at 18:40 Igor Berman <igor.ber...@gmail.com> wrote: >>> >>>> ok, >>>> I have working prototype already, if somebody is interested(attached) >>>> >>>> I might add it as PR latter(with tests etc) >>>> >>>> tested locally & with s3 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 21 April 2016 at 12:01, Aljoscha Krettek <aljos...@apache.org> >>>> wrote: >>>> >>>>> Hi, >>>>> as far as I know there is no one working on this. I'm only aware of >>>>> someone working on an ORC (from Hive) Writer. >>>>> >>>>> This would be a welcome addition! I think you are already on the right >>>>> track, the only thing required will probably be an AvroFileWriter and you >>>>> already started looking at SequenceFileWriter, which should be similar. >>>>> >>>>> Cheers, >>>>> Aljoscha >>>>> >>>>> On Thu, 21 Apr 2016 at 09:45 Igor Berman <igor.ber...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> Is there such implementation somewhere?(before I start to implement >>>>>> it myself, it seems not too difficult based on SequenceFileWriter >>>>>> example) >>>>>> >>>>>> anyway any ideas/pointers will be highly appreciated >>>>>> >>>>>> thanks in advance >>>>>> >>>>>> >>>> >>
