Sorry to revive this thread, I'm just circling back to this now. Is it possible to use https://issues.apache.org/jira/browse/FLINK-24565 with the DataStream API? I am not sure how to make use of AvroFileFormatFactory in the DataStream APi context, and couldn't find any examples.
On Mon, Jan 10, 2022 at 4:19 PM Kevin Lam <kevin....@shopify.com> wrote: > Hi David, > > Awesome, wasn't aware of FLINK-24565. That's the kind of thing we were > looking for and will take a look at it. Thanks for sharing that! > > > > On Fri, Jan 7, 2022 at 2:05 PM David Morávek <david.mora...@gmail.com> > wrote: > >> Hi Kevin, >> >> I'm not as familiar with initiatives around the new sources, but it seems >> that the BulkFormat for Avro [1] has been added recently and will be >> released with the Flink 1.15.x. >> >> [1] https://issues.apache.org/jira/browse/FLINK-24565 >> >> Best, >> D. >> >> On Fri, Jan 7, 2022 at 7:23 PM Kevin Lam <kevin....@shopify.com> wrote: >> >>> Hi all, >>> >>> We're looking into using the new FileSource >>> <https://nightlies.apache.org/flink/flink-docs-release-1.14/api/java/org/apache/flink/connector/file/src/FileSource.html> >>> API, we see that there is a BulkFormat >>> <https://nightlies.apache.org/flink/flink-docs-release-1.14/api/java/org/apache/flink/connector/file/src/reader/BulkFormat.html> >>> for Parquet, via ParquetColumnarRowFormat >>> <https://nightlies.apache.org/flink/flink-docs-release-1.14/api/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormat.html>. >>> Is there similar BulkFormat available or in the works for Avro files? >>> >>> I imagined it may be a common use-case in the community so wanted to >>> check here before we invest time implementing our own. >>> >>> Thanks in advance! >>> >>
