Hi David, Awesome, wasn't aware of FLINK-24565. That's the kind of thing we were looking for and will take a look at it. Thanks for sharing that!
On Fri, Jan 7, 2022 at 2:05 PM David Morávek <david.mora...@gmail.com> wrote: > Hi Kevin, > > I'm not as familiar with initiatives around the new sources, but it seems > that the BulkFormat for Avro [1] has been added recently and will be > released with the Flink 1.15.x. > > [1] https://issues.apache.org/jira/browse/FLINK-24565 > > Best, > D. > > On Fri, Jan 7, 2022 at 7:23 PM Kevin Lam <kevin....@shopify.com> wrote: > >> Hi all, >> >> We're looking into using the new FileSource >> <https://nightlies.apache.org/flink/flink-docs-release-1.14/api/java/org/apache/flink/connector/file/src/FileSource.html> >> API, we see that there is a BulkFormat >> <https://nightlies.apache.org/flink/flink-docs-release-1.14/api/java/org/apache/flink/connector/file/src/reader/BulkFormat.html> >> for Parquet, via ParquetColumnarRowFormat >> <https://nightlies.apache.org/flink/flink-docs-release-1.14/api/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormat.html>. >> Is there similar BulkFormat available or in the works for Avro files? >> >> I imagined it may be a common use-case in the community so wanted to >> check here before we invest time implementing our own. >> >> Thanks in advance! >> >
