the DataStream API should fully subsume the DataSet API (through bounded streams) in the long run [1] And you can consider use Table/SQL API in your project.
[1] https://flink.apache.org/roadmap.html#analytics-applications-and-the-roles-of-datastream-dataset-and-table-api *Best Regards,* *Zhenghua Gao* On Fri, Aug 16, 2019 at 11:52 PM Lian Jiang <jiangok2...@gmail.com> wrote: > Thanks. Which api (dataset or datastream) is recommended for file handling > (no window operation required)? > > We have similar scenario for real-time processing. May it make sense to > use datastream api for both batch and real-time for uniformity? > > Sent from my iPhone > > On Aug 16, 2019, at 00:38, Zhenghua Gao <doc...@gmail.com> wrote: > > Flink allows hadoop (mapreduce) OutputFormats in Flink jobs[1]. You can > have a try with Parquet OutputFormat[2]. > > And if you can turn to DataStream APIļ¼ > StreamingFileSink + ParquetBulkWriter meets your requirement[3][4]. > > [1] > https://github.com/apache/flink/blob/master/flink-connectors/flink-hadoop-compatibility/src/test/java/org/apache/flink/test/hadoopcompatibility/mapreduce/example/WordCount.java > [2] > https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetOutputFormat.java > [3] > https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java > [4] > https://github.com/apache/flink/blob/master/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetBulkWriter.java > > > *Best Regards,* > *Zhenghua Gao* > > > On Fri, Aug 16, 2019 at 1:04 PM Lian Jiang <jiangok2...@gmail.com> wrote: > >> Hi, >> >> I am using Flink 1.8.1 DataSet for a batch processing. The data source is >> avro files and I want to output the result into parquet. >> https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/batch/ >> only has no related information. What's the recommended way for doing this? >> Do I need to write adapters? Appreciate your help! >> >> >>
