Hi Rafi,

At the moment I do not see any support of Parquet in DataSet API except 
HadoopOutputFormat, mentioned in stack overflow question. I have cc’ed Fabian 
and Aljoscha, maybe they could provide more information.

Best,
Andrey

> On 25 Oct 2018, at 13:08, Rafi Aroch <rafi.ar...@gmail.com> wrote:
> 
> Hi,
> 
> I'm writing a Batch job which reads Parquet, does some aggregations and 
> writes back as Parquet files.
> I would like the output to be partitioned by year, month, day by event time. 
> Similarly to the functionality of the BucketingSink.
> 
> I was able to achieve the reading/writing to/from Parquet by using the 
> hadoop-compatibility features.
> I couldn't find a way to partition the data by year, month, day to create a 
> folder hierarchy accordingly. Everything is written to a single directory.
> 
> I could find an unanswered question about this issue: 
> https://stackoverflow.com/questions/52204034/apache-flink-does-dataset-api-support-writing-output-to-individual-file-partit
>  
> <https://stackoverflow.com/questions/52204034/apache-flink-does-dataset-api-support-writing-output-to-individual-file-partit>
> 
> Can anyone suggest a way to achieve this? Maybe there's a way to integrate 
> the BucketingSink with the DataSet API? Another solution?
> 
> Rafi

Reply via email to