Re: StreamingFileSink Avro batch size and compression

2019-01-26 Thread Taher Koitawala
Thanks, I'll check it out. On Sat 26 Jan, 2019, 10:48 PM Dawid Wysakowicz Hi, > > Generally speaking you can pass the batch size through RollingPolicy[1]. > Unfortunately BulkFormats uses OnCheckpointRollingPolicy and AFAIK it does > not allow adjusting its behavior on part size. Maybe Kostas hav

Re: StreamingFileSink Avro batch size and compression

2019-01-26 Thread Dawid Wysakowicz
Hi, Generally speaking you can pass the batch size through RollingPolicy[1]. Unfortunately BulkFormats uses OnCheckpointRollingPolicy and AFAIK it does not allow adjusting its behavior on part size. Maybe Kostas have an idea how to do that in the least invasive way. How to do it for non bulk forma

Re: StreamingFileSink Avro batch size and compression

2019-01-25 Thread Taher Koitawala
Can someone please help with this? On Fri 25 Jan, 2019, 1:47 PM Taher Koitawala Hi All, > Is there a way to specify *batch size* and *compression *properties > when using StreamingFileSink just like we did in bucketing sink? The only > parameters it is accepting is Inactivity bucket chec

StreamingFileSink Avro batch size and compression

2019-01-25 Thread Taher Koitawala
Hi All, Is there a way to specify *batch size* and *compression *properties when using StreamingFileSink just like we did in bucketing sink? The only parameters it is accepting is Inactivity bucket check interval and avro schema. We have numerous flink jobs pulling data from the