Re: StreamingFileSink.forBulkFormat() with CheckpointRollingPolicy issues

2021-01-06 Thread Yun Gao
Hi Mahendra, Sorry for the late reply. I noticed that in your code you implement a bucket assigner that reads to switch to a new bucket every minute, does it related to the current problems met ? Since different buckets would use different directories and files, when switching buckets new

StreamingFileSink.forBulkFormat() with CheckpointRollingPolicy issues

2020-12-29 Thread Mahendra Hegde
Hello, I am trying to use StreamingFileSink.forBulkFormat() for writing avro to S3. I have used ‘CheckpointRollingPolicy’ as DefaultRolling cannot be used with bulk formats. But when I use this I am facing 2 issues : 1. ‘shouldRollOnEvent’ method is getting called on each record addition but