StreamingFileSink.forBulkFormat() for Avro is not working with withRollingPolicyllingfile

2020-12-28 Thread Mahendra Hegde
Hi, I am trying to upload avro records to AWS S3 using StreamingFileSink. Avro file is getting created and uploaded with valid data but I want to add a Rolling policy which will roll the file after specific time or total part file size. With forBulkFormat() I am able to use only CheckpointRolli

StreamingFileSink.forBulkFormat() with CheckpointRollingPolicy issues

2020-12-29 Thread Mahendra Hegde
Hello, I am trying to use StreamingFileSink.forBulkFormat() for writing avro to S3. I have used ‘CheckpointRollingPolicy’ as DefaultRolling cannot be used with bulk formats. But when I use this I am facing 2 issues : 1. ‘shouldRollOnEvent’ method is getting called on each record addition but