Re: sink with BucketingSink to S3 files override

2018-02-20 Thread Aljoscha Krettek
Hi, I'm afraid the BucketingSink does not work well with S3 because of the eventually-consistent nature of S3. As you noticed in the code snipped you sent, the sink relies on the fact that directory listings are accurate, which is not the case with S3. The Flink community is aware of this prob

sink with BucketingSink to S3 files override

2018-02-19 Thread galantaa
Hey, I have some kind of a concurrency problem with Bucketing sink when I write to S3. I use the AvroKeyValueSinkWriter. The problem is that when I send events the suppose to be written to the same directory, but to a different part file (because of different event types), the files override each o