Re: Adding a part suffix setter to the BucketingSink

2018-07-12 Thread Kostas Kloudas
Hi Lakshmi, I meant Flink 1.5.1 (not 1.5.3) which was recently released. Cheers, Kostas > On Jul 12, 2018, at 7:34 PM, Lakshmi Gururaja Rao wrote: > > Hi Kostas, > > Thank you for replying. I am already using the ability to set part suffix. > I was not aware of this issue - https://issues.apa

Re: Adding a part suffix setter to the BucketingSink

2018-07-12 Thread Lakshmi Gururaja Rao
Hi Kostas, Thank you for replying. I am already using the ability to set part suffix. I was not aware of this issue - https://issues.apache.org/ jira/browse/FLINK-9603. Thanks for pointing out, I'll make sure to use the 1.5.3 version of the sink. Thanks Lakshmi On Thu, Jul 12, 2018 at 4:55 AM,

Re: Adding a part suffix setter to the BucketingSink

2018-07-12 Thread vino yang
Hi Kostas, good job! 2018-07-12 19:40 GMT+08:00 Kostas Kloudas : > Hi Lakshmi, > > Since Flink-1.5 you have the ability to set the part suffix. > As you said, you only want the .gzip to be the suffix of the final (or > “completed”) part files, which is exactly what is currently supported. > > If

Re: Adding a part suffix setter to the BucketingSink

2018-07-12 Thread Kostas Kloudas
Hi Lakshmi, Since Flink-1.5 you have the ability to set the part suffix. As you said, you only want the .gzip to be the suffix of the final (or “completed”) part files, which is exactly what is currently supported. If you want also intermediate files to have this suffix, then you can always set

Re: Adding a part suffix setter to the BucketingSink

2018-04-05 Thread Lakshmi Gururaja Rao
I can see two ways of achieving this: 1. Setting a suffix* **only*** for the completed part files. I don't necessarily think the suffix should be added for the intermediate files (as intermediate files should not really be ready for consumption by a downstream process?) 2. Be able to override this

Re: Adding a part suffix setter to the BucketingSink

2018-04-03 Thread Aljoscha Krettek
So you want to be able to set a "global" suffix that should be appended to all different kinds of files that the sink writes, including intermediate files? Aljoscha > On 29. Mar 2018, at 16:59, l...@lyft.com wrote: > > Sorry, I meant "I don't see a way of doing this apart from setting a part >

Re: Adding a part suffix setter to the BucketingSink

2018-03-29 Thread lrao
Sorry, I meant "I don't see a way of doing this apart from setting a part file *suffix* with the required file extension. " On 2018/03/29 14:55:43, l...@lyft.com wrote: > Currently the BucketingSink allows addition of part prefix, pending > prefix/suffix and in-progress prefix/suffix via sett