Re: StreamingFileSink causing AmazonS3Exception

2019-02-21 Thread Padarn Wilson
Thanks Kostas! On Mon, Feb 18, 2019 at 5:10 PM Kostas Kloudas wrote: > Hi Padarn, > > This is the jira issue: https://issues.apache.org/jira/browse/FLINK-11187 > and the fix, as you can see, was first included in version 1.7.2. > > Cheers, > Kostas > > On Mon, Feb 18, 2019 at 3:49 AM Padarn Wil

Re: StreamingFileSink causing AmazonS3Exception

2019-02-18 Thread Kostas Kloudas
Hi Padarn, This is the jira issue: https://issues.apache.org/jira/browse/FLINK-11187 and the fix, as you can see, was first included in version 1.7.2. Cheers, Kostas On Mon, Feb 18, 2019 at 3:49 AM Padarn Wilson wrote: > Hi Addison, Kostas, Steffan, > > I am also encountering this exact issue

Re: StreamingFileSink causing AmazonS3Exception

2019-02-17 Thread Padarn Wilson
Hi Addison, Kostas, Steffan, I am also encountering this exact issue. I cannot find a JIRA ticket on this, is there some planned work on implementing a fix? @Addison - Did you manage to find a fix that you could apply without modifying the Flink codebase? If possible it would be better not patch

Re: StreamingFileSink causing AmazonS3Exception

2018-12-17 Thread Addison Higham
Oh this is timely! I hope I can save you some pain Kostas! (cc-ing to flink dev to get feedback there for what I believe to be a confirmed bug) I was just about to open up a flink issue for this after digging (really) deep and figuring out the issue over the weekend. The problem arises due the

Re: StreamingFileSink causing AmazonS3Exception

2018-12-14 Thread Kostas Kloudas
Hi Steffen, Thanks for reporting this. Internally Flink does not keep any open connections to S3. It only keeps buffers data internally up till the point they reach a min-size limit (by default 5MB) and then uploads them as a part of an MPU on one go. Given this, I will have to dig a bit dipper