Re: StreamingFileSink causing AmazonS3Exception

2018-12-17 Thread Addison Higham
Issue opened here: https://issues.apache.org/jira/browse/FLINK-11187 On Mon, Dec 17, 2018 at 2:37 PM Addison Higham wrote: > Oh this is timely! > > I hope I can save you some pain Kostas! (cc-ing to flink dev to get > feedback there for what I believe to be a confirmed bug) >

[jira] [Created] (FLINK-11187) StreamingFileSink with S3 backend transient socket timeout issues

2018-12-17 Thread Addison Higham (JIRA)
Addison Higham created FLINK-11187: -- Summary: StreamingFileSink with S3 backend transient socket timeout issues Key: FLINK-11187 URL: https://issues.apache.org/jira/browse/FLINK-11187 Project

Re: StreamingFileSink causing AmazonS3Exception

2018-12-17 Thread Addison Higham
Oh this is timely! I hope I can save you some pain Kostas! (cc-ing to flink dev to get feedback there for what I believe to be a confirmed bug) I was just about to open up a flink issue for this after digging (really) deep and figuring out the issue over the weekend. The problem arises due the

Re: support/docs for compression in StreamingFileSink

2018-11-14 Thread Addison Higham
Just noticed one detail about using the BulkWriter interface, you no longer can assign a rolling policy. That makes sense for formats like orc/parquet, but perhaps not for simple text compression. On Wed, Nov 14, 2018 at 1:43 PM Addison Higham wrote: > HI all, > > I am moving some co

support/docs for compression in StreamingFileSink

2018-11-14 Thread Addison Higham
HI all, I am moving some code to use the StreamingFileSink. Currently, it doesn't look like there is any native support for compression (gzip or otherwise) built into flink when using the StreamingFileSink. It seems like this is a really common need that as far as I could tell, wasn't represented

StreamingFileSink Bug? Committing results on stream close

2018-11-07 Thread Addison Higham
Hi all, Just run into a bit of a problem and I am not sure what the behavior should be and if this should be considered a bug? Or if there is some other way this should be handled? I have a streaming job with a stream that eventually closes, this job sinks to a StreamingFileSink. The problem I am

Re: FLINK-9752 (s3 recoverable writer) not actually fixed in 1.6.2?

2018-10-30 Thread Addison Higham
> who worked on the S3 writer. @Klou did we intentionally not backport this > > feature? > > > > I think there should be nothing special about backporting this feature > and > > building your own version of Flink. > > > > Cheers, > > Till > > &

FLINK-9752 (s3 recoverable writer) not actually fixed in 1.6.2?

2018-10-30 Thread Addison Higham
Hi all, Been hitting my head against a wall for the last few hours. The release notes for 1.6.2 show https://issues.apache.org/jira/browse/FLINK-9752 as resolved in 1.6.2. I am trying to upgrade and switch some things to use the StreamingFileSink against s3. However, when doing so, I get the follo

[jira] [Created] (FLINK-9616) DatadogHttpReporter fails to be created due to missing shaded dependency

2018-06-19 Thread Addison Higham (JIRA)
Addison Higham created FLINK-9616: - Summary: DatadogHttpReporter fails to be created due to missing shaded dependency Key: FLINK-9616 URL: https://issues.apache.org/jira/browse/FLINK-9616 Project

Improvements to Mesos Deployments Using Docker

2018-06-18 Thread Addison Higham
I am currently in the process of getting flink 1.5 running in a mesos cluster using docker. I have come across a few improvements that I think could be helpful with this configuration (and will probably also apply for any future containerized deployments, like Kubernetes) I have already created t

[jira] [Created] (FLINK-9612) Add option for minimal artifacts being pulled in meseos

2018-06-18 Thread Addison Higham (JIRA)
Addison Higham created FLINK-9612: - Summary: Add option for minimal artifacts being pulled in meseos Key: FLINK-9612 URL: https://issues.apache.org/jira/browse/FLINK-9612 Project: Flink

[jira] [Created] (FLINK-9611) Allow for user-defined artifacts to be specified as part of a mesos overlay

2018-06-18 Thread Addison Higham (JIRA)
Addison Higham created FLINK-9611: - Summary: Allow for user-defined artifacts to be specified as part of a mesos overlay Key: FLINK-9611 URL: https://issues.apache.org/jira/browse/FLINK-9611 Project

[jira] [Created] (FLINK-7615) Under mesos when using a role, TaskManagers fail to schedule

2017-09-13 Thread Addison Higham (JIRA)
Addison Higham created FLINK-7615: - Summary: Under mesos when using a role, TaskManagers fail to schedule Key: FLINK-7615 URL: https://issues.apache.org/jira/browse/FLINK-7615 Project: Flink

[jira] [Created] (FLINK-5975) Mesos should support adding volumes to launched taskManagers

2017-03-06 Thread Addison Higham (JIRA)
Addison Higham created FLINK-5975: - Summary: Mesos should support adding volumes to launched taskManagers Key: FLINK-5975 URL: https://issues.apache.org/jira/browse/FLINK-5975 Project: Flink