Re: Clean shutdown of streaming job

Hi Ning, I don't think it is possible to pause a Kafka source upon taking a savepoint without making any changes to the implementation. I think your problem is that the Cassandra sink doesn't support exactly once guarantees when the Cassandra query isn't idempotent. If possible, the cleanest solu

Re: Manual trigger the window in fold operator or incremental aggregation

we distinguish within apply function of 'RichWindowFunction' whether > it was called due to onElement trigger call or onProcessingtime trigger > call of a custom Trigger ? > > Thanks! > > On Wed, 17 Oct 2018 at 12:51, Niels van Kaam wrote: > >> Hi Zhen Li, >>

Re: Checkpointing when one of the sources has completed

ely > Joshua > > On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam wrote: > >> Hi All, >> >> I am debugging an issue where the periodic checkpointing has halted. I >> noticed that one of the sources of my job has completed (finished). The >> other sources and o

Re: Manual trigger the window in fold operator or incremental aggregation

Hi Zhen Li, You can control when a windowed stream emits data with "Triggers". See: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#triggers Flink comes with a couple of default triggers, but you can also create your own by implementing https://ci.apache.o

Checkpointing when one of the sources has completed

Hi All, I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output. Does anyone know if Flink's periodic checkpoints are supposed to contin

Dealing with an asynchronous source (and sink) in Flink 1.5.0. Await.Result() does not complete.

Hi All, I have implemented a custom sourcefuntion on a datasource with an asynchronous API (the API calls return Scala futures). I need to perform calls to the asynchronous API during initialization of each individual (parallel) source instance, and when in exacly-once mode also during snapshotsta

Re: Akka Http used in custom RichSourceFunction

, because indeed it looks like > your akka’s version is leaking to Flink’s classloader. > > Piotrek > > > On 25 May 2018, at 09:40, Niels van Kaam wrote: > > Hi Piotrek, > > Thank you for your response! > > I am currently just testing the job in a local environ

Re: Akka Http used in custom RichSourceFunction

ou should shade your akka dependency. > > What is the full exception? Is it thrown when YOURS code tries to > shutdown, or when FLINK’s code tries to shutdown? > > Piotrek > > > On 24 May 2018, at 14:38, Niels van Kaam wrote: > > Hi All, > > I wrote a custom sou

Akka Http used in custom RichSourceFunction

Hi All, I wrote a custom source function (RichSourceFunction) which connects to a web socket using the Akka Http Library. The job using this source runs fine on a local environment until upon shutdown I see the following error in the log: "Exception in thread "main" java.lang.NoSuchMethodError: ak

Re: "Close()" aborts last transaction in TwoPhaseCommitSinkFunction

pping it and there might > be some corner cases that I haven’t thought about. > > Piotrek > > > On 9 Mar 2018, at 14:49, Niels van Kaam wrote: > > Hi, > > I'm working on a custom implementation of a sink which I would like to use > with exactly once semantics

"Close()" aborts last transaction in TwoPhaseCommitSinkFunction

Hi, I'm working on a custom implementation of a sink which I would like to use with exactly once semantics. Therefore I have implemented the TwoPhaseCommitSinkFunction class as mentioned in this recent post: https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html I