Re: Using Contextual Data

2017-04-16 Thread Tzu-Li (Gordon) Tai
Hi, It seems like you want to load the context tables from external databases when the CEP operators start, and allow pattern detection logic to refer to those tables. I’m not entirely sure if this possible with the Flink CEP library. I’ve looped in Klou who is more familiar with the CEP library

Re: Kafka offset commits

2017-04-16 Thread Tzu-Li (Gordon) Tai
Hi, The FlinkKafkaConsumer in 1.2 is able to restore from older version state snapshots and bridge the migration, so there should be no problem in reading the offsets from older state. The smallest or highest offsets will only be used if the offset no longer exists due to Kafka data retention s

Re: Flink errors out and job fails--IOException from CollectSink.open()

2017-04-16 Thread Tzu-Li (Gordon) Tai
Hi, The NPE should definitely be fixed. Here’s the filed JIRA for it:  https://issues.apache.org/jira/browse/FLINK-6311. @Sathi: When building Flink we suggest to avoid 3.3.x+ as you have experienced yourself, it is subject to incorrect shading of some of the dependencies. Cheers, Gordon On 1

Re: Data duplication on a High Availability activated cluster after a Task Manager failure recovery

2017-04-16 Thread Tzu-Li (Gordon) Tai
Hi, A few things to clarify first: 1. What is the sink you are using? Checkpointing in Flink allows for exactly-once state updates. Whether or not end-to-end exactly-once delivery can be achieved depends on the sink. For data store sinks such as Cassandra / Elasticsearch, this can be made effe

Data duplication on a High Availability activated cluster after a Task Manager failure recovery

2017-04-16 Thread F.Amara
Hi all, I'm using Flink 1.2.0. I have a distributed system where Flink High Availability feature is activated. Data is produced using a Kafka broker and on a TM failure scenario, the cluster restarts. Checkpointing is enabled with exactly once processing. Problem encountered is, at the end of data

Re: Flink errors out and job fails--IOException from CollectSink.open()

2017-04-16 Thread Ted Yu
Can you describe the transitive dependency issue in more detail (since a lot of people use 3.3.x) ? Thanks > On Apr 16, 2017, at 1:56 AM, Sathi Chowdhury > wrote: > > Ok finally able to run the job fine ...the culprit was an older version of > aws inside flink kinesis connector jar , also re

Re: Flink errors out and job fails--IOException from CollectSink.open()

2017-04-16 Thread Sathi Chowdhury
Ok finally able to run the job fine ...the culprit was an older version of aws inside flink kinesis connector jar , also rebuilt flink again from source with maven 3.25 ,I was on 3.3* and it had weird transitive dependency problems. On Apr 15, 2017, at 11:13 AM, Sathi Chowdhury mailto:sathi.