Re: Flink Kafka Consumer Behaviour

Hi Prabhu, cc Stephan, Robert, I was having similar issues where flink Kafka 09 consumer was not committing offsets to kafka. After digging into JobManager logs, I found that checkpoints were getting expired before getting completed and hence "checkpoint completed" message was being ignored. I

Re: Enriching a tuple mapped from a datastream with data coming from a JDBC source

Hi again, I implemented the RichMap Function (open method runs a JDBC query to populate a HashMap with data) which I am using in the map function. Now there is another RichMap.map function that would add to the HashMap that was initialized in the first function. How would I share the Map between th

Re: Exception while running Flink jobs (1.0.0)

I think you're running into the same exception I face sometimes..I've opened a jira for it [1]. Could you please try to apply that patch and see if things get better? https://issues.apache.org/jira/plugins/servlet/mobile#issue/FLINK-4719 Best, Flavio On 3 Oct 2016 22:09, "Tarandeep Singh" wrote

Re: Exception while running Flink jobs (1.0.0)

Now, when I ran it again (with lower task slots per machine) I got a different error- org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed. at org.apache.flink.client.program.Client.runBlocking(Client.java:381) at org.apache.flink.c

Exception while running Flink jobs (1.0.0)

Hi, I am using flink-1.0.0 and running ETL (batch) jobs on it for quite some time (few months) without any problem. Starting this morning, I have been getting errors like these- "Received an event in channel 3 while still having data from a record. This indicates broken serialization logic. If yo

Re: Regarding Late Elements

Not yet. I'm hoping a Flink export on this mailing list will reply. - LF From: vinay patil To: user@flink.apache.org Sent: Monday, October 3, 2016 8:09 AM Subject: Re: Regarding Late Elements Hi LF, So did you manage to get the workaround for it ? I am using a Custom Trigger whi

Re: Regarding Late Elements

Hi LF, So did you manage to get the workaround for it ? I am using a Custom Trigger which is similar to 1.0.3 with few changes Regards, Vinay Patil On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List archive.] wrote: > We have the same requirement - we cannot discard an

Re: Using Flink

Hi Gordon, - I'm currently running my flink program on 1.2 SNAPSHOT with kafka source and I have checkpoint enabled. When I look at the consumer offsets in kafka it appears to be stagnant and there is a huge lag. But I can see my flink program is in pace with kafka source in JMX metrics and output

Re: Regarding Late Elements

We have the same requirement - we cannot discard any data even if it arrives late.  - LF   From: Vinay Patil To: user@flink.apache.org Sent: Sunday, October 2, 2016 8:21 PM Subject: Regarding Late Elements Hi Guys, Just wanted to get an idea on Why Flink decided to completely dis

Re: Flink Checkpoint runs slow for low load stream

Hi Stephan, Is the Async kafka offset commit released in 1.3.1? Varaga On Wed, Sep 28, 2016 at 9:49 AM, Chakravarthy varaga < chakravarth...@gmail.com> wrote: > Hi Stephan, > > That should be great. Let me know once the fix is done and the > snapshot version to use, I'll check and reve

Side Inputs vs. Connected Streams

Hi, I read the Side Inputs design document. How does it compare to using ConnectedStreams with respect to handling the ordering of streams transparently? One of the challenges I have with ConnectedStreams is I

Re: Flink job fails to restore RocksDB state after upgrading to 1.2-SNAPSHOT

Hi Stefan, Sorry for the late reply - I was away last week. I've just got round to retrying my above scenario (run my job, take a savepoint, restore my job) using the latest Flink 1.2-SNAPSHOT -- and am now seeing a different exception when restoring the state: 10/03/2016 11:29:02 Job execution

Re: How to stop job through java API

Now, I am able to create an class of flink she'll front end UI class. Once I have this instance, I can launch cmd in Java. Cmd has features to connect remote resource manager to do most if actions. Thanks, Will > On Oct 3, 2016, at 4:31 AM, "LINZ, Arnaud" wrote: > > Hi, > > I have a similar

RE: How to stop job through java API

Hi, I have a similar issue. Here is how I deal with programmatically stopping permanent streaming jobs, and I’m interested in knowing if there is a better way now. Currently, I use hand-made streaming sources that periodically check for some flag and end if a stop request was made. Stopping th

Re: Using Flink

Hi! - I'm currently running my flink program on 1.2 SNAPSHOT with kafka source and I have checkpoint enabled. When I look at the consumer offsets in kafka it appears to be stagnant and there is a huge lag. But I can see my flink program is in pace with kafka source in JMX metrics and outputs. I