Re: FlinkKafkaConsumer -> KafkaSource State Migration

Hi, Sorry for the late reply but most of use were involved in the Flink Forward conference. The upgrade strategies for the Kafka sink and source are pretty similar. Source and sink do not rely on state migration but leveraging Kafka as source of truth. When running with FlinkKafkaConsumer Maso

database as stream source issue

Hi, I am trying to use postgres DB as the stream data source and push to kafka topic. Here is how I config database source. Looks like it didn't read out any data. But I didn't see any error from the flink log. I did a test, tried to insert wrong data to database, I saw flink throw below error. Lo

Re: Circular chaining of exception causes when ExceptionInChainedOperatorException is thrown

Hi! Thanks for reporting this issue. This flushException field is to record the exception from the timer thread and will fail the job once the main thread discovers its existence. I've created a jira ticket [1] about this issue and you can check the progress there. [1] https://issues.apache.org/j

Re: Flink SQL does not honor "table.exec.source.idle-timeout"

Hi! What's the value of your config.autowatermarkInterval()? It must be larger than 0 for table.exec.source.idle-timeout to work. More specifically, auto watermark is to avoid sending watermark for each record (which reduces the performance) but instead sends watermark once in each auto watermark

Possibility of supporting Reactive mode for native Kubernetes application mode

Hello Community, I am checking the reactive mode for Flink deployment. I noticed that this is supported in Kubernetes environment, but only for standalone Kubernetes as of now. I have read some previous discussion threads regarding this issue. See [1][2][3][4][5][6]. Question 1: It seems that

Flink SQL does not honor "table.exec.source.idle-timeout"

Hi all, I have a local Flink SQL app with Kafka source running with 3 partitions (0,1,2). I am running the following code: final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); final EnvironmentSettings settings = EnvironmentSettings.newInstance().u

Circular chaining of exception causes when ExceptionInChainedOperatorException is thrown

I am using flink-connector-jdbc 1.12.2 When the flink fails to write records to the database, as a result of some connection failure, then the exception thrown contains an almost 5000 of the following circular causes: Caused by: java.lang.RuntimeException: Writing records to JDBC failed.

Re: Checkpoint failures without exceptions

Hi Patrick, Could you also have a look at the stack of the tasks of the second function to see what the main thread and netty thread is doing during the checkpoint period ? Best, Yun --Original Mail -- Sender: Send Date:Wed Oct 27 22:05:40 2021 Recipients:Fli

Checkpoint failures without exceptions

Hi Flink Community, I have an issue with failing checkpoints on all stateful jobs in a session cluster which I’m unable to track down so far. The jobs sit between Kafka. Only the first checkpoint gets completed all others fail. The watermarks are progressing regularly and are aligned between sub

Re: RE: Duplicate Calls to Cep Filter

Hi Puneet, Sorry I'm not be an expert for CEP, but the underlying implementation of the CEP should be based on the NFA, and from the API documentation, `followedBy` does not require the two patterns are adjacent (namely the give pattern also accepts ['a', 'c', 'b']. Thus when recieved 'a', I th

Re: Flink handle both kafka source and db source

Thank you for your reply! I will check Hybrid source. How do we know the database table is fully scanned? And after the scan is completed, does flink scan the table again or keep idling? Thanks, Qihua On Tue, Oct 26, 2021 at 1:59 PM Rafi Aroch wrote: > Hi, > > Take a look at the new 1.14 featur

Re: Statefun embedded functions - parallel per partition, sequential per key

Thanks for your reply Igal The reason why I'm using data stream integration is that the messages on kafka are in .json, and I need to convert them to protobufs for embedded functions. If I was using remote functions I wouldn't need to do that. With regards to performance, in order to exclude the

Re: Re: Flink support for Kafka versions

Hi Prasanna, we cannot backport these changes as they change dependency coordinates. What I'd recommend you to do is to use the Flink 2_12 dependencies and then bump Kafka client to 2.8.X in your application jar. That should work in any case. Best, Arvid On Wed, Oct 27, 2021 at 12:05 PM Yun Ga

Re: Re: Flink support for Kafka versions

Hi Prasanna, I think the two issues would not be picked back to 1.12.x since they are all large modification and user-visiable and thus they should be not suitable to be picked back to the bugfix version. Which scala version are you going to use? Flink currently provided both version with scala

RE: FlinkKafkaConsumer -> KafkaSource State Migration

I would also be interested on instructions/discussion on how to state-migrate from pre-unified sources/sinks to unified ones (Kafka) 😊 Thias From: Mason Chen Sent: Mittwoch, 27. Oktober 2021 01:52 To: user Subject: FlinkKafkaConsumer -> KafkaSource State Migration Hi all, I read these instru

RE: Duplicate Calls to Cep Filter

Hi Puneet, … not able to answer your question, but I would be curious to also print out the value with your diagnostic message. … assuming we’ll see an ‘a’ and a ‘b’ for both filters resp. … simple explanation would be that the filters are applied to all input, regardless of the pattern matchi

Duplicate Calls to Cep Filter

Hi, I am facing an issue where a single event is causing execution of a cep filter multiple times. I went through this video explaining automata formation from pattern sequence but it still does not explain the behaviour that I am facing. Following is

Re: Getting Errors in Standby Jobmanager pod during installation & after restart on k8s

Roman's answer is on the point. The exception is really confusing and it comes from fabric8 kubernetes-client. We might try to create a PR for the upstream project :) Best, Yang Roman Khachatryan 于2021年10月25日周一 下午10:00写道： > Hi Amit, > > AFAIK, these exceptions are normal in HA mode as differen

Re: Not cleanup Kubernetes Configmaps after execution success

Hi, I think Roman is right. It seems that the JobManager is relaunched again by K8s after Flink has already deregister the application(aka delete the JobManager K8s deployment). One possible reason might be that kubelet is too late to know the JobManager deployment is deleted. So it relaunch the

Re: Statefun embedded functions - parallel per partition, sequential per key

Hello Fil, Indeed what you are describing is exactly what a remote function does. I am curious to learn more about the current performance limitations that you encounter with the remote functions. One thing to try in combination with the async transport, is to increase the total number of in fig