Synchronizing streams in coprocessfunction

2022-06-26 Thread Gopi Krishna M
Hi, I've a scenario where I use connected streams where one is a low throughput metadata stream and another one is a high throughput data stream. I use CoProcessFunction that operates on a data stream with behavior controlled by a metadata stream. Is there a way to slow down/pause the high through

Re: [ANNOUNCE] Apache Flink 1.14.5 released

2022-06-26 Thread Qingsheng Ren
Thanks Xingbo for driving this release! Best, Qingsheng > On Jun 22, 2022, at 11:50, Xingbo Huang wrote: > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.14.5, which is the fourth bugfix release for the Apache Flink 1.14 > series. > > Apache FlinkĀ® is

Re: Flink k8s Operator on AWS?

2022-06-26 Thread Yang Wang
Could you please share the JobManager logs of failed deployment? It will also help a lot if you could show the pending pod status via "kubectl describe ". Given that the current Flink Kubernetes Operator is built on top of native K8s integration[1], the Flink ResourceManager should allocate enough

Re: How to convert Table containing TIMESTAMP_LTZ into DataStream in PyFlink 1.15.0?

2022-06-26 Thread Dian Fu
Hi John, This seems like a bug and I have created a ticket https://issues.apache.org/jira/browse/FLINK-28253 to track it. For now, you could try replacing to_data_stream with to_append_stream` to see if it works. Regards, Dian On Sat, Jun 25, 2022 at 4:07 AM John Tipper wrote: > Hi, > > I hav

Re: How to use connectors in PyFlink 1.15.0 when not defined in Python API?

2022-06-26 Thread Dian Fu
Hi John, Kinesis and most of the other connectors will be supported in 1.16, see [1] for more details about kinesis. For versions prior to 1.16, you could try just as Andrew suggested or refer to the implementations which are already available in the master as examples. Regards, Dian [1] https:

Re: How to make current application cdc

2022-06-26 Thread yuxia
> I mean CDC should be handled on the Kafka side. What do you mean about that? Do you mean the the Kafka should store the message with the cdc format like debezium[1], Canal[2], MaxWell[3], OGG[4]? > Or should I need to use Table API I'm afraid not. Seems you can still use Flink Datastream AP

Re: How to make current application cdc

2022-06-26 Thread Sid
Hi team, Any help here please? Thanks, Sid On Sat, Jun 25, 2022 at 4:02 PM Sid wrote: > Hello, > > I have a current flow where the data from the Flink-Kafka connector is > captured and processed using Flink Datastream API and stored in Kafka > topics. However, I would like to make it CDC enabl