Re: Slack Invite

2024-05-30 Thread gongzhongqiang
Hi, The invite link : https://join.slack.com/t/apache-flink/shared_invite/zt-2jtsd06wy-31q_aELVkdc4dHsx0GMhOQ Best, Zhongqiang Gong Nelson de Menezes Neto 于2024年5月30日周四 15:01写道: > Hey guys! > > I want to join the slack community but the invite has expired.. > Can u send me a new one? >

Re: "Self-service ingestion pipelines with evolving schema via Flink and Iceberg" presentation recording from Flink Forward Seattle 2023

2024-05-26 Thread gongzhongqiang
Flink CDC 3.0 focuses on data integration scenarios, so you don't need to pay attention to the framework implementation, you just need to use the YAML format to describe the data source and target to quickly build a data synchronization task with schema evolution.And it supports rich source and sin

Re: What is the best way to aggregate data over a long window

2024-05-20 Thread gongzhongqiang
event time and > also clean the state. > > This way each state would always keep only one record, no matter for what > period we aggregate data for. > > Is this a better approach than windowing ? > > > Thanks > S

Re: [ANNOUNCE] Apache Flink CDC 3.1.0 released

2024-05-17 Thread gongzhongqiang
Congratulations ! Thanks for all contributors. Best, Zhongqiang Gong Qingsheng Ren 于 2024年5月17日周五 17:33写道: > The Apache Flink community is very happy to announce the release of > Apache Flink CDC 3.1.0. > > Apache Flink CDC is a distributed data integration tool for real time > data and batch

Re: What is the best way to aggregate data over a long window

2024-05-17 Thread gongzhongqiang
Hi Sachin, We can optimize this problem in the following ways: - use org.apache.flink.streaming.api.datastream.WindowedStream#aggregate(org.apache.flink.api.common.functions.AggregateFunction) to reduce number of data - use TTL to clean data which are not need - enble incremental checkpoint - us

Re: SSL Kafka PyFlink

2024-05-16 Thread gongzhongqiang
Hi Phil, The kafka configuration keys of ssl maybe not correct. You can refer the kafka document[1] to get the ssl configurations of client. [1] https://kafka.apache.org/documentation/#security_configclients Best, Zhongqiang Gong Phil Stavridis 于2024年5月17日周五 01:44写道: > Hi, > > I have a PyFl

Re: Checkpointing while loading causing issues

2024-05-14 Thread gongzhongqiang
Hi Lars, Currently, there is no configuration available to trigger a checkpoint immediately after the job starts in Flink. But we can address this issue from multiple perspectives using the insights provided in this document [1]. [1] https://nightlies.apache.org/flink/flink-docs-release-1.19/

Re: CSV format and hdfs

2024-04-28 Thread gongzhongqiang
Hi Artem, I research on this and open a issue[1] , Rob Young , Alexander Fedulov and I discuss on this. We also think this performance issue can be solved by manual flush. I had opened a pr[2]. You can cherry pick and package on your local, replace the jar in lib folder. I'm willing to hear from

Re: FlinkCEP

2024-04-23 Thread gongzhongqiang
Hi, After flink 1.5 , there have been no major changes to the CEP API. Best, Zhongqiang Gong Esa Heikkinen 于2024年4月23日周二 04:19写道: > Hi > > It's been over 5 years since I last did anything with FlinkCEP and Flink. > > Has there been any significant development in FlinkCEP during this time? > >

Re: One query just for curiosity

2024-03-28 Thread gongzhongqiang
Hi Ganesh, As Zhanghao Chen told before, He advise you two solutions for different scenarios. 1.Process record is a CPU-bound task: scale up parallelism of task and flink cluster to improve tps. 2.Process record is a IO-bound task: use Async-IO to reduce cost of resource and alse get better per

Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

2024-03-28 Thread gongzhongqiang
Congratulations! Best, Zhongqiang Gong Yu Li 于2024年3月28日周四 15:57写道: > CC the Flink user and dev mailing list. > > Paimon originated within the Flink community, initially known as Flink > Table Store, and all our incubating mentors are members of the Flink > Project Management Committee. I am c

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-21 Thread gongzhongqiang
Congrattulations! Thanks for the great work! Best, Zhongqiang Gong Leonard Xu 于2024年3月20日周三 21:36写道: > Hi devs and users, > > We are thrilled to announce that the donation of Flink CDC as a > sub-project of Apache Flink has completed. We invite you to explore the new > resources available: > >

Re: [ANNOUNCE] Apache Flink 1.19.0 released

2024-03-18 Thread gongzhongqiang
Congrats! Thanks to everyone involved! Best, Zhongqiang Gong Lincoln Lee 于2024年3月18日周一 16:27写道: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. > > Apache Flink® is an open-source stream pr