Re: Flink pipeline throughput

Hello Karthik, You may check the execution-buffer-timeout-interval parameter. This value is an important one for your case. I had a similar issue experienced in the past. https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/config/#execution-buffer-timeout-interval For your

退订

退订

Re:Re: Optimize exact deduplication for tens of billions data per day

Hi, Wang. What about just increasing the parallemism to reduce the number of keys processed per parallelism? Is the distribution of keys uneven? If so, you can use the datastream api to manually implement some optimization methods of flink sql.[1] [1] https://nightlies.apache.org/flink/flin

Re:Flink pipeline throughput

Hi, Kartik. On flink ui, is there any operator that maintains a relatively high busy? Could you also try using a flame graph to provide more information?[1] [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/flame_graphs/ -- Best！ Xuyang At 2024-03-30 1