Re: Urgent: Mitigating Slow Consumer Impact and Seeking Open-SourceSolutions in Apache Kafka Consumers

Hi Karthick, We’ve experienced the similar issue before. What we were doing at that time was to define multiple topics and each has a different # of partitions which means some of the topics with more partitions will have the high parallelisms for processing. And you can further divide the topic

Not cleanup Kubernetes Configmaps after execution success

Hi all, We have Flink jobs run on batch mode and get the job status via JobHandler. onJobExecuted ()[

Re: Not cleanup Kubernetes Configmaps after execution success

like KubernetesApplicationClusterEntrypoint is re-started in >> the middle of shutdown and, as a result, the resources it (re)creates >> aren't clean up. >> >> Could you please also share Kubernetes logs and resource definitions >> to validate the above assumption? &

How to accelerate state processor with a large savepoint

Hi team, We want to try to use state processor APIs[1] to clean up some legacy states. Here are our steps: 1. Create a new savepoint (~= 1.5TB) 2. Submit state processor jobs 3. Write results to a new savepoint We create 8 task managers with 120 slots to execute it. Here are the related configur

OOM errors cause by the new KafkaSink API

Hi all, Due to FlinkKafkaConsumer and FlinkKafkaProducer will be depreciated at Flink 1.15*[1]*, we are trying to migrate the APIs to KafkaSource and KafkaSink*[2]*. At the same time, we also modified the serilizers*[3]*. Our Kafka settings are not changed*[4]*. The services are very stable befor

Re: OOM errors cause by the new KafkaSink API

rsistent OOM, you can dump the memory and check which part > is taking up memory. > > > 2022年4月25日 上午11:44，Hua Wei Chen 写道： > > Hi all, > > Due to FlinkKafkaConsumer and FlinkKafkaProducer will be depreciated at > Flink 1.15*[1]*, we are trying to migrate the APIs to K

Re: OOM errors cause by the new KafkaSink API

regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser > > > On Mon, 25 Apr 2022 at 17:11, Hua Wei Chen > wrote: > >> Hi Huweihua, >> >> Thanks for the reply. Yes, we increased memory first. >> But we

Re: OOM errors cause by the new KafkaSink API

ation > to KafkaConsumer and KafkaProducer to validate if then the OOM appears. > > Best regards, > > Martijn > > On Tue, 10 May 2022 at 04:29, Hua Wei Chen > wrote: > >> Hi Martijn, >> >> Thanks for your response. >> >> > What&#