Re: Urgent: Mitigating Slow Consumer Impact and Seeking Open-SourceSolutions in Apache Kafka Consumers

Hi Karthick, We’ve experienced the similar issue before. What we were doing at that time was to define multiple topics and each has a different # of partitions which means some of the topics with more partitions will have the high parallelisms for processing. And you can further divide the topic

Re: OOM errors cause by the new KafkaSink API

ation > to KafkaConsumer and KafkaProducer to validate if then the OOM appears. > > Best regards, > > Martijn > > On Tue, 10 May 2022 at 04:29, Hua Wei Chen > wrote: > >> Hi Martijn, >> >> Thanks for your response. >> >> > What&#

Re: OOM errors cause by the new KafkaSink API

regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser > > > On Mon, 25 Apr 2022 at 17:11, Hua Wei Chen > wrote: > >> Hi Huweihua, >> >> Thanks for the reply. Yes, we increased memory first. >> But we

Re: OOM errors cause by the new KafkaSink API

rsistent OOM, you can dump the memory and check which part > is taking up memory. > > > 2022年4月25日 上午11:44，Hua Wei Chen 写道： > > Hi all, > > Due to FlinkKafkaConsumer and FlinkKafkaProducer will be depreciated at > Flink 1.15*[1]*, we are trying to migrate the APIs to K

OOM errors cause by the new KafkaSink API

Hi all, Due to FlinkKafkaConsumer and FlinkKafkaProducer will be depreciated at Flink 1.15*[1]*, we are trying to migrate the APIs to KafkaSource and KafkaSink*[2]*. At the same time, we also modified the serilizers*[3]*. Our Kafka settings are not changed*[4]*. The services are very stable befor

How to accelerate state processor with a large savepoint

Hi team, We want to try to use state processor APIs[1] to clean up some legacy states. Here are our steps: 1. Create a new savepoint (~= 1.5TB) 2. Submit state processor jobs 3. Write results to a new savepoint We create 8 task managers with 120 slots to execute it. Here are the related configur

Re: Not cleanup Kubernetes Configmaps after execution success

like KubernetesApplicationClusterEntrypoint is re-started in >> the middle of shutdown and, as a result, the resources it (re)creates >> aren't clean up. >> >> Could you please also share Kubernetes logs and resource definitions >> to validate the above assumption? &

Not cleanup Kubernetes Configmaps after execution success

Hi all, We have Flink jobs run on batch mode and get the job status via JobHandler. onJobExecuted ()[