Re: Kafka Consumer stop consuming data

[Jerome Li]

Hi Aeden,

Thanks for getting back.

Do you mean one of the partitions is in idle state and not new watermark 
generated from there and then it stunk all the downsteams and stop consuming 
data from Kafka? I didn’t use watermark in my application through.

I checked that all the Kafka partition has data consistently coming in.

Best wish,
Jerome

From: Aeden Jameson <aeden.jame...@gmail.com>
Date: Tuesday, July 13, 2021 at 3:22 PM
To: Jerome Li <l...@vmware.com>
Cc: user@flink.apache.org <user@flink.apache.org>
Subject: Re: Kafka Consumer stop consuming data
This can happen if you have an idle partition. Are all partitions
receiving data consistently?

On Tue, Jul 13, 2021 at 2:59 PM Jerome Li <l...@vmware.com> wrote:
>
> Hi,
>
>
>
> I got question about Flink Kafka consumer. I am facing the issue that the 
> Kafka consumer somehow stop consuming data from Kafka after start for few 
> minutes or after few hours. While stopping, I checked the backpressure and 
> cpu and memory consumption. It all looks like not data consuming instead of 
> stunk at busy computing. And not notable logs in either JM or TM.
>
>
>
> I want to know what scenarios could cause the Kafak consumer stop consuming?
>
>
>
> Any help would be appreciated!
>
>
>
> Best,
>
> Jerome



--
Cheers,
Aeden

GitHub: 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faedenj&amp;data=04%7C01%7Clije%40vmware.com%7Ca688c9c7b3954a8cb5f608d9464cb2ff%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637618117520228221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=3R%2Bqlz19tsts3omsEodLOAmkIr0oRA7nE4zCVOSXluc%3D&amp;reserved=0
Linked In: 
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Faedenjameson&amp;data=04%7C01%7Clije%40vmware.com%7Ca688c9c7b3954a8cb5f608d9464cb2ff%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637618117520228221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=wML9HQDVNvdTApj1pPxzUqKwxMShpcUq7084Oma28aY%3D&amp;reserved=0
Blah Blah Blah: 
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fdaliful&amp;data=04%7C01%7Clije%40vmware.com%7Ca688c9c7b3954a8cb5f608d9464cb2ff%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637618117520228221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=Re8zSJsvmx%2BNtkEmP4hf5gAlzzNpdtnSeJlaEi2LTi8%3D&amp;reserved=0