Abhinav Dixit created KAFKA-18018: ------------------------------------- Summary: Consumption failing with DefaultStatePersister for 1 million records Key: KAFKA-18018 URL: https://issues.apache.org/jira/browse/KAFKA-18018 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit
I was running some performance tests and I couldn't consume 1,000,000 records using 10 share consumers. I've narrowed down the issue to {{DefaultStatePersister}} . When I run the same tests using {{{}NoOpShareStatePersister{}}}, then I don't see the reliability issue. Steps to reproduce (could be maybe reproduced by other ways as well, but this is how I noticed it consistently) - # Create a topic with a single partition # Produce 1000 records into the topic and consume it with the help of 10 share consumers. # Again produce 1000 records into the topic and consume it with the help of 10 share consumers. # Produce 1,000,000 records into the topic and consume it with the help of 10 share consumers. You'll see the records not getting consumed even within 5 minutes (It shouldn't take more than 5-6 seconds to consume 1,000,000 records with 10 share consumers, based on my past experience) PS - I haven't noticed any issues using console share consumers, so probably an issue with scale -- This message was sent by Atlassian Jira (v8.20.10#820010)