Abhinav Dixit created KAFKA-18018:
-------------------------------------

             Summary: Consumption failing with DefaultStatePersister for 1 
million records
                 Key: KAFKA-18018
                 URL: https://issues.apache.org/jira/browse/KAFKA-18018
             Project: Kafka
          Issue Type: Sub-task
            Reporter: Abhinav Dixit


I was running some performance tests and I couldn't consume 1,000,000 records 
using 10 share consumers. I've narrowed down the issue to 
{{DefaultStatePersister}} . When I run the same tests using 
{{{}NoOpShareStatePersister{}}}, then I don't see the reliability issue.

Steps to reproduce (could be maybe reproduced by other ways as well, but this 
is how I noticed it consistently) - # Create a topic with a single partition
 # Produce 1000 records into the topic and consume it with the help of 10 share 
consumers.
 # Again produce 1000 records into the topic and consume it with the help of 10 
share consumers.
 # Produce 1,000,000 records into the topic and consume it with the help of 10 
share consumers.

You'll see the records not getting consumed even within 5 minutes (It shouldn't 
take more than 5-6 seconds to consume 1,000,000 records with 10 share 
consumers, based on my past experience)

PS - I haven't noticed any issues using console share consumers, so probably an 
issue with scale



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to