I realise that's a silly question, you must be if you're using auto commit.
When a consumer starts, it needs to do a few things. 1) Connect to a bootstrap server 2) Join an existing consumer group, or create a new one, if it doesn't exist. This may cause a stop the world rebalance as partitions are reassigned within the group. 3) Acquire metadata - which brokers are the partition leaders for my assigned partitions on? And what offsets am I consuming from? 4) Establish the long lived connections to those brokers. 5) Send fetch requests (I might not have the order correct) So yeah, this is why you're seeing that initial delay before consuming records. Kind regards, Liam Clarke-Hutchinson On Mon, 13 Dec 2021, 7:19 pm Liam Clarke-Hutchinson, <lclar...@redhat.com> wrote: > Hi, > > I'm assuming you're using consumer groups? E.g., group.id=X > > Cheers, > > Liam > > On Mon, 13 Dec 2021, 6:30 pm Jigar Shah, <jigar.shah1...@gmail.com> wrote: > >> Hello, >> I am trying to test the latency between message production and message >> consumption using Java Kafka-Client*(2.7.2)* library. >> The configuration of cluster is 3 KafkaBrokers*(2.7.2, Scala 2.13)*, 3 >> Zookeeper*(3.5.9)* >> Here is a pattern what I have observed >> Reference: >> ConsumerReadTimeStamp: Timestamp when record received in Kafka Consumer >> ProducerTimeStamp: Timestamp added before producer.send record >> RecordTimeStamp: CreateTimeStamp inside the record obtained at consumer >> >> [image: kafka1.png] >> >> *For 100 Messages* >> >> *ConsumerReadTimeStamp-ProducerTimeStamp(ms)* >> >> *ConsumerReadTimeStamp-RecordTimeStamp(ms)* >> >> *Average* >> >> *252.56* >> >> *238.85* >> >> *Max* >> >> *2723* >> >> *2016* >> >> *Min* >> >> *125* >> >> *125* >> >> >> On the consumer side it takes too much time for initial few messages but >> later on it is quite consistent. >> I have executed the above same test for large number of messages : >> 100,1000,10000,etc. and the pattern seems to be same >> Here are the configurations, mostly using default properties. >> Topic: >> partitions=16 >> min.insync.replica=2 >> replication.factor=3 >> >> >> Consumer: >> security.protocol=PLAINTEXT >> enable.auto.commit=true >> >> >> Producer: >> security.protocol=PLAINTEXT >> compression.type=gzip >> acks=all >> >> >> Is there any reason why there is huge latency at the beginning when a >> consumer is created please? >> Also please suggest some way to optimise configurations to have some >> better consistent results ? >> >> Thank you in advance for your feedback. >> >
