Re: Huge latency at consumer side ,testing performance for production and consumption

2022-01-28 Thread Jigar Shah
Hello Liam, Here is the image. I hope it is accessible now *Regards,* *Jigar* On Fri, 28 Jan 2022 at 15:04, Liam Clarke-Hutchinson wrote: > Hi Jigar, > > Your image attachment didn't come through again. > > Thanks, > > Liam > > On Fri, 28 Jan 2022, 5:35 pm Jigar Shah, wrote: > > > Hello aga

Re: Huge latency at consumer side ,testing performance for production and consumption

2022-01-28 Thread Liam Clarke-Hutchinson
Hi Jigar, Your image attachment didn't come through again. Thanks, Liam On Fri, 28 Jan 2022, 5:35 pm Jigar Shah, wrote: > Hello again, > Could someone please provide feedback on these findings ? > Thank you in advance for feedback. > > *Regards,* > *Jigar* > > > > On Mon, 17 Jan 2022 at 13:24

Re: Huge latency at consumer side ,testing performance for production and consumption

2022-01-27 Thread Jigar Shah
Hello again, Could someone please provide feedback on these findings ? Thank you in advance for feedback. *Regards,* *Jigar* On Mon, 17 Jan 2022 at 13:24, Jigar Shah wrote: > Hello again, > I had performed a few more tests on producer and consumer again and I > observed a pattern in Kafka Pro

Re: Huge latency at consumer side ,testing performance for production and consumption

2022-01-16 Thread Jigar Shah
Hello again, I had performed a few more tests on producer and consumer again and I observed a pattern in Kafka Producer creating large latency. Could you please confirm that my understanding is correct about the producer protocol? The configurations are the same as above. The producer is continuo

Re: Huge latency at consumer side ,testing performance for production and consumption

2021-12-14 Thread Jigar Shah
Hello, I agree with time taken for consumer initialization processes But actually in the test I am taking care of that and I am waiting for the consumer to be initiated and only then starting the producer to discount the initialization delay. So, are there any more processes happening during the po

Re: Huge latency at consumer side ,testing performance for production and consumption

2021-12-13 Thread Luke Chen
Hi Jigar, As Liam mentioned, those are necessary consumer initialization processes. So, I don't think you can speed it up by altering some timeouts/interval properties. Is there any reason why you need to care about the initial delay? If, like you said, the delay won't happen later on, I think the

Re: Huge latency at consumer side ,testing performance for production and consumption

2021-12-13 Thread Jigar Shah
Hello , Answering your first mail, indeed I am using consumer groups using group.id , I must have missed to add it in mentioned properties Also, thank you for information regarding the internal processes working behind creating a KafkaConsumer. I agree that following steps do add latency during ini

Re: Huge latency at consumer side ,testing performance for production and consumption

2021-12-12 Thread Liam Clarke-Hutchinson
I realise that's a silly question, you must be if you're using auto commit. When a consumer starts, it needs to do a few things. 1) Connect to a bootstrap server 2) Join an existing consumer group, or create a new one, if it doesn't exist. This may cause a stop the world rebalance as partitions

Re: Huge latency at consumer side ,testing performance for production and consumption

2021-12-12 Thread Liam Clarke-Hutchinson
Hi, I'm assuming you're using consumer groups? E.g., group.id=X Cheers, Liam On Mon, 13 Dec 2021, 6:30 pm Jigar Shah, wrote: > Hello, > I am trying to test the latency between message production and message > consumption using Java Kafka-Client*(2.7.2)* library. > The configuration of cluster