It depends on the use case. Look at lingerMs (10ms) So the latency is 10ms.
` Received invalid metadata error in produce request on partition inbound_topic-1 due to org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.. Going to request metadata update now` - Check your broker logs. My question remains same , Did you try to produce records using console producer in your setup ? On Wed, Jul 3, 2019 at 1:55 PM Shyam P <shyamabigd...@gmail.com> wrote: > thank you. > Senthil , > Why batch.size=65536 & retries=100, its big number right ? do we need > this much for streaming applications ? > > Regards, > Shyam > > On Wed, Jul 3, 2019 at 1:30 PM SenthilKumar K <senthilec...@gmail.com> > wrote: > >> `*Partition = -1` - *This explains why are you getting timeout error. >> >> Why dont you use Default Partitioner ?: >> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java >> >> Try only with below producer properties : >> bootstrap.servers=<<>>> >> acks =1 >> retries=100 >> batch.size=65536 >> linger.ms=10 >> key.serializer=<<>> >> value.serializer=<<>>> >> compression.type=<<snappy>> >> buffer.memory=104857600 >> >> --Senthil >> >> >> On Wed, Jul 3, 2019 at 1:18 PM Shyam P <shyamabigd...@gmail.com> wrote: >> >>> Hi SenthilKumar, >>> thanks a lot . >>> >>> Yeah I set up local set up and print the log with partition info which >>> shows as below >>> >>> 2019-07-03 02:48:28.822 INFO 7092 --- [lt-dispatcher-2] >>> c.s.c.p.p.CompanyInfoPartitioner : Topic : inbound_topic Key = 597736248- >>> Entropy Cayman Solar Ltd.-null-null-null *Partition = -1* 2019-07-03 >>> 02:48:28.931 ERROR 7092 --- [ad | producer-1] >>> o.s.k.support.LoggingProducerListener : Exception thrown when sending a >>> message with key='597736248- Entropy Cayman Solar Ltd.-null-null-null' and >>> payload='com.spgmi.ca.prescore.model.Company@8b12343' to topic >>> inbound_topic : >>> >>> org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) >>> for inbound_topic --1: 104 ms has passed since batch creation plus linger >>> time >>> >>> My topics inbound_topic has two partitions as you see below >>> C:\Software\kafka\kafka_2.11-2.1.1\bin\windows>kafka-topics.bat --describe >>> --zookeeper localhost:2181 --topic inbound_topic Topic:inbound_topic >>> PartitionCount:2 ReplicationFactor:1 Configs: Topic: inbound_topic >>> Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Topic: inbound_topic Partition: 1 >>> Leader: 0 Replicas: 0 Isr: 0 >>> >>> *But my producer seems to trying to send to Partition = -1.* >>> >>> My partition logic is as below >>> >>> int p = (((String)key).hashCode() * Integer.MAX_VALUE) % numPartitions; >>> logger.info("Topic : "+ topic + "\t Key = " + (String)key + " >>> Partition = " + p ); >>> >>> On key i am doing hashCode(). What need to be corrected here to avoid >>> this negative number partition number ? i.e. Partition = -1 >>> >>> What should be my partition key logic like ? >>> >>> any help highly appreciated. >>> Regards, >>> Shyam >>> >>> On Tue, Jul 2, 2019 at 8:48 PM SenthilKumar K <senthilec...@gmail.com> >>> wrote: >>> >>>> Does it happen to all partitions or only few partitions ? Can you make >>>> sure your local setup working fine ? Were you able to produce using >>>> console-producer ? >>>> >>>> Example : >>>> EVERE: Expiring 7 record(s) for topic-9{partition:9}: 30022 ms has >>>> passed since last append >>>> Expiring 9 record(s) for topic-2{partition:2}: 30015 ms has passed >>>> since batch creation plus linger time >>>> >>>> --Senthil >>>> >>>> On Tue, Jul 2, 2019 at 5:34 PM Shyam P <shyamabigd...@gmail.com> wrote: >>>> >>>>> Thanks a lot Senthil for quick reply. >>>>> I am using kafka_2.11-2.1.1 . >>>>> In your case Kafka Producer Client in One DataCenter and Kafka Broker >>>>> in other DataCenter but in my case I installed Kafka on the same machine >>>>> where Producer is running. >>>>> i.e. currently I am in development mode , so everything now on my >>>>> local for timebeing ...i.e. Kafka broker , zk and my producer code in >>>>> eclipse. >>>>> >>>>> If is is a set up issue at least it should run fine in my local right. >>>>> I tried several producer configurations combinations as explained in >>>>> the SOF link. >>>>> >>>>> So not sure now what is the issue and how to fix it ? >>>>> >>>>> Is in your case the issue fixed ? >>>>> >>>>> Regards, >>>>> Shyam >>>>> >>>>> On Tue, Jul 2, 2019 at 5:12 PM SenthilKumar K <senthilec...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Shyam, We also faced `TimeoutException: Expiring 1 record(s)` >>>>>> issue in >>>>>> our Kafka Producer Client. As described here >>>>>> < >>>>>> https://stackoverflow.com/questions/56807188/how-to-fix-kafka-common-errors-timeoutexception-expiring-1-records-xxx-ms-has >>>>>> > >>>>>> , >>>>>> first we tried increasing request timeout but that didn't help. We >>>>>> had >>>>>> setup like Kafka Producer Client in One DataCenter and Kafka Broker in >>>>>> other DataCenter & thats why the producer failed to push records to >>>>>> brokers >>>>>> on time due to network issue. In your case , Could be setup issue ? >>>>>> >>>>>> --Senthil >>>>>> >>>>>> On Tue, Jul 2, 2019 at 3:57 PM Shyam P <shyamabigd...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> > Hi, >>>>>> > I am facing the below issue. >>>>>> > >>>>>> > org.apache.kafka.common.errors.TimeoutException: Expiring 1 >>>>>> record(s) >>>>>> > for <topic-name> 229 ms has passed since batch creation plus linger >>>>>> > time >>>>>> > >>>>>> > >>>>>> > I tried many producer configuration settings. more details below : >>>>>> > >>>>>> > >>>>>> https://stackoverflow.com/questions/56807188/how-to-fix-kafka-common-errors-timeoutexception-expiring-1-records-xxx-ms-has >>>>>> > >>>>>> > But nothing working. >>>>>> > >>>>>> > Can anyone plz help me , what is wrong here and how to fix it ? >>>>>> > >>>>>> > thanks, >>>>>> > Shyam >>>>>> > >>>>>> >>>>>
