Re: Impact on having large number of consumers on producers / brokers

2019-10-21 Thread Shyam P
What are you trying to do here ? whats your objective ? On Sat, Oct 19, 2019 at 8:45 PM Hrishikesh Mishra wrote: > Can anyone please help me with this? > > On Fri, 18 Oct 2019 at 2:58 PM, Hrishikesh Mishra > wrote: > > > Hi all, > > > > I wanted to understand, having large numbers of consumers

Re: Does anyone fixed Producer TimeoutException problem ?

2019-07-03 Thread Shyam P
Yes I tried with console-producer , it is working fine. But its a plain message i.e. String .. but i have to send a 20 field json as message.

Re: Does anyone fixed Producer TimeoutException problem ?

2019-07-03 Thread Shyam P
retries=100 > batch.size=65536 > linger.ms=10 > key.serializer=<<>> > value.serializer=<<>>> > compression.type=<> > buffer.memory=104857600 > > --Senthil > > > On Wed, Jul 3, 2019 at 1:18 PM Shyam P wrote: > >> Hi SenthilKumar, >>

Re: Does anyone fixed Producer TimeoutException problem ?

2019-07-03 Thread Shyam P
s backoff time I am using below settings. acks: 1 retries: 1 batchSize: 10 lingerMs: 5 bufferMemory: 33554432 requestTimeoutMs: 90 What should be corrected here to fix above error. On Wed, Jul 3, 2019 at 1:18 PM Shyam P wrote: > Hi SenthilKumar, > thanks a lot . >

Re: Does anyone fixed Producer TimeoutException problem ?

2019-07-03 Thread Shyam P
r local setup working fine ? Were you able to produce using > console-producer ? > > Example : > EVERE: Expiring 7 record(s) for topic-9{partition:9}: 30022 ms has passed > since last append > Expiring 9 record(s) for topic-2{partition:2}: 30015 ms has passed since > batch cr

Re: Does anyone fixed Producer TimeoutException problem ?

2019-07-02 Thread Shyam P
ailed to push records to brokers > on time due to network issue. In your case , Could be setup issue ? > > --Senthil > > On Tue, Jul 2, 2019 at 3:57 PM Shyam P wrote: > > > Hi, > > I am facing the below issue. > > > > org.apache.kafka.common.errors.Timeout

Fwd: Does anyone fixed Producer TimeoutException problem ?

2019-07-02 Thread Shyam P
Hi, I am facing the below issue. org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for 229 ms has passed since batch creation plus linger time I tried many producer configuration settings. more details below : https://stackoverflow.com/questions/56807188/how-to-fix-kafka-co

Does anyone fixed Producer TimeoutException problem ?

2019-07-02 Thread Shyam P
Hi, I am facing the below issue. org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for 229 ms has passed since batch creation plus linger time I tried many producer configuration settings. more details below : https://stackoverflow.com/questions/56807188/how-to-fix-kafka-co

How spark structured streaming consumers initiated and invoked while reading multi-partitioned kafka topics?

2019-06-10 Thread Shyam P
Hi, Any suggestions regarding below issue? https://stackoverflow.com/questions/56524921/how-spark-structured-streaming-consumers-initiated-and-invoked-while-reading-mul Thanks, Shyam

Kafka TotalTimeMs time interval

2018-05-11 Thread Shyam P R
Hi All, I have a question regarding Kafka TotalTimeMs metrics and similar metrics. What is the time interval for which the metrics is calculated. For example let us take the MBean kafka.network:name=TotalTimeMs,request=Produce,type=RequestMetrics as an example. In this metric, there is a Mean and