At this point of investigation, I have no idea....
________________________________
De : Vishnu Manivannan <vis...@arrcus.com>
Envoyé : vendredi 20 juillet 2018 21:28:24
À : users@kafka.apache.org
Objet : Re: Kafka Connect: Increase Consumer Consumption
No, I'm just running one consumer now and trying to increase its consumption
before I increase the number of consumers.
Yeah I know it's weird, I thought I would see some changes in data consumption
after tweaking these parameters.
On 7/18/18, 11:48 PM, "adrien ruffie" <adriennolar...@hotmail.fr> wrote:
Strange enough ...
I don't really understand why.
Do you have multiple consumer ?
I don't know if a metric which gives the external consumption of a topic.
But if even simple metric exist, you can check if it's your consumer which
not really receive more than 1150 records,
and your topic gets out more than 5000 records.
Adrien
________________________________
De : Vishnu Manivannan <vis...@arrcus.com>
Envoyé : jeudi 19 juillet 2018 00:05:14
À : users@kafka.apache.org
Objet : Re: Kafka Connect: Increase Consumer Consumption
Hi Adrien,
I set fetch.max.wait.ms to 1500 ms and ran it again. It still isn't
crossing 1150 records per fetch. On the producer side (using
kakfa-producer-perf-test), its producing about 30,000 records/sec and a million
records in total.
I tried different configurations amongst these three parameters
(max.poll.records, fetch.min.bytes, fetch.max.wait.ms). All the other
parameters are unchanged and have their default values. I have not changed any
of the broker configs either. I read the Kafka documentation but could not find
any other parameters that could impact the fetch size.
Is there some other consumer or broker parameter that I might be missing?
Thanks,
Vishnu
On 7/18/18, 1:48 PM, "adrien ruffie" <adriennolar...@hotmail.fr> wrote:
Hi Vishnu,
do you have check your fetch.max.wait.ms value ?
it may not be long enough time to wait until you recover your 5000
records ...
maybe just enough time to recover only 1150 records.
fetch.max.wait.ms
By setting fetch.min.bytes, you tell Kafka to wait until it has enough
data to send before responding to the consumer. fetch.max.wait.ms lets you
control how long to wait. By default, Kafka will wait up to 500 ms. This
results in up to 500 ms of extra latency in case there is not enough data
flowing to the Kafka topic to satisfy the minimum amount of data to return. If
you want to limit the potential latency (usually due to SLAs controlling the
maximum latency of the application), you can set fetch.max.wait.ms to a lower
value. If you set fetch.max.wait.ms to 100 ms and fetch.min.bytes to 1 MB,
Kafka will receive a fetch request from the consumer and will respond with data
either when it has 1 MB of data to return or after 100 ms, whichever happens
first.
Best regards,
Adrien
________________________________
De : Vishnu Manivannan <vis...@arrcus.com>
Envoyé : mercredi 18 juillet 2018 21:00:50
À : users@kafka.apache.org
Objet : Kafka Connect: Increase Consumer Consumption
Hi,
I am currently working with a single Kafka broker and a single Kafka
consumer. I am trying to get the consumer to fetch more records, so I can
increase the batch size when I write the data to a DB.
Each record is about 1 KB and I am trying to fetch at least 5000
records each time. So, I changed the configurations for the following consumer
parameters:
* max.poll.records = 5000
* fetch.min.bytes = 5120000
For some reason, the maximum number of records fetched each time does
not go above 1150. Are there any other parameters that I should look into or
any changes I should make to the current configurations?
Thanks,
Vishnu
