Re: Impact of slow consumers

2014-03-18 Thread Jay Kreps
In general this is expected. Consumers generally read from the OS cache and so they do no I/O. However a slow consumer may fall out of this cached portion of the log and do real reads. These reads will compete with writes for disk bandwidth. However what is puzzling is that you mention that other

Re: Impact of slow consumers

2014-03-18 Thread Abhinav Anand
Hi Guozhang, The throughput refers to the broker side throughput. The peak time throughput of broker for the topic is around (9-10 MBps) but with slow consumers the throughput went down to 5-6 MBps. I consume data using a consumer group. Consumer group needs 5 consumers to consume the entire data

Re: Impact of slow consumers

2014-03-18 Thread Guozhang Wang
Hello Abhinav, When you say "slow consumer set on topic1", do you mean you have a consumer group consuming from topic1, with one of them slow, or do you mean all consumers are slow in that group, or do you use a simple consumer? When you say "throughput", do you mean consumer side throughput, or b