Imagine exposing Kafka over various remoting protocols, where incoming poll/read requests may come in concurrently for different consumer groups, especially in a case with lots of different consumer groups. If you create and destroy KafkaConsumer for each such request, response times and throughput will be very low, and doing that is one of the ways to reproduce https://issues.apache.org/jira/browse/KAFKA-1716
It would be better if one could reuse a (pool of) Consumer instances, and through a read operation parameter specify for which consumer group should read be performed. Kind regards, Stevo Slavic. On Tue, Oct 14, 2014 at 6:17 PM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > Stevo, > > The new consumer API is planned for 0.9, not 0.8.2. You can take a look at > a detailed javadoc here > < > http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > > . > > Can you explain why you would like to poll messages across consumer groups > using just one instance? > > Thanks, > Neha > > On Tue, Oct 14, 2014 at 1:03 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > Hello Apache Kafka community, > > > > Current (Kafka 0.8.1.1) high-level API's KafkaConsumer is not lightweight > > object, it's creation takes some time and resources, and it does not seem > > to be thread-safe. It's API also does not support reuse, for consuming > > messages from different consumer groups. > > > > I see even in the coming (0.8.2) redesigned API it will not be possible > to > > reuse consumer instance to poll messages from different consumer groups. > > > > Can something be done to support this? > > > > Would it help if there was consumer group as a separate entity from > > consumer, for all the subscription management tasks? > > > > Kind regards, > > Stevo Slavic > > >
