Re: Reusable consumer across consumer groups

Sorry for late reply. Not sure what more details you need. Here's example http://confluent.io/docs/current/kafka-rest/docs/intro.html of exposing Kafka through remoting (http/rest) :-) One can without looking into kafka rest proxy code see based on its limitations that it's using HL consumer, with

Re: Reusable consumer across consumer groups

I'm wondering how much of this can be done using careful system design vs building it within the consumer itself. You could distribute the several consumer instances across machines since it is built for distributed load balancing. That will sufficiently isolate the resources required to run the va

Re: Reusable consumer across consumer groups

Imagine exposing Kafka over various remoting protocols, where incoming poll/read requests may come in concurrently for different consumer groups, especially in a case with lots of different consumer groups. If you create and destroy KafkaConsumer for each such request, response times and throughput

Re: Reusable consumer across consumer groups

Stevo, The new consumer API is planned for 0.9, not 0.8.2. You can take a look at a detailed javadoc here . Can you explain why you would like to poll messages across consu

Reusable consumer across consumer groups

Hello Apache Kafka community, Current (Kafka 0.8.1.1) high-level API's KafkaConsumer is not lightweight object, it's creation takes some time and resources, and it does not seem to be thread-safe. It's API also does not support reuse, for consuming messages from different consumer groups. I see e