By design, Kafka stores data independent of the number of publishers or subscribers connecting to it. This provides high performance as the broker does not have to manage consumers and evict data based on the consumers position. This is one of the main reasons why Kafka is much more performance compared to the JMS queues.
It seems like your use case requires the concept of ephemeral topics where you would like to auto delete a topic once a particular consumer group has finished consuming data from it. Once 0.8.2 is released with the delete topic support, we intend to add auto expiration of topics that will delete topics that have not been accessed in some configurable time. Is there a reason why your application needs to create such short lived topics? Thanks, Neha On Thu, Aug 14, 2014 at 2:56 PM, Prunier, Dominique < dominique.prun...@emc.com> wrote: > Hi, > > I'm playing around with Kafka with the idea to implement a general purpose > message exchanger for a distributed application with high throughput > requirements (multiple hundred thousand messages per sec). > > In this context, i would like to be able to use a topic as some form of > private mailbox for a single consumer group. In this situation, once the > single consumer group has committed its offset on its private topic, the > messages there won't be used anymore and can be safely discarded. > Therefore, i was wondering if you'd see a way (in the current release or in > the future) to have a topic which expiration policy is based on consumer > offsets. > > Thanks, > > -- > Dominique Prunier > >
