On Thu, Aug 14, 2014 at 09:56:11PM +0000, Prunier, Dominique wrote: > Hi, > > I'm playing around with Kafka with the idea to implement a general purpose > message exchanger for a distributed application with high throughput > requirements (multiple hundred thousand messages per sec). > > In this context, i would like to be able to use a topic as some form of > private mailbox for a single consumer group. In this situation, once the > single consumer group has committed its offset on its private topic, the > messages there won't be used anymore and can be safely discarded. Therefore, > i was wondering if you'd see a way (in the current release or in the future) > to have a topic which expiration policy is based on consumer offsets.
Kafka does not provide any specific consumer-driven expiration policy. However, it is possible to do the following which I think should accomplish what you want, but I wouldn't really recommend this: Use log compaction. (http://kafka.apache.org/documentation.html#compaction) So you could set your topic's retention policy to "compact" and attach a unique key with every message. After your consuming application consumes a set of messages, you can "delete" those messages by having your consuming application produce a tombstone message to that topic with that key. Those messages will be cleaned out when the log segment rolls over and the cleaner runs. That said I think it is much simpler if you just use the standard retention policy (and set a relatively low retention period) and just monitor your consumer lag. Joel
