On Thu, Aug 14, 2014 at 09:56:11PM +0000, Prunier, Dominique wrote:
> Hi,
> 
> I'm playing around with Kafka with the idea to implement a general purpose 
> message exchanger for a distributed application with high throughput 
> requirements (multiple hundred thousand messages per sec).
> 
> In this context, i would like to be able to use a topic as some form of 
> private mailbox for a single consumer group. In this situation, once the 
> single consumer group has committed its offset on its private topic, the 
> messages there won't be used anymore and can be safely discarded. Therefore, 
> i was wondering if you'd see a way (in the current release or in the future) 
> to have a topic which expiration policy is based on consumer offsets.

Kafka does not provide any specific consumer-driven expiration policy.
However, it is possible to do the following which I think should
accomplish what you want, but I wouldn't really recommend this:

Use log compaction.
(http://kafka.apache.org/documentation.html#compaction) So you could
set your topic's retention policy to "compact" and attach a unique key
with every message. After your consuming application consumes a set of
messages, you can "delete" those messages by having your consuming
application produce a tombstone message to that topic with that key.
Those messages will be cleaned out when the log segment rolls over and
the cleaner runs.

That said I think it is much simpler if you just use the standard
retention policy (and set a relatively low retention period) and just
monitor your consumer lag.

Joel

Reply via email to