Hi Bill, Have you looked at: http://kafka.apache.org/documentation.html#compaction
It supports deletes so if you keyed by transaction_id you could, in theory, delete them. Cheers, Damian On 20 January 2016 at 18:35, Bill Warshaw <bill.wars...@appian.com> wrote: > Hello, > > I'm working on a team that is starting to use Kafka as a distributed > transaction log for a set of in-memory databases which can be replicated > across nodes. We decided to use Kafka instead of Bookkeeper for a variety > of reasons, but there are a couple spots where Kafka is not a perfect fit. > > The biggest issue facing us is deleting old transactions from the log after > checkpointing the database. We can't use any of the built-in size or > time-based deletion mechanisms efficiently, because we could get ourselves > into a dangerous state where we're deleting transactions that haven't been > checkpointed yet. The current approach we're looking at is rolling a new > topic each time we checkpoint, and deleting the old topic once all replicas > have consumed everything in it. > > Another idea we came up with is using a pluggable compaction policy; we > would set the message key as the offset or transaction id, and the policy > would delete all messages with a key smaller than that id. > I took a stab at implementing the hook in Kafka for pluggable compaction > policies at > > https://github.com/apache/kafka/compare/trunk...bill-warshaw:pluggable_compaction_policy > (rough implementation), and it seems fairly straightforward. One problem > that we run into is that the custom policy class can only access > information that is defined in the configuration, and the configuration > doesn't allow custom key-value pairs; if we wanted to pass it information > dynamically, we'd have to use some hack like calling Zookeeper from within > the class. > To get around this, my best idea is to add the ability to specify arbitrary > key-value pairs in the configuration, that our client could use to pass > information to the custom policy. Does this set off any alarm bells for > you guys? If so, are there other approaches we could take that come to > mind? > > > Thanks for your time, > Bill Warshaw > > -- > <http://appianworld.com> > This message and any attachments are solely for the intended recipient. If > you are not the intended recipient, disclosure, copying, use, or > distribution of the information included in this message is prohibited -- > please immediately and permanently delete this message. >
