Almost, Consider this example: |R1|R2|R1|R3|R2|R4|R4|R1|R2| <- this is an example of a stream where RX represents updates of a particular resource. I need to keep the history of changes forever for all the resources but only until resource is alive. If resource expires/dies I'd like to remove it completely. In this example consider that resource R2 dies but others are still alive. In such case I'd like to able to transform this into: |R1|R1|R3|R4|R4|R1| <- so, it's not compaction because I need the history of changes but neither I can simply remove 'old' messages because I need to do this based of the lifecycle of the resource not just their age.
Sincerely, Tomasz Kopacki DevOps Engineer @ Nokia -----Original Message----- From: Manikumar [mailto:manikumar.re...@gmail.com] Sent: Wednesday, March 21, 2018 10:17 AM To: users@kafka.apache.org Subject: Re: log compaction v log rotation - best of the two worlds We can enable both compaction and retention for a topic by setting cleanup.policy="delete,compact" http://kafka.apache.org/documentation/#topicconfigs Does this handle your requirement? On Wed, Mar 21, 2018 at 2:36 PM, Kopacki, Tomasz (Nokia - PL/Wroclaw) < tomasz.kopa...@nokia.com> wrote: > Hi, > I've been recently exploring log handling in kafka and I wonder if/how > can I mixed log compaction with log rotation. > A little background first: > I have an application that uses kafka topics as a backend for event > sourcing. Messages represents change of state of my 'resources'. Each > resource has UID that is also used as a key for the messages. > My resources have a lifecycle and when their life ends I don't need > them anymore and there is no point in keeping their history. Having > said that I thought that best choice for me will be log compaction > with tombstone feature but I also would like to have a possibility to > keep history of changes for the resources(only until they die). > With those requirements I'd love to have a possibility to use > tombstone feature for log rotation but I guess it ain't working like that. > > Does anyone here had similar requirements and solve that somehow ? > > > Sincerely, > Tomasz Kopacki > DevOps Engineer @ Nokia > >
