I think first solution is what I need. Your second proposal also looks fine but I don't like idea of keeping additional ledger. Thanks Svante ! I appreciate your help.
Sincerely, Tomasz Kopacki DevOps Engineer @ Nokia -----Original Message----- From: Svante Karlsson [mailto:svante.karls...@csi.se] Sent: Wednesday, March 21, 2018 11:21 AM To: users@kafka.apache.org Subject: Re: log compaction v log rotation - best of the two worlds alt1) if you can store a generation counter in the value of the "latest value" topic you could do as follows topic latest_value key [id] topic full_history key[id, generation] on delete get the latest_value.generation_counter and issue deletes on full_history key[id, 0..generation_counter] alt2) if you cannot store a generation_counter in "latest_value" store a timestamp or uuid to make each key unique topic latest_value key [id] topic full_history key[id, timestamp/uuid] on delete of "id" scan full_history topic from beginning and issue deletes on full_history key[id, timestamp] you could optimized this by having another topic that contains a "to be purged ids" /svante 2018-03-21 11:16 GMT+01:00 Manikumar <manikumar.re...@gmail.com>: > Sorry, I was wrong. For history topic, we can use regular topic with > sufficient retention period. > maybe others can give more ideas. > > On Wed, Mar 21, 2018 at 3:34 PM, Kopacki, Tomasz (Nokia - PL/Wroclaw) > < tomasz.kopa...@nokia.com> wrote: > > > Do you mean I can use tombstone if my clean policy is 'delete' and > > it still work ? > > > > Sincerely, > > Tomasz Kopacki > > DevOps Engineer @ Nokia > > > > -----Original Message----- > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > Sent: Wednesday, March 21, 2018 11:03 AM > > To: users@kafka.apache.org > > Subject: Re: log compaction v log rotation - best of the two worlds > > > > Not sure if understood requirement correctly. one option is to use > > two compacted topic topics. one is for current state of the resource > > and one > is > > for history. and use tombstones whenever you want to clear them. > > > > On Wed, Mar 21, 2018 at 2:53 PM, Kopacki, Tomasz (Nokia - > > PL/Wroclaw) < tomasz.kopa...@nokia.com> wrote: > > > > > Almost, > > > Consider this example: > > > > > > |R1|R2|R1|R3|R2|R4|R4|R1|R2| <- this is an example of a stream > > > |R1|R2|R1|R3|R2|R4|R4|R1|R2| where RX > > > represents updates of a particular resource. I need to keep the > > > history of changes forever for all the resources but only until > > > resource is alive. If resource expires/dies I'd like to remove it > > > completely. In this example consider that resource R2 dies but > > > others are still alive. In such case I'd like to able to transform this > > > into: > > > |R1|R1|R3|R4|R4|R1| <- so, it's not compaction because I need the > > > |R1|R1|R3|R4|R4|R1| history > > > of changes but neither I can simply remove 'old' messages because > > > I need to do this based of the lifecycle of the resource not just > > > their > > age. > > > > > > > > > > > > Sincerely, > > > Tomasz Kopacki > > > DevOps Engineer @ Nokia > > > > > > -----Original Message----- > > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > > Sent: Wednesday, March 21, 2018 10:17 AM > > > To: users@kafka.apache.org > > > Subject: Re: log compaction v log rotation - best of the two > > > worlds > > > > > > We can enable both compaction and retention for a topic by setting > > > cleanup.policy="delete,compact" > > > http://kafka.apache.org/documentation/#topicconfigs > > > > > > Does this handle your requirement? > > > > > > On Wed, Mar 21, 2018 at 2:36 PM, Kopacki, Tomasz (Nokia - > > > PL/Wroclaw) < tomasz.kopa...@nokia.com> wrote: > > > > > > > Hi, > > > > I've been recently exploring log handling in kafka and I wonder > > > > if/how can I mixed log compaction with log rotation. > > > > A little background first: > > > > I have an application that uses kafka topics as a backend for > > > > event sourcing. Messages represents change of state of my 'resources'. > > > > Each resource has UID that is also used as a key for the messages. > > > > My resources have a lifecycle and when their life ends I don't > > > > need them anymore and there is no point in keeping their > > > > history. Having said that I thought that best choice for me will > > > > be log compaction with tombstone feature but I also would like > > > > to have a possibility to keep history of changes for the resources(only > > > > until they die). > > > > With those requirements I'd love to have a possibility to use > > > > tombstone feature for log rotation but I guess it ain't working > > > > like > > > that. > > > > > > > > Does anyone here had similar requirements and solve that somehow ? > > > > > > > > > > > > Sincerely, > > > > Tomasz Kopacki > > > > DevOps Engineer @ Nokia > > > > > > > > > > > > > >
