Re: Kafka compacted topic question.

Thank you Naresh. It answers my question. On Sat, Jan 20, 2018 at 8:15 AM, naresh Goud wrote: > Hi Rahul, > > If I understand your question correctly, you are intrested only in latest > values for keys and don't want to maintain any older values immediately > after update happened in a value for

Re: Kafka compacted topic question.

Hi Rahul, If I understand your question correctly, you are intrested only in latest values for keys and don't want to maintain any older values immediately after update happened in a value for given key If you wanted all of segments to include in compaction consider this property value how much i

Re: Kafka compacted topic question.

Ok , so there is no attempt made for de-duplication while the row is still hot in memtable. Why is this behaviour? For compact topics we are only interested in last update for any key. thanks, Rahul On Fri, Jan 19, 2018 at 3:18 PM, Matthias J. Sax wrote: > Yes and no. > > There is a background

Re: Kafka compacted topic question.

Yes and no. There is a background compaction thread that runs periodically (you can configure the scheduling for this thread). Thus, compaction happens async. It's correct, that the current head segments is not considered for compaction. There is also no de-duplication on write, but message will

Re: Kafka compacted topic question.

Yeah, and I thought I answered your question? I think the compaction happens when new segments are created. Sorry if I’m still misunderstanding. > On Jan 19, 2018, at 3:55 PM, Rahul Bhattacharjee > wrote: > > Thanks Matt for the response .I was asking about the log compaction >

Re: Kafka compacted topic question.

Thanks Matt for the response .I was asking about the log compaction of kafka topics. On Fri, Jan 19, 2018 at 12:36 PM, Matt Farmer wrote: > Someone will need to correct me if I’m wrong, but my understanding is that > a topic log on disk is div

Re: Kafka compacted topic question.

Someone will need to correct me if I’m wrong, but my understanding is that a topic log on disk is divided into segments. Compaction will occur when a segment “rolls off” - so when a new active segment is created and the previous segment becomes inactive. Segments can be bounded by size and time

Kafka compacted topic question.

Let's say we have a compacted topic (log.cleanup.policy=compact) where lot of updates happen for relatively small set of keys. My question is when does the compaction happen. In memtable , when a new update comes for an already existing key in memtable , the value is simple replaced. or, all the u