Re: What's the behavior when Kafka is deleting messages and consumers are still reading

2014-06-24 Thread Guozhang Wang
I think we can probably update the documentation page for this update: https://kafka.apache.org/documentation.html#compaction On Tue, Jun 24, 2014 at 3:54 PM, Lian, Li wrote: > GuoZhang, > > Thanks for explaining. I thought there might be such kind of lock > mechanism but cannot confirm it in

Re: What's the behavior when Kafka is deleting messages and consumers are still reading

2014-06-24 Thread Lian, Li
GuoZhang, Thanks for explaining. I thought there might be such kind of lock mechanism but cannot confirm it in any documentation on Kafka website. It will be better if this could be written down in some Wiki or FAQ. Best regards, Lex Lian Email: ll...@ebay.com On 6/24/14, 11:17 PM, "Guozh

Re: What's the behavior when Kafka is deleting messages and consumers are still reading

2014-06-24 Thread Neha Narkhede
The behavior on the consumer in this case is governed by the value of the "auto.offset.reset" config. Depending on this config, it will reset it's offset to either the earliest or the latest in the log. Thanks, Neha On Tue, Jun 24, 2014 at 8:17 AM, Guozhang Wang wrote: > Hi Li, > > The log ope

Re: What's the behavior when Kafka is deleting messages and consumers are still reading

2014-06-24 Thread Guozhang Wang
Hi Li, The log operations are protected by a lock, so if there is a concurrent read on this partition it will not be deleted. But then when it is deleted the next fetch/read will result in a OffsetOutOfRange exception and the consumer needs to restart from a offset reset value. Guozhang On Tue,