Hi Karan, I think the issue is in verification step. Because the start and end offsets are not going to be reset when messages are deleted. Have you checked whether a consumer would see the messages that are supposed to be deleted? Thanks.
--Vahid From: karan alang <karan.al...@gmail.com> To: users@kafka.apache.org Date: 06/22/2017 06:09 PM Subject: Re: Deleting/Purging data from Kafka topics (Kafka 0.10) Hi Vahid, somehow, the changes suggested don't seem to be taking effect, and i dont see the data being purged from the topic. Here are the steps i followed - 1) topic is set with param -- retention.ms=1000 $KAFKA10_HOME/bin/kafka-topics.sh --describe --topic topicPurge --zookeeper localhost:2161 Topic:topicPurge PartitionCount:3 ReplicationFactor:3 Configs:retention.ms =1000 Topic: topicPurge Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Topic: topicPurge Partition: 1 Leader: 0 Replicas: 0,2,3 Isr: 0,2,3 Topic: topicPurge Partition: 2 Leader: 1 Replicas: 1,3,0 Isr: 1,3,0 2) There are 4 brokers, and in the server.properties (for each of the brokers), i've modified the following property log.retention.check.interval.ms=30000 I am expecting the data to be purged every 30 secs based on property - log.retention.check.interval.ms, however, that does not seem to be happening. 3) Here is the command to check the offsets $KAFKA10_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:6092,localhost:6093,localhost:6094,localhost:6095 --topic topicPurge --time -1 --partitions 0,1,2 topicPurge:0:67 topicPurge:1:67 topicPurge:2:66 Any ideas on what the issue might be ? On Thu, Jun 22, 2017 at 1:31 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Karan, > > The other broker config that plays a role here is > "log.retention.check.interval.ms". > For a low log retention time like in your example if this broker config > value is much higher, then the broker doesn't delete old logs regular > enough. > > --Vahid > > > > From: karan alang <karan.al...@gmail.com> > To: users@kafka.apache.org > Date: 06/22/2017 12:27 PM > Subject: Deleting/Purging data from Kafka topics (Kafka 0.10) > > > > Hi All - > How do i go about deleting data from Kafka Topics ? I've Kafka 0.10 > installed. > > I tried setting the parameter of the topic as shown below -> > > $KAFKA10_HOME/bin/kafka-topics.sh --zookeeper localhost:2161 --alter > --topic mmtopic6 --config retention.ms=1000 > I was expecting to have the data purged in about a min or so .. however, > i > dont see that happening .. > any ideas on what needs to be done ? > > > > >
