ok, thanks ... i'll check that and update in case i see issues. regds, Karan Alang
On Fri, Jun 23, 2017 at 12:01 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Karan, > > I think what you are seeing with `--time -1` and '--time -2` confirms that > the messages are deleted from the log. > The offset returned in both cases is the same, which means that the offset > start and offset end are both the same (i.e. the log is empty). > When messages are removed from the log the offsets won't reset to 0. The > offset index just keeps increasing, instead the offset start changes over > time when log retention occurs. > > So, in order to find the number of messages in a partition, you can just > get the difference of the offsets returned from `--time -1` and `--time > -2`. > I hope this answers your question. > > Thanks. > --Vahid > > > > > From: karan alang <karan.al...@gmail.com> > To: users@kafka.apache.org > Date: 06/22/2017 11:14 PM > Subject: Re: Deleting/Purging data from Kafka topics (Kafka 0.10) > > > > Hi Vahid, > here is the output of the GetOffsetShell commands (with --time -1 & -2) > > $KAFKA10_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell > --broker-list localhost:6092,localhost:6093,localhost:6094,localhost:6095 > --topic topicPurge --time -2 --partitions 0,1,2 > > topicPurge:0:67 > > topicPurge:1:67 > > topicPurge:2:66 > > Karans-MacBook-Pro-3:config karanalang$ > $KAFKA10_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell > --broker-list localhost:6092,localhost:6093,localhost:6094,localhost:6095 > --topic topicPurge --time -1 --partitions 0,1,2 > > topicPurge:0:67 > > topicPurge:1:67 > > topicPurge:2:66 > > > So, how do i interpret the above ? I was expecting the zookeeper to be > purged too .. & the offsets shown as 0, however that is not the case. (the > observation seem to tally with what you put in your email,i think) > > Also, the consumer is not able to read any data.. so i guess the data is > actually purged ? > > However, that also brings up additional questions .. > > I was using the GetOffsetShell command to get the count, but seems that is > not necessarily the right way .. > > What command should be used to get the count ? > > On Thu, Jun 22, 2017 at 8:34 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > Hi Karan, > > > > Just to clarify, with `--time -1` you are getting back the latest offset > > of the partition. > > If you do `--time -2` you'll get the earliest valid offset. > > > > So, let's say the latest offset of partition 0 of topic 'test' is 100. > > When you publish 5 messages to the partition, and before retention > policy > > kicks in, > > - with `--time -1` you should get test:0:105 > > - with `--time -2` you should get test:0:100 > > > > But after retention policy kicks in and old messages are removed, > > - with `--time -1` you should get test:0:105 > > - with `--time -2` you should get test:0:105 > > > > Could you please advise whether you're seeing a different behavior? > > > > Thanks. > > --Vahid > > > > > > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > To: users@kafka.apache.org > > Date: 06/22/2017 06:43 PM > > Subject: Re: Deleting/Purging data from Kafka topics (Kafka 0.10) > > > > > > > > Hi Karan, > > > > I think the issue is in verification step. Because the start and end > > offsets are not going to be reset when messages are deleted. > > Have you checked whether a consumer would see the messages that are > > supposed to be deleted? Thanks. > > > > --Vahid > > > > > > > > From: karan alang <karan.al...@gmail.com> > > To: users@kafka.apache.org > > Date: 06/22/2017 06:09 PM > > Subject: Re: Deleting/Purging data from Kafka topics (Kafka 0.10) > > > > > > > > Hi Vahid, > > > > somehow, the changes suggested don't seem to be taking effect, and i > dont > > see the data being purged from the topic. > > > > Here are the steps i followed - > > > > 1) topic is set with param -- retention.ms=1000 > > > > $KAFKA10_HOME/bin/kafka-topics.sh --describe --topic topicPurge > > --zookeeper > > localhost:2161 > > > > Topic:topicPurge PartitionCount:3 ReplicationFactor:3 > Configs:retention.ms > > =1000 > > > > Topic: topicPurge Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 > > > > Topic: topicPurge Partition: 1 Leader: 0 Replicas: 0,2,3 Isr: 0,2,3 > > > > Topic: topicPurge Partition: 2 Leader: 1 Replicas: 1,3,0 Isr: 1,3,0 > > > > > > 2) There are 4 brokers, and in the server.properties (for each of the > > brokers), i've modified the following property > > > > log.retention.check.interval.ms=30000 > > > > I am expecting the data to be purged every 30 secs based on property - > > log.retention.check.interval.ms, however, that does not seem to be > > happening. > > > > 3) Here is the command to check the offsets > > > > $KAFKA10_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell > > --broker-list > localhost:6092,localhost:6093,localhost:6094,localhost:6095 > > --topic topicPurge --time -1 --partitions 0,1,2 > > > > topicPurge:0:67 > > > > topicPurge:1:67 > > > > topicPurge:2:66 > > > > > > Any ideas on what the issue might be ? > > > > > > > > > > > > > > > > On Thu, Jun 22, 2017 at 1:31 PM, Vahid S Hashemian < > > vahidhashem...@us.ibm.com> wrote: > > > > > Hi Karan, > > > > > > The other broker config that plays a role here is > > > "log.retention.check.interval.ms". > > > For a low log retention time like in your example if this broker > config > > > value is much higher, then the broker doesn't delete old logs regular > > > enough. > > > > > > --Vahid > > > > > > > > > > > > From: karan alang <karan.al...@gmail.com> > > > To: users@kafka.apache.org > > > Date: 06/22/2017 12:27 PM > > > Subject: Deleting/Purging data from Kafka topics (Kafka 0.10) > > > > > > > > > > > > Hi All - > > > How do i go about deleting data from Kafka Topics ? I've Kafka 0.10 > > > installed. > > > > > > I tried setting the parameter of the topic as shown below -> > > > > > > $KAFKA10_HOME/bin/kafka-topics.sh --zookeeper localhost:2161 --alter > > > --topic mmtopic6 --config retention.ms=1000 > > > I was expecting to have the data purged in about a min or so .. > > however, > > > i > > > dont see that happening .. > > > any ideas on what needs to be done ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
