Hi, Thanks for the lucid explanation. So say if I run this bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.73.198:9092 --topic test-stream --time -1 --offsets 6 I get output like: test-stream:0:81658712,81616045,81299569,80987584,80673168,80364832
This means these are the offset of latest message on these segment files. So the current segment is holding (81658712 - 81616045) messages = 42667 message Previous one had (81616045 - 81299569) = 316476 messages Previous than that (81299569 - 80987584) = 311985 messages and so on... and when I run bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.73.198:9092 --topic test-stream --time -2 I get output like test-advice-stream:0:69258918 This means that earliest message offset in the segment file is 69258918 and messages before that have been deleted. I guess my understating is correct. Thanks Sachin On Sun, Dec 18, 2016 at 6:25 AM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > The tool writes output in the format: > > <topic>:<partition>:<comma separated list of valid offsets> > > So in the case of your example with --time -1 that returned > test-window-stream:0:724, it is saying that test-window-stream has > partition 0 with a valid log segment which has the first offset = 724. Note > that --time -1 is a special code for "only give the latest segment". For > the -2 option you saw 0:0 because the earliest segment is still currently > the first segment created for that topic, i.e. the one starting with offset > 0 (because your retention policy hasn't kicked in yet). In cases where it > returned multiple comma separated values, those are multiple starting > offsets for segments containing offsets before the timestamp you requested. > > Note that this is using the old simple consumer approach to querying for > offsets. That implementation could only tell you about offsets based on > entire log segments. If you're on a newer version of Kafka, you may be > interested in the time-based offset query, which you can access in the new > consumer API and gives you better granularity ( > http://kafka.apache.org/0101/ > javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html# > offsetsForTimes(java.util.Map)). > > -Ewen > > On Thu, Dec 15, 2016 at 3:52 AM, Sachin Mittal <sjmit...@gmail.com> wrote: > > > Folks any explanation for this. Or any link that can help me on that. > > > > On Tue, Dec 13, 2016 at 1:00 PM, Sachin Mittal <sjmit...@gmail.com> > wrote: > > > > > Hi, > > > I have some trouble interpreting the result of GetOffsetShell command. > > > > > > Say if I run > > > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell > --broker-list > > > localhost:9092 --topic test-window-stream --time -2 > > > test-window-stream:0:0 > > > > > > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat > > > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic > > > test-window-stream --time -1 > > > test-window-stream:0:724 > > > > > > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell > --broker-list > > > localhost:9092 --topic test-window-stream > > > test-window-stream:0:724 > > > > > > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat > > > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic > > > test-window-stream --offset 2 > > > test-window-stream:0:724,0 > > > > > > So what does 0:0 mean or for that mater 0:724 > > > > > > Also when I use -2 (earliest), why does it show 0:0, does this means > > > offset if at the start of the topic. > > > Also when I use -1 (latest), it shows 0:724, does it means none of the > > > records are processed. > > > However I was having a stream application running which did aggregate > > > atleast upto 720 records before I killed it. So why does it not show > > here? > > > > > > Also it is not clear what --offset flag means and when I pass something > > > greater than 1 I get ,0 appended to the earlier result. > > > > > > Thanks > > > Sachin > > > > > > > > >
