Hmm, I guess you're right Tod :) Just to confirm, you meant that, while you're changing the exported file it might happen that one of the segment files becomes eligible for cleanup by retention, which would then make the imported offsets out of range?
Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support Sematext <http://sematext.com/> | Contact <http://sematext.com/about/contact.html> On Fri, Dec 18, 2015 at 6:29 PM, Todd Palino <tpal...@gmail.com> wrote: > That works if you want to set to an arbitrary offset, Marko. However in the > case the OP described, wanting to reset to smallest, it is better to just > delete the consumer group and start the consumer with auto.offset.reset set > to smallest. The reason is that while you can pull the current smallest > offsets from the brokers and set them in Zookeeper for the consumer, by the > time you do that the smallest offset is likely no longer valid. This means > you’re going to resort to the offset reset logic anyways. > > -Todd > > > On Fri, Dec 18, 2015 at 7:10 AM, Marko Bonaći <marko.bon...@sematext.com> > wrote: > > > You can also do this: > > 1. stop consumers > > 2. export offsets from ZK > > 3. make changes to the exported file > > 4. import offsets to ZK > > 5. start consumers > > > > e.g. > > bin/kafka-run-class.sh kafka.tools.ExportZkOffsets --group group-name > > --output-file /tmp/zk-offsets --zkconnect localhost:2181 > > bin/kafka-run-class.sh kafka.tools.ImportZkOffsets --input-file > > /tmp/zk-offsets --zkconnect localhost:2181 > > > > Marko Bonaći > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > > Solr & Elasticsearch Support > > Sematext <http://sematext.com/> | Contact > > <http://sematext.com/about/contact.html> > > > > On Fri, Dec 18, 2015 at 4:06 PM, Jens Rantil <jens.ran...@tink.se> > wrote: > > > > > Hi, > > > > > > I noticed that a consumer in the new consumer API supports setting the > > > offset for a partition to beginning. I assume doing so also would > update > > > the offset in Zookeeper eventually. > > > > > > Cheers, > > > Jens > > > > > > On Friday, December 18, 2015, Akhilesh Pathodia < > > > pathodia.akhil...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > > > > > I want to reset the kafka offset in zookeeper so that the consumer > will > > > > start reading messages from first offset. I am using flume as a > > consumer > > > to > > > > kafka. I have set the kafka property kafka.auto.offset.reset to > > > "smallest", > > > > but it does not reset the offset in zookeeper and that's why flume > will > > > not > > > > read messages from first offset. > > > > > > > > Is there any way to reset kafka offset in zookeeper? > > > > > > > > Thanks, > > > > Akhilesh > > > > > > > > > > > > > -- > > > Jens Rantil > > > Backend engineer > > > Tink AB > > > > > > Email: jens.ran...@tink.se > > > Phone: +46 708 84 18 32 > > > Web: www.tink.se > > > > > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin > > > < > > > > > > http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary > > > > > > > Twitter <https://twitter.com/tink> > > > > > > > > > -- > *—-* > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
