A tool for reseting consumer group offsets is proposed here: KIP-122: https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
KIP is currently in the voting stage. On Thu, Mar 9, 2017 at 7:35 AM, Jeff Widman <j...@netskope.com> wrote: > Did you have to do anything different beyond what was already in the Gist? > > I'd be curious to see the code as I've considered putting together a small > repo of various python scripts I've found useful when working with Kafka > > On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > wrote: > > > Thank you Jeff and Robert. > > > > I've had success in getting the offset position to seek to a position in > > the queue, when all other consumers in the group are off. > > > > Would anyone like me to tidy the tool up enough that it could be included > > in Kafka or the docs? > > > > Regards > > -- > > Glen Ogilvie > > Open Systems Specialists > > Level 1, 162 Grafton Road > > http://www.oss.co.nz/ > > > > Ph: +64 9 984 3000 > > Mobile: +64 21 684 146 > > GPG Key: ACED9C17 > > > > > > From: "Robert Quinlivan" <rquinli...@signal.co> > > To: "users" <users@kafka.apache.org> > > Sent: Thursday, 9 March, 2017 6:42:15 AM > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X > > > > The best approach would be: > > - Have all consumers in your group shut down > > - Have an offset reset tool join with the same group name as above > > - Offset tool subscribes to all topic-partitions, seeks to the desired > > offset, and commits. > > - Offset tool shuts down > > - Consumers then restart and re-join the consumer group, resuming at the > > offsets that were last committed for each topic-partition > > > > On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <j...@netskope.com> wrote: > > > > > Yeah, that gist looks like it *should* work. I haven't tested it so > can't > > > guarantee. > > > > > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > > > wrote: > > > > > > > Hi Jeff, > > > > > > > > Yes, the work I am doing is ops work. Logstash is consuming from the > > > topic > > > > + consumer group, and I don't want it to start at the beginning, but > > > rather > > > > at a specific offset, > > > > so setting the offset for the consumer group externally, then > starting > > up > > > > logstash is my goal. > > > > > > > > I'm still a little unclear as to how to do this. > > > > > > > > Is this python script: http://pastebin.com/tvxj1wTX > > > > > > > > The right way to go about getting the offset set to a specific value > > > > (12345678 in this example) for a specific consumer group? > > > > > > > > Regards > > > > -- > > > > Glen Ogilvie > > > > Open Systems Specialists > > > > Level 1, 162 Grafton Road > > > > http://www.oss.co.nz/ > > > > > > > > Ph: +64 9 984 3000 > > > > Mobile: +64 21 684 146 > > > > GPG Key: ACED9C17 > > > > > > > > > > > > From: "Jeff Widman" <j...@netskope.com> > > > > To: "users" <users@kafka.apache.org> > > > > Sent: Wednesday, 8 March, 2017 1:41:17 PM > > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X > > > > > > > > Offsets for modern kafka consumers are stored in an internal Kafka > > topic, > > > > so they aren't as easy to change as zookeeper. > > > > > > > > To set a consumer offset, you need a consumer within a consumer group > > to > > > > call commit() with your explicit offset. If needed, you can create a > > > dummy > > > > consumer and tell it to join an existing consumer group. > > > > > > > > Take a look at this migration script, especially the part where it > > > commits > > > > the offset to see how it can work in Scala: > > > > https://github.com/apache/kafka/pull/2615/files > > > > > > > > It's fairly straightforward to do this within most other clients as > > well. > > > > If you're doing some quick ops work where you don't want to spin up > the > > > > JVM, then it's fairly easy to do this using kafka-python. > > > > > > > > > > > > > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <glen.ogil...@oss.co.nz > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure > > out > > > > if > > > > > I can manually > > > > > set a consumer offset, for a specific consumer when that consumer > is > > > > > stopped. > > > > > > > > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets > > and > > > > > kafka.tools.ImportZkOffsets > > > > > ( https://cwiki.apache.org/confluence/display/KAFKA/ > > > > > System+Tools#SystemTools-ConsumerOffsetChecker ) > > > > > > > > > > However, if my version they don't work, because they try and read > > from > > > > > zookeeper /consumers which is empty.. I think they are old tools. > > > > > > > > > > Does anyone know where in zookeeper, where the current kafka keeps > > > > > consumer offsets? > > > > > > > > > > Regards > > > > > -- > > > > > Glen Ogilvie > > > > > Open Systems Specialists > > > > > Level 1, 162 Grafton Road > > > > > http://www.oss.co.nz/ > > > > > > > > > > Ph: +64 9 984 3000 > > > > > Mobile: +64 21 684 146 > > > > > GPG Key: ACED9C17 > > > > > > > > > > > > > > > > > > > > -- > > Robert Quinlivan > > Software Engineer, Signal > > >
