Did you have to do anything different beyond what was already in the Gist? I'd be curious to see the code as I've considered putting together a small repo of various python scripts I've found useful when working with Kafka
On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> wrote: > Thank you Jeff and Robert. > > I've had success in getting the offset position to seek to a position in > the queue, when all other consumers in the group are off. > > Would anyone like me to tidy the tool up enough that it could be included > in Kafka or the docs? > > Regards > -- > Glen Ogilvie > Open Systems Specialists > Level 1, 162 Grafton Road > http://www.oss.co.nz/ > > Ph: +64 9 984 3000 > Mobile: +64 21 684 146 > GPG Key: ACED9C17 > > > From: "Robert Quinlivan" <rquinli...@signal.co> > To: "users" <users@kafka.apache.org> > Sent: Thursday, 9 March, 2017 6:42:15 AM > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X > > The best approach would be: > - Have all consumers in your group shut down > - Have an offset reset tool join with the same group name as above > - Offset tool subscribes to all topic-partitions, seeks to the desired > offset, and commits. > - Offset tool shuts down > - Consumers then restart and re-join the consumer group, resuming at the > offsets that were last committed for each topic-partition > > On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <j...@netskope.com> wrote: > > > Yeah, that gist looks like it *should* work. I haven't tested it so can't > > guarantee. > > > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > > wrote: > > > > > Hi Jeff, > > > > > > Yes, the work I am doing is ops work. Logstash is consuming from the > > topic > > > + consumer group, and I don't want it to start at the beginning, but > > rather > > > at a specific offset, > > > so setting the offset for the consumer group externally, then starting > up > > > logstash is my goal. > > > > > > I'm still a little unclear as to how to do this. > > > > > > Is this python script: http://pastebin.com/tvxj1wTX > > > > > > The right way to go about getting the offset set to a specific value > > > (12345678 in this example) for a specific consumer group? > > > > > > Regards > > > -- > > > Glen Ogilvie > > > Open Systems Specialists > > > Level 1, 162 Grafton Road > > > http://www.oss.co.nz/ > > > > > > Ph: +64 9 984 3000 > > > Mobile: +64 21 684 146 > > > GPG Key: ACED9C17 > > > > > > > > > From: "Jeff Widman" <j...@netskope.com> > > > To: "users" <users@kafka.apache.org> > > > Sent: Wednesday, 8 March, 2017 1:41:17 PM > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X > > > > > > Offsets for modern kafka consumers are stored in an internal Kafka > topic, > > > so they aren't as easy to change as zookeeper. > > > > > > To set a consumer offset, you need a consumer within a consumer group > to > > > call commit() with your explicit offset. If needed, you can create a > > dummy > > > consumer and tell it to join an existing consumer group. > > > > > > Take a look at this migration script, especially the part where it > > commits > > > the offset to see how it can work in Scala: > > > https://github.com/apache/kafka/pull/2615/files > > > > > > It's fairly straightforward to do this within most other clients as > well. > > > If you're doing some quick ops work where you don't want to spin up the > > > JVM, then it's fairly easy to do this using kafka-python. > > > > > > > > > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > > > wrote: > > > > > > > Hi, > > > > > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure > out > > > if > > > > I can manually > > > > set a consumer offset, for a specific consumer when that consumer is > > > > stopped. > > > > > > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets > and > > > > kafka.tools.ImportZkOffsets > > > > ( https://cwiki.apache.org/confluence/display/KAFKA/ > > > > System+Tools#SystemTools-ConsumerOffsetChecker ) > > > > > > > > However, if my version they don't work, because they try and read > from > > > > zookeeper /consumers which is empty.. I think they are old tools. > > > > > > > > Does anyone know where in zookeeper, where the current kafka keeps > > > > consumer offsets? > > > > > > > > Regards > > > > -- > > > > Glen Ogilvie > > > > Open Systems Specialists > > > > Level 1, 162 Grafton Road > > > > http://www.oss.co.nz/ > > > > > > > > Ph: +64 9 984 3000 > > > > Mobile: +64 21 684 146 > > > > GPG Key: ACED9C17 > > > > > > > > > > > > > -- > Robert Quinlivan > Software Engineer, Signal >
