Yeah, that gist looks like it *should* work. I haven't tested it so can't guarantee.
On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> wrote: > Hi Jeff, > > Yes, the work I am doing is ops work. Logstash is consuming from the topic > + consumer group, and I don't want it to start at the beginning, but rather > at a specific offset, > so setting the offset for the consumer group externally, then starting up > logstash is my goal. > > I'm still a little unclear as to how to do this. > > Is this python script: http://pastebin.com/tvxj1wTX > > The right way to go about getting the offset set to a specific value > (12345678 in this example) for a specific consumer group? > > Regards > -- > Glen Ogilvie > Open Systems Specialists > Level 1, 162 Grafton Road > http://www.oss.co.nz/ > > Ph: +64 9 984 3000 > Mobile: +64 21 684 146 > GPG Key: ACED9C17 > > > From: "Jeff Widman" <j...@netskope.com> > To: "users" <users@kafka.apache.org> > Sent: Wednesday, 8 March, 2017 1:41:17 PM > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X > > Offsets for modern kafka consumers are stored in an internal Kafka topic, > so they aren't as easy to change as zookeeper. > > To set a consumer offset, you need a consumer within a consumer group to > call commit() with your explicit offset. If needed, you can create a dummy > consumer and tell it to join an existing consumer group. > > Take a look at this migration script, especially the part where it commits > the offset to see how it can work in Scala: > https://github.com/apache/kafka/pull/2615/files > > It's fairly straightforward to do this within most other clients as well. > If you're doing some quick ops work where you don't want to spin up the > JVM, then it's fairly easy to do this using kafka-python. > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > wrote: > > > Hi, > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out > if > > I can manually > > set a consumer offset, for a specific consumer when that consumer is > > stopped. > > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets and > > kafka.tools.ImportZkOffsets > > ( https://cwiki.apache.org/confluence/display/KAFKA/ > > System+Tools#SystemTools-ConsumerOffsetChecker ) > > > > However, if my version they don't work, because they try and read from > > zookeeper /consumers which is empty.. I think they are old tools. > > > > Does anyone know where in zookeeper, where the current kafka keeps > > consumer offsets? > > > > Regards > > -- > > Glen Ogilvie > > Open Systems Specialists > > Level 1, 162 Grafton Road > > http://www.oss.co.nz/ > > > > Ph: +64 9 984 3000 > > Mobile: +64 21 684 146 > > GPG Key: ACED9C17 > > >
