The best approach would be: - Have all consumers in your group shut down - Have an offset reset tool join with the same group name as above - Offset tool subscribes to all topic-partitions, seeks to the desired offset, and commits. - Offset tool shuts down - Consumers then restart and re-join the consumer group, resuming at the offsets that were last committed for each topic-partition
On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <j...@netskope.com> wrote: > Yeah, that gist looks like it *should* work. I haven't tested it so can't > guarantee. > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > wrote: > > > Hi Jeff, > > > > Yes, the work I am doing is ops work. Logstash is consuming from the > topic > > + consumer group, and I don't want it to start at the beginning, but > rather > > at a specific offset, > > so setting the offset for the consumer group externally, then starting up > > logstash is my goal. > > > > I'm still a little unclear as to how to do this. > > > > Is this python script: http://pastebin.com/tvxj1wTX > > > > The right way to go about getting the offset set to a specific value > > (12345678 in this example) for a specific consumer group? > > > > Regards > > -- > > Glen Ogilvie > > Open Systems Specialists > > Level 1, 162 Grafton Road > > http://www.oss.co.nz/ > > > > Ph: +64 9 984 3000 > > Mobile: +64 21 684 146 > > GPG Key: ACED9C17 > > > > > > From: "Jeff Widman" <j...@netskope.com> > > To: "users" <users@kafka.apache.org> > > Sent: Wednesday, 8 March, 2017 1:41:17 PM > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X > > > > Offsets for modern kafka consumers are stored in an internal Kafka topic, > > so they aren't as easy to change as zookeeper. > > > > To set a consumer offset, you need a consumer within a consumer group to > > call commit() with your explicit offset. If needed, you can create a > dummy > > consumer and tell it to join an existing consumer group. > > > > Take a look at this migration script, especially the part where it > commits > > the offset to see how it can work in Scala: > > https://github.com/apache/kafka/pull/2615/files > > > > It's fairly straightforward to do this within most other clients as well. > > If you're doing some quick ops work where you don't want to spin up the > > JVM, then it's fairly easy to do this using kafka-python. > > > > > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > > wrote: > > > > > Hi, > > > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out > > if > > > I can manually > > > set a consumer offset, for a specific consumer when that consumer is > > > stopped. > > > > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets and > > > kafka.tools.ImportZkOffsets > > > ( https://cwiki.apache.org/confluence/display/KAFKA/ > > > System+Tools#SystemTools-ConsumerOffsetChecker ) > > > > > > However, if my version they don't work, because they try and read from > > > zookeeper /consumers which is empty.. I think they are old tools. > > > > > > Does anyone know where in zookeeper, where the current kafka keeps > > > consumer offsets? > > > > > > Regards > > > -- > > > Glen Ogilvie > > > Open Systems Specialists > > > Level 1, 162 Grafton Road > > > http://www.oss.co.nz/ > > > > > > Ph: +64 9 984 3000 > > > Mobile: +64 21 684 146 > > > GPG Key: ACED9C17 > > > > > > -- Robert Quinlivan Software Engineer, Signal
