Thank you Jeff and Robert. I've had success in getting the offset position to seek to a position in the queue, when all other consumers in the group are off.
Would anyone like me to tidy the tool up enough that it could be included in Kafka or the docs? Regards -- Glen Ogilvie Open Systems Specialists Level 1, 162 Grafton Road http://www.oss.co.nz/ Ph: +64 9 984 3000 Mobile: +64 21 684 146 GPG Key: ACED9C17 From: "Robert Quinlivan" <rquinli...@signal.co> To: "users" <users@kafka.apache.org> Sent: Thursday, 9 March, 2017 6:42:15 AM Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X The best approach would be: - Have all consumers in your group shut down - Have an offset reset tool join with the same group name as above - Offset tool subscribes to all topic-partitions, seeks to the desired offset, and commits. - Offset tool shuts down - Consumers then restart and re-join the consumer group, resuming at the offsets that were last committed for each topic-partition On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <j...@netskope.com> wrote: > Yeah, that gist looks like it *should* work. I haven't tested it so can't > guarantee. > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > wrote: > > > Hi Jeff, > > > > Yes, the work I am doing is ops work. Logstash is consuming from the > topic > > + consumer group, and I don't want it to start at the beginning, but > rather > > at a specific offset, > > so setting the offset for the consumer group externally, then starting up > > logstash is my goal. > > > > I'm still a little unclear as to how to do this. > > > > Is this python script: http://pastebin.com/tvxj1wTX > > > > The right way to go about getting the offset set to a specific value > > (12345678 in this example) for a specific consumer group? > > > > Regards > > -- > > Glen Ogilvie > > Open Systems Specialists > > Level 1, 162 Grafton Road > > http://www.oss.co.nz/ > > > > Ph: +64 9 984 3000 > > Mobile: +64 21 684 146 > > GPG Key: ACED9C17 > > > > > > From: "Jeff Widman" <j...@netskope.com> > > To: "users" <users@kafka.apache.org> > > Sent: Wednesday, 8 March, 2017 1:41:17 PM > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X > > > > Offsets for modern kafka consumers are stored in an internal Kafka topic, > > so they aren't as easy to change as zookeeper. > > > > To set a consumer offset, you need a consumer within a consumer group to > > call commit() with your explicit offset. If needed, you can create a > dummy > > consumer and tell it to join an existing consumer group. > > > > Take a look at this migration script, especially the part where it > commits > > the offset to see how it can work in Scala: > > https://github.com/apache/kafka/pull/2615/files > > > > It's fairly straightforward to do this within most other clients as well. > > If you're doing some quick ops work where you don't want to spin up the > > JVM, then it's fairly easy to do this using kafka-python. > > > > > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > > wrote: > > > > > Hi, > > > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out > > if > > > I can manually > > > set a consumer offset, for a specific consumer when that consumer is > > > stopped. > > > > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets and > > > kafka.tools.ImportZkOffsets > > > ( https://cwiki.apache.org/confluence/display/KAFKA/ > > > System+Tools#SystemTools-ConsumerOffsetChecker ) > > > > > > However, if my version they don't work, because they try and read from > > > zookeeper /consumers which is empty.. I think they are old tools. > > > > > > Does anyone know where in zookeeper, where the current kafka keeps > > > consumer offsets? > > > > > > Regards > > > -- > > > Glen Ogilvie > > > Open Systems Specialists > > > Level 1, 162 Grafton Road > > > http://www.oss.co.nz/ > > > > > > Ph: +64 9 984 3000 > > > Mobile: +64 21 684 146 > > > GPG Key: ACED9C17 > > > > > > -- Robert Quinlivan Software Engineer, Signal
