Hi Jeff, Yes, the work I am doing is ops work. Logstash is consuming from the topic + consumer group, and I don't want it to start at the beginning, but rather at a specific offset, so setting the offset for the consumer group externally, then starting up logstash is my goal.
I'm still a little unclear as to how to do this. Is this python script: http://pastebin.com/tvxj1wTX The right way to go about getting the offset set to a specific value (12345678 in this example) for a specific consumer group? Regards -- Glen Ogilvie Open Systems Specialists Level 1, 162 Grafton Road http://www.oss.co.nz/ Ph: +64 9 984 3000 Mobile: +64 21 684 146 GPG Key: ACED9C17 From: "Jeff Widman" <j...@netskope.com> To: "users" <users@kafka.apache.org> Sent: Wednesday, 8 March, 2017 1:41:17 PM Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X Offsets for modern kafka consumers are stored in an internal Kafka topic, so they aren't as easy to change as zookeeper. To set a consumer offset, you need a consumer within a consumer group to call commit() with your explicit offset. If needed, you can create a dummy consumer and tell it to join an existing consumer group. Take a look at this migration script, especially the part where it commits the offset to see how it can work in Scala: https://github.com/apache/kafka/pull/2615/files It's fairly straightforward to do this within most other clients as well. If you're doing some quick ops work where you don't want to spin up the JVM, then it's fairly easy to do this using kafka-python. On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> wrote: > Hi, > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out if > I can manually > set a consumer offset, for a specific consumer when that consumer is > stopped. > > It looks like it used to be done using: kafka.tools.ExportZkOffsets and > kafka.tools.ImportZkOffsets > ( https://cwiki.apache.org/confluence/display/KAFKA/ > System+Tools#SystemTools-ConsumerOffsetChecker ) > > However, if my version they don't work, because they try and read from > zookeeper /consumers which is empty.. I think they are old tools. > > Does anyone know where in zookeeper, where the current kafka keeps > consumer offsets? > > Regards > -- > Glen Ogilvie > Open Systems Specialists > Level 1, 162 Grafton Road > http://www.oss.co.nz/ > > Ph: +64 9 984 3000 > Mobile: +64 21 684 146 > GPG Key: ACED9C17 >
