Did you have to do anything different beyond what was already in the Gist?

I'd be curious to see the code as I've considered putting together a small
repo of various python scripts I've found useful when working with Kafka

On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> wrote:

> Thank you Jeff and Robert.
>
> I've had success in getting the offset position to seek to a position in
> the queue, when all other consumers in the group are off.
>
> Would anyone like me to tidy the tool up enough that it could be included
> in Kafka or the docs?
>
> Regards
> --
> Glen Ogilvie
> Open Systems Specialists
> Level 1, 162 Grafton Road
> http://www.oss.co.nz/
>
> Ph: +64 9 984 3000
> Mobile: +64 21 684 146
> GPG Key: ACED9C17
>
>
> From: "Robert Quinlivan" <rquinli...@signal.co>
> To: "users" <users@kafka.apache.org>
> Sent: Thursday, 9 March, 2017 6:42:15 AM
> Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
>
> The best approach would be:
> - Have all consumers in your group shut down
> - Have an offset reset tool join with the same group name as above
> - Offset tool subscribes to all topic-partitions, seeks to the desired
> offset, and commits.
> - Offset tool shuts down
> - Consumers then restart and re-join the consumer group, resuming at the
> offsets that were last committed for each topic-partition
>
> On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <j...@netskope.com> wrote:
>
> > Yeah, that gist looks like it *should* work. I haven't tested it so can't
> > guarantee.
> >
> > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogil...@oss.co.nz>
> > wrote:
> >
> > > Hi Jeff,
> > >
> > > Yes, the work I am doing is ops work. Logstash is consuming from the
> > topic
> > > + consumer group, and I don't want it to start at the beginning, but
> > rather
> > > at a specific offset,
> > > so setting the offset for the consumer group externally, then starting
> up
> > > logstash is my goal.
> > >
> > > I'm still a little unclear as to how to do this.
> > >
> > > Is this python script: http://pastebin.com/tvxj1wTX
> > >
> > > The right way to go about getting the offset set to a specific value
> > > (12345678 in this example) for a specific consumer group?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> > >
> > > From: "Jeff Widman" <j...@netskope.com>
> > > To: "users" <users@kafka.apache.org>
> > > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > >
> > > Offsets for modern kafka consumers are stored in an internal Kafka
> topic,
> > > so they aren't as easy to change as zookeeper.
> > >
> > > To set a consumer offset, you need a consumer within a consumer group
> to
> > > call commit() with your explicit offset. If needed, you can create a
> > dummy
> > > consumer and tell it to join an existing consumer group.
> > >
> > > Take a look at this migration script, especially the part where it
> > commits
> > > the offset to see how it can work in Scala:
> > > https://github.com/apache/kafka/pull/2615/files
> > >
> > > It's fairly straightforward to do this within most other clients as
> well.
> > > If you're doing some quick ops work where you don't want to spin up the
> > > JVM, then it's fairly easy to do this using kafka-python.
> > >
> > >
> > >
> > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <glen.ogil...@oss.co.nz>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure
> out
> > > if
> > > > I can manually
> > > > set a consumer offset, for a specific consumer when that consumer is
> > > > stopped.
> > > >
> > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets
> and
> > > > kafka.tools.ImportZkOffsets
> > > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > > >
> > > > However, if my version they don't work, because they try and read
> from
> > > > zookeeper /consumers which is empty.. I think they are old tools.
> > > >
> > > > Does anyone know where in zookeeper, where the current kafka keeps
> > > > consumer offsets?
> > > >
> > > > Regards
> > > > --
> > > > Glen Ogilvie
> > > > Open Systems Specialists
> > > > Level 1, 162 Grafton Road
> > > > http://www.oss.co.nz/
> > > >
> > > > Ph: +64 9 984 3000
> > > > Mobile: +64 21 684 146
> > > > GPG Key: ACED9C17
> > > >
> > >
> >
>
>
>
> --
> Robert Quinlivan
> Software Engineer, Signal
>

Reply via email to