You shouldn't need to seek at all. When you commit offsets for a partition,
they are stored in Kafka and become available to any member which is
assigned that partition. The default behavior after every rebalance is to
lookup the last committed offset for each assigned position and seek to
it. This
So would I seekToEnd in that case? I can't seek to the last commit I
manually comitted since I may not be assigned the same partitions as before.
On Wed, Feb 24, 2016 at 1:44 PM, Jason Gustafson wrote:
> Sure, but in that case, the commits are still being stored in Kafka, so
> resetting to the l
Sure, but in that case, the commits are still being stored in Kafka, so
resetting to the last committed position seems like what you want.
-Jason
On Wed, Feb 24, 2016 at 10:42 AM, craig w wrote:
> In this case I am managing offsets manually. When a message is done being
> processed, I invoke "c
In this case I am managing offsets manually. When a message is done being
processed, I invoke "commitSync" passing it the map of commits to sync.
On Wed, Feb 24, 2016 at 1:35 PM, Jason Gustafson wrote:
> I think the problem is the call to position() from within the callback.
> When onAssigned()
I think the problem is the call to position() from within the callback.
When onAssigned() gets invoked, we don't have a position yet, so calling
position() forces the consumer to request the last committed offset which
evidently causes an infinite loop. It might be worth opening a JIRA for
this sin
I am using the Java Kafka 0.9 client. When I subscribe to a topic I provide
a ConsumerRebalanceListener. In the "onPartitionsAssigned" method I am
doing this:
partitions.foreach( (tp: TopicPartition) => {
consumer.seek(tp, consumer.position(tp))
})
However, sometimes I end up an infinite loop
