Hmm... gist didn't come thru for some reason.Trying agin:
gist.github.com/gzoller/93fe2392fd3606bcb3b879e4ab2f8f6e
From: Greg Zoller <[email protected]>
To: "[email protected]" <[email protected]>
Sent: Thursday, April 7, 2016 5:25 PM
Subject: [0.9.0.1] Having problems with commitAsync() not updating offsets
Hello,
I'm having trouble with commitAsync() not updating my partition offsets. I
pre-populate 1,000,000 strings on topic 'lowercaseStrings' across 4 partitions.
I check the offsets and they're fine--250K records/partition.
Then I run some code that uses a single-thread with blocking queue to
communicate with the (multi-threaded) outside world.
This code does a poll and populates an iterator, which is bled dry by the
caller before another is requested.
For now I individually commit each and every ConsumerRecord I process until I
understand how this works... I'll figure out batched commits later.
I can confirm that I pull and process 1 million individual records. I can also
confirm my OffsetCommitCallback is called 1 million times, 250K times for each
of my 4 partitions. Exception is null in the callback for each of these times.
Everything appears in order
Now when I look at the partition offsets I see they're all over the place,
typically with two having around 16K lag behind head offset, and the others a
scattered few lag behind offset. That's not good.
For grins I added long delays after my test run (before shutting everything
down), to let things settle down--thinking maybe something still had to catch
up, but this didn't change behavior. I'm also perplexed why the callback's
numbers all appeared as expected but not the offsets. I would have expected
these to match (i.e. callback not called until actual commit happens).
Below is a gist of my consumer thread. Nothing fancy.
Any thoughts or ideas appreciated.Greg
Kafka 0.9.0.1 Consumer
|
|
|
| | |
|
|
|
| |
Kafka 0.9.0.1 Consumer
Kafka 0.9.0.1 Consumer | |
|
|
