Hi Gaurav (and others in this thread),
My apologies for the late reply, This e-mail appears to have missed my
inbox somehow.
>From your logs, It appears that there was a leadership change happening on
the Kafka side for this topic partition? If so, I would actually expect the
follower's offset to
Some more logs from Kakja
WARN [2017-05-01 15:21:19,132]
kafka.server.ReplicaFetcherThread:[Logging$class:warn:83] -
[ReplicaFetcherThread-0-3] - [ReplicaFetcherThread-0-3], Replica 0 for
partition [Topic3,17] reset its fetch offset from 45039137 to current
leader 3's latest offset 45039132
INFO
Looking further, the reason for this "jump back" seems not so straight
forward:
In Kafka's Simple Consumer code:
private def sendRequest(request: RequestOrResponse): NetworkReceive = {
lock synchronized {
var response: NetworkReceive = null
try {
getOrMakeConnection()
blockin
This also seems somewhat related to the mail on this group a few days back
- with subject 'Messages lost after broker failure'.
If someone had set auto.offset.reset to largest, then reverse would happen
- i.e samza skipping over kafka partition queue in face of such failures.
On Tue, May 2, 2017
Hi All,
We recently observed an issue in our Samza application (version
0.12.0) where we found that that the message offsets "jumped back" causing
many of the messages to be re-processed.
On digging deeper - here is what we found:
- There was a some network related issues at that time causing t
