You have to perform seeks (using the consumer) to the lowest unprocessed offset
for each partition returned by the poll, before the next poll.
________________________________
From: Peter Cipov <pci...@twilio.com.INVALID>
Sent: Thursday, February 18, 2021 1:20 PM
To: users@kafka.apache.org <users@kafka.apache.org>
Subject: Abort transaction semantics
Hello
I have a question regarding aborting transactions in kafka client 2.4.1.
lets have following code :
... propper transaction producer consumer creation, consumer autocommit =
false
producer.transactionInit();
while(true) {
records = consumer.poll();
logRecordOffsets(records)
producer.beginTransaction()
try {
doMagic()
} catch{
producer.AbortTransaction();
continue;
}
producer.sendOffsets(..);
producer.commitTransaction()
}
When doMagic crashes for some reason, abort is called and code will start
from beginning with doing poll.
Our assumption was that the next poll will start from the same offsets, but
as we saw from logs this is not the case. What we observed that offsets are
shifted and messages are lost, they will not be retried again.
What is the semantics for abort, we could not figure out from
documentation.
What is the recommended approach for retrying ?
Thank you
