Occasional checkpoint mismatch on Samza task reload

2019-10-31 Thread Malcolm McFarland
Hey folks, We're running Samza 0.14.1 on Hadoop 2.7.6. Every once in a while while restarting an application, the process will come up with some variation on this error: INFO Validating offset for topic and partition [,] WARN While refreshing brokers for [,]: org.apache.kafka.common.errors.Offse

Re: Occasional checkpoint mismatch on Samza task reload

2019-10-31 Thread Bharath Kumara Subramanian
Hi Malcolm, The warning is not particularly related to your restarts. It happens when the offset requested by the consumer no longer exists on the broker. It can typically happen if you are consuming from a topic with TTL enabled and the time retention kicked in and broker purged older offsets. Y