We've been playing around with the new Consumer API and have it an unfortunate bump in the road. When our onPartitionsRevoked() callback is called we'd like to be able to commit any data that we were processing to stable storage so we can then commit the offsets back to Kafka. This way we don't throw away in progress work. The problem is that onPartitionsRevoked() is called from the same thread running poll() which means that heartbeats are paused while the callback is processing. Our commits sometimes take a few minutes which means that we'll lose our Kafka session and the partitions we were trying to commit will get reassigned. The end result of that is more duplicates in our stored data.
Has anyone else encountered this? We're probably going to just do a rollback rather than a commit in the callback to return quickly, but I wanted to check to see if there was something we were missing. -Joey
