Re: Rebalance events when auto commit is disabled in 0.9 consumer

2016-04-26 Thread Liquan Pei
Hi Balthasar, It is possible that after rebalance, the topic partitions assigned to a consumer instance are changed. To prevent state loss, a custom ConsumerRebalanceListener is needed to properly save the state. In the onPartitionRevoked method, you need to trigger the write to database and wait

Rebalance events when auto commit is disabled in 0.9 consumer

2016-04-26 Thread Balthasar Schopman
Hi, Per consumer group we have many consumers that aggregate data for a predefined amount of time and then write the aggregated data to a db. We're looking to set enable.auto.commit to false and write the offsets only when the aggregated data has been written to db. In that context we're not su