Re: kafka manual commit vs auto commit

2018-05-24 Thread Shantanu Deshmukh
So here, when I received a message I run some business logic on it and try to send some email. Now sometimes we have a promotional campaign running millions of emails need to be delivered. For such numerous events is manual commit good? Will it generate too much network activity if I commit a singl

Re: kafka manual commit vs auto commit

2018-05-24 Thread M. Manna
Manual commit is important where event consumption eventually leads to some post-processing/database update/state change for your application. Without doing all those, you cannot truly say that you have "Received" the message. "Receiving" is interpreted differently and it's up to your target applic

kafka manual commit vs auto commit

2018-05-24 Thread Shantanu Deshmukh
Hello everyone, We have a 3 broker Kafka 0.10.1.0 cluster in production environment. Lately we are seeing a lot of "auto commit failed because poll() spend too much time processing" warning messages. Also, due to such events there is constant fear of duplicate messages and the same does happen. To