Re: Consumer hangs at poll() and never throw exception on client

2019-12-15 Thread Frank Zhou
Hi, You mean commitTransaction() or abortTransaction()? If so, then yes, either 1 will be called for each transaction. On Mon, Dec 16, 2019 at 12:46 PM Eric Azama wrote: > Consumers are unable to make any progress if the next records to be sent > are part of an open transaction. Are your produc

Re: Consumer hangs at poll() and never throw exception on client

2019-12-15 Thread Eric Azama
Consumers are unable to make any progress if the next records to be sent are part of an open transaction. Are your producers properly closing transactions? On Sun, Dec 15, 2019 at 7:49 PM Frank Zhou wrote: > More details, the first application's consumer has been closed and its > producers are r

Re: Consumer hangs at poll() and never throw exception on client

2019-12-15 Thread Frank Zhou
More details, the first application's consumer has been closed and its producers are running already when the second application starts, then second consumer hangs forever if transaction is enabled on producers. On Mon, Dec 16, 2019 at 11:43 AM Frank Zhou wrote: > Hi Matthias, > > Seems the issu

Re: Consumer hangs at poll() and never throw exception on client

2019-12-15 Thread Frank Zhou
Hi Matthias, Seems the issue is related to transaction. Let me expand a bit on our flow of this. Our application will use single consumer to fetch the records, deal with the data and then multiple producers to send data afterwards. There could be multiple applications sending data to a single topi

Re: Consumer hangs at poll() and never throw exception on client

2019-12-10 Thread Matthias J. Sax
Hard to say why the second consumer does not fetch data. I would try to inspect the logs at DEBUG level to get more insight what going on. -Matthias On 12/10/19 12:27 AM, Frank Zhou wrote: > "earliest", as we need everything from the start > > On Tue, Dec 10, 2019 at 4:22 PM Matthias J. Sax

Re: Consumer hangs at poll() and never throw exception on client

2019-12-10 Thread Frank Zhou
"earliest", as we need everything from the start On Tue, Dec 10, 2019 at 4:22 PM Matthias J. Sax wrote: > What is your `auto.offset.reset` policy? > > On 12/10/19 12:02 AM, Frank Zhou wrote: > > Hi, > > > > Thanks for your prompt reply. Consumers have different group.id in our > > case, as we j

Re: Consumer hangs at poll() and never throw exception on client

2019-12-10 Thread Matthias J. Sax
What is your `auto.offset.reset` policy? On 12/10/19 12:02 AM, Frank Zhou wrote: > Hi, > > Thanks for your prompt reply. Consumers have different group.id in our > case, as we just need all the data in the topic, so every time the > application starts, the consumer will be in a new group. For cur

Re: Consumer hangs at poll() and never throw exception on client

2019-12-10 Thread Frank Zhou
Hi, Thanks for your prompt reply. Consumers have different group.id in our case, as we just need all the data in the topic, so every time the application starts, the consumer will be in a new group. For current testing environment, we just one partition for one topic. though we have tried with mul

Re: Consumer hangs at poll() and never throw exception on client

2019-12-09 Thread Matthias J. Sax
Are both consumers in the same consumer group, ie, use the same `group.id`? If yes, how many partitions does the topic have? If it has only one partition, the observed behavior is expected, because a single partitions can only be read by a single consumer instance per consumer group. The second co

Consumer hangs at poll() and never throw exception on client

2019-12-09 Thread Frank Zhou
Hi, I am quite new to Kafka, and I have encountered a weird case during QA stage for our application. Now we have 2 consumers consuming same topic in kafka cluster. The first started consumer works fine and get closed after getting all the messages. After that, the second one started and just hang