Hi Matthias, Seems the issue is related to transaction. Let me expand a bit on our flow of this. Our application will use single consumer to fetch the records, deal with the data and then multiple producers to send data afterwards. There could be multiple applications sending data to a single topic as well. So the above issue happens when transactions are enabled when multiple applications are running(Transaction ids are different among producers and applications). Today I tested with transaction disabled, and everything seems fine. Do you see any issue for our usage on transactions?
On Tue, Dec 10, 2019 at 4:42 PM Matthias J. Sax <matth...@confluent.io> wrote: > Hard to say why the second consumer does not fetch data. > > I would try to inspect the logs at DEBUG level to get more insight what > going on. > > > -Matthias > > On 12/10/19 12:27 AM, Frank Zhou wrote: > > "earliest", as we need everything from the start > > > > On Tue, Dec 10, 2019 at 4:22 PM Matthias J. Sax <matth...@confluent.io> > > wrote: > > > >> What is your `auto.offset.reset` policy? > >> > >> On 12/10/19 12:02 AM, Frank Zhou wrote: > >>> Hi, > >>> > >>> Thanks for your prompt reply. Consumers have different group.id in our > >>> case, as we just need all the data in the topic, so every time the > >>> application starts, the consumer will be in a new group. For current > >>> testing environment, we just one partition for one topic. though we > have > >>> tried with multiple partitions, and situation happened as well. > >>> > >>> On Tue, Dec 10, 2019 at 3:56 PM Matthias J. Sax <matth...@confluent.io > > > >>> wrote: > >>> > >>>> Are both consumers in the same consumer group, ie, use the same ` > >> group.id > >>>> `? > >>>> > >>>> If yes, how many partitions does the topic have? If it has only one > >>>> partition, the observed behavior is expected, because a single > >>>> partitions can only be read by a single consumer instance per consumer > >>>> group. The second consumer won't have any assigned partitions to read > >> from. > >>>> > >>>> -Matthias > >>>> > >>>> On 12/9/19 11:12 PM, Frank Zhou wrote: > >>>>> Hi, > >>>>> > >>>>> I am quite new to Kafka, and I have encountered a weird case during > QA > >>>>> stage for our application. Now we have 2 consumers consuming same > topic > >>>>> in kafka cluster. The first started consumer works fine and get > closed > >>>>> after getting all the messages. After that, the second one started > and > >>>>> just hang there in the poll(), which we set the Duration to 10 > seconds, > >>>>> and will quit the loop if no records are fetched. From kafka server > >> log, > >>>>> I don't see warning or error for the second consumer. > >>>>> > >>>>> Attached client/server log related to both consumers. I am using > 2.3.1 > >>>>> client, 2.2.2 kafka servers. Looking forward to you feedback. > >>>>> > >>>>> > >>>>> Best regards, > >>>>> -- > >>>>> *Frank Zhou* > >>>>> R&D, Itiviti > >>>>> Java Developer > >>>>> D +852 2521 7480 > >>>>> frank.z...@itiviti.com <mailto:frank.z...@itiviti.com> > >>>>> > >>>>> ______________________________ > >>>>> > >>>>> itiviti.com <https://www.itiviti.com/> > >>>>> > >>>>> /The information contained in or attached to this email is strictly > >>>>> confidential. If you are not the intended recipient, please notify us > >>>>> immediately by telephone and return the message to us./ > >>>>> > >>>>> /Email communications by definition contain personal information. The > >>>>> Itiviti group of companies is subject to European data protection > >>>>> regulations. Itiviti’s Privacy Notice is available at > www.itiviti.com > >>>>> <http://www.itiviti.com/>. Itiviti expects the recipient of this > email > >>>>> to be compliant with Itiviti’s Privacy Notice and applicable > >>>>> regulations. Please advise us immediately at > >>>>> dataprotectiont...@itiviti.com if you are not compliant with these./ > >>>>> > >>>>> > >>>>> ______________________________ > >>>>> > >>>>> itiviti.com <https://www.itiviti.com/> > >>>>> <https://www.linkedin.com/company/itiviti> ^Follow Itiviti on > Linkedin > >>>>> <https://www.linkedin.com/company/10438325/> > >>>>> > >>>>> The information contained in or attached to this email is strictly > >>>>> confidential. If you are not the intended recipient, please notify us > >>>>> immediately by telephone and return the message to us. Email > >>>>> communications by definition contain personal information. The > Itiviti > >>>>> group of companies is subject to European data protection > regulations. > >>>>> > >>>>> Itiviti’s Privacy Notice is available at www.itiviti.com > >>>>> <http://www.itiviti.com/>. Itiviti expects the recipient of this > email > >>>>> to be compliant with Itiviti’s Privacy Notice and applicable > >>>>> regulations. Please advise us immediately at > >>>>> dataprotectiont...@itiviti.com if you are not compliant with these. > >>>>> > >>>> > >>>> > >>> > >> > >> > > > > -- *Frank Zhou* R&D, Itiviti Java Developer D +852 2521 7480 frank.z...@itiviti.com ______________________________ itiviti.com <https://www.itiviti.com/> *The information contained in or attached to this email is strictly confidential. If you are not the intended recipient, please notify us immediately by telephone and return the message to us.* *Email communications by definition contain personal information. The Itiviti group of companies is subject to European data protection regulations. Itiviti’s Privacy Notice is available at www.itiviti.com <http://www.itiviti.com/>. Itiviti expects the recipient of this email to be compliant with Itiviti’s Privacy Notice and applicable regulations. Please advise us immediately at dataprotectiont...@itiviti.com if you are not compliant with these.* -- ______________________________ itiviti.com <https://www.itiviti.com/> <https://www.linkedin.com/company/itiviti> Follow Itiviti on Linkedin <https://www.linkedin.com/company/10438325/> The information contained in or attached to this email is strictly confidential. If you are not the intended recipient, please notify us immediately by telephone and return the message to us. Email communications by definition contain personal information. The Itiviti group of companies is subject to European data protection regulations. Itiviti’s Privacy Notice is available at www.itiviti.com <http://www.itiviti.com/>. Itiviti expects the recipient of this email to be compliant with Itiviti’s Privacy Notice and applicable regulations. Please advise us immediately at dataprotectiont...@itiviti.com if you are not compliant with these.
