I'm very sorry, I was not active for a while on Beam (my bad). But I'm still happy to help. I will read the whole thread first :)
Regards JB On Tue, Jan 28, 2025 at 9:00 PM Radek Stankiewicz <radosl...@google.com> wrote: > > Hi JB, > > Thanks for that! > > Zack, I think you need to implement autoscaler otherwise Dataflow won't know > how to move splits between the workers. > In your pipeline, have you noticed any imbalance between the workers? > > > > Radek > > wt., 28 sty 2025, 20:54 użytkownik Jean-Baptiste Onofré <j...@nanthrax.net> > napisał: >> >> Hi >> >> Sorry for jumping late on this thread :) >> As I worked on the JmsIO while ago (I'm the original author :)), happy >> to help if I can. >> >> Regards >> JB >> >> On Tue, Jan 28, 2025 at 6:10 PM Zack Culberson >> <zack.culber...@albertsons.com> wrote: >> > >> > Hi Radek, >> > >> > I wouldn’t say it was empty but the queue was receiving about a 1/3 to ¼ >> > of the message one of the other Queue Managers was getting. But those >> > connections were still going up. I am using dataflow runner v2 and 2.61 >> > sdk. We saw the connections on MeshIq which we use to monitor the queues >> > and see connection count and if there are any stuck messages. >> > >> > thank you >> > >> > Zack Culberson >> > >> > >> > >> > From: Radek Stankiewicz <radosl...@google.com> >> > Sent: Tuesday, January 28, 2025 9:15 AM >> > To: Ahmet Altay <al...@google.com> >> > Cc: user@beam.apache.org; Yi Hu <ya...@google.com>; Zack Culberson >> > <zack.culber...@albertsons.com> >> > Subject: EXTERNAL Email: Re: JMSIO support >> > >> > >> > >> > hey Zack, >> > >> > JMSIO reader reads messages in a loop and it looks like whenever there is >> > a checkpoint Beam recreates the connection. Maybe it takes some time for >> > the driver and MQ to finalize the connection. >> > >> > Can you confirm which dataflow runner and sdk you are using? >> > >> > When you observed 2000 connections, do I understand correctly that this QM >> > had empty queues? >> > >> > Were 2000 connections reported by QM monitoring? >> > >> > >> > >> > Radek >> > >> > >> > >> > >> > >> > >> > >> > On Tue, Jan 28, 2025 at 1:28 AM Ahmet Altay <al...@google.com> wrote: >> > >> > I do not know the answer. Adding @Radek Stankiewicz & @Yi Hu in case they >> > might be able to help. >> > >> > >> > >> > On Mon, Jan 27, 2025 at 2:55 PM Zack Culberson >> > <zack.culber...@albertsons.com> wrote: >> > >> > Hi all, >> > >> > I was wondering if anyone would know, we are using the JMS IO read to read >> > from 3 IBM MQ’s. We setup the job to have a configurable amount of >> > readers, which right now is 9 so three readers per mq. Our question came >> > that at one point we were seeing the number of connections for one of our >> > Queue managers was going up to 2000 connections but all of the messages >> > were going to another Queue Manager whose connections stayed at about a >> > constant 100. Could some one help me understand why that is and does the >> > JMS leave the connection open when it connects to read or does it open and >> > close every time it connects ? >> > >> > Thank you >> > >> > >> > >> > Zack Culberson >> > >> > >> > >> > ________________________________ >> > >> > Warning: All e-mail sent to this address will be received by the corporate >> > e-mail system, and is subject to archival and review by someone other than >> > the recipient. This e-mail may contain proprietary information and is >> > intended only for the use of the intended recipient(s). If the reader of >> > this message is not the intended recipient(s), you are notified that you >> > have received this message in error and that any review, dissemination, >> > distribution or copying of this message is strictly prohibited. If you >> > have received this message in error, please notify the sender immediately. >> > >> > ________________________________ >> > >> > ________________________________ >> > Warning: All e-mail sent to this address will be received by the corporate >> > e-mail system, and is subject to archival and review by someone other than >> > the recipient. This e-mail may contain proprietary information and is >> > intended only for the use of the intended recipient(s). If the reader of >> > this message is not the intended recipient(s), you are notified that you >> > have received this message in error and that any review, dissemination, >> > distribution or copying of this message is strictly prohibited. If you >> > have received this message in error, please notify the sender immediately. >> > ________________________________
