I didn’t use a custom auto scaler though that is something I am relooking into, but I think I set the # of splits to 10.
Thanks Zack Culberson From: Radek Stankiewicz <radosl...@google.com> Sent: Tuesday, January 28, 2025 12:35 PM To: Zack Culberson <zack.culber...@albertsons.com> Cc: Ahmet Altay <al...@google.com>; user@beam.apache.org; Yi Hu <ya...@google.com> Subject: Re: EXTERNAL Email: Re: JMSIO support Any chance to share how you set up auto scaler and how many splits you are setting for this pipeline? https://github.com/apache/beam/blob/master/runners%2Fgoogle-cloud-dataflow-java%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Frunners%2Fdataflow%2Finternal%2FCustomSources.java#L95<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/runners*2Fgoogle-cloud-dataflow-java*2Fsrc*2Fmain*2Fjava*2Forg*2Fapache*2Fbeam*2Frunners*2Fdataflow*2Finternal*2FCustomSources.java*L95__;JSUlJSUlJSUlJSUj!!K1aXqncla1X7G90AkdLmCg!uanPPPKtotrPKuJ3Q4f4EIZzfcw3HNUvUkXlNTNhE0QbT53OcNpcw3kGDo3hcPgvDAU7fgfe8Ek-G30pNIwuQvxeKg$> here we use dataflow pipeline options to specify amount of splits. wt., 28 sty 2025, 18:47 użytkownik Zack Culberson <zack.culber...@albertsons.com<mailto:zack.culber...@albertsons.com>> napisał: Just to add this as well [cid:image001.png@01DB7181.9EF4BA00] So our middle queue has 7122 messages sitting on it but either the JMS IO didn’t scale up to handle those or our first queue that has 734 started getting more messages either way I was expecting all the Queue managers to behave the like the first to keep up with the throughput but that’s not what Im observing. Thanks Zack Culberson From: Radek Stankiewicz <radosl...@google.com<mailto:radosl...@google.com>> Sent: Tuesday, January 28, 2025 9:15 AM To: Ahmet Altay <al...@google.com<mailto:al...@google.com>> Cc: user@beam.apache.org<mailto:user@beam.apache.org>; Yi Hu <ya...@google.com<mailto:ya...@google.com>>; Zack Culberson <zack.culber...@albertsons.com<mailto:zack.culber...@albertsons.com>> Subject: EXTERNAL Email: Re: JMSIO support hey Zack, JMSIO reader reads messages in a loop and it looks like whenever there is a checkpoint Beam recreates the connection. Maybe it takes some time for the driver and MQ to finalize the connection. Can you confirm which dataflow runner and sdk you are using? When you observed 2000 connections, do I understand correctly that this QM had empty queues? Were 2000 connections reported by QM monitoring? Radek On Tue, Jan 28, 2025 at 1:28 AM Ahmet Altay <al...@google.com<mailto:al...@google.com>> wrote: I do not know the answer. Adding @Radek Stankiewicz<mailto:radosl...@google.com> & @Yi Hu<mailto:ya...@google.com> in case they might be able to help. On Mon, Jan 27, 2025 at 2:55 PM Zack Culberson <zack.culber...@albertsons.com<mailto:zack.culber...@albertsons.com>> wrote: Hi all, I was wondering if anyone would know, we are using the JMS IO read to read from 3 IBM MQ’s. We setup the job to have a configurable amount of readers, which right now is 9 so three readers per mq. Our question came that at one point we were seeing the number of connections for one of our Queue managers was going up to 2000 connections but all of the messages were going to another Queue Manager whose connections stayed at about a constant 100. Could some one help me understand why that is and does the JMS leave the connection open when it connects to read or does it open and close every time it connects ? Thank you Zack Culberson ________________________________ Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately. ________________________________ ________________________________ Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately. ________________________________ ________________________________ Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately. ________________________________
