hey Zach, Few recommendations: - level of parallelism dataflow v1 is set by desiredNumUnboundedSourceSplits (default N=0) which means that JmsIO will create N splits equal to the amount of your workers. Feel free to increase it. - you should implement AutoScaler <https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/AutoScaler.html> and set it via withAutoScaler - what JMS backend you are using? Some backends like MQ provide queue length so to estimate backlog you need to calculate moving average of payload size. - if you are using Solace, there is a new SolaceIO created.
2 msg/s is suspiciously low - have you tried to debug the pipeline with profiling to understand where the pipeline is wasting time? Radek On 2024/11/14 01:51:31 Zack Culberson wrote: > Hi All, > > I did end up getting my issue resolved with the kafka writer so thank you for that, I have some questions on JMS io as I am trying to increase the rate we are reading messages from the Queue we have. Right now it tops out at about 2msg/s and I am running on a n2-standard-8 machine. I have tried increasing the MaxBatchSize and that hasn't seemed to help. I was wondering on how to setup and use an autoscaler and is that the best course of action if we are looking to be reading 500 msg/s at peak time ? Any help on what I could do here to improve throughput I would create appreciate. > > > Zack Culberson > Data Engineer, Retail > [email protected]<[email protected]> > Albertsons Companies > Phone: +1.605.431.1112 <(605)%20431-1112> > [image005] > > ________________________________ > Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately. > ________________________________ >
