Masters and slaves don't communicate with each other. Masters are active, and slaves are inactive, waiting to become the master and activate. Do you mean that you have a Network of Brokers?
If so, if a broker that is consuming from another broker becomes a slow consumer, you should investigate whether maybe the message store (persistence or memory, depending on whether the messages are persistent or non-persistent) on that other broker got full and producer flow control kicked in. If so, you'd want to understand why real (non-broker) consumers aren't consuming messages fast enough to prevent the store from filling. Also, if you suspect that the brokers are simply overloaded by the volume of connections to them, one option would be to use a CPU sampler such as JVisualVM to do CPU sampling of the broker process, to determine where the majority of the time is being spent. Since you mentioned that everything was fine till you added more consumers, that might be a path worth pursuing. Tim On Fri, Mar 15, 2019 at 6:32 PM Konstantin Raskoshnyi <konra...@gmail.com> wrote: > Hi guys, > I set up activemq based on this document > https://puppet.com/docs/mcollective/current/deploy/middleware/activemq.html > > Everything worked fine until we got 6000 more servers subscribed. > > Sys memory: > > > brokers config > > transport = 0.0.0.0:61616?jms.prefetchPolicy.all=100000& > <http://0.0.0.0:61616/?jms.prefetchPolicy.all=100000&>; > > <bean id="prefetchPolicy" > class="org.apache.activemq.ActiveMQPrefetchPolicy"> > <property name="queuePrefetch" value="100000" /> > </bean> > > > > <systemUsage> > <systemUsage sendFailIfNoSpace="true"> > <memoryUsage> > <memoryUsage limit="2 gb"/> > </memoryUsage> > <storeUsage> > <storeUsage limit="5 gb" name="foo"/> > </storeUsage> > <tempUsage> > <tempUsage limit="1 gb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > > I see the next error after which master broker & slaves cannot communicate > to each other: > 2019-03-13 13:03:49,078 | WARN | TopicSubscription: > consumer=broker1->broker2-38486-1552507411577-12:1:1:1, destinations=6, > dispatched=1000, delivered=0, matched=1001, discarded=0: has twice its > prefetch limit pending, without an ack; it appears to be slow | > org.apache.activemq.broker.region.TopicSubscription | ActiveMQ Transport: > ssl:///192.168.12.6:45626 > > Is there any workarounds around this? > > > Thanks >
