Hi Nikolas- Images won’t come through the mail server. Best to post the picture to pastbin/imgur/etc and share the link.
Thanks, Matt Pavlovich > On Oct 25, 2023, at 6:25 AM, Nikolas Papaioannou > <niko...@mnemonic.no.INVALID> wrote: > > > > Reference image broke, trying again! > > As text: > Rest 1,2,3 produces towards a random broker 1,2,3, and service 1,2,3 > subscribes to random broker > > - Nikolas > > From: Nikolas Papaioannou <niko...@mnemonic.no.INVALID > <mailto:niko...@mnemonic.no.INVALID>> > Sent: 25 October 2023 13:15:11 > To: users@activemq.apache.org <mailto:users@activemq.apache.org> > Subject: Messages getting stuck on nodes without consumers, even with TTL > > > Hi there! > > > > We are having some issues and need a small sanity check :') > > > > We are running a network of brokers (multicast) with three nodes. Consumers > and producers can connect to any of three nodes in the network/cluster. From > time to time, we see that messages start to expire for one or more queues. > When this happens, it is usually the case that the node where the message > expiry occurs, does not have any directly connected consumers. It is as the > messages are stuck on the node where the expiry occurs. > > > > We are trying to figure out what might be wrong with our config, when > removing broker3 everything seems stable and working. Without any "stuck" > messages expiring. > > The nodes are strongly coupled with a 10G link, so we have no issues with > passing along messages between nodes as need be > > > > > > > > This is our current config (sans auth and queues/topics): > > > > > > > > > > > <beans > > xmlns="http://www.springframework.org/schema/beans"; > > xmlns:amq="http://activemq.apache.org/schema/core"; > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans.xsd > > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd";> > > > > <!-- Allows us to use system properties as variables in this configuration > file --> > > <bean > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > > <property > name="locations"> > > <value>file:${activemq.conf}/credentials.properties</value> > > </property> > > </bean> > > > > <broker > xmlns="http://activemq.apache.org/schema/core"; > brokerName="<%= @fqdn %>" > dataDirectory="${activemq.data}" > useJmx="true" > persistent="false" > > > > <plugins> > > <discardingDLQBrokerPlugin > dropAll="true" > /> > > <simpleAuthenticationPlugin> > > <users> > > <authenticationUser > username="example" > password="example" > groups="producers,consumers"/> > > </users> > > </simpleAuthenticationPlugin> > > <authorizationPlugin> > > <map> > > <authorizationMap> > > <authorizationEntries> > > <authorizationEntry > queue="Service.>" > read="consumers" > write="producers" > admin="admins"/> > > <authorizationEntry > topic="Service.>" > read="consumers" > write="producers" > admin="admins"/> > > <authorizationEntry > topic="ActiveMQ.Advisory.>" > read="consumers,producers,guests,users" > write="consumers,producers,guests,users" > admin="consumers,producers,guests,users,admins"/> > > </authorizationEntries> > > </authorizationMap> > > </map> > > </authorizationPlugin> > > </plugins> > > > > <destinationPolicy> > > <policyMap> > > <policyEntries> > > <!-- Set the following policy on all queues using the '>' wildcard --> > > <policyEntry > queue=">" > prioritizedMessages="true"/> > > <policyEntry > queue=">"> > > <!-- Tell the dead letter strategy not to process expired messages so that > they will just be discarded instead of being sent to the DLQ --> > > <deadLetterStrategy> > > <sharedDeadLetterStrategy > processExpired="false" > /> > > </deadLetterStrategy> > > </policyEntry> > > <policyEntry > topic="Service.>"> > > <networkBridgeFilterFactory> > > <conditionalNetworkBridgeFilterFactory > replayWhenNoConsumers="true"/> > > </networkBridgeFilterFactory> > > </policyEntry> > > </policyEntries> > > </policyMap> > > </destinationPolicy> > > > > <destinations> > > <queue > physicalName="Service.Example" > /> > > <topic > physicalName="Service.Example.ALL" > /> > > </destinations> > > > > <persistenceAdapter> > > <memoryPersistenceAdapter/> > > </persistenceAdapter> > > > > <transportConnectors> > > <!-- Some Puppet template marup, @fqdn is resolved to the hostname of the > node --> > > <transportConnector > name="listener" > uri="tcp://<%= @fqdn > %>:4001?maximumConnections=1000&wireformat.maxFrameSize=104857600" > rebalanceClusterClients="true" > updateClusterClients="true" > updateClusterClientsOnRemove="true"/> > > <transportConnector > name="multicast" > uri="tcp://<%= @fqdn %>:4200" > discoveryUri="multicast://default"/> > > </transportConnectors> > > > > <networkConnectors> > > <!-- Only Message TTL? Might need to set the other TTL too? --> > > <networkConnector > uri="multicast://default" > userName="broker" > password="password" > decreaseNetworkConsumerPriority="false" > messageTTL="5" > suppressDuplicateQueueSubscriptions="false" > conduitSubscriptions="false"/> > > </networkConnectors> > > > > <shutdownHooks> > > <bean > xmlns="http://www.springframework.org/schema/beans"; > class="org.apache.activemq.hooks.SpringContextHook" > /> > > </shutdownHooks> > > > > </broker> > > <import > resource="jetty.xml"/> > > </beans> > > > > > > > > -Nikolas Papaioannou
