Hi,
I have found simple reproduction scenarios for cluster-connection failure
and related message loss.
All is documented in details under
https://issues.apache.org/jira/browse/ARTEMIS-5086
In short, configuration must have FORCE on config-delete-addresses/
config-delete-queues.
When the configuration is refreshed and if an address (with force deletion)
is removed from configuration, then "Remote binding" queues are deleted at
the same time as the "Local binding" queue.
Regards,
Jean-Pascal
On Sat, Nov 23, 2024 at 3:01 PM Alexander Milovidov <milovid...@gmail.com>
wrote:
> Hi Justin,
>
> I also had this message in the logs. There were also two other queues with
> addresses which had been removed from configuration and undeployed during
> this configuration reload.
>
> The possible scenario to reproduce is:
> - create a cluster with 2 active nodes;
> - create two broker.xml files, one with only default addresses/queues,
> other with approx. 3000 addresses with queues;
> - the configuration file should include all types of objects (addresses,
> anycast and multicast queues);
> - cycle between these two configurations to deploy/undeploy addresses and
> queues - copy configuration file 1 or 2 to the broker.xml;
> - wait for the period needed to reload the configuration and repeat for
> each file.
>
> I'm not sure it will be reproduced because products in the test lab usually
> work as designed.
> Also it is possible to continuously send/receive messages to some queues
> during the entire test to make the test to be closer to the real.
>
> пт, 22 нояб. 2024 г. в 23:41, Justin Bertram <jbert...@apache.org>:
>
> > At this point I don't see how this could happen unless the relevant
> > address-setting had FORCE for the configDeleteQueues or
> > configDeleteAddresses. This code has been essentially the same for the
> last
> > 5 years or so.
> >
> > Do you have a way to reproduce this?
> >
> > Do you also see this logging?
> >
> > AMQ224076 Undeploying address...
> >
> >
> > Justin
> >
> > On Thu, Nov 21, 2024 at 4:31 PM Alexander Milovidov <
> milovid...@gmail.com>
> > wrote:
> >
> > > Hi Justin,
> > >
> > > Thanks for your help.
> > > I will define cluster addresses and queues in the configuration.
> Usually
> > we
> > > don't clean Artemis data on production servers, and IDs of the nodes
> > remain
> > > the same.
> > >
> > > > Can you invoke the "getAddressSettingsAsJSON" management method on
> the
> > > ActiveMQServerControl passing in "$.artemis.internal.sf.#" as the match
> > > and provide the output here?
> > >
> > > {
> > > "addressFullMessagePolicy": "PAGE",
> > > "maxSizeBytes": 524288000,
> > > "pageSizeBytes": 10485760,
> > > "maxDeliveryAttempts": 10,
> > > "messageCounterHistoryDayLimit": 10,
> > > "redeliveryDelay": 0,
> > > "deadLetterAddress": "DLQ",
> > > "minExpiryDelay": 1000,
> > > "maxExpiryDelay": 2592000000,
> > > "redistributionDelay": 1000,
> > > "slowConsumerThresholdMeasurementUnit": "MESSAGES_PER_SECOND",
> > > "autoCreateQueues": true,
> > > "autoDeleteQueues": false,
> > > "autoDeleteCreatedQueues": false,
> > > "configDeleteQueues": "OFF",
> > > "autoCreateAddresses": true,
> > > "autoDeleteAddresses": false,
> > > "configDeleteAddresses": "OFF",
> > > "managementBrowsePageSize": 200,
> > > "autoCreateDeadLetterResources": true,
> > > "deadLetterQueuePrefix": "DLQ.",
> > > "managementMessageAttributeSizeLimit": 10000
> > > }
> > >
> > > (formatted for better reading)
> > >
> > > чт, 21 нояб. 2024 г. в 20:01, Justin Bertram <jbert...@apache.org>:
> > >
> > > > > There are no address settings like autoDeleteQueues,
> > > > autoDeleteCreatedQueues, configDeleteQueues etc. I wonder how could
> > this
> > > > happen.
> > > >
> > > > Can you invoke the "getAddressSettingsAsJSON" management method on
> the
> > > > ActiveMQServerControl [1] passing in "$.artemis.internal.sf.#" as the
> > > match
> > > > and provide the output here?
> > > >
> > > >
> > > > Justin
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://activemq.apache.org/components/artemis/documentation/javadocs/javadoc-latest/org/apache/activemq/artemis/api/core/management/ActiveMQServerControl.html#getAddressSettingsAsJSON(java.lang.String)
> > > >
> > > > On Thu, Nov 21, 2024 at 3:41 AM Alexander Milovidov <
> > > milovid...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All!
> > > > >
> > > > > We have Artemis cluster with two primary / backups, and it worked
> > > > normally
> > > > > before. Suddenly, the cluster queue was undeployed on one of the
> > > cluster
> > > > > nodes during reload of the broker configuration. There was a log
> > > message
> > > > > with event id AMQ224077 Undeploying queue
> > > > > $.artemis.internal.sf.cluster-name.cluster-node-uuid.
> > > > >
> > > > > After this queue was undeployed, the messages which were routed to
> > > other
> > > > > cluster node were unrouted and discarded.
> > > > >
> > > > > There are no address settings like autoDeleteQueues,
> > > > > autoDeleteCreatedQueues, configDeleteQueues etc. I wonder how could
> > > this
> > > > > happen.
> > > > > The cluster queue was recreated after restart of the cluster
> > connector.
> > > > >
> > > > > I don't know the root cause of the problem and we would like to
> > prevent
> > > > > this situation in the future because it leads to message loss. Is
> it
> > ok
> > > > to
> > > > > make cluster addresses and queues to be configuration-managed on
> both
> > > > > cluster nodes?
> > > > >
> > > > > ActiveMQ Artemis version is 2.37.0.
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Alexander
> > > > >
> > > >
> > >
> >
> --
> Regards,
> Alexander
>
