I've been investigating this more the whole week, I would really appreciate a response. What I can see from inspecting the logs and metricbeat analysis, is that the mssql connection in the first broker starts getting very slow, which causes it to lose the master lease while messages are still inflight.
When the first broker fails to restore the lease it starts to shutdown and the messages are lost without being processed. The jdbc persistence adapter in the activemq.xml file does not set failIfLocked so that should be false. Is there a way to stop the broker from shutting down when it fails to renew its lease? Secondly, when the second broker starts and retries the failed messages, it gets a Primary Key violation in object 'dbo.ACTIVEMQ_MSGS'. I believe this is in the temp store within the server. Is there a way to flush this temp store on a failure? We are using ActiveMq classic v5.18.3 and these failovers started not long after upgrading from v5.13.2. Thanks -----Original Message----- From: Jack Fry (They/Them) <j...@scottlogic.com.INVALID> Sent: Monday, February 12, 2024 4:16 PM To: users@activemq.apache.org Subject: RE: Message delivery failure and Primary Key violation on load balanced ActiveMQ Broker [You don't often get email from j...@scottlogic.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Looking at the logs again, we noticed this warning on the broker as well: Store limit is 30720 mb (current store usage is 0 mb). The data directory: D:\Apps\apache-activemq-5.18.3\data only has 25672 mb of usable space. - resetting to maximum available disk space: 25672 mb Could this be related? If there was not enough available disk space would It cause this exception to occur? -----Original Message----- From: Jack Fry (They/Them) <j...@scottlogic.com.INVALID> Sent: Monday, February 12, 2024 2:35 PM To: users@activemq.apache.org Subject: Message delivery failure and Primary Key violation on load balanced ActiveMQ Broker [You don't often get email from j...@scottlogic.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] (Re-sent this as I wanted to change the subject) Hi, Last week our ActiveMQ message broker lost a message from the queue. We have a load balanced system with two separate brokers sharing a data store. At some point during the transfer of the lease, the original lease holder started shutting down, which started a cascade of failures as the Transport Connection failed to deliver the message. 16:54:07.869 Starting Job Scheduler Store 16:54:07.869 Persistence Adapter successfully started 16:54:08.456 Apache ActiveMQ 5.18.3 (gbldnsrv4pw4564, ID:GBLDNSRV4PW4564-50268-1705135858444-0:3) is starting 16:54:10.807 gbldnsrv4pw4563, no longer able to keep the exclusive lock so giving up being a master 16:54:10.807 Apache ActiveMQ 5.18.3 (gbldnsrv4pw4563, ID:GBLDNSRV4PW4563-63611-1706742264783-0:2) is shutting down 16:54:10.807 Transport Connection to: tcp://10.18.136.56:51504 failed: Broker BrokerService[gbldnsrv4pw4563] is being stopped 16:54:10.807 socketQueue interrupted - stopping 16:54:10.807 Could not accept connection during shutdown : null (null) 16:54:10.823 Transport Connection to: tcp://10.18.136.38:57430 failed: Broker BrokerService[gbldnsrv4pw4563] is being stopped 16:54:10.823 Failed delivery for (MessageId: ID-GBLDNSRV4PW4563-1706742293708-1-145409 on ExchangeId: ID-GBLDNSRV4PW4563-1706742293708-1-145309). On delivery attempt: 0 caught: org.springframework.jms.UncategorizedJmsException: Uncategorized exception occurred during JMS processing; nested exception is javax.jms.JMSException: Peer (vm://localhost#35661) disposed. However, when the second broker came to process the failed messages, there was an Primary Key exception from the sqldb that the message was already stored in the database. 16:54:37.357 Error while closing connection: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684907). 16:54:37.374 Ignoring SQLException, java.io.IOException: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684907). 16:54:37.421 Ignoring SQLException, java.io.IOException: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684909). 16:54:37.421 Commit failed: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684909). 16:54:37.483 Store COMMIT FAILED: 16:54:40.824 Failed delivery for (MessageId: ID-GBLDNSRV4PW4563-1706742293708-1-145409 on ExchangeId: ID-GBLDNSRV4PW4563-1706742293708-1-145309). On delivery attempt: 1 caught: java.lang.IllegalStateException: SendProcessor has not been started: sendTo(activemq://queue: 16:54:40.824 Failed delivery for (MessageId: ID-GBLDNSRV4PW4563-1706742293708-1-145410 on ExchangeId: ID-GBLDNSRV4PW4563-1706742293708-1-145312). On delivery attempt: 1 caught: java.lang.IllegalStateException: SendProcessor has not been started: sendTo(activemq://queue:) Would anyone here know what happened here? Is this a bug? Many thanks, Jack
