Hi Jack- This looks like you are using the JDBC backend. I would recommend having the DBA review logs for the database for other indicators. You may have a couple things going on. For sure it appears you have a duplicate publishing attempt based on the PRIMARY KEY violation.
-Matt Pavlovich > On Feb 12, 2024, at 8:34 AM, Jack Fry (They/Them) > <j...@scottlogic.com.INVALID> wrote: > > (Re-sent this as I wanted to change the subject) > > Hi, > > Last week our ActiveMQ message broker lost a message from the queue. We have > a load balanced system with two separate brokers sharing a data store. At > some point during the transfer of the lease, the original lease holder > started shutting down, which started a cascade of failures as the Transport > Connection failed to deliver the message. > > 16:54:07.869 Starting Job Scheduler Store > 16:54:07.869 Persistence Adapter successfully started > 16:54:08.456 Apache ActiveMQ 5.18.3 (gbldnsrv4pw4564, > ID:GBLDNSRV4PW4564-50268-1705135858444-0:3) is starting > 16:54:10.807 gbldnsrv4pw4563, no longer able to keep the exclusive lock so > giving up being a master > 16:54:10.807 Apache ActiveMQ 5.18.3 (gbldnsrv4pw4563, > ID:GBLDNSRV4PW4563-63611-1706742264783-0:2) is shutting down > 16:54:10.807 Transport Connection to: tcp://10.18.136.56:51504 failed: Broker > BrokerService[gbldnsrv4pw4563] is being stopped > 16:54:10.807 socketQueue interrupted - stopping > 16:54:10.807 Could not accept connection during shutdown : null (null) > 16:54:10.823 Transport Connection to: tcp://10.18.136.38:57430 failed: Broker > BrokerService[gbldnsrv4pw4563] is being stopped > 16:54:10.823 Failed delivery for (MessageId: > ID-GBLDNSRV4PW4563-1706742293708-1-145409 on ExchangeId: > ID-GBLDNSRV4PW4563-1706742293708-1-145309). On delivery attempt: 0 caught: > org.springframework.jms.UncategorizedJmsException: Uncategorized exception > occurred during JMS processing; nested exception is javax.jms.JMSException: > Peer (vm://localhost#35661) disposed. > > However, when the second broker came to process the failed messages, there > was an Primary Key exception from the sqldb that the message was already > stored in the database. > > 16:54:37.357 Error while closing connection: Violation of PRIMARY KEY > constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in > object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684907). > 16:54:37.374 Ignoring SQLException, java.io.IOException: Violation of PRIMARY > KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key > in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684907). > 16:54:37.421 Ignoring SQLException, java.io.IOException: Violation of PRIMARY > KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key > in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684909). > 16:54:37.421 Commit failed: Violation of PRIMARY KEY constraint > 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object > 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684909). > 16:54:37.483 Store COMMIT FAILED: > 16:54:40.824 Failed delivery for (MessageId: > ID-GBLDNSRV4PW4563-1706742293708-1-145409 on ExchangeId: > ID-GBLDNSRV4PW4563-1706742293708-1-145309). On delivery attempt: 1 caught: > java.lang.IllegalStateException: SendProcessor has not been started: > sendTo(activemq://queue: > 16:54:40.824 Failed delivery for (MessageId: > ID-GBLDNSRV4PW4563-1706742293708-1-145410 on ExchangeId: > ID-GBLDNSRV4PW4563-1706742293708-1-145312). On delivery attempt: 1 caught: > java.lang.IllegalStateException: SendProcessor has not been started: > sendTo(activemq://queue:) > > Would anyone here know what happened here? Is this a bug? > > Many thanks, > Jack
