Thanks! There is definitely something odd about our K8S deployment, but we haven’t figured out what it is yet
[rg] <https://www.redpointglobal.com/> John Lilley Data Management Chief Architect, Redpoint Global Inc. 888 Worcester Street, Suite 200 Wellesley, MA 02482 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: Justin Bertram <jbert...@apache.org> Sent: Thursday, March 16, 2023 7:03 AM To: users@activemq.apache.org Subject: Re: Can someone help explain journal-related error? *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** I don't see anything in the log to indicate why the broker shutdown which typically indicates that it was shutdown administratively. Justin On Thu, Mar 16, 2023 at 7:53 AM John Lilley <john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>> wrote: Hi Justin, Thanks for the quick response! My colleague saved the artemis log here: https://drive.google.com/file/d/1y9vK7tiTBEP3-2m4K5DFA6DyPG2AGg8W/view?usp=share_link<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fdrive.google.com%2ffile%2fd%2f1y9vK7tiTBEP3-2m4K5DFA6DyPG2AGg8W%2fview%3fusp%3dshare_link&c=E,1,O9YvN7CNx7dp_5P5SpJgjGiv7kqd7BgvgssEt4a5ETweU3pTaXc0NIzgsCBcICcsfb81jqVJxjpCAmuG7VXx1o3sfYj5GsqRT2m6HxSoxs8tOLQ,&typo=1> He did not see anything else concerning, but maybe there is. Any help much appreciated. Thanks John [rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,g5RKoSAFFUzeXjmpIUM5S5TR2AB18Tq_z4G3_CI7cyTvnVZK0Z3XKaFA-drK9OOgmVdAXN4DYAxzAv-N2S7h2lqnpf0e9GWwaB_JYJDr6s30Z8nJUuYhxNfkvg,,&typo=1> John Lilley Data Management Chief Architect, Redpoint Global Inc. 888 Worcester Street, Suite 200 Wellesley, MA 02482 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: Justin Bertram <jbert...@apache.org<mailto:jbert...@apache.org>> Sent: Wednesday, March 15, 2023 11:17 AM To: users@activemq.apache.org<mailto:users@activemq.apache.org> Subject: Re: Can someone help explain journal-related error? *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** The WARN messages about ActiveMQShutdownException are likely a symptom and not a cause of the broker shutdown. Those exceptions are being thrown because a few threads are still trying to access the journal after it's been shut down (as part of the overall broker shutdown process). You'd need to provide more of the log for additional investigation about the root cause of the broker shutdown. The WARN message about AMQ222269 is just indicating that you're using an unfixed value for journal-pool-files in broker.xml (i.e. "-1"). It's safer to use a fixed value so that the pool of journal files can grow and especially shrink as necessary. You can read more about journal-pool-files in the documentation [1]. Justin [1] https://activemq.apache.org/components/artemis/documentation/latest/persistence.html#configuring-the-message-journal On Wed, Mar 15, 2023 at 10:54 AM John Lilley <john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>> wrote: We recently encountered a fatal error in which Artemis stopped. Can anyone help explain how we might have got here? Does this indicate issues with journal storage? I don’t expect we’ll find a smoking gun, but perhaps you can suggest other log entries to look for or other diagnostics to capture? WARN [org.apache.activemq.artemis.core.server] AMQ222153: Cannot locate record for message id = 530583 on Journal org.apache.activemq.artemis.api.core.ActiveMQShutdownException: Journal must be in state=LOADED, was [STOPPED] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkJournalIsLoaded(JournalImpl.java:1301<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fJournalImpl.java%3a1301&c=E,1,O0DQ0AqgApwFJoQ_sgLy80QvEOBSEO_UqaEY5KmwX9uSVjdbLXWtm3cwkcu2_WfHLVDVRqnO9ira0yka7LvTf67xSf1SIZp3HArFr4vvk8h4E7A,&typo=1&ancr_add=1>) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.tryAppendDeleteRecord(JournalImpl.java:1171<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fJournalImpl.java%3a1171&c=E,1,Wu0WFTBK6Q26DULiBdUP-AF7OFIV7Dm84oN5RWfhwpt_O7ehlGGSP4aQX04ykkKmyRy3NqGMuyMfv9I464X86SjAz1Hkcs_kp9qeaS8FLly750PJM5c-apot&typo=1&ancr_add=1>) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.deleteMessage(AbstractJournalStorageManager.java:454<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractJournalStorageManager.java%3a454&c=E,1,V3sQK3gEkM677jvQDfqk5UDcS2TkycZhuR8YaBhDJQZlzbZc48_-eHnKZK8E2UY0sywp9PeM22Mnvi4qrHkbqnUW65qbH8ELom3YBdNC4a3iIzdzKUzTOADLhqAW&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.QueueImpl.postAcknowledge(QueueImpl.java:4054<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fQueueImpl.java%3a4054&c=E,1,DUpOyJIsPf5qAnR9KPIPETnWOpiUPBjVEYoRL642Xv-e7U_331MaaQ03EaM0svAQpDrJ_QEpO0_blejashZJCsxw7Mq8niwMmNBUIY4Qc86ujJKwgA,,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.afterCommit(RefsOperation.java:188<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRefsOperation.java%3a188&c=E,1,DF4hAUBFAGcsf1TzKa-U1tVeImCRua4b5gxN0ZH-siZrDosvP4nFyF0T1dps-9nCIdBvSklQVCymZG6VmuDYNkYBep5Te-Rty5atyjxB_rkGO6jWdUyvMg,,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:560<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fTransactionImpl.java%3a560&c=E,1,UPOChCL0Vj1snm_Rjj0PO-99Fyk6BbWa5WGgMjd1USlUmg5N1fXluOOrvsiJxK4bLYHZUk1VCqERBFyeUHs-c9HkVeXT5g6Fa97nCaPdftyqVdxh&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:303<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fTransactionImpl.java%3a303&c=E,1,V73eN2Nu8h6kMnWOR_zgxEKjof99MkDr81KYrTLyt9NHvs0V5rsZlrdzFeNGDAXLbSFEPKhGdJYVX5wr8VsPSM0dcpHe7b-nAp9h55DB73MThthjTTYKTpzDFZXS&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,WE1dX34m8h4dKsy4VpThvIQEF7o0w0q7eO-XKtogy3l-ogb4-HQ_e20mHD9DnohAaldLjPL0I9ag-BGqMuGzpWWCDjybqUQGVjWR4eS_yZuG8ndw4Dw,&typo=1&ancr_add=1>(OperationContextImpl.java:284<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOperationContextImpl.java%3a284&c=E,1,zt1rSgXpYdC2EAVYtpHUHNzvWej9Dm1BMS8TeByg6nwGubREY5ufqTH_-qL_rNOxWIBQ6QfmlLXQd90FZUGWxvJjKH1jmoSzPjumVh87&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a58&c=E,1,S5UGFe9nzI1JTByFY8MYVUYLEpNEAe0W8GkFrKGLbBm-qOlg0q42X4z_NdjsQUHCDhK8Q2l4tQu0KZbgVxKxorKi0hpn72fzN_j5zhA1pj9e-JYDvl-NZQ19-us,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a33&c=E,1,dKB8TheP0P6QfgZRk4lbJe5GAgY_34UFL8_7XmmZkfwsRCRujZEEKVLiesYJ894Dy1j5Y3zd5VwVsg9xdoJADxBMPpPO7WvEjbVYk7G9F3fAZg5cIs87M8xB99A,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a69&c=E,1,zstzdDtlnynrdIIzKpSOmLPIEXrrGF3k7-HVlN_SxjjdhlggrY34Suou-4KijsrqtcrSc-ZMRHLntclshRkF0D2PRMi550T6beVEpLE1vr9XYXAEADQ,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,BRmPyDPFszCgAxyVFBBx77FjMG8u5zhwKKzXiJKY9nU0Vlb52sv6IVzYZXhQj0iaI16lTuLyn7uutDWS0JcPK654JIRXVn_h4s_5pdgztEx_TTOi&typo=1&ancr_add=1>(Unknown Source) ~[?:?] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,wI-dBAlpi9SqYLDirWakqU7HB2OBxZh-3xx13OOzq66ZGJXQ8koHdi8AlJqPUsajuVT2VWZWBLqr7YiCQu0MOEwnZQPhQeI1qcD3PBumqFCJLw,,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,BompecMXwlyoz2d5OHvwdtLTQyMeBdkWWNcnzjvAk8S7SZVFbtGV-cMRznb6LFs2r8DW43yiz7xJDVz3jqITcIlO4FLIGjtfoBbN2gnK1S0CpZ2M_-zB8Q,,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] WARN [org.apache.activemq.artemis.core.server] AMQ222154: Error checking DLQ org.apache.activemq.artemis.api.core.ActiveMQShutdownException: Journal must be in state=LOADED, was [STOPPED] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkJournalIsLoaded(JournalImpl.java:1301<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fJournalImpl.java%3a1301&c=E,1,atonxTRrm15UcRe3WM9O9b2Ks0Z55QfSGNYtXHLVbTronxOgCENgQuusU9XmNg6fFRJIT1U8e7oZiO6tGDDZnZkt4-emiMb5OMfSsxGLrhrD6_pPTQ5tidWhmg,,&typo=1&ancr_add=1>) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.tryAppendUpdateRecord(JournalImpl.java:1053<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fJournalImpl.java%3a1053&c=E,1,jD6Ai80KdEc2DPD2HysKfVZeBAsexwXCOtC_7er_PvYFXa2NPuh_ubmY9LfukV09ZlFingV_bcRpp5-rFsQPjCi0kEfULeQHozqJ13186EQLMHjDqCq2kc7Y-L1-&typo=1&ancr_add=1>) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.Journal.tryAppendUpdateRecord(Journal.java:152<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fJournal.java%3a152&c=E,1,z_88VOCCi3wl_usxwk0QVxiYEHYRRfA6euIQQFjS6yVh25xCI5nVAFf5q1D5PWfo7x2orXREf_b2nm2C0r90G05ijhyfbEpTjXX1Zj2YF89kYywCfw,,&typo=1&ancr_add=1>) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.updateDeliveryCount(AbstractJournalStorageManager.java:709<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractJournalStorageManager.java%3a709&c=E,1,4u12x648NGdntHZFDGDmn6iDP2TFIXyzxKMX1akoH9EvuneQn1gRaFUbGo5w2OzP7WQ_bZ5Ti2ENNOj5tdrICqXnyn01twLbBqzEd6JhELACjLE9FBkVGHnc&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.QueueImpl.checkRedelivery(QueueImpl.java:3401<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fQueueImpl.java%3a3401&c=E,1,AUcBEh9wSwBVqRMb-T5v1YAAuS6nmOBnlOomxwT4FQ2x32jy73fpF59E6nQCyobPFuZ6fZZ5gASVrmYbMb2ooH6UVUjLOgAvGpb6utWcEcXizkR84NrJTK_M&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.rollbackRedelivery(RefsOperation.java:169<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRefsOperation.java%3a169&c=E,1,7PNiwBPkEFWIaHp4F-DlDZGMjioLeZzF6hMhIcg4WGeNlbN7x-aK8LMRqBdPkDYc2oHo0X8V6jKIMpDfNb1x5tBwnng-jdnCCI3e8dgV0MVJ9UMFVyA,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.afterRollback(RefsOperation.java:119<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRefsOperation.java%3a119&c=E,1,nzSYdbfK33BMmviqcfWbzMK5G1A0y8dVHltap8QG_jFSNUOSJ4cPq14DljFqNaFx3gn3eswe17dX-1y91h1iI0ZfuFiv_EmynNeO1dl5ftjCXwS4jg,,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterRollback(TransactionImpl.java:570<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fTransactionImpl.java%3a570&c=E,1,pDJ8npZzDNLspbrGnQZA0XmQrocILD6-pdKVuwZvjio0SNHWLvpGGPwiIzTUmxdT5WNrBR0HAB2DsINGw7crQtnBU03tKboRYBTOjgHM-64DB3NoSFG4&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$4.done(TransactionImpl.java:421<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fTransactionImpl.java%3a421&c=E,1,CXQuE3x16uqJIEiM3c-EwsBTsN5U64GTS7GSA9DNLfI0nqrf4RI_hJqIhinrcGkUDEFpY6Uz3JcqENyjIZuxeSxSkiLBwk-GYvYrtTv4goUFBLsiIYs,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,iuRg2Lc-IDNRKkyrXRnZjYXVXPwEnJ0MbAOQFYNLSKcw25NcqGdTP2sM2W6mVSrESFaj7M8rrQxhgJzca6B1NcpyGQwU0_ffwpZH6TsbCMChRyhqdA,,&typo=1&ancr_add=1>(OperationContextImpl.java:284<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOperationContextImpl.java%3a284&c=E,1,5u9iA_0tHzg52XXgoL_SRFQMvgckRfVQzM3mFCoYeL2Ctd1dWd0wyE-E-AGGeBFCu-VsYdFz2tlxTL6VlUZfKwOLaL1fpwKvdHDsOwwHFAEgOPsBLR80CQ,,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a58&c=E,1,bkEYT3XeLzx5SfsmNUyFewIwe6v20d40EXzK2cEZqE4kaD3pOk0ttZtF20fpbI-P3rC1eWjseONXyiueMfqLZjvxCH-ZxAA00uHzZtgIPSILETN1UbZVjhs,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a33&c=E,1,1SYeX3Czr_dfRBuc9eiJb569ungLd2sxiYbSy4mZVZqqvHmJjYPq4ZJ6g29bknzYrvLc-IN-wm9hoBsvMwuy4Q4Psw3VMiir-o_fd-6MNw,,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a69&c=E,1,_vflAYzEZTWDquPieW3TOSNYVe4bSxlVm04KULNZoJguY6RJoc1Lzpk9BWdp0jQz8gFepGrDwzZzAyU3ZBUAcXiMGNDFMyWy4r5Ycg7jtFwbHtEMlEfpAex62t_l&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,BsKB8Dv9EdJ4G3iN4jpP7wIJirf73r-hRmyEMy6XLXKx7-Jz5MFJ1VBdZriIM9Q3V101YhFstel3NcjFqW75theqgCRukgNh_ohniQBUwOiqE7nc&typo=1&ancr_add=1>(Unknown Source) ~[?:?] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,5fayBW3o4J0FMGZ2NGJjy74pON1NR6lQB6KD97NIJEQZxY7QMI9sIx4bqykVg41FRdjrWAGUEW_2uEAP-yTD04vGQTDaxcauCgVV2LRWVSCkRlOWwvTaW9BWQg,,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,O-GwJ4OlI5OwRSHP1Wzo6_doSpdPer8xhB06Av32aR5yQyh787JI5suPLFe_ySzG8GYzWvig2DzKm6aZVWcoSdK6kPJhwU7PyaY_WTjiZNrdKh_sie_miFw,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?]----- a bunch more of these and then this final one ----- WARN [org.apache.activemq.artemis.core.server] AMQ222154: Error checking DLQ org.apache.activemq.artemis.api.core.ActiveMQShutdownException: Journal must be in state=LOADED, was [STOPPED] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkJournalIsLoaded(JournalImpl.java:1301<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fJournalImpl.java%3a1301&c=E,1,51dMCBHFLMcO-pnijR-xYhcR6WaIVpHxuSSlxYj5PRyfeKkBW-mwT2u05MZ8XJ-AnsqiZjjP1kscslt6F5VuAZSDaM9Ru7VXkI2MdBcUj1k,&typo=1&ancr_add=1>) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.impl.JournalImpl.tryAppendUpdateRecord(JournalImpl.java:1053<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fJournalImpl.java%3a1053&c=E,1,HjHMTST3GX1G95NCN-0RLn93wZrI98xYLQTYvcnnXyZBiybGT7lJS4Ov3mSK5Iirb9jJheuqkrWb5arKUBK7Ly-p-tZCDIHiCDrBueJFMAtEom9MAgV4Hho,&typo=1&ancr_add=1>) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.journal.Journal.tryAppendUpdateRecord(Journal.java:152<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fJournal.java%3a152&c=E,1,I87CBecNT1rNGnCSr5NO9XJubWlNJujaX-wB7tcieNjNW_kqOno2zKLR7xhrfvBjV4txAi8SEXFwTTteFbEqO-BeS_YcKwfdU26HaFZ5mg,,&typo=1&ancr_add=1>) ~[artemis-journal-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.updateDeliveryCount(AbstractJournalStorageManager.java:709<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractJournalStorageManager.java%3a709&c=E,1,izmH3um73NLioWDJVkai_4E4ymzU6JN1wKRX_VVkgkDV8tt4e9q-A6A-NpaUPHziIwxPuUj1slAmiXCGwadkMO-n58HgUQyswSoy3NQFd1KxGI3F6JSZypyveXrg&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.QueueImpl.checkRedelivery(QueueImpl.java:3401<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fQueueImpl.java%3a3401&c=E,1,reDbIBoTs1LQoyZqKaKSaQulIg_PyHBXSzwBupem9brEttw4ZjiMLyAieiYCX4GornwtOxpGpmK1dvCzu2Ezf4XPL9FQvEMCcCM3nlfnXnkn2g,,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.rollbackRedelivery(RefsOperation.java:169<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRefsOperation.java%3a169&c=E,1,pp1x3cpWv8jL250N2UqhJDlAD-zB8UkW4YcqB-faamJKZflSz8uXCF9BYv1LIsZHIElo2Ewul_4IlM2__ekotRzVQSfDD4-O5SGl-1GF4MNfHz8YFcEve37PgcA,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.server.impl.RefsOperation.afterRollback(RefsOperation.java:119<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRefsOperation.java%3a119&c=E,1,lDRnPqyO86dUDfjpGNyuPetVyUz3SxL7SkLNZ2NooISBYXlWFTK0LXlAs9K0kW26O-Tp-TJg8AAkWmlsKFJfSAYZ1ltKv4UJEin3ZO_er-ZPXwY,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterRollback(TransactionImpl.java:570<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fTransactionImpl.java%3a570&c=E,1,Lo3-ks6ucDicj2RA4KJwOnJHN5a2rJ60_ArfiCpZkig41Hf5m_x8w4dOdovmC5aZhgKCtfmBsdBzsqJgt-JhUwS7vcgj0wFV7qghqy88UH2Up317Ls7pHrck7d8,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$4.done(TransactionImpl.java:421<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fTransactionImpl.java%3a421&c=E,1,agJ4AN6MSKX-usUvLDIEHbFEYvuMmCmvH-ifsgGcNRixJDVNzyxLW9Faqh9sL1pQx10oomtJPEy9z8i-voixSKdzSPdEjZ7d28bM-GLLmS2cVdXOkA,,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,RtGG9O-OfBOgCseVf2765pxOnHuYUoo5vI35hgMDucr3jLfTTbLGT_2ftxXzIezxjFBNrpPZ3u64kJr8fRpbPAm26WxyOR6sEAFVn7HJb_Xs1g,,&typo=1&ancr_add=1>(OperationContextImpl.java:284<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOperationContextImpl.java%3a284&c=E,1,DDuVr8NuE_AUPP6LpxHQAz0cIQ2nf7HCkEO_h2uvDlyVPBDXl7JZ8PVCYS0ZsWr8_uWIXNvqzf4sXGA0kK9pPWL745V1Z1XL6p7tCRlPEdQtHaEW4Ig,&typo=1&ancr_add=1>) ~[artemis-server-2.27.1.jar:2.27.1] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a58&c=E,1,VfyQY9UEpjDpfbCLwBH8Jk3Kk2cOIqX1Wc4cTnPqc4Hy_btQVR2IxEC_yXDtJFQHaP2g5o7Qvxx94Z9NWwf5GNKmmcBlnnou7OR3QxY2hNu4rw,,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a33&c=E,1,TL5Cv3NLFhs1ptDgb3Ibuxqt36JsXjqJchL8i0efXJV7o36xOmYIj1-rwORWCyxp4t1KdIMtU65vzdVZq9eh90bXUz8St8W-7fzDuaD0A15GEAzp0R5H5Vo,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a69&c=E,1,c1oiOz-6UAo1BYPhLhbrYYBOr8RKpbrzc-2wDEMce2UOXlt45NcBXQM-fakBkPj_6VTY4-bq7IuUBcgyAxJESHVlN2-IYkaJXdJrL3EvMhLmftn7aWTeHw,,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,srwpagZSQY_AlxFmoOxtJs2z8E6RToW5NtRkQbPnopjoCv85Gm9PP4Dr6dCBL2llAM118S5EmIzXAvNi25kQb5qLc1fMO-PrscBuVfrvkXhGd-Z2ez9AUw,,&typo=1&ancr_add=1>(Unknown Source) ~[?:?] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,IKVHhWGOSKPrh3FOyAb1MvzjW2wGTALyl0ZR_x4sUvWCMrUqjSIonhc6shbV2wYjKcvH9E9DfffjDyzf2nYpMW5ATtW9Yc3IZDQnQPCisbnn8w,,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,roRcBDTE_0WfDaNbaZQHOLuUibpViYYsxTASqKxs00koyMPlVfwJ5FSnTWGO_lCd0KDPKLd5hYD0Znwty09Lbe0tVbAOy8xbq9orW1KjTA,,&typo=1&ancr_add=1>) ~[artemis-commons-2.27.1.jar:?] INFO [org.apache.activemq.artemis] AMQ241005: Stopping embedded web server INFO [io.hawt.web.auth.AuthenticationFilter] Destroying hawtio authentication filter INFO [io.hawt.HawtioContextListener] Destroying hawtio services INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin INFO [org.apache.activemq.artemis] AMQ241006: Stopped embedded web server INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.27.1 [f22970c9-bdd1-11ed-bdc3-22fe978b5405] stopped, uptime 2 hours 13 minutes Upon startup we are seeing this WARN msg, not sure what it means but it may have something to do with this issue. Maybe we have something miss-configured and that is why we are seeing this issues? WARN [org.apache.activemq.artemis.core.server] AMQ222269: Please use a fixed value for "journal-pool-files". Default changed per https://issues.apache.org/jira/browse/ARTEMIS-1628 Thanks john <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,ryr8RBuJZl5w30UbOu1jQGSaCBE9mAcv23icM_mUj4uN5XfeOB0RXxxSWInCDHaLuGZ2lGFgmkKR3mrIheifGXQn1mH8zbW2CtBak-BMTTELEvU,&typo=1> John Lilley <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,ryr8RBuJZl5w30UbOu1jQGSaCBE9mAcv23icM_mUj4uN5XfeOB0RXxxSWInCDHaLuGZ2lGFgmkKR3mrIheifGXQn1mH8zbW2CtBak-BMTTELEvU,&typo=1> Data Management Chief Architect, Redpoint Global Inc. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,ryr8RBuJZl5w30UbOu1jQGSaCBE9mAcv23icM_mUj4uN5XfeOB0RXxxSWInCDHaLuGZ2lGFgmkKR3mrIheifGXQn1mH8zbW2CtBak-BMTTELEvU,&typo=1> 888 Worcester Street, Suite 200 Wellesley, MA 02482 <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,ryr8RBuJZl5w30UbOu1jQGSaCBE9mAcv23icM_mUj4uN5XfeOB0RXxxSWInCDHaLuGZ2lGFgmkKR3mrIheifGXQn1mH8zbW2CtBak-BMTTELEvU,&typo=1> M: +1 7209385761 | john.lil...@redpointglobal.com <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,ryr8RBuJZl5w30UbOu1jQGSaCBE9mAcv23icM_mUj4uN5XfeOB0RXxxSWInCDHaLuGZ2lGFgmkKR3mrIheifGXQn1mH8zbW2CtBak-BMTTELEvU,&typo=1> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,ryr8RBuJZl5w30UbOu1jQGSaCBE9mAcv23icM_mUj4uN5XfeOB0RXxxSWInCDHaLuGZ2lGFgmkKR3mrIheifGXQn1mH8zbW2CtBak-BMTTELEvU,&typo=1> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,ryr8RBuJZl5w30UbOu1jQGSaCBE9mAcv23icM_mUj4uN5XfeOB0RXxxSWInCDHaLuGZ2lGFgmkKR3mrIheifGXQn1mH8zbW2CtBak-BMTTELEvU,&typo=1> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
