We have now upgraded one environment to 2.20.0 and notice some difference in the logs. It might be pointing to a certain queue that receives over 100 kb sized messages. 2.20.0 is still crashing in similar fashion. There's a huge IOWAIT reported from our monitoring tools during the crash/restart and wrapper.log is filled with hundreds, if not thousands, of errors like this, all during 1-2 second time period:
INFO | jvm 2 | 2022/11/23 08:02:12 | WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 90a07ca5-6a2e-11ed-9aa1-767acc7119a2 INFO | jvm 2 | 2022/11/23 08:02:12 | WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 90a07ca5-6a2e-11ed-9aa1-767acc7119a2 INFO | jvm 2 | 2022/11/23 08:02:12 | WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 90ce4367-6a2e-11ed-9aa1-767acc7119a2 INFO | jvm 2 | 2022/11/23 08:02:12 | WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 90ce4367-6a2e-11ed-9aa1-767acc7119a2 On Tue, Nov 8, 2022 at 2:27 PM Thomas Wahlberg <thomas.wahlb...@nextup.fi> wrote: > Thanks for your reply. We can indeed update to 2.20.0 and we're moving > forward with this now. Newer versions seemed to have new possible issues so > we'll try with 2.20.0 for starters. > > On Tue, Nov 8, 2022 at 12:52 PM Roskvist Anton <anton.roskv...@volvo.com> > wrote: > >> Hello Thomas, >> >> I might be mistaken but I believe this has been fixed in broker version >> 2.20.0... it might be a good idea to update the broker version regardless >> since there has been quite a few bug fixes, added features and improvements >> made since 2.19.0. Be sure to follow the "Versions" documentation if >> upgrading: >> https://activemq.apache.org/components/artemis/documentation/latest/versions.html >> >> If that's not an option you could try to set the >> "critical-analyzer-policy" to "LOG" instead to keep it from killing the >> broker. This has the drawback that if there are any actual critical errors >> in the broker it will not self-terminate. Also note that I am just another >> user of the broker so any advise regarding this should be taken with a >> spoonful of "read the docs ". >> >> Br, >> Anton >> >> -----Original Message----- >> From: Thomas Wahlberg <thomas.wahlb...@nextup.fi> >> Sent: den 8 november 2022 11:27 >> To: users@activemq.apache.org >> Subject: Artemis: Artemis crashing and restarting randomly due to IO >> timeout >> >> [You don't often get email from thomas.wahlb...@nextup.fi. Learn why >> this is important at https://aka.ms/LearnAboutSenderIdentification ] >> >> CAUTION: This email originated from outside of the organization. If >> suspicious, please report it. >> >> Hi, >> we're using artemis 2.19.0 and we've noticed this is happening randomly. >> Sometimes it happens twice per day, sometimes once a month. It doesn't >> seem to be related to any cron jobs or any server load. Sometimes during >> the night, sometimes during the day. We're using Python with Artemis. The >> server monitoring tools don't report anything special regarding IO or >> IOWAIT during the time of the crash. Can you help us in any way? Any >> pointers on what to look for? >> >> Thank you in advance! >> >> Blow are some details: >> Artemis: 2.19.0 >> OS: Centos 7 >> openjdk version "1.8.0_131" >> OpenJDK Runtime Environment (build 1.8.0_131-b11) OpenJDK 64-Bit Server >> VM (build 25.131-b11, mixed mode) >> >> INFO | jvm 6 | ***************** | WARN >> [org.apache.activemq.artemis.utils.critical.CriticalMeasure] Component >> org.apache.activemq.artemis.core.io.buffer.TimedBuffer is expired on path 4 >> INFO | jvm 6 | ***************** | ERROR >> [org.apache.activemq.artemis.core.server] AMQ224079: The process for the >> virtual machine will be killed, as component >> org.apache.activemq.artemis.core.io.buffer.TimedBuffer@726de316 is not >> responsive >> INFO | jvm 6 | ***************** | WARN >> [org.apache.activemq.artemis.core.server] AMQ222199: Thread dump: >> >> ******************************************************************************* >> INFO | jvm 6 | ***************** | Complete Thread dump >> INFO | jvm 6 | ***************** | "Reference Handler" Id=2 RUNNABLE >> INFO | jvm 6 | ***************** | at >> java.base@11.0.14 >> /java.lang.ref.Reference.waitForReferencePendingList(Native >> Method) >> INFO | jvm 6 | ***************** | at java.base@11.0.14 >> /java.lang.ref.Reference.processPendingReferences(Reference.java:241) >> INFO | jvm 6 | ***************** | at java.base@11.0.14 >> /java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213) >> INFO | jvm 6 | ***************** | >> >> ... >> >> INFO | jvm 6 | ***************** | >> INFO | jvm 6 | ***************** | >> >> =============================================================================== >> INFO | jvm 6 | ***************** | End Thread dump >> INFO | jvm 6 | ***************** | >> >> ******************************************************************************* >> INFO | jvm 6 | ***************** | >> ERROR | wrapper | ***************** | JVM exited unexpectedly. >> STATUS | wrapper | ***************** | Reloading Wrapper configuration... >> STATUS | wrapper | ***************** | Launching a JVM... >> >> -- >> >> Thomas Wahlberg >> Chief Technology Officer >> Quriiri / NextUp Oy >> Puh. +358407031990 >> thomas.wahlb...@quriiri.fi >> >> This email message (including its attachments) is confidential and may >> contain privileged information and is intended solely for the use of the >> individual and/or entity to whom it is addressed. If you are not the >> intended recipient of this e-mail you may not disseminate, distribute or >> copy this e-mail (including its attachments), or any part thereof. If this >> e-mail is received in error, please notify the sender immediately by return >> e-mail and make sure that this e-mail (including its attachments), and all >> copies thereof, are immediately deleted from your system. Please further >> note that when you communicate with us via email or visit our website we >> process your personal data. See our privacy policy for more information >> about how we process it: https://www.volvogroup.com/en-en/privacy.html >> > > > -- > > Thomas Wahlberg > Chief Technology Officer > Quriiri / NextUp Oy > Puh. +358407031990 > thomas.wahlb...@quriiri.fi > -- Thomas Wahlberg Chief Technology Officer Quriiri / NextUp Oy Puh. +358407031990 thomas.wahlb...@quriiri.fi
