this could be ARTEMIS-4084: this commit, which is not part of a
release yet. We should make 2.27.1 next week:
I'm in vacation mode this week due to the US Thanksgiving, if nobody
sends a release this week I will cut a release on monday.
commit 03b82142eb0844b9de02ca3d7ed365d849e3ac02
Author: Clebert Suconic <clebertsuco...@apache.org>
Date: Tue Nov 8 09:42:16 2022 -0500
ARTEMIS-4084 Fixing addSorted with large transactions
when cancelling a large number of messages, the addSorted could be
holding a lock for too long causing the server to crash under
CriticalAnalyzer
co-authored: AntonRoskvist <anton.roskv...@volvo.com> (discovering
the issue and providing the test ClientCrashMassiveRollbackTest.java)
On Wed, Nov 23, 2022 at 9:16 AM Clebert Suconic
<clebert.suco...@gmail.com> wrote:
>
> It was said your issue was fixed in 2.20. But it was not said to move to
> that version.
>
>
> When you upgrade always go to the latest.
>
>
> On Wed, Nov 23, 2022 at 8:53 AM Thomas Wahlberg <thomas.wahlb...@nextup.fi>
> wrote:
>>
>> Hi,
>> it was first recommendation here. Second, I found some troubling posts
>> about missing messages in newer versions, not sure if they're real problems
>> or not. Third, 2.19 to 2.20 didn't require us to do any other configuration
>> updates, which newer version might (?) require
>>
>> On Wed, Nov 23, 2022 at 3:44 PM Clebert Suconic <clebert.suco...@gmail.com>
>> wrote:
>>
>> > Why did you move to 2.20 and not the latest ?
>> >
>> >
>> > On Wed, Nov 23, 2022 at 2:37 AM Thomas Wahlberg <thomas.wahlb...@nextup.fi
>> > >
>> > wrote:
>> >
>> > > We have now upgraded one environment to 2.20.0 and notice some difference
>> > > in the logs. It might be pointing to a certain queue that receives over
>> > 100
>> > > kb sized messages. 2.20.0 is still crashing in similar fashion. There's a
>> > > huge IOWAIT reported from our monitoring tools during the crash/restart
>> > and
>> > > wrapper.log is filled with hundreds, if not thousands, of errors like
>> > this,
>> > > all during 1-2 second time period:
>> > >
>> > > INFO | jvm 2 | 2022/11/23 08:02:12 | WARN
>> > > [org.apache.activemq.artemis.core.server] AMQ222061: Client connection
>> > > failed, clearing up resources for session
>> > > 90a07ca5-6a2e-11ed-9aa1-767acc7119a2
>> > > INFO | jvm 2 | 2022/11/23 08:02:12 | WARN
>> > > [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up
>> > resources
>> > > for session 90a07ca5-6a2e-11ed-9aa1-767acc7119a2
>> > > INFO | jvm 2 | 2022/11/23 08:02:12 | WARN
>> > > [org.apache.activemq.artemis.core.server] AMQ222061: Client connection
>> > > failed, clearing up resources for session
>> > > 90ce4367-6a2e-11ed-9aa1-767acc7119a2
>> > > INFO | jvm 2 | 2022/11/23 08:02:12 | WARN
>> > > [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up
>> > resources
>> > > for session 90ce4367-6a2e-11ed-9aa1-767acc7119a2
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Nov 8, 2022 at 2:27 PM Thomas Wahlberg <
>> > thomas.wahlb...@nextup.fi>
>> > > wrote:
>> > >
>> > > > Thanks for your reply. We can indeed update to 2.20.0 and we're moving
>> > > > forward with this now. Newer versions seemed to have new possible
>> > issues
>> > > so
>> > > > we'll try with 2.20.0 for starters.
>> > > >
>> > > > On Tue, Nov 8, 2022 at 12:52 PM Roskvist Anton <
>> > anton.roskv...@volvo.com
>> > > >
>> > > > wrote:
>> > > >
>> > > >> Hello Thomas,
>> > > >>
>> > > >> I might be mistaken but I believe this has been fixed in broker
>> > version
>> > > >> 2.20.0... it might be a good idea to update the broker version
>> > > regardless
>> > > >> since there has been quite a few bug fixes, added features and
>> > > improvements
>> > > >> made since 2.19.0. Be sure to follow the "Versions" documentation if
>> > > >> upgrading:
>> > > >>
>> > >
>> > https://activemq.apache.org/components/artemis/documentation/latest/versions.html
>> > > >>
>> > > >> If that's not an option you could try to set the
>> > > >> "critical-analyzer-policy" to "LOG" instead to keep it from killing
>> > the
>> > > >> broker. This has the drawback that if there are any actual critical
>> > > errors
>> > > >> in the broker it will not self-terminate. Also note that I am just
>> > > another
>> > > >> user of the broker so any advise regarding this should be taken with a
>> > > >> spoonful of "read the docs ".
>> > > >>
>> > > >> Br,
>> > > >> Anton
>> > > >>
>> > > >> -----Original Message-----
>> > > >> From: Thomas Wahlberg <thomas.wahlb...@nextup.fi>
>> > > >> Sent: den 8 november 2022 11:27
>> > > >> To: users@activemq.apache.org
>> > > >> Subject: Artemis: Artemis crashing and restarting randomly due to IO
>> > > >> timeout
>> > > >>
>> > > >> [You don't often get email from thomas.wahlb...@nextup.fi. Learn why
>> > > >> this is important at https://aka.ms/LearnAboutSenderIdentification ]
>> > > >>
>> > > >> CAUTION: This email originated from outside of the organization. If
>> > > >> suspicious, please report it.
>> > > >>
>> > > >> Hi,
>> > > >> we're using artemis 2.19.0 and we've noticed this is happening
>> > randomly.
>> > > >> Sometimes it happens twice per day, sometimes once a month. It doesn't
>> > > >> seem to be related to any cron jobs or any server load. Sometimes
>> > during
>> > > >> the night, sometimes during the day. We're using Python with Artemis.
>> > > The
>> > > >> server monitoring tools don't report anything special regarding IO or
>> > > >> IOWAIT during the time of the crash. Can you help us in any way? Any
>> > > >> pointers on what to look for?
>> > > >>
>> > > >> Thank you in advance!
>> > > >>
>> > > >> Blow are some details:
>> > > >> Artemis: 2.19.0
>> > > >> OS: Centos 7
>> > > >> openjdk version "1.8.0_131"
>> > > >> OpenJDK Runtime Environment (build 1.8.0_131-b11) OpenJDK 64-Bit
>> > Server
>> > > >> VM (build 25.131-b11, mixed mode)
>> > > >>
>> > > >> INFO | jvm 6 | ***************** | WARN
>> > > >> [org.apache.activemq.artemis.utils.critical.CriticalMeasure]
>> > Component
>> > > >> org.apache.activemq.artemis.core.io.buffer.TimedBuffer is expired on
>> > > path 4
>> > > >> INFO | jvm 6 | ***************** | ERROR
>> > > >> [org.apache.activemq.artemis.core.server] AMQ224079: The process for
>> > the
>> > > >> virtual machine will be killed, as component
>> > > >> org.apache.activemq.artemis.core.io.buffer.TimedBuffer@726de316 is
>> > not
>> > > >> responsive
>> > > >> INFO | jvm 6 | ***************** | WARN
>> > > >> [org.apache.activemq.artemis.core.server] AMQ222199: Thread dump:
>> > > >>
>> > > >>
>> > >
>> > *******************************************************************************
>> > > >> INFO | jvm 6 | ***************** | Complete Thread dump
>> > > >> INFO | jvm 6 | ***************** | "Reference Handler" Id=2
>> > > RUNNABLE
>> > > >> INFO | jvm 6 | ***************** | at
>> > > >> java.base@11.0.14
>> > > >> /java.lang.ref.Reference.waitForReferencePendingList(Native
>> > > >> Method)
>> > > >> INFO | jvm 6 | ***************** | at java.base@11.0.14
>> > > >> /java.lang.ref.Reference.processPendingReferences(Reference.java:241)
>> > > >> INFO | jvm 6 | ***************** | at java.base@11.0.14
>> > > >> /java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
>> > > >> INFO | jvm 6 | ***************** |
>> > > >>
>> > > >> ...
>> > > >>
>> > > >> INFO | jvm 6 | ***************** |
>> > > >> INFO | jvm 6 | ***************** |
>> > > >>
>> > > >>
>> > >
>> > ===============================================================================
>> > > >> INFO | jvm 6 | ***************** | End Thread dump
>> > > >> INFO | jvm 6 | ***************** |
>> > > >>
>> > > >>
>> > >
>> > *******************************************************************************
>> > > >> INFO | jvm 6 | ***************** |
>> > > >> ERROR | wrapper | ***************** | JVM exited unexpectedly.
>> > > >> STATUS | wrapper | ***************** | Reloading Wrapper
>> > > configuration...
>> > > >> STATUS | wrapper | ***************** | Launching a JVM...
>> > > >>
>> > > >> --
>> > > >>
>> > > >> Thomas Wahlberg
>> > > >> Chief Technology Officer
>> > > >> Quriiri / NextUp Oy
>> > > >> Puh. +358407031990
>> > > >> thomas.wahlb...@quriiri.fi
>> > > >>
>> > > >> This email message (including its attachments) is confidential and may
>> > > >> contain privileged information and is intended solely for the use of
>> > the
>> > > >> individual and/or entity to whom it is addressed. If you are not the
>> > > >> intended recipient of this e-mail you may not disseminate, distribute
>> > or
>> > > >> copy this e-mail (including its attachments), or any part thereof. If
>> > > this
>> > > >> e-mail is received in error, please notify the sender immediately by
>> > > return
>> > > >> e-mail and make sure that this e-mail (including its attachments), and
>> > > all
>> > > >> copies thereof, are immediately deleted from your system. Please
>> > further
>> > > >> note that when you communicate with us via email or visit our website
>> > we
>> > > >> process your personal data. See our privacy policy for more
>> > information
>> > > >> about how we process it:
>> > https://www.volvogroup.com/en-en/privacy.html
>> > > >>
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > Thomas Wahlberg
>> > > > Chief Technology Officer
>> > > > Quriiri / NextUp Oy
>> > > > Puh. +358407031990
>> > > > thomas.wahlb...@quriiri.fi
>> > > >
>> > >
>> > >
>> > > --
>> > >
>> > > Thomas Wahlberg
>> > > Chief Technology Officer
>> > > Quriiri / NextUp Oy
>> > > Puh. +358407031990
>> > > thomas.wahlb...@quriiri.fi
>> > >
>> > --
>> > Clebert Suconic
>> >
>>
>>
>> --
>>
>> Thomas Wahlberg
>> Chief Technology Officer
>> Quriiri / NextUp Oy
>> Puh. +358407031990
>> thomas.wahlb...@quriiri.fi
>
> --
> Clebert Suconic
--
Clebert Suconic
