@Ben Warrick: BTW When I said "you are likely" hitting the bug, I was just using polite words...
I am 100% sure you are hitting that condition by creating one session per message. You should definitely look to upgrade or cherry-pick the fix to your own branch. On Tue, Jun 7, 2022 at 3:14 PM Clebert Suconic <clebert.suco...@gmail.com> wrote: > > The release candidate : > > > https://dist.apache.org/repos/dist/dev/activemq/activemq-artemis/2.23.0 > > I would appreciate your feedback here or even better on the dev list at the > voting thread. > > On Tue, Jun 7, 2022 at 12:52 PM Clebert Suconic <clebert.suco...@gmail.com> > wrote: >> >> right, but since I'm just releasing it would be great to hear his >> feedback on top of the release candidate itself. >> >> On Tue, Jun 7, 2022 at 12:38 PM Justin Bertram <jbert...@apache.org> wrote: >> > >> > A snapshot [1] is always a possibility as well. >> > >> > >> > Justin >> > >> > [1] >> > https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-artemis/2.23.0-SNAPSHOT/ >> > >> > On Tue, Jun 7, 2022 at 11:34 AM Clebert Suconic <clebert.suco...@gmail.com> >> > wrote: >> > >> > > I am preparing artemis 2.23.0 at the moment... >> > > >> > > The way a release works in apache, you send it for vote and it will be >> > > officially released after 72 hours of a passing vote. but you should >> > > have a candidate release today that you can try it out. Based on the >> > > current diffs it should be fairly safe for you to upgrade. >> > > >> > > 2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will require >> > > JDK 11 minimal. >> > > >> > > >> > > I will send you a link to the Candidate release as soon as it's ready >> > > on this thread. >> > > >> > > On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick <horseatingwe...@gmail.com> >> > > wrote: >> > > > >> > > > Hi Clebert, >> > > > >> > > > Yeah, I can use whatever I want after I prove it out in lower >> > > environments. >> > > > >> > > > Thanks, >> > > > >> > > > Ben >> > > > >> > > > On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic < >> > > clebert.suco...@gmail.com> >> > > > wrote: >> > > > >> > > > > Can you use jdk 11? >> > > > > >> > > > > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic < >> > > clebert.suco...@gmail.com> >> > > > > wrote: >> > > > > >> > > > > > You are likely hitting the bug I fixed at: >> > > > > > >> > > > > > https://github.com/apache/activemq-artemis/pull/4098 >> > > > > > >> > > > > > I introduced a Thread Local Leakage. I thought by setting >> > > > > > threadLocal.set(null) I would be clearing it... and in fact I just >> > > > > > updated the threadLocal variable with null. >> > > > > > >> > > > > > That ThreadLocal is created for every new session. >> > > > > > >> > > > > > >> > > > > > I was going to start a release today but I couldn't start it today >> > > > > > as >> > > > > > I'm a bit sick today. I should be able to start a release tomorrow >> > > and >> > > > > > then you could try it out. >> > > > > > >> > > > > > >> > > > > > your other option would be to cherry-pick the change and build the >> > > > > > version yourself so you can run it in production. >> > > > > > >> > > > > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro >> > > > > > <nigro....@gmail.com> >> > > > > > wrote: >> > > > > > > >> > > > > > > Some more info about it here >> > > https://netty.io/wiki/tcp-fast-open.html >> > > > > > > >> > > > > > > Given that Netty can makes use of it and Artemis can as well: if >> > > it's >> > > > > not >> > > > > > > present in the doc it means we need to expose it in the acceptor >> > > > > > > configuration (for epoll) and...we love contributions :) >> > > > > > > But I suggest first to give it a shot first in a test env to see >> > > if it >> > > > > > > solves your problem! >> > > > > > > >> > > > > > > Il ven 3 giu 2022, 22:21 Ben Warrick <horseatingwe...@gmail.com> >> > > ha >> > > > > > scritto: >> > > > > > > >> > > > > > > > Hi Francesco, >> > > > > > > > >> > > > > > > > I don't find anything in the Artemis docs about TCP_FAST_OPEN. >> > > > > > > > It >> > > > > must >> > > > > > not >> > > > > > > > be an exposed configuration on Artemis? Where do I find more on >> > > this? >> > > > > > > > >> > > > > > > > Thanks, >> > > > > > > > >> > > > > > > > Ben >> > > > > > > > >> > > > > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro < >> > > nigro....@gmail.com> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Are coming, not commenting: typo introduced by the the phone, >> > > > > sorry! >> > > > > > > > > >> > > > > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro >> > > > > > > > > <nigro....@gmail.com> >> > > ha >> > > > > > > > scritto: >> > > > > > > > > >> > > > > > > > > > Hi Ben, >> > > > > > > > > > >> > > > > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case >> > > > > connections >> > > > > > are >> > > > > > > > > > commenting from the came clients (I don't remember if is an >> > > > > exposes >> > > > > > > > Netty >> > > > > > > > > > configuration) or...submit a PR to let Artemis separate >> > > > > (multiple) >> > > > > > > > > threads >> > > > > > > > > > to accept connections but, as you said, this is just an >> > > > > > anti-pattern >> > > > > > > > and >> > > > > > > > > in >> > > > > > > > > > order to reduce the client code you can think about using a >> > > > > > transparent >> > > > > > > > > > connection pooling mechanism, instead. >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick < >> > > horseatingwe...@gmail.com> >> > > > > > ha >> > > > > > > > > > scritto: >> > > > > > > > > > >> > > > > > > > > >> I'm running Artemis 2.14, and one of the systems >> > > > > > > > > >> publishing >> > > to >> > > > > it >> > > > > > > > > creates >> > > > > > > > > >> a >> > > > > > > > > >> lot of separate tasks -- from its perspective. I can see >> > > > > > > > > >> on >> > > > > > Artemis >> > > > > > > > that >> > > > > > > > > >> the Total connection count vs Total messages acknowledged >> > > > > > > > > >> is >> > > > > > pretty >> > > > > > > > > close, >> > > > > > > > > >> so it's apparent that the client system is making use of >> > > > > > > > > >> the >> > > > > > > > > anti-pattern >> > > > > > > > > >> of creating a new connection for each message it >> > > > > > > > > >> publishes. >> > > I >> > > > > > don't >> > > > > > > > have >> > > > > > > > > >> access to the client system implementation, but is there >> > > > > anything >> > > > > > I >> > > > > > > > can >> > > > > > > > > do >> > > > > > > > > >> on the Artemis end to make it more efficient? Latency with >> > > this >> > > > > > system >> > > > > > > > > is >> > > > > > > > > >> becoming an issue. Also every 30 minutes or so there is a >> > > spike >> > > > > > in CPU >> > > > > > > > > >> usage (~50%) which I assume is garbage collection. During >> > > that >> > > > > > spike >> > > > > > > > > >> message publishing slows down a lot. >> > > > > > > > > >> >> > > > > > > > > >> Thanks, >> > > > > > > > > >> >> > > > > > > > > >> Ben >> > > > > > > > > >> >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > Clebert Suconic >> > > > > > >> > > > > -- >> > > > > Clebert Suconic >> > > > > >> > > >> > > >> > > >> > > -- >> > > Clebert Suconic >> > > >> > > >> >> >> >> -- >> Clebert Suconic > > -- > Clebert Suconic -- Clebert Suconic
