@Clebert: Thanks for finding this. Here is some more evidence for you. This is only one monkey on my back right now, but I do plan on upgrading. I'll get you some feedback for sure. If I can manage, I want to try reproducing this in a lower environment.
[image: image.png] On Tue, Jun 7, 2022 at 5:07 PM Clebert Suconic <clebert.suco...@gmail.com> wrote: > @Ben Warrick: BTW When I said "you are likely" hitting the bug, I was > just using polite words... > > > I am 100% sure you are hitting that condition by creating one session > per message. You should definitely look to upgrade or cherry-pick the > fix to your own branch. > > On Tue, Jun 7, 2022 at 3:14 PM Clebert Suconic > <clebert.suco...@gmail.com> wrote: > > > > The release candidate : > > > > > > https://dist.apache.org/repos/dist/dev/activemq/activemq-artemis/2.23.0 > > > > I would appreciate your feedback here or even better on the dev list at > the voting thread. > > > > On Tue, Jun 7, 2022 at 12:52 PM Clebert Suconic < > clebert.suco...@gmail.com> wrote: > >> > >> right, but since I'm just releasing it would be great to hear his > >> feedback on top of the release candidate itself. > >> > >> On Tue, Jun 7, 2022 at 12:38 PM Justin Bertram <jbert...@apache.org> > wrote: > >> > > >> > A snapshot [1] is always a possibility as well. > >> > > >> > > >> > Justin > >> > > >> > [1] > >> > > https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-artemis/2.23.0-SNAPSHOT/ > >> > > >> > On Tue, Jun 7, 2022 at 11:34 AM Clebert Suconic < > clebert.suco...@gmail.com> > >> > wrote: > >> > > >> > > I am preparing artemis 2.23.0 at the moment... > >> > > > >> > > The way a release works in apache, you send it for vote and it will > be > >> > > officially released after 72 hours of a passing vote. but you should > >> > > have a candidate release today that you can try it out. Based on the > >> > > current diffs it should be fairly safe for you to upgrade. > >> > > > >> > > 2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will > require > >> > > JDK 11 minimal. > >> > > > >> > > > >> > > I will send you a link to the Candidate release as soon as it's > ready > >> > > on this thread. > >> > > > >> > > On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick < > horseatingwe...@gmail.com> > >> > > wrote: > >> > > > > >> > > > Hi Clebert, > >> > > > > >> > > > Yeah, I can use whatever I want after I prove it out in lower > >> > > environments. > >> > > > > >> > > > Thanks, > >> > > > > >> > > > Ben > >> > > > > >> > > > On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic < > >> > > clebert.suco...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > Can you use jdk 11? > >> > > > > > >> > > > > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic < > >> > > clebert.suco...@gmail.com> > >> > > > > wrote: > >> > > > > > >> > > > > > You are likely hitting the bug I fixed at: > >> > > > > > > >> > > > > > https://github.com/apache/activemq-artemis/pull/4098 > >> > > > > > > >> > > > > > I introduced a Thread Local Leakage. I thought by setting > >> > > > > > threadLocal.set(null) I would be clearing it... and in fact I > just > >> > > > > > updated the threadLocal variable with null. > >> > > > > > > >> > > > > > That ThreadLocal is created for every new session. > >> > > > > > > >> > > > > > > >> > > > > > I was going to start a release today but I couldn't start it > today as > >> > > > > > I'm a bit sick today. I should be able to start a release > tomorrow > >> > > and > >> > > > > > then you could try it out. > >> > > > > > > >> > > > > > > >> > > > > > your other option would be to cherry-pick the change and > build the > >> > > > > > version yourself so you can run it in production. > >> > > > > > > >> > > > > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro < > nigro....@gmail.com> > >> > > > > > wrote: > >> > > > > > > > >> > > > > > > Some more info about it here > >> > > https://netty.io/wiki/tcp-fast-open.html > >> > > > > > > > >> > > > > > > Given that Netty can makes use of it and Artemis can as > well: if > >> > > it's > >> > > > > not > >> > > > > > > present in the doc it means we need to expose it in the > acceptor > >> > > > > > > configuration (for epoll) and...we love contributions :) > >> > > > > > > But I suggest first to give it a shot first in a test env > to see > >> > > if it > >> > > > > > > solves your problem! > >> > > > > > > > >> > > > > > > Il ven 3 giu 2022, 22:21 Ben Warrick < > horseatingwe...@gmail.com> > >> > > ha > >> > > > > > scritto: > >> > > > > > > > >> > > > > > > > Hi Francesco, > >> > > > > > > > > >> > > > > > > > I don't find anything in the Artemis docs about > TCP_FAST_OPEN. It > >> > > > > must > >> > > > > > not > >> > > > > > > > be an exposed configuration on Artemis? Where do I find > more on > >> > > this? > >> > > > > > > > > >> > > > > > > > Thanks, > >> > > > > > > > > >> > > > > > > > Ben > >> > > > > > > > > >> > > > > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro < > >> > > nigro....@gmail.com> > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Are coming, not commenting: typo introduced by the the > phone, > >> > > > > sorry! > >> > > > > > > > > > >> > > > > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro < > nigro....@gmail.com> > >> > > ha > >> > > > > > > > scritto: > >> > > > > > > > > > >> > > > > > > > > > Hi Ben, > >> > > > > > > > > > > >> > > > > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case > >> > > > > connections > >> > > > > > are > >> > > > > > > > > > commenting from the came clients (I don't remember if > is an > >> > > > > exposes > >> > > > > > > > Netty > >> > > > > > > > > > configuration) or...submit a PR to let Artemis > separate > >> > > > > (multiple) > >> > > > > > > > > threads > >> > > > > > > > > > to accept connections but, as you said, this is just > an > >> > > > > > anti-pattern > >> > > > > > > > and > >> > > > > > > > > in > >> > > > > > > > > > order to reduce the client code you can think about > using a > >> > > > > > transparent > >> > > > > > > > > > connection pooling mechanism, instead. > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick < > >> > > horseatingwe...@gmail.com> > >> > > > > > ha > >> > > > > > > > > > scritto: > >> > > > > > > > > > > >> > > > > > > > > >> I'm running Artemis 2.14, and one of the systems > publishing > >> > > to > >> > > > > it > >> > > > > > > > > creates > >> > > > > > > > > >> a > >> > > > > > > > > >> lot of separate tasks -- from its perspective. I can > see on > >> > > > > > Artemis > >> > > > > > > > that > >> > > > > > > > > >> the Total connection count vs Total messages > acknowledged is > >> > > > > > pretty > >> > > > > > > > > close, > >> > > > > > > > > >> so it's apparent that the client system is making > use of the > >> > > > > > > > > anti-pattern > >> > > > > > > > > >> of creating a new connection for each message it > publishes. > >> > > I > >> > > > > > don't > >> > > > > > > > have > >> > > > > > > > > >> access to the client system implementation, but is > there > >> > > > > anything > >> > > > > > I > >> > > > > > > > can > >> > > > > > > > > do > >> > > > > > > > > >> on the Artemis end to make it more efficient? > Latency with > >> > > this > >> > > > > > system > >> > > > > > > > > is > >> > > > > > > > > >> becoming an issue. Also every 30 minutes or so there > is a > >> > > spike > >> > > > > > in CPU > >> > > > > > > > > >> usage (~50%) which I assume is garbage collection. > During > >> > > that > >> > > > > > spike > >> > > > > > > > > >> message publishing slows down a lot. > >> > > > > > > > > >> > >> > > > > > > > > >> Thanks, > >> > > > > > > > > >> > >> > > > > > > > > >> Ben > >> > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > Clebert Suconic > >> > > > > > > >> > > > > -- > >> > > > > Clebert Suconic > >> > > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > Clebert Suconic > >> > > > >> > > > >> > >> > >> > >> -- > >> Clebert Suconic > > > > -- > > Clebert Suconic > > > > -- > Clebert Suconic >
