Hi Shiv, you could enable network tracing for .NET, see https://learn.microsoft.com/en-us/dotnet/framework/network-programming/how-to-configure-network-tracing
Regards, Domenico On Thu, 17 Apr 2025 at 09:17, Shiv Kumar Dixit <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > Hi Justin, > Thanks for input. > > "Regarding the underlying problem...Have you explored the possibility of > having the client itself somehow report that it's having a problem > connecting rather than trying to work backwards from the broker to the > client?" > > Most of our customers are OpenWire based (Apache NMS) and uses > failover-based connection string. They just get very generic error like > below. They have transport.timeout=5000 in broker failover URI. So, they > report connection problem but they wont get any concrete error info about > the underlying problem. > > Failover oneway timed out after 5000 milliseconds. > at > Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.Oneway(Command > command) > at Apache.NMS.ActiveMQ.Transport.MutexTransport.Oneway(Command command) > at > Apache.NMS.ActiveMQ.Transport.ResponseCorrelator.AsyncRequest(Command > command) > at Apache.NMS.ActiveMQ.Transport.ResponseCorrelator.Request(Command > command, TimeSpan timeout) > at Apache.NMS.ActiveMQ.Connection.CheckConnected() > at Apache.NMS.ActiveMQ.Connection.Start() > at > com.test.TestSender.TestMessageSender.SendMessage(CreateMessageContext > messageContext, Boolean isForTopic, Int32 priority)." > > Best Regards > Shiv > > -----Original Message----- > From: Justin Bertram <jbert...@apache.org> > Sent: 16 April 2025 09:56 PM > To: users@activemq.apache.org > Subject: Re: Additional Info on certificate based authentication errors > > > > Unverified Sender: The sender of this email has not been verified. Review > the content of the message carefully and verify the identity of the sender > before acting on this email: replying, opening attachments or clicking > links. > > > > On startup it logs below message which I guess seems normal. > > That message is normal. You should see it no matter what SSL provider > you're using. > > > If OPENSSL keeps the broker unaware of such SSL issues or I need to > enable certain logging flags? > > I'm not aware of any additional logging you need to see SSL-related errors > when using OpenSSL. My guess is that this difference comes down to the > specifics of how Netty integrates with OpenSSL. > > > I had broker logging at DEBUG level along with ' set > JAVA_ARGS=%JAVA_ARGS% -Djavax.net.debug=ssl,handshake' in > artemis.profile.cmd file. > > I don't think -Djavax.net.debug=ssl,handshake will actually do anything if > you're using OpenSSL. > > > Also, I tried to download > netty-tcnative-boringssl-static-2.0.70.Final-linux-x86_64.jar but could not > do it. Can you please share its download link as well? > > All the relevant files are here [1]. > The netty-tcnative-boringssl-static-2.0.70.Final-linux-x86_64.jar file is > here [2]. > > > Regarding the underlying problem...Have you explored the possibility of > having the client itself somehow report that it's having a problem > connecting rather than trying to work backwards from the broker to the > client? > > > Justin > > [1] > > https://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.70.Final/ > [2] > > https://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.70.Final/netty-tcnative-boringssl-static-2.0.70.Final-linux-x86_64.jar > > On Wed, Apr 16, 2025 at 9:50 AM Shiv Kumar Dixit > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > Thanks Domenico. > > > > I could download both jars and placed them in my Windows laptop's > > artemis_home/lib folder. Broker started fine now with > sslProvider=OPENSSL. > > On startup it logs below message which I guess seems normal. > > 2025-04-16 12:45:10,826 INFO [org.apache.activemq.artemis.core.server] > > AMQ221053: Disallowing use of vulnerable protocol 'SSLv2Hello' on > acceptor > > 'artemis-ssl'. See > > > http://www.oracle.com/technetwork/topics/security/poodlecve-2014-3566-2339408.html > > for more details. > > > > 1. I can send message using correct certificate. > > > > 2. When I tried with invalid certificate, it threw error to client but > > there is no SSL error logged in broker. It was not the case when > > sslProvider by JDK by default. With JDK, I used to see ' AMQ222208: SSL > > handshake failed for client from /1.2.3.4:52053: > > javax.net.ssl.SSLHandshakeException: Empty server certificate chain'. Now > > it no longer coming for OPENSSL. If OPENSSL keeps the broker unaware of > > such SSL issues or I need to enable certain logging flags? > > > > 3. I had broker logging at DEBUG level along with ' set > > JAVA_ARGS=%JAVA_ARGS% -Djavax.net.debug=ssl,handshake' in > > artemis.profile.cmd file. > > > > Also, I tried to download > > netty-tcnative-boringssl-static-2.0.70.Final-linux-x86_64.jar but could > not > > do it. Can you please share its download link as well? > > > > Best Regards > > Shiv > > > > -----Original Message----- > > From: Domenico Francesco Bruscino <bruscin...@gmail.com> > > Sent: 16 April 2025 12:13 PM > > To: users@activemq.apache.org > > Subject: Re: Additional Info on certificate based authentication errors > > > > > > > > Unverified Sender: The sender of this email has not been verified. Review > > the content of the message carefully and verify the identity of the > sender > > before acting on this email: replying, opening attachments or clicking > > links. > > > > > > Hi Shiv, > > > > the netty-tcnative-boringssl-static-{os_arch} 2.0.70.Final for windows is > > netty-tcnative-boringssl-static-2.0.70.Final-windows-x86_64.jar[1] and > you > > also need netty-tcnative-classes[2]. > > > > [1] > > > > > https://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/2.0.70.Final/netty-tcnative-boringssl-static-2.0.70.Final-windows-x86_64.jar > > [2] > > > > > https://repo1.maven.org/maven2/io/netty/netty-tcnative-classes/2.0.70.Final/netty-tcnative-classes-2.0.70.Final.jar > > > > Regards, > > Domenico > > > > On Tue, 15 Apr 2025 at 14:41, Shiv Kumar Dixit > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > Hi Domenico/Justin, > > > Somehow Using OPENSSL as sslProvider seems complicated process. As per > > > documentation (https://netty.io/wiki/forked-tomcat-native.html), I > tried > > > to use below as it seems statically link approach and I don’t have to > > worry > > > about installation process. I may not have permission to install as > well. > > > > > > netty-tcnative-boringssl-static -- This is an uber jar that contains > the > > > statically linked libraries for all supported > > > netty-tcnative-boringssl-static-{os_arch}. While this tends to be a > > fairly > > > large jar, it greatly simplifies the getting started experience since > an > > > application need not be concerned with obtaining the correct jar for > the > > > platform. > > > > > > This jar is there in maven central ( > > > > > > https://mvnrepository.com/artifact/io.netty/netty-tcnative-boringssl-static/2.0.70.Final > > ) > > > but it's just 3 KB and does not look like a fat jar as per above > > > description. > > > > > > For testing, I am running my broker on a Windows laptop but it could be > > > deployed in Linux as systemctl service or a pod in K8s. Any input will > be > > > helpful. > > > > > > Best Regards > > > Shiv > > > > > > -----Original Message----- > > > From: Domenico Francesco Bruscino <bruscin...@gmail.com> > > > Sent: 14 April 2025 02:22 PM > > > To: users@activemq.apache.org > > > Subject: Re: Additional Info on certificate based authentication errors > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > Review > > > the content of the message carefully and verify the identity of the > > sender > > > before acting on this email: replying, opening attachments or clicking > > > links. > > > > > > > > > Hi Shiv, > > > > > > to use OPENSSL as sslProvider with the netty-tcnative default artifact, > > > your system must have both libapr-1 and OpenSSL installed and > configured. > > > Alternatively, you can use one of the netty-tcnative artifacts > statically > > > linked, for further details, see > > > https://netty.io/wiki/forked-tomcat-native.html > > > > > > Regards, > > > Domenico > > > > > > On Mon, 14 Apr 2025 at 10:42, Shiv Kumar Dixit > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > Hi Domenico, > > > > I am putting netty-tcnative-2.0.70.Final.jar in broker_home/lib > folder > > > > but still the same error. I am running broker on a Windows laptop. Do > > > > I need to have openssl installed as pre-condition on my laptop as the > > > > documentation says " If used with OPENSSL you can add netty-tcnative > > > > to your classpath to use the native installed openssl". So openssl > > > > installation is precondition to use OPENSSL as sslProvider? > > > > > > > > Best Regards > > > > Shiv > > > > > > > > -----Original Message----- > > > > From: Domenico Francesco Bruscino <bruscin...@gmail.com> > > > > Sent: 14 April 2025 01:52 PM > > > > To: users@activemq.apache.org > > > > Subject: Re: Additional Info on certificate based authentication > > > > errors > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > > Review the content of the message carefully and verify the identity > of > > > > the sender before acting on this email: replying, opening attachments > > > > or clicking links. > > > > > > > > > > > > Hi Shiv, > > > > > > > > you need to add `netty-tcnative` to your classpath to use the native > > > > installed openssl. > > > > > > > > Regards, > > > > Domenico > > > > > > > > On Fri, 11 Apr 2025 at 11:37, Shiv Kumar Dixit > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > Hi Justin, > > > > > I set connectionsAllowed=3 on acceptor. I then ran 3 good listener > > > apps. > > > > > It created 3 good connections and then 1 consumer per connection. I > > > > > then ran 4th good listener but broker rejected the connection. When > > > > > I ran 4th bad listener (it has wrong certificate name in its > > > > > config), it tried to create connection with broker and failed with > > > > > "javax.net.ssl.SSLHandshakeException: Empty server certificate > > chain". > > > > > So, connectionsAllowed can only restrict good connections. > > > > > > > > > > Regarding switching to OPENSSL, I set the value as per your > > > > > suggestion, but broker won't start due to below error: > > > > > 1. Do we need to copy additional jars? > > > > > 2. Do we need OpenSSL installed locally? > > > > > > > > > > 2025-04-11 14:40:56,715 ERROR > > > > > [org.apache.activemq.artemis.core.server] > > > > > AMQ224097: Failed to start server > > > > > java.lang.NoClassDefFoundError: > > > > > io/netty/internal/tcnative/SSLPrivateKeyMethod > > > > > at > > > > > > io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext. > > > > > ja > > > > > va:480) ~[netty-handler-4.1.112.Final.jar:4.1.112.Final] > > > > > at > > > > > > io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java: > > > > > 64 > > > > > 3) ~[netty-handler-4.1.112.Final.jar:4.1.112.Final] > > > > > at > > > > > > org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.create > > > > > Ne > > > > > ttyContext(SSLSupport.java:240) > > > > > ~[artemis-core-client-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.remoting.impl.ssl.DefaultOpenSSLCon > > > > > te > > > > > xtFactory.getServerSslContext(DefaultOpenSSLContextFactory.java:60) > > > > > ~[artemis-core-client-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.l > > > > > oa > > > > > dSSLContext(NettyAcceptor.java:397) > > > > > ~[artemis-server-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.< > > > > > in > > > > > it>(NettyAcceptor.java:349) > > > > > ~[artemis-server-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptorFa > > > > > ct > > > > > ory.createAcceptor(NettyAcceptorFactory.java:43) > > > > > ~[artemis-server-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServic > > > > > eI > > > > > mpl.createAcceptor(RemotingServiceImpl.java:267) > > > > > ~[artemis-server-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServic > > > > > eI > > > > > mpl.start(RemotingServiceImpl.java:210) > > > > > ~[artemis-server-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.init > > > > > ia > > > > > lisePart2(ActiveMQServerImpl.java:3560) > > > > > ~[artemis-server-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.server.impl.PrimaryOnlyActivation.r > > > > > un > > > > > (PrimaryOnlyActivation.java:78) > > > > > ~[artemis-server-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.inte > > > > > rn > > > > > alStart(ActiveMQServerImpl.java:742) > > > > > ~[artemis-server-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.star > > > > > t( > > > > > ActiveMQServerImpl.java:632) > > > > > [artemis-server-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker. > > > > > ja > > > > > va:66) > > > > > [artemis-cli-2.37.0.jar:2.37.0] > > > > > at > > > > > org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130) > > > > > [artemis-cli-2.37.0.jar:2.37.0] > > > > > at > > > > > > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java > > > > > :2 > > > > > 21) > > > > > [artemis-cli-2.37.0.jar:2.37.0] > > > > > at > > > > > org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:167) > > > > > [artemis-cli-2.37.0.jar:2.37.0] > > > > > at > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nati > > > > > ve > > > > > Method) ~[?:?] > > > > > at > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Nativ > > > > > eM > > > > > ethodAccessorImpl.java:62) > > > > > ~[?:?] > > > > > at > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(D > > > > > el > > > > > egatingMethodAccessorImpl.java:43) > > > > > ~[?:?] > > > > > at > > > > > java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > > ~[?:?] > > > > > at > > > > > org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:152) > > > > > [artemis-boot.jar:2.37.0] > > > > > at > > > > > org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64) > > > > > [artemis-boot.jar:2.37.0] > > > > > Caused by: java.lang.ClassNotFoundException: > > > > > io.netty.internal.tcnative.SSLPrivateKeyMethod > > > > > at > > > > > java.base/java.net > .URLClassLoader.findClass(URLClassLoader.java:476) > > > > > ~[?:?] > > > > > at > > > > > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594) > > > > > ~[?:?] > > > > > at > > > > > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) > > > > > ~[?:?] > > > > > ... 23 more > > > > > java.lang.NoClassDefFoundError: > > > > > io/netty/internal/tcnative/SSLPrivateKeyMethod > > > > > at > > > > > > > > > > io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.ja > > > > va:480) > > > > > at > > > > > > > > > io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:643) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.createNe > > > > ttyContext(SSLSupport.java:240) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.remoting.impl.ssl.DefaultOpenSSLConte > > > > xtFactory.getServerSslContext(DefaultOpenSSLContextFactory.java:60) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.loa > > > > dSSLContext(NettyAcceptor.java:397) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.<in > > > > it>(NettyAcceptor.java:349) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptorFact > > > > ory.createAcceptor(NettyAcceptorFactory.java:43) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceI > > > > mpl.createAcceptor(RemotingServiceImpl.java:267) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceI > > > > mpl.start(RemotingServiceImpl.java:210) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initia > > > > lisePart2(ActiveMQServerImpl.java:3560) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.server.impl.PrimaryOnlyActivation.run > > > > (PrimaryOnlyActivation.java:78) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.intern > > > > alStart(ActiveMQServerImpl.java:742) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start( > > > > ActiveMQServerImpl.java:632) > > > > > at > > > > > > > > > > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.ja > > > > va:66) > > > > > at > > > > > org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130) > > > > > at > > > > > > > > > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:221) > > > > > at > > > > > org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:167) > > > > > at > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nati > > > > > ve > > > > > Method) > > > > > at > > > > > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM > > > > ethodAccessorImpl.java:62) > > > > > at > > > > > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del > > > > egatingMethodAccessorImpl.java:43) > > > > > at > java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > > at > > > > > org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:152) > > > > > at > > > > > org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64) > > > > > Caused by: java.lang.ClassNotFoundException: > > > > > io.netty.internal.tcnative.SSLPrivateKeyMethod > > > > > at java.base/java.net > > > > > .URLClassLoader.findClass(URLClassLoader.java:476) > > > > > at > > > > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594) > > > > > at > > > > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) > > > > > ... 23 more > > > > > > > > > > Best Regards > > > > > Shiv > > > > > > > > > > -----Original Message----- > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > Sent: 11 April 2025 01:55 AM > > > > > To: users@activemq.apache.org > > > > > Subject: Re: Additional Info on certificate based authentication > > > > > errors > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > > > Review the content of the message carefully and verify the identity > > > > > of the sender before acting on this email: replying, opening > > > > > attachments or clicking links. > > > > > > > > > > > > > > > > I just defined sslProvider=openssl... > > > > > > > > > > You should be using this as noted in the documentation [1]: > > > > > > > > > > sslProvider=OPENSSL > > > > > > > > > > > Is there any log or other indication that now SSL process has > > > > > > moved from > > > > > JDK to OPENSSL? > > > > > > > > > > I couldn't find anything that logs which sslProvider is in use for > > > > > an acceptor. > > > > > > > > > > > Unfortunately, I could not find any article on using OPENSSL with > > > > > Artemis. > > > > > > > > > > I'm not aware of any specific articles. However, this documentation > > > > > [1] should be sufficient. Let me know if more details are required. > > > > > > > > > > > ...regarding connectionsAllowed on acceptor, will it protect > > > > > > against > > > > > misbehaving producer/consumer? I understood that it is only good > > > > > enough to restrict good connections. > > > > > > > > > > I'm not really sure what you're asking here. > > > > > > > > > > Technically speaking you can only create a producer or consumer > once > > > > > a connection is established so if an application creates a > > > > > connection and then uses that connection to spam create a bunch of > > > > > producers and/or consumers then connectionsAllowed will not prevent > > > that. > > > > > However, if an application is spamming connections in the process > of > > > > > producing and/or consuming messages then connectionsAllowed will > > > > > stop > > > > that. > > > > > > > > > > > > > > > Justin > > > > > > > > > > [1] > > > > > > > > > > https://ac/ > > > > > > ti%2F&data=05%7C02%7C%7Cc2dd0998e0f64808f14b08dd7b31b658%7C1a1dce202 > > > > > > 1b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638802175649312259%7CUnknown%7CTWF > > > > > > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiI > > > > > > sIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=OFzPPH8QwnopQep > > > > > MALRRY%2F7zhMygdC1c6VI9%2Ff1kA5Q%3D&reserved=0 > > > > > vemq.apache.org > %2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fco > > > > > nf > > > > > > iguring-transports.html%23configuring-netty-ssl%3F%3A~%3Atext%3DsslP > > > > > ro > > > > > > vider&data=05%7C02%7C%7C24e1fb3bd58841ee2d1108dd7b2d76ac%7C1a1dce202 > > > > > 1b > > > > > > 14beaa9d2130e9f1f6e2f%7C0%7C0%7C638802157373991188%7CUnknown%7CTWFpb > > > > > GZ > > > > > > sb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF > > > > > OI > > > > > > joiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=j1nFu5HBrb48dSJybiOGn > > > > > Lv > > > > > N1%2FSBO%2F3TRuRou5xrNgU%3D&reserved=0 > > > > > > > > > > On Thu, Apr 10, 2025 at 11:57 AM Shiv Kumar Dixit > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > Hi Justin, > > > > > > While moving from JDK to OPENSSL, I just defined > > > > > > sslProvider=openssl on the acceptor and broker seems to start > > > > > > fine. Is there any log or other indication that now SSL process > has > > > moved from JDK to OPENSSL? > > > > > > Unfortunately, I could not find any article on using OPENSSL with > > > > > Artemis. > > > > > > Any input will be useful here. > > > > > > > > > > > > Secondly regarding, "first line of defense would actually be in > > > > > > the network exposure rather than in the broker configuration", we > > > > > > have network segregation but all of the misbehaving clients come > > > > > > from trusted > > > > > network. > > > > > > One same VM, we could have 2 listeners - one has correctly > > > > > > configured the certificate details while another has wrong > > > > > > configuration. Second one is causing " > > > > > > javax.net.ssl.SSLHandshakeException: Empty server certificate > > chain" > > > > > > error on the broker. Given we have dozens of VMs connecting to > > > > > > broker cluster and each VM hosting many producers/consumers where > > > > > > both good and bad configuration can co-exist, we are defenceless > > > > > > against this kind of issue. Here only option is to find IP and > > > > > > scan producer/consumer configuration and see if they have any > > > > misconfiguration causing trouble to brokers. > > > > > > > > > > > > Thirdly, regarding connectionsAllowed on acceptor, will it > protect > > > > > > against misbehaving producer/consumer? I understood that it is > > > > > > only good enough to restrict good connections. > > > > > > > > > > > > Best Regards > > > > > > Shiv > > > > > > > > > > > > -----Original Message----- > > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > > Sent: 09 April 2025 07:28 PM > > > > > > To: users@activemq.apache.org > > > > > > Subject: Re: Additional Info on certificate based authentication > > > > > > errors > > > > > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been > verified. > > > > > > Review the content of the message carefully and verify the > > > > > > identity of the sender before acting on this email: replying, > > > > > > opening attachments or clicking links. > > > > > > > > > > > > > > > > > > > ...any article or whitepaper from community on how to handle > > > > > > > (D)DoS > > > > > > attack on Artemis broker? > > > > > > > > > > > > I'm not aware of any such whitepaper. > > > > > > > > > > > > That said, I think the first line of defense would actually be in > > > > > > the network exposure rather than in the broker configuration. In > > > > > > other words, only expose your broker to the segment of the > network > > > > > > where legitimate clients are located. You could even potentially > > > > > > use a firewall to narrow exposure down to a set of specific IP > > > addresses. > > > > > > > > > > > > As far as broker configuration is concerned I recommend you look > > > > > > at setting connectionsAllowed [1] on the relevant acceptors in > > > > > > broker.xml and take a look at resource-limits-settings [2]. > > > > > > > > > > > > And of course enforce security so that only legitimate clients > can > > > > > connect. > > > > > > > > > > > > Hope that helps! > > > > > > > > > > > > > > > > > > Justin > > > > > > > > > > > > [1] > > > > > > > > > > > > https://acti/ > > > > > > vemq.apache.org > %2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2F > > > > > > co > > > > > > nf > > > > > > > iguring-transports.html%23configuring-netty-tcp&data=05%7C02%7C%7C > > > > > > d6 > > > > > > c8 > > > > > > > 503c5b0c4c80724008dd786de948%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C > > > > > > 0% > > > > > > 7C > > > > > > > 0%7C638799135676402406%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnR > > > > > > yd > > > > > > WU > > > > > > > sIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D > > > > > > %3 > > > > > > D% > > > > > > > 7C0%7C%7C%7C&sdata=soScwxyUF6yuCvnYhnBx5vi%2BosXLSgZgNU%2BpDv7aJc4 > > > > > > %3 > > > > > > D& > > > > > > reserved=0 > > > > > > [2] > > > > > > > > > > > > https://acti/ > > > > > > vemq.apache.org > %2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2F > > > > > > re > > > > > > so > > > > > > > urce-limits.html%23resource-limits&data=05%7C02%7C%7Cd6c8503c5b0c4 > > > > > > c8 > > > > > > 07 > > > > > > > 24008dd786de948%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63879 > > > > > > 91 > > > > > > 35 > > > > > > > 676430981%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIw > > > > > > Lj > > > > > > Au > > > > > > > MDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C > > > > > > %7 > > > > > > C& > > > > > > > sdata=TveTJr9mCrsXXUKFD%2FnILP43%2FX2cQVgGIXEMqVgEcmk%3D&reserved= > > > > > > 0 > > > > > > > > > > > > On Wed, Apr 9, 2025 at 8:20 AM Shiv Kumar Dixit > > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > > > Thanks Justin for input. > > > > > > > > > > > > > > I am trying with OpenSSL and handshake-timeout=30 changes for > > now. > > > > > > > I will update how it goes. Post that I will collect thread > dumps > > > > > > > and share for further analysis. > > > > > > > > > > > > > > On the side note - if there any article or whitepaper from > > > > > > > community on how to handle (D)DoS attack on Artemis broker? > > > > > > > > > > > > > > Best Regards > > > > > > > Shiv > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > > > Sent: 08 April 2025 11:38 PM > > > > > > > To: users@activemq.apache.org > > > > > > > Subject: Re: Additional Info on certificate based > authentication > > > > > > > errors > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been > > verified. > > > > > > > Review the content of the message carefully and verify the > > > > > > > identity of the sender before acting on this email: replying, > > > > > > > opening attachments or clicking links. > > > > > > > > > > > > > > > > > > > > > > If JDK handles these handshakes as part of heap or native > > memory? > > > > > > > > > > > > > > I believe it would be part of the heap, but Netty uses direct > > > > > > > buffers so perhaps that is related. > > > > > > > > > > > > > > > Here broker became unresponsive after lot of SSL handshake > > > > > > > > errors as we > > > > > > > could see lot of 'AMQ224088: Timeout (10 seconds)... > > > > > > > > > > > > > > A high volume of SSL connections can overwhelm the hardware on > > > > > > > which the broker is running due to the CPU intensive nature of > > > > > > > the cryptographic operations as well as memory requirements. > > > > > > > This is one reason why folks use OpenSSL rather than the JDK. > > > > > > > OpenSSL is typically faster and more efficient with hardware > > > > > > > resources. It's impossible to say what's really happening based > > > > > > > on the available evidence but you may simply be hitting > > > > > > > hardware/OS limits. All the SSL connection work is being done > > > > > > > before the broker is really > > > > involved. > > > > > > > > > > > > > > In order to deal with the AMQ224088 errors you might try > setting > > > > > > > "handshake-timeout=30" on the relevant acceptor in broker.xml. > > > > > > > The default timeout is 10 seconds. > > > > > > > > > > > > > > If you want to investigate further I suggest getting a handful > > > > > > > of thread dumps during your test. Monitoring CPU utilization > > > > > > > might be helpful as well. Also, if you're exporting metrics [1] > > > > > > > (highly > > > > > > > recommended!) you can enable optional metrics for Netty [2] to > > > > > > > see details about direct and heap memory usage. > > > > > > > > > > > > > > > > > > > > > Justin > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > https://acti/ > > > > > > > vemq.apache.org > %2Fcomponents%2Fartemis%2Fdocumentation%2Flatest% > > > > > > > 2F > > > > > > > me > > > > > > > tr > > > > > > > > ics.html%23metrics&data=05%7C02%7C%7Cfb52ba57391b458763f208dd776 > > > > > > > e9 > > > > > > > fd > > > > > > > a% > > > > > > > > 7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638798039227600972% > > > > > > > 7C > > > > > > > Un > > > > > > > kn > > > > > > > > own%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIl > > > > > > > Ai > > > > > > > Oi > > > > > > > JX > > > > > > > > aW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nT41 > > > > > > > PG > > > > > > > vq > > > > > > > Oe > > > > > > > rhYBh9VfodoTIFZerwDDE0cWp79YSX5OU%3D&reserved=0 > > > > > > > [2] > > > > > > > > > > > > > > https://acti/ > > > > > > > vemq.apache.org > %2Fcomponents%2Fartemis%2Fdocumentation%2Flatest% > > > > > > > 2F > > > > > > > me > > > > > > > tr > > > > > > > > ics.html%23optional-metrics&data=05%7C02%7C%7Cfb52ba57391b458763 > > > > > > > f2 > > > > > > > 08 > > > > > > > dd > > > > > > > > 776e9fda%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387980392 > > > > > > > 27 > > > > > > > 61 > > > > > > > 38 > > > > > > > > 53%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM > > > > > > > DA > > > > > > > wM > > > > > > > CI > > > > > > > > sIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&s > > > > > > > da > > > > > > > ta > > > > > > > =C > > > > > > > cTlTKQ%2B%2F6MIGiZRIrBa3pUPD75E4phfF5vbIJD9tcE%3D&reserved=0 > > > > > > > > > > > > > > On Tue, Apr 8, 2025 at 4:14 AM Shiv Kumar Dixit > > > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > > > > > Hi Justin, > > > > > > > > Yes, we are getting exception in broker log > > > > > > > > (javax.net.ssl.SSLHandshakeException: Empty server > certificate > > > > > > > > chain) and we are looking for more information along with > just > > > > > > > > client > > > > > > IP. > > > > > > > > > > > > > > > > If JDK handles these handshakes as part of heap or native > > memory? > > > > > > > > Our brokers are running in K8s (6 GB heap and 10 GB to pod) > > > > > > > > and as per observations, pod is getting killed with OOM. We > > > > > > > > have OOM dump instruction in artemis.profile but it is not > > > > > > > > happening which means broker process was still running while > > > > > > > > pod was killed and restarted by K8s. If the large native > > > > > > > > memory usage caused by SSL handshake can cause pod to OOM and > > > get killed? > > > > > > > > > > > > > > > > We tried to simulate similar case in VM based broker. Here > > > > > > > > broker became unresponsive after lot of SSL handshake errors > > > > > > > > as we could see lot of > > > > > > > > 'AMQ224088: Timeout (10 seconds) on acceptor > > > "artemis-ssl-customer" > > > > > > > > during protocol handshake with /1.2.3.4:62403 has occurred.' > > > > > > > > Where clients are trying to connect back. After some time it > > > > > > > > got restarted > > > > > > > with OOM. > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Shiv > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > > > > Sent: 04 April 2025 09:26 PM > > > > > > > > To: users@activemq.apache.org > > > > > > > > Subject: Re: Additional Info on certificate based > > > > > > > > authentication errors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been > > > verified. > > > > > > > > Review the content of the message carefully and verify the > > > > > > > > identity of the sender before acting on this email: replying, > > > > > > > > opening attachments or clicking links. > > > > > > > > > > > > > > > > > > > > > > > > > How can we find out when a client connects to broker using > > > > > > > > > invalid > > > > > > > > certificate... > > > > > > > > > > > > > > > > According to the information you've provided so far you're > > > > > > > > already being notified about this in the broker log. Is that > > > > > > > > not > > > > the case? > > > > > > > > I was under the impression that you simply wanted more > details > > > > > > > > about the > > > > > > > failure. > > > > > > > > > > > > > > > > > ...SSL handshake is terminated at what stage like client > > > > > > > > > hello, server > > > > > > > > hello or anything further? > > > > > > > > > > > > > > > > I'm not an SSL expert by any means, but I believe the failure > > > > > > > > comes several steps after the client and server hello (i.e. > > > > > > > > the very first steps in the handshake). After the broker > sends > > > > > > > > its certificate to the client and the client verifies it the > > > > > > > > broker then requests the client's > > > > > > > certificate. > > > > > > > > When the client sends its certificate the broker verifies it. > > > > > > > > I believe this is where the failure is happening. > > > > > > > > > > > > > > > > As noted previously, this handshake is being done by the JDK > > > > > > > > (or potentially OpenSSL if you've chosen OPENSSL for the > > > > > > > > sslProvider > > > > > [1]). > > > > > > > > The broker doesn't control what information is available when > > > > > > > > a failure > > > > > > > occurs. > > > > > > > > As far as I'm aware the broker is already providing all the > > > > > > > > information available regarding an SSL handshake failure. > > > > > > > > > > > > > > > > > > > > > > > > Justin > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > https://acti/ > > > > > > > > vemq.apache.org > %2Fcomponents%2Fartemis%2Fdocumentation%2Flates > > > > > > > > t% > > > > > > > > 2F > > > > > > > > co > > > > > > > > nf > > > > > > > > > iguring-transports.html%23configuring-netty-ssl&data=05%7C02%7 > > > > > > > > C% > > > > > > > > 7C > > > > > > > > 49 > > > > > > > > f8 > > > > > > > > > 42368dc64754902208dd76c85945%7C1a1dce2021b14beaa9d2130e9f1f6e2 > > > > > > > > f% > > > > > > > > 7C > > > > > > > > 0% > > > > > > > > 7C > > > > > > > > > 0%7C638797325066726960%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGk > > > > > > > > iO > > > > > > > > nR > > > > > > > > yd > > > > > > > > WU > > > > > > > > > sIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyf > > > > > > > > Q% > > > > > > > > 3D > > > > > > > > %3 > > > > > > > > D% > > > > > > > > > 7C0%7C%7C%7C&sdata=esE%2FGWoSZocxz93n%2Fs8rye3Vhk%2BNFPLbtFEwh > > > > > > > > RQ > > > > > > > > vS > > > > > > > > Og > > > > > > > > %3 > > > > > > > > D&reserved=0 > > > > > > > > > > > > > > > > On Thu, Apr 3, 2025 at 9:58 AM Shiv Kumar Dixit > > > > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > > > > > > > Hi Justin, > > > > > > > > > We have defined our acceptors with needClientAuth=true as > we > > > > > > > > > want 2-way SSL. How can we find out when a client connects > > > > > > > > > to broker using invalid certificate, SSL handshake is > > > > > > > > > terminated at what stage like client hello, server hello or > > > > > > > > > anything > > > > further? > > > > > > > > > In our case, we have a .net client which connects using URL > > > > > > > > > like ssl://host:port?transport.clientCertSubject=CN=ABC1. > If > > > > > > > > > ABC1 cert is install on the client, it works fine but if > > > > > > > > > actual CN is ABC and someone misconfigured CN as ABC1 then > > > > > > > > > we get > > > > > > > > javax.net.ssl.SSLHandshakeException: > > > > > > > > > Empty server certificate chain error at broker. > > > > > > > > > > > > > > > > > > As per my understanding, client hello should go well as it > > > > > > > > > has no client cert details but info like TLS version, > cipher > > > > > > > > > details > > > > > etc. > > > > > > > > > Server hello should also be fine and now server should be > > > > > > > > > expecting client certificate details along with passing > > > > > > > > > server > > > > > > certificate. > > > > > > > > > Now when client responds, it should be passing its > > > > > > > > > certificate, and this validation should be failing at > server > > > > > > > > > side resulting in error > > > > > > > like: > > > > > > > > > AMQ222208: SSL handshake failed for client from / > > 1.2.3.4:53838 > > > : > > > > > > > > > javax.net.ssl.SSLHandshakeException: Empty server > > > > > > > > > certificate > > > > > chain. > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > > Shiv > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > > > > > Sent: 03 April 2025 08:05 PM > > > > > > > > > To: users@activemq.apache.org > > > > > > > > > Subject: Re: Additional Info on certificate based > > > > > > > > > authentication errors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been > > > > verified. > > > > > > > > > Review the content of the message carefully and verify the > > > > > > > > > identity of the sender before acting on this email: > > > > > > > > > replying, opening attachments or clicking links. > > > > > > > > > > > > > > > > > > > > > > > > > > > Are you wanting to extract cert data on the client or the > > > broker? > > > > > > > > > The broker already uses code like this to get cert data > [1]. > > > > > > > > > The problem in your case is that the SSL handshake fails so > > > > > > > > > the broker has no chance to inspect the cert data on the > > > > connection. > > > > > > > > > > > > > > > > > > > > > > > > > > > Justin > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://gith/ > > > > > > > > > ub.com > %2Fapache%2Factivemq-artemis%2Fblob%2Fmain%2Fartemis-c > > > > > > > > > or > > > > > > > > > e- > > > > > > > > > cl > > > > > > > > > ie > > > > > > > > > nt > > > > > > > > > > %2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Fc > > > > > > > > > or > > > > > > > > > e% > > > > > > > > > 2F > > > > > > > > > re > > > > > > > > > mo > > > > > > > > > > ting%2FCertificateUtil.java&data=05%7C02%7C%7C04b98fbbd31044 > > > > > > > > > 31 > > > > > > > > > 0c > > > > > > > > > 05 > > > > > > > > > 08 > > > > > > > > > dd > > > > > > > > > > 73915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638793 > > > > > > > > > 79 > > > > > > > > > 04 > > > > > > > > > 28 > > > > > > > > > 19 > > > > > > > > > 81 > > > > > > > > > > 82%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL > > > > > > > > > jA > > > > > > > > > uM > > > > > > > > > DA > > > > > > > > > wM > > > > > > > > > CI > > > > > > > > > > sIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C% > > > > > > > > > 7C > > > > > > > > > &s > > > > > > > > > da > > > > > > > > > ta > > > > > > > > > =% > > > > > > > > > 2BRmolH7RtdgtR0JDo6le65BYsJhxyCPXDBUxbNEPnSo%3D&reserved=0 > > > > > > > > > > > > > > > > > > On Thu, Apr 3, 2025 at 2:16 AM Shiv Kumar Dixit > > > > > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > Hi Justin, > > > > > > > > > > Thanks for input. > > > > > > > > > > > > > > > > > > > > Also, we came across this link https://stac/ > > > > > > > > > > koverflow.com > %2Fquestions%2F9777864%2Fhow-can-i-get-the-cl > > > > > > > > > > ie > > > > > > > > > > nt > > > > > > > > > > -c > > > > > > > > > > er > > > > > > > > > > ti > > > > > > > > > > fi > > > > > > > > > > > cate-in-netty-handler-to-identify-user&data=05%7C02%7C%7Ce > > > > > > > > > > e4 > > > > > > > > > > e7 > > > > > > > > > > ef > > > > > > > > > > 7f > > > > > > > > > > 3c > > > > > > > > > > 84 > > > > > > > > > > > c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0 > > > > > > > > > > %7 > > > > > > > > > > C0 > > > > > > > > > > %7 > > > > > > > > > > C6 > > > > > > > > > > 38 > > > > > > > > > > 79 > > > > > > > > > > > 2877389315248%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy > > > > > > > > > > dW > > > > > > > > > > Us > > > > > > > > > > Il > > > > > > > > > > Yi > > > > > > > > > > Oi > > > > > > > > > > Iw > > > > > > > > > > > LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D% > > > > > > > > > > 3D > > > > > > > > > > %7 > > > > > > > > > > C0 > > > > > > > > > > %7 > > > > > > > > > > C% > > > > > > > > > > 7C > > > > > > > > > > > %7C&sdata=M0a4HbeUymmwbtVFCOGPQRp5ZH%2Fmq7eLebjNuQT%2FVKk% > > > > > > > > > > 3D > > > > > > > > > > &r > > > > > > > > > > es > > > > > > > > > > er > > > > > > > > > > ve > > > > > > > > > > d= > > > > > > > > > > 0 where it seems we can extract SSL information from > > > > > > > > > > ChannelHandlerContext ctx object. Do you see if it is > > > feasible? > > > > > > > > > > > > > > > > > > > > ChannelHandlerContext ctx = ... > > > > > > > > > > SslHandler sslhandler = (SslHandler) > > > > > > > > > > ctx.channel().pipeline().get("ssl"); > > > > > > > > > > > > > > > > > > > > > > > sslhandler.engine().getSession().getPeerCertificateChain()[0]. > > > > > > > > > > ge > > > > > > > > > > tS > > > > > > > > > > ub > > > > > > > > > > je > > > > > > > > > > ctDN()); > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > > > Shiv > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > > > > > > Sent: 28 March 2025 10:42 PM > > > > > > > > > > To: users@activemq.apache.org > > > > > > > > > > Subject: Re: Additional Info on certificate based > > > > > > > > > > authentication errors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been > > > > > verified. > > > > > > > > > > Review the content of the message carefully and verify > the > > > > > > > > > > identity of the sender before acting on this email: > > > > > > > > > > replying, opening attachments or clicking links. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One option to improve this situation for Core [1] and > > > > > > > > > > OpenWire [2] clients is to specify the "local" port which > > > > > > > > > > will then be printed on in broker log along with the IP > > > > > > > > > > address. By default both clients use a random, available > > > > > > > > > > port which makes correlation difficult, if not > impossible, > > > > > > > > > > in > > > > my circumstances. > > > > > > > > > > Of course, every client which does this will need to use > a > > > > > > > > > > unique port to ensure there are no conflicts if they are > > > > > > > > > > all running on the same > > > > > > > machine. > > > > > > > > > > > > > > > > > > > > Let me know where you reached out to the Netty community > > > > > > > > > > and I'll follow along. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Justin > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://acti/ > > > > > > > > > > vemq.apache.org > %2Fcomponents%2Fartemis%2Fdocumentation%2Fl > > > > > > > > > > at > > > > > > > > > > es > > > > > > > > > > t% > > > > > > > > > > 2F > > > > > > > > > > co > > > > > > > > > > nf > > > > > > > > > > > iguring-transports.html%23configuring-the-transport%23%3A~ > > > > > > > > > > %3 > > > > > > > > > > At > > > > > > > > > > ex > > > > > > > > > > t% > > > > > > > > > > 3D > > > > > > > > > > lo > > > > > > > > > > > calPort&data=05%7C02%7C%7Cee4e7ef7f3c84c0dd87208dd72bccbc9 > > > > > > > > > > %7 > > > > > > > > > > C1 > > > > > > > > > > a1 > > > > > > > > > > dc > > > > > > > > > > e2 > > > > > > > > > > 02 > > > > > > > > > > > 1b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638792877389342921%7CUnk > > > > > > > > > > no > > > > > > > > > > wn > > > > > > > > > > %7 > > > > > > > > > > CT > > > > > > > > > > WF > > > > > > > > > > pb > > > > > > > > > > > GZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJ > > > > > > > > > > Xa > > > > > > > > > > W4 > > > > > > > > > > zM > > > > > > > > > > iI > > > > > > > > > > sI > > > > > > > > > > kF > > > > > > > > > > > OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3wj4B1Un% > > > > > > > > > > 2F > > > > > > > > > > Ur > > > > > > > > > > Pl > > > > > > > > > > sk > > > > > > > > > > Lh > > > > > > > > > > OE > > > > > > > > > > yDo7SQ%2FN0ou2OGnTgPvfp4nk%3D&reserved=0 > > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > https://acti/ > > > > > > > > > > vemq.apache.org > %2Fcomponents%2Fclassic%2Fdocumentation%2Fh > > > > > > > > > > ow > > > > > > > > > > -d > > > > > > > > > > o- > > > > > > > > > > i- > > > > > > > > > > de > > > > > > > > > > fi > > > > > > > > > > > ne-a-local-address-and-local-port-for-tcp-or-ssl&data=05%7 > > > > > > > > > > C0 > > > > > > > > > > 2% > > > > > > > > > > 7C > > > > > > > > > > %7 > > > > > > > > > > Ce > > > > > > > > > > e4 > > > > > > > > > > > e7ef7f3c84c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f > > > > > > > > > > 1f > > > > > > > > > > 6e > > > > > > > > > > 2f > > > > > > > > > > %7 > > > > > > > > > > C0 > > > > > > > > > > %7 > > > > > > > > > > > C0%7C638792877389363848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU > > > > > > > > > > 1h > > > > > > > > > > cG > > > > > > > > > > ki > > > > > > > > > > On > > > > > > > > > > Ry > > > > > > > > > > dW > > > > > > > > > > > UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldU > > > > > > > > > > Ij > > > > > > > > > > oy > > > > > > > > > > fQ > > > > > > > > > > %3 > > > > > > > > > > D% > > > > > > > > > > 3D > > > > > > > > > > > %7C0%7C%7C%7C&sdata=juvStcSQ%2BDcKH%2F14xA%2BdHuAUQUzrKHV0 > > > > > > > > > > PK > > > > > > > > > > 3m > > > > > > > > > > M9 > > > > > > > > > > A9 > > > > > > > > > > ue > > > > > > > > > > 4% > > > > > > > > > > 3D&reserved=0 > > > > > > > > > > > > > > > > > > > > On Fri, Mar 28, 2025 at 10:12 AM Shiv Kumar Dixit > > > > > > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > Hello Justin, > > > > > > > > > > > Thanks for the response. > > > > > > > > > > > > > > > > > > > > > > We have clients using Core/AMQP/OpenWire with > > certificates. > > > > > > > > > > > > > > > > > > > > > > Passing SSL debugging flag is good idea but given we > > > > > > > > > > > have many clients using broker concurrently, I fear we > > > > > > > > > > > will have lot of logs and it will be difficult to > > > > > > > > > > > analyse. I am also checking with netty community if > they > > > > > > > > > > > can provide some > > > > input. > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > > > > Shiv > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > > > > > > > Sent: 28 March 2025 08:33 AM > > > > > > > > > > > To: users@activemq.apache.org > > > > > > > > > > > Subject: Re: Additional Info on certificate based > > > > > > > > > > > authentication errors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not > been > > > > > > verified. > > > > > > > > > > > Review the content of the message carefully and verify > > > > > > > > > > > the identity of the sender before acting on this email: > > > > > > > > > > > replying, opening attachments or clicking links. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unfortunately I don't think the broker itself will be > > > > > > > > > > > able to provide additional information because it > simply > > > > > > > > > > > doesn't > > > > > have it. > > > > > > > > > > > The SSL handshake is delegated by Netty to the JDK (or > > > > > > > > > > > potentially to OpenSSL) and it's the first thing done > > > > > > > > > > > when the > > > > > > > client connects. > > > > > > > > > > > It happens before the messaging protocol handshake > takes > > > > > > > > > > > place so there's definitely no way to get the client ID > > > > > > > > > > > because it hasn't been sent yet. Netty only provides a > > > > > > > > > > > Throwable [1] and we print the relevant > > > > > > > > > > information from it. > > > > > > > > > > > > > > > > > > > > > > It's possible you could get some relevant information > > > > > > > > > > > from Java's SSL debug logging by passing something like > > > > > > > > > > > -Djavax.net.debug=ssl:handshake to the JVM at startup. > > > > > > > > > > > > > > > > > > > > > > What kind of client are you using to connect to the > > > > > > > > > > > broker in this > > > > > > > > > case? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Justin > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > https://gith/ > > > > > > > > > > > ub.com > %2Fapache%2Factivemq-artemis%2Fblob%2Fc4b315ba0c1f > > > > > > > > > > > af > > > > > > > > > > > be > > > > > > > > > > > 2d > > > > > > > > > > > 2c > > > > > > > > > > > 3e > > > > > > > > > > > bf > > > > > > > > > > > 0c > > > > > > > > > > > > dd84f921f42b5a%2Fartemis-server%2Fsrc%2Fmain%2Fjava%2For > > > > > > > > > > > g% > > > > > > > > > > > 2F > > > > > > > > > > > ap > > > > > > > > > > > ac > > > > > > > > > > > he > > > > > > > > > > > %2 > > > > > > > > > > > Fa > > > > > > > > > > > > ctivemq%2Fartemis%2Fcore%2Fremoting%2Fimpl%2Fnetty%2FNet > > > > > > > > > > > ty > > > > > > > > > > > Ac > > > > > > > > > > > ce > > > > > > > > > > > pt > > > > > > > > > > > or > > > > > > > > > > > .j > > > > > > > > > > > av > > > > > > > > > > > > a%23L1014&data=05%7C02%7C%7C2d25dad5c33e4432379808dd6e1b > > > > > > > > > > > bb > > > > > > > > > > > e3 > > > > > > > > > > > %7 > > > > > > > > > > > C1 > > > > > > > > > > > a1 > > > > > > > > > > > dc > > > > > > > > > > > e2 > > > > > > > > > > > > 021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638787787610427088%7 > > > > > > > > > > > CU > > > > > > > > > > > nk > > > > > > > > > > > no > > > > > > > > > > > wn > > > > > > > > > > > %7 > > > > > > > > > > > CT > > > > > > > > > > > WF > > > > > > > > > > > > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlA > > > > > > > > > > > iO > > > > > > > > > > > iJ > > > > > > > > > > > Xa > > > > > > > > > > > W4 > > > > > > > > > > > zM > > > > > > > > > > > iI > > > > > > > > > > > sI > > > > > > > > > > > > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nJBF0 > > > > > > > > > > > dM > > > > > > > > > > > qm > > > > > > > > > > > Oi > > > > > > > > > > > PJ > > > > > > > > > > > dl > > > > > > > > > > > UM > > > > > > > > > > > Bj > > > > > > > > > > > 3q3Ia01NII0ykmP6XQisV%2Bao%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 27, 2025 at 2:03 AM Shiv Kumar Dixit > > > > > > > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi team, > > > > > > > > > > > > Currently we are using Apache Artemis 2.37.0. We have > > > > > > > > > > > > certificate-based customers. Many times, we see > errors > > > > > > > > > > > > like below in > > > > > > > > > > > broker log: > > > > > > > > > > > > > > > > > > > > > > > > AMQ222208: SSL handshake failed for client from > > > > > /a.b.c.d:53838: > > > > > > > > > > > > javax.net.ssl.SSLHandshakeException: Empty server > > > > > > > > > > > > certificate > > > > > > > > chain. > > > > > > > > > > > > > > > > > > > > > > > > AMQ222208: SSL handshake failed for client from > > > > > /a.b.c.d:59132: > > > > > > > > > > > > java.security.cert.CertificateExpiredException: > > NotAfter: > > > > > > > > > > > > Tue Mar > > > > > > > > > > > > 25 > > > > > > > > > > > > 00:00:11 IST 2025 > > > > > > > > > > > > > > > > > > > > > > > > AMQ224088: Timeout (10 seconds) on acceptor "artemis" > > > > > > > > > > > > during protocol handshake with /a.b.c.d:62403 has > > > occurred. > > > > > > > > > > > > > > > > > > > > > > > > Here we only get customer IP along with error > message. > > > > > > > > > > > > In real world, we have many customer applications > > > > > > > > > > > > running from same IP but not all use expired > > > > > > > > > > > > certificate or invalid certificate. We may have only > > > > > > > > > > > > of such misbehaving > > > > > client. > > > > > > > > > > > > Troubleshooting such cases require validating all > > > > > > > > > > > > customer applications on single IP to trace the > issue. > > > > > > > > > > > > It is very time consuming as we need to check > customer > > > > > > > > > > > > configuration, start/stop applications to see how it > > > > > > > > > > > > impacts the > > > > > > > > > > broker error etc. > > > > > > > > > > > > > > > > > > > > > > > > Can we get some additional information such as client > > > > > > > > > > > > id, certificate CN details or any other certificate > > > > > > > > > > > > information which can help us in identifying the > > > > > > > > > > > > erring > > > > > client faster? > > > > > > > > > > > > If there is any log level change which can enable to > > > > > > > > > > > > put such information in broker > > > > > > > > > > log will be helpful. > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > Shiv > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------- > > > > > > > > > > > -- > > > > > > > > > > > -- > > > > > > > > > > > -- > > > > > > > > > > > -- > > > > > > > > > > > -- > > > > > > > > > > > -- > > > > > > > > > > > - To unsubscribe, e-mail: > > > > > > > > > > > users-unsubscr...@activemq.apache.org > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > > users-h...@activemq.apache.org For further > information, > > > > visit: > > > > > > > > > > > https://acti/ > > > > > > > > > > > vemq.apache.org > %2Fcontact&data=05%7C02%7C%7C2d25dad5c33e > > > > > > > > > > > 44 > > > > > > > > > > > 32 > > > > > > > > > > > 37 > > > > > > > > > > > 98 > > > > > > > > > > > 08 > > > > > > > > > > > dd > > > > > > > > > > > 6e > > > > > > > > > > > > 1bbbe3%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387 > > > > > > > > > > > 87 > > > > > > > > > > > 78 > > > > > > > > > > > 76 > > > > > > > > > > > 10 > > > > > > > > > > > 43 > > > > > > > > > > > 97 > > > > > > > > > > > 97 > > > > > > > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiI > > > > > > > > > > > wL > > > > > > > > > > > jA > > > > > > > > > > > uM > > > > > > > > > > > DA > > > > > > > > > > > wM > > > > > > > > > > > CI > > > > > > > > > > > sI > > > > > > > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7 > > > > > > > > > > > C% > > > > > > > > > > > 7C > > > > > > > > > > > &s > > > > > > > > > > > da > > > > > > > > > > > ta > > > > > > > > > > > =X > > > > > > > > > > > Rb > > > > > > > > > > > > daTuyVGNgBH%2FDdvihBuYzv%2BiATnFUfURp8IU2L%2B4%3D&reserv > > > > > > > > > > > ed > > > > > > > > > > > =0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------- > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > - To unsubscribe, e-mail: > > > > > > > > > > users-unsubscr...@activemq.apache.org > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > users-h...@activemq.apache.org For further information, > > > visit: > > > > > > > > > > https://acti/ > > > > > > > > > > vemq.apache.org > %2Fcontact&data=05%7C02%7C%7Cee4e7ef7f3c84c > > > > > > > > > > 0d > > > > > > > > > > d8 > > > > > > > > > > 72 > > > > > > > > > > 08 > > > > > > > > > > dd > > > > > > > > > > 72 > > > > > > > > > > > bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638792 > > > > > > > > > > 87 > > > > > > > > > > 73 > > > > > > > > > > 89 > > > > > > > > > > 38 > > > > > > > > > > 52 > > > > > > > > > > 94 > > > > > > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL > > > > > > > > > > jA > > > > > > > > > > uM > > > > > > > > > > DA > > > > > > > > > > wM > > > > > > > > > > CI > > > > > > > > > > sI > > > > > > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C% > > > > > > > > > > 7C > > > > > > > > > > &s > > > > > > > > > > da > > > > > > > > > > ta > > > > > > > > > > =P > > > > > > > > > > Uy > > > > > > > > > > z3W9mbHPKRghYMO58s5HWPMCKuIyITGbbJMIy%2BY0%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------ > > > > > > > > > -- > > > > > > > > > -- > > > > > > > > > -- > > > > > > > > > -- > > > > > > > > > - To unsubscribe, e-mail: > > > > > > > > > users-unsubscr...@activemq.apache.org > > > > > > > > > For additional commands, e-mail: > > > > > > > > > users-h...@activemq.apache.org For further information, > > visit: > > > > > > > > > https://acti/ > > > > > > > > > vemq.apache.org > %2Fcontact&data=05%7C02%7C%7C04b98fbbd3104431 > > > > > > > > > 0c > > > > > > > > > 05 > > > > > > > > > 08 > > > > > > > > > dd > > > > > > > > > 73 > > > > > > > > > > 915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63879379 > > > > > > > > > 04 > > > > > > > > > 28 > > > > > > > > > 22 > > > > > > > > > 95 > > > > > > > > > 91 > > > > > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjA > > > > > > > > > uM > > > > > > > > > DA > > > > > > > > > wM > > > > > > > > > CI > > > > > > > > > sI > > > > > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C > > > > > > > > > &s > > > > > > > > > da > > > > > > > > > ta > > > > > > > > > =k > > > > > > > > > aL > > > > > > > > > FLEBaBPzcqcq%2BJYRMJ4UbXRKaKHQzQthxEdfNHPw%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > > > -- > > > > > > > > -- > > > > > > > > -- > > > > > > > > - To unsubscribe, e-mail: > > > > > > > > users-unsubscr...@activemq.apache.org > > > > > > > > For additional commands, e-mail: > > > > > > > > users-h...@activemq.apache.org For further information, > visit: > > > > > > > > https://acti/ > > > > > > > > vemq.apache.org > %2Fcontact&data=05%7C02%7C%7C49f842368dc6475490 > > > > > > > > 22 > > > > > > > > 08 > > > > > > > > dd > > > > > > > > 76 > > > > > > > > > c85945%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387973250 > > > > > > > > 66 > > > > > > > > 73 > > > > > > > > 96 > > > > > > > > 57 > > > > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM > > > > > > > > DA > > > > > > > > wM > > > > > > > > CI > > > > > > > > sI > > > > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&s > > > > > > > > da > > > > > > > > ta > > > > > > > > =H > > > > > > > > 7m > > > > > > > > 8XMqQ5GLVSPXcp25%2FyBx%2F3XdkKBQ6bM%2BkeloHL5U%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------- > > > > > > > -- > > > > > > > -- > > > > > > > - To unsubscribe, e-mail: > users-unsubscr...@activemq.apache.org > > > > > > > For additional commands, e-mail: > users-h...@activemq.apache.org > > > > > > > For further information, visit: > > > > > > > https://acti/ > > > > > > > vemq.apache.org > %2Fcontact&data=05%7C02%7C%7Cfb52ba57391b458763f2 > > > > > > > 08 > > > > > > > dd > > > > > > > 77 > > > > > > > > 6e9fda%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638798039227 > > > > > > > 62 > > > > > > > 68 > > > > > > > 34 > > > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > > > > > > > wM > > > > > > > CI > > > > > > > sI > > > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sda > > > > > > > ta > > > > > > > =O > > > > > > > Sm > > > > > > > JsMP3PS2rdYtqvDI34HvFvG2zztpNv%2FU2ZDoNUTs%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > -- > > > > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > > > > For additional commands, e-mail: users-h...@activemq.apache.org > > > > > > For further information, visit: > > > > > > https://acti/ > > > > > > vemq.apache.org > %2Fcontact&data=05%7C02%7C%7Cd6c8503c5b0c4c80724008 > > > > > > dd > > > > > > 78 > > > > > > > 6de948%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63879913567645 > > > > > > 27 > > > > > > 73 > > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwM > > > > > > CI > > > > > > sI > > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata > > > > > > =s > > > > > > Hk > > > > > > 8yVgb24q77V5qQ39xx6etduy0KhqVak9vcf5S5YQ%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > > > For additional commands, e-mail: users-h...@activemq.apache.org > For > > > > > further information, visit: > > > > > https://ac/ > > > > > > ti%2F&data=05%7C02%7C%7Cc2dd0998e0f64808f14b08dd7b31b658%7C1a1dce202 > > > > > > 1b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638802175649339067%7CUnknown%7CTWF > > > > > > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiI > > > > > > sIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nw7JxrI8lMqVm9o > > > > > p%2ByAI70yXlrTZbCoFBcgWZ5iiJ6Y%3D&reserved=0 > > > > > vemq.apache.org > %2Fcontact&data=05%7C02%7C%7C24e1fb3bd58841ee2d1108dd > > > > > 7b > > > > > > 2d76ac%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6388021573740442 > > > > > 10 > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCI > > > > > sI > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=G > > > > > MK > > > > > r73S0iMWDJYQl9AmwKQxy6mjMML08UneehEHcxfs%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > For additional commands, e-mail: users-h...@activemq.apache.org > > > For further information, visit: https://activemq.apache.org/contact > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > For additional commands, e-mail: users-h...@activemq.apache.org > > For further information, visit: https://activemq.apache.org/contact > > > > >
