Hi Shiv, to use OPENSSL as sslProvider with the netty-tcnative default artifact, your system must have both libapr-1 and OpenSSL installed and configured. Alternatively, you can use one of the netty-tcnative artifacts statically linked, for further details, see https://netty.io/wiki/forked-tomcat-native.html
Regards, Domenico On Mon, 14 Apr 2025 at 10:42, Shiv Kumar Dixit <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > Hi Domenico, > I am putting netty-tcnative-2.0.70.Final.jar in broker_home/lib folder but > still the same error. I am running broker on a Windows laptop. Do I need to > have openssl installed as pre-condition on my laptop as the documentation > says " If used with OPENSSL you can add netty-tcnative to your classpath to > use the native installed openssl". So openssl installation is precondition > to use OPENSSL as sslProvider? > > Best Regards > Shiv > > -----Original Message----- > From: Domenico Francesco Bruscino <bruscin...@gmail.com> > Sent: 14 April 2025 01:52 PM > To: users@activemq.apache.org > Subject: Re: Additional Info on certificate based authentication errors > > > > Unverified Sender: The sender of this email has not been verified. Review > the content of the message carefully and verify the identity of the sender > before acting on this email: replying, opening attachments or clicking > links. > > > Hi Shiv, > > you need to add `netty-tcnative` to your classpath to use the native > installed openssl. > > Regards, > Domenico > > On Fri, 11 Apr 2025 at 11:37, Shiv Kumar Dixit > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > Hi Justin, > > I set connectionsAllowed=3 on acceptor. I then ran 3 good listener apps. > > It created 3 good connections and then 1 consumer per connection. I > > then ran 4th good listener but broker rejected the connection. When I > > ran 4th bad listener (it has wrong certificate name in its config), it > > tried to create connection with broker and failed with > > "javax.net.ssl.SSLHandshakeException: Empty server certificate chain". > > So, connectionsAllowed can only restrict good connections. > > > > Regarding switching to OPENSSL, I set the value as per your > > suggestion, but broker won't start due to below error: > > 1. Do we need to copy additional jars? > > 2. Do we need OpenSSL installed locally? > > > > 2025-04-11 14:40:56,715 ERROR > > [org.apache.activemq.artemis.core.server] > > AMQ224097: Failed to start server > > java.lang.NoClassDefFoundError: > > io/netty/internal/tcnative/SSLPrivateKeyMethod > > at > > io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.ja > > va:480) ~[netty-handler-4.1.112.Final.jar:4.1.112.Final] > > at > > io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:64 > > 3) ~[netty-handler-4.1.112.Final.jar:4.1.112.Final] > > at > > org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.createNe > > ttyContext(SSLSupport.java:240) > > ~[artemis-core-client-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.remoting.impl.ssl.DefaultOpenSSLConte > > xtFactory.getServerSslContext(DefaultOpenSSLContextFactory.java:60) > > ~[artemis-core-client-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.loa > > dSSLContext(NettyAcceptor.java:397) > > ~[artemis-server-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.<in > > it>(NettyAcceptor.java:349) > > ~[artemis-server-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptorFact > > ory.createAcceptor(NettyAcceptorFactory.java:43) > > ~[artemis-server-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceI > > mpl.createAcceptor(RemotingServiceImpl.java:267) > > ~[artemis-server-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceI > > mpl.start(RemotingServiceImpl.java:210) > > ~[artemis-server-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initia > > lisePart2(ActiveMQServerImpl.java:3560) > > ~[artemis-server-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.server.impl.PrimaryOnlyActivation.run > > (PrimaryOnlyActivation.java:78) > > ~[artemis-server-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.intern > > alStart(ActiveMQServerImpl.java:742) > > ~[artemis-server-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start( > > ActiveMQServerImpl.java:632) > > [artemis-server-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.ja > > va:66) > > [artemis-cli-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130) > > [artemis-cli-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:2 > > 21) > > [artemis-cli-2.37.0.jar:2.37.0] > > at > > org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:167) > > [artemis-cli-2.37.0.jar:2.37.0] > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) ~[?:?] > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM > > ethodAccessorImpl.java:62) > > ~[?:?] > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del > > egatingMethodAccessorImpl.java:43) > > ~[?:?] > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > ~[?:?] > > at > > org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:152) > > [artemis-boot.jar:2.37.0] > > at > > org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64) > > [artemis-boot.jar:2.37.0] > > Caused by: java.lang.ClassNotFoundException: > > io.netty.internal.tcnative.SSLPrivateKeyMethod > > at > > java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476) > > ~[?:?] > > at > > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594) > > ~[?:?] > > at > > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) > > ~[?:?] > > ... 23 more > > java.lang.NoClassDefFoundError: > > io/netty/internal/tcnative/SSLPrivateKeyMethod > > at > > > io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:480) > > at > > io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:643) > > at > > > org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.createNettyContext(SSLSupport.java:240) > > at > > > org.apache.activemq.artemis.core.remoting.impl.ssl.DefaultOpenSSLContextFactory.getServerSslContext(DefaultOpenSSLContextFactory.java:60) > > at > > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.loadSSLContext(NettyAcceptor.java:397) > > at > > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.<init>(NettyAcceptor.java:349) > > at > > > org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptorFactory.createAcceptor(NettyAcceptorFactory.java:43) > > at > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl.createAcceptor(RemotingServiceImpl.java:267) > > at > > > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl.start(RemotingServiceImpl.java:210) > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:3560) > > at > > > org.apache.activemq.artemis.core.server.impl.PrimaryOnlyActivation.run(PrimaryOnlyActivation.java:78) > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:742) > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:632) > > at > > > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66) > > at > > org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130) > > at > > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:221) > > at > > org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:167) > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > at > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > > org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:152) > > at > > org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64) > > Caused by: java.lang.ClassNotFoundException: > > io.netty.internal.tcnative.SSLPrivateKeyMethod > > at java.base/java.net > > .URLClassLoader.findClass(URLClassLoader.java:476) > > at > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594) > > at > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) > > ... 23 more > > > > Best Regards > > Shiv > > > > -----Original Message----- > > From: Justin Bertram <jbert...@apache.org> > > Sent: 11 April 2025 01:55 AM > > To: users@activemq.apache.org > > Subject: Re: Additional Info on certificate based authentication > > errors > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > Review the content of the message carefully and verify the identity of > > the sender before acting on this email: replying, opening attachments > > or clicking links. > > > > > > > I just defined sslProvider=openssl... > > > > You should be using this as noted in the documentation [1]: > > > > sslProvider=OPENSSL > > > > > Is there any log or other indication that now SSL process has moved > > > from > > JDK to OPENSSL? > > > > I couldn't find anything that logs which sslProvider is in use for an > > acceptor. > > > > > Unfortunately, I could not find any article on using OPENSSL with > > Artemis. > > > > I'm not aware of any specific articles. However, this documentation > > [1] should be sufficient. Let me know if more details are required. > > > > > ...regarding connectionsAllowed on acceptor, will it protect against > > misbehaving producer/consumer? I understood that it is only good > > enough to restrict good connections. > > > > I'm not really sure what you're asking here. > > > > Technically speaking you can only create a producer or consumer once a > > connection is established so if an application creates a connection > > and then uses that connection to spam create a bunch of producers > > and/or consumers then connectionsAllowed will not prevent that. > > However, if an application is spamming connections in the process of > > producing and/or consuming messages then connectionsAllowed will stop > that. > > > > > > Justin > > > > [1] > > > > https://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Facti > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fconf > > iguring-transports.html%23configuring-netty-ssl%3F%3A~%3Atext%3DsslPro > > vider&data=05%7C02%7C%7C24e1fb3bd58841ee2d1108dd7b2d76ac%7C1a1dce2021b > > 14beaa9d2130e9f1f6e2f%7C0%7C0%7C638802157373991188%7CUnknown%7CTWFpbGZ > > sb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOI > > joiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=j1nFu5HBrb48dSJybiOGnLv > > N1%2FSBO%2F3TRuRou5xrNgU%3D&reserved=0 > > > > On Thu, Apr 10, 2025 at 11:57 AM Shiv Kumar Dixit > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > Hi Justin, > > > While moving from JDK to OPENSSL, I just defined sslProvider=openssl > > > on the acceptor and broker seems to start fine. Is there any log or > > > other indication that now SSL process has moved from JDK to OPENSSL? > > > Unfortunately, I could not find any article on using OPENSSL with > > Artemis. > > > Any input will be useful here. > > > > > > Secondly regarding, "first line of defense would actually be in the > > > network exposure rather than in the broker configuration", we have > > > network segregation but all of the misbehaving clients come from > > > trusted > > network. > > > One same VM, we could have 2 listeners - one has correctly > > > configured the certificate details while another has wrong > > > configuration. Second one is causing " > > > javax.net.ssl.SSLHandshakeException: Empty server certificate chain" > > > error on the broker. Given we have dozens of VMs connecting to > > > broker cluster and each VM hosting many producers/consumers where > > > both good and bad configuration can co-exist, we are defenceless > > > against this kind of issue. Here only option is to find IP and scan > > > producer/consumer configuration and see if they have any > misconfiguration causing trouble to brokers. > > > > > > Thirdly, regarding connectionsAllowed on acceptor, will it protect > > > against misbehaving producer/consumer? I understood that it is only > > > good enough to restrict good connections. > > > > > > Best Regards > > > Shiv > > > > > > -----Original Message----- > > > From: Justin Bertram <jbert...@apache.org> > > > Sent: 09 April 2025 07:28 PM > > > To: users@activemq.apache.org > > > Subject: Re: Additional Info on certificate based authentication > > > errors > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > Review the content of the message carefully and verify the identity > > > of the sender before acting on this email: replying, opening > > > attachments or clicking links. > > > > > > > > > > ...any article or whitepaper from community on how to handle > > > > (D)DoS > > > attack on Artemis broker? > > > > > > I'm not aware of any such whitepaper. > > > > > > That said, I think the first line of defense would actually be in > > > the network exposure rather than in the broker configuration. In > > > other words, only expose your broker to the segment of the network > > > where legitimate clients are located. You could even potentially use > > > a firewall to narrow exposure down to a set of specific IP addresses. > > > > > > As far as broker configuration is concerned I recommend you look at > > > setting connectionsAllowed [1] on the relevant acceptors in > > > broker.xml and take a look at resource-limits-settings [2]. > > > > > > And of course enforce security so that only legitimate clients can > > connect. > > > > > > Hope that helps! > > > > > > > > > Justin > > > > > > [1] > > > > > > https://acti/ > > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fco > > > nf > > > iguring-transports.html%23configuring-netty-tcp&data=05%7C02%7C%7Cd6 > > > c8 > > > 503c5b0c4c80724008dd786de948%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0% > > > 7C > > > 0%7C638799135676402406%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRyd > > > WU > > > sIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3 > > > D% > > > 7C0%7C%7C%7C&sdata=soScwxyUF6yuCvnYhnBx5vi%2BosXLSgZgNU%2BpDv7aJc4%3 > > > D& > > > reserved=0 > > > [2] > > > > > > https://acti/ > > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fre > > > so > > > urce-limits.html%23resource-limits&data=05%7C02%7C%7Cd6c8503c5b0c4c8 > > > 07 > > > 24008dd786de948%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387991 > > > 35 > > > 676430981%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLj > > > Au > > > MDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7 > > > C& > > > sdata=TveTJr9mCrsXXUKFD%2FnILP43%2FX2cQVgGIXEMqVgEcmk%3D&reserved=0 > > > > > > On Wed, Apr 9, 2025 at 8:20 AM Shiv Kumar Dixit > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > Thanks Justin for input. > > > > > > > > I am trying with OpenSSL and handshake-timeout=30 changes for now. > > > > I will update how it goes. Post that I will collect thread dumps > > > > and share for further analysis. > > > > > > > > On the side note - if there any article or whitepaper from > > > > community on how to handle (D)DoS attack on Artemis broker? > > > > > > > > Best Regards > > > > Shiv > > > > > > > > -----Original Message----- > > > > From: Justin Bertram <jbert...@apache.org> > > > > Sent: 08 April 2025 11:38 PM > > > > To: users@activemq.apache.org > > > > Subject: Re: Additional Info on certificate based authentication > > > > errors > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > > Review the content of the message carefully and verify the > > > > identity of the sender before acting on this email: replying, > > > > opening attachments or clicking links. > > > > > > > > > > > > > If JDK handles these handshakes as part of heap or native memory? > > > > > > > > I believe it would be part of the heap, but Netty uses direct > > > > buffers so perhaps that is related. > > > > > > > > > Here broker became unresponsive after lot of SSL handshake > > > > > errors as we > > > > could see lot of 'AMQ224088: Timeout (10 seconds)... > > > > > > > > A high volume of SSL connections can overwhelm the hardware on > > > > which the broker is running due to the CPU intensive nature of the > > > > cryptographic operations as well as memory requirements. This is > > > > one reason why folks use OpenSSL rather than the JDK. OpenSSL is > > > > typically faster and more efficient with hardware resources. It's > > > > impossible to say what's really happening based on the available > > > > evidence but you may simply be hitting hardware/OS limits. All the > > > > SSL connection work is being done before the broker is really > involved. > > > > > > > > In order to deal with the AMQ224088 errors you might try setting > > > > "handshake-timeout=30" on the relevant acceptor in broker.xml. The > > > > default timeout is 10 seconds. > > > > > > > > If you want to investigate further I suggest getting a handful of > > > > thread dumps during your test. Monitoring CPU utilization might be > > > > helpful as well. Also, if you're exporting metrics [1] (highly > > > > recommended!) you can enable optional metrics for Netty [2] to see > > > > details about direct and heap memory usage. > > > > > > > > > > > > Justin > > > > > > > > [1] > > > > > > > > https://acti/ > > > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2F > > > > me > > > > tr > > > > ics.html%23metrics&data=05%7C02%7C%7Cfb52ba57391b458763f208dd776e9 > > > > fd > > > > a% > > > > 7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638798039227600972%7C > > > > Un > > > > kn > > > > own%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAi > > > > Oi > > > > JX > > > > aW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nT41PG > > > > vq > > > > Oe > > > > rhYBh9VfodoTIFZerwDDE0cWp79YSX5OU%3D&reserved=0 > > > > [2] > > > > > > > > https://acti/ > > > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2F > > > > me > > > > tr > > > > ics.html%23optional-metrics&data=05%7C02%7C%7Cfb52ba57391b458763f2 > > > > 08 > > > > dd > > > > 776e9fda%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638798039227 > > > > 61 > > > > 38 > > > > 53%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > > > > wM > > > > CI > > > > sIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sda > > > > ta > > > > =C > > > > cTlTKQ%2B%2F6MIGiZRIrBa3pUPD75E4phfF5vbIJD9tcE%3D&reserved=0 > > > > > > > > On Tue, Apr 8, 2025 at 4:14 AM Shiv Kumar Dixit > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > Hi Justin, > > > > > Yes, we are getting exception in broker log > > > > > (javax.net.ssl.SSLHandshakeException: Empty server certificate > > > > > chain) and we are looking for more information along with just > > > > > client > > > IP. > > > > > > > > > > If JDK handles these handshakes as part of heap or native memory? > > > > > Our brokers are running in K8s (6 GB heap and 10 GB to pod) and > > > > > as per observations, pod is getting killed with OOM. We have OOM > > > > > dump instruction in artemis.profile but it is not happening > > > > > which means broker process was still running while pod was > > > > > killed and restarted by K8s. If the large native memory usage > > > > > caused by SSL handshake can cause pod to OOM and get killed? > > > > > > > > > > We tried to simulate similar case in VM based broker. Here > > > > > broker became unresponsive after lot of SSL handshake errors as > > > > > we could see lot of > > > > > 'AMQ224088: Timeout (10 seconds) on acceptor "artemis-ssl-customer" > > > > > during protocol handshake with /1.2.3.4:62403 has occurred.' > > > > > Where clients are trying to connect back. After some time it got > > > > > restarted > > > > with OOM. > > > > > > > > > > Best Regards > > > > > Shiv > > > > > > > > > > -----Original Message----- > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > Sent: 04 April 2025 09:26 PM > > > > > To: users@activemq.apache.org > > > > > Subject: Re: Additional Info on certificate based authentication > > > > > errors > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > > > Review the content of the message carefully and verify the > > > > > identity of the sender before acting on this email: replying, > > > > > opening attachments or clicking links. > > > > > > > > > > > > > > > > How can we find out when a client connects to broker using > > > > > > invalid > > > > > certificate... > > > > > > > > > > According to the information you've provided so far you're > > > > > already being notified about this in the broker log. Is that not > the case? > > > > > I was under the impression that you simply wanted more details > > > > > about the > > > > failure. > > > > > > > > > > > ...SSL handshake is terminated at what stage like client > > > > > > hello, server > > > > > hello or anything further? > > > > > > > > > > I'm not an SSL expert by any means, but I believe the failure > > > > > comes several steps after the client and server hello (i.e. the > > > > > very first steps in the handshake). After the broker sends its > > > > > certificate to the client and the client verifies it the broker > > > > > then requests the client's > > > > certificate. > > > > > When the client sends its certificate the broker verifies it. I > > > > > believe this is where the failure is happening. > > > > > > > > > > As noted previously, this handshake is being done by the JDK (or > > > > > potentially OpenSSL if you've chosen OPENSSL for the sslProvider > > [1]). > > > > > The broker doesn't control what information is available when a > > > > > failure > > > > occurs. > > > > > As far as I'm aware the broker is already providing all the > > > > > information available regarding an SSL handshake failure. > > > > > > > > > > > > > > > Justin > > > > > > > > > > [1] > > > > > > > > > > https://acti/ > > > > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest% > > > > > 2F > > > > > co > > > > > nf > > > > > iguring-transports.html%23configuring-netty-ssl&data=05%7C02%7C% > > > > > 7C > > > > > 49 > > > > > f8 > > > > > 42368dc64754902208dd76c85945%7C1a1dce2021b14beaa9d2130e9f1f6e2f% > > > > > 7C > > > > > 0% > > > > > 7C > > > > > 0%7C638797325066726960%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiO > > > > > nR > > > > > yd > > > > > WU > > > > > sIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ% > > > > > 3D > > > > > %3 > > > > > D% > > > > > 7C0%7C%7C%7C&sdata=esE%2FGWoSZocxz93n%2Fs8rye3Vhk%2BNFPLbtFEwhRQ > > > > > vS > > > > > Og > > > > > %3 > > > > > D&reserved=0 > > > > > > > > > > On Thu, Apr 3, 2025 at 9:58 AM Shiv Kumar Dixit > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > Hi Justin, > > > > > > We have defined our acceptors with needClientAuth=true as we > > > > > > want 2-way SSL. How can we find out when a client connects to > > > > > > broker using invalid certificate, SSL handshake is terminated > > > > > > at what stage like client hello, server hello or anything > further? > > > > > > In our case, we have a .net client which connects using URL > > > > > > like ssl://host:port?transport.clientCertSubject=CN=ABC1. If > > > > > > ABC1 cert is install on the client, it works fine but if > > > > > > actual CN is ABC and someone misconfigured CN as ABC1 then we > > > > > > get > > > > > javax.net.ssl.SSLHandshakeException: > > > > > > Empty server certificate chain error at broker. > > > > > > > > > > > > As per my understanding, client hello should go well as it has > > > > > > no client cert details but info like TLS version, cipher > > > > > > details > > etc. > > > > > > Server hello should also be fine and now server should be > > > > > > expecting client certificate details along with passing server > > > certificate. > > > > > > Now when client responds, it should be passing its > > > > > > certificate, and this validation should be failing at server > > > > > > side resulting in error > > > > like: > > > > > > AMQ222208: SSL handshake failed for client from /1.2.3.4:53838: > > > > > > javax.net.ssl.SSLHandshakeException: Empty server certificate > > chain. > > > > > > > > > > > > Best Regards > > > > > > Shiv > > > > > > > > > > > > -----Original Message----- > > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > > Sent: 03 April 2025 08:05 PM > > > > > > To: users@activemq.apache.org > > > > > > Subject: Re: Additional Info on certificate based > > > > > > authentication errors > > > > > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been > verified. > > > > > > Review the content of the message carefully and verify the > > > > > > identity of the sender before acting on this email: replying, > > > > > > opening attachments or clicking links. > > > > > > > > > > > > > > > > > > Are you wanting to extract cert data on the client or the broker? > > > > > > The broker already uses code like this to get cert data [1]. > > > > > > The problem in your case is that the SSL handshake fails so > > > > > > the broker has no chance to inspect the cert data on the > connection. > > > > > > > > > > > > > > > > > > Justin > > > > > > > > > > > > [1] > > > > > > > > > > > > https://gith/ > > > > > > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2Fmain%2Fartemis-cor > > > > > > e- > > > > > > cl > > > > > > ie > > > > > > nt > > > > > > %2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Fcor > > > > > > e% > > > > > > 2F > > > > > > re > > > > > > mo > > > > > > ting%2FCertificateUtil.java&data=05%7C02%7C%7C04b98fbbd3104431 > > > > > > 0c > > > > > > 05 > > > > > > 08 > > > > > > dd > > > > > > 73915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63879379 > > > > > > 04 > > > > > > 28 > > > > > > 19 > > > > > > 81 > > > > > > 82%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjA > > > > > > uM > > > > > > DA > > > > > > wM > > > > > > CI > > > > > > sIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C > > > > > > &s > > > > > > da > > > > > > ta > > > > > > =% > > > > > > 2BRmolH7RtdgtR0JDo6le65BYsJhxyCPXDBUxbNEPnSo%3D&reserved=0 > > > > > > > > > > > > On Thu, Apr 3, 2025 at 2:16 AM Shiv Kumar Dixit > > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > > > Hi Justin, > > > > > > > Thanks for input. > > > > > > > > > > > > > > Also, we came across this link https://stac/ > > > > > > > koverflow.com%2Fquestions%2F9777864%2Fhow-can-i-get-the-clie > > > > > > > nt > > > > > > > -c > > > > > > > er > > > > > > > ti > > > > > > > fi > > > > > > > cate-in-netty-handler-to-identify-user&data=05%7C02%7C%7Cee4 > > > > > > > e7 > > > > > > > ef > > > > > > > 7f > > > > > > > 3c > > > > > > > 84 > > > > > > > c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7 > > > > > > > C0 > > > > > > > %7 > > > > > > > C6 > > > > > > > 38 > > > > > > > 79 > > > > > > > 2877389315248%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydW > > > > > > > Us > > > > > > > Il > > > > > > > Yi > > > > > > > Oi > > > > > > > Iw > > > > > > > LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D > > > > > > > %7 > > > > > > > C0 > > > > > > > %7 > > > > > > > C% > > > > > > > 7C > > > > > > > %7C&sdata=M0a4HbeUymmwbtVFCOGPQRp5ZH%2Fmq7eLebjNuQT%2FVKk%3D > > > > > > > &r > > > > > > > es > > > > > > > er > > > > > > > ve > > > > > > > d= > > > > > > > 0 where it seems we can extract SSL information from > > > > > > > ChannelHandlerContext ctx object. Do you see if it is feasible? > > > > > > > > > > > > > > ChannelHandlerContext ctx = ... > > > > > > > SslHandler sslhandler = (SslHandler) > > > > > > > ctx.channel().pipeline().get("ssl"); > > > > > > > > > > > > > > sslhandler.engine().getSession().getPeerCertificateChain()[0]. > > > > > > > ge > > > > > > > tS > > > > > > > ub > > > > > > > je > > > > > > > ctDN()); > > > > > > > > > > > > > > Best Regards > > > > > > > Shiv > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > > > Sent: 28 March 2025 10:42 PM > > > > > > > To: users@activemq.apache.org > > > > > > > Subject: Re: Additional Info on certificate based > > > > > > > authentication errors > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been > > verified. > > > > > > > Review the content of the message carefully and verify the > > > > > > > identity of the sender before acting on this email: > > > > > > > replying, opening attachments or clicking links. > > > > > > > > > > > > > > > > > > > > > One option to improve this situation for Core [1] and > > > > > > > OpenWire [2] clients is to specify the "local" port which > > > > > > > will then be printed on in broker log along with the IP > > > > > > > address. By default both clients use a random, available > > > > > > > port which makes correlation difficult, if not impossible, in > my circumstances. > > > > > > > Of course, every client which does this will need to use a > > > > > > > unique port to ensure there are no conflicts if they are all > > > > > > > running on the same > > > > machine. > > > > > > > > > > > > > > Let me know where you reached out to the Netty community and > > > > > > > I'll follow along. > > > > > > > > > > > > > > > > > > > > > Justin > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > https://acti/ > > > > > > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flat > > > > > > > es > > > > > > > t% > > > > > > > 2F > > > > > > > co > > > > > > > nf > > > > > > > iguring-transports.html%23configuring-the-transport%23%3A~%3 > > > > > > > At > > > > > > > ex > > > > > > > t% > > > > > > > 3D > > > > > > > lo > > > > > > > calPort&data=05%7C02%7C%7Cee4e7ef7f3c84c0dd87208dd72bccbc9%7 > > > > > > > C1 > > > > > > > a1 > > > > > > > dc > > > > > > > e2 > > > > > > > 02 > > > > > > > 1b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638792877389342921%7CUnkno > > > > > > > wn > > > > > > > %7 > > > > > > > CT > > > > > > > WF > > > > > > > pb > > > > > > > GZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa > > > > > > > W4 > > > > > > > zM > > > > > > > iI > > > > > > > sI > > > > > > > kF > > > > > > > OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3wj4B1Un%2F > > > > > > > Ur > > > > > > > Pl > > > > > > > sk > > > > > > > Lh > > > > > > > OE > > > > > > > yDo7SQ%2FN0ou2OGnTgPvfp4nk%3D&reserved=0 > > > > > > > [2] > > > > > > > > > > > > > > https://acti/ > > > > > > > vemq.apache.org%2Fcomponents%2Fclassic%2Fdocumentation%2Fhow > > > > > > > -d > > > > > > > o- > > > > > > > i- > > > > > > > de > > > > > > > fi > > > > > > > ne-a-local-address-and-local-port-for-tcp-or-ssl&data=05%7C0 > > > > > > > 2% > > > > > > > 7C > > > > > > > %7 > > > > > > > Ce > > > > > > > e4 > > > > > > > e7ef7f3c84c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f1f > > > > > > > 6e > > > > > > > 2f > > > > > > > %7 > > > > > > > C0 > > > > > > > %7 > > > > > > > C0%7C638792877389363848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1h > > > > > > > cG > > > > > > > ki > > > > > > > On > > > > > > > Ry > > > > > > > dW > > > > > > > UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj > > > > > > > oy > > > > > > > fQ > > > > > > > %3 > > > > > > > D% > > > > > > > 3D > > > > > > > %7C0%7C%7C%7C&sdata=juvStcSQ%2BDcKH%2F14xA%2BdHuAUQUzrKHV0PK > > > > > > > 3m > > > > > > > M9 > > > > > > > A9 > > > > > > > ue > > > > > > > 4% > > > > > > > 3D&reserved=0 > > > > > > > > > > > > > > On Fri, Mar 28, 2025 at 10:12 AM Shiv Kumar Dixit > > > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > > > > > Hello Justin, > > > > > > > > Thanks for the response. > > > > > > > > > > > > > > > > We have clients using Core/AMQP/OpenWire with certificates. > > > > > > > > > > > > > > > > Passing SSL debugging flag is good idea but given we have > > > > > > > > many clients using broker concurrently, I fear we will > > > > > > > > have lot of logs and it will be difficult to analyse. I am > > > > > > > > also checking with netty community if they can provide some > input. > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Shiv > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > > > > Sent: 28 March 2025 08:33 AM > > > > > > > > To: users@activemq.apache.org > > > > > > > > Subject: Re: Additional Info on certificate based > > > > > > > > authentication errors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been > > > verified. > > > > > > > > Review the content of the message carefully and verify the > > > > > > > > identity of the sender before acting on this email: > > > > > > > > replying, opening attachments or clicking links. > > > > > > > > > > > > > > > > > > > > > > > > Unfortunately I don't think the broker itself will be able > > > > > > > > to provide additional information because it simply > > > > > > > > doesn't > > have it. > > > > > > > > The SSL handshake is delegated by Netty to the JDK (or > > > > > > > > potentially to OpenSSL) and it's the first thing done when > > > > > > > > the > > > > client connects. > > > > > > > > It happens before the messaging protocol handshake takes > > > > > > > > place so there's definitely no way to get the client ID > > > > > > > > because it hasn't been sent yet. Netty only provides a > > > > > > > > Throwable [1] and we print the relevant > > > > > > > information from it. > > > > > > > > > > > > > > > > It's possible you could get some relevant information from > > > > > > > > Java's SSL debug logging by passing something like > > > > > > > > -Djavax.net.debug=ssl:handshake to the JVM at startup. > > > > > > > > > > > > > > > > What kind of client are you using to connect to the broker > > > > > > > > in this > > > > > > case? > > > > > > > > > > > > > > > > > > > > > > > > Justin > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > https://gith/ > > > > > > > > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2Fc4b315ba0c1faf > > > > > > > > be > > > > > > > > 2d > > > > > > > > 2c > > > > > > > > 3e > > > > > > > > bf > > > > > > > > 0c > > > > > > > > dd84f921f42b5a%2Fartemis-server%2Fsrc%2Fmain%2Fjava%2Forg% > > > > > > > > 2F > > > > > > > > ap > > > > > > > > ac > > > > > > > > he > > > > > > > > %2 > > > > > > > > Fa > > > > > > > > ctivemq%2Fartemis%2Fcore%2Fremoting%2Fimpl%2Fnetty%2FNetty > > > > > > > > Ac > > > > > > > > ce > > > > > > > > pt > > > > > > > > or > > > > > > > > .j > > > > > > > > av > > > > > > > > a%23L1014&data=05%7C02%7C%7C2d25dad5c33e4432379808dd6e1bbb > > > > > > > > e3 > > > > > > > > %7 > > > > > > > > C1 > > > > > > > > a1 > > > > > > > > dc > > > > > > > > e2 > > > > > > > > 021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638787787610427088%7CU > > > > > > > > nk > > > > > > > > no > > > > > > > > wn > > > > > > > > %7 > > > > > > > > CT > > > > > > > > WF > > > > > > > > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiO > > > > > > > > iJ > > > > > > > > Xa > > > > > > > > W4 > > > > > > > > zM > > > > > > > > iI > > > > > > > > sI > > > > > > > > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nJBF0dM > > > > > > > > qm > > > > > > > > Oi > > > > > > > > PJ > > > > > > > > dl > > > > > > > > UM > > > > > > > > Bj > > > > > > > > 3q3Ia01NII0ykmP6XQisV%2Bao%3D&reserved=0 > > > > > > > > > > > > > > > > On Thu, Mar 27, 2025 at 2:03 AM Shiv Kumar Dixit > > > > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > > > > > > > Hi team, > > > > > > > > > Currently we are using Apache Artemis 2.37.0. We have > > > > > > > > > certificate-based customers. Many times, we see errors > > > > > > > > > like below in > > > > > > > > broker log: > > > > > > > > > > > > > > > > > > AMQ222208: SSL handshake failed for client from > > /a.b.c.d:53838: > > > > > > > > > javax.net.ssl.SSLHandshakeException: Empty server > > > > > > > > > certificate > > > > > chain. > > > > > > > > > > > > > > > > > > AMQ222208: SSL handshake failed for client from > > /a.b.c.d:59132: > > > > > > > > > java.security.cert.CertificateExpiredException: NotAfter: > > > > > > > > > Tue Mar > > > > > > > > > 25 > > > > > > > > > 00:00:11 IST 2025 > > > > > > > > > > > > > > > > > > AMQ224088: Timeout (10 seconds) on acceptor "artemis" > > > > > > > > > during protocol handshake with /a.b.c.d:62403 has occurred. > > > > > > > > > > > > > > > > > > Here we only get customer IP along with error message. > > > > > > > > > In real world, we have many customer applications > > > > > > > > > running from same IP but not all use expired certificate > > > > > > > > > or invalid certificate. We may have only of such > > > > > > > > > misbehaving > > client. > > > > > > > > > Troubleshooting such cases require validating all > > > > > > > > > customer applications on single IP to trace the issue. > > > > > > > > > It is very time consuming as we need to check customer > > > > > > > > > configuration, start/stop applications to see how it > > > > > > > > > impacts the > > > > > > > broker error etc. > > > > > > > > > > > > > > > > > > Can we get some additional information such as client > > > > > > > > > id, certificate CN details or any other certificate > > > > > > > > > information which can help us in identifying the erring > > client faster? > > > > > > > > > If there is any log level change which can enable to put > > > > > > > > > such information in broker > > > > > > > log will be helpful. > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > Shiv > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------- > > > > > > > > -- > > > > > > > > -- > > > > > > > > -- > > > > > > > > -- > > > > > > > > -- > > > > > > > > - To unsubscribe, e-mail: > > > > > > > > users-unsubscr...@activemq.apache.org > > > > > > > > For additional commands, e-mail: > > > > > > > > users-h...@activemq.apache.org For further information, > visit: > > > > > > > > https://acti/ > > > > > > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C2d25dad5c33e44 > > > > > > > > 32 > > > > > > > > 37 > > > > > > > > 98 > > > > > > > > 08 > > > > > > > > dd > > > > > > > > 6e > > > > > > > > 1bbbe3%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638787 > > > > > > > > 78 > > > > > > > > 76 > > > > > > > > 10 > > > > > > > > 43 > > > > > > > > 97 > > > > > > > > 97 > > > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL > > > > > > > > jA > > > > > > > > uM > > > > > > > > DA > > > > > > > > wM > > > > > > > > CI > > > > > > > > sI > > > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C% > > > > > > > > 7C > > > > > > > > &s > > > > > > > > da > > > > > > > > ta > > > > > > > > =X > > > > > > > > Rb > > > > > > > > daTuyVGNgBH%2FDdvihBuYzv%2BiATnFUfURp8IU2L%2B4%3D&reserved > > > > > > > > =0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------ > > > > > > > -- > > > > > > > -- > > > > > > > -- > > > > > > > -- > > > > > > > - To unsubscribe, e-mail: > > > > > > > users-unsubscr...@activemq.apache.org > > > > > > > For additional commands, e-mail: > > > > > > > users-h...@activemq.apache.org For further information, visit: > > > > > > > https://acti/ > > > > > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7Cee4e7ef7f3c84c0d > > > > > > > d8 > > > > > > > 72 > > > > > > > 08 > > > > > > > dd > > > > > > > 72 > > > > > > > bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63879287 > > > > > > > 73 > > > > > > > 89 > > > > > > > 38 > > > > > > > 52 > > > > > > > 94 > > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjA > > > > > > > uM > > > > > > > DA > > > > > > > wM > > > > > > > CI > > > > > > > sI > > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C > > > > > > > &s > > > > > > > da > > > > > > > ta > > > > > > > =P > > > > > > > Uy > > > > > > > z3W9mbHPKRghYMO58s5HWPMCKuIyITGbbJMIy%2BY0%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > -- > > > > > > -- > > > > > > -- > > > > > > - To unsubscribe, e-mail: > > > > > > users-unsubscr...@activemq.apache.org > > > > > > For additional commands, e-mail: > > > > > > users-h...@activemq.apache.org For further information, visit: > > > > > > https://acti/ > > > > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C04b98fbbd31044310c > > > > > > 05 > > > > > > 08 > > > > > > dd > > > > > > 73 > > > > > > 915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387937904 > > > > > > 28 > > > > > > 22 > > > > > > 95 > > > > > > 91 > > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM > > > > > > DA > > > > > > wM > > > > > > CI > > > > > > sI > > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&s > > > > > > da > > > > > > ta > > > > > > =k > > > > > > aL > > > > > > FLEBaBPzcqcq%2BJYRMJ4UbXRKaKHQzQthxEdfNHPw%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------- > > > > > -- > > > > > -- > > > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > > > For additional commands, e-mail: users-h...@activemq.apache.org > > > > > For further information, visit: > > > > > https://acti/ > > > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C49f842368dc647549022 > > > > > 08 > > > > > dd > > > > > 76 > > > > > c85945%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638797325066 > > > > > 73 > > > > > 96 > > > > > 57 > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > > > > > wM > > > > > CI > > > > > sI > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sda > > > > > ta > > > > > =H > > > > > 7m > > > > > 8XMqQ5GLVSPXcp25%2FyBx%2F3XdkKBQ6bM%2BkeloHL5U%3D&reserved=0 > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > > For additional commands, e-mail: users-h...@activemq.apache.org > > > > For further information, visit: > > > > https://acti/ > > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7Cfb52ba57391b458763f208 > > > > dd > > > > 77 > > > > 6e9fda%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63879803922762 > > > > 68 > > > > 34 > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwM > > > > CI > > > > sI > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata > > > > =O > > > > Sm > > > > JsMP3PS2rdYtqvDI34HvFvG2zztpNv%2FU2ZDoNUTs%3D&reserved=0 > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > For additional commands, e-mail: users-h...@activemq.apache.org For > > > further information, visit: > > > https://acti/ > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7Cd6c8503c5b0c4c80724008dd > > > 78 > > > 6de948%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387991356764527 > > > 73 > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCI > > > sI > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=s > > > Hk > > > 8yVgb24q77V5qQ39xx6etduy0KhqVak9vcf5S5YQ%3D&reserved=0 > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > For additional commands, e-mail: users-h...@activemq.apache.org For > > further information, visit: > > https://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Facti > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C24e1fb3bd58841ee2d1108dd7b > > 2d76ac%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638802157374044210 > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsI > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GMK > > r73S0iMWDJYQl9AmwKQxy6mjMML08UneehEHcxfs%3D&reserved=0 > > > > >
