Hi Justin, While moving from JDK to OPENSSL, I just defined sslProvider=openssl on the acceptor and broker seems to start fine. Is there any log or other indication that now SSL process has moved from JDK to OPENSSL? Unfortunately, I could not find any article on using OPENSSL with Artemis. Any input will be useful here.
Secondly regarding, "first line of defense would actually be in the network exposure rather than in the broker configuration", we have network segregation but all of the misbehaving clients come from trusted network. One same VM, we could have 2 listeners - one has correctly configured the certificate details while another has wrong configuration. Second one is causing " javax.net.ssl.SSLHandshakeException: Empty server certificate chain" error on the broker. Given we have dozens of VMs connecting to broker cluster and each VM hosting many producers/consumers where both good and bad configuration can co-exist, we are defenceless against this kind of issue. Here only option is to find IP and scan producer/consumer configuration and see if they have any misconfiguration causing trouble to brokers. Thirdly, regarding connectionsAllowed on acceptor, will it protect against misbehaving producer/consumer? I understood that it is only good enough to restrict good connections. Best Regards Shiv -----Original Message----- From: Justin Bertram <jbert...@apache.org> Sent: 09 April 2025 07:28 PM To: users@activemq.apache.org Subject: Re: Additional Info on certificate based authentication errors Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links. > ...any article or whitepaper from community on how to handle (D)DoS attack on Artemis broker? I'm not aware of any such whitepaper. That said, I think the first line of defense would actually be in the network exposure rather than in the broker configuration. In other words, only expose your broker to the segment of the network where legitimate clients are located. You could even potentially use a firewall to narrow exposure down to a set of specific IP addresses. As far as broker configuration is concerned I recommend you look at setting connectionsAllowed [1] on the relevant acceptors in broker.xml and take a look at resource-limits-settings [2]. And of course enforce security so that only legitimate clients can connect. Hope that helps! Justin [1] https://activemq.apache.org/components/artemis/documentation/latest/configuring-transports.html#configuring-netty-tcp [2] https://activemq.apache.org/components/artemis/documentation/latest/resource-limits.html#resource-limits On Wed, Apr 9, 2025 at 8:20 AM Shiv Kumar Dixit <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > Thanks Justin for input. > > I am trying with OpenSSL and handshake-timeout=30 changes for now. I > will update how it goes. Post that I will collect thread dumps and > share for further analysis. > > On the side note - if there any article or whitepaper from community > on how to handle (D)DoS attack on Artemis broker? > > Best Regards > Shiv > > -----Original Message----- > From: Justin Bertram <jbert...@apache.org> > Sent: 08 April 2025 11:38 PM > To: users@activemq.apache.org > Subject: Re: Additional Info on certificate based authentication > errors > > > > Unverified Sender: The sender of this email has not been verified. > Review the content of the message carefully and verify the identity of > the sender before acting on this email: replying, opening attachments > or clicking links. > > > > If JDK handles these handshakes as part of heap or native memory? > > I believe it would be part of the heap, but Netty uses direct buffers > so perhaps that is related. > > > Here broker became unresponsive after lot of SSL handshake errors as > > we > could see lot of 'AMQ224088: Timeout (10 seconds)... > > A high volume of SSL connections can overwhelm the hardware on which > the broker is running due to the CPU intensive nature of the > cryptographic operations as well as memory requirements. This is one > reason why folks use OpenSSL rather than the JDK. OpenSSL is typically > faster and more efficient with hardware resources. It's impossible to > say what's really happening based on the available evidence but you > may simply be hitting hardware/OS limits. All the SSL connection work > is being done before the broker is really involved. > > In order to deal with the AMQ224088 errors you might try setting > "handshake-timeout=30" on the relevant acceptor in broker.xml. The > default timeout is 10 seconds. > > If you want to investigate further I suggest getting a handful of > thread dumps during your test. Monitoring CPU utilization might be > helpful as well. Also, if you're exporting metrics [1] (highly > recommended!) you can enable optional metrics for Netty [2] to see > details about direct and heap memory usage. > > > Justin > > [1] > > https://acti/ > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fmetr > ics.html%23metrics&data=05%7C02%7C%7Cfb52ba57391b458763f208dd776e9fda% > 7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638798039227600972%7CUnkn > own%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJX > aW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nT41PGvqOe > rhYBh9VfodoTIFZerwDDE0cWp79YSX5OU%3D&reserved=0 > [2] > > https://acti/ > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fmetr > ics.html%23optional-metrics&data=05%7C02%7C%7Cfb52ba57391b458763f208dd > 776e9fda%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387980392276138 > 53%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCI > sIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=C > cTlTKQ%2B%2F6MIGiZRIrBa3pUPD75E4phfF5vbIJD9tcE%3D&reserved=0 > > On Tue, Apr 8, 2025 at 4:14 AM Shiv Kumar Dixit > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > Hi Justin, > > Yes, we are getting exception in broker log > > (javax.net.ssl.SSLHandshakeException: Empty server certificate > > chain) and we are looking for more information along with just client IP. > > > > If JDK handles these handshakes as part of heap or native memory? > > Our brokers are running in K8s (6 GB heap and 10 GB to pod) and as > > per observations, pod is getting killed with OOM. We have OOM dump > > instruction in artemis.profile but it is not happening which means > > broker process was still running while pod was killed and restarted > > by K8s. If the large native memory usage caused by SSL handshake can > > cause pod to OOM and get killed? > > > > We tried to simulate similar case in VM based broker. Here broker > > became unresponsive after lot of SSL handshake errors as we could > > see lot of > > 'AMQ224088: Timeout (10 seconds) on acceptor "artemis-ssl-customer" > > during protocol handshake with /1.2.3.4:62403 has occurred.' Where > > clients are trying to connect back. After some time it got restarted > with OOM. > > > > Best Regards > > Shiv > > > > -----Original Message----- > > From: Justin Bertram <jbert...@apache.org> > > Sent: 04 April 2025 09:26 PM > > To: users@activemq.apache.org > > Subject: Re: Additional Info on certificate based authentication > > errors > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > Review the content of the message carefully and verify the identity > > of the sender before acting on this email: replying, opening > > attachments or clicking links. > > > > > > > How can we find out when a client connects to broker using invalid > > certificate... > > > > According to the information you've provided so far you're already > > being notified about this in the broker log. Is that not the case? I > > was under the impression that you simply wanted more details about > > the > failure. > > > > > ...SSL handshake is terminated at what stage like client hello, > > > server > > hello or anything further? > > > > I'm not an SSL expert by any means, but I believe the failure comes > > several steps after the client and server hello (i.e. the very first > > steps in the handshake). After the broker sends its certificate to > > the client and the client verifies it the broker then requests the > > client's > certificate. > > When the client sends its certificate the broker verifies it. I > > believe this is where the failure is happening. > > > > As noted previously, this handshake is being done by the JDK (or > > potentially OpenSSL if you've chosen OPENSSL for the sslProvider [1]). > > The broker doesn't control what information is available when a > > failure > occurs. > > As far as I'm aware the broker is already providing all the > > information available regarding an SSL handshake failure. > > > > > > Justin > > > > [1] > > > > https://acti/ > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fco > > nf > > iguring-transports.html%23configuring-netty-ssl&data=05%7C02%7C%7C49 > > f8 > > 42368dc64754902208dd76c85945%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0% > > 7C > > 0%7C638797325066726960%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRyd > > WU > > sIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3 > > D% > > 7C0%7C%7C%7C&sdata=esE%2FGWoSZocxz93n%2Fs8rye3Vhk%2BNFPLbtFEwhRQvSOg > > %3 > > D&reserved=0 > > > > On Thu, Apr 3, 2025 at 9:58 AM Shiv Kumar Dixit > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > Hi Justin, > > > We have defined our acceptors with needClientAuth=true as we want > > > 2-way SSL. How can we find out when a client connects to broker > > > using invalid certificate, SSL handshake is terminated at what > > > stage like client hello, server hello or anything further? In our > > > case, we have a .net client which connects using URL like > > > ssl://host:port?transport.clientCertSubject=CN=ABC1. If ABC1 cert > > > is install on the client, it works fine but if actual CN is ABC > > > and someone misconfigured CN as ABC1 then we get > > javax.net.ssl.SSLHandshakeException: > > > Empty server certificate chain error at broker. > > > > > > As per my understanding, client hello should go well as it has no > > > client cert details but info like TLS version, cipher details etc. > > > Server hello should also be fine and now server should be > > > expecting client certificate details along with passing server > > > certificate. > > > Now when client responds, it should be passing its certificate, > > > and this validation should be failing at server side resulting in > > > error > like: > > > AMQ222208: SSL handshake failed for client from /1.2.3.4:53838: > > > javax.net.ssl.SSLHandshakeException: Empty server certificate chain. > > > > > > Best Regards > > > Shiv > > > > > > -----Original Message----- > > > From: Justin Bertram <jbert...@apache.org> > > > Sent: 03 April 2025 08:05 PM > > > To: users@activemq.apache.org > > > Subject: Re: Additional Info on certificate based authentication > > > errors > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > Review the content of the message carefully and verify the > > > identity of the sender before acting on this email: replying, > > > opening attachments or clicking links. > > > > > > > > > Are you wanting to extract cert data on the client or the broker? > > > The broker already uses code like this to get cert data [1]. The > > > problem in your case is that the SSL handshake fails so the broker > > > has no chance to inspect the cert data on the connection. > > > > > > > > > Justin > > > > > > [1] > > > > > > https://gith/ > > > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2Fmain%2Fartemis-core-cl > > > ie > > > nt > > > %2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Fcore%2F > > > re > > > mo > > > ting%2FCertificateUtil.java&data=05%7C02%7C%7C04b98fbbd31044310c05 > > > 08 > > > dd > > > 73915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638793790428 > > > 19 > > > 81 > > > 82%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > > > wM > > > CI > > > sIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sda > > > ta > > > =% > > > 2BRmolH7RtdgtR0JDo6le65BYsJhxyCPXDBUxbNEPnSo%3D&reserved=0 > > > > > > On Thu, Apr 3, 2025 at 2:16 AM Shiv Kumar Dixit > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > Hi Justin, > > > > Thanks for input. > > > > > > > > Also, we came across this link > > > > https://stac/ > > > > koverflow.com%2Fquestions%2F9777864%2Fhow-can-i-get-the-client-c > > > > er > > > > ti > > > > fi > > > > cate-in-netty-handler-to-identify-user&data=05%7C02%7C%7Cee4e7ef > > > > 7f > > > > 3c > > > > 84 > > > > c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7 > > > > C6 > > > > 38 > > > > 79 > > > > 2877389315248%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIl > > > > Yi > > > > Oi > > > > Iw > > > > LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0 > > > > %7 > > > > C% > > > > 7C > > > > %7C&sdata=M0a4HbeUymmwbtVFCOGPQRp5ZH%2Fmq7eLebjNuQT%2FVKk%3D&res > > > > er > > > > ve > > > > d= > > > > 0 where it seems we can extract SSL information from > > > > ChannelHandlerContext ctx object. Do you see if it is feasible? > > > > > > > > ChannelHandlerContext ctx = ... > > > > SslHandler sslhandler = (SslHandler) > > > > ctx.channel().pipeline().get("ssl"); > > > > > > > > sslhandler.engine().getSession().getPeerCertificateChain()[0].ge > > > > tS > > > > ub > > > > je > > > > ctDN()); > > > > > > > > Best Regards > > > > Shiv > > > > > > > > -----Original Message----- > > > > From: Justin Bertram <jbert...@apache.org> > > > > Sent: 28 March 2025 10:42 PM > > > > To: users@activemq.apache.org > > > > Subject: Re: Additional Info on certificate based authentication > > > > errors > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > > Review the content of the message carefully and verify the > > > > identity of the sender before acting on this email: replying, > > > > opening attachments or clicking links. > > > > > > > > > > > > One option to improve this situation for Core [1] and OpenWire > > > > [2] clients is to specify the "local" port which will then be > > > > printed on in broker log along with the IP address. By default > > > > both clients use a random, available port which makes > > > > correlation difficult, if not impossible, in my circumstances. > > > > Of course, every client which does this will need to use a > > > > unique port to ensure there are no conflicts if they are all > > > > running on the same > machine. > > > > > > > > Let me know where you reached out to the Netty community and > > > > I'll follow along. > > > > > > > > > > > > Justin > > > > > > > > [1] > > > > > > > > https://acti/ > > > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest% > > > > 2F > > > > co > > > > nf > > > > iguring-transports.html%23configuring-the-transport%23%3A~%3Atex > > > > t% > > > > 3D > > > > lo > > > > calPort&data=05%7C02%7C%7Cee4e7ef7f3c84c0dd87208dd72bccbc9%7C1a1 > > > > dc > > > > e2 > > > > 02 > > > > 1b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638792877389342921%7CUnknown%7 > > > > CT > > > > WF > > > > pb > > > > GZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zM > > > > iI > > > > sI > > > > kF > > > > OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3wj4B1Un%2FUrPl > > > > sk > > > > Lh > > > > OE > > > > yDo7SQ%2FN0ou2OGnTgPvfp4nk%3D&reserved=0 > > > > [2] > > > > > > > > https://acti/ > > > > vemq.apache.org%2Fcomponents%2Fclassic%2Fdocumentation%2Fhow-do- > > > > i- > > > > de > > > > fi > > > > ne-a-local-address-and-local-port-for-tcp-or-ssl&data=05%7C02%7C > > > > %7 > > > > Ce > > > > e4 > > > > e7ef7f3c84c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f > > > > %7 > > > > C0 > > > > %7 > > > > C0%7C638792877389363848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGki > > > > On > > > > Ry > > > > dW > > > > UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ > > > > %3 > > > > D% > > > > 3D > > > > %7C0%7C%7C%7C&sdata=juvStcSQ%2BDcKH%2F14xA%2BdHuAUQUzrKHV0PK3mM9 > > > > A9 > > > > ue > > > > 4% > > > > 3D&reserved=0 > > > > > > > > On Fri, Mar 28, 2025 at 10:12 AM Shiv Kumar Dixit > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > Hello Justin, > > > > > Thanks for the response. > > > > > > > > > > We have clients using Core/AMQP/OpenWire with certificates. > > > > > > > > > > Passing SSL debugging flag is good idea but given we have many > > > > > clients using broker concurrently, I fear we will have lot of > > > > > logs and it will be difficult to analyse. I am also checking > > > > > with netty community if they can provide some input. > > > > > > > > > > Best Regards > > > > > Shiv > > > > > > > > > > -----Original Message----- > > > > > From: Justin Bertram <jbert...@apache.org> > > > > > Sent: 28 March 2025 08:33 AM > > > > > To: users@activemq.apache.org > > > > > Subject: Re: Additional Info on certificate based > > > > > authentication errors > > > > > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > > > Review the content of the message carefully and verify the > > > > > identity of the sender before acting on this email: replying, > > > > > opening attachments or clicking links. > > > > > > > > > > > > > > > Unfortunately I don't think the broker itself will be able to > > > > > provide additional information because it simply doesn't have it. > > > > > The SSL handshake is delegated by Netty to the JDK (or > > > > > potentially to OpenSSL) and it's the first thing done when the > client connects. > > > > > It happens before the messaging protocol handshake takes place > > > > > so there's definitely no way to get the client ID because it > > > > > hasn't been sent yet. Netty only provides a Throwable [1] and > > > > > we print the relevant > > > > information from it. > > > > > > > > > > It's possible you could get some relevant information from > > > > > Java's SSL debug logging by passing something like > > > > > -Djavax.net.debug=ssl:handshake to the JVM at startup. > > > > > > > > > > What kind of client are you using to connect to the broker in > > > > > this > > > case? > > > > > > > > > > > > > > > Justin > > > > > > > > > > [1] > > > > > > > > > > https://gith/ > > > > > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2Fc4b315ba0c1fafbe2d > > > > > 2c > > > > > 3e > > > > > bf > > > > > 0c > > > > > dd84f921f42b5a%2Fartemis-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fap > > > > > ac > > > > > he > > > > > %2 > > > > > Fa > > > > > ctivemq%2Fartemis%2Fcore%2Fremoting%2Fimpl%2Fnetty%2FNettyAcce > > > > > pt > > > > > or > > > > > .j > > > > > av > > > > > a%23L1014&data=05%7C02%7C%7C2d25dad5c33e4432379808dd6e1bbbe3%7 > > > > > C1 > > > > > a1 > > > > > dc > > > > > e2 > > > > > 021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638787787610427088%7CUnkno > > > > > wn > > > > > %7 > > > > > CT > > > > > WF > > > > > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa > > > > > W4 > > > > > zM > > > > > iI > > > > > sI > > > > > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nJBF0dMqmOi > > > > > PJ > > > > > dl > > > > > UM > > > > > Bj > > > > > 3q3Ia01NII0ykmP6XQisV%2Bao%3D&reserved=0 > > > > > > > > > > On Thu, Mar 27, 2025 at 2:03 AM Shiv Kumar Dixit > > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > > > Hi team, > > > > > > Currently we are using Apache Artemis 2.37.0. We have > > > > > > certificate-based customers. Many times, we see errors like > > > > > > below in > > > > > broker log: > > > > > > > > > > > > AMQ222208: SSL handshake failed for client from /a.b.c.d:53838: > > > > > > javax.net.ssl.SSLHandshakeException: Empty server > > > > > > certificate > > chain. > > > > > > > > > > > > AMQ222208: SSL handshake failed for client from /a.b.c.d:59132: > > > > > > java.security.cert.CertificateExpiredException: NotAfter: > > > > > > Tue Mar > > > > > > 25 > > > > > > 00:00:11 IST 2025 > > > > > > > > > > > > AMQ224088: Timeout (10 seconds) on acceptor "artemis" during > > > > > > protocol handshake with /a.b.c.d:62403 has occurred. > > > > > > > > > > > > Here we only get customer IP along with error message. In > > > > > > real world, we have many customer applications running from > > > > > > same IP but not all use expired certificate or invalid > > > > > > certificate. We may have only of such misbehaving client. > > > > > > Troubleshooting such cases require validating all customer > > > > > > applications on single IP to trace the issue. It is very > > > > > > time consuming as we need to check customer configuration, > > > > > > start/stop applications to see how it impacts the > > > > broker error etc. > > > > > > > > > > > > Can we get some additional information such as client id, > > > > > > certificate CN details or any other certificate information > > > > > > which can help us in identifying the erring client faster? > > > > > > If there is any log level change which can enable to put > > > > > > such information in broker > > > > log will be helpful. > > > > > > > > > > > > Thanks > > > > > > Shiv > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > > > > -- > > > > > -- > > > > > -- > > > > > - To unsubscribe, e-mail: > > > > > users-unsubscr...@activemq.apache.org > > > > > For additional commands, e-mail: > > > > > users-h...@activemq.apache.org For further information, visit: > > > > > https://acti/ > > > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C2d25dad5c33e443237 > > > > > 98 > > > > > 08 > > > > > dd > > > > > 6e > > > > > 1bbbe3%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387877876 > > > > > 10 > > > > > 43 > > > > > 97 > > > > > 97 > > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM > > > > > DA > > > > > wM > > > > > CI > > > > > sI > > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&s > > > > > da > > > > > ta > > > > > =X > > > > > Rb > > > > > daTuyVGNgBH%2FDdvihBuYzv%2BiATnFUfURp8IU2L%2B4%3D&reserved=0 > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------- > > > > -- > > > > -- > > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > > For additional commands, e-mail: users-h...@activemq.apache.org > > > > For further information, visit: > > > > https://acti/ > > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7Cee4e7ef7f3c84c0dd872 > > > > 08 > > > > dd > > > > 72 > > > > bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638792877389 > > > > 38 > > > > 52 > > > > 94 > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > > > > wM > > > > CI > > > > sI > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sda > > > > ta > > > > =P > > > > Uy > > > > z3W9mbHPKRghYMO58s5HWPMCKuIyITGbbJMIy%2BY0%3D&reserved=0 > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > -- > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > For additional commands, e-mail: users-h...@activemq.apache.org > > > For further information, visit: > > > https://acti/ > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C04b98fbbd31044310c0508 > > > dd > > > 73 > > > 915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63879379042822 > > > 95 > > > 91 > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwM > > > CI > > > sI > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata > > > =k > > > aL > > > FLEBaBPzcqcq%2BJYRMJ4UbXRKaKHQzQthxEdfNHPw%3D&reserved=0 > > > > > > > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > For additional commands, e-mail: users-h...@activemq.apache.org For > > further information, visit: > > https://acti/ > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C49f842368dc64754902208dd > > 76 > > c85945%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387973250667396 > > 57 > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCI > > sI > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=H > > 7m > > 8XMqQ5GLVSPXcp25%2FyBx%2F3XdkKBQ6bM%2BkeloHL5U%3D&reserved=0 > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > For additional commands, e-mail: users-h...@activemq.apache.org For > further information, visit: > https://acti/ > vemq.apache.org%2Fcontact&data=05%7C02%7C%7Cfb52ba57391b458763f208dd77 > 6e9fda%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638798039227626834 > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsI > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=OSm > JsMP3PS2rdYtqvDI34HvFvG2zztpNv%2FU2ZDoNUTs%3D&reserved=0 > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org For additional commands, e-mail: users-h...@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
