Thanks Justin for input. I am trying with OpenSSL and handshake-timeout=30 changes for now. I will update how it goes. Post that I will collect thread dumps and share for further analysis.
On the side note - if there any article or whitepaper from community on how to handle (D)DoS attack on Artemis broker? Best Regards Shiv -----Original Message----- From: Justin Bertram <jbert...@apache.org> Sent: 08 April 2025 11:38 PM To: users@activemq.apache.org Subject: Re: Additional Info on certificate based authentication errors Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links. > If JDK handles these handshakes as part of heap or native memory? I believe it would be part of the heap, but Netty uses direct buffers so perhaps that is related. > Here broker became unresponsive after lot of SSL handshake errors as > we could see lot of 'AMQ224088: Timeout (10 seconds)... A high volume of SSL connections can overwhelm the hardware on which the broker is running due to the CPU intensive nature of the cryptographic operations as well as memory requirements. This is one reason why folks use OpenSSL rather than the JDK. OpenSSL is typically faster and more efficient with hardware resources. It's impossible to say what's really happening based on the available evidence but you may simply be hitting hardware/OS limits. All the SSL connection work is being done before the broker is really involved. In order to deal with the AMQ224088 errors you might try setting "handshake-timeout=30" on the relevant acceptor in broker.xml. The default timeout is 10 seconds. If you want to investigate further I suggest getting a handful of thread dumps during your test. Monitoring CPU utilization might be helpful as well. Also, if you're exporting metrics [1] (highly recommended!) you can enable optional metrics for Netty [2] to see details about direct and heap memory usage. Justin [1] https://activemq.apache.org/components/artemis/documentation/latest/metrics.html#metrics [2] https://activemq.apache.org/components/artemis/documentation/latest/metrics.html#optional-metrics On Tue, Apr 8, 2025 at 4:14 AM Shiv Kumar Dixit <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > Hi Justin, > Yes, we are getting exception in broker log > (javax.net.ssl.SSLHandshakeException: Empty server certificate chain) > and we are looking for more information along with just client IP. > > If JDK handles these handshakes as part of heap or native memory? Our > brokers are running in K8s (6 GB heap and 10 GB to pod) and as per > observations, pod is getting killed with OOM. We have OOM dump > instruction in artemis.profile but it is not happening which means > broker process was still running while pod was killed and restarted by > K8s. If the large native memory usage caused by SSL handshake can > cause pod to OOM and get killed? > > We tried to simulate similar case in VM based broker. Here broker > became unresponsive after lot of SSL handshake errors as we could see > lot of > 'AMQ224088: Timeout (10 seconds) on acceptor "artemis-ssl-customer" > during protocol handshake with /1.2.3.4:62403 has occurred.' Where > clients are trying to connect back. After some time it got restarted with OOM. > > Best Regards > Shiv > > -----Original Message----- > From: Justin Bertram <jbert...@apache.org> > Sent: 04 April 2025 09:26 PM > To: users@activemq.apache.org > Subject: Re: Additional Info on certificate based authentication > errors > > > > Unverified Sender: The sender of this email has not been verified. > Review the content of the message carefully and verify the identity of > the sender before acting on this email: replying, opening attachments > or clicking links. > > > > How can we find out when a client connects to broker using invalid > certificate... > > According to the information you've provided so far you're already > being notified about this in the broker log. Is that not the case? I > was under the impression that you simply wanted more details about the > failure. > > > ...SSL handshake is terminated at what stage like client hello, > > server > hello or anything further? > > I'm not an SSL expert by any means, but I believe the failure comes > several steps after the client and server hello (i.e. the very first > steps in the handshake). After the broker sends its certificate to the > client and the client verifies it the broker then requests the client's > certificate. > When the client sends its certificate the broker verifies it. I > believe this is where the failure is happening. > > As noted previously, this handshake is being done by the JDK (or > potentially OpenSSL if you've chosen OPENSSL for the sslProvider [1]). > The broker doesn't control what information is available when a failure > occurs. > As far as I'm aware the broker is already providing all the > information available regarding an SSL handshake failure. > > > Justin > > [1] > > https://acti/ > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fconf > iguring-transports.html%23configuring-netty-ssl&data=05%7C02%7C%7C49f8 > 42368dc64754902208dd76c85945%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C > 0%7C638797325066726960%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWU > sIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D% > 7C0%7C%7C%7C&sdata=esE%2FGWoSZocxz93n%2Fs8rye3Vhk%2BNFPLbtFEwhRQvSOg%3 > D&reserved=0 > > On Thu, Apr 3, 2025 at 9:58 AM Shiv Kumar Dixit > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > Hi Justin, > > We have defined our acceptors with needClientAuth=true as we want > > 2-way SSL. How can we find out when a client connects to broker > > using invalid certificate, SSL handshake is terminated at what stage > > like client hello, server hello or anything further? In our case, we > > have a .net client which connects using URL like > > ssl://host:port?transport.clientCertSubject=CN=ABC1. If ABC1 cert is > > install on the client, it works fine but if actual CN is ABC and > > someone misconfigured CN as ABC1 then we get > javax.net.ssl.SSLHandshakeException: > > Empty server certificate chain error at broker. > > > > As per my understanding, client hello should go well as it has no > > client cert details but info like TLS version, cipher details etc. > > Server hello should also be fine and now server should be expecting > > client certificate details along with passing server certificate. > > Now when client responds, it should be passing its certificate, and > > this validation should be failing at server side resulting in error like: > > AMQ222208: SSL handshake failed for client from /1.2.3.4:53838: > > javax.net.ssl.SSLHandshakeException: Empty server certificate chain. > > > > Best Regards > > Shiv > > > > -----Original Message----- > > From: Justin Bertram <jbert...@apache.org> > > Sent: 03 April 2025 08:05 PM > > To: users@activemq.apache.org > > Subject: Re: Additional Info on certificate based authentication > > errors > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > Review the content of the message carefully and verify the identity > > of the sender before acting on this email: replying, opening > > attachments or clicking links. > > > > > > Are you wanting to extract cert data on the client or the broker? > > The broker already uses code like this to get cert data [1]. The > > problem in your case is that the SSL handshake fails so the broker > > has no chance to inspect the cert data on the connection. > > > > > > Justin > > > > [1] > > > > https://gith/ > > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2Fmain%2Fartemis-core-clie > > nt > > %2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Fcore%2Fre > > mo > > ting%2FCertificateUtil.java&data=05%7C02%7C%7C04b98fbbd31044310c0508 > > dd > > 73915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63879379042819 > > 81 > > 82%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwM > > CI > > sIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata > > =% > > 2BRmolH7RtdgtR0JDo6le65BYsJhxyCPXDBUxbNEPnSo%3D&reserved=0 > > > > On Thu, Apr 3, 2025 at 2:16 AM Shiv Kumar Dixit > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > Hi Justin, > > > Thanks for input. > > > > > > Also, we came across this link > > > https://stac/ > > > koverflow.com%2Fquestions%2F9777864%2Fhow-can-i-get-the-client-cer > > > ti > > > fi > > > cate-in-netty-handler-to-identify-user&data=05%7C02%7C%7Cee4e7ef7f > > > 3c > > > 84 > > > c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6 > > > 38 > > > 79 > > > 2877389315248%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYi > > > Oi > > > Iw > > > LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7 > > > C% > > > 7C > > > %7C&sdata=M0a4HbeUymmwbtVFCOGPQRp5ZH%2Fmq7eLebjNuQT%2FVKk%3D&reser > > > ve > > > d= > > > 0 where it seems we can extract SSL information from > > > ChannelHandlerContext ctx object. Do you see if it is feasible? > > > > > > ChannelHandlerContext ctx = ... > > > SslHandler sslhandler = (SslHandler) > > > ctx.channel().pipeline().get("ssl"); > > > > > > sslhandler.engine().getSession().getPeerCertificateChain()[0].getS > > > ub > > > je > > > ctDN()); > > > > > > Best Regards > > > Shiv > > > > > > -----Original Message----- > > > From: Justin Bertram <jbert...@apache.org> > > > Sent: 28 March 2025 10:42 PM > > > To: users@activemq.apache.org > > > Subject: Re: Additional Info on certificate based authentication > > > errors > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > Review the content of the message carefully and verify the > > > identity of the sender before acting on this email: replying, > > > opening attachments or clicking links. > > > > > > > > > One option to improve this situation for Core [1] and OpenWire [2] > > > clients is to specify the "local" port which will then be printed > > > on in broker log along with the IP address. By default both > > > clients use a random, available port which makes correlation > > > difficult, if not impossible, in my circumstances. Of course, > > > every client which does this will need to use a unique port to > > > ensure there are no conflicts if they are all running on the same machine. > > > > > > Let me know where you reached out to the Netty community and I'll > > > follow along. > > > > > > > > > Justin > > > > > > [1] > > > > > > https://acti/ > > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2F > > > co > > > nf > > > iguring-transports.html%23configuring-the-transport%23%3A~%3Atext% > > > 3D > > > lo > > > calPort&data=05%7C02%7C%7Cee4e7ef7f3c84c0dd87208dd72bccbc9%7C1a1dc > > > e2 > > > 02 > > > 1b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638792877389342921%7CUnknown%7CT > > > WF > > > pb > > > GZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiI > > > sI > > > kF > > > OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3wj4B1Un%2FUrPlsk > > > Lh > > > OE > > > yDo7SQ%2FN0ou2OGnTgPvfp4nk%3D&reserved=0 > > > [2] > > > > > > https://acti/ > > > vemq.apache.org%2Fcomponents%2Fclassic%2Fdocumentation%2Fhow-do-i- > > > de > > > fi > > > ne-a-local-address-and-local-port-for-tcp-or-ssl&data=05%7C02%7C%7 > > > Ce > > > e4 > > > e7ef7f3c84c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7 > > > C0 > > > %7 > > > C0%7C638792877389363848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOn > > > Ry > > > dW > > > UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3 > > > D% > > > 3D > > > %7C0%7C%7C%7C&sdata=juvStcSQ%2BDcKH%2F14xA%2BdHuAUQUzrKHV0PK3mM9A9 > > > ue > > > 4% > > > 3D&reserved=0 > > > > > > On Fri, Mar 28, 2025 at 10:12 AM Shiv Kumar Dixit > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > Hello Justin, > > > > Thanks for the response. > > > > > > > > We have clients using Core/AMQP/OpenWire with certificates. > > > > > > > > Passing SSL debugging flag is good idea but given we have many > > > > clients using broker concurrently, I fear we will have lot of > > > > logs and it will be difficult to analyse. I am also checking > > > > with netty community if they can provide some input. > > > > > > > > Best Regards > > > > Shiv > > > > > > > > -----Original Message----- > > > > From: Justin Bertram <jbert...@apache.org> > > > > Sent: 28 March 2025 08:33 AM > > > > To: users@activemq.apache.org > > > > Subject: Re: Additional Info on certificate based authentication > > > > errors > > > > > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > > Review the content of the message carefully and verify the > > > > identity of the sender before acting on this email: replying, > > > > opening attachments or clicking links. > > > > > > > > > > > > Unfortunately I don't think the broker itself will be able to > > > > provide additional information because it simply doesn't have it. > > > > The SSL handshake is delegated by Netty to the JDK (or > > > > potentially to OpenSSL) and it's the first thing done when the client > > > > connects. > > > > It happens before the messaging protocol handshake takes place > > > > so there's definitely no way to get the client ID because it > > > > hasn't been sent yet. Netty only provides a Throwable [1] and we > > > > print the relevant > > > information from it. > > > > > > > > It's possible you could get some relevant information from > > > > Java's SSL debug logging by passing something like > > > > -Djavax.net.debug=ssl:handshake to the JVM at startup. > > > > > > > > What kind of client are you using to connect to the broker in > > > > this > > case? > > > > > > > > > > > > Justin > > > > > > > > [1] > > > > > > > > https://gith/ > > > > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2Fc4b315ba0c1fafbe2d2c > > > > 3e > > > > bf > > > > 0c > > > > dd84f921f42b5a%2Fartemis-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapac > > > > he > > > > %2 > > > > Fa > > > > ctivemq%2Fartemis%2Fcore%2Fremoting%2Fimpl%2Fnetty%2FNettyAccept > > > > or > > > > .j > > > > av > > > > a%23L1014&data=05%7C02%7C%7C2d25dad5c33e4432379808dd6e1bbbe3%7C1 > > > > a1 > > > > dc > > > > e2 > > > > 021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638787787610427088%7CUnknown > > > > %7 > > > > CT > > > > WF > > > > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 > > > > zM > > > > iI > > > > sI > > > > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nJBF0dMqmOiPJ > > > > dl > > > > UM > > > > Bj > > > > 3q3Ia01NII0ykmP6XQisV%2Bao%3D&reserved=0 > > > > > > > > On Thu, Mar 27, 2025 at 2:03 AM Shiv Kumar Dixit > > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > > > Hi team, > > > > > Currently we are using Apache Artemis 2.37.0. We have > > > > > certificate-based customers. Many times, we see errors like > > > > > below in > > > > broker log: > > > > > > > > > > AMQ222208: SSL handshake failed for client from /a.b.c.d:53838: > > > > > javax.net.ssl.SSLHandshakeException: Empty server certificate > chain. > > > > > > > > > > AMQ222208: SSL handshake failed for client from /a.b.c.d:59132: > > > > > java.security.cert.CertificateExpiredException: NotAfter: Tue > > > > > Mar > > > > > 25 > > > > > 00:00:11 IST 2025 > > > > > > > > > > AMQ224088: Timeout (10 seconds) on acceptor "artemis" during > > > > > protocol handshake with /a.b.c.d:62403 has occurred. > > > > > > > > > > Here we only get customer IP along with error message. In real > > > > > world, we have many customer applications running from same IP > > > > > but not all use expired certificate or invalid certificate. We > > > > > may have only of such misbehaving client. Troubleshooting such > > > > > cases require validating all customer applications on single > > > > > IP to trace the issue. It is very time consuming as we need to > > > > > check customer configuration, start/stop applications to see > > > > > how it impacts the > > > broker error etc. > > > > > > > > > > Can we get some additional information such as client id, > > > > > certificate CN details or any other certificate information > > > > > which can help us in identifying the erring client faster? If > > > > > there is any log level change which can enable to put such > > > > > information in broker > > > log will be helpful. > > > > > > > > > > Thanks > > > > > Shiv > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------- > > > > -- > > > > -- > > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > > For additional commands, e-mail: users-h...@activemq.apache.org > > > > For further information, visit: > > > > https://acti/ > > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C2d25dad5c33e44323798 > > > > 08 > > > > dd > > > > 6e > > > > 1bbbe3%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638787787610 > > > > 43 > > > > 97 > > > > 97 > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > > > > wM > > > > CI > > > > sI > > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sda > > > > ta > > > > =X > > > > Rb > > > > daTuyVGNgBH%2FDdvihBuYzv%2BiATnFUfURp8IU2L%2B4%3D&reserved=0 > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > -- > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > For additional commands, e-mail: users-h...@activemq.apache.org > > > For further information, visit: > > > https://acti/ > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7Cee4e7ef7f3c84c0dd87208 > > > dd > > > 72 > > > bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63879287738938 > > > 52 > > > 94 > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwM > > > CI > > > sI > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata > > > =P > > > Uy > > > z3W9mbHPKRghYMO58s5HWPMCKuIyITGbbJMIy%2BY0%3D&reserved=0 > > > > > > > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > For additional commands, e-mail: users-h...@activemq.apache.org For > > further information, visit: > > https://acti/ > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C04b98fbbd31044310c0508dd > > 73 > > 915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387937904282295 > > 91 > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCI > > sI > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k > > aL > > FLEBaBPzcqcq%2BJYRMJ4UbXRKaKHQzQthxEdfNHPw%3D&reserved=0 > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > For additional commands, e-mail: users-h...@activemq.apache.org For > further information, visit: > https://acti/ > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C49f842368dc64754902208dd76 > c85945%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638797325066739657 > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsI > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=H7m > 8XMqQ5GLVSPXcp25%2FyBx%2F3XdkKBQ6bM%2BkeloHL5U%3D&reserved=0 > >
