Hi Justin, Yes, we are getting exception in broker log (javax.net.ssl.SSLHandshakeException: Empty server certificate chain) and we are looking for more information along with just client IP.
If JDK handles these handshakes as part of heap or native memory? Our brokers are running in K8s (6 GB heap and 10 GB to pod) and as per observations, pod is getting killed with OOM. We have OOM dump instruction in artemis.profile but it is not happening which means broker process was still running while pod was killed and restarted by K8s. If the large native memory usage caused by SSL handshake can cause pod to OOM and get killed? We tried to simulate similar case in VM based broker. Here broker became unresponsive after lot of SSL handshake errors as we could see lot of 'AMQ224088: Timeout (10 seconds) on acceptor "artemis-ssl-customer" during protocol handshake with /1.2.3.4:62403 has occurred.' Where clients are trying to connect back. After some time it got restarted with OOM. Best Regards Shiv -----Original Message----- From: Justin Bertram <jbert...@apache.org> Sent: 04 April 2025 09:26 PM To: users@activemq.apache.org Subject: Re: Additional Info on certificate based authentication errors Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links. > How can we find out when a client connects to broker using invalid certificate... According to the information you've provided so far you're already being notified about this in the broker log. Is that not the case? I was under the impression that you simply wanted more details about the failure. > ...SSL handshake is terminated at what stage like client hello, server hello or anything further? I'm not an SSL expert by any means, but I believe the failure comes several steps after the client and server hello (i.e. the very first steps in the handshake). After the broker sends its certificate to the client and the client verifies it the broker then requests the client's certificate. When the client sends its certificate the broker verifies it. I believe this is where the failure is happening. As noted previously, this handshake is being done by the JDK (or potentially OpenSSL if you've chosen OPENSSL for the sslProvider [1]). The broker doesn't control what information is available when a failure occurs. As far as I'm aware the broker is already providing all the information available regarding an SSL handshake failure. Justin [1] https://activemq.apache.org/components/artemis/documentation/latest/configuring-transports.html#configuring-netty-ssl On Thu, Apr 3, 2025 at 9:58 AM Shiv Kumar Dixit <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > Hi Justin, > We have defined our acceptors with needClientAuth=true as we want > 2-way SSL. How can we find out when a client connects to broker using > invalid certificate, SSL handshake is terminated at what stage like > client hello, server hello or anything further? In our case, we have a > .net client which connects using URL like > ssl://host:port?transport.clientCertSubject=CN=ABC1. If ABC1 cert is > install on the client, it works fine but if actual CN is ABC and > someone misconfigured CN as ABC1 then we get > javax.net.ssl.SSLHandshakeException: > Empty server certificate chain error at broker. > > As per my understanding, client hello should go well as it has no > client cert details but info like TLS version, cipher details etc. > Server hello should also be fine and now server should be expecting > client certificate details along with passing server certificate. Now > when client responds, it should be passing its certificate, and this > validation should be failing at server side resulting in error like: > AMQ222208: SSL handshake failed for client from /1.2.3.4:53838: > javax.net.ssl.SSLHandshakeException: Empty server certificate chain. > > Best Regards > Shiv > > -----Original Message----- > From: Justin Bertram <jbert...@apache.org> > Sent: 03 April 2025 08:05 PM > To: users@activemq.apache.org > Subject: Re: Additional Info on certificate based authentication > errors > > > > Unverified Sender: The sender of this email has not been verified. > Review the content of the message carefully and verify the identity of > the sender before acting on this email: replying, opening attachments > or clicking links. > > > Are you wanting to extract cert data on the client or the broker? The > broker already uses code like this to get cert data [1]. The problem > in your case is that the SSL handshake fails so the broker has no > chance to inspect the cert data on the connection. > > > Justin > > [1] > > https://gith/ > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2Fmain%2Fartemis-core-client > %2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Fcore%2Fremo > ting%2FCertificateUtil.java&data=05%7C02%7C%7C04b98fbbd31044310c0508dd > 73915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387937904281981 > 82%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCI > sIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=% > 2BRmolH7RtdgtR0JDo6le65BYsJhxyCPXDBUxbNEPnSo%3D&reserved=0 > > On Thu, Apr 3, 2025 at 2:16 AM Shiv Kumar Dixit > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > Hi Justin, > > Thanks for input. > > > > Also, we came across this link > > https://stac/ > > koverflow.com%2Fquestions%2F9777864%2Fhow-can-i-get-the-client-certi > > fi > > cate-in-netty-handler-to-identify-user&data=05%7C02%7C%7Cee4e7ef7f3c > > 84 > > c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638 > > 79 > > 2877389315248%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOi > > Iw > > LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C% > > 7C > > %7C&sdata=M0a4HbeUymmwbtVFCOGPQRp5ZH%2Fmq7eLebjNuQT%2FVKk%3D&reserve > > d= > > 0 where it seems we can extract SSL information from > > ChannelHandlerContext ctx object. Do you see if it is feasible? > > > > ChannelHandlerContext ctx = ... > > SslHandler sslhandler = (SslHandler) > > ctx.channel().pipeline().get("ssl"); > > > > sslhandler.engine().getSession().getPeerCertificateChain()[0].getSub > > je > > ctDN()); > > > > Best Regards > > Shiv > > > > -----Original Message----- > > From: Justin Bertram <jbert...@apache.org> > > Sent: 28 March 2025 10:42 PM > > To: users@activemq.apache.org > > Subject: Re: Additional Info on certificate based authentication > > errors > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > Review the content of the message carefully and verify the identity > > of the sender before acting on this email: replying, opening > > attachments or clicking links. > > > > > > One option to improve this situation for Core [1] and OpenWire [2] > > clients is to specify the "local" port which will then be printed on > > in broker log along with the IP address. By default both clients use > > a random, available port which makes correlation difficult, if not > > impossible, in my circumstances. Of course, every client which does > > this will need to use a unique port to ensure there are no conflicts > > if they are all running on the same machine. > > > > Let me know where you reached out to the Netty community and I'll > > follow along. > > > > > > Justin > > > > [1] > > > > https://acti/ > > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fco > > nf > > iguring-transports.html%23configuring-the-transport%23%3A~%3Atext%3D > > lo > > calPort&data=05%7C02%7C%7Cee4e7ef7f3c84c0dd87208dd72bccbc9%7C1a1dce2 > > 02 > > 1b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638792877389342921%7CUnknown%7CTWF > > pb > > GZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsI > > kF > > OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3wj4B1Un%2FUrPlskLh > > OE > > yDo7SQ%2FN0ou2OGnTgPvfp4nk%3D&reserved=0 > > [2] > > > > https://acti/ > > vemq.apache.org%2Fcomponents%2Fclassic%2Fdocumentation%2Fhow-do-i-de > > fi > > ne-a-local-address-and-local-port-for-tcp-or-ssl&data=05%7C02%7C%7Ce > > e4 > > e7ef7f3c84c0dd87208dd72bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0 > > %7 > > C0%7C638792877389363848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy > > dW > > UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D% > > 3D > > %7C0%7C%7C%7C&sdata=juvStcSQ%2BDcKH%2F14xA%2BdHuAUQUzrKHV0PK3mM9A9ue > > 4% > > 3D&reserved=0 > > > > On Fri, Mar 28, 2025 at 10:12 AM Shiv Kumar Dixit > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > Hello Justin, > > > Thanks for the response. > > > > > > We have clients using Core/AMQP/OpenWire with certificates. > > > > > > Passing SSL debugging flag is good idea but given we have many > > > clients using broker concurrently, I fear we will have lot of logs > > > and it will be difficult to analyse. I am also checking with netty > > > community if they can provide some input. > > > > > > Best Regards > > > Shiv > > > > > > -----Original Message----- > > > From: Justin Bertram <jbert...@apache.org> > > > Sent: 28 March 2025 08:33 AM > > > To: users@activemq.apache.org > > > Subject: Re: Additional Info on certificate based authentication > > > errors > > > > > > > > > > > > Unverified Sender: The sender of this email has not been verified. > > > Review the content of the message carefully and verify the > > > identity of the sender before acting on this email: replying, > > > opening attachments or clicking links. > > > > > > > > > Unfortunately I don't think the broker itself will be able to > > > provide additional information because it simply doesn't have it. > > > The SSL handshake is delegated by Netty to the JDK (or potentially > > > to OpenSSL) and it's the first thing done when the client connects. > > > It happens before the messaging protocol handshake takes place so > > > there's definitely no way to get the client ID because it hasn't > > > been sent yet. Netty only provides a Throwable [1] and we print > > > the relevant > > information from it. > > > > > > It's possible you could get some relevant information from Java's > > > SSL debug logging by passing something like > > > -Djavax.net.debug=ssl:handshake to the JVM at startup. > > > > > > What kind of client are you using to connect to the broker in this > case? > > > > > > > > > Justin > > > > > > [1] > > > > > > https://gith/ > > > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2Fc4b315ba0c1fafbe2d2c3e > > > bf > > > 0c > > > dd84f921f42b5a%2Fartemis-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache > > > %2 > > > Fa > > > ctivemq%2Fartemis%2Fcore%2Fremoting%2Fimpl%2Fnetty%2FNettyAcceptor > > > .j > > > av > > > a%23L1014&data=05%7C02%7C%7C2d25dad5c33e4432379808dd6e1bbbe3%7C1a1 > > > dc > > > e2 > > > 021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638787787610427088%7CUnknown%7 > > > CT > > > WF > > > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zM > > > iI > > > sI > > > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=nJBF0dMqmOiPJdl > > > UM > > > Bj > > > 3q3Ia01NII0ykmP6XQisV%2Bao%3D&reserved=0 > > > > > > On Thu, Mar 27, 2025 at 2:03 AM Shiv Kumar Dixit > > > <shivkumar.di...@it.eurofinseu.com.invalid> wrote: > > > > > > > Hi team, > > > > Currently we are using Apache Artemis 2.37.0. We have > > > > certificate-based customers. Many times, we see errors like > > > > below in > > > broker log: > > > > > > > > AMQ222208: SSL handshake failed for client from /a.b.c.d:53838: > > > > javax.net.ssl.SSLHandshakeException: Empty server certificate chain. > > > > > > > > AMQ222208: SSL handshake failed for client from /a.b.c.d:59132: > > > > java.security.cert.CertificateExpiredException: NotAfter: Tue > > > > Mar > > > > 25 > > > > 00:00:11 IST 2025 > > > > > > > > AMQ224088: Timeout (10 seconds) on acceptor "artemis" during > > > > protocol handshake with /a.b.c.d:62403 has occurred. > > > > > > > > Here we only get customer IP along with error message. In real > > > > world, we have many customer applications running from same IP > > > > but not all use expired certificate or invalid certificate. We > > > > may have only of such misbehaving client. Troubleshooting such > > > > cases require validating all customer applications on single IP > > > > to trace the issue. It is very time consuming as we need to > > > > check customer configuration, start/stop applications to see how > > > > it impacts the > > broker error etc. > > > > > > > > Can we get some additional information such as client id, > > > > certificate CN details or any other certificate information > > > > which can help us in identifying the erring client faster? If > > > > there is any log level change which can enable to put such > > > > information in broker > > log will be helpful. > > > > > > > > Thanks > > > > Shiv > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > -- > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > For additional commands, e-mail: users-h...@activemq.apache.org > > > For further information, visit: > > > https://acti/ > > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C2d25dad5c33e4432379808 > > > dd > > > 6e > > > 1bbbe3%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C63878778761043 > > > 97 > > > 97 > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwM > > > CI > > > sI > > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata > > > =X > > > Rb > > > daTuyVGNgBH%2FDdvihBuYzv%2BiATnFUfURp8IU2L%2B4%3D&reserved=0 > > > > > > > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > For additional commands, e-mail: users-h...@activemq.apache.org For > > further information, visit: > > https://acti/ > > vemq.apache.org%2Fcontact&data=05%7C02%7C%7Cee4e7ef7f3c84c0dd87208dd > > 72 > > bccbc9%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C6387928773893852 > > 94 > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCI > > sI > > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=P > > Uy > > z3W9mbHPKRghYMO58s5HWPMCKuIyITGbbJMIy%2BY0%3D&reserved=0 > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > For additional commands, e-mail: users-h...@activemq.apache.org For > further information, visit: > https://acti/ > vemq.apache.org%2Fcontact&data=05%7C02%7C%7C04b98fbbd31044310c0508dd73 > 915e0f%7C1a1dce2021b14beaa9d2130e9f1f6e2f%7C0%7C0%7C638793790428229591 > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsI > lAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kaL > FLEBaBPzcqcq%2BJYRMJ4UbXRKaKHQzQthxEdfNHPw%3D&reserved=0 > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org For additional commands, e-mail: users-h...@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
