Hi everyone,
Recently we've started to use SSLEngine to create our own TLS server.
However we sometimes got the following exception "SSLHandshakeException:
Insufficient buffer remaining for AEAD cipher fragment (2)." when
dealing with client disconnections at server side during the TLS
handshake. This has been tested with latest JDK21.
We started to investigate and traced it down to the following scenario:
- Setup: Standard SSLSocket as client (all defaults) connects to our TLS
server using Java SSLEngine as internal engine. Both client and server
have support for TLSv1.3
1. Client sends hello
"client version" : "TLSv1.2",
"supported_versions (43)": {
"versions": [TLSv1.3, TLSv1.2]
},
2. Server processes the client hello, and attempts to send server hello:
"server version" : "TLSv1.2",
"supported_versions (43)": {
"selected version": [TLSv1.3]
},
3. From this point, server has switched to TLSv1.3 and is using a
GcmReadCipher (from SSLCipher class)
4. However, in this scenario, due to network congestion or server
application being slow, the server hello response is delayed and did not
reach the client yet
5. The client, still assuming it is operating under TLSv1.2, gets a
timeout and decides to close the socket. This produces TLSv1.2 alerts
(user canceled and close notification) and reaches the server
6. Server tries to decrypt the TLSv1.2 alerts, but since TLSv1.2 alerts
are not encrypted, gets the "SSLHandshakeException: Insufficient buffer
remaining for AEAD cipher fragment (2)." exception.
If the client did receive the server hello response, it would switch to
TLSv1.3 properly and would encrypt TLSv1.3 alerts from then on as expected.
Additionally, if we use the same scenario but force TLSv1.2 on the
server side, then the server does not use the GcmReadCipher at this
point, and it can process the TLSv1.2 alerts properly.
So it seems the JDK SSLEngine did not take the above scenario into
account, or are we missing something?
Best regards,
Domien
