Hello Christopher and Tomcat crew
This TLS protocol violation has been seen to cause issues where a client will
"linger" with the unclosed connection (usually by default for a minute for each
webservice call) until Tomcat timesout the connection and closes the socket
(this "linger" occurs for clients who more strictly follow the TLS protocol
than others). For clients that require making many webservice calls to a Tomcat
configured with the Nio+OpenSSL connector, this can extremely slow down the
client code.
While a TCP dump is the most conclusive way to demonstrate the fact that the
Tomcat server does not return the server's close_notify in response to the
client's close_notify (when using Nio+OpenSSL), I have a Python script that can
demonstrate the lack of a server close_notify which I have put after my email
signature. When the ssl_socket_instance.unwrap() is called, the client sends
its close_notify, and waits for the close_notify from the server, which for
Nio+JSSE or APR+OpenSSL, is received promptly, but for Nio+OpenSSL, it never
arrives and instead an EOF on the socket is received after a timeout is reached
(on the Tomcat side).
Unfortunately, even an extremely verbose curl will not show enough of the ssl
communication. You can see the different behaviors by sending "Q" to an openssl
s_client (built with the ability to use the -debug flag).
openssl s_client -connect <host>:<port> -state -debug <<< "Q"
For the Nio+JSSE or Apr+OpenSSL connector, you see the client close_notify go
out, followed by the client reading the server's close_notify:
---
DONE
write to 0x5b93da131c40 [0x5b93da13cee3] (31 bytes => 31 (0x1F))
0000 - 15 03 03 00 1a 83 0b b7-5a e0 00 ad f3 2c 8f 9c ........Z....,..
0010 - 13 8c 04 8f 57 48 e9 7a-15 a6 ef 9c b0 16 b2 ....WH.z.......
SSL3 alert write:warning:close notify
read from 0x5b93da131c40 [0x5b93da089670] (8192 bytes => 31 (0x1F))
0000 - 15 03 03 00 1a a0 f4 7f-e5 65 84 ed df 3a a4 ec .........e...:..
0010 - 76 42 bd c6 37 28 cf 21-03 ca 9a dc 9f 5c 23 vB..7(.!.....\#
read from 0x5b93da131c40 [0x5b93da089670] (8192 bytes => 0)
For the Nio+OpenSSL connector, you will not see the close_notify returned by
the server.
---
DONE
write to 0x5714dc99ac40 [0x5714dc9a5ee3] (31 bytes => 31 (0x1F))
0000 - 15 03 03 00 1a f9 41 8b-50 31 2f 8b 6c 7a 77 c3 ......A.P1/.lzw.
0010 - 12 ca 2d 15 fc 7d 33 cd-ad ee 5c 3d 23 aa 11 ..-..}3...\=#..
SSL3 alert write:warning:close notify
read from 0x5714dc99ac40 [0x5714dc8f2670] (8192 bytes => 0)
You will not see the same "lingering" behavior though because openssl s_client
will disconnect quickly if the server does not return a close_notify.
The Python code after my email signature demonstrates the "lingering" behavior
seen for some clients which wait for the server's close_notify where the client
"unwraps" the SSL layer from the socket, but keeps the socket open.
Please let me know if you'd like any more diagnostic information
Best,
Isaac Klickstein
##########################
##### PYTHON EXAMPLE #####
# USAGE:
# python3 tomcat_ssl_unwrap_example.py <tomcat host> <connector port>
<username> <password>
import sys
import socket
import ssl
import os
import time
import errno
from urllib.parse import urlparse
import base64
def call_manager_app(url,username,password):
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
ssl_context.verify_mode = ssl.CERT_NONE
parsed_url = urlparse(url)
hostname, port = parsed_url.netloc.split(":") if ":" in parsed_url.netloc
else (parsed_url.netloc, 443)
credentials = f"{username}:{password}"
encoded_credentials = base64.b64encode(credentials.encode()).decode()
socket_instance = socket.socket(socket.AF_INET, socket.SOCK_STREAM);
# socket_instance.setblocking(True)
ssl_socket_instance = ssl_context.wrap_socket(socket_instance,
server_hostname=hostname)
ssl_socket_instance.settimeout(1)
ssl_socket_instance.connect((hostname,int(port)))
request = f"GET {parsed_url.path} HTTP/1.1\r\nHost: {hostname}\r\n"
request += "Accept: */*\r\n"
request += f"Authorization: Basic {encoded_credentials}\r\n\r\n"
ssl_socket_instance.send(request.encode())
response_data = b""
while True:
try:
data = ssl_socket_instance.recv(16384)
except socket.timeout:
break
response_data += data
response = response_data.decode()
headers, body = response.split("\r\n\r\n", 1)
print("Headers:")
print(headers)
print("Body:")
print(body)
# Unwrap the socket, and waiting for the server's close notify
ssl_socket_instance.settimeout(120)
socket_instance = ssl_socket_instance.unwrap()
socket_instance.close()
host=sys.argv[1]
port=sys.argv[2]
username=sys.argv[3]
password=sys.argv[4]
# Supply the URL of the Tomcat manager page's /text/list endpoint
#url = "https://localhost:9015/manager/text/list";
#
#
url = "https://"; + host + ":" + port + "/manager/text/list"
print(url)
call_manager_app(url,username,password)
######## END OF PYTHON #############
####################################
Sent with Proton Mail secure email.
On Thursday, August 29th, 2024 at 9:05 AM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> Isaac,
>
> On 8/26/24 13:24, Isaac Klickstein wrote:
>
> > > What is the "Tomcat Native Client"?
> >
> > I am using the Tomcat Native software (maybe "client" was wrong) available
> > here to build the OpenSSLImplementation:
> > https://tomcat.apache.org/download-native.cgi#2.0.8
> >
> > I then link to this library using LD_LIBRARY_PATH in the Tomcat's setenv.sh
> > script.
>
>
> Aah, okay. You are just using libtcnative on the server side. curl is
> the client.
>
> > > How do you trigger this behavior? Just any request like "curl
> >
> > I have been using the manager app, something like
> >
> > curl --cacert </path/to/ca file> --url 'https://<tomcat host>:<tomcat https
> > connector port>/manager/text/list' --user robot:robotpw
> >
> > but any request to the ROOT/manager/other hosted would do.
> >
> > I have been breaking down the behavior based on protocol=NIO/NIO2/APR and
> > the sslImplementationName JSSE/OpenSSL
> >
> > NIO/NIO2+JSSE = good
> > NIO/NIO2+OpenSSL = bad
> > APR+OpenSSL = good
>
>
> That's interesting. When using APR+OpenSSL, the Tomcat code is entirely
> responsible for the connection management (e.g. socket, buffers, etc.)
> and the crypto (using OpenSSL, of course).
>
> When using NIO+OpenSSL, Java is responsible for the connection
> management AND the orchestration of the use of the cryptographic module.
> The use of OpenSSL versus some other cryptographic module (e.g. built-in
> JSSE) should not affect whether a close_notify is sent. :/
>
> > I have TCP dumps for each of these configurations saved and could upload
> > them as well as the configuration of the connectors.
>
>
> Is a TCP dump required to observe this behavior, or will e.g. curl -vvv
> show it as well?
>
> Is this causing any actual issues in your environment, or are you more
> reporting a spec violation that needs to be cleaned-up. It seems to be
> that if the client asks the server to close the connection, if the
> connection is closed then it's closed whether or not this particular
> message is transmitted before termination of the connection.
>
> -chris
>
> > On Monday, August 26th, 2024 at 11:40 AM, Christopher Schultz
> > ch...@christopherschultz.net wrote:
> >
> > > Isaac,
> > >
> > > On 8/25/24 13:27, Isaac Klickstein wrote:
> > >
> > > > Hello Tomcat Users
> > > >
> > > > Tomcat Version: 10.1.28
> > > > OpenSSL version: 3.0.14
> > > > Tomcat Native Client: 2.0.8
> > >
> > > What is the "Tomcat Native Client"?
> > >
> > > > I have configured an HTTPS connector with the
> > > > org.apache.coyote.http11.Http11NioProtocol protocol and the
> > > > org.apache.tomcat.util.net.openssl.OpenSSLImplementation
> > > > sslImplementationName using TLSv1.2
> > > >
> > > > When I tcpdump any request to this connector, Tomcat is not returning a
> > > > "close_notify" in response to a client's close_notify, and I cannot
> > > > figure out how to force Tomcat to return a close_notify. This seems to
> > > > be a violation of the TLS protocol which demands both sides issue a
> > > > close_notify.
> > >
> > > Careful: both the client and the server are always allowed to be
> > > powered-off before they respond to any network stimulus. This is what
> > > timeouts are for. TLS cannot place any more requirements on the network
> > > peers than TCP has already done.
> > >
> > > > Recreating this situation, as far as I can tell, only requires
> > > > combining the Http11NioProtocol with the OpenSSLImplementation (Tomcat9
> > > > or Tomcat10, TLSv1.2 or TLSv1.3, OpenSSL 3.0, 3.1, and 3.2, all exhibit
> > > > this behavior).
> > >
> > > How do you trigger this behavior? Just any request like "curl
> > > https://example.com/"; ?
> > >
> > > > Other notes, switching the sslImplementationName to
> > > > org.apache.tomcat.util.net.jsse.JSSEImplementation does return a
> > > > close_notify by the server in response to the client's close_notify.
> > > >
> > > > Also, switching back to Tomcat9, and using the
> > > > org.apache.coyote.http11.Http11AprProtocol, Tomcat also returns a
> > > > close_notify in response to a client's close_notify.
> > > >
> > > > I have run out of ideas, googling this behavior has turned up nothing
> > > > related to Tomcat (although there does appear to be a similar behavior
> > > > noticed in Netty also using the OpenSSLEngine
> > > > https://github.com/netty/netty/issues/6167)
> > > >
> > > > Any help would be greatly appreciated, I am happy to send along any
> > > > other information that would be informational for diagnostics
> > >
> > > So...
> > >
> > > Tomcat 10.1 + NIO/JSSE+OpenSSLImplementation+tcnative = bad
> > > Tomcat 9.0 + APR+tcnative = good
> > > Tomcat 9.0 + NIO/JSSE+OpenSSLImplementation+tcnative = ?
> > >
> > > -chris
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
