On 20/03/18 07:52, Richard Tearle wrote:
> Hello
>
> We have 4 applications built on the same architecture with a web UI
> and camel based ESB running in separate Tomcat's, using REST/XML to
> communicate between the two. This is all deployed within separate
> Docker containers but on the same VM (at least for test), either on
> Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've
> been upgrading to Tomcat 8.5.x since November last year, but this has
> been hampered by what looks to be random connection closed errors when
> our UI communicates to the ESB. We have a series of Selenium based
> autotests which will fail in different places, but with the same
> error:
<snip/>
> We've seen this on all of the versions of Tomcat 8.5 we've tried from
> 8.5.23 onwards - and I've just tried Tomcat 9.0.6 with the same error.
> We're using Oracle Java version 1.8u162.
>
> Development have looked at this off and on over the last couple of
> months, but without success, so I was asked to reproduce this with a
> simple test case (I'm devops usually, but with a development
> background). The client stack uses a
> PoolingNHttpClientConnectionManager, CloseableHttpAsyncClient and
> AsyncRestTemplate to communicate to the ESB. My simple test case
> replaces the web UI with a simple jar based application, but it still
> replicates the issue. The ESB is a simple Camel route that adds two
> supplied numbers, waits for a time (supplied by the UI), and returns
> the result. The UI repeatedly calls the ESB (on a single thread, with
> no overlapping calls), with two random numbers to be summed, and a set
> amount of time to wait before the ESB returns. Between each call to
> the ESB there's a set amount of time to wait before the next call. It
> seems that when the ESB wait is >= 5secs the time take to fail is
> greatly decreased.
>
> The connection between the UI and the ESB is TLSv1.2 with the ESB
> configured with certificateVerification="required". The connector
> configuration with the ESB's server.xml is:
>
> <Connector port="${servers.port}"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> maxThreads="150" SSLEnabled="true" scheme="https"
> secure="true" server="Apache" maxPostSize="100000">
> <SSLHostConfig certificateVerification="required" protocols="TLSv1.2"
> caCertificateFile="/usr/local/tomcat/ssl/ca-cert.pem" >
> <Certificate
> certificateFile="/usr/local/tomcat/ssl/server-cert.pem"
>
> certificateKeyFile="/usr/local/tomcat/ssl/server-key.pem"
> certificateKeyPassword="${keystore.pass}">
> </Certificate>
> </SSLHostConfig>
> </Connector>
>
> We are using PEM encoded certificates, and using Tomcat Native Library
> version 1.2.16. If I take out certificateVerification="required", we
> don't get the error (with my simple test case running for an hour)
> I've tried using PKCS12 and JKS certificates, but I can't get Tomcat
> 8.5.x to work with our self signed trust store (we get the
> "trustAnchors parameter must be non-empty" errors), although Tomcat
> 8.0.x seems happy enough with them. So I can't tell if it's a
> difference between using OpenSSL and JSSE.
There are rather too many factors at play here. It would be good to try
and eliminate some of them.
What are the known working 8.0.x versions?
I looks like you are using JSSE with 8.0.x. It should be possible to use
the exact same configuration with 8.5.x. No need to use the native
library and no need to switch to the new configuration style.
Lets try and get 8.5.x working with JSSE. That should help narrow down
the root cause. What happens when you transfer the working 8.0.x config
to 8.5.x?
Also, anything you can do to reduce the complexity of the test
application (ideally reducing it to simple Servlets/JSPs) would make it
easier for others to reproduce the issue.
> Enabling SSL debug traces on the client shows that in the error case,
> after the initial successful handshaking between the client and the
> server, the ESB shuts down the connection just after the client has
> sent the data:
>
> *** CertificateVerify
> Signature Algorithm SHA512withRSA
> I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 520
> I/O dispatcher 1, WRITE: TLSv1.2 Change Cipher Spec, length = 1
> *** Finished
> verify_data: { 197, 39, 73, 181, 14, 87, 139, 81, 247, 181, 32, 17 }
> ***
> I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 40
> I/O dispatcher 1, READ: TLSv1.2 Change Cipher Spec, length = 1
> I/O dispatcher 1, READ: TLSv1.2 Handshake, length = 40
> *** Finished
> verify_data: { 84, 164, 177, 160, 235, 23, 31, 20, 252, 149, 214, 245 }
> ***
> %% Cached client session: [Session-101,
> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
> I/O dispatcher 1, WRITE: TLSv1.2 Application Data, length = 391
> I/O dispatcher 1, READ: TLSv1.2 Alert, length = 26
> I/O dispatcher 1, RECV TLSv1.2 ALERT: warning, close_notify
> I/O dispatcher 1, closeInboundInternal()
> I/O dispatcher 1, closeOutboundInternal()
> I/O dispatcher 1, SEND TLSv1.2 ALERT: warning, description = close_notify
> I/O dispatcher 1, WRITE: TLSv1.2 Alert, length = 26
>
> The failed call doesn't make it into the ESB application logs, and I
> can't see any errors in any of the ESB logs.
Hmm. That looks like a controlled shutdown. Random thought, does setting
disableSessionTickets="true" help at all when using OpenSSL?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
