Hello
We have 4 applications built on the same architecture with a web UI
and camel based ESB running in separate Tomcat's, using REST/XML to
communicate between the two. This is all deployed within separate
Docker containers but on the same VM (at least for test), either on
Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've
been upgrading to Tomcat 8.5.x since November last year, but this has
been hampered by what looks to be random connection closed errors when
our UI communicates to the ESB. We have a series of Selenium based
autotests which will fail in different places, but with the same
error:
09:30:46.057 [main] ERROR test.ui.Main - Catching
java.util.concurrent.ExecutionException:
org.apache.http.ConnectionClosedException: Connection closed
at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:70)
~[test-ui.jar:?]
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:80)
~[test-ui.jar:?]
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
~[test-ui.jar:?]
at
org.springframework.util.concurrent.FutureAdapter.get(FutureAdapter.java:81)
~[test-ui.jar:?]
at
org.springframework.util.concurrent.FutureAdapter.get(FutureAdapter.java:81)
~[test-ui.jar:?]
at test.ui.Main.post(Main.java:182) ~[test-ui.jar:?]
at test.ui.Main.callServer(Main.java:148) [test-ui.jar:?]
at test.ui.Main.main(Main.java:109) [test-ui.jar:?]
Caused by: org.apache.http.ConnectionClosedException: Connection closed
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:345)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
~[test-ui.jar:?]
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
~[test-ui.jar:?]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_162]
We've seen this on all of the versions of Tomcat 8.5 we've tried from
8.5.23 onwards - and I've just tried Tomcat 9.0.6 with the same error.
We're using Oracle Java version 1.8u162.
Development have looked at this off and on over the last couple of
months, but without success, so I was asked to reproduce this with a
simple test case (I'm devops usually, but with a development
background). The client stack uses a
PoolingNHttpClientConnectionManager, CloseableHttpAsyncClient and
AsyncRestTemplate to communicate to the ESB. My simple test case
replaces the web UI with a simple jar based application, but it still
replicates the issue. The ESB is a simple Camel route that adds two
supplied numbers, waits for a time (supplied by the UI), and returns
the result. The UI repeatedly calls the ESB (on a single thread, with
no overlapping calls), with two random numbers to be summed, and a set
amount of time to wait before the ESB returns. Between each call to
the ESB there's a set amount of time to wait before the next call. It
seems that when the ESB wait is >= 5secs the time take to fail is
greatly decreased.
The connection between the UI and the ESB is TLSv1.2 with the ESB
configured with certificateVerification="required". The connector
configuration with the ESB's server.xml is:
<Connector port="${servers.port}"
protocol="org.apache.coyote.http11.Http11NioProtocol"
maxThreads="150" SSLEnabled="true" scheme="https"
secure="true" server="Apache" maxPostSize="100000">
<SSLHostConfig certificateVerification="required" protocols="TLSv1.2"
caCertificateFile="/usr/local/tomcat/ssl/ca-cert.pem" >
<Certificate
certificateFile="/usr/local/tomcat/ssl/server-cert.pem"
certificateKeyFile="/usr/local/tomcat/ssl/server-key.pem"
certificateKeyPassword="${keystore.pass}">
</Certificate>
</SSLHostConfig>
</Connector>
We are using PEM encoded certificates, and using Tomcat Native Library
version 1.2.16. If I take out certificateVerification="required", we
don't get the error (with my simple test case running for an hour)
I've tried using PKCS12 and JKS certificates, but I can't get Tomcat
8.5.x to work with our self signed trust store (we get the
"trustAnchors parameter must be non-empty" errors), although Tomcat
8.0.x seems happy enough with them. So I can't tell if it's a
difference between using OpenSSL and JSSE.
Enabling SSL debug traces on the client shows that in the error case,
after the initial successful handshaking between the client and the
server, the ESB shuts down the connection just after the client has
sent the data:
*** CertificateVerify
Signature Algorithm SHA512withRSA
I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 520
I/O dispatcher 1, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data: { 197, 39, 73, 181, 14, 87, 139, 81, 247, 181, 32, 17 }
***
I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 40
I/O dispatcher 1, READ: TLSv1.2 Change Cipher Spec, length = 1
I/O dispatcher 1, READ: TLSv1.2 Handshake, length = 40
*** Finished
verify_data: { 84, 164, 177, 160, 235, 23, 31, 20, 252, 149, 214, 245 }
***
%% Cached client session: [Session-101,
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
I/O dispatcher 1, WRITE: TLSv1.2 Application Data, length = 391
I/O dispatcher 1, READ: TLSv1.2 Alert, length = 26
I/O dispatcher 1, RECV TLSv1.2 ALERT: warning, close_notify
I/O dispatcher 1, closeInboundInternal()
I/O dispatcher 1, closeOutboundInternal()
I/O dispatcher 1, SEND TLSv1.2 ALERT: warning, description = close_notify
I/O dispatcher 1, WRITE: TLSv1.2 Alert, length = 26
The failed call doesn't make it into the ESB application logs, and I
can't see any errors in any of the ESB logs.
I appreciate that I haven't included all the information required to
help resolve this issue; so please tell me what extra information is
required.
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended recipient of this email you must: (i) not
disclose, copy or distribute its contents to any other person nor use its
contents in any way or you may be acting unlawfully; (ii) contact
Northgate Public Services immediately on +44(0)1442 768445 quoting the name
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that
no viruses are contained in this email, but does not accept any
responsibility once this email has been transmitted. You should scan
attachments (if any) for viruses.
Northgate Public Services (UK) Limited, registered in England and Wales
under number 00968498 with a registered address of Peoplebuilding 2,
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2
4NW. Rave Technologies (India) Pvt Limited, registered in India under
number 117068 with a registered address of 2nd Floor, Ballard House, Adi
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
