Hi there,
Versions: Tomcat 8.5.3, JDK 1.8 + JCE, Bouncy Castle 1.48, Ubuntu 14.04 &
16.04,Windows 2012 R2
I’m running into an issue where we are getting NullPointerExceptions from the
Coyote connector in a Tomcat web application.
This is an existing, stable web application that was recently upgraded to
Tomcat 8.5.3. When we repeatedly send requests over HTTPS eventually (after 30
or so rapid requests) the web app becomes largely unresponsive and the catalina
log fills up with errors like this:
-------- BEGIN LOG --------
18-Jul-2016 10:47:30.066 WARNING [Tomcat-7]
org.apache.tomcat.util.net.AbstractEndpoint.countDownConnection Incorrect
connection count, multiple socket.close called on the same socket.
18-Jul-2016 10:47:36.707 INFO [Tomcat-6]
org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request
header
Note: further occurrences of HTTP header parsing errors will be logged at
DEBUG level.
java.lang.IllegalStateException: Unexpected state: headers already parsed.
Buffer not recycled?
at
org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:587)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1010)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
18-Jul-2016 10:47:36.815 SEVERE [Tomcat-12]
org.apache.coyote.http11.Http11Processor.service Error processing request
java.lang.NullPointerException
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:389)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1110)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
18-Jul-2016 10:47:36.816 SEVERE [Tomcat-12]
org.apache.coyote.http11.Http11Processor.endRequest Error finishing response
java.lang.NullPointerException
at
org.apache.coyote.http11.Http11OutputBuffer.flushBuffer(Http11OutputBuffer.java:533)
at
org.apache.coyote.http11.Http11OutputBuffer.endRequest(Http11OutputBuffer.java:318)
at
org.apache.coyote.http11.Http11Processor.endRequest(Http11Processor.java:1794)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1149)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
18-Jul-2016 10:56:54.739 INFO [https-jsse-nio-8443-ClientPoller-1]
org.apache.catalina.connector.CoyoteAdapter.checkRecycled Encountered a
non-recycled request and recycled it forcedly.
org.apache.catalina.connector.CoyoteAdapter$RecycleRequiredException
at
org.apache.catalina.connector.CoyoteAdapter.checkRecycled(CoyoteAdapter.java:494)
at
org.apache.coyote.http11.Http11Processor.recycle(Http11Processor.java:1840)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.release(AbstractProtocol.java:955)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.release(AbstractProtocol.java:976)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:730)
at org.apache.tomcat.util.net.NioEndpoint.close(NioEndpoint.java:505)
at
org.apache.tomcat.util.net.NioEndpoint.access$600(NioEndpoint.java:69)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.processSendfile(NioEndpoint.java:964)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:844)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:826)
at java.lang.Thread.run(Thread.java:745)
18-Jul-2016 10:59:13.195 SEVERE [Tomcat-23]
org.apache.coyote.http11.Http11Processor.endRequest Error finishing response
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:244)
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:343)
at sun.nio.ch.IOUtil.write(IOUtil.java:60)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at
org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:143)
at
org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:632)
at
org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at
org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1241)
at
org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:428)
at
org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:418)
at
org.apache.coyote.http11.Http11OutputBuffer.flushBuffer(Http11OutputBuffer.java:533)
at
org.apache.coyote.http11.Http11OutputBuffer.endRequest(Http11OutputBuffer.java:318)
at
org.apache.coyote.http11.Http11Processor.endRequest(Http11Processor.java:1794)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1149)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
-------- END LOG --------
We’ve looked for the usual suspects like referencing an old ServletRequest or
ServletResponse and things like that, but I’m skeptical it’d be something like
that simply because the web application works without issue over HTTP and
continues to work without issue at the same time that we’re seeing this over
HTTPS. Requests over port 8080 go without failure, but over 8443 we see these
errors in the server log and on the client side things like
net::ERR_CONTENT_LENGTH_MISMATCH and net::ERR_SSL_PROTOCOL_ERROR (from Chrome).
Particularly due to the “Incorrect connection count” warning, it seems like
Tomcat may be recycling a request or response object twice or attempting to
operate on a recycled object.
I realize this isn’t exact reproduction steps, but I’m really starting to run
out of places to look for a root cause, much less a solution to this. Does
anyone have any idea where I should look? I’ve included the server.xml below.
-------- BEGIN server.xml --------
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--><!-- Note: A "Server" is not itself a "Container", so you may not
define subcomponents such as "Valves" at this level.
Documentation at /docs/config/server.html
--><Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener" />
<!-- Security listener. Documentation at /docs/config/listeners.html
<Listener className="org.apache.catalina.security.SecurityListener" />
-->
<!--APR library loader. Documentation at /docs/apr.html -->
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<!-- Prevent memory leaks due to use of particular java/javax APIs-->
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<!-- Global JNDI resources
Documentation at /docs/jndi-resources-howto.html
-->
<GlobalNamingResources>
<!-- Editable user database that can also be used by
UserDatabaseRealm to authenticate users
-->
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase" description="User database that can be
updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<!-- A "Service" is a collection of one or more "Connectors" that share
a single "Container" Note: A "Service" is not itself a "Container",
so you may not define subcomponents such as "Valves" at this level.
Documentation at /docs/config/service.html
-->
<Service name="Catalina">
<!--The connectors can use a shared executor, you can define one or more
named thread pools-->
<Executor name="tomcatThreadPool" namePrefix="Tomcat-" />
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
-->
<Connector URIEncoding="UTF-8" port="8080" executor="tomcatThreadPool"
protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />
<!-- A "Connector" using the shared thread pool-->
<!--
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
-->
<!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
This connector uses the NIO implementation with the JSSE engine. When
using the JSSE engine, the JSSE configuration attributes must be used.
-->
<!--
<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
maxThreads="150" SSLEnabled="true">
<SSLHostConfig>
<Certificate certificateKeystoreFile="conf/localhost-rsa.jks"
type="RSA" />
</SSLHostConfig>
</Connector>
-->
<!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2
This connector uses the APR/native implementation. When using the
APR/native implementation or the OpenSSL engine with NIO or NIO2 then
the OpenSSL configuration attributes must be used.
-->
<!--
<Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
maxThreads="150" SSLEnabled="true" >
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
<SSLHostConfig>
<Certificate certificateKeyFile="conf/localhost-rsa-key.pem"
certificateFile="conf/localhost-rsa-cert.pem"
certificateChainFile="conf/localhost-rsa-chain.pem"
type="RSA" />
</SSLHostConfig>
</Connector>
-->
<Connector URIEncoding="UTF-8" server="Apache" port="8443"
executor="tomcatThreadPool" SSLEnabled="true" maxPostSize="-1" scheme="https"
protocol="HTTP/1.1" secure="true" clientAuth="false" sslProtocol="TLS"
sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1" keystoreFile="/usr/secret"
keystorePass="1234"
ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA"></Connector>
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3"
redirectPort="8443" />
<!-- An Engine represents the entry point (within Catalina) that processes
every request. The Engine implementation for Tomcat stand alone
analyzes the HTTP headers included with the request, and passes them
on to the appropriate Host (virtual host).
Documentation at /docs/config/engine.html -->
<!-- You should set jvmRoute to support load-balancing via AJP ie :
<Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
-->
<Engine name="Catalina" defaultHost="localhost">
<!--For clustering, please take a look at documentation at:
/docs/cluster-howto.html (simple how to)
/docs/config/cluster.html (reference documentation) -->
<!--
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
-->
<!-- Use the LockOutRealm to prevent attempts to guess user passwords
via a brute-force attack -->
<Realm className="org.apache.catalina.realm.LockOutRealm">
<!-- This Realm uses the UserDatabase configured in the global JNDI
resources under the key "UserDatabase". Any edits
that are performed against this UserDatabase are immediately
available for use by the Realm. -->
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase" />
</Realm>
<Host name="localhost" appBase="webapps" unpackWARs="true"
autoDeploy="true">
<!-- SingleSignOn valve, share authentication between web applications
Documentation at: /docs/config/valve.html -->
<!--
<Valve className="org.apache.catalina.authenticator.SingleSignOn" />
-->
<!-- Access log processes all example.
Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common" -->
<!--
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" resolveHosts="false"/>
-->
</Host>
</Engine>
</Service>
</Server>
-------- END server.xml --------
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
