We experienced a very similar situation with Oracle connections not releasing.
We did not hit an error state because our DBA team alerted us and we began to
immediately fail-over to freshly restarted Tomcat instances.
We recently upgraded Tomcat from v9.0.38 to v9.0.65. We also are using Spring
in our application.
When we saw this thread, for a rough test, we started up a new Tomcat v9.0.38
instance in the same production environment as the problematic v9.0.65
instances and started draining customers to the downgraded instance. The
v9.0.38 instance is growing the number of connections within the bounds defined
for our connection pool and is correctly releasing the connections when demand
drops.
We have since spun up several new v9.0.38 Tomcat instances and taken all of the
9.0.65 instances offline.
Sorry, since our DBA team was pro-active enough to alert us before we started
generating stack traces, I don't have any to share. Likewise we do not have GC
or JVM dumps to share. We were in production break/fix mode and in a rush to
keep our production environment responsive to our customers.
R. Mark Harris
Information Technology Systems
Oregon Department of Corrections
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Saturday, March 4, 2023 12:45 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 9.0.71 Anomalies
On 03/03/2023 20:19, jonmcalexan...@wellsfargo.com.INVALID wrote:
> Hi Mark,
>
> On the slowness, this is when they are retrieving random .js files from the
> exploded war file after deployment.
To clarify, these are .js files loaded directly from the file system?
They are not packaged in a JAR file?
> It's taking an a long
> amount of time. Some of these are quite large, like 2MB or more. When the
> issue shows, doing a curl we get to here and then it pauses for some time
> before it feeds back the data.
>
> * Trying **.**.**.**:8443...
> * TCP_NODELAY set
> * Connected to server port 8443 (#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> * SSL connection using TLSv1.2 / <cipher>
> * ALPN, server accepted to use http/1.1
> * Server certificate:
> * subject:
> * start date:
> * expire date:
> * issuer:
> * SSL certificate verify result: unable to get local issuer certificate
> (20), continuing anyway.
>> GET <URI> HTTP/1.1
>> Host:
>> User-Agent: curl/7.65.3
>> Accept: */*
>>
>
> And it just hangs out here before finally getting the requested file.
How repeatable is this?
How long does it hang before delivering content? Is it always the same or does
it vary.
Which connector are you using?
What Tomcat version did you upgrade from?
How does the problem before the upgrade compare to the problem after the
upgrade?
What component is serving the content? Is it Tomcat's default servlet or is it
something else?
When it happens, take 3 thread dumps a few seconds apart. The aim is to figure
out why it is hanging.
> In looking at the catalina.out log file, I am not seeing any
> errors/stack-traces.
>
> Any ideas as to what may be causing this?
Not at the moment. The information requested above should at least narrow down
which parts we need to think about.
Mark
>
> Thank you,
>
> Dream * Excel * Explore * Inspire
> Jon McAlexander
> Senior Infrastructure Engineer
> Asst. Vice President
> He/His
>
> Middleware Product Engineering
> Enterprise CIO | EAS | Middleware | Infrastructure Solutions
>
> 8080 Cobblestone Rd | Urbandale, IA 50322
> MAC: F4469-010
> Tel 515-988-2508 | Cell 515-988-2508
>
> jonmcalexan...@wellsfargo.com
> This message may contain confidential and/or privileged information. If you
> are not the addressee or authorized to receive this for the addressee, you
> must not use, copy, disclose, or take any action based on this message or any
> information herein. If you have received this message in error, please advise
> the sender immediately by reply e-mail and delete this message. Thank you for
> your cooperation.
>
>> -----Original Message-----
>> From: Mark Thomas <ma...@apache.org>
>> Sent: Friday, March 3, 2023 1:32 AM
>> To: users@tomcat.apache.org
>> Subject: Re: Tomcat 9.0.71 Anomalies
>>
>> On 02/03/2023 21:54, jonmcalexan...@wellsfargo.com.INVALID wrote:
>>> Hello gentle beings,
>>>
>>> I have a couple of application teams having issues since getting
>>> upgraded to
>> Tomcat 9.0.71.
>>
>> Upgrading from which Tomcat version?
>>
>>> The main one has to do with an application that has run fine in the
>>> past is
>> now exceeding max cursors with their Oracle Database datasource. They
>> are using spring framework to control the Database operations. Here
>> is what their resource looks like:
>>>
>>> <Resource name="jdbc/f***" auth="Container"
>> type="javax.sql.DataSource"
>>> maxTotal="100" maxIdle="30" maxWaitMillis="15000"
>>> username="${datasource.fm.username}"
>> password="${datasource.fm.password}"
>> driverClassName="oracle.jdbc.OracleDriver"
>>>
>> url="jdbc:oracle:thin:@(Description=(CONNECT_TIMEOUT=10)(RETRY_COUN
>> T=3)(ADDRESS_LIST=(LOAD_BALANCE=on)(FAILOVER=ON)(ADDRESS=(PROT
>> OCOL=tcp)(HOST=***************)(PORT=3203))(ADDRESS=(PROTOCOL=
>> tcp)(HOST=*****************)(PORT=3203)))(CONNECT_DATA=(SERVICE
>> _NAME=ceofm_u)))"/>
>>>
>>> Here is an error from the log file:
>>>
>>> [3/2/2023 1:50 PM] Burgos, Maria D.:
>>> here is the error in catalina.out
>>> 02-Mar-2023 13:05:30.944 SEVERE [https-jsse-nio-28443-exec-8]
>> org.apache.catalina.core.StandardWrapperValve.invoke
>> Servlet.service() for servlet [f***servlet] in context with path
>> [/f***] threw exception [Request processing failed; nested exception
>> is com.wellsfargo.fms.common.exception.FMSException] with root cause
>>> com.wellsfargo.f**.common.exception.F**Exception
>>> at
>> org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTem
>> pl
>> ate.java:1542)
>>> at
>> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:
>> 66
>> 7)
>>
>> Any chance we can see a root cause for that? Something from the JDBC
>> driver?
>>
>>> The other team is having performance issues when using static .jar
>> resources in their war file. One part of the app, on the browser side
>> measurements, is taking 2.2 minutes to load. This started getting
>> worse after moving to 9.0.71. Are there any known issues that could be
>> causing this?
>> Currently waiting for 9.0.73 to get release. :)
>>
>> Do you mean static resources stored in a JAR's META-INF/resources
>> directory?
>>
>> Is the WAR deployed in packed or unpacked form?
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
