On 7/29/2015 3:39 PM, Mark Thomas wrote:
On 29/07/2015 20:25, Jerry Malcolm wrote:
Well, it appears that we are slowly getting to the bottom of this. But
with every answer, I get a few more questions....

First, I installed the latest TC8 on my laptop, copied my server.conf
and conf/catalina folder to it and started it up just to see what errors
I got.  After changing out an obsolete listener, it came up.  I found
all of the <resource> parm exceptions in stderr.  So that question is
cleared up.  Thanks for clarifying where to find that.

The site is a wedding vendor advertiser site that spans two major
cities.  There is no user login.  Simply a very huge online catalog.
I'm certain it's deployed only once.  Whether I need that many
connections is a valid question.  As far as I know, I haven't hit the
limit in normal operation until now.  Could possibly reduce the count if
I collect statistics.

I've been monitoring the production server logs all day watching to be
sure connection pool doesn't dry up again.  About an hour ago, there was
a single huge dump in stdout of approx 2000 'logAbandoned' exceptions.
They showed connections from 1am right after my last bounce of the
server thru 1:35pm.   The good news is with the stack trace on one of
them I was able to see the bug causing the leak. But why did it decide
to wait over 12 hours accumulating abandoned connections before dumping
them back in the pool?  I realize from now knowing the code bug that the
leak is a slow drip that is continually leaking on a regular basis.  But
since that last 12-hour accumulated dump, the abandoning has returned to
silence.   Since leaks are occurring regularly and would be timing out
regularly, shouldn't I see a similar 'slow drip' of logAbandoned entries
in stdout instead of a big dump every 12 hours?

It's going to take a day or two to fix the leak, test, and deploy. If
indeed abandoned connections are now correctly being returned to the
pool, then I presume we are back to working the way it did on TC7.
Still not sure why it started working now.  But I guess once I get the
leak fixed and if TC8 is now configured to handle abandoned connections,
I'm good.  Still would like to know about the mega-dump vs. trickle of
abandoned connections being logged.
That sounds like a bug that has just been fixed in Pool:
https://issues.apache.org/jira/browse/POOL-300

The fix for that should be in the next Tomcat 8 release.

Mark
Thanks, Mark..... Yep that would definitely explain why with a smaller count of abandoned connections, I wasn't seeing any output.

Jerry

Thanks again for all of the info.

Jerry

On 7/29/2015 1:15 PM, Christopher Schultz wrote:
Jerry,

On 7/29/15 12:06 PM, Jerry Malcolm wrote:
On 7/29/2015 10:11 AM, Christopher Schultz wrote:

I guess I could move up to the latest 8.  But it'll require some
client down time, and they aren't thrilled with that at this
point.
Understood, though trying various <Resource> configurations is
certainly resulting in downtime, no?

Off topic, but relevant.... it sure would be nice if the xml
parser for <resource> tags and other config items would flag
with errors any parameter misspellings such as using the old
obsolete 'removeAbandoned' instead of the new
'removeAbandonedOnBorrow', etc. or even just typo misspells.
It does:

Jul 02, 2015 10:17:25 AM
org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
ObjectInstance WARNING: Name = diagnosis Property maxActive is
not used in DBCP2, use maxTotal instead. maxTotal default value
is 8. You have set value of "1" for "maxActive" property, which
is being ignored.

...

WARNING: Name = diagnosis Property removeAbandoned is not used
in DBCP2, use one or both of removeAbandonedOnBorrow or
removeAbandonedOnMaintenance instead. Both have default value set
to false. You have set value of "true" for "removeAbandoned"
property, which is being ignored.


(I have a context.xml that I want to use between both Tomcat 7
and Tomcat 8, so I actually have both spellings of those
attributes in my XML.)
This is really strange.  Are the above log entries in stdout?
Yes, so they typically end up in catalina.out. Those logs are from my
dev system's catalina.out from a recent restart.

I had 'removeAbandoned' in my resource statements until yesterday.
But I am not seeing anything in stdout like the statement above.
And after changing removeAbandoned to the correct spelling, I'm
also not seeing any abandoned log entries either.
Hmm.

Is there some sort of 'log level' that may be throttling back my
log entries?  If my code has leaks, and I'm running out of
connections but not seeing any abandoned entries in the log, that
should be a concern, right?
Possibly.

Perhaps you are deploying your application more times than you
thought, and are making many more connections than you expected.

What does your <Resource> element actually look like? (Remember
to remove sensitive information such as passwords.)
<Resource name="jdbc/wgbe" testOnBorrow="true"
validationQuery="SELECT 1"
validationQuery might not be necessary anymore. I think DBCP2 does a
lightweight connection test instead issuing an actual query, unless
you override it with a validationQuery.

Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
itself will perform the lightweight connection test instead of
actually issuing a query. Much faster than going through the SQL
parser, etc.

auth="Container" type="javax.sql.DataSource" maxTotal="300"
That's a LOT of connections IMO. What is your total connection limit?
Is it per-user?

maxIdle="30" maxWaitMillis="10000" removeAbandonedOnBorrow="true"
removeAbandonedOnMaintenance="true" removeAbandonedTimeout="30"
logAbandoned="true" username="xxxxxxx" password="xxxxxxxxx"
driverClassName="org.gjt.mm.mysql.Driver"
url="jdbc:mysql://localhost:3306/wgbe?autoReconnect=true" />
Do you have the 'manager' web application installed? Can you check to
see what applications are actually deployed? If you have 2 copies
deployed, you might see 600 connections open.

Also, do you have any stray context.xml files hanging around in
conf/*/* that you weren't expecting? If you are making changes to your
configuration and they don't appear to be having any effect, it's
possible that your new configuration isn't being applied because
another config is masking it.

-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date:
07/29/15

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date: 07/29/15


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Reply via email to