On 29/07/2015 20:25, Jerry Malcolm wrote: > Well, it appears that we are slowly getting to the bottom of this. But > with every answer, I get a few more questions.... > > First, I installed the latest TC8 on my laptop, copied my server.conf > and conf/catalina folder to it and started it up just to see what errors > I got. After changing out an obsolete listener, it came up. I found > all of the <resource> parm exceptions in stderr. So that question is > cleared up. Thanks for clarifying where to find that. > > The site is a wedding vendor advertiser site that spans two major > cities. There is no user login. Simply a very huge online catalog. > I'm certain it's deployed only once. Whether I need that many > connections is a valid question. As far as I know, I haven't hit the > limit in normal operation until now. Could possibly reduce the count if > I collect statistics. > > I've been monitoring the production server logs all day watching to be > sure connection pool doesn't dry up again. About an hour ago, there was > a single huge dump in stdout of approx 2000 'logAbandoned' exceptions. > They showed connections from 1am right after my last bounce of the > server thru 1:35pm. The good news is with the stack trace on one of > them I was able to see the bug causing the leak. But why did it decide > to wait over 12 hours accumulating abandoned connections before dumping > them back in the pool? I realize from now knowing the code bug that the > leak is a slow drip that is continually leaking on a regular basis. But > since that last 12-hour accumulated dump, the abandoning has returned to > silence. Since leaks are occurring regularly and would be timing out > regularly, shouldn't I see a similar 'slow drip' of logAbandoned entries > in stdout instead of a big dump every 12 hours? > > It's going to take a day or two to fix the leak, test, and deploy. If > indeed abandoned connections are now correctly being returned to the > pool, then I presume we are back to working the way it did on TC7. > Still not sure why it started working now. But I guess once I get the > leak fixed and if TC8 is now configured to handle abandoned connections, > I'm good. Still would like to know about the mega-dump vs. trickle of > abandoned connections being logged.
That sounds like a bug that has just been fixed in Pool: https://issues.apache.org/jira/browse/POOL-300 The fix for that should be in the next Tomcat 8 release. Mark > > Thanks again for all of the info. > > Jerry > > On 7/29/2015 1:15 PM, Christopher Schultz wrote: > Jerry, > > On 7/29/15 12:06 PM, Jerry Malcolm wrote: >>>> On 7/29/2015 10:11 AM, Christopher Schultz wrote: >>>> >>>> I guess I could move up to the latest 8. But it'll require some >>>> client down time, and they aren't thrilled with that at this >>>> point. > Understood, though trying various <Resource> configurations is > certainly resulting in downtime, no? > >>>>>> Off topic, but relevant.... it sure would be nice if the xml >>>>>> parser for <resource> tags and other config items would flag >>>>>> with errors any parameter misspellings such as using the old >>>>>> obsolete 'removeAbandoned' instead of the new >>>>>> 'removeAbandonedOnBorrow', etc. or even just typo misspells. >>>>> It does: >>>>> >>>>> Jul 02, 2015 10:17:25 AM >>>>> org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get >>>>> ObjectInstance WARNING: Name = diagnosis Property maxActive is >>>>> not used in DBCP2, use maxTotal instead. maxTotal default value >>>>> is 8. You have set value of "1" for "maxActive" property, which >>>>> is being ignored. >>>>> >>>>> ... >>>>> >>>>> WARNING: Name = diagnosis Property removeAbandoned is not used >>>>> in DBCP2, use one or both of removeAbandonedOnBorrow or >>>>> removeAbandonedOnMaintenance instead. Both have default value set >>>>> to false. You have set value of "true" for "removeAbandoned" >>>>> property, which is being ignored. >>>>> >>>>> >>>>> (I have a context.xml that I want to use between both Tomcat 7 >>>>> and Tomcat 8, so I actually have both spellings of those >>>>> attributes in my XML.) >>>> This is really strange. Are the above log entries in stdout? > Yes, so they typically end up in catalina.out. Those logs are from my > dev system's catalina.out from a recent restart. > >>>> I had 'removeAbandoned' in my resource statements until yesterday. >>>> But I am not seeing anything in stdout like the statement above. >>>> And after changing removeAbandoned to the correct spelling, I'm >>>> also not seeing any abandoned log entries either. > Hmm. > >>>> Is there some sort of 'log level' that may be throttling back my >>>> log entries? If my code has leaks, and I'm running out of >>>> connections but not seeing any abandoned entries in the log, that >>>> should be a concern, right? > Possibly. > > Perhaps you are deploying your application more times than you > thought, and are making many more connections than you expected. > >>>>> What does your <Resource> element actually look like? (Remember >>>>> to remove sensitive information such as passwords.) >>>> <Resource name="jdbc/wgbe" testOnBorrow="true" >>>> validationQuery="SELECT 1" > validationQuery might not be necessary anymore. I think DBCP2 does a > lightweight connection test instead issuing an actual query, unless > you override it with a validationQuery. > > Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver > itself will perform the lightweight connection test instead of > actually issuing a query. Much faster than going through the SQL > parser, etc. > >>>> auth="Container" type="javax.sql.DataSource" maxTotal="300" > That's a LOT of connections IMO. What is your total connection limit? > Is it per-user? > >>>> maxIdle="30" maxWaitMillis="10000" removeAbandonedOnBorrow="true" >>>> removeAbandonedOnMaintenance="true" removeAbandonedTimeout="30" >>>> logAbandoned="true" username="xxxxxxx" password="xxxxxxxxx" >>>> driverClassName="org.gjt.mm.mysql.Driver" >>>> url="jdbc:mysql://localhost:3306/wgbe?autoReconnect=true" /> > Do you have the 'manager' web application installed? Can you check to > see what applications are actually deployed? If you have 2 copies > deployed, you might see 600 connections open. > > Also, do you have any stray context.xml files hanging around in > conf/*/* that you weren't expecting? If you are making changes to your > configuration and they don't appear to be having any effect, it's > possible that your new configuration isn't being applied because > another config is masking it. > > -chris >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> >> ----- >> No virus found in this message. >> Checked by AVG - www.avg.com >> Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date: >> 07/29/15 > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
