Thanks for the pointer to jstack. Looking through the output now, but the problem doesn't persist. I've only had it happen twice in 3 or 4 days.
I guess I need to dump a full all-thread stacktrace like jstack when the problem occurs to see what is going on. I'm also looking through the poolmanager source to see if there's any additional logging I could turn on, but I'm not seeing anything cayenne-related executing at the moment. And, interestingly enough, I only have 8 "PoolManagerCleanup" threads, and not the 9 that I expected. Maybe they are create-on-demand and one of my runtimes hasn't needed it yet? On Fri, Jun 12, 2015 at 10:56 AM, Andrus Adamchik <and...@objectstyle.org> wrote: > So there's a leak somewhere. Do you see any hanging threads with jstack, or > maybe some unclosed ResultIterators? > > Andrus > >> On Jun 12, 2015, at 5:51 PM, Mike Kienenberger <mkien...@gmail.com> wrote: >> >> ....And upping my connection pool size from 1 to 3 didn't help as I >> got the error again today, but this time with all 3 in use. >> >> Caused by: java.sql.SQLException: Can't obtain connection. Request >> timed out. Total used connections: 3 >> >> >> >> On Thu, Jun 11, 2015 at 11:41 AM, Mike Kienenberger <mkien...@gmail.com> >> wrote: >>> So what I had when I originally deployed would have been three pools >>> of one connection each? That's fine. And that explains why it >>> worked without problems. >>> >>> So when I increased to 3 connections, I actually have 9 now, three in >>> each of the three pools which I definitely do not want. >>> >>> So apparently I only hit this problem in a single runtime's connection >>> pool, and not in all of them. That should make it easier to track >>> down what caused it. Thanks. >>> >>> >>> >>> On Thu, Jun 11, 2015 at 11:32 AM, Andrus Adamchik >>> <and...@objectstyle.org> wrote: >>>>> My guess is that all three runtimes share the same pool >>>> >>>> >>>> Actually no. If you have 3 runtimes, and let Cayenne instantiate its pool >>>> (as the stack trace implies) instead of using say container JNDI >>>> DataSource, you should have 3 instances of the pool. You can try >>>> confirming the number of PoolManager instances with VisualVM or jmap. >>>> >>>> Andrus >>>> >>>>> On Jun 11, 2015, at 6:18 PM, Mike Kienenberger <mkien...@gmail.com> wrote: >>>>> >>>>> So I finally put my old Cayenne 1.x-upgraded to 3.x app into production. >>>>> >>>>> A couple of hours later, I got this error [1]. Note that my app >>>>> synchronizes requests so only one thread should ever be running >>>>> Cayenne requests at a time. >>>>> >>>>> My app has run for years with only a single connection. My 3.x app >>>>> ran in development for a year before it was deployed to production. >>>>> >>>>> Is this a fluke? I did raise my connection pool to 3 the next day, >>>>> but I don't want to arbitrarily assign a higher number and think I've >>>>> magically solved the problem. >>>>> >>>>> Does having three active server runtimes affect this? Before, my >>>>> query qualifiers were all handled inside the main runtime, but now I >>>>> have three separate runtimes configured. My guess is that all three >>>>> runtimes share the same pool, so this is probably not the cause. And >>>>> since the problem happened a couple hours later and the three runtimes >>>>> were being used well before that without an error, I'd say experience >>>>> bears this out. >>>>> >>>>> >>>>> [1] >>>>> >>>>> Caused by: java.sql.SQLException: Can't obtain connection. Request >>>>> timed out. Total used connections: 1 >>>>> at >>>>> org.apache.cayenne.conn.PoolManager.uncheckPooledConnection(PoolManager.java:452) >>>>> at >>>>> org.apache.cayenne.conn.PoolManager.getConnection(PoolManager.java:382) >>>>> at >>>>> org.apache.cayenne.conn.PoolManager.getConnection(PoolManager.java:371) >>>>> at >>>>> org.apache.cayenne.access.DataNode$TransactionDataSource.getConnection(DataNode.java:351) >>>>> at >>>>> org.apache.cayenne.access.DataNode.performQueries(DataNode.java:259) >>>>> >>>> >> >
