Thanks for sharing the outcome. I'm glad you got to the bottom of it properly.
On Sat, Aug 23, 2025, 16:52 Daniel Schwartz <d...@danielgschwartz.com> wrote: > Robert, and all who have responded to my postings, > > You were correct! There was a memory leak. > > Robert suggested that I go back and take another look at the original > version of my program. I did this and saw that I had been overlooking some > exceptions that were being thrown. These were occasional one-liners, null > pointer exceptions, somewhat buried in a lengthy log file. On digging > deeper, I found that these were being thrown in a segment of the code that > followed a connection-open statement, but before the corresponding > connection-close statement, so the close statement was being skipped, > thereby constituting a memory leak. > > The exceptions are being thrown by queries submitted by web crawlers, > which by-pass the user interface and submit bogus requests; for example, > asking for holidays information for a city called "Scotland". This invokes > an internal table lookup (not a db query) that tries to return an ID for > the city, and not finding one, it returns "null". This in turn causes a > null pointer exception. > > This problem was indeed resolved either by using the try-with-resources > construct or moving the db connection-close operation into a finally > clause. Both of these seem to work. The exceptions are still being > thrown, but the program keeps running with no memory leak. > > Thanks again to everyone for your comments. It's been very helpful, and I > have learned a lot in the process. > > Dan > > -----Original Message----- > From: Robert Turner <rtur...@e-djuster.ca.INVALID> > Sent: Tuesday, August 19, 2025 8:27 PM > To: Tomcat Users List <users@tomcat.apache.org> > Subject: Re: [EXTERNAL EMAIL] How to access a REST service > > Daniel, > > If you would like to validate your theory, simply remove the > try-with-resources and revert to your previous logic. However, I strongly > believe your theory is completely incorrect and makes no sense to me at all. > > I'm glad it's working much better for you now. Best of luck, > > Robert > > On Tue, Aug 19, 2025 at 8:21 PM Daniel Schwartz <d...@danielgschwartz.com> > wrote: > > > Chuck, Robert, Chris, > > > > I believe that this issue is finally resolved. My system has been > > running flawlessly for most of two days now with the minimum and > > maximum pool sizes set to the defaults of 8 and 32 respectively. It is > still getting frequent > > hits by web crawlers and it is handling this with no problem. So I'm > > prepared to just leave it at this. > > > > After all that has been said and done, I suspect that the problem all > > along was that, after many months of running Glassfish with no > > maintenance, the system developed some internal glitch that made it > > create 250+ connection objects that weren't being removed. But when I > > shut Glassfish down yesterday, deleted the .war file, and then > > restarted it with an empty domain, it reset itself and resumed running > > normally. Anyway, this is my speculation at this point. > > > > I did do a test today to see if the production version running on AWS > > will report thrown exceptions and found that it does. As an > > experiment I put the line "float something = 1/0;" in the method that > > retrieves the list of countries and then ran my test query. It > > retrieved the list of countries and output the expected "Other > exception" message in the system.log file. > > Based on this, I believe that the system was never throwing any > > exceptions. There was some speculation that exceptions were being > > "swallowed", i.e., being thrown without any reports, but this doesn't > > seem to be happening. The fact that I have never seen any such > > exception reports I take as confirmation that this was never the problem. > > > > Also, before I ran my test today, which required replacing the current > > .war file with my test .war file, I checked the current contents of > > the JDBC pooling monitor and saw that the number of connections > > released equaled the number acquired, and the number of potential > > leaks was still at zero. I take this as further confirmation that > everything is now okay. > > > > I do appreciate your observation that my JDBC pool size was abnormally > > high and that this was something that needed to be addressed. Whether > > this had to do with my code changes or the issue I just described, I'm > > thankful for everyone's efforts and that the issue now seems to be > resolved. > > > > Best regards, > > > > Dan > > > > From: Chuck Caldarale <n82...@gmail.com> > > Sent: Tuesday, August 19, 2025 1:33 AM > > To: Tomcat Users List <users@tomcat.apache.org> > > Subject: Re: [EXTERNAL EMAIL] How to access a REST service > > > > > On 2025 Aug 18, at 21:22, Daniel Schwartz <d...@danielgschwartz.com > > <mailto:d...@danielgschwartz.com>> wrote: > > I wasn't resisting using > > the try-with-resources construct, only when I tried to do this, I was > > getting compiler errors. But I evidently wasn't doing > > NkdkJdXPPEBannerStart Be Careful With This Message From (Chuck > > Caldarale <n82...@gmail.com>)< > > https://godaddy1.cloud-protect.net/email-details/?k=k1&payload=53616c7 > > 465645f5f51a287f6939989554348daf64676e0bf5f4ffa1456b85165104baabd11d2f > > 4f5bc0540d6b7871afb306ea1aca1c07fdba45a24a58aaaf383c3a1273ce3b65804c22 > > 46118bcb75f9a17caf373b92d7f2d854d41c3082963c5f125ae63845f041a91581d495 > > 3459459aa2c4234a2139a28123b2078ebbe1692068d6901194786f9343be17d758d90e > > b09e3dae7ff54cd3793b11432fdf9d37bd8db5d655c3c25c0886896cda9112c4fb9137 > > c2d65eecb0c1a06af06bf9910a86cefb6cb06aeddf8c5b5c98eb750cbd2d035efc232c > > cb97a5a636e0ea51701e39e165cf615cbe618288fc0f3b7969034 > > > > > Learn More< > > https://godaddy1.cloud-protect.net/email-details/?k=k1&payload=53616c7 > > 465645f5f51a287f6939989554348daf64676e0bf5f4ffa1456b85165104baabd11d2f > > 4f5bc0540d6b7871afb306ea1aca1c07fdba45a24a58aaaf383c3a1273ce3b65804c22 > > 46118bcb75f9a17caf373b92d7f2d854d41c3082963c5f125ae63845f041a91581d495 > > 3459459aa2c4234a2139a28123b2078ebbe1692068d6901194786f9343be17d758d90e > > b09e3dae7ff54cd3793b11432fdf9d37bd8db5d655c3c25c0886896cda9112c4fb9137 > > c2d65eecb0c1a06af06bf9910a86cefb6cb06aeddf8c5b5c98eb750cbd2d035efc232c > > cb97a5a636e0ea51701e39e165cf615cbe618288fc0f3b7969034 > > > > > Potential Impersonation > > The sender's identity could not be verified and someone may be > > impersonating the sender. Take caution when interacting with this > message. > > > > NkdkJdXPPEBannerEnd > > > > > > > > > On 2025 Aug 18, at 21:22, Daniel Schwartz <d...@danielgschwartz.com > > <mailto:d...@danielgschwartz.com>> wrote: > > > > > > > > > > I wasn't resisting using the try-with-resources construct, only when > > > I > > tried to do this, I was getting compiler errors. But I evidently > > wasn't doing it right, because I tried it again and this time it worked. > > > > > > > > > > Following is the new code for getting the list of countries. It > > > uses > > this construct and in addition has an extra catch clause for any kind > > of exception. I have done this in all six places where a connection > > is being created. > > > > > > > > > > I restarted Glassfish earlier today and dropped in the new .war file. > > It has been running for 6-8 hours now. The log files show no evidence > > of any exceptions being thrown. > > > > > > > > > > > > When you’re running GlassFish in your production environment, is it > > running as a service? If so, you’re not going to see anything written > > via System.out (or System.err), unless GlassFish has an option to > > redirect that to its internal logging mechanism. Tomcat has such an > > option, and GlassFish started as a fork of Tomcat many years ago, so > > such an option may well exist. > > > > > > > > > > > > > The current contents of the JDBC monitor page are copied below. I > > > don't > > know how to interpret most of this, but I do note that the number of > > connections released equals the number acquired. Also, I have set the > > Connection Leak Timeout to be 10 seconds, and the NumPotentialConnLeak > > is still at 0. > > > > > > > > > > > > However, there are a couple of stats that might be a bit concerning: > > of the 20 physical connections created, only 1 is free and 19 were > destroyed. > > I would think that a pool would destroy connections only when it had > > to (eg, the connection is considered unsafe for reuse), although > > GlassFish does appear to have a mode where it discards connections > > after a while just in case. There’s also something odd with having > > only 1 free connection, since the pool is supposed to maintain a > > minimum of 8. Or were these statistics captured when the minimum was set > to 1? > > > > > > > > > > > > > In addition, I did some further experimenting with the pool size, > > > and > > now I find that if I set the minimum back to 8 and the maximum to 32, > > which are the defaults, the system keeps running and doesn't output > > that "can't allocate more connections" message. For me this is very > > puzzling. Did my code changes have anything to do with this? > > > > > > > > > > > > Not much else changed, so in all likelihood, yes. Note that the > > highest number of connections concurrently used is just 2, which is > > what we’d expect for an app like yours. > > > > > > > > > > > > > Anyway, my interpretation of all this is as before, that there are > > > no > > memory leaks. Is this correct? > > > > > > > > > > > > There don’t appear to be any with your revised code, at least up to > > this point (other than perhaps those 19 destroyed connections), but > > let it run for a few days and see what happens if the web crawlers > > start feeding garbage to your web service again. > > > > > > > > - Chuck > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org<mailto: > > users-unsubscr...@tomcat.apache.org> > > > > For additional commands, e-mail: users-h...@tomcat.apache.org<mailto: > > users-h...@tomcat.apache.org> > > > > > > >
