Hello Chris, Definitely you have to increase the logging level. If your tomcat instance is using JULI and you suspect from struts perhaps you could try to increase the debugging level of struts in your $CATALINA_BASE/conf/logging.properties:
org.apache.struts.level=FINE or FINEST Hope it helps, Luis 2018-08-06 18:08 GMT+02:00 Louis Zipes <louis.zi...@gmcr.com>: > Hi, > Not an answer but just letting you my thinking on where I would look for > additional error messages that might help tell more of the story. > > 1) Any additional information in the individual Java Plugin Logs that tell > more of the story? > > 2) Can you increase the logging on the Tomcat side to try to capture more > of the error? > > 3) What about the SUSE logs themselves? Do they tell you anything? > > Thanks, Louis > > -----Original Message----- > From: Chrifister [mailto:chrifis...@gmail.com] > Sent: Monday, August 06, 2018 11:54 AM > To: users@tomcat.apache.org > Subject: Tomcat 8.x-9.x + Struts 1.3.x - Applications will get into a > state where they won't serve random bunch of static resources, 500 errors > > - - - external message, proceed with caution - - - > > > Hi, > > Our current setup is Tomcat 9.0.8 running SuSE Enterprise. This server is > running a dozen web applications built with Struts 1.3.8 with some newer > Spring applications on the horizon. There is a large user base with some > applications seeing heavy usage. Applications are currently using Java 1.7 > and 1.8. > > We were originally running Tomcat 7.x but were having issues with perm gen > maxing out very quickly for unknown reasons but possibly related to a buggy > third party "enterprise-grade" reporting Java library. We had to restart > the server nightly to try and keep perm gen from maxing out. Part of the > reason was this third-party library spawned immortal threads that would > prevent an application from unloading and being garbage collected when a > newer build of an application was deployed (the developers behind it never > expected the library would be run on a server with multiple > applications....). So we upgraded Tomcat to 8.5.x first and then to 9.x > recently. This fixed the perm gen issue. > > Our current issue we are having is that for some unknown reason and after > seemingly random lengths of time, an application will get into a state and > will start having issues which results in failed page loads or pages not > loading correctly. According to Chrome's network tab in developer console, > a random bunch of static resources (javascript, css, images) are returning > 500 errors and not being served. Whether the page loads or not depends on > exactly which resources were not returned. Every time you access any page > in that application, another random bunch of resources have 500 errors. > There's no indication in any of Tomcat's log files that an application is > in this state. The application will stay in this unusable state until it is > restarted or the server is restarted. > > We've resorted to once again scheduling the server to restart nightly which > has cut down on the frequency of this happening which hints at this being > related to usage, but it is still happening once a week and sometimes more. > The applications that seem to experience this the most are I believe the > more heavily used applications. > > No Spring application has experienced this issue on our other servers which > leads me to tentatively say that Spring is not affected and/or is not a > cause of the issue but upgrading all applications to Spring is not feasible > at the moment. > > We've tried upgrading Struts in the most frequently affected applications > to 1.3.10 but it did not solve the issue and actually afflicted us with > another issue stemming from a bug in that Struts version. So we had to go > back to 1.3.8. > > I spoke with a couple of people in Tomcat's IRC channel and they seemed to > think it was a third-party library or a problem/race condition between the > Struts and Tomcat servlets. While this may be important information, I have > no idea what to do with it. > > I'm not sure debugging is a possibility because it's a remote server and I > wouldn't even know what to look for. I also can't allow a production > application to remain in this state for very long. > > I can't file a bug report because I can't reproduce it at will and I am > unable to provide thread or heap dumps. > > I have a suspicion it may be caused by that third part library although I > don't see how that library would affect Tomcat's serving of static > resources. > > This issue has never happened to our test server or our local instances of > tomcat. Since I suspect it's related to usage, this is not surprising. > > Any help would be greatly appreciated. > > Chris > --------------------------------------- > CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and > may contain information that is confidential, proprietary or exempt from > disclosure. If you are not the intended recipient, please contact the > sender immediately. Unauthorized use or distribution is prohibited and may > be unlawful. > -- "Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better." - Samuel Beckett
