On Sat, Feb 23, 2013 at 5:44 PM, Leo Donahue - RDSA IT < leodona...@mail.maricopa.gov> wrote:
> Tomcat 7.0.32 (..upgrading right after this email...) > > it would be nice to know if you had tomcat's (ROOT) web app while you were running tomcat 7.0.32. You definitely should have seen the same WARNINGs and tomcat-login attempts against your server. IMO, when you migrated from 7.0.32 to 7.0.37, I think you had configuration tightened up/down to prevent these tomcat login attempts, and now that you migrated to 7.0.37, i am 'assuming' that you forgot to tighten up/down your tomcat configuration. Deploying a third party app is causing Out of Memory errors on our web > server. > > Will increasing these: -Xms1024m -Xmx1024m -XX:PermSize=256m > -XX:MaxPermSize=356m just delay the inevitable? > > Feb 22, 2013 3:17:44 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > at java.util.Formatter.parse(Formatter.java:2480) > at java.util.Formatter.format(Formatter.java:2414) > at java.util.Formatter.format(Formatter.java:2367) > at java.lang.String.format(String.java:2769) > at > com.esri.rest.mapserver.MapInfo.writeTileToResponse(MapInfo.java:733) > at > com.esri.rest.mapserver.MapServerServlet.d(MapServerServlet.java:403) > at > com.esri.rest.mapserver.MapServerServlet.service(MapServerServlet.java:152) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:722) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749) > at > org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487) > at > org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379) > at > org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339) > at > com.esri.rest.catalog.CatalogServlet.service(CatalogServlet.java:181) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:722) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749) > at > org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487) > at > org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379) > at > org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339) > at > com.esri.rest.DispatchServlet.service(DispatchServlet.java:123) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:722) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at > com.esri.rest.security.SecurityFilter.doFilter(SecurityFilter.java:79) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at com.esri.rest.RestFilter.doFilter(RestFilter.java:81) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > Feb 22, 2013 3:17:46 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > at java.util.Formatter.parse(Formatter.java:2480) > at java.util.Formatter.format(Formatter.java:2414) > at java.util.Formatter.format(Formatter.java:2367) > at java.lang.String.format(String.java:2769) > at > com.esri.rest.mapserver.MapInfo.writeTileToResponse(MapInfo.java:733) > at > com.esri.rest.mapserver.MapServerServlet.d(MapServerServlet.java:403) > at > com.esri.rest.mapserver.MapServerServlet.service(MapServerServlet.java:152) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:722) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749) > at > org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487) > at > org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379) > at > org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339) > at > com.esri.rest.catalog.CatalogServlet.service(CatalogServlet.java:181) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:722) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749) > at > org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487) > at > org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379) > at > org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339) > at > com.esri.rest.DispatchServlet.service(DispatchServlet.java:123) > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:722) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at > com.esri.rest.security.SecurityFilter.doFilter(SecurityFilter.java:79) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > at com.esri.rest.RestFilter.doFilter(RestFilter.java:81) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) > Feb 22, 2013 3:17:47 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:17:49 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:17:49 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:17:49 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:17:49 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:17:50 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:17:51 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:17:53 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:18:04 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:18:06 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:18:09 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:18:23 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > Feb 22, 2013 3:18:24 PM com.esri.rest.RestFilter doFilter > SEVERE: Unable to process request > java.lang.OutOfMemoryError: PermGen space > "http-apr-80-exec-19" java.lang.OutOfMemoryError: PermGen space > Exception in thread "http-apr-80-exec-26" java.lang.OutOfMemoryError: > PermGen space > Exception in thread "http-apr-80-exec-28" java.lang.OutOfMemoryError: > PermGen space > ... more of the same > > The following catalina logs were the result of a start/stop Tomcat. > > Feb 23, 2013 3:15:59 PM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/rest] appears to have started a thread named > [PDUManagerImpl-openConnection] but has failed to stop it. This is very > likely to create a memory leak. > Feb 23, 2013 3:15:59 PM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/rest] appears to have started a thread named > [PDUManagerImpl-openConnection] but has failed to stop it. This is very > likely to create a memory leak. > Feb 23, 2013 3:15:59 PM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > SEVERE: The web application [/rest] appears to have started a thread named > [PDUManagerImpl-openConnection] but has failed to stop it. This is very > likely to create a memory leak. > Feb 23, 2013 3:15:59 PM org.apache.catalina.loader.WebappClassLoader > checkThreadLocalMapForLeaks > SEVERE: The web application [/rest] created a ThreadLocal with key of type > [com.sun.xml.bind.v2.ClassFactory$1] (value > [com.sun.xml.bind.v2.ClassFactory$1@1488560]) and a value of type > [java.util.WeakHashMap] (value [{class > com.esri.arcgisws.CacheDescriptionInfo=java.lang.ref.WeakReference@1919e73, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfLayerDescriptionAdapter=java.lang.ref.WeakReference@139c41d, > class com.esri.arcgisws.Fields=java.lang.ref.WeakReference@17a6a4b, class > com.esri.arcgisws.adapters.Adapters$ArrayOfLabelClassDescriptionAdapter=java.lang.ref.WeakReference@1d889aa, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfMapServerLegendInfoAdapter=java.lang.ref.WeakReference@136083b, > class > com.esri.arcgisws.GetLegendInfoResponse=java.lang.ref.WeakReference@567aef, > class > com.esri.arcgisws.ArrayOfMapServerLegendGroup=java.lang.ref.WeakReference@1a102ef, > class > com.esri.arcgisws.GetCacheStorageInfoResponse=java.lang.ref.WeakReference@1c7b77e, > class > com.esri.arcgisws.ArrayOfLayerCacheInfo=java.lang.ref.WeakReference@dfd603, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfMapServerBookmarkAdapter=java.lang.ref.WeakReference@15b6382, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfLayerDrawingDescriptionAdapter=java.lang.ref.WeakReference@e1fbe3, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfLayerCacheInfoAdapter=java.lang.ref.WeakReference@60090d, > class com.esri.arcgisws.TileCacheInfo=java.lang.ref.WeakReference@f23ad3, > class > com.esri.arcgisws.ArrayOfPropertySetProperty=java.lang.ref.WeakReference@f42ea0, > class > com.esri.arcgisws.FeatureLayerDrawingDescription=java.lang.ref.WeakReference@12130d1, > class com.esri.arcgisws.PointN=java.lang.ref.WeakReference@c1d35, class > com.esri.arcgisws.adapters.Adapters$ArrayOfPropertySetPropertyAdapter=java.lang.ref.WeakReference@151b6ea, > class > com.esri.arcgisws.MapServerLegendClass=java.lang.ref.WeakReference@1bfcfca, > class > com.esri.arcgisws.ArrayOfMapServerBookmark=java.lang.ref.WeakReference@1e2e2f4, > class com.esri.arcgisws.SimpleTextSymbol=java.lang.ref.WeakReference@a21ac2, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfMapServerLegendGroupAdapter=java.lang.ref.WeakReference@8aca5e, > class com.esri.arcgisws.CacheStorageInfo=java.lang.ref.WeakReference@1ec10c9, > class com.esri.arcgisws.LayerCacheInfo=java.lang.ref.WeakReference@1420a71, > class com.esri.arcgisws.ImageResult=java.lang.ref.WeakReference@bb94d9, > class com.esri.arcgisws.EnvelopeN=java.lang.ref.WeakReference@129642c, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfLODInfoAdapter=java.lang.ref.WeakReference@6fa30c, > class > com.esri.arcgisws.adapters.JAXBElementAdapter=java.lang.ref.WeakReference@1397663, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfMapServerLegendClassAdapter=java.lang.ref.WeakReference@18b697e, > class > com.esri.arcgisws.ArrayOfMapServerLegendInfo=java.lang.ref.WeakReference@1baeec3, > class > org.xmlsoap.schemas.soap.envelope.Body=java.lang.ref.WeakReference@1914a39, > class > com.esri.arcgisws.LineLabelPlacementDescription=java.lang.ref.WeakReference@19ff359, > class > com.esri.arcgisws.ArrayOfLayerDrawingDescription=java.lang.ref.WeakReference@216f75, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfStringAdapter=java.lang.ref.WeakReference@11c9ba1, > class com.esri.arcgisws.CacheControlInfo=java.lang.ref.WeakReference@12d813e, > class > com.esri.arcgisws.MapServerLegendInfo=java.lang.ref.WeakReference@801b62, > class com.esri.arcgisws.SimpleFillSymbol=java.lang.ref.WeakReference@85e41d, > class > com.esri.arcgisws.GetDocumentInfoResponse=java.lang.ref.WeakReference@43563e, > class > com.esri.arcgisws.LabelingDescription=java.lang.ref.WeakReference@d32baa, > class com.esri.arcgisws.TileImageInfo=java.lang.ref.WeakReference@1d7ae87, > class java.util.ArrayList=java.lang.ref.WeakReference@1032d4b, class > com.esri.arcgisws.LabelClassDescription=java.lang.ref.WeakReference@192c07a, > class com.esri.arcgisws.RgbColor=java.lang.ref.WeakReference@19a9de2, > class com.esri.arcgisws.LODInfo=java.lang.ref.WeakReference@1f36a7c, > class > com.esri.arcgisws.ArrayOfMapServerLegendClass=java.lang.ref.WeakReference@1d3f905, > class com.esri.arcgisws.GeometryDef=java.lang.ref.WeakReference@181c482, > class > com.esri.arcgisws.GetDefaultLayerDrawingDescriptionsResponse=java.lang.ref.WeakReference@8c491e, > class > com.esri.arcgisws.ProjectedCoordinateSystem=java.lang.ref.WeakReference@147bef, > class com.esri.arcgisws.PropertySet=java.lang.ref.WeakReference@d4abda, > class com.esri.arcgisws.MapServerInfo=java.lang.ref.WeakReference@5eac01, > class > com.esri.arcgisws.GetCacheDescriptionInfoResponse=java.lang.ref.WeakReference@16e8e76, > class com.esri.arcgisws.LayerDescription=java.lang.ref.WeakReference@14af0fe, > class > com.esri.arcgisws.GetServiceConfigurationInfoResponse=java.lang.ref.WeakReference@113e285, > class > com.esri.arcgisws.ArrayOfLayerDescription=java.lang.ref.WeakReference@7c9492, > class > com.esri.arcgisws.GetServerInfoResponse=java.lang.ref.WeakReference@fe2479, > class com.esri.arcgisws.Field=java.lang.ref.WeakReference@10c8785, class > org.xmlsoap.schemas.soap.envelope.Envelope=java.lang.ref.WeakReference@1953895, > class > com.esri.arcgisws.MapServerLegendGroup=java.lang.ref.WeakReference@1b5dab5, > class com.esri.arcgisws.SimpleRenderer=java.lang.ref.WeakReference@29b46a, > class com.esri.arcgisws.MapDescription=java.lang.ref.WeakReference@3bf22, > class com.esri.arcgisws.ArrayOfLODInfo=java.lang.ref.WeakReference@1f9805f, > class com.esri.arcgisws.ArrayOfInt=java.lang.ref.WeakReference@15b1b88, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfFieldAdapter=java.lang.ref.WeakReference@6cb4c8, > class com.esri.arcgisws.MapExtent=java.lang.ref.WeakReference@d0cc3a, > class com.esri.arcgisws.ArrayOfString=java.lang.ref.WeakReference@101828, > class > com.esri.arcgisws.ArrayOfMapLayerInfo=java.lang.ref.WeakReference@a205d2, > class com.esri.arcgisws.SimpleLineSymbol=java.lang.ref.WeakReference@12c4c7a, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfIntAdapter=java.lang.ref.WeakReference@151000a, > class com.esri.arcgisws.ArrayOfField=java.lang.ref.WeakReference@d99967, > class > com.esri.arcgisws.PropertySetProperty=java.lang.ref.WeakReference@1e668d0, > class com.esri.arcgisws.MapLayerInfo=java.lang.ref.WeakReference@1eb59b1, > class > com.esri.arcgisws.ArrayOfLabelClassDescription=java.lang.ref.WeakReference@4dabd7, > class > com.esri.arcgisws.adapters.Adapters$ArrayOfMapLayerInfoAdapter=java.lang.ref.WeakReference@14cecee}]) > but failed to remove it when the web application was stopped. Threads are > going to be renewed over time to try and avoid a probable memory leak. > Feb 23, 2013 3:15:59 PM org.apache.catalina.loader.WebappClassLoader > checkThreadLocalMapForLeaks > SEVERE: The web application [/rest] created a ThreadLocal with key of type > [com.sun.xml.bind.v2.runtime.Coordinator$1] (value > [com.sun.xml.bind.v2.runtime.Coordinator$1@1573137]) and a value of type > [java.lang.Object[]] (value [[Ljava.lang.Object;@16020ed]) but failed to > remove it when the web application was stopped. Threads are going to be > renewed over time to try and avoid a probable memory leak. > > Leo > >