Summary: Because of our use case (hundreds of webapps per instance), we cannot feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to the massive increase in memory the upgrade causes.
Background: We’ve been running Tomcat 6 for several years, on perhaps an unusual scale: several dozen Tomcat instances, each running up to a *few hundred webapps* in a single container. (Picture a cloud service provider with several products and hundreds of customers, with one dedicated webapp per-customer-per-product, resulting in many thousands of web applications. It’s not ideal but it’s how the software was designed). Typical max heap sizes per container are 1GB to 4GB depending on utilization, and max container startup times are in the neighborhood of 5 to 10 minutes. · When Tomcat 7 was released, our testing showed a 10-to-20-fold increase in memory consumption and a 3-to-5-fold increase in startup time for our test case (100 deployed webapps). After some digging we determined that these increases were related to JAR scanning (tomcat.util.scan.DefaultJarScanner). When setting jarsToSkip=*, the memory consumption was decreased to something somewhat more reasonable (2-3x increase over Tomcat 6) and the startup time went back to what we were used to with Tomcat 6. However at that time we opted to stay on Tomcat 6, because our apps were not functional with jarsToSkip=*, and we deemed it impractical to manage a very large “blacklist” in the jarsToSkip Property. · When Tomcat 8 was released, we noted that it included a “whitelist” version of the jarsToSkip Property, jarsToScan. Our hopes were high that we could skip from Tomcat 6 to Tomcat 8, but our test case (again 100 deployed webapps) quickly deflated those hopes: similar startup times but at least a *30-fold increase in memory consumption*, regardless of how we configure jarsToScan or jarsToSkip. · A heap dump analysis showed that the Tomcat 8 memory increase was largely due to the size/count of org.apache.catalina.webresources.JarResourceSet objects. We attempted to reduce this by setting the Context/Resources@cachingAllowed atttribute to “false” and also tried to tune the “cacheMaxSize” and “cacheObjectMaxSize” attributes. The only effect that came from these changes was a change in the object that took up all the space: instead of org.apache.catalina.webresources.JarResourceSet objects, they are org.apache.catalina.webresources.StandardResourceSet objects, taking up roughly the same amount of space. Question: Is there anything else we can adjust to make Tomcat 8’s memory consumption closer to that of Tomcat 6’s, for our use case? If not, we are faced with either running Tomcat 6 past its EOL, or possibly maintaining a very large jarsToSkip blacklist in Tomcat 7 (until its EOL). It is not economically feasible for us to increase the physical memory of our servers by 30x so we can run Tomcat 8. Test procedure: - Create a "dummy" webapp (using default Maven archetype) named test.war - Add several popular Maven dependencies such that the total size of the JARs in WEB-INF/lib is about 30MB - Download the latest versions of Tomcat 6, 7, and 8. - Set Java environment variables for a JMX listener and a 3GB max heap - Make 100 copies of test.war (test1.war, test2.war, …, test100.war) and drop them in /webapps. - Start the container, note the reported startup time, perform an explicit major GC, and note the heap utilization. - Run each test several times and average the results Test results: Case 1 (baseline): +---------+--------------+---------------------------+ | version | startup time | heap usage after major GC | +---------+--------------+---------------------------+ | tomcat6 | 36,711ms | 21,163,288 | | tomcat7 | 104,517ms | 489,992,264 | | tomcat8 | 156,094ms | 1,010,512,568 | +---------+--------------+---------------------------+ Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”) +---------+--------------+---------------------------+ | version | startup time | heap usage after major GC | +---------+--------------+---------------------------+ | tomcat6 | 36,711ms | 21,163,288 | | tomcat7 | 38,979ms | 72,359,840 | | tomcat8 | 52,040ms | 633,682,336 | +---------+--------------+---------------------------+
