On 16/02/2016 16:13, Ty wrote: > Summary: > > Because of our use case (hundreds of webapps per instance), we cannot > feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to > the massive increase in memory the upgrade causes. > > Background: > > We’ve been running Tomcat 6 for several years, on perhaps an unusual scale: > several dozen Tomcat instances, each running up to a *few hundred webapps* > in a single container. (Picture a cloud service provider with several > products and hundreds of customers, with one dedicated webapp > per-customer-per-product, resulting in many thousands of web > applications. It’s > not ideal but it’s how the software was designed). Typical max heap sizes > per container are 1GB to 4GB depending on utilization, and max container > startup times are in the neighborhood of 5 to 10 minutes. > > · When Tomcat 7 was released, our testing showed a 10-to-20-fold > increase in memory consumption and a 3-to-5-fold increase in startup time > for our test case (100 deployed webapps). After some digging we determined > that these increases were related to JAR scanning > (tomcat.util.scan.DefaultJarScanner).
JAR scanning should be transient. What was it that was causing the 10-20 fold increase? > · When Tomcat 8 was released, we noted that it included a > “whitelist” version of the jarsToSkip Property, jarsToScan. Our hopes were > high that we could skip from Tomcat 6 to Tomcat 8, but our test case (again > 100 deployed webapps) quickly deflated those hopes: similar startup times > but at least a *30-fold increase in memory consumption*, regardless of how > we configure jarsToScan or jarsToSkip. > > · A heap dump analysis showed that the Tomcat 8 memory increase was > largely due to the size/count of > org.apache.catalina.webresources.JarResourceSet objects. Those are only going to be created when a JAR is found that contains META-INF/resources. The spec requires Tomcat to expose those as static resources and that requires plumbing. Note Tomcat 6 will just ignore these because the spec is to old for this feature. > We attempted to > reduce this by setting the Context/Resources@cachingAllowed atttribute to > “false” and also tried to tune the “cacheMaxSize” and “cacheObjectMaxSize” > attributes. Those attributes should have no impact on the number or size of the JarResourceSet instances. > The only effect that came from these changes was a change in > the object that took up all the space: instead of > org.apache.catalina.webresources.JarResourceSet objects, they are > org.apache.catalina.webresources.StandardResourceSet objects, taking up > roughly the same amount of space. Strange. Very strange. > Question: > > Is there anything else we can adjust to make Tomcat 8’s memory consumption > closer to that of Tomcat 6’s, for our use case? It is hard to see exactly what is going on here. The symptoms you describe and the resulting changes with configuration are not consistent with how I would expect Tomcat to behave nor can I see how it might behave in the way described. > If not, we are faced with > either running Tomcat 6 past its EOL, or possibly maintaining a very large > jarsToSkip blacklist in Tomcat 7 (until its EOL). It is not economically > feasible for us to increase the physical memory of our servers by 30x so we > can run Tomcat 8. > > Test procedure: > > > - Create a "dummy" webapp (using default Maven archetype) named test.war > - Add several popular Maven dependencies such that the total size of the > JARs in WEB-INF/lib is about 30MB Can you provide the sample project? It makes sense to ensure we are working from the same baseline. I wonder if the libraries themselves are a factor here. > - Download the latest versions of Tomcat 6, 7, and 8. > - Set Java environment variables for a JMX listener and a 3GB max heap > - Make 100 copies of test.war (test1.war, test2.war, …, test100.war) and > drop them in /webapps. > - Start the container, note the reported startup time, perform an > explicit major GC, and note the heap utilization. > - Run each test several times and average the results That is a nice simple test we should be able to repeat. > Test results: > > Case 1 (baseline): > > > > +---------+--------------+---------------------------+ > | version | startup time | heap usage after major GC | > +---------+--------------+---------------------------+ > | tomcat6 | 36,711ms | 21,163,288 | > | tomcat7 | 104,517ms | 489,992,264 | > | tomcat8 | 156,094ms | 1,010,512,568 | > +---------+--------------+---------------------------+ > > > Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”) > > +---------+--------------+---------------------------+ > | version | startup time | heap usage after major GC | > +---------+--------------+---------------------------+ > | tomcat6 | 36,711ms | 21,163,288 | > | tomcat7 | 38,979ms | 72,359,840 | > | tomcat8 | 52,040ms | 633,682,336 | > +---------+--------------+---------------------------+ >From those figures my impression is that we should be able to do something about the memory. The start-up time with JAR scanning is what it is. Without JAR scanning there might be scope for improvement. Mark --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
