George, thanks for sharing your experience, good to know I'm not the first down this path. I'd be interested to know if you've tested at all with Tomcat 8 and what your findings were, if so. Based on my experience I'd expect you to be in a similar boat to us.
Mark, thanks for the reply. I've uploaded my "dummy" webapp here: http://filebin.ca/2XCPi8oAdy84/test.war As for the JAR scanning / startup times, with Tomcat 8's jarsToScan setting I'm confident we'd be able to manage that, and am happy to leapfrog over Tomcat 7. It's the memory utilization in Tomcat 8 specifically, independent of JAR scanning, that's the show-stopper for us. Let me know if I can provide any other info, and thanks again. On Tue, Feb 16, 2016 at 12:58 PM Mark Thomas <ma...@apache.org> wrote: > On 16/02/2016 16:13, Ty wrote: > > Summary: > > > > Because of our use case (hundreds of webapps per instance), we cannot > > feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to > > the massive increase in memory the upgrade causes. > > > > Background: > > > > We’ve been running Tomcat 6 for several years, on perhaps an unusual > scale: > > several dozen Tomcat instances, each running up to a *few hundred > webapps* > > in a single container. (Picture a cloud service provider with several > > products and hundreds of customers, with one dedicated webapp > > per-customer-per-product, resulting in many thousands of web > > applications. It’s > > not ideal but it’s how the software was designed). Typical max heap > sizes > > per container are 1GB to 4GB depending on utilization, and max container > > startup times are in the neighborhood of 5 to 10 minutes. > > > > · When Tomcat 7 was released, our testing showed a 10-to-20-fold > > increase in memory consumption and a 3-to-5-fold increase in startup time > > for our test case (100 deployed webapps). After some digging we > determined > > that these increases were related to JAR scanning > > (tomcat.util.scan.DefaultJarScanner). > > JAR scanning should be transient. What was it that was causing the 10-20 > fold increase? > > > · When Tomcat 8 was released, we noted that it included a > > “whitelist” version of the jarsToSkip Property, jarsToScan. Our hopes > were > > high that we could skip from Tomcat 6 to Tomcat 8, but our test case > (again > > 100 deployed webapps) quickly deflated those hopes: similar startup > times > > but at least a *30-fold increase in memory consumption*, regardless of > how > > we configure jarsToScan or jarsToSkip. > > > > · A heap dump analysis showed that the Tomcat 8 memory increase > was > > largely due to the size/count of > > org.apache.catalina.webresources.JarResourceSet objects. > > Those are only going to be created when a JAR is found that contains > META-INF/resources. The spec requires Tomcat to expose those as static > resources and that requires plumbing. Note Tomcat 6 will just ignore > these because the spec is to old for this feature. > > > We attempted to > > reduce this by setting the Context/Resources@cachingAllowed atttribute > to > > “false” and also tried to tune the “cacheMaxSize” and > “cacheObjectMaxSize” > > attributes. > > Those attributes should have no impact on the number or size of the > JarResourceSet instances. > > > The only effect that came from these changes was a change in > > the object that took up all the space: instead of > > org.apache.catalina.webresources.JarResourceSet objects, they are > > org.apache.catalina.webresources.StandardResourceSet objects, taking up > > roughly the same amount of space. > > Strange. Very strange. > > > Question: > > > > Is there anything else we can adjust to make Tomcat 8’s memory > consumption > > closer to that of Tomcat 6’s, for our use case? > > It is hard to see exactly what is going on here. The symptoms you > describe and the resulting changes with configuration are not consistent > with how I would expect Tomcat to behave nor can I see how it might > behave in the way described. > > > If not, we are faced with > > either running Tomcat 6 past its EOL, or possibly maintaining a very > large > > jarsToSkip blacklist in Tomcat 7 (until its EOL). It is not economically > > feasible for us to increase the physical memory of our servers by 30x so > we > > can run Tomcat 8. > > > > Test procedure: > > > > > > - Create a "dummy" webapp (using default Maven archetype) named > test.war > > - Add several popular Maven dependencies such that the total size of > the > > JARs in WEB-INF/lib is about 30MB > > Can you provide the sample project? It makes sense to ensure we are > working from the same baseline. I wonder if the libraries themselves are > a factor here. > > > - Download the latest versions of Tomcat 6, 7, and 8. > > - Set Java environment variables for a JMX listener and a 3GB max heap > > - Make 100 copies of test.war (test1.war, test2.war, …, test100.war) > and > > drop them in /webapps. > > - Start the container, note the reported startup time, perform an > > explicit major GC, and note the heap utilization. > > - Run each test several times and average the results > > That is a nice simple test we should be able to repeat. > > > Test results: > > > > Case 1 (baseline): > > > > > > > > +---------+--------------+---------------------------+ > > | version | startup time | heap usage after major GC | > > +---------+--------------+---------------------------+ > > | tomcat6 | 36,711ms | 21,163,288 | > > | tomcat7 | 104,517ms | 489,992,264 | > > | tomcat8 | 156,094ms | 1,010,512,568 | > > +---------+--------------+---------------------------+ > > > > > > Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”) > > > > +---------+--------------+---------------------------+ > > | version | startup time | heap usage after major GC | > > +---------+--------------+---------------------------+ > > | tomcat6 | 36,711ms | 21,163,288 | > > | tomcat7 | 38,979ms | 72,359,840 | > > | tomcat8 | 52,040ms | 633,682,336 | > > +---------+--------------+---------------------------+ > > From those figures my impression is that we should be able to do > something about the memory. The start-up time with JAR scanning is what > it is. Without JAR scanning there might be scope for improvement. > > Mark > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
