> JAR scanning should be transient. What was it that was causing the 10-20 > fold increase?
>From memory (this was a while ago), it was related to java.util.jar.Manifest objects, or what they contained/referenced. I'd be glad to do another heap analysis and summarize it if it would help, but even more glad to just skip Tomcat 7 altogether. On Tue, Feb 16, 2016 at 2:11 PM Ty <ty733...@gmail.com> wrote: > George, thanks for sharing your experience, good to know I'm not the first > down this path. I'd be interested to know if you've tested at all with > Tomcat 8 and what your findings were, if so. Based on my experience I'd > expect you to be in a similar boat to us. > > Mark, thanks for the reply. I've uploaded my "dummy" webapp here: > > http://filebin.ca/2XCPi8oAdy84/test.war > > As for the JAR scanning / startup times, with Tomcat 8's jarsToScan > setting I'm confident we'd be able to manage that, and am happy to leapfrog > over Tomcat 7. It's the memory utilization in Tomcat 8 specifically, > independent of JAR scanning, that's the show-stopper for us. Let me know > if I can provide any other info, and thanks again. > > > > On Tue, Feb 16, 2016 at 12:58 PM Mark Thomas <ma...@apache.org> wrote: > >> On 16/02/2016 16:13, Ty wrote: >> > Summary: >> > >> > Because of our use case (hundreds of webapps per instance), we cannot >> > feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to >> > the massive increase in memory the upgrade causes. >> > >> > Background: >> > >> > We’ve been running Tomcat 6 for several years, on perhaps an unusual >> scale: >> > several dozen Tomcat instances, each running up to a *few hundred >> webapps* >> > in a single container. (Picture a cloud service provider with several >> > products and hundreds of customers, with one dedicated webapp >> > per-customer-per-product, resulting in many thousands of web >> > applications. It’s >> > not ideal but it’s how the software was designed). Typical max heap >> sizes >> > per container are 1GB to 4GB depending on utilization, and max container >> > startup times are in the neighborhood of 5 to 10 minutes. >> > >> > · When Tomcat 7 was released, our testing showed a 10-to-20-fold >> > increase in memory consumption and a 3-to-5-fold increase in startup >> time >> > for our test case (100 deployed webapps). After some digging we >> determined >> > that these increases were related to JAR scanning >> > (tomcat.util.scan.DefaultJarScanner). >> >> JAR scanning should be transient. What was it that was causing the 10-20 >> fold increase? >> >> > · When Tomcat 8 was released, we noted that it included a >> > “whitelist” version of the jarsToSkip Property, jarsToScan. Our hopes >> were >> > high that we could skip from Tomcat 6 to Tomcat 8, but our test case >> (again >> > 100 deployed webapps) quickly deflated those hopes: similar startup >> times >> > but at least a *30-fold increase in memory consumption*, regardless of >> how >> > we configure jarsToScan or jarsToSkip. >> > >> > · A heap dump analysis showed that the Tomcat 8 memory increase >> was >> > largely due to the size/count of >> > org.apache.catalina.webresources.JarResourceSet objects. >> >> Those are only going to be created when a JAR is found that contains >> META-INF/resources. The spec requires Tomcat to expose those as static >> resources and that requires plumbing. Note Tomcat 6 will just ignore >> these because the spec is to old for this feature. >> >> > We attempted to >> > reduce this by setting the Context/Resources@cachingAllowed atttribute >> to >> > “false” and also tried to tune the “cacheMaxSize” and >> “cacheObjectMaxSize” >> > attributes. >> >> Those attributes should have no impact on the number or size of the >> JarResourceSet instances. >> >> > The only effect that came from these changes was a change in >> > the object that took up all the space: instead of >> > org.apache.catalina.webresources.JarResourceSet objects, they are >> > org.apache.catalina.webresources.StandardResourceSet objects, taking up >> > roughly the same amount of space. >> >> Strange. Very strange. >> >> > Question: >> > >> > Is there anything else we can adjust to make Tomcat 8’s memory >> consumption >> > closer to that of Tomcat 6’s, for our use case? >> >> It is hard to see exactly what is going on here. The symptoms you >> describe and the resulting changes with configuration are not consistent >> with how I would expect Tomcat to behave nor can I see how it might >> behave in the way described. >> >> > If not, we are faced with >> > either running Tomcat 6 past its EOL, or possibly maintaining a very >> large >> > jarsToSkip blacklist in Tomcat 7 (until its EOL). It is not >> economically >> > feasible for us to increase the physical memory of our servers by 30x >> so we >> > can run Tomcat 8. >> > >> > Test procedure: >> > >> > >> > - Create a "dummy" webapp (using default Maven archetype) named >> test.war >> > - Add several popular Maven dependencies such that the total size of >> the >> > JARs in WEB-INF/lib is about 30MB >> >> Can you provide the sample project? It makes sense to ensure we are >> working from the same baseline. I wonder if the libraries themselves are >> a factor here. >> >> > - Download the latest versions of Tomcat 6, 7, and 8. >> > - Set Java environment variables for a JMX listener and a 3GB max >> heap >> > - Make 100 copies of test.war (test1.war, test2.war, …, test100.war) >> and >> > drop them in /webapps. >> > - Start the container, note the reported startup time, perform an >> > explicit major GC, and note the heap utilization. >> > - Run each test several times and average the results >> >> That is a nice simple test we should be able to repeat. >> >> > Test results: >> > >> > Case 1 (baseline): >> > >> > >> > >> > +---------+--------------+---------------------------+ >> > | version | startup time | heap usage after major GC | >> > +---------+--------------+---------------------------+ >> > | tomcat6 | 36,711ms | 21,163,288 | >> > | tomcat7 | 104,517ms | 489,992,264 | >> > | tomcat8 | 156,094ms | 1,010,512,568 | >> > +---------+--------------+---------------------------+ >> > >> > >> > Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”) >> > >> > +---------+--------------+---------------------------+ >> > | version | startup time | heap usage after major GC | >> > +---------+--------------+---------------------------+ >> > | tomcat6 | 36,711ms | 21,163,288 | >> > | tomcat7 | 38,979ms | 72,359,840 | >> > | tomcat8 | 52,040ms | 633,682,336 | >> > +---------+--------------+---------------------------+ >> >> From those figures my impression is that we should be able to do >> something about the memory. The start-up time with JAR scanning is what >> it is. Without JAR scanning there might be scope for improvement. >> >> Mark >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
