On 16/02/2016 16:13, Ty wrote:
Summary:
Because of our use case (hundreds of webapps per instance), we cannot
feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to
the massive increase in memory the upgrade causes.
Background:
We’ve been running Tomcat 6 for several years, on perhaps an unusual
scale:
several dozen Tomcat instances, each running up to a *few hundred
webapps*
in a single container. (Picture a cloud service provider with several
products and hundreds of customers, with one dedicated webapp
per-customer-per-product, resulting in many thousands of web
applications. It’s
not ideal but it’s how the software was designed). Typical max heap
sizes
per container are 1GB to 4GB depending on utilization, and max container
startup times are in the neighborhood of 5 to 10 minutes.
· When Tomcat 7 was released, our testing showed a 10-to-20-fold
increase in memory consumption and a 3-to-5-fold increase in startup time
for our test case (100 deployed webapps). After some digging we
determined
that these increases were related to JAR scanning
(tomcat.util.scan.DefaultJarScanner).
JAR scanning should be transient. What was it that was causing the 10-20
fold increase?
· When Tomcat 8 was released, we noted that it included a
“whitelist” version of the jarsToSkip Property, jarsToScan. Our hopes
were
high that we could skip from Tomcat 6 to Tomcat 8, but our test case
(again
100 deployed webapps) quickly deflated those hopes: similar startup
times
but at least a *30-fold increase in memory consumption*, regardless of
how
we configure jarsToScan or jarsToSkip.
· A heap dump analysis showed that the Tomcat 8 memory increase
was
largely due to the size/count of
org.apache.catalina.webresources.JarResourceSet objects.
Those are only going to be created when a JAR is found that contains
META-INF/resources. The spec requires Tomcat to expose those as static
resources and that requires plumbing. Note Tomcat 6 will just ignore
these because the spec is to old for this feature.
We attempted to
reduce this by setting the Context/Resources@cachingAllowed atttribute
to
“false” and also tried to tune the “cacheMaxSize” and
“cacheObjectMaxSize”
attributes.
Those attributes should have no impact on the number or size of the
JarResourceSet instances.
The only effect that came from these changes was a change in
the object that took up all the space: instead of
org.apache.catalina.webresources.JarResourceSet objects, they are
org.apache.catalina.webresources.StandardResourceSet objects, taking up
roughly the same amount of space.
Strange. Very strange.
Question:
Is there anything else we can adjust to make Tomcat 8’s memory
consumption
closer to that of Tomcat 6’s, for our use case?
It is hard to see exactly what is going on here. The symptoms you
describe and the resulting changes with configuration are not consistent
with how I would expect Tomcat to behave nor can I see how it might
behave in the way described.
If not, we are faced with
either running Tomcat 6 past its EOL, or possibly maintaining a very
large
jarsToSkip blacklist in Tomcat 7 (until its EOL). It is not economically
feasible for us to increase the physical memory of our servers by 30x so
we
can run Tomcat 8.
Test procedure:
- Create a "dummy" webapp (using default Maven archetype) named
test.war
- Add several popular Maven dependencies such that the total size of
the
JARs in WEB-INF/lib is about 30MB
Can you provide the sample project? It makes sense to ensure we are
working from the same baseline. I wonder if the libraries themselves are
a factor here.
- Download the latest versions of Tomcat 6, 7, and 8.
- Set Java environment variables for a JMX listener and a 3GB max heap
- Make 100 copies of test.war (test1.war, test2.war, …, test100.war)
and
drop them in /webapps.
- Start the container, note the reported startup time, perform an
explicit major GC, and note the heap utilization.
- Run each test several times and average the results
That is a nice simple test we should be able to repeat.
Test results:
Case 1 (baseline):
+---------+--------------+---------------------------+
| version | startup time | heap usage after major GC |
+---------+--------------+---------------------------+
| tomcat6 | 36,711ms | 21,163,288 |
| tomcat7 | 104,517ms | 489,992,264 |
| tomcat8 | 156,094ms | 1,010,512,568 |
+---------+--------------+---------------------------+
Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”)
+---------+--------------+---------------------------+
| version | startup time | heap usage after major GC |
+---------+--------------+---------------------------+
| tomcat6 | 36,711ms | 21,163,288 |
| tomcat7 | 38,979ms | 72,359,840 |
| tomcat8 | 52,040ms | 633,682,336 |
+---------+--------------+---------------------------+
From those figures my impression is that we should be able to do
something about the memory. The start-up time with JAR scanning is what
it is. Without JAR scanning there might be scope for improvement.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
