On 8/7/14, 8:56 AM, "Daniel Mikusa" <dmik...@pivotal.io> wrote:
>On Thu, Aug 7, 2014 at 11:35 AM, Peter Rifel <pri...@mixpo.com> wrote: > >> Dan, >> >> On 8/6/14, 4:52 AM, "Daniel Mikusa" <dmik...@pivotal.io> wrote: >> >> >On Tue, Aug 5, 2014 at 7:54 PM, Peter Rifel <pri...@mixpo.com> wrote: >> > >> >> Hello, >> >> >> >> I am in the process of upgrading from Tomcat 7.0.54 to Tomcat 8.0.9 >>and >> >>am >> >> seeing some performance degradation in my webapp. >> >> >> >> Configuration info: >> >> >> >> Ubuntu 14.04.1 LTS >> >> java version "1.8.0_11" >> >> Java(TM) SE Runtime Environment (build 1.8.0_11-b12) >> >> Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode) >> >> >> >> Both versions of tomcat use the APR Connector with TC Native 1.1.31 >> >> statically linked against APR 1.5.1 and OpenSSL 1.0.1h >> >> >> >> >> >> I'm running identical requests to identical web apps between these >> >>tomcats >> >> and in Tomcat 8 I'm seeing an ~20% response time increase and as >>such a >> >> similar reqs / sec decrease. After using VisualVM to profile the >> >> application I've narrowed it down to the JRE's JAXP library taking >> >> significantly longer to create its xml validation "SchemaFactory" >>which >> >> doesn't make sense to me because I'm using the same JRE in both >>tests. >> >>I >> >> know that the Resources implementation was overhauled in Tomcat 8 so >>is >> >> there any possibility that this could be a side effect? For a better >> >>idea >> >> of what I'm talking about, see the below VisualVM screenshots of a >> >> profiling snapshot's call tree: >> >> >> >> >> >> Tomcat 7 - http://i.imgur.com/pbX5xBG.jpg >> >> Notice how the time spent in the javax.xml.validation.* methods is >><6% >> >> >> >> Tomcat 8 - http://i.imgur.com/Txv20lg.jpg >> >> Notice how the time spent in these methods is significantly higher. >> >> >> >> >> >> Again, I'm confused as to how the performance of the JRE's library >> >>changes >> >> between tomcat versions, can anyone explain this to me or point me in >> >>the >> >> right direction on where to look? Is it as simple as a config >>change or >> >> will it require a java code change? The XmlUtils.validate method you >> >>see >> >> in the screenshot is simply the following: >> >> >> >> javax.xml.validation.SchemaFactory factory = >> >> SchemaFactory.newInstance("http://www.w3.org/2001/XMLSchema";); >> >> >> >> Here are the Tomcat configurations: >> >> >> >> >> >> Tomcat 8 server.xml: http://pastebin.com/Tn3d6bXd >> >> Tomcat 8 context.xml: http://pastebin.com/reuJCGNq >> >> >> >> Tomcat 7 server.xml: http://pastebin.com/Kq6SfFd0 >> >> Tomcat 7 context.xml: http://pastebin.com/vdp0PGPh >> >> >> >> >> >> Does anyone have any idea why this might be performing so differently >> >> between versions of tomcat? Could it be a bug in the JRE? I can try >> >>and >> >> provide test code for repro if it helps. >> >> >> > >> >Are you using the "endorsed" directory with Tomcat 7 (and not with 8) >>by >> >any chance? >> > >> > >> > >> >>http://tomcat.apache.org/tomcat-7.0-doc/class-loader-howto.html#XML_Parse >>r >> >s_and_Java >> > >> >Dan >> >> I double checked, and no I'm not using that or any special class-loading >> in either tomcat 7 or 8. For reference, here is my CATALINA_OPTS for >> both: (The Xverify:none at the end was to fix a problem with the >>profiler) >> >> >>-Djava.library.path=$CATALINA_HOME/lib/apr/osx:$CATALINA_HOME/lib/apr/lin >>ux >> -javaagent:$CATALINA_HOME/bin/classmexer.jar -XX:+PrintGCDetails >> -XX:+PrintGCTimeStamps >> -Xloggc:$CATALINA_HOME/logs/garbage-collection_$(date >> +'%y_%m_%d__%H_%M').log >> -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n >> -Dcom.sun.management.jmxremote.port=64850 >> -Dcom.sun.management.jmxremote.authenticate=false >> -Dcom.sun.management.jmxremote.ssl=false -Djava.net.preferIPv4Stack=true >> -Xms128m -Xmx5120m -XX:MaxPermSize=256m -Xverify:none >> > >You could add the "-verbose" option just to double check. That would show >you where the classes are being loaded from. It would catch an >unintentional change to the class path. > >Not saying that's the issue, but different classes could certainly be a >cause. Verifying that the same versions are being used in both setups >would help to rule out that possibility. Good idea, but both tomcats load their javax.xml.* classes from the same location: [Loaded javax.xml.parsers.SAXParserFactory from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] [Loaded javax.xml.parsers.FactoryFinder from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] [Loaded javax.xml.parsers.SecuritySupport from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] [Loaded javax.xml.parsers.SecuritySupport$2 from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] [Loaded javax.xml.parsers.SecuritySupport$5 from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] [Loaded javax.xml.parsers.FactoryFinder$1 from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] [Loaded javax.xml.parsers.SecuritySupport$1 from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] [Loaded javax.xml.parsers.SAXParser from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] [Loaded javax.xml.parsers.DocumentBuilderFactory from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] [Loaded javax.xml.parsers.DocumentBuilder from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar] > >Also, you mentioned a test case. Do you have a small test case that can >reproduce the problem that you can share? > >Dan I've created the following test case which reproduces the issue: TestingServlet.java - http://pastebin.com/ANYJ3wV5 /WEB-INF/test.xsd - http://pastebin.com/wnHSWtTP I ran a 10 minute jmeter test with 20 threads making GET requests to that servlet and the following response time histograms shows the results: Tomcat 7 - http://i.imgur.com/epM5512.png Tomcat 8 - http://i.imgur.com/bcuDdkQ.png Peter --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
