Hi, We are running CAS 6.3 with tomcat 9 and Java 11, and have SAML2 and oauth dependencies installed with hazelcast as ticket registry and json files for service registry. We have noticed that after a few days of running the CPU usage for tomcat spikes to above 100% and requires a restart for it to come back down. When we check the load on the server there isn't to many authentications happening. We had a similar tomcat configuration when running CAS 5.3 with tomcat 8.5 and didn't really see this issue. What we notice in the CAS logs is the metadata being loaded on several occasions over 10 times for a single authentication. [org.apereo.cas.support.saml.services.idp.metadata.SamlRegisteredServiceServiceProviderMetadataFacade] - <Resolved metadata chain from [/etc/cas/saml/sp_metadata
In our tomcat configuration we are using nio connector together with the opensslimplementation, we have also tried switching to nio2 but we see similar behavior. We have even bumped up the CPU on the server but still encounter the same problem. Doing various thread dumps the issue doesn't seem to be with GC, or hazelcast. Below is a snippet of one of the threads that was showing up with high cpu during a thread dump. "https-openssl-nio-443-exec-63" #340 daemon prio=5 os_prio=0 cpu=783837.57ms elapsed=335561.54s tid=0x00007f44bc289800 nid=0x7c5f runnable [0x00007f4468f5d000] java.lang.Thread.State: RUNNABLE at java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames([email protected]/Native Method) at java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames([email protected]/StackStreamFactory.java:386) at java.lang.StackStreamFactory$AbstractStackWalker.getNextBatch([email protected]/StackStreamFactory.java:322) at java.lang.StackStreamFactory$AbstractStackWalker.peekFrame([email protected]/StackStreamFactory.java:263) at java.lang.StackStreamFactory$AbstractStackWalker.hasNext([email protected]/StackStreamFactory.java:351) at java.lang.StackStreamFactory$StackFrameTraverser.tryAdvance([email protected]/StackStreamFactory.java:593) at java.util.stream.ReferencePipeline.forEachWithCancel([email protected]/ReferencePipeline.java:127) at java.util.stream.AbstractPipeline.copyIntoWithCancel([email protected]/AbstractPipeline.java:502) at java.util.stream.AbstractPipeline.copyInto([email protected]/AbstractPipeline.java:488) at java.util.stream.AbstractPipeline.wrapAndCopyInto([email protected]/AbstractPipeline.java:474) at java.util.stream.FindOps$FindOp.evaluateSequential([email protected]/FindOps.java:150) at java.util.stream.AbstractPipeline.evaluate([email protected]/AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.findFirst([email protected]/ReferencePipeline.java:543) at org.apache.logging.log4j.util.StackLocator.lambda$getCallerClass$6(StackLocator.java:55) at org.apache.logging.log4j.util.StackLocator$$Lambda$163/0x0000000800376840.apply(Unknown Source) at java.lang.StackStreamFactory$StackFrameTraverser.consumeFrames([email protected]/StackStreamFactory.java:534) at java.lang.StackStreamFactory$AbstractStackWalker.doStackWalk([email protected]/StackStreamFactory.java:306) at java.lang.StackStreamFactory$AbstractStackWalker.callStackWalk([email protected]/Native Method) at java.lang.StackStreamFactory$AbstractStackWalker.beginStackWalk([email protected]/StackStreamFactory.java:370) at java.lang.StackStreamFactory$AbstractStackWalker.walk([email protected]/StackStreamFactory.java:243) at java.lang.StackWalker.walk([email protected]/StackWalker.java:498) at org.apache.logging.log4j.util.StackLocator.getCallerClass(StackLocator.java:54) at org.apache.logging.log4j.util.StackLocatorUtil.getCallerClass(StackLocatorUtil.java:67) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:51) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:354) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:379) at org.opensaml.core.xml.AbstractXMLObject.<init>(AbstractXMLObject.java:48) at org.opensaml.saml.saml2.metadata.impl.EndpointImpl.<init>(EndpointImpl.java:59) at org.opensaml.saml.saml2.metadata.impl.SingleSignOnServiceImpl.<init>(SingleSignOnServiceImpl.java:40) at org.opensaml.saml.saml2.metadata.impl.SingleSignOnServiceBuilder.buildObject(SingleSignOnServiceBuilder.java:49) at org.opensaml.saml.saml2.metadata.impl.SingleSignOnServiceBuilder.buildObject(SingleSignOnServiceBuilder.java:31) at org.opensaml.core.xml.AbstractXMLObjectBuilder.buildObject(AbstractXMLObjectBuilder.java:58) at org.opensaml.core.xml.AbstractXMLObjectBuilder.buildObject(AbstractXMLObjectBuilder.java:73) at org.opensaml.core.xml.io.AbstractXMLObjectUnmarshaller.buildXMLObject(AbstractXMLObjectUnmarshaller.java:182) at org.opensaml.core.xml.io.AbstractXMLObjectUnmarshaller.unmarshall(AbstractXMLObjectUnmarshaller.java:104) at org.opensaml.core.xml.io.AbstractXMLObjectUnmarshaller.unmarshallChildElement(AbstractXMLObjectUnmarshaller.java:337) at org.opensaml.core.xml.io.AbstractXMLObjectUnmarshaller.unmarshall(AbstractXMLObjectUnmarshaller.java:128) at org.opensaml.core.xml.io.AbstractXMLObjectUnmarshaller.unmarshallChildElement(AbstractXMLObjectUnmarshaller.java:337) at org.opensaml.core.xml.io.AbstractXMLObjectUnmarshaller.unmarshall(AbstractXMLObjectUnmarshaller.java:128) at org.opensaml.saml.metadata.resolver.impl.DOMMetadataResolver.initMetadataResolver(DOMMetadataResolver.java:68) at org.apereo.cas.support.saml.idp.metadata.locator.SamlIdPMetadataResolver.initMetadataResolver(SamlIdPMetadataResolver.java:64) at org.opensaml.saml.metadata.resolver.impl.AbstractMetadataResolver.doInitialize(AbstractMetadataResolver.java:289) at net.shibboleth.utilities.java.support.component.AbstractInitializableComponent.initialize(AbstractInitializableComponent.java:65) - locked <0x00000005850f62c0> (a org.apereo.cas.support.saml.idp.metadata.locator.SamlIdPMetadataResolver) .. Has anyone noticed something similar? Thanks! ___________________ Juan Quintanilla -- - Website: https://apereo.github.io/cas - Gitter Chatroom: https://gitter.im/apereo/cas - List Guidelines: https://goo.gl/1VRrw7 - Contributions: https://goo.gl/mh7qDG --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/BN6PR05MB3474E121288FEA5212432C27861A9%40BN6PR05MB3474.namprd05.prod.outlook.com.
