It ended up not being an IO issue. We use https://www.jenkins.io/projects/jcasc/ and the official Jenkins docker image to deploy it to AWS ECS. We have a startup script that does some cleanup in the mounted jenkins home directory to make sure that updated plugins are installed properly. We noticed some saml*.xml files in the home directory. We updated the startup script to remove these and now the auth flow with Okta works as expected.
On Friday, September 25, 2020 at 9:07:23 AM UTC-4 [email protected] wrote: > as I said your problem is the IO, if you enter en the Instance by ssh and > check the iostats you will see more than 5-10% of your operations waiting > for IO. NFS, EFS, and in general network filesystems works well with > big files but with small files and write concurrence is where the problems > start > > El vie., 25 sept. 2020 a las 15:02, Mark Schroering (<[email protected]>) > escribió: > >> We are using AWS EFS for the Jenkins Home mount. It was configured for >> burst throughput, and after reading >> https://aws.amazon.com/blogs/storage/best-practices-for-using-amazon-efs-for-container-storage/ >> >> we just changed it to provisioned throughput of 150 MiB/s. The change did >> not help with the slow login times. We are still digging through the logs, >> but are not sure what is causing the big time gaps. >> >> >> >> On Thursday, September 24, 2020 at 7:05:59 PM UTC-4 [email protected] >> wrote: >> >>> Is your Jenkins home in a NFS or other network storage? I think so for >>> the mount point, when a user enter a few files are written, because your IO >>> is slow the IO operations are blocked waiting to finish that make the login >>> slower than expected. You probably has more performance issues, I usually >>> recommend to not use NFS file systems for the Jenkins home, take a look to >>> this KB >>> https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide >>> >>> El jueves, 24 de septiembre de 2020 a las 15:52:05 UTC+2, >>> [email protected] escribió: >>> >>>> Here are the logs in a better format. >>>> >>>> Sep 24, 2020 7:52:17 AM >>>> FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding >>>> attribute value mark.schroering@*****.com for attribute null >>>> Sep 24, 2020 7:52:17 AM >>>> FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>>> key: email / value: [mark.schroering@*****.com] / class >>>> java.util.ArrayList >>>> Sep 24, 2020 7:52:17 AM >>>> FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>>> key: notBefore / value: 2020-09-24T11:46:38.907Z / class >>>> org.joda.time.DateTime >>>> Sep 24, 2020 7:52:17 AM >>>> FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>>> key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class >>>> org.joda.time.DateTime >>>> Sep 24, 2020 7:52:17 AM >>>> FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL >>>> Sep 24, 2020 7:53:35 AM >>>> FINE >>>> org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin >>>> >>>> called. Using consumerServiceUrl >>>> https://ci.infra.lifeomic.com/securityRealm/finishLogin >>>> Sep 24, 2020 7:53:35 AM >>>> FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: >>>> / >>>> Sep 24, 2020 7:53:35 AM >>>> FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL >>>> Sep 24, 2020 7:53:45 AM >>>> FINE >>>> org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin >>>> >>>> called. Using consumerServiceUrl >>>> https://ci.infra.lifeomic.com/securityRealm/finishLogin >>>> Sep 24, 2020 7:53:45 AM >>>> FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: >>>> / >>>> Sep 24, 2020 7:53:45 AM >>>> FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL >>>> Sep 24, 2020 7:54:13 AM >>>> INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using >>>> SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin >>>> Sep 24, 2020 7:54:13 AM >>>> INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver >>>> resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml >>>> Sep 24, 2020 7:54:13 AM >>>> INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver >>>> resolveAttempting to create directory structure for /mnt/jenkins_home >>>> >>>> On Thursday, September 24, 2020 at 9:37:54 AM UTC-4 Mark Schroering >>>> wrote: >>>> >>>>> We have noticed it taking a very long time (up to 60s) to complete the >>>>> SAML auth flow. Here are some logs showing the bigger time gaps. We are >>>>> on version 1.1.7 of the SAML plugin and running Jenkins version 2.257. >>>>> >>>>> >>>>> Sep 24, 2020 7:52:17 AM FINE org.pac4j.saml.client.SAML2Client >>>>> retrieveUserProfileAdding attribute value mark.schroering@*****.com for >>>>> attribute null Sep 24, 2020 7:52:17 >>>>> AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>>>> key: email / value: [mark.schroering@*****.com] / class >>>>> java.util.ArrayList >>>>> Sep 24, 2020 7:52:17 AM FINE org.pac4j.core.profile.UserProfile >>>>> addAttributeno conversion => key: notBefore / value: >>>>> 2020-09-24T11:46:38.907Z / class org.joda.time.DateTime Sep 24, 2020 >>>>> 7:52:17 AM FINE org.pac4j.core.profile.UserProfile addAttributeno >>>>> conversion => key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class >>>>> org.joda.time.DateTime Sep 24, 2020 7:52:17 >>>>> AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL Sep 24, >>>>> 2020 >>>>> 7:53:35 >>>>> AM FINE >>>>> org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin >>>>> >>>>> called. Using consumerServiceUrl >>>>> https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 >>>>> 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL >>>>> redirection: / Sep 24, 2020 7:53:35 >>>>> AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, >>>>> 2020 >>>>> 7:53:45 >>>>> AM FINE >>>>> org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin >>>>> >>>>> called. Using consumerServiceUrl >>>>> https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 >>>>> 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL >>>>> redirection: / Sep 24, 2020 7:53:45 >>>>> AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, >>>>> 2020 >>>>> 7:54:13 >>>>> AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver >>>>> Using >>>>> SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin >>>>> Sep 24, 2020 7:54:13 >>>>> AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver >>>>> resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml Sep >>>>> 24, 2020 7:54:13 >>>>> AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver >>>>> resolveAttempting to create directory structure for /mnt/jenkins_home >>>>> >>>>> Looking at the browser tools on page load: >>>>> >>>>> GET /securityRealm/commenceLogin <-- 57s >>>>> GET /securityRealm/finishLogin <--- 38s >>>>> >>>>> the Okta SSO parts in between seem to be quick as expected. >>>>> >>>>> Any tips on how to further debug or troubleshoot would be appreciated. >>>>> >>>>> Thanks for the help. >>>>> >>>>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Jenkins Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jenkinsci-users/WgAwcT0OGvk/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/1caf58e5-020d-44ab-a682-00a974054fbcn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/jenkinsci-users/1caf58e5-020d-44ab-a682-00a974054fbcn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Un Saludo > Iván Fernández Calvo > https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033 > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/39933fc2-07b4-46bc-96e7-df335091ebbfn%40googlegroups.com.
