Hi, As mentioned in the thread "Jenkins and transient failures", we've had an unusually high number of transient failures in Jenkins lately. We've done several things in response to that:
1) Monty identified a problem with our pypi mirror which was the cause of many of the errors, and corrected it. 2) Monty is continuing to work on the single dependency list which should allow us to switch to using our local pypi mirror exclusively, further reducing transient network errors, as well as significantly speeding up test run time. 3) Several transient errors were caused by failed fetches from Gerrit. While consulting with the Gerrit authors about tuning, they discovered a bug in Gerrit where a 5 minute timeout was being interpreted as a 5 millisecond timeout. I have updated our gerrit configuration to work around that. 4) Clark Boylan implemented automatic retrying for the git fetches that we use with Jenkins. I hope that we'll get to the point where we have almost no transient network errors when testing, but we know it will never be perfect, so at the CI meeting we discussed how best to implement retriggering with Zuul. Clark added a comment filter that will retrigger Jenkins if you leave a comment that matches a regex. We currently run two kinds of jobs in Jenkins, the check job and the gate job. The check jobs run immediately when a patchset is uploaded and vote +/-1. The gate jobs run on approval, queue up across all projects and vote +/-2 (if they fail, jobs behind them in the gate pipeline may need to run again). To retrigger the initial Jenkins check job, just leave a comment on the review in Gerrit with only the text "recheck". To retrigger the Jenkins merge gate job, leave a comment with only the text "reverify", or if you are a core reviewer, just leave another "Approved" vote. (Don't leave a "reverify" comment if the change hasn't been approved yet, it still won't be merged and will slow Jenkins down.) -Jim _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
