OPNFV Jenkins Failures Root Cause Analysis: 2017-05-21 When:
Between May 21 2017 17:45 UTC and May 22 2017 14:25 UTC. What: On May 21 2017 17:45 UTC, Jenkins was upgraded to 2.46.2 and all installed plugins were updated to their latest versions. Any job triggered between May 21 2017 17:45 UTC and May 22 2017 14:25 UTC failed with the notice: > *** Please tell me who you are. > > Run > > git config --global user.email "y...@example.com" > git config --global user.name "Your Name" > > to set your account's default identity. > Omit --global to set the identity only in this repository. This error was posted to the Tech-Discuss list at 4:05 UTC, May 22 2017. On May 22 2017 14:25 UTC, the cause of the issue was identified: the global git.user.name and git.user.email were not set in Jenkins. This issue was fixed and the resolution reported to the Tech-Discuss list. Root Cause: OPNFV Jenkins uses Jenkins Job Builder (jjb) 1.6.2. This software generates XML from a YAML configuration of a Jenkins job. For the configuration relating to the Git SCM plugin, jjb outputs the 'skipTag' setting to False if it is not set. This is reflected in the Jenkins UI as 'Create a tag for every build' under the SCM section of a build. Before the upgrade this setting was ignored by the Git SCM plugin (2.3.5) since it was not yet a supported feature. After upgrading the plugin (3.3.0) this setting went into effect, causing the above error on builds. Process Improvements: To ensure we're providing OPNFV with a higher level of support, we are taking the following actions: We are updating the testing process to include running genuine OPNFV builds in the test environment, and are taking care to ensure they do not vote in Gerrit. This will help us proactively detect and fix problems which may be caused by Jenkins or related plugin upgrades before they affect the production systems. Prior testing was focused on changes to the JNLP protocol and to ensure builders reconnected after the update; mock builds were run before and after upgrading. We are also improving our monitoring of errors. We currently have poor visibility into Jenkins build errors and we will be configuring Jenkins to notify us of build failures. This will help us identify any errors that may be missed in the testing process, as well as improve the general stability and reliability of the OPNFV CI systems. _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
