The 8.x ref guide builds are running on a node they've never run on before: "websites2". The master ref guide builds are running on the "websites1" node, which has not exhibited the same build problems. Both nodes are pinned to label "git-websites", which can result in builds on either the "websites1" node or the "websites2" node. AFAICT there's some kind of routing based on the identity (name maybe?) of the job, which results in effectively permanent assignment to either one or the other node.
I put the key download commands from the error message into the 8.x job config and triggered a run. This enabled verification. But there is now another problem related to RVM's installation of Ruby 2.3.3: compilation is failing, with details in a log file on the "websites2" machine, to which I don't have access. I'll contact Infra about it. In the meantime, I pinned the 8.x build to "websites1" and triggered a build, which succeeded. I left in the key download commands, so leaving them in permanently apparently wouldn't cause any trouble. (I'll make an issue to modify our build script to include a check for the keys and download them if they don't exist.) -- Steve > On Jun 27, 2019, at 9:22 AM, Cassandra Targett <[email protected]> wrote: > > Thanks Steve, that makes sense. Let me know if there’s anything I can do to > help. > > If possible, maybe the script could check for the keys and import them if not > available? At the very least, maybe I should add something about this > scenario to internal Ref Guide docs. I totally forgot this had happened > before. > > Sorry, I didn’t mean to imply you needed to disable the 8.1 job - I forgot to > mention I recently gave myself permissions to do that in Jenkins - but thank > you for doing it. > > Cassandra > On Jun 27, 2019, 8:12 AM -0500, Steve Rowe <[email protected]>, wrote: >> This has happened previously. Usually it's a new machine that has never >> built the ref guide previously. IIRC there was an Infra notice a couple >> days ago about one of the 2 machines used to build the ref guide ("website1" >> OSLT) being taken offline. Probably they put in place a replacement for it. >> >> In the past I've modified the job config to do what's suggested in the error >> message: download the key to the new machine, trigger a run, then comment >> out the key download in the job config. >> >> I suppose we could always download the key on every run? >> >> I'll work on it later today. >> >> I'll go disable the 8.1 ref guide job now. >> >> -- >> Steve >> >>> On Jun 27, 2019, at 8:02 AM, Cassandra Targett <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> This seems to be a persistent problem. It’s complaining about the keys for >>> checking the RVM download. Right now it’s the 8.x job that’s failing (and >>> 8.1, which needs to be disabled), but the master branch build appears to be >>> fine. >>> >>> The jenkins.build.ref.guide.sh does the build, and I note in the beginning >>> of the script that it seems to assume that the RVM package keys are already >>> imported on the server (see >>> https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/jenkins.build.ref.guide.sh#L13 >>> >>> <https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/jenkins.build.ref.guide.sh#L13>). >>> >>> Steve, you wrote this script and set up the overall build processes, do you >>> think the problem here might be that some changes have been made on the >>> Jenkins servers that removed those keys? The work to set this up was done >>> in https://issues.apache.org/jira/browse/SOLR-10568 >>> <https://issues.apache.org/jira/browse/SOLR-10568>, but it doesn’t mention >>> the keys or how they got imported onto those machines, so that part isn’t >>> clear to me. >>> >>> Thanks, >>> >>> Cassandra >>> On Jun 27, 2019, 4:52 AM -0500, Apache Jenkins Server >>> <[email protected] <mailto:[email protected]>>, wrote: >>>> Build: https://builds.apache.org/job/Solr-reference-guide-8.x/3944/ >>>> <https://builds.apache.org/job/Solr-reference-guide-8.x/3944/> >>>> >>>> Log: >>>> Started by timer >>>> [EnvInject] - Loading node environment variables. >>>> Building remotely on websites2 (git-websites svn-websites) in workspace >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-8.x >>>> No credentials specified >>>>> git rev-parse --is-inside-work-tree # timeout=10 >>>> Fetching changes from the remote Git repository >>>>> git config remote.origin.url >>>>> https://gitbox.apache.org/repos/asf/lucene-solr.git >>>>> <https://gitbox.apache.org/repos/asf/lucene-solr.git> # timeout=10 >>>> Cleaning workspace >>>>> git rev-parse --verify HEAD # timeout=10 >>>> Resetting working tree >>>>> git reset --hard # timeout=10 >>>>> git clean -fdx # timeout=10 >>>> Fetching upstream changes from >>>> https://gitbox.apache.org/repos/asf/lucene-solr.git >>>> <https://gitbox.apache.org/repos/asf/lucene-solr.git> >>>>> git --version # timeout=10 >>>>> git fetch --tags --progress >>>>> https://gitbox.apache.org/repos/asf/lucene-solr.git >>>>> <https://gitbox.apache.org/repos/asf/lucene-solr.git> >>>>> +refs/heads/*:refs/remotes/origin/* >>>>> git rev-parse refs/remotes/origin/branch_8x^{commit} # timeout=10 >>>>> git rev-parse refs/remotes/origin/origin/branch_8x^{commit} # timeout=10 >>>> Checking out Revision 4e58515b0c6aa297be2416c5e5d70e9eda4fb1db >>>> (refs/remotes/origin/branch_8x) >>>>> git config core.sparsecheckout # timeout=10 >>>>> git checkout -f 4e58515b0c6aa297be2416c5e5d70e9eda4fb1db >>>> Commit message: "LUCENE-8886: Fix TestMutablePointsReaderUtils tests" >>>>> git rev-list --no-walk cf443ad9f756407fa8db5ad5bfd39c26367acbc9 # >>>>> timeout=10 >>>> No emails were triggered. >>>> [Solr-reference-guide-8.x] $ /bin/bash -xe >>>> /tmp/jenkins9997498791593161379.sh >>>> + bash dev-tools/scripts/jenkins.build.ref.guide.sh >>>> + set -e >>>> + RVM_PATH=/home/jenkins/.rvm >>>> + RUBY_VERSION=ruby-2.3.3 >>>> + GEMSET=solr-refguide-gemset >>>> + curl -sSL https://get.rvm.io <https://get.rvm.io/> >>>> + bash -s -- --ignore-dotfiles stable >>>> Turning on ignore dotfiles mode. >>>> Downloading https://github.com/rvm/rvm/archive/1.29.8.tar.gz >>>> <https://github.com/rvm/rvm/archive/1.29.8.tar.gz> >>>> Downloading >>>> https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc >>>> <https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc> >>>> gpg: Signature made Wed 08 May 2019 02:14:49 PM UTC >>>> gpg: using RSA key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB >>>> gpg: Can't check signature: No public key >>>> GPG signature verification failed for >>>> '/home/jenkins/.rvm/archives/rvm-1.29.8.tgz' - >>>> 'https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc >>>> <https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc>'! >>>> Try to install GPG v2 and then fetch the public key: >>>> >>>> gpg --keyserver hkp://pool.sks-keyservers.net >>>> <hkp://pool.sks-keyservers.net> --recv-keys >>>> 409B6B1796C275462A1703113804BB82D39DC0E3 >>>> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB >>>> >>>> or if it fails: >>>> >>>> command curl -sSL https://rvm.io/mpapis.asc <https://rvm.io/mpapis.asc> | >>>> gpg --import - >>>> command curl -sSL https://rvm.io/pkuczynski.asc >>>> <https://rvm.io/pkuczynski.asc> | gpg --import - >>>> >>>> In case of further problems with validation please refer to >>>> https://rvm.io/rvm/security <https://rvm.io/rvm/security> >>>> >>>> Build step 'Execute shell' marked build as failure >>>> Archiving artifacts >>>> Publishing Javadoc >>>> Email was triggered for: Failure - Any >>>> Sending email for trigger: Failure - Any >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> <mailto:[email protected]> >>>> For additional commands, e-mail: [email protected] >>>> <mailto:[email protected]>
