And the master job finished successfully (I checked and made sure it ran on the new server, and it did), so now I’ve backported the script change to branch_8x and updated the 8.x Ref Guide job to use the tag “git-websites” again. It just also finished successfully.
Thanks again Steve for your earlier help - I’ve let Infra know we’re all set in INFRA-18666. Cassandra On Jul 1, 2019, 10:27 AM -0500, Cassandra Targett <[email protected]>, wrote: > I pushed a change to master and updated the Ref Guide master build to use > “git-websites” again. I’ll monitor the next time it runs (~30 minutes from > now) to see what happens. If it works, I’ll backport the change to branch_8x. > > Cassandra > On Jun 30, 2019, 3:13 PM -0500, Erick Erickson <[email protected]>, > wrote: > > AFAIC, just push it. Even if you do find a way to test it, we’d still > > potentially have something _else_ different between your test bed and > > websites2….. > > > > > On Jun 30, 2019, at 12:46 PM, Cassandra Targett <[email protected]> > > > wrote: > > > > > > Thanks for filing those issues, Steve. > > > > > > The master branch Ref Guide build started failing this weekend with the > > > Ruby installation problem, so I pinned that build to “websites1” for now. > > > > > > I saw the comments in the INFRA issue about using a newer version of > > > Ruby, but wasn’t really sure how to test installing Ruby 2.5.1 instead. > > > Apparently Gavin tried it and it worked all right after he initialized > > > with the ant ivy-bootstrap target. Should I just commit the version > > > change to the script, or try to test it some more first (somehow)? > > > > > > Cassandra > > > On Jun 27, 2019, 10:41 AM -0500, Steve Rowe <[email protected]>, wrote: > > > > INFRA JIRA for "websites2" problem: > > > > https://issues.apache.org/jira/browse/INFRA-18666 > > > > > > > > JIRA to add GPG key download to the ref guide build script: > > > > https://issues.apache.org/jira/browse/SOLR-13582 > > > > > > > > -- > > > > Steve > > > > > > > > > On Jun 27, 2019, at 10:50 AM, Steve Rowe <[email protected]> wrote: > > > > > > > > > > The 8.x ref guide builds are running on a node they've never run on > > > > > before: "websites2". The master ref guide builds are running on the > > > > > "websites1" node, which has not exhibited the same build problems. > > > > > Both nodes are pinned to label "git-websites", which can result in > > > > > builds on either the "websites1" node or the "websites2" node. AFAICT > > > > > there's some kind of routing based on the identity (name maybe?) of > > > > > the job, which results in effectively permanent assignment to either > > > > > one or the other node. > > > > > > > > > > I put the key download commands from the error message into the 8.x > > > > > job config and triggered a run. This enabled verification. But there > > > > > is now another problem related to RVM's installation of Ruby 2.3.3: > > > > > compilation is failing, with details in a log file on the "websites2" > > > > > machine, to which I don't have access. I'll contact Infra about it. > > > > > > > > > > In the meantime, I pinned the 8.x build to "websites1" and triggered > > > > > a build, which succeeded. I left in the key download commands, so > > > > > leaving them in permanently apparently wouldn't cause any trouble. > > > > > (I'll make an issue to modify our build script to include a check for > > > > > the keys and download them if they don't exist.) > > > > > > > > > > -- > > > > > Steve > > > > > > > > > > > On Jun 27, 2019, at 9:22 AM, Cassandra Targett > > > > > > <[email protected]> wrote: > > > > > > > > > > > > Thanks Steve, that makes sense. Let me know if there’s anything I > > > > > > can do to help. > > > > > > > > > > > > If possible, maybe the script could check for the keys and import > > > > > > them if not available? At the very least, maybe I should add > > > > > > something about this scenario to internal Ref Guide docs. I totally > > > > > > forgot this had happened before. > > > > > > > > > > > > Sorry, I didn’t mean to imply you needed to disable the 8.1 job - I > > > > > > forgot to mention I recently gave myself permissions to do that in > > > > > > Jenkins - but thank you for doing it. > > > > > > > > > > > > Cassandra > > > > > > On Jun 27, 2019, 8:12 AM -0500, Steve Rowe <[email protected]>, > > > > > > wrote: > > > > > > > This has happened previously. Usually it's a new machine that has > > > > > > > never built the ref guide previously. IIRC there was an Infra > > > > > > > notice a couple days ago about one of the 2 machines used to > > > > > > > build the ref guide ("website1" OSLT) being taken offline. > > > > > > > Probably they put in place a replacement for it. > > > > > > > > > > > > > > In the past I've modified the job config to do what's suggested > > > > > > > in the error message: download the key to the new machine, > > > > > > > trigger a run, then comment out the key download in the job > > > > > > > config. > > > > > > > > > > > > > > I suppose we could always download the key on every run? > > > > > > > > > > > > > > I'll work on it later today. > > > > > > > > > > > > > > I'll go disable the 8.1 ref guide job now. > > > > > > > > > > > > > > -- > > > > > > > Steve > > > > > > > > > > > > > > > On Jun 27, 2019, at 8:02 AM, Cassandra Targett > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > This seems to be a persistent problem. It’s complaining about > > > > > > > > the keys for checking the RVM download. Right now it’s the 8.x > > > > > > > > job that’s failing (and 8.1, which needs to be disabled), but > > > > > > > > the master branch build appears to be fine. > > > > > > > > > > > > > > > > The jenkins.build.ref.guide.sh does the build, and I note in > > > > > > > > the beginning of the script that it seems to assume that the > > > > > > > > RVM package keys are already imported on the server (see > > > > > > > > https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/jenkins.build.ref.guide.sh#L13). > > > > > > > > > > > > > > > > Steve, you wrote this script and set up the overall build > > > > > > > > processes, do you think the problem here might be that some > > > > > > > > changes have been made on the Jenkins servers that removed > > > > > > > > those keys? The work to set this up was done in > > > > > > > > https://issues.apache.org/jira/browse/SOLR-10568, but it > > > > > > > > doesn’t mention the keys or how they got imported onto those > > > > > > > > machines, so that part isn’t clear to me. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Cassandra > > > > > > > > On Jun 27, 2019, 4:52 AM -0500, Apache Jenkins Server > > > > > > > > <[email protected]>, wrote: > > > > > > > > > Build: > > > > > > > > > https://builds.apache.org/job/Solr-reference-guide-8.x/3944/ > > > > > > > > > > > > > > > > > > Log: > > > > > > > > > Started by timer > > > > > > > > > [EnvInject] - Loading node environment variables. > > > > > > > > > Building remotely on websites2 (git-websites svn-websites) in > > > > > > > > > workspace > > > > > > > > > /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-8.x > > > > > > > > > No credentials specified > > > > > > > > > > git rev-parse --is-inside-work-tree # timeout=10 > > > > > > > > > Fetching changes from the remote Git repository > > > > > > > > > > git config remote.origin.url > > > > > > > > > > https://gitbox.apache.org/repos/asf/lucene-solr.git # > > > > > > > > > > timeout=10 > > > > > > > > > Cleaning workspace > > > > > > > > > > git rev-parse --verify HEAD # timeout=10 > > > > > > > > > Resetting working tree > > > > > > > > > > git reset --hard # timeout=10 > > > > > > > > > > git clean -fdx # timeout=10 > > > > > > > > > Fetching upstream changes from > > > > > > > > > https://gitbox.apache.org/repos/asf/lucene-solr.git > > > > > > > > > > git --version # timeout=10 > > > > > > > > > > git fetch --tags --progress > > > > > > > > > > https://gitbox.apache.org/repos/asf/lucene-solr.git > > > > > > > > > > +refs/heads/*:refs/remotes/origin/* > > > > > > > > > > git rev-parse refs/remotes/origin/branch_8x^{commit} # > > > > > > > > > > timeout=10 > > > > > > > > > > git rev-parse refs/remotes/origin/origin/branch_8x^{commit} > > > > > > > > > > # timeout=10 > > > > > > > > > Checking out Revision > > > > > > > > > 4e58515b0c6aa297be2416c5e5d70e9eda4fb1db > > > > > > > > > (refs/remotes/origin/branch_8x) > > > > > > > > > > git config core.sparsecheckout # timeout=10 > > > > > > > > > > git checkout -f 4e58515b0c6aa297be2416c5e5d70e9eda4fb1db > > > > > > > > > Commit message: "LUCENE-8886: Fix > > > > > > > > > TestMutablePointsReaderUtils tests" > > > > > > > > > > git rev-list --no-walk > > > > > > > > > > cf443ad9f756407fa8db5ad5bfd39c26367acbc9 # timeout=10 > > > > > > > > > No emails were triggered. > > > > > > > > > [Solr-reference-guide-8.x] $ /bin/bash -xe > > > > > > > > > /tmp/jenkins9997498791593161379.sh > > > > > > > > > + bash dev-tools/scripts/jenkins.build.ref.guide.sh > > > > > > > > > + set -e > > > > > > > > > + RVM_PATH=/home/jenkins/.rvm > > > > > > > > > + RUBY_VERSION=ruby-2.3.3 > > > > > > > > > + GEMSET=solr-refguide-gemset > > > > > > > > > + curl -sSL https://get.rvm.io > > > > > > > > > + bash -s -- --ignore-dotfiles stable > > > > > > > > > Turning on ignore dotfiles mode. > > > > > > > > > Downloading https://github.com/rvm/rvm/archive/1.29.8.tar.gz > > > > > > > > > Downloading > > > > > > > > > https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc > > > > > > > > > gpg: Signature made Wed 08 May 2019 02:14:49 PM UTC > > > > > > > > > gpg: using RSA key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB > > > > > > > > > gpg: Can't check signature: No public key > > > > > > > > > GPG signature verification failed for > > > > > > > > > '/home/jenkins/.rvm/archives/rvm-1.29.8.tgz' - > > > > > > > > > 'https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc'! > > > > > > > > > Try to install GPG v2 and then fetch the public key: > > > > > > > > > > > > > > > > > > gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys > > > > > > > > > 409B6B1796C275462A1703113804BB82D39DC0E3 > > > > > > > > > 7D2BAF1CF37B13E2069D6956105BD0E739499BDB > > > > > > > > > > > > > > > > > > or if it fails: > > > > > > > > > > > > > > > > > > command curl -sSL https://rvm.io/mpapis.asc | gpg --import - > > > > > > > > > command curl -sSL https://rvm.io/pkuczynski.asc | gpg > > > > > > > > > --import - > > > > > > > > > > > > > > > > > > In case of further problems with validation please refer to > > > > > > > > > https://rvm.io/rvm/security > > > > > > > > > > > > > > > > > > Build step 'Execute shell' marked build as failure > > > > > > > > > Archiving artifacts > > > > > > > > > Publishing Javadoc > > > > > > > > > Email was triggered for: Failure - Any > > > > > > > > > Sending email for trigger: Failure - Any > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > >
