Thanh, I'm not quite sure the logic of having it at that particular point either. Something to investigate.
Ed On Thu, Jan 19, 2017 at 8:44 PM, Thanh Ha <thanh...@linuxfoundation.org> wrote: > FWIW in OpenDaylight we don't typically run yum update or apt-get update > in our init-scripts on VM spinup. At the job level we only install > dependencies needed by the build. I'm not sure why fd.io is running > upgrades but it was existing in the script when I looked at it. System > upgrades during VM spinup is not something the OpenDaylight project does at > least. > > Regards, > Thanh > > > On Thu, Jan 19, 2017 at 10:38 PM, Dave Wallace <dwallac...@gmail.com> > wrote: > >> Ed, Thanh, Vanessa, >> >> IMHO, updating the ubuntu packages every time a VM is spun up is a bug >> wrt. being able to reproduce some (hopefully rare) build/test issues. >> Since every VM is potentially running with different versions of OS >> components, when a failure occurs (e.g. in "make test"), then it may be >> necessary to recreate the exact run-time environment in order to reproduce >> the failure. Unless the complete package list is being archived for every >> VM instance that is spun up, this may not be possible. >> >> My experience is that those rare cases where a tool or environment issue >> causes a failure, the cost to find the issue is extraordinarily high if you >> do not have the ability to recreate the EXACT build/run-time environment. >> This >> is why CSIT does not update OS components in the VM initialization scripts >> and the VM images are built from a specific package list instead of pulling >> the latest versions from the apt repositories. >> >> My recommendation is that the VM images be updated periodically (weekly >> or whenever a new security update is released) and the package lists >> archived for each VM image version. Each VM image should also be verified >> against a known good VPP commit version as is done with CSIT branches. >> Ideally we should build a fully automated continuous deployment model to >> reduce the amount of work to update the VM images to running a Jenkins job >> to build/test/deploy a new VM image from the latest packages versions. >> >> With that automation in place, this mechanism could be extended for use >> by CSIT as well as "make test", thus ensuring that all of our testing was >> done with the same OS component version. Ideally, all projects should be >> using the same OS components to ensure that everything is tested in the >> same run-time environment. >> >> Thanks, >> -daw- >> >> On 1/19/2017 8:31 PM, Thanh Ha via RT wrote: >> >> The issue with the 16.04 Ubuntu image is fixed now (but we may require some >> additional actions which I'll send to Vanessa to in case this issue comes up >> again). We fixed this issue tonight by rebuilding ubuntu1604 and deploying >> the new image. >> >> I'm going to close this ticket as resolved and we'll take the additional >> task to find a way to ensure this doesn't appear again off of this ticket. >> >> If you're not interested in the detailed analysis you can stop reading now. >> >> For those interested I suspect that the lock issue will appear again >> (although I could be wrong). The reason I believe so is that our vm init >> script runs "apt-get update" as an initialization step when the VM boots up >> at creation time via this script [0]. Ed mentioned that we didn't see this >> in the past and it only started appear again recently as we deployed another >> patch to disable Ubuntu's unattended updates. >> >> I believe a possible reason we will see this issue appear again due to [0] >> is because of we switched from using JClouds to OpenStack Jenkins plugins >> for node spinnup and there's difference in how the init-script is executed >> depending on which plugin is being used. >> >> JClouds Plugin: >> >> 1) boot vm >> 2) wait for ssh access >> 3) copies init-script into vm via ssh >> 4) executes init-script, and doesn't continue processing until script is >> complete >> 5) once init-script is complete, passes vm over to job and job starts >> >> OpenStack Plugin: >> >> 1) boot vm and passes init-script in as User Data >> 2) init-script runs inside vm without Jenkins intervention, thus is a >> non-blocking function >> 3) in parallel jenkins waits for ssh access to vm >> 4) ssh's into vm and passes vm over to job and job starts running >> >> In the OpenStack plugin case step 4 can execute while step 2 is still >> running apt-get update in the background because it was a non-blocking >> function. >> >> A few ideas I have to get around this. >> >> a) Allow init-script to continue running apt-get update however have a shell >> script at the start of Ubuntu jobs that waits for the lock to get released >> before allowing the job to start >> >> b) Remove apt-get update from init-script and make the job run apt-get >> update at the beginning of it's execution >> >> c) Regularly update VMs to ensure that apt-get update always runs quickly >> >> Regards, >> Thanh >> >> [0] >> https://git.fd.io/ci-management/tree/jenkins-scripts/basic_settings.sh#n14 >> >> >> On Thu Jan 19 19:23:59 2017, hagbard wrote: >> >> FYI... helpdesk is on it, and its being worked in #fdio-infra on IRC >> >> Ed >> >> On Thu, Jan 19, 2017 at 4:31 PM, Ed Warnicke <hagb...@gmail.com> >> <hagb...@gmail.com> wrote: >> >> >> Looping in help desk. >> On Thu, Jan 19, 2017 at 4:16 PM Dave Barach (dbarach) <dbar...@cisco.com> >> <dbar...@cisco.com> >> wrote: >> >> >> Folks, >> >> >> >> See https://jenkins.fd.io/job/vpp-verify-master-ubuntu1604/3378/console >> >> >> >> 11:00:46 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource >> temporarily unavailable) >> >> 11:00:46 E: Unable to lock the administration directory (/var/lib/dpkg/), >> is another process using it? >> >> >> >> I recognize this failure from my own Ubuntu 16.04 system: a cron-job >> starts “apt-get -q”, which for whatever reason does not terminate. As a >> workaround, “sudo killall apt-get || true” before trying to acquire build >> dependencies... >> >> >> >> HTH... Dave >> >> >> _______________________________________________ >> >> vpp-dev mailing list >> vpp-dev@lists.fd.io >> https://lists.fd.io/mailman/listinfo/vpp-dev >> >> _______________________________________________ >> vpp-dev mailing >> listvpp-...@lists.fd.iohttps://lists.fd.io/mailman/listinfo/vpp-dev >> >> >> >
_______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
