So PL, these are just the slaves and not jenkins, right? If we can incorporate some monitoring, that would be nice. (no emotional attachements to my hack has formed yet)
On Thu, Oct 1, 2015 at 2:18 PM, Pierre-Luc Dion <pdion...@apache.org> wrote: > Daan: I like your style! > > I was looking at using Chef [1] to managed Jenkins slaves, I've started > deploying it as Proof of concept for some slaves I've recently deployed, > within this scope of idea was to deploy a monitoring tool at some point to > alerts on event such as disk full. > > I was not aware that puppet was already used for automated deployment of > Jenkins. I guest if we are going to managed Jenkins slaves, we all need to > agree on the tool to use with the majority of the community/dev right? > > [1] cookbooks repo: https://github.com/CloudStack-extras/ci-chefrepo > > > > On Wed, Sep 30, 2015 at 6:59 AM, Daan Hoogland <daan.hoogl...@gmail.com> > wrote: > > > So I did, expressing my worst operator practices I created a small > script: > > > > <script> > > ! /bin/bash > > > > THRESHOLD=95 > > USAGE=`/bin/df -P /var/lib/jenkins/workspace | /bin/grep -v Capacity | > > /bin/cut -c 47- | /bin/cut -c 6-7 ` > > > > function maildev() { > > /bin/mail -s 'disk usage at jenkins.bac.o' dev@cloudstack.apache.org > << > > EOT > > `/bin/df -P /var/lib/jenkins/workspace` > > EOT > > } > > > > [[ $USAGE > $THRESHOLD ]] && maildev > > </script> > > > > and added it to cron: > > > > */10 * * * * ( cd /var/lib/jenkins && ./check_workspace.sh ) >/dev/null > > > > There I noticed some traces of puppet usage. Should I implement this in > > puppet somehow? > > > > willing to do some more, > > Daan > > > > On Wed, Sep 30, 2015 at 11:40 AM, David Nalley <da...@gnsa.us> wrote: > > > > > SGTM, Feel free to. > > > > > > > > > > > > On Wed, Sep 30, 2015 at 5:34 AM, Daan Hoogland < > daan.hoogl...@gmail.com> > > > wrote: > > > > H David, I just tried and it works indeed. Forgot about that. Does it > > > make > > > > sense to have a script containing du mail to dev@ if something like > > > > /var/lib/jenkins/workspace or / exceeds a threshold? (yes me > > > volunteering) > > > > > > > > On Wed, Sep 30, 2015 at 10:48 AM, Raja Pullela < > > raja.pull...@citrix.com> > > > > wrote: > > > > > > > >> Btw, edison, pradeep, rayeesn - are not active anymore - talluri may > > be? > > > >> I think. > > > >> > > > >> -----Original Message----- > > > >> From: David Nalley [mailto:da...@gnsa.us] > > > >> Sent: Wednesday, September 30, 2015 2:01 PM > > > >> To: dev@cloudstack.apache.org > > > >> Cc: Pierre-Luc Dion <pdion...@apache.org>; tall...@apache.org > > > >> Subject: Re: jenkins disk full ? > > > >> > > > >> Right now these folks have access: > > > >> > > > >> ewanm:x:1002:1002::/home/ewanm:/bin/bash > > > >> prasanna:x:1003:1003::/home/prasanna:/bin/bash > > > >> salvatore:x:1004:1004::/home/salvatore:/bin/bash > > > >> edison:x:1005:1005::/home/edison:/bin/bash > > > >> pradeep:x:1006:1006::/home/pradeep:/bin/bash > > > >> rayeesn:x:1007:1007::/home/rayeesn:/bin/bash > > > >> hugot:x:1008:1008::/home/hugot:/bin/bash > > > >> talluri:x:1009:1009::/home/talluri:/bin/bash > > > >> dahn:x:1010:1010::/home/dahn:/bin/bash > > > >> > > > >> PL or other folks who want access, send me a ssh pub key off list. > > > >> > > > >> --David > > > >> > > > >> On Wed, Sep 30, 2015 at 4:20 AM, Rajani Karuturi < > > > >> rajani.karut...@citrix.com> wrote: > > > >> > Thanks david. Do you know if anyone else has access to this? if > not, > > > can > > > >> you give access to PL please? > > > >> > > > > >> > ~Rajani > > > >> > > > > >> > > > > >> > > > > >> > On 30-Sep-2015, at 1:13 pm, David Nalley <da...@gnsa.us> wrote: > > > >> > > > > >> >> The below is before I cleaned up a bit. We now have 3.2GB free. > > > >> >> > > > >> >> [root@domU-12-31-39-15-22-2D jobs]# du -h -d 1 1.3G > > > >> >> ./package-rhel63-master 200M > ./cloudstack-4.3-maven-build-noredist > > > >> >> 638M ./package-rhel63-4.5 > > > >> >> 1.1G ./cloudstack-4.4-systemvm64 > > > >> >> 1.3M ./build-4.5 > > > >> >> 8.0K ./cloudstack-master-with-patch > > > >> >> 186M ./build-test-archive-4.4 > > > >> >> 2.9M ./test-smoke-matrix-master > > > >> >> 100K ./Create-Release-Packages > > > >> >> 1.5M ./test-packaging-4.4 > > > >> >> 214M ./build-test-archive-master > > > >> >> 612K ./cloudstack.apt-get.eu-systemvm 88M ./built-test-archive > 4.2M > > > >> >> ./simulator-4.5-singlerun 424K ./test-tabnanny-4.4 2.2G > > > >> >> ./build-master 124M ./cloudstack-4.4-systemvm 331M > > > >> >> ./package-centos7-4.4.4-noredist 804K > ./cloudstack-4.4-maven-build > > > >> >> 3.4M ./cloudstack-4.3-maven-build 645M ./package-centos7-master > 80M > > > >> >> ./test-fetch-marvin-master 180M ./parameterized-slowbuild 508K > > > >> >> ./cloudstack.apt-get.eu-noredis-rhel > > > >> >> 104K ./dockerbuild-master-marvin > > > >> >> 9.1M ./test-regression-matrix-master > > > >> >> 345M ./cloudstack-4.3-package-deb > > > >> >> 3.8G ./parameterized-sytemvm > > > >> >> 26M ./HyperVAgent4.3 > > > >> >> 510M ./cloudstack-4.3-package-rpm > > > >> >> 1.5M ./test-packaging-master > > > >> >> 7.2M ./build-docs-admin-master > > > >> >> 8.3M ./cloudstack-marvin-master > > > >> >> 1013M ./cloudstack-4.3-systemvm > > > >> >> 131M ./run-checkin-tests > > > >> >> 2.5M ./simulator-gate > > > >> >> 145M ./cloudstack-master-twiceweekly-coverity > > > >> >> 3.2M ./cloudstack-4.4-hyperv-agent > > > >> >> 104K ./test-setup-advancedzone-master 1.1G ./master-slowbuild 44M > > > >> >> ./cloudstack-master 301M ./package-centos7-master-noredist 396K > > > >> >> ./test-tabnanny-master 816K ./simulator-build 20K ./test-libcloud > > > >> >> 236K ./test-smoke-matrix-4.3 4.9M ./dockerbuild-4.5-simulator > 331M > > > >> >> ./package-centos7-4.4-noredist 632K > > > >> >> ./cloudstack.apt-get.eu-noredis-deb > > > >> >> 144K ./dockerbuild-master-management_centos6 > > > >> >> 292M ./package-centos6-4.6 > > > >> >> 9.4M ./stop-jetty > > > >> >> 326M ./cloudstack-4.4-package-rpm > > > >> >> 40K ./mgmt-build-reviewboard-requests 20K > > ./build-test-centos-builtin > > > >> >> 76K ./simulator-deploydb 20K ./Master - Python Static Code > Analysis > > > >> >> 53M ./test-smoke-matrix-4.4 442M ./build-master-slowbuild 376K > > > >> >> ./build-docs-niciranvp-master 840K ./test-matrix-4.4 2.7M > > > >> >> ./dockerbuild-master-simulator 21M ./build-docs-install-master > 50M > > > >> >> ./test-fetch-marvin-4.4 789M > > > >> >> ./cloudstack-rpm-packages-with-branch-parameter > > > >> >> 5.3G ./build-systemvm-4.5 > > > >> >> 1.1G ./cloudstack-4.3-systemvm64 > > > >> >> 3.2M ./build-docs-release-notes-master 8.0K > > > >> >> ./cloudstack-master-codecoverage 121M ./fastsimulatorbuild 4.3M > > > >> >> ./cloudstack-4.4-marvin 352M ./build-master-noredist 7.2M > > > >> >> ./test-setup-advancedzone-4.4 3.3M > ./cloudstack-master-hyperv-agent > > > >> >> 1.9G ./build-systemvm64-master 29M ./start-jetty 3.4M > > > >> >> ./simulator-4.4-integration-tests 264M ./build-4.5-noredist 2.0G > > > >> >> ./package-deb-master 576K ./build-master-simulator 137M > > > >> >> ./mgmt-update-nonoss-libs 11M ./test-matrix-extended-master 235M > > > >> >> ./cloudstack-4.4-maven-build-noredist > > > >> >> 5.3M ./simulator-hotfix-trigger > > > >> >> 8.0K ./cloudstack-maven-build-with-branch-parameter > > > >> >> 2.7M ./test-yumrepo-refresh-master > > > >> >> 1.3G ./cloudstack-4.5-hyperv-agent > > > >> >> 1.6M ./Build-by-Branch > > > >> >> 3.5G ./build-systemvm-master > > > >> >> 369M ./package-deb-4.6 > > > >> >> 172K ./test-maintenance-operations-4.4 48K > > > >> >> ./cloudstack-4.3-package-rpm-noredist > > > >> >> 532K ./build-docs-vxlan-master > > > >> >> 18M ./build-docs-devguide-master > > > >> >> 456K ./test-environment-refresh-master 876K ./test-matrix-master > > 2.1M > > > >> >> ./test-yumrepo-refresh-4.4 51M ./test-environment-refresh-4.4 > 6.3M > > > >> >> ./simulator-singlerun 724M ./package-deb-4.5 16M > > > >> >> ./docs-4.3-gsoc-guide 384K ./test-matrix-extended-4.4 20K > > > >> >> ./pull-requests 204K ./build-docs-midonet-master 358M > > > >> >> ./cloudstack-4.4-package-deb 506M ./build-master-jdk18 2.3M > > > >> >> ./cloudstack.apt-get.eu-noredis-centos7 > > > >> >> 499M ./package-centos7-4.6 > > > >> >> 119M ./build-4.5-simulator > > > >> >> 12K ./cloudstack-4.2-systemvm_test > > > >> >> 1.6M ./test-regression-matrix-4.4 > > > >> >> 144K ./build-systemvm64-master-with-debian8 > > > >> >> 12K ./cloudstack-4.2-systemvm > > > >> >> 12K ./cloudstack-4.2-systemvm64 > > > >> >> 3.1M ./cloudstack-4.4-auto-marvin > > > >> >> 334M ./package-rhel63-master-noredist 6.8M > > > >> >> ./test-maintenance-operations-master > > > >> >> 36G . > > > >> >> > > > >> >> On Tue, Sep 29, 2015 at 9:04 AM, Pierre-Luc Dion < > > > pdion...@apache.org> > > > >> wrote: > > > >> >>> David, Do you think you could do another cleanup on the master > of > > > >> j.bac.o ? > > > >> >>> > > > >> >>> Could you list the jobs that consume most of the spaces? > > > >> >>> > > > >> >>> Thanks, > > > >> >>> > > > >> >>> > > > >> >>> On Wed, Sep 23, 2015 at 12:22 AM, Rajani Karuturi > > > >> >>> <rajani.karut...@citrix.com> wrote: > > > >> >>>> > > > >> >>>> Hi David, > > > >> >>>> disk space issue again. Can you please do a little more > cleanup? > > > >> >>>> > > > >> >>>> > > > >> >>>> > > > http://jenkins.buildacloud.org/computer/coohq-slave-01/executors/0/ > > > >> >>>> causeOfDeath > > > >> >>>> > > > >> >>>> ~Rajani > > > >> >>>> > > > >> >>>> > > > >> >>>> > > > >> >>>> On 18-Sep-2015, at 11:59 pm, David Nalley <da...@gnsa.us> > wrote: > > > >> >>>> > > > >> >>>>> I freed up a little bit of space, but we'll have to do > something > > > >> >>>>> different in the long term. > > > >> >>>>> > > > >> >>>>> --David > > > >> >>>>> > > > >> >>>>> On Fri, Sep 18, 2015 at 7:37 AM, Pierre-Luc Dion > > > >> >>>>> <pd...@cloudops.com> > > > >> >>>>> wrote: > > > >> >>>>>> Hi, > > > >> >>>>>> > > > >> >>>>>> Srikanteswararao (or any one having access to jenkins > server), > > do > > > >> >>>>>> you think you could do some more magic here? We ran out of > > space > > > >> >>>>>> again on the master jenkins server for j.bac.o. > > > >> >>>>>> > > > >> >>>>>> Thanks, > > > >> >>>>>> > > > >> >>>>>> PL > > > >> >>>>>> > > > >> >>>>>> On Mon, Jun 1, 2015 at 4:26 PM, Pierre-Luc Dion > > > >> >>>>>> <pd...@cloudops.com> > > > >> >>>>>> wrote: > > > >> >>>>>> > > > >> >>>>>>> thanks Srikanteswararao, it worked and we have centos63 > rpm > > > >> >>>>>>> now for master branch! > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> *Pierre-Luc DION* > > > >> >>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect t > > > >> >>>>>>> 855.652.5683 > > > >> >>>>>>> > > > >> >>>>>>> *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions > > > >> >>>>>>> Experts > > > >> >>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w > > cloudops.com > > > >> >>>>>>> *|* tw @CloudOps_ > > > >> >>>>>>> > > > >> >>>>>>> On Mon, Jun 1, 2015 at 3:09 PM, Srikanteswararao talluri < > > > >> >>>>>>> tall...@apache.org> wrote: > > > >> >>>>>>> > > > >> >>>>>>>> There is no space . > > > >> >>>>>>>> > > > >> >>>>>>>> /dev/xvdf1 50G 47G 0 100% > > > /var/lib/jenkins/workspace > > > >> >>>>>>>> > > > >> >>>>>>>> I have cleaned up builds for 'fastsimulatorbuild' job which > > > >> >>>>>>>> were using 9G. > > > >> >>>>>>>> Modified the job to delete old builds and artefacts after > 30 > > > >> >>>>>>>> builds/30days. > > > >> >>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>>> ~Talluri > > > >> >>>>>>>> > > > >> >>>>>>>> On Mon, Jun 1, 2015 at 8:14 PM, Pierre-Luc Dion > > > >> >>>>>>>> <pd...@cloudops.com> > > > >> >>>>>>>> wrote: > > > >> >>>>>>>> > > > >> >>>>>>>>> Can someone that have access to the principal jenkins > server > > > >> >>>>>>>>> to check > > > >> >>>>>>>> disk > > > >> >>>>>>>>> space left? > > > >> >>>>>>>>> > > > >> >>>>>>>>> getting this error on few slave, the one I've check does > not > > > >> >>>>>>>>> have disk > > > >> >>>>>>>> full > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> Thread has died > > > >> >>>>>>>>> > > > >> >>>>>>>>> java.io.IOException: No space left on device > > > >> >>>>>>>>> at java.io.FileOutputStream.writeBytes(Native > Method) > > > >> >>>>>>>>> at > > > >> java.io.FileOutputStream.write(FileOutputStream.java:318) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) > > > >> >>>>>>>>> at > > > >> sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316) > > > >> >>>>>>>>> at > > > sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) > > > >> >>>>>>>>> at > > java.io.BufferedWriter.close(BufferedWriter.java:266) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > hudson.util.AtomicFileWriter.close(AtomicFileWriter.java:94) > > > >> >>>>>>>>> at > > > >> >>>>>>>> > > hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:109) > > > >> >>>>>>>>> at hudson.util.TextFile.write(TextFile.java:121) > > > >> >>>>>>>>> at > hudson.model.Job.saveNextBuildNumber(Job.java:274) > > > >> >>>>>>>>> at hudson.model.Job.assignBuildNumber(Job.java:332) > > > >> >>>>>>>>> at hudson.model.Run.<init>(Run.java:286) > > > >> >>>>>>>>> at > > > >> hudson.model.AbstractBuild.<init>(AbstractBuild.java:167) > > > >> >>>>>>>>> at hudson.model.Build.<init>(Build.java:92) > > > >> >>>>>>>>> at > > > >> hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:34) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > > >> >>>>>>>>> Method) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>>> > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCon > > > >> >>>>>>>> structorAccessorImpl.java:57) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>>> > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Deleg > > > >> >>>>>>>> atingConstructorAccessorImpl.java:45) > > > >> >>>>>>>>> at > > > >> >>>>>>>> > > java.lang.reflect.Constructor.newInstance(Constructor.java:525) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:186) > > > >> >>>>>>>>> at > > > >> >>>>>>>> > > hudson.model.AbstractProject.newBuild(AbstractProject.java:1000 > > > >> >>>>>>>> ) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> > > hudson.model.AbstractProject.createExecutable(AbstractProject.java:1193) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> > > hudson.model.AbstractProject.createExecutable(AbstractProject.java:143) > > > >> >>>>>>>>> at hudson.model.Executor.run(Executor.java:213) > > > >> >>>>>>>>> > > > >> >>>>>>>>> more info > > > >> >>>>>>>>> <http://wiki.jenkins-ci.org/display/JENKINS/Dead+Executor > > > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> ref: > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>>> > > > http://jenkins.buildacloud.org/computer/cca-slave-02/executors/ > > > >> >>>>>>>> 0/causeOfDeath > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> Thanks! > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>> > > > >> >>> > > > >> > > > > >> > > > > > > > > > > > > > > > > -- > > > > Daan > > > > > > > > > > > -- > > Daan > > > -- Daan
