Disks is full againt on master jenkins server. I did not cleanup the disk yet as I don't have access to the server.
I'll delete some jobs see how it goes... On Thu, Oct 1, 2015 at 11:11 AM, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > So PL, these are just the slaves and not jenkins, right? > If we can incorporate some monitoring, that would be nice. (no emotional > attachements to my hack has formed yet) > > On Thu, Oct 1, 2015 at 2:18 PM, Pierre-Luc Dion <pdion...@apache.org> > wrote: > > > Daan: I like your style! > > > > I was looking at using Chef [1] to managed Jenkins slaves, I've started > > deploying it as Proof of concept for some slaves I've recently deployed, > > within this scope of idea was to deploy a monitoring tool at some point > to > > alerts on event such as disk full. > > > > I was not aware that puppet was already used for automated deployment of > > Jenkins. I guest if we are going to managed Jenkins slaves, we all need > to > > agree on the tool to use with the majority of the community/dev right? > > > > [1] cookbooks repo: https://github.com/CloudStack-extras/ci-chefrepo > > > > > > > > On Wed, Sep 30, 2015 at 6:59 AM, Daan Hoogland <daan.hoogl...@gmail.com> > > wrote: > > > > > So I did, expressing my worst operator practices I created a small > > script: > > > > > > <script> > > > ! /bin/bash > > > > > > THRESHOLD=95 > > > USAGE=`/bin/df -P /var/lib/jenkins/workspace | /bin/grep -v Capacity | > > > /bin/cut -c 47- | /bin/cut -c 6-7 ` > > > > > > function maildev() { > > > /bin/mail -s 'disk usage at jenkins.bac.o' dev@cloudstack.apache.org > > << > > > EOT > > > `/bin/df -P /var/lib/jenkins/workspace` > > > EOT > > > } > > > > > > [[ $USAGE > $THRESHOLD ]] && maildev > > > </script> > > > > > > and added it to cron: > > > > > > */10 * * * * ( cd /var/lib/jenkins && ./check_workspace.sh ) >/dev/null > > > > > > There I noticed some traces of puppet usage. Should I implement this in > > > puppet somehow? > > > > > > willing to do some more, > > > Daan > > > > > > On Wed, Sep 30, 2015 at 11:40 AM, David Nalley <da...@gnsa.us> wrote: > > > > > > > SGTM, Feel free to. > > > > > > > > > > > > > > > > On Wed, Sep 30, 2015 at 5:34 AM, Daan Hoogland < > > daan.hoogl...@gmail.com> > > > > wrote: > > > > > H David, I just tried and it works indeed. Forgot about that. Does > it > > > > make > > > > > sense to have a script containing du mail to dev@ if something > like > > > > > /var/lib/jenkins/workspace or / exceeds a threshold? (yes me > > > > volunteering) > > > > > > > > > > On Wed, Sep 30, 2015 at 10:48 AM, Raja Pullela < > > > raja.pull...@citrix.com> > > > > > wrote: > > > > > > > > > >> Btw, edison, pradeep, rayeesn - are not active anymore - talluri > may > > > be? > > > > >> I think. > > > > >> > > > > >> -----Original Message----- > > > > >> From: David Nalley [mailto:da...@gnsa.us] > > > > >> Sent: Wednesday, September 30, 2015 2:01 PM > > > > >> To: dev@cloudstack.apache.org > > > > >> Cc: Pierre-Luc Dion <pdion...@apache.org>; tall...@apache.org > > > > >> Subject: Re: jenkins disk full ? > > > > >> > > > > >> Right now these folks have access: > > > > >> > > > > >> ewanm:x:1002:1002::/home/ewanm:/bin/bash > > > > >> prasanna:x:1003:1003::/home/prasanna:/bin/bash > > > > >> salvatore:x:1004:1004::/home/salvatore:/bin/bash > > > > >> edison:x:1005:1005::/home/edison:/bin/bash > > > > >> pradeep:x:1006:1006::/home/pradeep:/bin/bash > > > > >> rayeesn:x:1007:1007::/home/rayeesn:/bin/bash > > > > >> hugot:x:1008:1008::/home/hugot:/bin/bash > > > > >> talluri:x:1009:1009::/home/talluri:/bin/bash > > > > >> dahn:x:1010:1010::/home/dahn:/bin/bash > > > > >> > > > > >> PL or other folks who want access, send me a ssh pub key off list. > > > > >> > > > > >> --David > > > > >> > > > > >> On Wed, Sep 30, 2015 at 4:20 AM, Rajani Karuturi < > > > > >> rajani.karut...@citrix.com> wrote: > > > > >> > Thanks david. Do you know if anyone else has access to this? if > > not, > > > > can > > > > >> you give access to PL please? > > > > >> > > > > > >> > ~Rajani > > > > >> > > > > > >> > > > > > >> > > > > > >> > On 30-Sep-2015, at 1:13 pm, David Nalley <da...@gnsa.us> wrote: > > > > >> > > > > > >> >> The below is before I cleaned up a bit. We now have 3.2GB free. > > > > >> >> > > > > >> >> [root@domU-12-31-39-15-22-2D jobs]# du -h -d 1 1.3G > > > > >> >> ./package-rhel63-master 200M > > ./cloudstack-4.3-maven-build-noredist > > > > >> >> 638M ./package-rhel63-4.5 > > > > >> >> 1.1G ./cloudstack-4.4-systemvm64 > > > > >> >> 1.3M ./build-4.5 > > > > >> >> 8.0K ./cloudstack-master-with-patch > > > > >> >> 186M ./build-test-archive-4.4 > > > > >> >> 2.9M ./test-smoke-matrix-master > > > > >> >> 100K ./Create-Release-Packages > > > > >> >> 1.5M ./test-packaging-4.4 > > > > >> >> 214M ./build-test-archive-master > > > > >> >> 612K ./cloudstack.apt-get.eu-systemvm 88M ./built-test-archive > > 4.2M > > > > >> >> ./simulator-4.5-singlerun 424K ./test-tabnanny-4.4 2.2G > > > > >> >> ./build-master 124M ./cloudstack-4.4-systemvm 331M > > > > >> >> ./package-centos7-4.4.4-noredist 804K > > ./cloudstack-4.4-maven-build > > > > >> >> 3.4M ./cloudstack-4.3-maven-build 645M ./package-centos7-master > > 80M > > > > >> >> ./test-fetch-marvin-master 180M ./parameterized-slowbuild 508K > > > > >> >> ./cloudstack.apt-get.eu-noredis-rhel > > > > >> >> 104K ./dockerbuild-master-marvin > > > > >> >> 9.1M ./test-regression-matrix-master > > > > >> >> 345M ./cloudstack-4.3-package-deb > > > > >> >> 3.8G ./parameterized-sytemvm > > > > >> >> 26M ./HyperVAgent4.3 > > > > >> >> 510M ./cloudstack-4.3-package-rpm > > > > >> >> 1.5M ./test-packaging-master > > > > >> >> 7.2M ./build-docs-admin-master > > > > >> >> 8.3M ./cloudstack-marvin-master > > > > >> >> 1013M ./cloudstack-4.3-systemvm > > > > >> >> 131M ./run-checkin-tests > > > > >> >> 2.5M ./simulator-gate > > > > >> >> 145M ./cloudstack-master-twiceweekly-coverity > > > > >> >> 3.2M ./cloudstack-4.4-hyperv-agent > > > > >> >> 104K ./test-setup-advancedzone-master 1.1G ./master-slowbuild > 44M > > > > >> >> ./cloudstack-master 301M ./package-centos7-master-noredist 396K > > > > >> >> ./test-tabnanny-master 816K ./simulator-build 20K > ./test-libcloud > > > > >> >> 236K ./test-smoke-matrix-4.3 4.9M ./dockerbuild-4.5-simulator > > 331M > > > > >> >> ./package-centos7-4.4-noredist 632K > > > > >> >> ./cloudstack.apt-get.eu-noredis-deb > > > > >> >> 144K ./dockerbuild-master-management_centos6 > > > > >> >> 292M ./package-centos6-4.6 > > > > >> >> 9.4M ./stop-jetty > > > > >> >> 326M ./cloudstack-4.4-package-rpm > > > > >> >> 40K ./mgmt-build-reviewboard-requests 20K > > > ./build-test-centos-builtin > > > > >> >> 76K ./simulator-deploydb 20K ./Master - Python Static Code > > Analysis > > > > >> >> 53M ./test-smoke-matrix-4.4 442M ./build-master-slowbuild 376K > > > > >> >> ./build-docs-niciranvp-master 840K ./test-matrix-4.4 2.7M > > > > >> >> ./dockerbuild-master-simulator 21M ./build-docs-install-master > > 50M > > > > >> >> ./test-fetch-marvin-4.4 789M > > > > >> >> ./cloudstack-rpm-packages-with-branch-parameter > > > > >> >> 5.3G ./build-systemvm-4.5 > > > > >> >> 1.1G ./cloudstack-4.3-systemvm64 > > > > >> >> 3.2M ./build-docs-release-notes-master 8.0K > > > > >> >> ./cloudstack-master-codecoverage 121M ./fastsimulatorbuild 4.3M > > > > >> >> ./cloudstack-4.4-marvin 352M ./build-master-noredist 7.2M > > > > >> >> ./test-setup-advancedzone-4.4 3.3M > > ./cloudstack-master-hyperv-agent > > > > >> >> 1.9G ./build-systemvm64-master 29M ./start-jetty 3.4M > > > > >> >> ./simulator-4.4-integration-tests 264M ./build-4.5-noredist > 2.0G > > > > >> >> ./package-deb-master 576K ./build-master-simulator 137M > > > > >> >> ./mgmt-update-nonoss-libs 11M ./test-matrix-extended-master > 235M > > > > >> >> ./cloudstack-4.4-maven-build-noredist > > > > >> >> 5.3M ./simulator-hotfix-trigger > > > > >> >> 8.0K ./cloudstack-maven-build-with-branch-parameter > > > > >> >> 2.7M ./test-yumrepo-refresh-master > > > > >> >> 1.3G ./cloudstack-4.5-hyperv-agent > > > > >> >> 1.6M ./Build-by-Branch > > > > >> >> 3.5G ./build-systemvm-master > > > > >> >> 369M ./package-deb-4.6 > > > > >> >> 172K ./test-maintenance-operations-4.4 48K > > > > >> >> ./cloudstack-4.3-package-rpm-noredist > > > > >> >> 532K ./build-docs-vxlan-master > > > > >> >> 18M ./build-docs-devguide-master > > > > >> >> 456K ./test-environment-refresh-master 876K > ./test-matrix-master > > > 2.1M > > > > >> >> ./test-yumrepo-refresh-4.4 51M ./test-environment-refresh-4.4 > > 6.3M > > > > >> >> ./simulator-singlerun 724M ./package-deb-4.5 16M > > > > >> >> ./docs-4.3-gsoc-guide 384K ./test-matrix-extended-4.4 20K > > > > >> >> ./pull-requests 204K ./build-docs-midonet-master 358M > > > > >> >> ./cloudstack-4.4-package-deb 506M ./build-master-jdk18 2.3M > > > > >> >> ./cloudstack.apt-get.eu-noredis-centos7 > > > > >> >> 499M ./package-centos7-4.6 > > > > >> >> 119M ./build-4.5-simulator > > > > >> >> 12K ./cloudstack-4.2-systemvm_test > > > > >> >> 1.6M ./test-regression-matrix-4.4 > > > > >> >> 144K ./build-systemvm64-master-with-debian8 > > > > >> >> 12K ./cloudstack-4.2-systemvm > > > > >> >> 12K ./cloudstack-4.2-systemvm64 > > > > >> >> 3.1M ./cloudstack-4.4-auto-marvin > > > > >> >> 334M ./package-rhel63-master-noredist 6.8M > > > > >> >> ./test-maintenance-operations-master > > > > >> >> 36G . > > > > >> >> > > > > >> >> On Tue, Sep 29, 2015 at 9:04 AM, Pierre-Luc Dion < > > > > pdion...@apache.org> > > > > >> wrote: > > > > >> >>> David, Do you think you could do another cleanup on the > master > > of > > > > >> j.bac.o ? > > > > >> >>> > > > > >> >>> Could you list the jobs that consume most of the spaces? > > > > >> >>> > > > > >> >>> Thanks, > > > > >> >>> > > > > >> >>> > > > > >> >>> On Wed, Sep 23, 2015 at 12:22 AM, Rajani Karuturi > > > > >> >>> <rajani.karut...@citrix.com> wrote: > > > > >> >>>> > > > > >> >>>> Hi David, > > > > >> >>>> disk space issue again. Can you please do a little more > > cleanup? > > > > >> >>>> > > > > >> >>>> > > > > >> >>>> > > > > http://jenkins.buildacloud.org/computer/coohq-slave-01/executors/0/ > > > > >> >>>> causeOfDeath > > > > >> >>>> > > > > >> >>>> ~Rajani > > > > >> >>>> > > > > >> >>>> > > > > >> >>>> > > > > >> >>>> On 18-Sep-2015, at 11:59 pm, David Nalley <da...@gnsa.us> > > wrote: > > > > >> >>>> > > > > >> >>>>> I freed up a little bit of space, but we'll have to do > > something > > > > >> >>>>> different in the long term. > > > > >> >>>>> > > > > >> >>>>> --David > > > > >> >>>>> > > > > >> >>>>> On Fri, Sep 18, 2015 at 7:37 AM, Pierre-Luc Dion > > > > >> >>>>> <pd...@cloudops.com> > > > > >> >>>>> wrote: > > > > >> >>>>>> Hi, > > > > >> >>>>>> > > > > >> >>>>>> Srikanteswararao (or any one having access to jenkins > > server), > > > do > > > > >> >>>>>> you think you could do some more magic here? We ran out of > > > space > > > > >> >>>>>> again on the master jenkins server for j.bac.o. > > > > >> >>>>>> > > > > >> >>>>>> Thanks, > > > > >> >>>>>> > > > > >> >>>>>> PL > > > > >> >>>>>> > > > > >> >>>>>> On Mon, Jun 1, 2015 at 4:26 PM, Pierre-Luc Dion > > > > >> >>>>>> <pd...@cloudops.com> > > > > >> >>>>>> wrote: > > > > >> >>>>>> > > > > >> >>>>>>> thanks Srikanteswararao, it worked and we have centos63 > > rpm > > > > >> >>>>>>> now for master branch! > > > > >> >>>>>>> > > > > >> >>>>>>> > > > > >> >>>>>>> > > > > >> >>>>>>> *Pierre-Luc DION* > > > > >> >>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect t > > > > >> >>>>>>> 855.652.5683 > > > > >> >>>>>>> > > > > >> >>>>>>> *CloudOps* Votre partenaire infonuagique* | *Cloud > Solutions > > > > >> >>>>>>> Experts > > > > >> >>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w > > > cloudops.com > > > > >> >>>>>>> *|* tw @CloudOps_ > > > > >> >>>>>>> > > > > >> >>>>>>> On Mon, Jun 1, 2015 at 3:09 PM, Srikanteswararao talluri < > > > > >> >>>>>>> tall...@apache.org> wrote: > > > > >> >>>>>>> > > > > >> >>>>>>>> There is no space . > > > > >> >>>>>>>> > > > > >> >>>>>>>> /dev/xvdf1 50G 47G 0 100% > > > > /var/lib/jenkins/workspace > > > > >> >>>>>>>> > > > > >> >>>>>>>> I have cleaned up builds for 'fastsimulatorbuild' job > which > > > > >> >>>>>>>> were using 9G. > > > > >> >>>>>>>> Modified the job to delete old builds and artefacts after > > 30 > > > > >> >>>>>>>> builds/30days. > > > > >> >>>>>>>> > > > > >> >>>>>>>> > > > > >> >>>>>>>> ~Talluri > > > > >> >>>>>>>> > > > > >> >>>>>>>> On Mon, Jun 1, 2015 at 8:14 PM, Pierre-Luc Dion > > > > >> >>>>>>>> <pd...@cloudops.com> > > > > >> >>>>>>>> wrote: > > > > >> >>>>>>>> > > > > >> >>>>>>>>> Can someone that have access to the principal jenkins > > server > > > > >> >>>>>>>>> to check > > > > >> >>>>>>>> disk > > > > >> >>>>>>>>> space left? > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> getting this error on few slave, the one I've check does > > not > > > > >> >>>>>>>>> have disk > > > > >> >>>>>>>> full > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> Thread has died > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> java.io.IOException: No space left on device > > > > >> >>>>>>>>> at java.io.FileOutputStream.writeBytes(Native > > Method) > > > > >> >>>>>>>>> at > > > > >> java.io.FileOutputStream.write(FileOutputStream.java:318) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>>> > > sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) > > > > >> >>>>>>>>> at > > > > >> sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316) > > > > >> >>>>>>>>> at > > > > sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>>> > > > java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) > > > > >> >>>>>>>>> at > > > java.io.BufferedWriter.close(BufferedWriter.java:266) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>>> > > hudson.util.AtomicFileWriter.close(AtomicFileWriter.java:94) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>> > > > hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:109) > > > > >> >>>>>>>>> at hudson.util.TextFile.write(TextFile.java:121) > > > > >> >>>>>>>>> at > > hudson.model.Job.saveNextBuildNumber(Job.java:274) > > > > >> >>>>>>>>> at > hudson.model.Job.assignBuildNumber(Job.java:332) > > > > >> >>>>>>>>> at hudson.model.Run.<init>(Run.java:286) > > > > >> >>>>>>>>> at > > > > >> hudson.model.AbstractBuild.<init>(AbstractBuild.java:167) > > > > >> >>>>>>>>> at hudson.model.Build.<init>(Build.java:92) > > > > >> >>>>>>>>> at > > > > >> hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:34) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>>> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > > > >> >>>>>>>>> Method) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>>> > > > > >> >>>>>>>> > > > > >> >>>>>>>> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCon > > > > >> >>>>>>>> structorAccessorImpl.java:57) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>>> > > > > >> >>>>>>>> > > > > >> >>>>>>>> > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Deleg > > > > >> >>>>>>>> atingConstructorAccessorImpl.java:45) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>> > > > java.lang.reflect.Constructor.newInstance(Constructor.java:525) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>>> > > > > >> > jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:186) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>> > > > hudson.model.AbstractProject.newBuild(AbstractProject.java:1000 > > > > >> >>>>>>>> ) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> > > > > >> > > > > hudson.model.AbstractProject.createExecutable(AbstractProject.java:1193) > > > > >> >>>>>>>>> at > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> > > > > >> > > > hudson.model.AbstractProject.createExecutable(AbstractProject.java:143) > > > > >> >>>>>>>>> at hudson.model.Executor.run(Executor.java:213) > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> more info > > > > >> >>>>>>>>> < > http://wiki.jenkins-ci.org/display/JENKINS/Dead+Executor > > > > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> ref: > > > > >> >>>>>>>>> > > > > >> >>>>>>>> > > > > >> >>>>>>>> > > > > http://jenkins.buildacloud.org/computer/cca-slave-02/executors/ > > > > >> >>>>>>>> 0/causeOfDeath > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> Thanks! > > > > >> >>>>>>>>> > > > > >> >>>>>>>> > > > > >> >>>>>>> > > > > >> >>>>>>> > > > > >> >>>> > > > > >> >>> > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > -- > > > > > Daan > > > > > > > > > > > > > > > > -- > > > Daan > > > > > > > > > -- > Daan >
