We're refining our monitoring and some cron jobs to try and keep the disk space pruned. But when space balloons due to poor cleanup, there is only so much we can do.
The short answer is that the multitude of groups at the Foundation using these shared resources need to be cognizant of that fact, and cooperate on ways to keep them available for all. We've seen disk space problems, too many jobs, too-long jobs, etc. Infra will keep working and improving our management of the build nodes. In the meantime, please keep reporting problems, and we'll do our best to keep the system up for y'all. Cheers, Greg InfraAdmin, ASF On Sun, Sep 30, 2018 at 10:54 PM Mike Jumper <mjum...@apache.org> wrote: > Is there any way Jenkins could be configured to take nodes offline > automatically when free space is below a certain threshold? Granted > space-hungry builds and proper cleanup remain a problem, but at least there > wouldn't be a random chance of jobs failing due to being assigned to a full > node. > > - Mike > > > On Sun, Sep 30, 2018 at 3:57 PM, Chris Lambertus <c...@apache.org> wrote: > > > The main culprit on H31 appears to be pulsar ws-cleanup jobs consuming > > almost 40% of the disk. (The problem on H21 was something > docker-related.) > > > > > > This is the second project I have found (Brooklyn was the first) which is > > leaving ws-cleanup directories all over the workspace and filling up the > > disks. Each one of these was 500+ MB. Pulsar, please address ASAP. > > > > > > > > > > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:19 > > pulsar_precommit_cpp@2_ws-cleanup_1535737112280 > > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:18 > > pulsar_precommit_cpp@2_ws-cleanup_1536174576053 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 06:04 > > pulsar_precommit_cpp@2_ws-cleanup_1537292379178 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 22:45 > > pulsar_precommit_cpp@2_ws-cleanup_1537397120862 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45 > > pulsar_precommit_cpp@2_ws-cleanup_1537403378279 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 21 23:31 > > pulsar_precommit_cpp@2_ws-cleanup_1537572701829 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 26 16:04 > > pulsar_precommit_cpp@2_ws-cleanup_1537977879341 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 27 22:23 > > pulsar_precommit_cpp@2_ws-cleanup_1538087035027 > > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:14 > > pulsar_precommit_cpp_ws-cleanup_1535699830230 > > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:16 > > pulsar_precommit_cpp_ws-cleanup_1535735511423 > > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:13 > > pulsar_precommit_cpp_ws-cleanup_1535786064397 > > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:18 > > pulsar_precommit_cpp_ws-cleanup_1536096963586 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 21 06:47 > > pulsar_precommit_cpp_ws-cleanup_1536114938910 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 21 06:45 > > pulsar_precommit_cpp_ws-cleanup_1536173165909 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:38 > > pulsar_precommit_cpp_ws-cleanup_1536175584418 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:50 > > pulsar_precommit_cpp_ws-cleanup_1536180707506 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:38 > > pulsar_precommit_cpp_ws-cleanup_1536182927940 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:51 > > pulsar_precommit_cpp_ws-cleanup_1536185193716 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:50 > > pulsar_precommit_cpp_ws-cleanup_1536273709029 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 05:59 > > pulsar_precommit_cpp_ws-cleanup_1536316964794 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 06:06 > > pulsar_precommit_cpp_ws-cleanup_1536675650893 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 28 06:08 > > pulsar_precommit_cpp_ws-cleanup_1536879616962 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 30 19:53 > > pulsar_precommit_cpp_ws-cleanup_1536891056968 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 30 19:47 > > pulsar_precommit_cpp_ws-cleanup_1536966466683 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 18 00:43 > > pulsar_precommit_cpp_ws-cleanup_1537216387387 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 00:45 > > pulsar_precommit_cpp_ws-cleanup_1537292309413 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 00:45 > > pulsar_precommit_cpp_ws-cleanup_1537308526251 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45 > > pulsar_precommit_cpp_ws-cleanup_1537331780420 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45 > > pulsar_precommit_cpp_ws-cleanup_1537379886715 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:46 > > pulsar_precommit_cpp_ws-cleanup_1537386197893 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:46 > > pulsar_precommit_cpp_ws-cleanup_1537397073464 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:46 > > pulsar_precommit_cpp_ws-cleanup_1537403247575 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:43 > > pulsar_precommit_cpp_ws-cleanup_1537413954888 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:45 > > pulsar_precommit_cpp_ws-cleanup_1537417322151 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:45 > > pulsar_precommit_cpp_ws-cleanup_1537421381129 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:44 > > pulsar_precommit_cpp_ws-cleanup_1537469348852 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:45 > > pulsar_precommit_cpp_ws-cleanup_1537489399468 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 22 00:44 > > pulsar_precommit_cpp_ws-cleanup_1537566528660 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 23:12 > > pulsar_precommit_cpp_ws-cleanup_1537571549978 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 22 00:44 > > pulsar_precommit_cpp_ws-cleanup_1537573605258 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 26 00:43 > > pulsar_precommit_cpp_ws-cleanup_1537855939828 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 25 11:18 > > pulsar_precommit_cpp_ws-cleanup_1537874305575 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 25 12:41 > > pulsar_precommit_cpp_ws-cleanup_1537879296622 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 26 01:39 > > pulsar_precommit_cpp_ws-cleanup_1537925955967 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 26 13:17 > > pulsar_precommit_cpp_ws-cleanup_1537967826838 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 26 15:55 > > pulsar_precommit_cpp_ws-cleanup_1537977326736 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 26 17:34 > > pulsar_precommit_cpp_ws-cleanup_1537983268177 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 27 22:04 > > pulsar_precommit_cpp_ws-cleanup_1538085846793 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 28 00:20 > > pulsar_precommit_cpp_ws-cleanup_1538094024821 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 28 17:42 > > pulsar_precommit_cpp_ws-cleanup_1538156543343 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 06:02 > > pulsar_precommit_integrationtests_ws-cleanup_1536298125841 > > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 06:00 > > pulsar_precommit_integrationtests_ws-cleanup_1536968128660 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 18 00:43 > > pulsar_precommit_integrationtests_ws-cleanup_1537218541598 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45 > > pulsar_precommit_integrationtests_ws-cleanup_1537331703324 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 22 00:43 > > pulsar_precommit_integrationtests_ws-cleanup_1537553748859 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 24 00:43 > > pulsar_precommit_integrationtests_ws-cleanup_1537730402937 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 24 07:48 > > pulsar_precommit_integrationtests_ws-cleanup_1537775336586 > > drwxr-xr-x 54 jenkins jenkins 4096 Sep 24 23:30 > > pulsar_precommit_integrationtests_ws-cleanup_1537831851711 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 25 23:08 > > pulsar_precommit_integrationtests_ws-cleanup_1537916928485 > > drwxr-xr-x 55 jenkins jenkins 4096 Sep 27 15:21 > > pulsar_precommit_integrationtests_ws-cleanup_1538061702243 > > > > > > -Chris > > ASF Infra > > > > > > > > > > > > > On Sep 30, 2018, at 3:12 PM, Chris Thistlethwaite < > > cthistl...@outlook.com> wrote: > > > > > > I've been working on H21 and H31. They both had a some docker issues > > where the process was locked and I couldn't do anything with a few > > containers that were over 8 days old. > > > > > > Thought we might be good this morning so I threw them back into > > rotation. Guess not, so I'll take them back out. > > > > > > -Chris T. > > > ________________________________ > > > From: Mike Jumper <mjum...@apache.org> > > > Sent: Sunday, September 30, 2018 6:03:51 PM > > > To: builds@apache.org > > > Subject: No space on H21 (Was: Fwd: Build failed in Jenkins: > > guacamole-client-coverity #73) > > > > > > FYI - we're currently seeing disk space issues on H21. > > > > > > - Mike > > > > > > > > > ---------- Forwarded message --------- > > > From: Apache Jenkins Server <jenk...@builds.apache.org> > > > Date: Sun, Sep 30, 2018, 14:37 > > > Subject: Build failed in Jenkins: guacamole-client-coverity #73 > > > To: <d...@guacamole.apache.org> > > > > > > > > > See < > > > https://builds.apache.org/job/guacamole-client-coverity/73/d > > isplay/redirect> > > > > > > ------------------------------------------ > > > Started by user mjumper > > > [EnvInject] - Loading node environment variables. > > > Building remotely on H21 (ubuntu xenial) in workspace < > > > https://builds.apache.org/job/guacamole-client-coverity/ws/> > > > [WS-CLEANUP] Deleting project workspace... > > > Cloning the remote Git repository > > > Cloning repository > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git > > >> git init < > > > https://builds.apache.org/job/guacamole-client-coverity/ws/g > > uacamole-client> > > > # timeout=10 > > > Fetching upstream changes from > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git > > >> git --version # timeout=10 > > >> git fetch --tags --progress > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git > > > +refs/heads/*:refs/remotes/origin/* > > > ERROR: Error cloning remote repo 'origin' > > > hudson.plugins.git.GitException: Command "git fetch --tags --progress > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git > > > +refs/heads/*:refs/remotes/origin/*" returned status code 128: > > > stdout: > > > stderr: remote: Counting objects: 62166, done. > > > remote: Compressing objects: 0% (1/16096) remote: > Compressing > > > objects: 1% (161/16096) remote: Compressing objects: 2% > > > (322/16096) remote: Compressing objects: 3% (483/16096) > > > remote: Compressing objects: 4% (644/16096) remote: > > > Compressing objects: 5% (805/16096) remote: Compressing > > > objects: 6% (966/16096) remote: Compressing objects: 7% > > > (1127/16096) remote: Compressing objects: 8% (1288/16096) > > > remote: Compressing objects: 9% (1449/16096) remote: > > > Compressing objects: 10% (1610/16096) remote: Compressing > > > objects: 11% (1771/16096) remote: Compressing objects: 12% > > > (1932/16096) remote: Compressing objects: 13% (2093/16096) > > > remote: Compressing objects: 14% (2254/16096) remote: > > > Compressing objects: 15% (2415/16096) remote: Compressing > > > objects: 16% (2576/16096) remote: Compressing objects: 17% > > > (2737/16096) remote: Compressing objects: 18% (2898/16096) > > > remote: Compressing objects: 19% (3059/16096) remote: > > > Compressing objects: 20% (3220/16096) remote: Compressing > > > objects: 21% (3381/16096) remote: Compressing objects: 22% > > > (3542/16096) remote: Compressing objects: 23% (3703/16096) > > > remote: Compressing objects: 24% (3864/16096) remote: > > > Compressing objects: 25% (4024/16096) remote: Compressing > > > objects: 26% (4185/16096) remote: Compressing objects: 27% > > > (4346/16096) remote: Compressing objects: 28% (4507/16096) > > > remote: Compressing objects: 29% (4668/16096) remote: > > > Compressing objects: 30% (4829/16096) remote: Compressing > > > objects: 31% (4990/16096) remote: Compressing objects: 32% > > > (5151/16096) remote: Compressing objects: 33% (5312/16096) > > > remote: Compressing objects: 34% (5473/16096) remote: > > > Compressing objects: 35% (5634/16096) remote: Compressing > > > objects: 36% (5795/16096) remote: Compressing objects: 37% > > > (5956/16096) remote: Compressing objects: 38% (6117/16096) > > > remote: Compressing objects: 39% (6278/16096) remote: > > > Compressing objects: 40% (6439/16096) remote: Compressing > > > objects: 41% (6600/16096) remote: Compressing objects: 42% > > > (6761/16096) remote: Compressing objects: 43% (6922/16096) > > > remote: Compressing objects: 44% (7083/16096) remote: > > > Compressing objects: 45% (7244/16096) remote: Compressing > > > objects: 46% (7405/16096) remote: Compressing objects: 47% > > > (7566/16096) remote: Compressing objects: 48% (7727/16096) > > > remote: Compressing objects: 49% (7888/16096) remote: > > > Compressing objects: 50% (8048/16096) remote: Compressing > > > objects: 51% (8209/16096) remote: Compressing objects: 52% > > > (8370/16096) remote: Compressing objects: 53% (8531/16096) > > > remote: Compressing objects: 54% (8692/16096) remote: > > > Compressing objects: 55% (8853/16096) remote: Compressing > > > objects: 56% (9014/16096) remote: Compressing objects: 57% > > > (9175/16096) remote: Compressing objects: 58% (9336/16096) > > > remote: Compressing objects: 59% (9497/16096) remote: > > > Compressing objects: 60% (9658/16096) remote: Compressing > > > objects: 61% (9819/16096) remote: Compressing objects: 62% > > > (9980/16096) remote: Compressing objects: 63% (10141/16096) > > > remote: Compressing objects: 64% (10302/16096) remote: > > > Compressing objects: 65% (10463/16096) remote: Compressing > > > objects: 66% (10624/16096) remote: Compressing objects: 67% > > > (10785/16096) remote: Compressing objects: 68% (10946/16096) > > > remote: Compressing objects: 69% (11107/16096) remote: > > > Compressing objects: 70% (11268/16096) remote: Compressing > > > objects: 71% (11429/16096) remote: Compressing objects: 72% > > > (11590/16096) remote: Compressing objects: 73% (11751/16096) > > > remote: Compressing objects: 74% (11912/16096) remote: > > > Compressing objects: 75% (12072/16096) remote: Compressing > > > objects: 76% (12233/16096) remote: Compressing objects: 77% > > > (12394/16096) remote: Compressing objects: 78% (12555/16096) > > > remote: Compressing objects: 79% (12716/16096) remote: > > > Compressing objects: 80% (12877/16096) remote: Compressing > > > objects: 81% (13038/16096) remote: Compressing objects: 82% > > > (13199/16096) remote: Compressing objects: 83% (13360/16096) > > > remote: Compressing objects: 84% (13521/16096) remote: > > > Compressing objects: 85% (13682/16096) remote: Compressing > > > objects: 86% (13843/16096) remote: Compressing objects: 87% > > > (14004/16096) remote: Compressing objects: 88% (14165/16096) > > > remote: Compressing objects: 89% (14326/16096) remote: > > > Compressing objects: 90% (14487/16096) remote: Compressing > > > objects: 91% (14648/16096) remote: Compressing objects: 92% > > > (14809/16096) remote: Compressing objects: 93% (14970/16096) > > > remote: Compressing objects: 94% (15131/16096) remote: > > > Compressing objects: 95% (15292/16096) remote: Compressing > > > objects: 96% (15453/16096) remote: Compressing objects: 97% > > > (15614/16096) remote: Compressing objects: 98% (15775/16096) > > > remote: Compressing objects: 99% (15936/16096) remote: > > > Compressing objects: 100% (16096/16096) remote: Compressing > > > objects: 100% (16096/16096), done. > > > Receiving objects: 0% (1/62166) Receiving objects: 1% (622/62166) > > > Receiving objects: 2% (1244/62166) Receiving objects: 3% > > > (1865/62166) Receiving objects: 4% (2487/62166) Receiving > objects: > > > 5% (3109/62166) Receiving objects: 6% (3730/62166) Receiving > > > objects: 7% (4352/62166) Receiving objects: 8% (4974/62166) > > > Receiving objects: 9% (5595/62166) Receiving objects: 10% > > > (6217/62166) Receiving objects: 11% (6839/62166) Receiving > objects: > > > 12% (7460/62166) Receiving objects: 13% (8082/62166) Receiving > > > objects: 14% (8704/62166) Receiving objects: 15% (9325/62166) > > > Receiving objects: 16% (9947/62166) Receiving objects: 17% > > > (10569/62166) Receiving objects: 18% (11190/62166) Receiving > > objects: > > > 19% (11812/62166) Receiving objects: 20% (12434/62166) Receiving > > > objects: 21% (13055/62166) Receiving objects: 22% (13677/62166) > > > fatal: write error: No space left on device > > > fatal: index-pack failed > > > > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI > > n(CliGitAPIImpl.java:2002) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandW > > ithCredentials(CliGitAPIImpl.java:1721) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300( > > CliGitAPIImpl.java:72) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute( > > CliGitAPIImpl.java:405) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute( > > CliGitAPIImpl.java:614) > > > at > > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat > > ionHandler$1.call(RemoteGitImpl.java:153) > > > at > > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat > > ionHandler$1.call(RemoteGitImpl.java:146) > > > at hudson.remoting.UserRequest.perform(UserRequest.java:212) > > > at hudson.remoting.UserRequest.perform(UserRequest.java:54) > > > at hudson.remoting.Request$2.run(Request.java:369) > > > at > > > hudson.remoting.InterceptingExecutorService$1.call(Intercept > > ingExecutorService.java:72) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > > Executor.java:1142) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > > lExecutor.java:617) > > > at java.lang.Thread.run(Thread.java:748) > > > Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote > > call > > > to H21 > > > at > > > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > > > at > > > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserR > > equest.java:357) > > > at hudson.remoting.Channel.call(Channel.java:955) > > > at > > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat > > ionHandler.execute(RemoteGitImpl.java:146) > > > at sun.reflect.GeneratedMethodAccessor918.invoke(Unknown > > > Source) > > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > > thodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > at > > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat > > ionHandler.invoke(RemoteGitImpl.java:132) > > > at com.sun.proxy.$Proxy142.execute(Unknown Source) > > > at > > > hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1146) > > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > > > at hudson.scm.SCM.checkout(SCM.java:504) > > > at > > > hudson.model.AbstractProject.checkout(AbstractProject.java:1208) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe > > ckout(AbstractBuild.java:574) > > > at > > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra > > ctBuild.java:499) > > > at hudson.model.Run.execute(Run.java:1794) > > > at hudson.model.FreeStyleBuild.ru > > n(FreeStyleBuild.java:43) > > > at > > > hudson.model.ResourceController.execute(ResourceController.java:97) > > > at hudson.model.Executor.run(Executor.java:429) > > > ERROR: Error cloning remote repo 'origin' > > > Retrying after 10 seconds > > >> git rev-parse --is-inside-work-tree # timeout=10 > > > Fetching changes from the remote Git repository > > >> git config remote.origin.url > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git # > > timeout=10 > > > Fetching upstream changes from > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git > > >> git --version # timeout=10 > > >> git fetch --tags --progress > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git > > > +refs/heads/*:refs/remotes/origin/* > > > ERROR: Error fetching remote repo 'origin' > > > hudson.plugins.git.GitException: Failed to fetch from > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git > > > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888) > > > at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155) > > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > > > at hudson.scm.SCM.checkout(SCM.java:504) > > > at hudson.model.AbstractProject.checkout(AbstractProject.java:1 > > 208) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe > > ckout(AbstractBuild.java:574) > > > at > > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra > > ctBuild.java:499) > > > at hudson.model.Run.execute(Run.java:1794) > > > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > > > at > > > hudson.model.ResourceController.execute(ResourceController.java:97) > > > at hudson.model.Executor.run(Executor.java:429) > > > Caused by: hudson.plugins.git.GitException: Command "git fetch --tags > > > --progress > https://git-wip-us.apache.org/repos/asf/guacamole-client.git > > > +refs/heads/*:refs/remotes/origin/*" returned status code 128: > > > stdout: > > > stderr: remote: Counting objects: 62166, done. > > > remote: Compressing objects: 0% (1/16096) remote: > Compressing > > > objects: 1% (161/16096) remote: Compressing objects: 2% > > > (322/16096) remote: Compressing objects: 3% (483/16096) > > > remote: Compressing objects: 4% (644/16096) remote: > > > Compressing objects: 5% (805/16096) remote: Compressing > > > objects: 6% (966/16096) remote: Compressing objects: 7% > > > (1127/16096) remote: Compressing objects: 8% (1288/16096) > > > remote: Compressing objects: 9% (1449/16096) remote: > > > Compressing objects: 10% (1610/16096) remote: Compressing > > > objects: 11% (1771/16096) remote: Compressing objects: 12% > > > (1932/16096) remote: Compressing objects: 13% (2093/16096) > > > remote: Compressing objects: 14% (2254/16096) remote: > > > Compressing objects: 15% (2415/16096) remote: Compressing > > > objects: 16% (2576/16096) remote: Compressing objects: 17% > > > (2737/16096) remote: Compressing objects: 18% (2898/16096) > > > remote: Compressing objects: 19% (3059/16096) remote: > > > Compressing objects: 20% (3220/16096) remote: Compressing > > > objects: 21% (3381/16096) remote: Compressing objects: 22% > > > (3542/16096) remote: Compressing objects: 23% (3703/16096) > > > remote: Compressing objects: 24% (3864/16096) remote: > > > Compressing objects: 25% (4024/16096) remote: Compressing > > > objects: 26% (4185/16096) remote: Compressing objects: 27% > > > (4346/16096) remote: Compressing objects: 28% (4507/16096) > > > remote: Compressing objects: 29% (4668/16096) remote: > > > Compressing objects: 30% (4829/16096) remote: Compressing > > > objects: 31% (4990/16096) remote: Compressing objects: 32% > > > (5151/16096) remote: Compressing objects: 33% (5312/16096) > > > remote: Compressing objects: 34% (5473/16096) remote: > > > Compressing objects: 35% (5634/16096) remote: Compressing > > > objects: 36% (5795/16096) remote: Compressing objects: 37% > > > (5956/16096) remote: Compressing objects: 38% (6117/16096) > > > remote: Compressing objects: 39% (6278/16096) remote: > > > Compressing objects: 40% (6439/16096) remote: Compressing > > > objects: 41% (6600/16096) remote: Compressing objects: 42% > > > (6761/16096) remote: Compressing objects: 43% (6922/16096) > > > remote: Compressing objects: 44% (7083/16096) remote: > > > Compressing objects: 45% (7244/16096) remote: Compressing > > > objects: 46% (7405/16096) remote: Compressing objects: 47% > > > (7566/16096) remote: Compressing objects: 48% (7727/16096) > > > remote: Compressing objects: 49% (7888/16096) remote: > > > Compressing objects: 50% (8048/16096) remote: Compressing > > > objects: 51% (8209/16096) remote: Compressing objects: 52% > > > (8370/16096) remote: Compressing objects: 53% (8531/16096) > > > remote: Compressing objects: 54% (8692/16096) remote: > > > Compressing objects: 55% (8853/16096) remote: Compressing > > > objects: 56% (9014/16096) remote: Compressing objects: 57% > > > (9175/16096) remote: Compressing objects: 58% (9336/16096) > > > remote: Compressing objects: 59% (9497/16096) remote: > > > Compressing objects: 60% (9658/16096) remote: Compressing > > > objects: 61% (9819/16096) remote: Compressing objects: 62% > > > (9980/16096) remote: Compressing objects: 63% (10141/16096) > > > remote: Compressing objects: 64% (10302/16096) remote: > > > Compressing objects: 65% (10463/16096) remote: Compressing > > > objects: 66% (10624/16096) remote: Compressing objects: 67% > > > (10785/16096) remote: Compressing objects: 68% (10946/16096) > > > remote: Compressing objects: 69% (11107/16096) remote: > > > Compressing objects: 70% (11268/16096) remote: Compressing > > > objects: 71% (11429/16096) remote: Compressing objects: 72% > > > (11590/16096) remote: Compressing objects: 73% (11751/16096) > > > remote: Compressing objects: 74% (11912/16096) remote: > > > Compressing objects: 75% (12072/16096) remote: Compressing > > > objects: 76% (12233/16096) remote: Compressing objects: 77% > > > (12394/16096) remote: Compressing objects: 78% (12555/16096) > > > remote: Compressing objects: 79% (12716/16096) remote: > > > Compressing objects: 80% (12877/16096) remote: Compressing > > > objects: 81% (13038/16096) remote: Compressing objects: 82% > > > (13199/16096) remote: Compressing objects: 83% (13360/16096) > > > remote: Compressing objects: 84% (13521/16096) remote: > > > Compressing objects: 85% (13682/16096) remote: Compressing > > > objects: 86% (13843/16096) remote: Compressing objects: 87% > > > (14004/16096) remote: Compressing objects: 88% (14165/16096) > > > remote: Compressing objects: 89% (14326/16096) remote: > > > Compressing objects: 90% (14487/16096) remote: Compressing > > > objects: 91% (14648/16096) remote: Compressing objects: 92% > > > (14809/16096) remote: Compressing objects: 93% (14970/16096) > > > remote: Compressing objects: 94% (15131/16096) remote: > > > Compressing objects: 95% (15292/16096) remote: Compressing > > > objects: 96% (15453/16096) remote: Compressing objects: 97% > > > (15614/16096) remote: Compressing objects: 98% (15775/16096) > > > remote: Compressing objects: 99% (15936/16096) remote: > > > Compressing objects: 100% (16096/16096) remote: Compressing > > > objects: 100% (16096/16096), done. > > > Receiving objects: 0% (1/62166) fatal: write error: No space left > on > > > device > > > fatal: index-pack failed > > > > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI > > n(CliGitAPIImpl.java:2002) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandW > > ithCredentials(CliGitAPIImpl.java:1721) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300( > > CliGitAPIImpl.java:72) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute( > > CliGitAPIImpl.java:405) > > > at > > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat > > ionHandler$1.call(RemoteGitImpl.java:153) > > > at > > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat > > ionHandler$1.call(RemoteGitImpl.java:146) > > > at hudson.remoting.UserRequest.perform(UserRequest.java:212) > > > at hudson.remoting.UserRequest.perform(UserRequest.java:54) > > > at hudson.remoting.Request$2.run(Request.java:369) > > > at > > > hudson.remoting.InterceptingExecutorService$1.call(Intercept > > ingExecutorService.java:72) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > > Executor.java:1142) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > > lExecutor.java:617) > > > at java.lang.Thread.run(Thread.java:748) > > > Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote > > call > > > to H21 > > > at > > > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > > > at > > > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserR > > equest.java:357) > > > at hudson.remoting.Channel.call(Channel.java:955) > > > at > > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat > > ionHandler.execute(RemoteGitImpl.java:146) > > > at sun.reflect.GeneratedMethodAccessor918.invoke(Unknown > > > Source) > > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > > thodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > at > > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat > > ionHandler.invoke(RemoteGitImpl.java:132) > > > at com.sun.proxy.$Proxy119.execute(Unknown Source) > > > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:886) > > > at > > > hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155) > > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > > > at hudson.scm.SCM.checkout(SCM.java:504) > > > at > > > hudson.model.AbstractProject.checkout(AbstractProject.java:1208) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe > > ckout(AbstractBuild.java:574) > > > at > > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra > > ctBuild.java:499) > > > at hudson.model.Run.execute(Run.java:1794) > > > at hudson.model.FreeStyleBuild.ru > > n(FreeStyleBuild.java:43) > > > at > > > hudson.model.ResourceController.execute(ResourceController.java:97) > > > at hudson.model.Executor.run(Executor.java:429) > > > ERROR: Error fetching remote repo 'origin' > > > Retrying after 10 seconds > > >> git rev-parse --is-inside-work-tree # timeout=10 > > > Fetching changes from the remote Git repository > > >> git config remote.origin.url > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git # > > timeout=10 > > > ERROR: Error fetching remote repo 'origin' > > > hudson.plugins.git.GitException: Failed to fetch from > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git > > > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888) > > > at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155) > > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > > > at hudson.scm.SCM.checkout(SCM.java:504) > > > at hudson.model.AbstractProject.checkout(AbstractProject.java:1 > > 208) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe > > ckout(AbstractBuild.java:574) > > > at > > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra > > ctBuild.java:499) > > > at hudson.model.Run.execute(Run.java:1794) > > > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > > > at > > > hudson.model.ResourceController.execute(ResourceController.java:97) > > > at hudson.model.Executor.run(Executor.java:429) > > > Caused by: hudson.plugins.git.GitException: Command "git config > > > remote.origin.url > > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git"; returned > > > status code 4: > > > stdout: > > > stderr: error: failed to write new configuration file < > > > https://builds.apache.org/job/guacamole-client-coverity/ws/g > > uacamole-client/.git/config.lock > > >> > > > > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI > > n(CliGitAPIImpl.java:2002) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI > > n(CliGitAPIImpl.java:1970) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI > > n(CliGitAPIImpl.java:1966) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand( > > CliGitAPIImpl.java:1597) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand( > > CliGitAPIImpl.java:1609) > > > at > > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl( > > CliGitAPIImpl.java:1243) > > > at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce > > ssorImpl.java:62) > > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > > thodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > at > > > hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(R > > emoteInvocationHandler.java:929) > > > at > > > hudson.remoting.RemoteInvocationHandler$RPCRequest.call(Remo > > teInvocationHandler.java:903) > > > at > > > hudson.remoting.RemoteInvocationHandler$RPCRequest.call(Remo > > teInvocationHandler.java:855) > > > at hudson.remoting.UserRequest.perform(UserRequest.java:212) > > > at hudson.remoting.UserRequest.perform(UserRequest.java:54) > > > at hudson.remoting.Request$2.run(Request.java:369) > > > at > > > hudson.remoting.InterceptingExecutorService$1.call(Intercept > > ingExecutorService.java:72) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > > Executor.java:1142) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > > lExecutor.java:617) > > > at java.lang.Thread.run(Thread.java:748) > > > Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote > > call > > > to H21 > > > at > > > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > > > at > > > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserR > > equest.java:357) > > > at hudson.remoting.Channel.call(Channel.java:955) > > > at > > > hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocat > > ionHandler.java:283) > > > at com.sun.proxy.$Proxy118.setRemoteUrl(Unknown Source) > > > at > > > org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl( > > RemoteGitImpl.java:295) > > > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:876) > > > at > > > hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155) > > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > > > at hudson.scm.SCM.checkout(SCM.java:504) > > > at > > > hudson.model.AbstractProject.checkout(AbstractProject.java:1208) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe > > ckout(AbstractBuild.java:574) > > > at > > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) > > > at > > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra > > ctBuild.java:499) > > > at hudson.model.Run.execute(Run.java:1794) > > > at hudson.model.FreeStyleBuild.ru > > n(FreeStyleBuild.java:43) > > > at > > > hudson.model.ResourceController.execute(ResourceController.java:97) > > > at hudson.model.Executor.run(Executor.java:429) > > > ERROR: Error fetching remote repo 'origin' > > > > >
