Oh, and I'm not trying to avoid your question about auto-removal. I don't have that answer, and will leave it for others.
On Sun, Sep 30, 2018 at 11:22 PM Greg Stein <gst...@gmail.com> wrote: > We're refining our monitoring and some cron jobs to try and keep the disk > space pruned. But when space balloons due to poor cleanup, there is only so > much we can do. > > The short answer is that the multitude of groups at the Foundation using > these shared resources need to be cognizant of that fact, and cooperate on > ways to keep them available for all. We've seen disk space problems, too > many jobs, too-long jobs, etc. > > Infra will keep working and improving our management of the build nodes. > In the meantime, please keep reporting problems, and we'll do our best to > keep the system up for y'all. > > Cheers, > Greg > InfraAdmin, ASF > > > On Sun, Sep 30, 2018 at 10:54 PM Mike Jumper <mjum...@apache.org> wrote: > >> Is there any way Jenkins could be configured to take nodes offline >> automatically when free space is below a certain threshold? Granted >> space-hungry builds and proper cleanup remain a problem, but at least >> there >> wouldn't be a random chance of jobs failing due to being assigned to a >> full >> node. >> >> - Mike >> >> >> On Sun, Sep 30, 2018 at 3:57 PM, Chris Lambertus <c...@apache.org> wrote: >> >> > The main culprit on H31 appears to be pulsar ws-cleanup jobs consuming >> > almost 40% of the disk. (The problem on H21 was something >> docker-related.) >> > >> > >> > This is the second project I have found (Brooklyn was the first) which >> is >> > leaving ws-cleanup directories all over the workspace and filling up the >> > disks. Each one of these was 500+ MB. Pulsar, please address ASAP. >> > >> > >> > >> > >> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:19 >> > pulsar_precommit_cpp@2_ws-cleanup_1535737112280 >> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:18 >> > pulsar_precommit_cpp@2_ws-cleanup_1536174576053 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 06:04 >> > pulsar_precommit_cpp@2_ws-cleanup_1537292379178 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 22:45 >> > pulsar_precommit_cpp@2_ws-cleanup_1537397120862 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45 >> > pulsar_precommit_cpp@2_ws-cleanup_1537403378279 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 21 23:31 >> > pulsar_precommit_cpp@2_ws-cleanup_1537572701829 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 26 16:04 >> > pulsar_precommit_cpp@2_ws-cleanup_1537977879341 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 27 22:23 >> > pulsar_precommit_cpp@2_ws-cleanup_1538087035027 >> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:14 >> > pulsar_precommit_cpp_ws-cleanup_1535699830230 >> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:16 >> > pulsar_precommit_cpp_ws-cleanup_1535735511423 >> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:13 >> > pulsar_precommit_cpp_ws-cleanup_1535786064397 >> > drwxr-xr-x 52 jenkins jenkins 4096 Sep 19 06:18 >> > pulsar_precommit_cpp_ws-cleanup_1536096963586 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 21 06:47 >> > pulsar_precommit_cpp_ws-cleanup_1536114938910 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 21 06:45 >> > pulsar_precommit_cpp_ws-cleanup_1536173165909 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:38 >> > pulsar_precommit_cpp_ws-cleanup_1536175584418 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:50 >> > pulsar_precommit_cpp_ws-cleanup_1536180707506 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:38 >> > pulsar_precommit_cpp_ws-cleanup_1536182927940 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:51 >> > pulsar_precommit_cpp_ws-cleanup_1536185193716 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 22 05:50 >> > pulsar_precommit_cpp_ws-cleanup_1536273709029 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 05:59 >> > pulsar_precommit_cpp_ws-cleanup_1536316964794 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 06:06 >> > pulsar_precommit_cpp_ws-cleanup_1536675650893 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 28 06:08 >> > pulsar_precommit_cpp_ws-cleanup_1536879616962 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 30 19:53 >> > pulsar_precommit_cpp_ws-cleanup_1536891056968 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 30 19:47 >> > pulsar_precommit_cpp_ws-cleanup_1536966466683 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 18 00:43 >> > pulsar_precommit_cpp_ws-cleanup_1537216387387 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 00:45 >> > pulsar_precommit_cpp_ws-cleanup_1537292309413 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 19 00:45 >> > pulsar_precommit_cpp_ws-cleanup_1537308526251 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45 >> > pulsar_precommit_cpp_ws-cleanup_1537331780420 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45 >> > pulsar_precommit_cpp_ws-cleanup_1537379886715 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:46 >> > pulsar_precommit_cpp_ws-cleanup_1537386197893 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:46 >> > pulsar_precommit_cpp_ws-cleanup_1537397073464 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:46 >> > pulsar_precommit_cpp_ws-cleanup_1537403247575 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:43 >> > pulsar_precommit_cpp_ws-cleanup_1537413954888 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:45 >> > pulsar_precommit_cpp_ws-cleanup_1537417322151 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:45 >> > pulsar_precommit_cpp_ws-cleanup_1537421381129 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:44 >> > pulsar_precommit_cpp_ws-cleanup_1537469348852 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 00:45 >> > pulsar_precommit_cpp_ws-cleanup_1537489399468 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 22 00:44 >> > pulsar_precommit_cpp_ws-cleanup_1537566528660 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 21 23:12 >> > pulsar_precommit_cpp_ws-cleanup_1537571549978 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 22 00:44 >> > pulsar_precommit_cpp_ws-cleanup_1537573605258 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 26 00:43 >> > pulsar_precommit_cpp_ws-cleanup_1537855939828 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 25 11:18 >> > pulsar_precommit_cpp_ws-cleanup_1537874305575 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 25 12:41 >> > pulsar_precommit_cpp_ws-cleanup_1537879296622 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 26 01:39 >> > pulsar_precommit_cpp_ws-cleanup_1537925955967 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 26 13:17 >> > pulsar_precommit_cpp_ws-cleanup_1537967826838 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 26 15:55 >> > pulsar_precommit_cpp_ws-cleanup_1537977326736 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 26 17:34 >> > pulsar_precommit_cpp_ws-cleanup_1537983268177 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 27 22:04 >> > pulsar_precommit_cpp_ws-cleanup_1538085846793 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 28 00:20 >> > pulsar_precommit_cpp_ws-cleanup_1538094024821 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 28 17:42 >> > pulsar_precommit_cpp_ws-cleanup_1538156543343 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 06:02 >> > pulsar_precommit_integrationtests_ws-cleanup_1536298125841 >> > drwxr-xr-x 53 jenkins jenkins 4096 Sep 25 06:00 >> > pulsar_precommit_integrationtests_ws-cleanup_1536968128660 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 18 00:43 >> > pulsar_precommit_integrationtests_ws-cleanup_1537218541598 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 20 00:45 >> > pulsar_precommit_integrationtests_ws-cleanup_1537331703324 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 22 00:43 >> > pulsar_precommit_integrationtests_ws-cleanup_1537553748859 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 24 00:43 >> > pulsar_precommit_integrationtests_ws-cleanup_1537730402937 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 24 07:48 >> > pulsar_precommit_integrationtests_ws-cleanup_1537775336586 >> > drwxr-xr-x 54 jenkins jenkins 4096 Sep 24 23:30 >> > pulsar_precommit_integrationtests_ws-cleanup_1537831851711 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 25 23:08 >> > pulsar_precommit_integrationtests_ws-cleanup_1537916928485 >> > drwxr-xr-x 55 jenkins jenkins 4096 Sep 27 15:21 >> > pulsar_precommit_integrationtests_ws-cleanup_1538061702243 >> > >> > >> > -Chris >> > ASF Infra >> > >> > >> > >> > >> > >> > > On Sep 30, 2018, at 3:12 PM, Chris Thistlethwaite < >> > cthistl...@outlook.com> wrote: >> > > >> > > I've been working on H21 and H31. They both had a some docker issues >> > where the process was locked and I couldn't do anything with a few >> > containers that were over 8 days old. >> > > >> > > Thought we might be good this morning so I threw them back into >> > rotation. Guess not, so I'll take them back out. >> > > >> > > -Chris T. >> > > ________________________________ >> > > From: Mike Jumper <mjum...@apache.org> >> > > Sent: Sunday, September 30, 2018 6:03:51 PM >> > > To: builds@apache.org >> > > Subject: No space on H21 (Was: Fwd: Build failed in Jenkins: >> > guacamole-client-coverity #73) >> > > >> > > FYI - we're currently seeing disk space issues on H21. >> > > >> > > - Mike >> > > >> > > >> > > ---------- Forwarded message --------- >> > > From: Apache Jenkins Server <jenk...@builds.apache.org> >> > > Date: Sun, Sep 30, 2018, 14:37 >> > > Subject: Build failed in Jenkins: guacamole-client-coverity #73 >> > > To: <d...@guacamole.apache.org> >> > > >> > > >> > > See < >> > > https://builds.apache.org/job/guacamole-client-coverity/73/d >> > isplay/redirect> >> > > >> > > ------------------------------------------ >> > > Started by user mjumper >> > > [EnvInject] - Loading node environment variables. >> > > Building remotely on H21 (ubuntu xenial) in workspace < >> > > https://builds.apache.org/job/guacamole-client-coverity/ws/> >> > > [WS-CLEANUP] Deleting project workspace... >> > > Cloning the remote Git repository >> > > Cloning repository >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git >> > >> git init < >> > > https://builds.apache.org/job/guacamole-client-coverity/ws/g >> > uacamole-client> >> > > # timeout=10 >> > > Fetching upstream changes from >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git >> > >> git --version # timeout=10 >> > >> git fetch --tags --progress >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git >> > > +refs/heads/*:refs/remotes/origin/* >> > > ERROR: Error cloning remote repo 'origin' >> > > hudson.plugins.git.GitException: Command "git fetch --tags --progress >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git >> > > +refs/heads/*:refs/remotes/origin/*" returned status code 128: >> > > stdout: >> > > stderr: remote: Counting objects: 62166, done. >> > > remote: Compressing objects: 0% (1/16096) remote: >> Compressing >> > > objects: 1% (161/16096) remote: Compressing objects: 2% >> > > (322/16096) remote: Compressing objects: 3% (483/16096) >> > > remote: Compressing objects: 4% (644/16096) remote: >> > > Compressing objects: 5% (805/16096) remote: Compressing >> > > objects: 6% (966/16096) remote: Compressing objects: 7% >> > > (1127/16096) remote: Compressing objects: 8% (1288/16096) >> > > remote: Compressing objects: 9% (1449/16096) remote: >> > > Compressing objects: 10% (1610/16096) remote: Compressing >> > > objects: 11% (1771/16096) remote: Compressing objects: 12% >> > > (1932/16096) remote: Compressing objects: 13% (2093/16096) >> > > remote: Compressing objects: 14% (2254/16096) remote: >> > > Compressing objects: 15% (2415/16096) remote: Compressing >> > > objects: 16% (2576/16096) remote: Compressing objects: 17% >> > > (2737/16096) remote: Compressing objects: 18% (2898/16096) >> > > remote: Compressing objects: 19% (3059/16096) remote: >> > > Compressing objects: 20% (3220/16096) remote: Compressing >> > > objects: 21% (3381/16096) remote: Compressing objects: 22% >> > > (3542/16096) remote: Compressing objects: 23% (3703/16096) >> > > remote: Compressing objects: 24% (3864/16096) remote: >> > > Compressing objects: 25% (4024/16096) remote: Compressing >> > > objects: 26% (4185/16096) remote: Compressing objects: 27% >> > > (4346/16096) remote: Compressing objects: 28% (4507/16096) >> > > remote: Compressing objects: 29% (4668/16096) remote: >> > > Compressing objects: 30% (4829/16096) remote: Compressing >> > > objects: 31% (4990/16096) remote: Compressing objects: 32% >> > > (5151/16096) remote: Compressing objects: 33% (5312/16096) >> > > remote: Compressing objects: 34% (5473/16096) remote: >> > > Compressing objects: 35% (5634/16096) remote: Compressing >> > > objects: 36% (5795/16096) remote: Compressing objects: 37% >> > > (5956/16096) remote: Compressing objects: 38% (6117/16096) >> > > remote: Compressing objects: 39% (6278/16096) remote: >> > > Compressing objects: 40% (6439/16096) remote: Compressing >> > > objects: 41% (6600/16096) remote: Compressing objects: 42% >> > > (6761/16096) remote: Compressing objects: 43% (6922/16096) >> > > remote: Compressing objects: 44% (7083/16096) remote: >> > > Compressing objects: 45% (7244/16096) remote: Compressing >> > > objects: 46% (7405/16096) remote: Compressing objects: 47% >> > > (7566/16096) remote: Compressing objects: 48% (7727/16096) >> > > remote: Compressing objects: 49% (7888/16096) remote: >> > > Compressing objects: 50% (8048/16096) remote: Compressing >> > > objects: 51% (8209/16096) remote: Compressing objects: 52% >> > > (8370/16096) remote: Compressing objects: 53% (8531/16096) >> > > remote: Compressing objects: 54% (8692/16096) remote: >> > > Compressing objects: 55% (8853/16096) remote: Compressing >> > > objects: 56% (9014/16096) remote: Compressing objects: 57% >> > > (9175/16096) remote: Compressing objects: 58% (9336/16096) >> > > remote: Compressing objects: 59% (9497/16096) remote: >> > > Compressing objects: 60% (9658/16096) remote: Compressing >> > > objects: 61% (9819/16096) remote: Compressing objects: 62% >> > > (9980/16096) remote: Compressing objects: 63% (10141/16096) >> > > remote: Compressing objects: 64% (10302/16096) remote: >> > > Compressing objects: 65% (10463/16096) remote: Compressing >> > > objects: 66% (10624/16096) remote: Compressing objects: >> 67% >> > > (10785/16096) remote: Compressing objects: 68% >> (10946/16096) >> > > remote: Compressing objects: 69% (11107/16096) >> remote: >> > > Compressing objects: 70% (11268/16096) remote: Compressing >> > > objects: 71% (11429/16096) remote: Compressing objects: >> 72% >> > > (11590/16096) remote: Compressing objects: 73% >> (11751/16096) >> > > remote: Compressing objects: 74% (11912/16096) >> remote: >> > > Compressing objects: 75% (12072/16096) remote: Compressing >> > > objects: 76% (12233/16096) remote: Compressing objects: >> 77% >> > > (12394/16096) remote: Compressing objects: 78% >> (12555/16096) >> > > remote: Compressing objects: 79% (12716/16096) >> remote: >> > > Compressing objects: 80% (12877/16096) remote: Compressing >> > > objects: 81% (13038/16096) remote: Compressing objects: >> 82% >> > > (13199/16096) remote: Compressing objects: 83% >> (13360/16096) >> > > remote: Compressing objects: 84% (13521/16096) >> remote: >> > > Compressing objects: 85% (13682/16096) remote: Compressing >> > > objects: 86% (13843/16096) remote: Compressing objects: >> 87% >> > > (14004/16096) remote: Compressing objects: 88% >> (14165/16096) >> > > remote: Compressing objects: 89% (14326/16096) >> remote: >> > > Compressing objects: 90% (14487/16096) remote: Compressing >> > > objects: 91% (14648/16096) remote: Compressing objects: >> 92% >> > > (14809/16096) remote: Compressing objects: 93% >> (14970/16096) >> > > remote: Compressing objects: 94% (15131/16096) >> remote: >> > > Compressing objects: 95% (15292/16096) remote: Compressing >> > > objects: 96% (15453/16096) remote: Compressing objects: >> 97% >> > > (15614/16096) remote: Compressing objects: 98% >> (15775/16096) >> > > remote: Compressing objects: 99% (15936/16096) >> remote: >> > > Compressing objects: 100% (16096/16096) remote: Compressing >> > > objects: 100% (16096/16096), done. >> > > Receiving objects: 0% (1/62166) Receiving objects: 1% >> (622/62166) >> > > Receiving objects: 2% (1244/62166) Receiving objects: 3% >> > > (1865/62166) Receiving objects: 4% (2487/62166) Receiving >> objects: >> > > 5% (3109/62166) Receiving objects: 6% (3730/62166) Receiving >> > > objects: 7% (4352/62166) Receiving objects: 8% (4974/62166) >> > > Receiving objects: 9% (5595/62166) Receiving objects: 10% >> > > (6217/62166) Receiving objects: 11% (6839/62166) Receiving >> objects: >> > > 12% (7460/62166) Receiving objects: 13% (8082/62166) Receiving >> > > objects: 14% (8704/62166) Receiving objects: 15% (9325/62166) >> > > Receiving objects: 16% (9947/62166) Receiving objects: 17% >> > > (10569/62166) Receiving objects: 18% (11190/62166) Receiving >> > objects: >> > > 19% (11812/62166) Receiving objects: 20% (12434/62166) Receiving >> > > objects: 21% (13055/62166) Receiving objects: 22% (13677/62166) >> > > fatal: write error: No space left on device >> > > fatal: index-pack failed >> > > >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI >> > n(CliGitAPIImpl.java:2002) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandW >> > ithCredentials(CliGitAPIImpl.java:1721) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300( >> > CliGitAPIImpl.java:72) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute( >> > CliGitAPIImpl.java:405) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute( >> > CliGitAPIImpl.java:614) >> > > at >> > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat >> > ionHandler$1.call(RemoteGitImpl.java:153) >> > > at >> > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat >> > ionHandler$1.call(RemoteGitImpl.java:146) >> > > at hudson.remoting.UserRequest.perform(UserRequest.java:212) >> > > at hudson.remoting.UserRequest.perform(UserRequest.java:54) >> > > at hudson.remoting.Request$2.run(Request.java:369) >> > > at >> > > hudson.remoting.InterceptingExecutorService$1.call(Intercept >> > ingExecutorService.java:72) >> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> > > at >> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> > Executor.java:1142) >> > > at >> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> > lExecutor.java:617) >> > > at java.lang.Thread.run(Thread.java:748) >> > > Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote >> > call >> > > to H21 >> > > at >> > > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) >> > > at >> > > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserR >> > equest.java:357) >> > > at hudson.remoting.Channel.call(Channel.java:955) >> > > at >> > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat >> > ionHandler.execute(RemoteGitImpl.java:146) >> > > at >> sun.reflect.GeneratedMethodAccessor918.invoke(Unknown >> > > Source) >> > > at >> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> > thodAccessorImpl.java:43) >> > > at java.lang.reflect.Method.invoke(Method.java:498) >> > > at >> > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat >> > ionHandler.invoke(RemoteGitImpl.java:132) >> > > at com.sun.proxy.$Proxy142.execute(Unknown Source) >> > > at >> > > hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1146) >> > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) >> > > at hudson.scm.SCM.checkout(SCM.java:504) >> > > at >> > > hudson.model.AbstractProject.checkout(AbstractProject.java:1208) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe >> > ckout(AbstractBuild.java:574) >> > > at >> > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra >> > ctBuild.java:499) >> > > at hudson.model.Run.execute(Run.java:1794) >> > > at hudson.model.FreeStyleBuild.ru >> > n(FreeStyleBuild.java:43) >> > > at >> > > hudson.model.ResourceController.execute(ResourceController.java:97) >> > > at hudson.model.Executor.run(Executor.java:429) >> > > ERROR: Error cloning remote repo 'origin' >> > > Retrying after 10 seconds >> > >> git rev-parse --is-inside-work-tree # timeout=10 >> > > Fetching changes from the remote Git repository >> > >> git config remote.origin.url >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git # >> > timeout=10 >> > > Fetching upstream changes from >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git >> > >> git --version # timeout=10 >> > >> git fetch --tags --progress >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git >> > > +refs/heads/*:refs/remotes/origin/* >> > > ERROR: Error fetching remote repo 'origin' >> > > hudson.plugins.git.GitException: Failed to fetch from >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git >> > > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888) >> > > at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155) >> > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) >> > > at hudson.scm.SCM.checkout(SCM.java:504) >> > > at hudson.model.AbstractProject.checkout(AbstractProject.java:1 >> > 208) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe >> > ckout(AbstractBuild.java:574) >> > > at >> > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra >> > ctBuild.java:499) >> > > at hudson.model.Run.execute(Run.java:1794) >> > > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >> > > at >> > > hudson.model.ResourceController.execute(ResourceController.java:97) >> > > at hudson.model.Executor.run(Executor.java:429) >> > > Caused by: hudson.plugins.git.GitException: Command "git fetch --tags >> > > --progress >> https://git-wip-us.apache.org/repos/asf/guacamole-client.git >> > > +refs/heads/*:refs/remotes/origin/*" returned status code 128: >> > > stdout: >> > > stderr: remote: Counting objects: 62166, done. >> > > remote: Compressing objects: 0% (1/16096) remote: >> Compressing >> > > objects: 1% (161/16096) remote: Compressing objects: 2% >> > > (322/16096) remote: Compressing objects: 3% (483/16096) >> > > remote: Compressing objects: 4% (644/16096) remote: >> > > Compressing objects: 5% (805/16096) remote: Compressing >> > > objects: 6% (966/16096) remote: Compressing objects: 7% >> > > (1127/16096) remote: Compressing objects: 8% (1288/16096) >> > > remote: Compressing objects: 9% (1449/16096) remote: >> > > Compressing objects: 10% (1610/16096) remote: Compressing >> > > objects: 11% (1771/16096) remote: Compressing objects: 12% >> > > (1932/16096) remote: Compressing objects: 13% (2093/16096) >> > > remote: Compressing objects: 14% (2254/16096) remote: >> > > Compressing objects: 15% (2415/16096) remote: Compressing >> > > objects: 16% (2576/16096) remote: Compressing objects: 17% >> > > (2737/16096) remote: Compressing objects: 18% (2898/16096) >> > > remote: Compressing objects: 19% (3059/16096) remote: >> > > Compressing objects: 20% (3220/16096) remote: Compressing >> > > objects: 21% (3381/16096) remote: Compressing objects: 22% >> > > (3542/16096) remote: Compressing objects: 23% (3703/16096) >> > > remote: Compressing objects: 24% (3864/16096) remote: >> > > Compressing objects: 25% (4024/16096) remote: Compressing >> > > objects: 26% (4185/16096) remote: Compressing objects: 27% >> > > (4346/16096) remote: Compressing objects: 28% (4507/16096) >> > > remote: Compressing objects: 29% (4668/16096) remote: >> > > Compressing objects: 30% (4829/16096) remote: Compressing >> > > objects: 31% (4990/16096) remote: Compressing objects: 32% >> > > (5151/16096) remote: Compressing objects: 33% (5312/16096) >> > > remote: Compressing objects: 34% (5473/16096) remote: >> > > Compressing objects: 35% (5634/16096) remote: Compressing >> > > objects: 36% (5795/16096) remote: Compressing objects: 37% >> > > (5956/16096) remote: Compressing objects: 38% (6117/16096) >> > > remote: Compressing objects: 39% (6278/16096) remote: >> > > Compressing objects: 40% (6439/16096) remote: Compressing >> > > objects: 41% (6600/16096) remote: Compressing objects: 42% >> > > (6761/16096) remote: Compressing objects: 43% (6922/16096) >> > > remote: Compressing objects: 44% (7083/16096) remote: >> > > Compressing objects: 45% (7244/16096) remote: Compressing >> > > objects: 46% (7405/16096) remote: Compressing objects: 47% >> > > (7566/16096) remote: Compressing objects: 48% (7727/16096) >> > > remote: Compressing objects: 49% (7888/16096) remote: >> > > Compressing objects: 50% (8048/16096) remote: Compressing >> > > objects: 51% (8209/16096) remote: Compressing objects: 52% >> > > (8370/16096) remote: Compressing objects: 53% (8531/16096) >> > > remote: Compressing objects: 54% (8692/16096) remote: >> > > Compressing objects: 55% (8853/16096) remote: Compressing >> > > objects: 56% (9014/16096) remote: Compressing objects: 57% >> > > (9175/16096) remote: Compressing objects: 58% (9336/16096) >> > > remote: Compressing objects: 59% (9497/16096) remote: >> > > Compressing objects: 60% (9658/16096) remote: Compressing >> > > objects: 61% (9819/16096) remote: Compressing objects: 62% >> > > (9980/16096) remote: Compressing objects: 63% (10141/16096) >> > > remote: Compressing objects: 64% (10302/16096) remote: >> > > Compressing objects: 65% (10463/16096) remote: Compressing >> > > objects: 66% (10624/16096) remote: Compressing objects: >> 67% >> > > (10785/16096) remote: Compressing objects: 68% >> (10946/16096) >> > > remote: Compressing objects: 69% (11107/16096) >> remote: >> > > Compressing objects: 70% (11268/16096) remote: Compressing >> > > objects: 71% (11429/16096) remote: Compressing objects: >> 72% >> > > (11590/16096) remote: Compressing objects: 73% >> (11751/16096) >> > > remote: Compressing objects: 74% (11912/16096) >> remote: >> > > Compressing objects: 75% (12072/16096) remote: Compressing >> > > objects: 76% (12233/16096) remote: Compressing objects: >> 77% >> > > (12394/16096) remote: Compressing objects: 78% >> (12555/16096) >> > > remote: Compressing objects: 79% (12716/16096) >> remote: >> > > Compressing objects: 80% (12877/16096) remote: Compressing >> > > objects: 81% (13038/16096) remote: Compressing objects: >> 82% >> > > (13199/16096) remote: Compressing objects: 83% >> (13360/16096) >> > > remote: Compressing objects: 84% (13521/16096) >> remote: >> > > Compressing objects: 85% (13682/16096) remote: Compressing >> > > objects: 86% (13843/16096) remote: Compressing objects: >> 87% >> > > (14004/16096) remote: Compressing objects: 88% >> (14165/16096) >> > > remote: Compressing objects: 89% (14326/16096) >> remote: >> > > Compressing objects: 90% (14487/16096) remote: Compressing >> > > objects: 91% (14648/16096) remote: Compressing objects: >> 92% >> > > (14809/16096) remote: Compressing objects: 93% >> (14970/16096) >> > > remote: Compressing objects: 94% (15131/16096) >> remote: >> > > Compressing objects: 95% (15292/16096) remote: Compressing >> > > objects: 96% (15453/16096) remote: Compressing objects: >> 97% >> > > (15614/16096) remote: Compressing objects: 98% >> (15775/16096) >> > > remote: Compressing objects: 99% (15936/16096) >> remote: >> > > Compressing objects: 100% (16096/16096) remote: Compressing >> > > objects: 100% (16096/16096), done. >> > > Receiving objects: 0% (1/62166) fatal: write error: No space left >> on >> > > device >> > > fatal: index-pack failed >> > > >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI >> > n(CliGitAPIImpl.java:2002) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandW >> > ithCredentials(CliGitAPIImpl.java:1721) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300( >> > CliGitAPIImpl.java:72) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute( >> > CliGitAPIImpl.java:405) >> > > at >> > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat >> > ionHandler$1.call(RemoteGitImpl.java:153) >> > > at >> > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat >> > ionHandler$1.call(RemoteGitImpl.java:146) >> > > at hudson.remoting.UserRequest.perform(UserRequest.java:212) >> > > at hudson.remoting.UserRequest.perform(UserRequest.java:54) >> > > at hudson.remoting.Request$2.run(Request.java:369) >> > > at >> > > hudson.remoting.InterceptingExecutorService$1.call(Intercept >> > ingExecutorService.java:72) >> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> > > at >> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> > Executor.java:1142) >> > > at >> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> > lExecutor.java:617) >> > > at java.lang.Thread.run(Thread.java:748) >> > > Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote >> > call >> > > to H21 >> > > at >> > > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) >> > > at >> > > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserR >> > equest.java:357) >> > > at hudson.remoting.Channel.call(Channel.java:955) >> > > at >> > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat >> > ionHandler.execute(RemoteGitImpl.java:146) >> > > at >> sun.reflect.GeneratedMethodAccessor918.invoke(Unknown >> > > Source) >> > > at >> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> > thodAccessorImpl.java:43) >> > > at java.lang.reflect.Method.invoke(Method.java:498) >> > > at >> > > org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocat >> > ionHandler.invoke(RemoteGitImpl.java:132) >> > > at com.sun.proxy.$Proxy119.execute(Unknown Source) >> > > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:886) >> > > at >> > > hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155) >> > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) >> > > at hudson.scm.SCM.checkout(SCM.java:504) >> > > at >> > > hudson.model.AbstractProject.checkout(AbstractProject.java:1208) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe >> > ckout(AbstractBuild.java:574) >> > > at >> > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra >> > ctBuild.java:499) >> > > at hudson.model.Run.execute(Run.java:1794) >> > > at hudson.model.FreeStyleBuild.ru >> > n(FreeStyleBuild.java:43) >> > > at >> > > hudson.model.ResourceController.execute(ResourceController.java:97) >> > > at hudson.model.Executor.run(Executor.java:429) >> > > ERROR: Error fetching remote repo 'origin' >> > > Retrying after 10 seconds >> > >> git rev-parse --is-inside-work-tree # timeout=10 >> > > Fetching changes from the remote Git repository >> > >> git config remote.origin.url >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git # >> > timeout=10 >> > > ERROR: Error fetching remote repo 'origin' >> > > hudson.plugins.git.GitException: Failed to fetch from >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git >> > > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888) >> > > at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155) >> > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) >> > > at hudson.scm.SCM.checkout(SCM.java:504) >> > > at hudson.model.AbstractProject.checkout(AbstractProject.java:1 >> > 208) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe >> > ckout(AbstractBuild.java:574) >> > > at >> > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra >> > ctBuild.java:499) >> > > at hudson.model.Run.execute(Run.java:1794) >> > > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >> > > at >> > > hudson.model.ResourceController.execute(ResourceController.java:97) >> > > at hudson.model.Executor.run(Executor.java:429) >> > > Caused by: hudson.plugins.git.GitException: Command "git config >> > > remote.origin.url >> > > https://git-wip-us.apache.org/repos/asf/guacamole-client.git"; >> returned >> > > status code 4: >> > > stdout: >> > > stderr: error: failed to write new configuration file < >> > > https://builds.apache.org/job/guacamole-client-coverity/ws/g >> > uacamole-client/.git/config.lock >> > >> >> > > >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI >> > n(CliGitAPIImpl.java:2002) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI >> > n(CliGitAPIImpl.java:1970) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandI >> > n(CliGitAPIImpl.java:1966) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand( >> > CliGitAPIImpl.java:1597) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand( >> > CliGitAPIImpl.java:1609) >> > > at >> > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl( >> > CliGitAPIImpl.java:1243) >> > > at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160) >> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > > at >> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >> > ssorImpl.java:62) >> > > at >> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> > thodAccessorImpl.java:43) >> > > at java.lang.reflect.Method.invoke(Method.java:498) >> > > at >> > > hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(R >> > emoteInvocationHandler.java:929) >> > > at >> > > hudson.remoting.RemoteInvocationHandler$RPCRequest.call(Remo >> > teInvocationHandler.java:903) >> > > at >> > > hudson.remoting.RemoteInvocationHandler$RPCRequest.call(Remo >> > teInvocationHandler.java:855) >> > > at hudson.remoting.UserRequest.perform(UserRequest.java:212) >> > > at hudson.remoting.UserRequest.perform(UserRequest.java:54) >> > > at hudson.remoting.Request$2.run(Request.java:369) >> > > at >> > > hudson.remoting.InterceptingExecutorService$1.call(Intercept >> > ingExecutorService.java:72) >> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> > > at >> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> > Executor.java:1142) >> > > at >> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> > lExecutor.java:617) >> > > at java.lang.Thread.run(Thread.java:748) >> > > Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote >> > call >> > > to H21 >> > > at >> > > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) >> > > at >> > > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserR >> > equest.java:357) >> > > at hudson.remoting.Channel.call(Channel.java:955) >> > > at >> > > hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocat >> > ionHandler.java:283) >> > > at com.sun.proxy.$Proxy118.setRemoteUrl(Unknown Source) >> > > at >> > > org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl( >> > RemoteGitImpl.java:295) >> > > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:876) >> > > at >> > > hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155) >> > > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) >> > > at hudson.scm.SCM.checkout(SCM.java:504) >> > > at >> > > hudson.model.AbstractProject.checkout(AbstractProject.java:1208) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.defaultChe >> > ckout(AbstractBuild.java:574) >> > > at >> > > jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) >> > > at >> > > hudson.model.AbstractBuild$AbstractBuildExecution.run(Abstra >> > ctBuild.java:499) >> > > at hudson.model.Run.execute(Run.java:1794) >> > > at hudson.model.FreeStyleBuild.ru >> > n(FreeStyleBuild.java:43) >> > > at >> > > hudson.model.ResourceController.execute(ResourceController.java:97) >> > > at hudson.model.Executor.run(Executor.java:429) >> > > ERROR: Error fetching remote repo 'origin' >> > >> > >> >
