[JIRA] (JENKINS-38723) Stash failed with IOException during scalability testing

[alob...@cloudbees.com (JIRA)]

Title: Message Title

Alvaro Lobato created an issue   Jenkins / JENKINS-38723 Stash failed with IOException during scalability testing Issue Type: Improvement Assignee: Unassigned Components: core Created: 2016/Oct/05 8:22 AM Priority: Minor Reporter: Alvaro Lobato During an scalability test, one of the executions failed on stash while executing this pipeline script in a high concurrency environment (15 nodes and 4 executors and a start queue of about 700 jobs) def branch(name) { return { node { sh "sleep 30 && head -c 52428800 /dev/urandom > ${name}.bin" stash includes: "${name}.bin", name: "${name}" } } } stage "Thinking" for (i = 0; i < 5; i++) { sleep time: 250, unit: 'MILLISECONDS' echo "Thinking $i" } stage "Working" def branches = [:] branches["b1"] = branch("b1") branches["b2"] = branch("b2") branches["b3"] = branch("b3") parallel branches stage "Resting" for (i = 0; i < 5; i++) { sleep time: 150, unit: 'MILLISECONDS' echo "Resting $i" } This was the result of the execution: Started [Pipeline] stage (Thinking) Entering stage Thinking Proceeding [Pipeline] sleep Sleeping for 0.25 sec [Pipeline] echo Thinking 0 [Pipeline] sleep Sleeping for 0.25 sec [Pipeline] echo Thinking 1 [Pipeline] sleep Sleeping for 0.25 sec [Pipeline] echo Thinking 2 [Pipeline] sleep Sleeping for 0.25 sec [Pipeline] echo Thinking 3 [Pipeline] sleep Sleeping for 0.25 sec [Pipeline] echo Thinking 4 [Pipeline] stage (Working) Entering stage Working Proceeding [Pipeline] parallel [Pipeline] [b1] { (Branch: b1) [Pipeline] [b2] { (Branch: b2) [Pipeline] [b3] { (Branch: b3) [Pipeline] [b1] node [Pipeline] [b2] node [Pipeline] [b3] node [b1] Still waiting to schedule task [b1] Waiting for next available executor [b2] Still waiting to schedule task [b2] Waiting for next available executor [b3] Still waiting to schedule task [b3] Waiting for next available executor [b1] Running on node-10.0.0.58 in /home/ec2-user/workspace/Pipeline@4 [Pipeline] [b1] { [Pipeline] [b1] sh [b1] [Pipeline@4] Running shell script [b1] + sleep 30 [b2] Running on node-10.0.0.61 in /home/ec2-user/workspace/Pipeline@3 [Pipeline] [b2] { [Pipeline] [b2] sh [b2] [Pipeline@3] Running shell script [b2] + sleep 30 [b1] + head -c 52428800 /dev/urandom [Pipeline] [b1] stash [b2] + head -c 52428800 /dev/urandom [Pipeline] [b2] stash [b1] Stashed 1 file(s) [Pipeline] } [Pipeline] // node [Pipeline] } [b3] Running on EC2_CJOC (i-0695a7fdd8011c959) in /home/ec2-user/workspace/Pipeline@2 [Pipeline] [b3] { [Pipeline] [b3] sh [b3] [Pipeline@2] Running shell script [b3] + sleep 30 [b2] Stashed 1 file(s) [Pipeline] } [Pipeline] // node [Pipeline] } [b3] + head -c 52428800 /dev/urandom [Pipeline] [b3] stash [Pipeline] } [Pipeline] // node [Pipeline] } Failed in branch b3 [Pipeline] // parallel [Pipeline] End of Pipeline java.io.IOException: remote file operation failed: /home/ec2-user/workspace/Pipeline@2 at hudson.remoting.Channel@5f30d359:EC2_CJOC (i-0695a7fdd8011c959): java.io.IOException: This archives contains unclosed entries. at hudson.FilePath.act(FilePath.java:1014) at hudson.FilePath.act(FilePath.java:996) at hudson.FilePath.archive(FilePath.java:459) at org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:90) at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:95) at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:85) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: This archives contains unclosed entries. at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:225) at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:241) at hudson.util.io.TarArchiver.close(TarArchiver.java:111) at hudson.FilePath$1.invoke(FilePath.java:465) at hudson.FilePath$1.invoke(FilePath.java:459) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to EC2_CJOC (i-0695a7fdd8011c959)(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) at hudson.remoting.Channel.call(Channel.java:781) at hudson.FilePath.act(FilePath.java:1007) ... 13 more Finished: FAILURE This was tested on O.S.: Linux on AWS Jenkins 2.7.3 JDK version: JDK8 Add Comment   This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.