Yes, archiving artifacts is very slow, this has apparently always been the case with Jenkins - there are numerous jira issues that have come and been resolved without fixes to this. It is what it is..
As I commented here <https://stackoverflow.com/questions/21268327/rsync-alternative-to-jenkins-copy-artifacts-plugin#comment117378767_25530456>, I'm also seeking massive performance gains by replacing copyArtifact with a shell call in my pipelines. In lieu of installing a proper artifact management system and replacing all archive/copyArtifact with calls to its REST API (which I'll be doing later this year), I'm hoping to find a quick alternative. HTTP/wget/curl is problematic when you want to fetch anything but a single artifact or all artifacts, bc HTTP doesn't support a notion of a directory, so you have to fetch the index and preprocess before fetching what you really want. With scp I could just use unix glob pattern matching to fetch what I desire in a single simple call. HTTP/wget/curl is also problematic bc you have to use Jenkins API tokens and authentication. I'm using ansible to setup my jenkins infra inside a firewalled LAN, and the jenkins users on all nodes are already set up to be able to freely ssh back and forth without password. So, SCP would be a slam dunk for me if I could construct the source path correctly. The problem is that Jenkins uses an algorithm to create a unique folder name, both for workspace names, and for branch job names, but I'm not sure if that's consistent, and therefore I do not know if it would be safe to attempt to re-construct and reference job paths on the controller's disk. E.g. to fetch artifacts from the corresponding branch of an upstream multibranch pipeline job with Full project name of "ProjectFolder/MyProject/feature%2Ffoo", in the downstream multibranch pipeline, I would do something like: scp -r jenkins-controller:<JENKINS_HOME>/jobs/ProjectFolder/jobs/MyProject/branches/ *<HOW_DO_I_COMPUTE_THE_BRANCH_PORTION_OF_PATH?>* /lastSuccessfulBuild/artifact/<GLOB> On Wednesday, May 30, 2018 at 11:02:48 AM UTC-7 ok999 wrote: > Copying files from jenkins workspace , is always slow. I have seen that > in the past. > > If u r using jenkins to checkout an artifact, and then copy to a path on a > remote. Just use wget/rsync (as Mark suggested) . U can trigger that from > jenkins too > > On Wed, May 30, 2018 at 6:12 AM Mark Waite <mark.ea...@gmail.com> wrote: > >> I thought I remembered reading advice that archiving artifacts was known >> to be slow with large files. I couldn't find the reference, so I may be >> incorrect in this case. >> >> I'd suggest using a different technique to store your large artifacts, >> rather than having Jenkins perform the copy. If on Unix, consider rsync or >> other copy program. If on Windows, consider robocopy. >> >> Mark Waite >> >> On Wed, May 30, 2018 at 1:44 AM panpaliamahen < >> mahendra...@buhlergroup.com> wrote: >> >>> Hi, >>> >>> I am using Jenkins version 2.107.3, latest one and on windows. I am using >>> Jenkins Slave node to run a Job which require to copy an artefact of size >>> ~6GH. And we found when jenkins copy this it takes ~10 minutes. (Note >>> artefact are copied from master). >>> >>> Also observed it takes ~10 minutes when jenkins job archived back to >>> master. >>> >>> Where as same can be copied in ~1 minute in same network without >>> Jenkins. >>> >>> Please can someone help me? >>> What should I do? >>> Do I need to install/upgrade any additional plugin on Jenkins? >>> >>> Please it is very urgent and reducing efficiency with in our development >>> teams. >>> >>> Thanks and regards, >>> Mahendra >>> >>> >>> >>> >>> -- >>> Sent from: >>> http://jenkins-ci.361315.n4.nabble.com/Jenkins-users-f361316.html >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jenkinsci-use...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/1527665457279-0.post%40n4.nabble.com >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-use...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHYp8p3civwUYu4_pAyi63PB4O8exWjt4YS0qimgLa8tg%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHYp8p3civwUYu4_pAyi63PB4O8exWjt4YS0qimgLa8tg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . > > >> For more options, visit https://groups.google.com/d/optout. >> > -- > Sent from mobile device, excuse typos if any. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/203463c9-0c2d-4d5f-a9c0-0fb77aae41d1n%40googlegroups.com.
