Yes, archiving artifacts is very slow, this has apparently always been the case with Jenkins - there are numerous jira issues that have come and been resolved without fixes to this. It is what it is..
As I commented here <https://stackoverflow.com/questions/21268327/rsync-alternative-to-jenkins-copy-artifacts-plugin#comment117378767_25530456>, I'm also seeking massive performance gains by replacing copyArtifact with a shell call in my pipelines. In lieu of installing a proper artifact management system and replacing all archive/copyArtifact with calls to its REST API (which I'll be doing later this year), I'm hoping to find a quick alternative. HTTP/wget/curl is problematic when you want to fetch anything but a single artifact or all artifacts, bc HTTP doesn't support a notion of a directory, so you have to fetch the index and preprocess before fetching what you really want. With scp I could just use unix glob pattern matching to fetch what I desire in a single simple call. HTTP/wget/curl is also problematic bc you have to use Jenkins API tokens and authentication. I'm using ansible to setup my jenkins infra inside a firewalled LAN, and the jenkins users on all nodes are already set up to be able to freely ssh back and forth without password. So, SCP would be a slam dunk for me if I could construct the source path correctly. The problem is that Jenkins uses an algorithm to create a unique folder name, both for workspace names, and for branch job names, but I'm not sure if that's consistent, and therefore I do not know if it would be safe to attempt to re-construct and reference job paths on the controller's disk. E.g. to fetch artifacts from the corresponding branch of an upstream multibranch pipeline job with Full project name of "ProjectFolder/MyProject/feature%2Ffoo", in the downstream multibranch pipeline, I would do something like: scp -r jenkins-controller:<JENKINS_HOME>/jobs/ProjectFolder/jobs/MyProject/branches/ *<HOW_DO_I_COMPUTE_THE_BRANCH_PORTION_OF_PATH?>* /lastSuccessfulBuild/artifact/<GLOB> On Wednesday, May 30, 2018 at 11:02:48 AM UTC-7 ok999 wrote: > Copying files from jenkins workspace , is always slow. I have seen that > in the past. > > If u r using jenkins to checkout an artifact, and then copy to a path on a > remote. Just use wget/rsync (as Mark suggested) . U can trigger that from > jenkins too > > On Wed, May 30, 2018 at 6:12 AM Mark Waite <[email protected]> wrote: > >> I thought I remembered reading advice that archiving artifacts was known >> to be slow with large files. I couldn't find the reference, so I may be >> incorrect in this case. >> >> I'd suggest using a different technique to store your large artifacts, >> rather than having Jenkins perform the copy. If on Unix, consider rsync or >> other copy program. If on Windows, consider robocopy. >> >> Mark Waite >> >> On Wed, May 30, 2018 at 1:44 AM panpaliamahen < >> [email protected]> wrote: >> >>> Hi, >>> >>> I am using Jenkins version 2.107.3, latest one and on windows. I am using >>> Jenkins Slave node to run a Job which require to copy an artefact of size >>> ~6GH. And we found when jenkins copy this it takes ~10 minutes. (Note >>> artefact are copied from master). >>> >>> Also observed it takes ~10 minutes when jenkins job archived back to >>> master. >>> >>> Where as same can be copied in ~1 minute in same network without >>> Jenkins. >>> >>> Please can someone help me? >>> What should I do? >>> Do I need to install/upgrade any additional plugin on Jenkins? >>> >>> Please it is very urgent and reducing efficiency with in our development >>> teams. >>> >>> Thanks and regards, >>> Mahendra >>> >>> >>> >>> >>> -- >>> Sent from: >>> http://jenkins-ci.361315.n4.nabble.com/Jenkins-users-f361316.html >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/1527665457279-0.post%40n4.nabble.com >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHYp8p3civwUYu4_pAyi63PB4O8exWjt4YS0qimgLa8tg%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHYp8p3civwUYu4_pAyi63PB4O8exWjt4YS0qimgLa8tg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . > > >> For more options, visit https://groups.google.com/d/optout. >> > -- > Sent from mobile device, excuse typos if any. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/203463c9-0c2d-4d5f-a9c0-0fb77aae41d1n%40googlegroups.com.
