On Mon, Aug 4, 2014 at 12:24 PM, Brian Moffat <crd...@gmail.com> wrote:
> Thanks Mark. > I have a couple of follow-up questions, if you don't mind. > > 1. I've seen it work - meaning that the matrix project would do one > checkout and the downstream projects would run one after the other with no > lengthy pause between. So I'm left feeling like there should be a way to > "get back to that" without the suggestions you made - since I've not > modified the project configs since seeing it work. It may be that at the > time it was an "unintended feature" that got removed by a plug-in update, > but that's pure conjecture on my part. > I'm reasonably sure that it has always behaved that way. Each job in a matrix (multi-configuration) project may be on a separate machine, so each job performs a "git fetch" followed by a "git checkout". However, if you can find versions of the git plugin and the git client plugin which behave the way you want, you could compare the source code of the current plugin and the version behaving the way you want to see if you can identify the cause of the behavioral change. > 2. Why does the matrix job always do a "get a full copy" of the > repository before invoking the downstream job? If that's what I wanted, I > would have the downstream job do the pull. To me it looks like a bug, not > an enhancement request. What am I missing? > I believe that each job in the matrix job needs an independent copy of the repository so that it can work independent of the other jobs in the matrix. > -Brian > > On Monday, August 4, 2014 1:41:08 PM UTC-4, Mark Waite wrote: >> >> >> >> >> On Mon, Aug 4, 2014 at 10:11 AM, Brian Moffat <crd...@gmail.com> wrote: >> >>> I have a matrix project to build different C# projects/solutions. There >>> is one git repository for all solutions. >>> >>> The matrix job is configured for Git to checkout the master branch to a >>> local directory, and then invokes a downstream project for each matrix >>> element (solution). That downstream project has not Git configuration, and >>> simply builds the solution using MSBuild. >>> >>> What I am observing (looking at the console output for the Matrix build >>> job instance) is the matrix job performs the git checkout, which takes >>> about 4 minutes. It then says "Triggering <matrix element>", but that >>> downstream project doesn't show up in the build list for another 4-5 >>> minutes. Based on noticing that the content of the checkout directory is >>> changing (total size differences) during the period between when it says >>> "Triggering ..." and when that downstream job actually starts, my guess is >>> that another git checkout is occurring behind the scenes. >>> >> >> Your guess is correct. Each instance gets a full copy of the repository. >> >> If your repository is large, you could add the "Additional behaviours" >> for "Advanced clone options" and use a reference repository on the build >> machine(s). For example, if you created a bare repository copy of your >> central git repo on each build machine at c:\bare\your-git-repo.git, then >> you could list c:\bare\your-git-repo.git as the reference repository >> location. Git will then reference the contents of that bare repository and >> use that to reduce the "fetch" time. >> >> That won't reduce the "checkout" time, just the "fetch" time. If you >> also need to reduce the checkout time, then you might consider using >> "Sparse checkout" to only checkout those subdirectories which are actually >> required for your build. >> >> Another alternative would be to implement "git-work-dir" support in the >> git plugin, and submit a pull request for it. Refer to >> https://issues.jenkins-ci.org/browse/JENKINS-23594 for an enhancement >> request asking for that support. That enhancement request seems to match >> your use case quite well. >> >> >>> Obviously, this adds a significant amount of time to the total length of >>> the matrix build. The builds of each solution take under 1 minute. So for >>> the 7 solutions, I would expect the length of time to be 4+(7*1) or roughly >>> 11 minutes, but with the 4-5 minute delay between each matrix element, it >>> takes (4*7) + (7*1) or roughly 30 minutes. Our CI/CD processes are >>> maturing, and the lengthy delay between checkin at deploy to staging is now >>> slowing us down. >>> >>> Can anyone provide me guidance on what is going on behind the scenes >>> (nothing in any Jenkins log that I can find), or tell me how I can >>> configure the matrix job "properly" to not do this? >>> >>> This has run fast, sometimes - but I think perhaps it stopped working >>> after updating plugins, or may somehow be related to the plugins installed >>> - I just don't know where to start to look. >>> >>> The git section of the matrix job has: >>> - Additional Behaviors : Checkout to a subdirectory: >>> C:\Build\Source\WebPortal\Master\Dev >>> - Advanced sub-module behavior: Recursively update submodules (checked) >>> - Prune stale remote-tracking branches (not sure this is necessary?) >>> >>> Here is the console log for the matrix job: >>> >>> Started by user < <http://autobuild.blueport.com/user/bmoffat>me> >>> [EnvInject] - Loading node environment variables. >>> Building in workspace C:\Tools\Jenkins\jobs\BuildAllWebPortals\workspace >>> > C:\Tools\Git\cmd\git.exe rev-parse --is-inside-work-tree >>> Fetching changes from the remote Git repository >>> > C:\Tools\Git\cmd\git.exe config remote.origin.url <our git repository >>> url> >>> >>> Pruning obsolete local branches >>> Fetching upstream changes from <our git repository url> >>> >>> > C:\Tools\Git\cmd\git.exe --version >>> > C:\Tools\Git\cmd\git.exe fetch --tags --progress <our git repository >>> url> +refs/heads/master:refs/remotes/origin/master --prune >>> > C:\Tools\Git\cmd\git.exe rev-parse "origin/master^{commit}" >>> Checking out Revision b18aa8cff0aa9da9166f740877845cef27b612ff >>> (origin/master) >>> > C:\Tools\Git\cmd\git.exe config core.sparsecheckout >>> > C:\Tools\Git\cmd\git.exe checkout -f >>> b18aa8cff0aa9da9166f740877845cef27b612ff >>> > C:\Tools\Git\cmd\git.exe rev-list >>> b18aa8cff0aa9da9166f740877845cef27b612ff >>> > C:\Tools\Git\cmd\git.exe remote >>> > C:\Tools\Git\cmd\git.exe submodule init >>> > C:\Tools\Git\cmd\git.exe submodule sync >>> > C:\Tools\Git\cmd\git.exe config --get remote.origin.url >>> > C:\Tools\Git\cmd\git.exe submodule update --init --recursive >>> Email was triggered for: Before Build >>> Sending email for trigger: Before Build >>> Sending email to: <distribution list> >>> Triggering MobileASI >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileASI/> >>> >>> MobileASI >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileASI/> >>> completed with result SUCCESS >>> Triggering Furniture >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=Furniture/> >>> >>> Furniture >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=Furniture/> >>> completed with result SUCCESS >>> Triggering RetailPortal >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=RetailPortal/> >>> >>> RetailPortal >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=RetailPortal/> >>> completed with result SUCCESS >>> Triggering MobileLeons >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileLeons/> >>> >>> MobileLeons >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileLeons/> >>> completed with result SUCCESS >>> Triggering Leons >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=Leons/> >>> >>> Leons <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=Leons/> >>> completed with result SUCCESS >>> Triggering MobileVCF >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileVCF/> >>> >>> MobileVCF >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileVCF/> >>> completed with result SUCCESS >>> Triggering RetailTool >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=RetailTool/> >>> >>> RetailTool >>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=RetailTool/> >>> completed with result SUCCESS >>> No emails were triggered. >>> Finished: SUCCESS >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jenkinsci-use...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Thanks! >> Mark Waite >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Thanks! Mark Waite -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
