On Tuesday, July 2, 2019 at 8:38:46 AM UTC-7, Mark Waite wrote: > That's great news. A git repository that needs more than 10 minutes to clone > is a really good candidate for reference repositories and all the other > techniques suggested in the "Git in the Large" video. The first preference > with repositories that large is to reduce their size by removing large files > from the repository. Unfortunately, most organizations can't do that. A > previous organization where I worked had a 25 GB repository that just kept > growing due to large binary checkins. Git large file storage (LFS) is > another lifesaver to avoid creating large repositories. Good luck! > > > On Tue, Jul 2, 2019 at 8:49 AM Kevin Stevens <kevs...@gmail.com> wrote: > > Hi Mark - whilst looking for a place to stash my reference repo I discovered > that Jenkins is keeping a cache of git checkouts in the > /var/lib/jenkins/cache folder, and by deleting these files the multibranch > pipeline scan now succeeds, so there was obviously something corrupt in that > cache (possibly caused by a recent instability issue with the Jenkins server > hardware), so I've not implemented a reference repo. > Thanks for your help ;-) > > > Kevin > > On Tuesday, 2 July 2019 15:25:41 UTC+1, Kevin Stevens wrote: > Thanks for the responses and the issue link Mark, I've increased the Git > timeout value as described, and confirmed in the log that the 30 minute > timeout is in effect, but I'm still seeing the same failure (i.e. fatal error > in Scan Multibranch Pipeline Now). > I was pretty confident that there's not a timeout issue as the failure occurs > in a matter of seconds, so I'm going to try the reference repo approach which > seems like it should work, but it would appear to be an issue within the git > client parsing to me. I'll confirm my outcome. > > > Regards, Kevin > > On Tuesday, 2 July 2019 13:37:41 UTC+1, Mark Waite wrote: > Sorry, I didn't read your description thoroughly enough. You said: > > I'm getting a fatal error reported when doing a repository scan (Scan > Multibranch Pipeline Now) which was previously working. > > > > The command line git log output indicates that your repository is large > enough that it may be reaching the default 10 minute clone timeout. Refer to > https://issues.jenkins-ci.org/browse/JENKINS-38973 for a discussion of > possible workarounds. > > > On Tue, Jul 2, 2019 at 6:31 AM Mark Waite <mark.e...@gmail.com> wrote: > > The repository in the workspace on the agent running that build is probably > damaged. Wipe the workspace on the agent and run the job again. > > > If wiping the workspace and running the job again does not resolve it, you > may also be encountering a timeout while cloning the repository. Increase > the timeout for the repository clone from its default of 10 minutes to > something large enough that it will allow you to clone the repository. > > > > The amount of output you're showing hints that the repository is large or the > network connection between the agent and the upstream cloned repository is > slow. In either of those cases, you probably want to reduce the clone time > and the disc space use by applying one or more of the techniques described in > "Git in the Large". Those techniques include (1) reference repositories on > the agent, (2) narrow refspecs, (3) shallow clones, and (4) sparse checkouts. > > > If this is a Pipeline repository and the failure is during the initial clone > of the repository, then you may also need to use lightweight checkout to only > checkout the Jenkinsfile rather than the entire repository. Alternately for > Pipelines, if your git provider is GitHub, Bitbucket, or Gitea, you can > significantly improve performance by using those branch source plugins to > manage the Pipeline instead of relying on low-level command line git calls. > > Reference repositories are usually the most effective technique to reduce > data transfer time and disc space use for large git repositories. Allan > Burdajewicz of CloudBees wrote a great article on reference repositories at > https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository > . > > > On Tue, Jul 2, 2019 at 5:51 AM Kevin Stevens <kevs...@gmail.com> wrote: > > I'm getting a fatal error reported when doing a repository scan (Scan > Multibranch Pipeline Now) which was previously working. > I've run the git commands manually from the command line (copied and pasted) > and they appear to work correctly, so it seems like a problem with the > Jenkins git client plugin. > > > The scan log is below (repository name and account information modified for > security reasons). > I'm not clear if this is a GIT or a Jenkins git client problem. I've cleared > the Jenkins workspace and updated to the latest Jenkins plugins (running on > Ubuntu 18.04). > Is anyone able to offer help in debugging this issue please? > > > > > > [Tue Jul 02 09:48:00 BST 2019] Starting branch indexing... > > git --version # timeout=10 > using GIT_ASKPASS to set credentials Jenkins Bitbucket User > > git ls-remote --symref > g...@bitbucket.org:my-company-name/my-repository-name.git # timeout=10 > > git rev-parse --is-inside-work-tree # timeout=10 > Setting origin to g...@bitbucket.org:my-company-name/my-repository-name.git > > git config remote.origin.url > g...@bitbucket.org:my-company-name/my-repository-name.git # timeout=10 > Fetching & pruning origin... > Listing remote references... > > git config --get remote.origin.url # timeout=10 > > git --version # timeout=10 > using GIT_ASKPASS to set credentials Jenkins Bitbucket User > > git ls-remote -h g...@bitbucket.org:my-company-name/my-repository-name.git > # timeout=10 > Fetching upstream changes from origin > > git config --get remote.origin.url # timeout=10 > using GIT_ASKPASS to set credentials Jenkins Bitbucket User > > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* > --prune > ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from source > 498872a6-3888-419f-9d2f-a2baa4520968 > hudson.plugins.git.GitException: Command "git fetch --tags --progress origin > +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128: > stdout: > stderr: remote: Counting objects: 114, done. > remote: Compressing objects: 0% (1/114) > remote: Compressing objects: 1% (2/114) > remote: Compressing objects: 2% (3/114) > remote: Compressing objects: 3% (4/114) > remote: Compressing objects: 4% (5/114) > remote: Compressing objects: 5% (6/114) > remote: Compressing objects: 6% (7/114) > remote: Compressing objects: 7% (8/114) > remote: Compressing objects: 8% (10/114) > remote: Compressing objects: 9% (11/114) > remote: Compressing objects: 10% (12/114) > remote: Compressing objects: 11% (13/114) > remote: Compressing objects: 12% (14/114) > remote: Compressing objects: 13% (15/114) > remote: Compressing objects: 14% (16/114) > remote: Compressing objects: 15% (18/114) > remote: Compressing objects: 16% (19/114) > remote: Compressing objects: 17% (20/114) > remote: Compressing objects: 18% (21/114) > remote: Compressing objects: 19% (22/114) > remote: Compressing objects: 20% (23/114) > remote: Compressing objects: 21% (24/114) > remote: Compressing objects: 22% (26/114) > remote: Compressing objects: 23% (27/114) > remote: Compressing objects: 24% (28/114) > remote: Compressing objects: 25% (29/114) > remote: Compressing objects: 26% (30/114) > remote: Compressing objects: 27% (31/114) > remote: Compressing objects: 28% (32/114) > remote: Compressing objects: 29% (34/114) > remote: Compressing objects: 30% (35/114) > remote: Compressing objects: 31% (36/114) > remote: Compressing objects: 32% (37/114) > remote: Compressing objects: 33% (38/114) > remote: Compressing objects: 34% (39/114) > remote: Compressing objects: 35% (40/114) > remote: Compressing objects: 36% (42/114) > remote: Compressing objects: 37% (43/114) > remote: Compressing objects: 38% (44/114) > remote: Compressing objects: 39% (45/114) > remote: Compressing objects: 40% (46/114) > remote: Compressing objects: 41% (47/114) > remote: Compressing objects: 42% (48/114) > remote: Compressing objects: 43% (50/114) > remote: Compressing objects: 44% (51/114) > remote: Compressing objects: 45% (52/114) > remote: Compressing objects: 46% (53/114) > remote: Compressing objects: 47% (54/114) > remote: Compressing objects: 48% (55/114) > remote: Compressing objects: 49% (56/114) > remote: Compressing objects: 50% (57/114) > remote: Compressing objects: 51% (59/114) > remote: Compressing objects: 52% (60/114) > remote: Compressing objects: 53% (61/114) > remote: Compressing objects: 54% (62/114) > remote: Compressing objects: 55% (63/114) > remote: Compressing objects: 56% (64/114) > remote: Compressing objects: 57% (65/114) > remote: Compressing objects: 58% (67/114) > remote: Compressing objects: 59% (68/114) > remote: Compressing objects: 60% (69/114) > remote: Compressing objects: 61% (70/114) > remote: Compressing objects: 62% (71/114) > remote: Compressing objects: 63% (72/114) > remote: Compressing objects: 64% (73/114) > remote: Compressing objects: 65% (75/114) > remote: Compressing objects: 66% (76/114) > remote: Compressing objects: 67% (77/114) > remote: Compressing objects: 68% (78/114) > remote: Compressing objects: 69% (79/114) > remote: Compressing objects: 70% (80/114) > remote: Compressing objects: 71% (81/114) > remote: Compressing objects: 72% (83/114) > remote: Compressing objects: 73% (84/114) > remote: Compressing objects: 74% (85/114) > remote: Compressing objects: 75% (86/114) > remote: Compressing objects: 76% (87/114) > remote: Compressing objects: 77% (88/114) > remote: Compressing objects: 78% (89/114) > remote: Compressing objects: 79% (91/114) > remote: Compressing objects: 80% (92/114) > remote: Compressing objects: 81% (93/114) > remote: Compressing objects: 82% (94/114) > remote: Compressing objects: 83% (95/114) > remote: Compressing objects: 84% (96/114) > remote: Compressing objects: 85% (97/114) > remote: Compressing objects: 86% (99/114) > remote: Compressing objects: 87% (100/114) > remote: Compressing objects: 88% (101/114) > remote: Compressing objects: 89% (102/114) > remote: Compressing objects: 90% (103/114) > remote: Compressing objects: 91% (104/114) > remote: Compressing objects: 92% (105/114) > remote: Compressing objects: 93% (107/114) > remote: Compressing objects: 94% (108/114) > remote: Compressing objects: 95% (109/114) > remote: Compressing objects: 96% (110/114) > remote: Compressing objects: 97% (111/114) > remote: Compressing objects: 98% (112/114) > remote: Compressing objects: 99% (113/114) > remote: Compressing objects: 100% (114/114) > remote: Compressing objects: 100% (114/114), done. > Receiving objects: 0% (1/114) > Receiving objects: 1% (2/114) > Receiving objects: 2% (3/114) > Receiving objects: 3% (4/114) > Receiving objects: 4% (5/114) > Receiving objects: 5% (6/114) > Receiving objects: 6% (7/114) > Receiving objects: 7% (8/114) > Receiving objects: 8% (10/114) > Receiving objects: 9% (11/114) > Receiving objects: 10% (12/114) > Receiving objects: 11% (13/114) > Receiving objects: 12% (14/114) > Receiving objects: 13% (15/114) > Receiving objects: 14% (16/114) > error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is > empty > fatal: cannot read existing object info > 21a28795e9c6fa55f026860e0c2f0b08d1b31611 > fatal: index-pack failed > > > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72) > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442) > at > jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575) > at > jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556) > at > jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367) > at > jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556) > at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) > at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283) > at > jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634) > at > com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277) > at > com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164) > at > jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025) > at hudson.model.ResourceController.execute(ResourceController.java:97) > at hudson.model.Executor.run(Executor.java:429) > [Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took 10 sec > FATAL: Failed to recompute children of my-repository-name > hudson.plugins.git.GitException: Command "git fetch --tags --progress origin > +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128: > stdout: > stderr: remote: Counting objects: 114, done. > remote: Compressing objects: 0% (1/114) > remote: Compressing objects: 1% (2/114) > remote: Compressing objects: 2% (3/114) > remote: Compressing objects: 3% (4/114) > remote: Compressing objects: 4% (5/114) > remote: Compressing objects: 5% (6/114) > remote: Compressing objects: 6% (7/114) > remote: Compressing objects: 7% (8/114) > remote: Compressing objects: 8% (10/114) > remote: Compressing objects: 9% (11/114) > remote: Compressing objects: 10% (12/114) > remote: Compressing objects: 11% (13/114) > remote: Compressing objects: 12% (14/114) > remote: Compressing objects: 13% (15/114) > remote: Compressing objects: 14% (16/114) > remote: Compressing objects: 15% (18/114) > remote: Compressing objects: 16% (19/114) > remote: Compressing objects: 17% (20/114) > remote: Compressing objects: 18% (21/114) > remote: Compressing objects: 19% (22/114) > remote: Compressing objects: 20% (23/114) > remote: Compressing objects: 21% (24/114) > remote: Compressing objects: 22% (26/114) > remote: Compressing objects: 23% (27/114) > remote: Compressing objects: 24% (28/114) > remote: Compressing objects: 25% (29/114) > remote: Compressing objects: 26% (30/114) > remote: Compressing objects: 27% (31/114) > remote: Compressing objects: 28% (32/114) > remote: Compressing objects: 29% (34/114) > remote: Compressing objects: 30% (35/114) > remote: Compressing objects: 31% (36/114) > remote: Compressing objects: 32% (37/114) > remote: Compressing objects: 33% (38/114) > remote: Compressing objects: 34% (39/114) > remote: Compressing objects: 35% (40/114) > remote: Compressing objects: 36% (42/114) > remote: Compressing objects: 37% (43/114) > remote: Compressing objects: 38% (44/114) > remote: Compressing objects: 39% (45/114) > remote: Compressing objects: 40% (46/114) > remote: Compressing objects: 41% (47/114) > remote: Compressing objects: 42% (48/114) > remote: Compressing objects: 43% (50/114) > remote: Compressing objects: 44% (51/114) > remote: Compressing objects: 45% (52/114) > remote: Compressing objects: 46% (53/114) > remote: Compressing objects: 47% (54/114) > remote: Compressing objects: 48% (55/114) > remote: Compressing objects: 49% (56/114) > remote: Compressing objects: 50% (57/114) > remote: Compressing objects: 51% (59/114) > remote: Compressing objects: 52% (60/114) > remote: Compressing objects: 53% (61/114) > remote: Compressing objects: 54% (62/114) > remote: Compressing objects: 55% (63/114) > remote: Compressing objects: 56% (64/114) > remote: Compressing objects: 57% (65/114) > remote: Compressing objects: 58% (67/114) > remote: Compressing objects: 59% (68/114) > remote: Compressing objects: 60% (69/114) > remote: Compressing objects: 61% (70/114) > remote: Compressing objects: 62% (71/114) > remote: Compressing objects: 63% (72/114) > remote: Compressing objects: 64% (73/114) > remote: Compressing objects: 65% (75/114) > remote: Compressing objects: 66% (76/114) > remote: Compressing objects: 67% (77/114) > remote: Compressing objects: 68% (78/114) > remote: Compressing objects: 69% (79/114) > remote: Compressing objects: 70% (80/114) > remote: Compressing objects: 71% (81/114) > remote: Compressing objects: 72% (83/114) > remote: Compressing objects: 73% (84/114) > remote: Compressing objects: 74% (85/114) > remote: Compressing objects: 75% (86/114) > remote: Compressing objects: 76% (87/114) > remote: Compressing objects: 77% (88/114) > remote: Compressing objects: 78% (89/114) > remote: Compressing objects: 79% (91/114) > remote: Compressing objects: 80% (92/114) > remote: Compressing objects: 81% (93/114) > remote: Compressing objects: 82% (94/114) > remote: Compressing objects: 83% (95/114) > remote: Compressing objects: 84% (96/114) > remote: Compressing objects: 85% (97/114) > remote: Compressing objects: 86% (99/114) > remote: Compressing objects: 87% (100/114) > remote: Compressing objects: 88% (101/114) > remote: Compressing objects: 89% (102/114) > remote: Compressing objects: 90% (103/114) > remote: Compressing objects: 91% (104/114) > remote: Compressing objects: 92% (105/114) > remote: Compressing objects: 93% (107/114) > remote: Compressing objects: 94% (108/114) > remote: Compressing objects: 95% (109/114) > remote: Compressing objects: 96% (110/114) > remote: Compressing objects: 97% (111/114) > remote: Compressing objects: 98% (112/114) > remote: Compressing objects: 99% (113/114) > remote: Compressing objects: 100% (114/114) > remote: Compressing objects: 100% (114/114), done. > Receiving objects: 0% (1/114) > Receiving objects: 1% (2/114) > Receiving objects: 2% (3/114) > Receiving objects: 3% (4/114) > Receiving objects: 4% (5/114) > Receiving objects: 5% (6/114) > Receiving objects: 6% (7/114) > Receiving objects: 7% (8/114) > Receiving objects: 8% (10/114) > Receiving objects: 9% (11/114) > Receiving objects: 10% (12/114) > Receiving objects: 11% (13/114) > Receiving objects: 12% (14/114) > Receiving objects: 13% (15/114) > Receiving objects: 14% (16/114) > error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is > empty > fatal: cannot read existing object info > 21a28795e9c6fa55f026860e0c2f0b08d1b31611 > fatal: index-pack failed > > > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72) > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442) > at > jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575) > at > jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556) > at > jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367) > at > jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556) > at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) > at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283) > at > jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634) > at > com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277) > at > com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164) > at > jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025) > at hudson.model.ResourceController.execute(ResourceController.java:97) > at hudson.model.Executor.run(Executor.java:429) > Finished: FAILURE > > > > > > > -- > > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkins...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > > Thanks! > Mark Waite > > > > -- > > > Thanks! > Mark Waite > > > > > > -- > > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkins...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/b30a6241-1427-46bf-b8a6-34a42b939608%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > > Thanks! > Mark Waite
Hi. I am in a similar situation where I have several very large Git repositories, and I want to build multibranch pipelines for each of them, and my reference repositories don’t seem to be used for scanning/indexing. My issue can’t be solved by a timeout increase. My issue is storage. I see in one of the pipelines, which was small enough to be cloned on my Jenkins master, that the reference repository is used for the checkouts in the per-branch workspaces. But my master node simply doesn’t have enough writable storage to fetch all objects of more than a dozen very large Git repositories. Mark Waite, if this is something you know how to fix but prefer to delegate, I’m happy to volunteer. I’m new to writing Jenkins plugins, but I should be able to help if I get some direction. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e970d393-e3e6-4add-acd0-8d57144dbc8c%40googlegroups.com.
