Hard links do not work across file systems. And I need to have a local clone on every slave. And I do not actually maintain this clone. My slaves are actually virtual machines and I recreate them regularly and the final step of the vm setup is to clone the git repo.
The local clone doesn't really need to be very up-to-date to provide big savings. If your slaves are persistent, then sure, you'd want to update the local clone every now and then, maybe once a week or so. I do believe I've heard that windows git supports hard links. But don't take my word for it. I've never used it myself. -- Sami 2012/2/21 Gergely Nagy <gsz...@gmail.com>: > Sami, > > Yes, I had been thinking about that too - but I actually already have a > gitolite mirror repo (+git-svn clone) on the master... > .. so I was already using "local remote urls"... however that mirror is on > a different partition (LV). .. which breaks the hard links, I assume. > > So thanks for the reminder - that will help on the master and linux > slaves... BTW, you'd have one clone for each slave right? > Also, is it a recommended practice to set up a Jenkins job solely to > maintain this clone? > If so, it would be nice to have plugin support (e.g a wizard to add > "mirrored git clones" across master/slaves, and SCM type "Git local clone") > - maybe I could start something like that, unless someone has already:) > > Still, a more general underlying mechanism would be great, e.g. one that > even works on Windows slaves.. the object sharing would not work the same > way (at least not with hard links - I assume). > > Thank you, > Gergo > > > On Mon, Feb 20, 2012 at 11:51 PM, Sami Tikka <sjti...@gmail.com> wrote: >> >> You can already achieve the same benefit by making a local clone of the >> git repo (use --bare for this) and then configuring each job to have 2 >> repos: the first should be /path/to/local/repo and the second can be the >> location where you usually clone from. >> >> This way most git objects will be shared because a local git clone will >> use hard links. >> >> My build slaves at work have small but fast ssd disks and we use this >> trick (plus running git clean -fxd as a post-task step) to keep disk space >> usage in control. >> >> -- Sami >> >> Gergely Nagy <gsz...@gmail.com> kirjoitti 15.2.2012 kello 19.15: >> >> Thanks Mark, >> that's great info - to me it sounds like the way to go. >> Gergo >> >> On Wed, Feb 15, 2012 at 3:03 AM, Mark Waite <markwa...@yahoo.com> wrote: >>> >>> The git plugin rework discussions mentioned the possibility of including >>> the "---reference <existing-repository>" argument to git clone so the pack >>> files for a single repository could be reused in multiple repositories on >>> the same machine. Then you could clone to a single directory on the slave, >>> and reference that clone rather than copying the pack files to each of the >>> workspace copies. >>> >>> I don't think it has been implemented yet, but the plugin developers may >>> be willing to share their ideas in case they have an even better idea than >>> using the --reference argument to git clone. >>> >>> Mark Waite >>> >>> From: Gergely Nagy <gsz...@gmail.com> >>> To: jenkinsci-users@googlegroups.com >>> Sent: Tuesday, February 14, 2012 1:23 PM >>> Subject: git: reduce clones' disk space >>> >>> Hi Jenkins gurus, >>> >>> I have a load of jobs (50+ I think) which clone the same repository, but >>> different branches, to build/unit/test/functional test stages. >>> >>> Also, it's a special application of the "job splitting pattern" >>> (https://wiki.jenkins-ci.org/display/JENKINS/Splitting+a+big+job+into+smaller+jobs): >>> the tarball that downstream jobs receive is a much smaller than the >>> entire workspace: it only contains unknown files(git ls-files -oz: the build >>> artifacts), which is "just" 400m >>> vs 1.8G. Downstream jobs unpack this on top of a pristine clone to get up >>> to speed. This is quite fast (most files are there already) and also seems >>> to do better change tracking. >>> >>> However it costs space - each of the workspace is ~ 4-5G - half of which >>> is the git clone. >>> While git has a good reason to clone everything with all the branches, I >>> don't need that duplicated 50 times on the Jenkins box. >>> So am wondering if there is a way to optimise this? >>> I guess, i'd rather have one single full clone, and let jobs have the >>> work directories (+index?).. >>> >>> Any enlightments/alternative ideas are appreciated. >>> thanks, >>> Gergo >>> >>> >>> >>> >>> >> >
