Re: [gentoo-user] Portage, git and shallow cloning
On Friday, 6 July 2018 06:34:01 BST Davyd McColl wrote: > 1) `sync-depth` has been deprecated (should now use `clone-depth`) But to what value should clone-depth be set? And why is the recent news item referring to instructions to use sync-depth? -- Regards, Peter.
Re: [gentoo-user] Portage, git and shallow cloning
On Sat, Jul 14, 2018 at 4:30 AM Peter Humphrey wrote: > > On Friday, 6 July 2018 06:34:01 BST Davyd McColl wrote: > > > 1) `sync-depth` has been deprecated (should now use `clone-depth`) > > But to what value should clone-depth be set? That comes down to personal taste. Do you want any history to be able to browse it? More depth means more history. If all you want is the current tree without history then you want a depth of 1, and of course you'll need to set up a cron job or something to go cleaning up past history (you never NEED more than the last commit). If you browse the online git repo you can see about how many commits there are in a day and estimate how many you want based on how many days you want. Also, this value only matters for the first sync. After that portage currently doesn't try to discard past commits, and it will always fetch all commits between your current state and the new head. If you want you could set up a script to manually purge history, and then do an initial sync with 1 depth. Then anytime you sync you could review the history since the last time you synced, and then run the purge command to discard all history up to the current commit. In doing this you'll always see all the history since the last time you reviewed it. -- Rich
Re: [gentoo-user] Multiboot USB - GRUB2 loop device
On Saturday, 14 July 2018 04.14.07 CEST Alex Luehm wrote: > I've recently taken it upon myself to create a multiboot USB with isos > that I tend to frequently use. So far I've been successful in adding > Clonezilla and the Archlinux live ISOs. I've attempted to add the Gentoo > install ISO in a similar manner (helped with the grub config within the > iso itself but can't seem to get GRUB to recogonize the image. My > DuckDuckGo-foo has returned useless results (a near hit, yet useless > inquiry being found here: > https://forums.gentoo.org/viewtopic-p-6527390.html). You may find the following projects interesting: - https://github.com/aguslr/multibootusb - https://github.com/thias/glim The first project contains GRUB configurations for both Gentoo and System Rescue CD while the second one only contains GRUB configurations for System Rescue CD. Please note I haven't tried anything yet. It's just on my (long) todo list. -- https://fturco.gitlab.com/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Portage, git and shallow cloning
On Saturday, 14 July 2018 11:40:03 BST Rich Freeman wrote: > On Sat, Jul 14, 2018 at 4:30 AM Peter Humphrey wrote: > > On Friday, 6 July 2018 06:34:01 BST Davyd McColl wrote: > > > 1) `sync-depth` has been deprecated (should now use `clone-depth`) > > > > But to what value should clone-depth be set? > > That comes down to personal taste. Do you want any history to be able > to browse it? More depth means more history. If all you want is the > current tree without history then you want a depth of 1... That's all I need for the portage tree, unless removing everything at lower depths will remove the change records. > ...and of course you'll need to set up a cron job or something to go > cleaning up past history (you never NEED more than the last commit). If you > browse the online git repo you can see about how many commits there are in a > day and estimate how many you want based on how many days you want. > > Also, this value only matters for the first sync. After that portage > currently doesn't try to discard past commits, and it will always > fetch all commits between your current state and the new head. > > If you want you could set up a script to manually purge history, and > then do an initial sync with 1 depth. Then anytime you sync you could > review the history since the last time you synced, and then run the > purge command to discard all history up to the current commit. In > doing this you'll always see all the history since the last time you > reviewed it. Is there something in git to do that purging? If not, perhaps a simple monthly script to delete /usr/portage/* - but not packages or distfiles, which are on separate partitions here - would do the trick. -- Regards, Peter.
[gentoo-user] Re: Portage, git and shallow cloning
Peter Humphrey wrote: > On Friday, 6 July 2018 06:34:01 BST Davyd McColl wrote: > >> 1) `sync-depth` has been deprecated (should now use `clone-depth`) > > [...] And why is the recent news item referring to instructions > to use sync-depth? Things have changed with portage-2.3.42: sync-depth is again supported (in addition to clone-depth). clone-depth is for the first cloning, sync-depth is for the runnnig system. The strategy which I had mentioned (with --merge) is used if sync-depth is set (and nonzero). Only the git index for people with overlayfs is still missing, although there are plans to introduce this also, optionally.
Re: [gentoo-user] Portage, git and shallow cloning
On Sat, Jul 14, 2018 at 8:00 AM Peter Humphrey wrote: > > That's all I need for the portage tree, unless removing everything at lower > depths will remove the change records. If you clone with a depth of one you'll see the current state of the tree, and a commit message from the CI bot, and that is it. You'll have zero change history for anything. If you clone with a dept of 10 you'll see one or two CI bot messages, and then the last 8 or so actual changes to the tree. You'll also have access to what the tree looked like when each of those changes was made. Note that git uses COW and compression, so the cost of increasing your depth isn't very high. A depth of 1 costs you about 670M, and a depth of 236000 costs you 1.5G. I'd expect the cost to be roughly linear between these. > > Is there something in git to do that purging? If not, perhaps a simple monthly > script to delete /usr/portage/* - but not packages or distfiles, which are on > separate partitions here - would do the trick. That delete would certainly work, though it would cost you a full sync (which would go back to your depth setting). I'd suggest moving distfiles outside of the repo if you're going to do that (really, it shouldn't be inside anyway), just to make it easier. git has no facilities to do this automatically, probably because it isn't something Linus does and git is very much his thing. However, I found that this works for me: git rev-parse HEAD >! .git/shallow git reflog expire --expire=all --all git gc --prune=now (This is a combination of: https://stackoverflow.com/a/34829535 (which doesn't work) and https://stackoverflow.com/a/46004595 (which is incomplete)) It runs in about 14s for me in a tmpfs. Another option would be to a local shallow clone and swap the repositories. You'll find tons of guides online for throwing out history that involve rebasing. You do NOT want to do this here. These will change the hash of the HEAD, which means that the next git pull won't be a fast-forward, and it will be a mess in general. You just want to discard local history, not rewrite the repository to say that there never was any history. Also note that the first line in this little script depends somewhat on git internals and may or may not work in the distant future. In any case, I suggest trying it. If it somehow eats your repo for breakfast just delete it and the next sync will re-fetch. -- Rich
Re: [gentoo-user] Portage, git and shallow cloning
On Sat, Jul 14, 2018 at 11:06 AM Rich Freeman wrote: > > git rev-parse HEAD >! .git/shallow > git reflog expire --expire=all --all > git gc --prune=now > Before anybody bangs their head against the wall too much I did end up having syncing issues with this. I suspect the fix is a one-liner, but which one-liner has defied a fair bit of messing around with it. In general the git authors aren't really big on supporting this sort of thing, so it is just a big hack. Doing a local sync to discard history might be an option. Just deleting the repo and re-syncing is another option. But, if somebody comes up with a good fix I'm all ears. -- Rich
