Hi Rich, I will experiment with a local metadata - that seems like it might give a worthwhile speedup (its currently about 910Mb so not too big)
Bill K. On 28/2/20 3:44 am, Rich Freeman wrote: > On Thu, Feb 27, 2020 at 2:27 PM james <gar...@verizon.net> wrote: >> On 2/26/20 7:08 PM, William Kenworthy wrote: >>> Hi, >>> >>> ��� due to space considerations on my laptop I have moved portage >>> onto a >>> network share (moosfs, mfsmounted) - slower but works fine.� However, >>> being a laptop trying to update/install when out and about is both very >>> slow and network intensive through a vpn - again, it works but is even >>> slower to the point of not always being practical >>> >>> Is there a way to localise/speedup portage scanning parts of the >>> update/install process? >> >> A simpler, much less sophisticated update, what I do is >> use emerge -f option to 'fetch only' option first. The selectively >> update; or you can use a usb-3.1+ device for fast easy upgrades, due to >> laptop limitations, but the communications data channel limitations >> leave you at the mercy of the available wireless bandwidth characteristics. > So, a few thoughts here: > > 1. Definitely make sure you're splitting out distfiles. Any new > install does this already, but mixing up the tree and distfiles is > mixing very different types of data with different requirements. > 2. Consider whether you REALLY need to share all this stuff. The > whole tree isn't THAT big, so it might be easier to just locally sync. > If network bandwidth is an issue, consider setting up a local mirror > so that you sync (and fetch distfiles) within your network when > available, and then just sync directly from the internet when you're > remote. > 3. If you REALLY need that portage tree to be on the network, > consider keeping just the metadata directory local. That can be > generated on-demand or synced, but either way it needs to be updated > anytime the repo itself is updated. Generally with rsync we > distribute both together. I suspect that having local metadata will > cut down on your network IO when you're doing dep resolution/etc. > 4. Squashfs was mentioned. In theory that is a nice compact > solution. In practice you might find it to be a pain to update since > it is read-only. But you could update your network tree, then create > a squashfs and stick that on your network, and then sync the squashfs > file for local mounting. > > I'm not sure how well moosefs performs for this sort of thing in > general. I personally use lizardfs to store large multimedia files > and it works fine, but streaming multi-GB multimedia that ends up > getting buffered is a very different access pattern than sourcing a > bazillion ebuild scripts. I'd be interested in how you're liking > moosefs for this sort of thing in general. For accessing tons of > little files I think all your latencies matter, but in particular you > want low latency to your master server as that is going to end up > limiting every read/open you perform. You might be able to play with > the cache settings on the client side, though at least in lizardfs > I've found some of those to be buggy - they're generally passed as > mount options. > > In the past I've had one main Gentoo box sync its repo from the web, > and had other hosts in the LAN either bind-mount this (containers), or > sync this over the network. Likewise I generally share my main host > distfiles as a mirror which is preferentially used by all my other > hosts - you don't get a 100% hit rate and the main host won't get a > copy of missed files, but it probably cuts down on 80+% of the network > fetching as long as the main host does its fetches before the other > hosts do. If the main host lacks a distfile then the MIRRORS setting > has a regular Gentoo mirror listed next. > > Honestly, unless you're running a completely private repo for whatever > reason I would not try to go remotely mounting your repo over the > internet from your moosefs server. I bet you could fetch anything > needed faster from a regular internet mirror in that case, and if > portage could use mirrors seamlessly over internet latencies chances > are we'd all be doing that already. Pretty much every distro > maintains a local repo on every host for this reason. >
