On Thu, Feb 27, 2020 at 2:27 PM james <gar...@verizon.net> wrote: > > On 2/26/20 7:08 PM, William Kenworthy wrote: > > Hi, > > > > ��� due to space considerations on my laptop I have moved portage > > onto a > > network share (moosfs, mfsmounted) - slower but works fine.� However, > > being a laptop trying to update/install when out and about is both very > > slow and network intensive through a vpn - again, it works but is even > > slower to the point of not always being practical > > > > Is there a way to localise/speedup portage scanning parts of the > > update/install process? > > > A simpler, much less sophisticated update, what I do is > use emerge -f option to 'fetch only' option first. The selectively > update; or you can use a usb-3.1+ device for fast easy upgrades, due to > laptop limitations, but the communications data channel limitations > leave you at the mercy of the available wireless bandwidth characteristics.
So, a few thoughts here: 1. Definitely make sure you're splitting out distfiles. Any new install does this already, but mixing up the tree and distfiles is mixing very different types of data with different requirements. 2. Consider whether you REALLY need to share all this stuff. The whole tree isn't THAT big, so it might be easier to just locally sync. If network bandwidth is an issue, consider setting up a local mirror so that you sync (and fetch distfiles) within your network when available, and then just sync directly from the internet when you're remote. 3. If you REALLY need that portage tree to be on the network, consider keeping just the metadata directory local. That can be generated on-demand or synced, but either way it needs to be updated anytime the repo itself is updated. Generally with rsync we distribute both together. I suspect that having local metadata will cut down on your network IO when you're doing dep resolution/etc. 4. Squashfs was mentioned. In theory that is a nice compact solution. In practice you might find it to be a pain to update since it is read-only. But you could update your network tree, then create a squashfs and stick that on your network, and then sync the squashfs file for local mounting. I'm not sure how well moosefs performs for this sort of thing in general. I personally use lizardfs to store large multimedia files and it works fine, but streaming multi-GB multimedia that ends up getting buffered is a very different access pattern than sourcing a bazillion ebuild scripts. I'd be interested in how you're liking moosefs for this sort of thing in general. For accessing tons of little files I think all your latencies matter, but in particular you want low latency to your master server as that is going to end up limiting every read/open you perform. You might be able to play with the cache settings on the client side, though at least in lizardfs I've found some of those to be buggy - they're generally passed as mount options. In the past I've had one main Gentoo box sync its repo from the web, and had other hosts in the LAN either bind-mount this (containers), or sync this over the network. Likewise I generally share my main host distfiles as a mirror which is preferentially used by all my other hosts - you don't get a 100% hit rate and the main host won't get a copy of missed files, but it probably cuts down on 80+% of the network fetching as long as the main host does its fetches before the other hosts do. If the main host lacks a distfile then the MIRRORS setting has a regular Gentoo mirror listed next. Honestly, unless you're running a completely private repo for whatever reason I would not try to go remotely mounting your repo over the internet from your moosefs server. I bet you could fetch anything needed faster from a regular internet mirror in that case, and if portage could use mirrors seamlessly over internet latencies chances are we'd all be doing that already. Pretty much every distro maintains a local repo on every host for this reason. -- Rich
