So it sounds like my first steps are to re-implement the downloads using aria2c. This would affect the minimum base package, no? Can I get some buy-in from maintainers that such changes are acceptable?
On Fri, Oct 13, 2023 at 2:06 PM James R. Haigh (+ML.GNU.Guix subaddress) <jrhaigh+ml.gnu.g...@runbox.com> wrote: > > Hi Josh, > > At Z-0400=2023-10-13Fri12:36:01, Josh Marshall sent: > > This is to parallelize connections which should never hurt downloading but > > can help. Mirroring would be parallelizing for providing packages, what I > > want to implement is to parallelize obtaining packages. Server side vs > > client side. > > Please, if you are going to do something like this, please use a > torrent architecture like BitTorrent or GNUnet – I suggest Aria2c as a very > good CLI download backend that can be daemonised and sent instructions over a > socket to add, pause, remove downloads, etc., and it supports magnet URLs > including the existing nontorrent servers (via ‘as’ parameters, iirc.). > > I actually implemented this in a local copy of APT Daemon many years > ago (circa 2011), but the change was not accepted upstream to Launchpad > (because I was not on bleeding-edge; I was too slow to keep-up with the > upstream development). My fork got forgotten about, because to get the full > benefit the server would have had to have added a BitTorrent Info Hash (BTIH) > to the metadata of each package, along with the MD5, SHA-256, etc. that it > already did (not a big ask, really). That said, without the full benefit of > having the metadata, it did provide immediate benefit and I used it for many > years, not upgrading my Ubuntu 11.04 Natty Narwhal that I was using back then > until I really had to. > > The immediate benefit that it provided was exactly as you described: > It allowed parallelisation of nontorrent downloads, be it from the same > server or from multiple mirrors. Iirc., I achieved this by simply passing > the download list to Aria2c in daemon mode, I think I also converted all the > HTTP URLs to ‘as’ parameters in magnet links, so that multiple mirrors could > be passed using multiple ‘as’ parameters in each magnet link. Then I simply > relied on Aria2c being amazing at parallelising everything that I had given > it! I then also implemented progress updates such that APT Daemon could > reflect where Aria2c was up to. > > The way I implemented this using Aria2c and magnet URLs meant that if > additional hashes were known, they could be used as well, and so if the > server metadata made the simple addition of adding BTIHs, it allows swarming > to occur, which in-turn would massively reduce load on the central servers, > and allow anyone who want to be a mirror to be a mirror simply by seeding > indefinitely. A default share ratio of 1.0 means that no user is a burden on > the network, unless they deliberately change that. Users can donate to the > running costs of the project simply by increasing their share ratio, which > adds another means of contribution that they may find easier than the others. > > Anyone keen to keep old packages online can simply seed them > indefinitely, so this is also really great for archival purposes. Even if > the central project loses interest in the old packages and deletes them, > anyone else can keep them up. The hashes ensure that they have not been > tampered with. > > There is also a really cool benefit that occurs, or can occur, on a > LAN. An entire network of computers can all swarm locally with each other, > thus needing each package to only need downloading through the metered last > mile bottleneck from the WAN precisely once – providing that local > broadcasting is supported. I think this requires Avahi, and I seem to > remember that Aria2c supports this but I can't remember. I don't ever > remember getting this bit working but also I did not try hard because it > would have required the metadata that I didn't have until after download, so > even if I got it working it would not have been directly useful unless the > APT repositories that I was using would include the BTIHs. > > So yeah, loads of great benefits to this architecture, and I > highly-recommend it: convert all existing URLs to magnet links (can be done > client-side as I did; or server-side); optionally add any additional mirrors > as additional ‘as’ parameters (again client-side or server-side); add ‘btih’ > parameters to the magnet links (the BTIH must be included in the server > metadata to get the full benefit of the swarming, but conversion to magnet > link format can be done client-side or server-side); then simply pass all > this to a really good parallelising backend such as Aria2c; then update any > progress data and relay pause, resume, cancel, etc. to the backend. > > One final note, as I am sure that there are a lot of GNUnet fans on > this list, is that I would try Aria2c first to see how well it can work, and > then try GNUnet or whatever else once you have a standard to benchmark > against. Both are Free Software, so no concern there. Aria2c is an > all-round download manager CLI that works with or without swarming, i.e. it > is just as good at HTTPS as it is BitTorrent, and can do both at the same > time. GNUnet has the advantage of working from SHA-256 iirc., which is > generally already included in the metadata of the repositories of various > distributions, but I think it lacks a lot of other features and stability and > ecosystem of alternative backends, compared to the BitTorrent network. > > Of course, there is no harm in including other hashes along with > BTIH, to allow people to experiment with alternative backends, while always > ensuring that what works works well. Another hash that may be useful to > include is the Tiger Tree Hash, which is structurally very similar to BTIH, > but stronger, iirc.. > > The first thing that the Guix project can do to signal interest in > this architecture is to simply include the BTIH of each package in the > repository metadata. Be it in magnet URL form or not does not matter because > the client can later convert that as needed. The important thing is an > authoritative statement in metadata that this version of this package has > this BTIH. Once that metadata is available, the game is on to implement > swarming support, be it with Aria2c as a backend (as I recommend at least > starting with) or otherwise. > > I know that this architecture works well out of first-hand experience > with APT Daemon written in Python. The only failure I had with it was lack > of upstream support. So I consider it important to first attain the upstream > approval before really investing more time into this. I seem to remember > suggesting this to the Nix project many years ago and didn't get anywhere, > and now I don't have the energy to try to improve upstream projects if they > reject my ideas, so I'll be interested to see whether you have any success > with your attempt to do the same. > > Good luck! ;-) > > Kind regards, > James. > -- > Wealth doesn't bring happiness, but poverty brings sadness. > Sent from Debian with Claws Mail, using email subaddressing as an alternative > to error-prone heuristical spam filtering. > Postal: James R. Haigh, Middle Farm, Vennington, nr. Westbury, nr. > Shrewsbury, Salop, SY5 9RG, Britain
