On 27.01.2020 13.15, Giancarlo Razzolini wrote:
Em janeiro 26, 2020 13:19 Kristian Klausen via arch-mirrors escreveu:
So instead of mirroring the whole thing, the idea is to mirror only
the database files (core.db etc) and download the packages on demand
from a Tier 1 mirror (and let nginx cache them). By doing it that
way, I only download requested packages from the Tier 1 mirrors,
instead of downloading the whole thing (saving Tier 1 bandwidth).
How you actually serve the files is irrelevant to us, but it might not be
irrelevant to the users, even if they don't actually *know* how you're
serving
files. I see some latency problems that can happen with such scenario.
I'm not sure latency is a big issue, the mirror won't be faster than the
"weakest link" (the upstream Tier 1 server) if the package isn't cached
and it cost a few roundtrips + TCP ramp up, but that's about it.
(nginx doesn't need to receive the whole response first, it can "stream"
the data to the client as it receive it from the upstream server)
Also, as pointed already, Tier 1 mirrors don't need their bandwidth
saved really.
I got that impression from the wiki, but private mirror !=
official/public mirror.
https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors
To provide even better performance a CDN (ex: Cloudflare) could be
used to provide more caching. So we end up with a setup like this:
Cloudflare -> Nginx cache -> Tier1 mirrors (nginx with multiple
upstream)
You'll have even more latency issues for packages you have not cached
yet.
See my previous comment.
I wonder how are you going to do invalidatios.
I'm under the impression that package files never changes? That database
files do change though, but I think that can be easily handled by either
using a low "max-age" (1-5 min) or simply using "no-cache" ("Caches must
check with the origin server for validation before using the cached copy.")
Do I miss something? Is this a bad idea?
If I do setup a mirror like that, is there any chance it could be
added as a official mirror?
As I've said, we won't care how you serve the files. As long as you
serve them and
serve the right file. But users will.
In that case, I will see if I can come up with a solution and open a
"feature request" to be a official mirror if it works out.
Regards,
Giancarlo Razzolini
