On 30.01.2020 17.04, Konstantin Ryabitsev wrote:
On Sun, 26 Jan 2020 at 11:19, Kristian Klausen via arch-mirrors
<arch-mirrors@archlinux.org> wrote:
I'm considering setting up a Arch Linux mirror and I'm considering a
different design.
So instead of mirroring the whole thing, the idea is to mirror only the
database files (core.db etc) and download the packages on demand from a
Tier 1 mirror (and let nginx cache them). By doing it that way, I only
download requested packages from the Tier 1 mirrors, instead of
downloading the whole thing (saving Tier 1 bandwidth).
To provide even better performance a CDN (ex: Cloudflare) could be used
to provide more caching. So we end up with a setup like this:
Cloudflare -> Nginx cache -> Tier1 mirrors (nginx with multiple upstream)
Do I miss something? Is this a bad idea?
If you are trying to save Tier1 some bandwidth, you'll probably
actually end up causing them more problems due to increased random
seek waits. Tier1 mirrors may not necessarily have fast storage -- for
example, all kernel.org mirror nodes have terabytes of spinning rust
and about half-a-TB of ssd used via lvm-cache. It works great for
Tier1 setups because most Tier2 mirrors want the same set of recent
updates that are served out of ssd cache. If a new mirror comes along
and wants to slurp and entire distro, that is fine too, because even
if there's higher iowait latency, the Tier2 mirror isn't working
against any HTTP timeouts or impatient clients and doesn't care if the
data arrives at a slower rate due to higher iowait. Tier1 can also
tell Tier2 mirror "I'm overloaded right now, please try again later"
and it'll be fine as most Tier2 mirrors can wait an hour or two before
receiving updates.
Making Tier1 mirrors a "cold cache" for your setup will likely cause
more disk thrash for them, but will also result in poorer service for
people using your mirror due to the reasons I listed above.
Tier 1 mirrors is also used directly by end-users (correct me if I'm wrong)
So worst-case (cache miss) my SSD-backed shared cache won't be
noticeable slower than pulling directly from the Tier 1 mirror.
Best-case (cache hit) I'm saving the Tier 1 mirror some bandwidth and
disk usage.
My idea is basically tiered caching (CDN -> Nginx SSD-backed shared
cache -> Tier 1 mirror(s)), is that worse than status quo? :)
If someone
tries to install a package and watches their download bar sit at 0 for
half a minute due to backend proxies fetching data from Tier1 origin,
that's going to result in frustrated people.
Nginx streams the data as it is received from the upstream server, so
worst-case (cache miss) the data can be delivered as fast as received
from the upstream server.
TL;DR: If you can afford CDN-fronting your mirror, that should be
mostly fine, but I would recommend against using Tier1 as your
cache-miss backend. Storage is cheap and most Tier1 mirrors have
unlimited bandwidth, so just run a Tier2 mirror (with slow/fast
storage caching) and keep local copies of everything.
-K
(mirrors.kernel.org administrator)
