Hi, Christopher Baines <m...@cbaines.net> skribis:
> Ludovic Courtès <l...@gnu.org> writes: > >>> The code isn't great, there's some difficulty in extracting the single >>> file from the nar, but the biggest problem is a limitation in the guile >>> fibers web server. Currently, responses have to be read in to memory, >>> which is fine for we pages, but not great if you're trying to serve >>> files which can be multiple gigabytes in size. This also means that the >>> first byte of the response is available when all the bytes are >>> available, so the download is slow to start. >> >> That, and in practice a cache (with some eviction mechanism) would be >> necessary so nars don’t need to be extracted every time and so we can >> use sendfile(2). > > I'd actually imagined that this would be used infrequently, but yeah, if > the decompression does become a bottleneck, then some caching reverse > proxy could help reduce that. Relying on a proxy may be insufficient, because you still have incoming requests that can trigger unbounded peaks of I/O and CPU usage, and these requests may not be satisfied in time (the client may hang up before the server is done processing the nar.) >> IWBN to share as much code as possible with ‘guix publish’, which has >> great test suite coverage and is being hammered every day. Clearly the >> bit about extracting nars is specific to the nar-herder though, so that >> may prove difficult. > > I'm going to look at the Guile Fibers web server, hopefully that can be > improved to support streaming responses, which would allow removing a > lot of custom code from guix publish. By “streaming responses”, do you mean pipelining? How would that affect ‘guix publish’? > There isn't all that much code to the nar-herder though, and most of > waht is there is doing different things to guix publish, so I'm not sure > there's all that much to share. > > What I was getting at here though, ignoring the implementation, was > whether this is worthwhile to do? As in, is there benefit to having this > and being able to extend the content addressed mirrors that Guix uses? Having more content-addressed mirrors is worthwhile IMO, yes. Having two different implementations of the same interfaces may not be ideal, though, in terms of long-term maintenance cost. Thanks, Ludo’.
