bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-05-03 Thread Ludovic Courtès
Hello, Mark H Weaver skribis: > Actually, IIUC, the build slaves are _already_ compressing everything, > and they always have. They compress the build outputs for transmission > back to the master machine. In the current framework, the master > machine immediately decompresses them upon receip

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-05-03 Thread Mark H Weaver
Reviving an old thread... Tobias Geerinckx-Rice writes: >> IMO, the best solution is to *never* generate nars on Hydra in response >> to client requests, but rather to have the build slaves pack and >> compress the nars, copy them to Hydra, and then serve them as static >> files using nginx. > >

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-04-18 Thread Ludovic Courtès
l...@gnu.org (Ludovic Courtès) skribis: > 2. Produce a narinfo and corresponding nar the first time they are > requested. So, the first time we receive “GET foo.narinfo”, return > 404 and spawn a thread to compute foo.narinfo and foo.nar. Return > 200 only when both are ready. >

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-04-17 Thread Ludovic Courtès
Hello, l...@gnu.org (Ludovic Courtès) skribis: > Other solutions I’ve thought about: > > 1. Produce narinfos and nars periodically rather than on-demand and > serve them as static files. > > pros: better HTTP latency and bandwidth > pros: allows us to add a Content-Length for nar

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-28 Thread Ludovic Courtès
Hey! Tobias Geerinckx-Rice skribis: > On 26/03/17 19:35, Tobias Geerinckx-Rice wrote: >> I can try to do some simple tests tomorrow. > > Two observations: > > - ‘proxy_cache_lock_timeout’ alone won't suffice to serialise requests; > ‘proxy_cache_lock_age’ must also be set to an equally ridicul

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-27 Thread Tobias Geerinckx-Rice
Guix, On 26/03/17 19:35, Tobias Geerinckx-Rice wrote: > I can try to do some simple tests tomorrow. Two observations: - ‘proxy_cache_lock_timeout’ alone won't suffice to serialise requests; ‘proxy_cache_lock_age’ must also be set to an equally ridiculously long span. Otherwise, multiple requ

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-26 Thread Tobias Geerinckx-Rice
Mark, On 24/03/17 09:12, Mark H Weaver wrote: > IIUC, with "proxy_cache_lock on", we have two choices of how other > client requests will be treated: > > [badly, ed.] Eh. You're probably (and disappointingly) right. When configuring my little cache, I had a clear idea of how such a cache sho

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-24 Thread Ludovic Courtès
Hi! Mark H Weaver skribis: > Tobias Geerinckx-Rice writes: [...] >> Are you sure? I was under the impression¹ that this is exactly what >> ‘proxy_cache_lock on;’ prevents. I'm no nginx guru, obviously, so please >> — anyone! — correct me if I'm misguided. > > I agree that "proxy_cache_lock on

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-24 Thread Mark H Weaver
Hi, Tobias Geerinckx-Rice writes: > On 23/03/17 19:36, Mark H Weaver wrote: >> One question: what will happen in the case of multiple concurrent >> requests for the same nar? Will multiple nar-pack-and-bzip2 processes >> be run on-demand? > > I think this used to be the case with the previous n

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-23 Thread Tobias Geerinckx-Rice
Ludo', On 22/03/17 23:06, Ludovic Courtès wrote: > Tobias Geerinckx-Rice skribis: >> proxy_cache_lock on; >> proxy_cache_lock_timeout 3h; #yolo >> proxy_cache_use_stale error timeout >> http_500 http_502 http_503 http_504; > I didn’t

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-23 Thread Tobias Geerinckx-Rice
Mark, On 23/03/17 19:36, Mark H Weaver wrote: > One question: what will happen in the case of multiple concurrent > requests for the same nar? Will multiple nar-pack-and-bzip2 processes > be run on-demand? I think this used to be the case with the previous nginx configuration, but the recent cha

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-23 Thread Mark H Weaver
l...@gnu.org (Ludovic Courtès) writes: > Hi again! > > Until now hydra.gnu.org was using Hydra (the software) to serve not only > the Web interface but also all the .narinfo and /nar URLs (substitute > meta-data and substitutes). > > Starting from now, hydra.gnu.org directs all .narinfo and corres

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-23 Thread Ricardo Wurmus
Ludovic Courtès writes: > Until now hydra.gnu.org was using Hydra (the software) to serve not only > the Web interface but also all the .narinfo and /nar URLs (substitute > meta-data and substitutes). > > Starting from now, hydra.gnu.org directs all .narinfo and corresponding > nar requests to ‘

bug#26201: hydra.gnu.org uses ‘guix publish’ for nars and narinfos

2017-03-22 Thread Ludovic Courtès
Hi again! Until now hydra.gnu.org was using Hydra (the software) to serve not only the Web interface but also all the .narinfo and /nar URLs (substitute meta-data and substitutes). Starting from now, hydra.gnu.org directs all .narinfo and corresponding nar requests to ‘guix publish’ instead of Hy