bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hi there! I have some good news! Time has passed but we didn’t remain idle :-) and the scary bug at is finding a happy end! First, Josselin introduced bindings to ‘posix_spawn’¹ in Guile, to use as the foundation of a new ‘spawn’ procedure, but also for ‘syste

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hello Maxime, Thanks for your review. I took your comments into account and actually wrote some more patches to clean up start_child of potential thread unsafe practices. Since these are not guix-specific, I sent the whole patchset to guile-devel instead [1]. Maxime Devos writes: > The process

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Josselin Poiret via Bug reports for GNU Guix schreef op do 26-05-2022 om 22:49 [+0200]: > Hello everyone, > > With all the recent platform changes, I was looking at `make > cuirass-jobs` for the first time and it left some deadlocked inferiors > as well, so I figured I could try my own solution, s

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hello everyone, With all the recent platform changes, I was looking at `make cuirass-jobs` for the first time and it left some deadlocked inferiors as well, so I figured I could try my own solution, since I've already dabbled in Guile POSIX internals! Here's a patch that rewrites open-bidirection

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Ludovic Courtès skribis: > For now, I’m going to go with the solution below, which is to use an > older revision of Guix (one where ‘open-inferior’ was using > ‘open-pipe*’) as the dependency of the ‘cuirass’ package. > > I’m running “cuirass evaluate” manually on berlin to make sure it > actuall

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hi, Christopher Baines skribis: > Maybe the store connection caching could be optional when calling > inferior-eval-with-store, and that could also switch between using > open-pipe* and primitive-fork for starting the inferior process. I don’t think that’s desirable as an interface. To me, the

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Ludovic Courtès writes: > For now, I’m going to go with the solution below, which is to use an > older revision of Guix (one where ‘open-inferior’ was using > ‘open-pipe*’) as the dependency of the ‘cuirass’ package. > > I’m running “cuirass evaluate” manually on berlin to make sure it > actuall

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

For now, I’m going to go with the solution below, which is to use an older revision of Guix (one where ‘open-inferior’ was using ‘open-pipe*’) as the dependency of the ‘cuirass’ package. I’m running “cuirass evaluate” manually on berlin to make sure it actually works. If everything goes well, I’l

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Ludovic Courtès skribis: > Notice there’s a single thread: it very much looks like the random > results one gets when forking a multithreaded process (in this case, > this one thread is a finalization thread, except it’s running in a > process that doesn’t actually have the other Guile threads).

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hi! Ludovic Courtès skribis: > Fixed in Guix commit a4994d739306abcf3f36706012fb88b35a970e6b with a > test that reproduces the issue. > > Commit d02b7abe24fac84ef1fb1880f51d56fc9fb6cfef updates the ‘guix’ > package so we should be able to reconfigure berlin now and hopefully > (crossing fingers!

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hi! Ludovic Courtès skribis: > wait4(83882,# hangs forever > > > The question to me becomes: why is ‘guix repl’ not getting EOF from its > read(0, …) call? The answer was that another child process (an inferior) was holding a copy of the socketpair file descriptor connected to that ‘guix re

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hi, Ludovic Courtès skribis: > Hi, > > Maxim Cournoyer skribis: > >> Berlin was reconfigured with this commit of Cuirass, and is now running >> the derivations with it, but so far still "In progress..." after more >> than 100 minutes [0] > > Yeah, I’m not sure about the backtrace you report, ho

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hi, Maxim Cournoyer skribis: > I'm not yet very knowledgeable about POSIX threads and shortcomings, but > this looks like a dangerous trap to have lying around :-). Is there an > alternative implementation we could come up with that would avoid the > potential problem (and warnings) entirely?

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hi, Maxim Cournoyer skribis: > Berlin was reconfigured with this commit of Cuirass, and is now running > the derivations with it, but so far still "In progress..." after more > than 100 minutes [0] Yeah, I’m not sure about the backtrace you report, however there’s again a bunch of ‘cuirass eval

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hello, Ludovic Courtès writes: > Ludovic Courtès skribis: > >> I’ve added a missing call to ‘close-inferior’. It’s a good idea, though >> I’m not entirely convinced yet it’ll solve the problem: >> >> >> https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=f087aaf685dbc7cc18f02

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hi Ludo and Mathieu! Ludovic Courtès writes: > Hi! > > Mathieu Othacehe skribis: > >> I noticed this error: https://ci.guix.gnu.org/eval/320682/log/raw. >> >> The following lines can be caused by having a multi-threaded program: >> (cuirass scripts evaluate) because of its par-for-each procedur

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Ludovic Courtès skribis: > I’ve added a missing call to ‘close-inferior’. It’s a good idea, though > I’m not entirely convinced yet it’ll solve the problem: > > > https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=f087aaf685dbc7cc18f0254895f4a4b0dfaba631 I tested it like this

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Ludovic Courtès skribis: > A plausible scenario is a synchronization issue: ‘cuirass evaluate’ got > all the data it asked for from ‘guix repl’, but it’s still waiting on > read(2) even though there’s nothing left. (There are two child ‘guix > repl’ processes and both are waiting on read(0, …).)

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hi! Mathieu Othacehe skribis: > I noticed this error: https://ci.guix.gnu.org/eval/320682/log/raw. > > The following lines can be caused by having a multi-threaded program: > (cuirass scripts evaluate) because of its par-for-each procedure, call > "primitive-fork" in open-bidirectional-pipe of (

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hello Maxim, Thanks for the report, the root cause here is the evaluation hanging. The Postgres slots starvation is just a symptom caused by having many concurrent (stuck) evaluation processes using slots. I noticed this error: https://ci.guix.gnu.org/eval/320682/log/raw. The following lines c

bug#55441: [cuirass] hang in "In progress..."; runs out of pgsql connections

Hello Guix, Since a couple days, Craps exhibits a new problem, where the evaluations on master appear stalled, showing as "In progress..." forever. Eventually, it seems the maximum number of PostgreSQL connections allowed (300, if what Ricardo wrote on #guix is accurate :-)) is reached, and we ge