Hi,
Mathieu Othacehe skribis:
>> Isn’t that the real problem, that we’re doing one transaction per
>> derivation?
>
> Is it really better in term of performance to send batch of queries
> within a single transaction? I haven't tried it yet.
I’m definitely not a database expert, but I think ther
Hey Ludo,
> Isn’t that the real problem, that we’re doing one transaction per
> derivation?
Is it really better in term of performance to send batch of queries
within a single transaction? I haven't tried it yet.
I think that the real bottleneck was having N fibers fighting over 4
workers to e
Hello!
Mathieu Othacehe skribis:
> Turns out, once an evaluation is over, new builds are registered. This
> registration tries to insert a new build for each derivation returned by
> the evaluation phase. If the new build does not add a new output, the
> insertion query is then rollbacked. This
Hello,
> I'm trying locally to spawn database workers dedicated to registration,
> that execute the evaluation queries in a single batch. Let's see if it
> helps.
I added 4 registration dedicated database workers and removed from
registration all accesses to the store. Now the evaluation return
Hello,
> In the best case of one pending evaluation, the registration duration is
> reduced from 1800 seconds to 320 seconds. I think that the gain is way
> larger when there are multiple pending evaluations.
>
> Pushed a fix as 461e07e14e1c8013343c0a2cb26c0e022e10d5e4.
While this improves the
For future reference, here's a small addition. As I said, we are using
the Cuirass SQLite database in WAL mode. This means that insertion
queries are added to a separate cuirass.db-wal file. Once in a while,
this file is supposed to be merged in the main cuirass.db file.
This mechanism known as
Hello,
> As each evaluation has to register around 50k builds, there might be
> some sort of "writing" contention on the database, explaining the long
> build registration time.
Turns out, once an evaluation is over, new builds are registered. This
registration tries to insert a new build for e
Hello,
The attached screenshot shows that 9 evaluations are currently "In
progress" for "guix-master" specification.
Evaluations 16725 to 16738 are completed, 7 hours ago and 56 minutes ago
respectively. They are still shown as "In progress" because the
"register" phase in "build-packages" is no
