Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

Hello Martin, Very good detective work, this subtle issue was causing concurrent database access even if the batch mode was disabled. I have pushed the patch into the Branch-9.6 git repository. Best Regards, Eric > What is the output of "show catalog" in bconsole? My guess is that you will > se

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 19.01.21 19:40, Martin Simmons wrote: I say "in theory" because it looks like Bacula 9.6.4 broke this! What is the output of "show catalog" in bconsole? My guess is that you will see db_driver=MySQ i.e. missing the final "L" of MySQL. This will prevent it from reusing the BDB object, leadi

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/19/21 3:17 PM, Phil Stracchino wrote: > Fix applied. I'll send Gentoo a patch for 9.6.7. Bug report and patch submitted to Gentoo. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/19/21 1:40 PM, Martin Simmons wrote: > To fix it, replace db_driver_len with db_driver_len+1 in this call to > bstrncpy. This has been fixed in Bacula 11 because the seemingly minor > problem with "status catalog" was reported in > https://bugs.bacula.org/view.php?id=2551 but it looks like th

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/19/21 1:40 PM, Martin Simmons wrote: > In theory jobs running without batch insert share a single BDB object in > memory, but it is used with a lock by a per-job thread (which is why the db > library has to be thread-safe). See setup_job(). > > I say "in theory" because it looks like Bacula

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

> On Mon, 18 Jan 2021 16:17:04 -0500, Phil Stracchino said: > > On 1/18/21 2:08 PM, Martin Simmons wrote: > > Looks reasonable to me. (*) > > > > If the INSERT failed for some other reason then the original mysql_query() > > should have failed rather than mysql_affected_rows() returning 0.

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/18/21 2:08 PM, Martin Simmons wrote: > Looks reasonable to me. (*) > > If the INSERT failed for some other reason then the original mysql_query() > should have failed rather than mysql_affected_rows() returning 0. I think if > a subsequent SELECT returns a PathId then it would be safe to as

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

> On Fri, 15 Jan 2021 14:18:32 -0500, Phil Stracchino said: > > The proper way to fix this is as follows: > > INSERT IGNORE into the table. > If mysql_affected_rows() returns 1, i.e we inserted exactly one row; > return mysql_insert_id(), which will be correct. > If mysql_affected_rows() retu

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/15/21 5:50 PM, Phil Stracchino wrote: > On 1/15/21 2:18 PM, Phil Stracchino wrote: >> If we INSERT IGNORE and there IS an existing matching record, then the >> record is NOT inserted, no error is thrown, but mysql_affected_rows() >> and mysql_insert_id() both return 0, and so does >> sql_inser

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/15/21 2:18 PM, Phil Stracchino wrote: > If we INSERT IGNORE and there IS an existing matching record, then the > record is NOT inserted, no error is thrown, but mysql_affected_rows() > and mysql_insert_id() both return 0, and so does > sql_insert_autokey_record(). Footnote I omitted: while

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/15/21 1:31 PM, Martin Simmons wrote: > Will sql_insert_autokey_record (i.e. mysql_insert_id) still return the correct > id after this? If not, then you will get a subtle mess. That is a good question. Per official documentation, "If you use INSERT IGNORE and the row is ignored, the AUTO_IN

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/15/21 1:37 PM, Martin Simmons wrote: >> On Fri, 15 Jan 2021 12:19:29 -0500, Phil Stracchino said: >> >> This both filters already-existing Paths out of the batch set much more >> efficiently, AND does not fail if another job inserts those same Paths >> first. If you compare the execution

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

> On Fri, 15 Jan 2021 12:19:29 -0500, Phil Stracchino said: > > This both filters already-existing Paths out of the batch set much more > efficiently, AND does not fail if another job inserts those same Paths > first. If you compare the execution plans for the two queries you will > see that

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

> On Fri, 15 Jan 2021 11:11:15 -0500, Phil Stracchino said: > > On 1/15/21 9:54 AM, Phil Stracchino wrote: > > I will TRY to find time to look at the code and see whether I can patch > > it myself and test. I might get time today or this weekend. > > Actually, I managed to fit in a look righ

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/15/21 11:23 AM, Phil Stracchino wrote: > On 1/15/21 11:11 AM, Phil Stracchino wrote: >> On 1/15/21 9:54 AM, Phil Stracchino wrote: >>> I will TRY to find time to look at the code and see whether I can patch >>> it myself and test. I might get time today or this weekend. >> >> Actually, I mana

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/15/21 11:11 AM, Phil Stracchino wrote: > On 1/15/21 9:54 AM, Phil Stracchino wrote: >> I will TRY to find time to look at the code and see whether I can patch >> it myself and test. I might get time today or this weekend. > > Actually, I managed to fit in a look right now. I've now rebuilt

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/15/21 9:54 AM, Phil Stracchino wrote: > I will TRY to find time to look at the code and see whether I can patch > it myself and test. I might get time today or this weekend. Actually, I managed to fit in a look right now. I've now rebuilt with the following patch: _ --- src/cats/sql_c

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

On 1/15/21 8:21 AM, Radosław Korzeniewski wrote: > Hello Phil, > > czw., 14 sty 2021 o 19:19 Phil Stracchino > napisał(a): > > I am increasingly convinced one of the changes in 9.6.7 introduced a but > of some kind here, PROBABLY involving a race condition be

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

Hello Phil, czw., 14 sty 2021 o 19:19 Phil Stracchino napisał(a): > I am increasingly convinced one of the changes in 9.6.7 introduced a but > of some kind here, PROBABLY involving a race condition between > concurrent jobs. > Could you share your thoughts on this? Please. -- Radosław Korzeni

Re: [Bacula-devel] Something amiss in bacula-9.6.7: Possible race condition

Three more jobs failed last night, all on duplicate entries in an INSERT INTO Path. Once again, the errored Jobs did not terminate. I am increasingly convinced one of the changes in 9.6.7 introduced a but of some kind here, PROBABLY involving a race condition between concurrent jobs. -- Phil