Did you also get an email about this crash (see the postfix lines in the log)? If not, check /usr/sbin/btraceback for the email address. It should contain the gdb information needed to diagnose this. If so, please post it here.
__Martin >>>>> On Sun, 1 Mar 2020 23:22:59 +0000, Chaz Vidal said: > > Greetings all, > Our Bacula system crashed on Friday with a segmentation violation. > > The system has been attempting to do a full backup of over 130TB of data over > the past few weeks which we've appeared to have lost because of the crash. > > Feb 28 09:56:31 <<servername>> bacula-dir[4211]: Bacula interrupted by signal > 11: Segmentation violation > Feb 28 09:56:31 <<servername>> bacula-dir[4211]: Kaboom! bacula-dir, > bacula-dir got signal 11 - Segmentation violation at 28-Feb-2020 09:56:31. > Attempting traceback. > Feb 28 09:56:31 <<servername>> bacula-dir[4211]: Kaboom! exepath=/usr/sbin/ > Feb 28 09:56:31 <<servername>> bacula-dir: Bacula interrupted by signal 11: > Segmentation violation > Feb 28 09:56:31 <<servername>> bacula-dir[4211]: Calling: > /usr/sbin/btraceback /usr/sbin/bacula-dir 4211 /var/lib/bacula > Feb 28 09:56:31 <<servername>> postfix/smtpd[59719]: connect from > localhost[127.0.0.1] > Feb 28 09:56:31 <<servername>> postfix/smtpd[59719]: 71CC36008A: > client=localhost[127.0.0.1] > Feb 28 09:56:31 <<servername>> postfix/cleanup[59722]: 71CC36008A: > message-id=<20200227232631.71CC36008A@<<servername>>.company.com> > Feb 28 09:56:31 <<servername>> postfix/qmgr[14399]: 71CC36008A: > from=<root@<<servername>>.company.com>, size=593, nrcpt=1 (queue active) > Feb 28 09:56:31 <<servername>> postfix/smtpd[59719]: disconnect from > localhost[127.0.0.1] helo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 > Feb 28 09:56:31 <<servername>> bacula-dir[4211]: It looks like the traceback > worked... > Feb 28 09:56:31 <<servername>> bacula-dir[4211]: LockDump: > /var/lib/bacula/bacula.4211.traceback > Feb 28 09:56:31 <<servername>> bacula-dir[4211]: bacula-dir: lockmgr.c:1221-0 > lockmgr disabled > > I do not know how to read a traceback file to understand what may have been > going on. We are attempting to restart the backup again but unless we > understand what happened the crash may appear again. > > We are running Bacula Version: 9.4.2 > > Appreciate if anyone can share any insight? > > Attempt to dump current JCRs. njcrs=7 > threadid=0x7fb497491f40 JobId=0 JobStatus=R jcr=0x55980a04a4f8 > name=*JobMonitor*.2020-02-11_15.29.48_01 > use_count=1 killable=0 > JobType=I JobLevel= > sched_time=11-Feb-2020 15:29 start_time=11-Feb-2020 15:29 > end_time=01-Jan-1970 09:30 wait_time=01-Jan-1970 09:30 > db=(nil) db_batch=(nil) batch_started=0 > wstore=0x55980a01ff28 rstore=0x55980a01ff28 wjcr=(nil) > client=0x55980a026128 reschedule_count=0 SD_msg_chan_started=0 > threadid=0x7fb495897700 JobId=104686 JobStatus=R jcr=0x7fb48806aea8 > name=job1.2020-02-11_17.38.56_13 > use_count=2 killable=1 > JobType=B JobLevel=F > sched_time=11-Feb-2020 17:38 start_time=11-Feb-2020 17:38 > end_time=01-Jan-1970 09:30 wait_time=21-Feb-2020 16:50 > db=0x7fb4880059a8 db_batch=(nil) batch_started=0 > wstore=0x7fb48803fc18 rstore=(nil) wjcr=(nil) client=0x7fb4880481a8 > reschedule_count=0 SD_msg_chan_started=1 > BDB=0x7fb4880059a8 db_name=bacula db_user=bacula connected=true > cmd="UPDATE Media SET InChanger=0, Slot=0 WHERE Slot=25 AND StorageId > IN (10) AND MediaId!=794" changes=1814 > RWLOCK=0x7fb4880059c0 w_active=0 w_wait=0 > threadid=0x7fb47e7fc700 JobId=104687 JobStatus=R jcr=0x7fb488068978 > name=job2.2020-02-11_17.40.43_14 > use_count=2 killable=1 > JobType=B JobLevel=F > sched_time=11-Feb-2020 17:40 start_time=11-Feb-2020 17:40 > end_time=01-Jan-1970 09:30 wait_time=01-Jan-1970 09:30 > db=0x7fb4880059a8 db_batch=(nil) batch_started=0 > wstore=0x7fb48803fc18 rstore=(nil) wjcr=(nil) client=0x7fb4880481a8 > reschedule_count=0 SD_msg_chan_started=1 > BDB=0x7fb4880059a8 db_name=bacula db_user=bacula connected=true > cmd="UPDATE Media SET InChanger=0, Slot=0 WHERE Slot=25 AND StorageId > IN (10) AND MediaId!=794" changes=1814 > RWLOCK=0x7fb4880059c0 w_active=0 w_wait=0 > threadid=0x7fb43f7fe700 JobId=104928 JobStatus=R jcr=0x7fb44805fa88 > name=job3.2020-02-14_15.47.06_47 > use_count=2 killable=1 > JobType=B JobLevel=F > sched_time=14-Feb-2020 15:46 start_time=14-Feb-2020 15:47 > end_time=01-Jan-1970 09:30 wait_time=27-Feb-2020 22:21 > db=0x7fb4880059a8 db_batch=(nil) batch_started=0 > wstore=0x7fb448034678 rstore=(nil) wjcr=(nil) client=0x7fb44803c148 > reschedule_count=0 SD_msg_chan_started=1 > BDB=0x7fb4880059a8 db_name=bacula db_user=bacula connected=true > cmd="UPDATE Media SET InChanger=0, Slot=0 WHERE Slot=25 AND StorageId > IN (10) AND MediaId!=794" changes=1814 > RWLOCK=0x7fb4880059c0 w_active=0 w_wait=0 > threadid=0x7fb43e7fc700 JobId=105616 JobStatus=R jcr=0x55980a005fe8 > name=job4.2020-02-21_21.30.01_16 > use_count=2 killable=1 > JobType=B JobLevel=F > sched_time=21-Feb-2020 21:30 start_time=24-Feb-2020 23:36 > end_time=01-Jan-1970 09:30 wait_time=01-Jan-1970 09:30 > db=0x7fb4880059a8 db_batch=(nil) batch_started=0 > wstore=0x7fb448033b78 rstore=(nil) wjcr=(nil) client=0x7fb44803e9e8 > reschedule_count=0 SD_msg_chan_started=1 > BDB=0x7fb4880059a8 db_name=bacula db_user=bacula connected=true > cmd="UPDATE Media SET InChanger=0, Slot=0 WHERE Slot=25 AND StorageId > IN (10) AND MediaId!=794" changes=1814 > RWLOCK=0x7fb4880059c0 w_active=0 w_wait=0 > threadid=0x7fb43effd700 JobId=0 JobStatus=C jcr=0x7fb47800b2e8 > name=-Console-.2020-02-27_08.39.19_09 > use_count=1 killable=0 > JobType=U JobLevel=F > sched_time=27-Feb-2020 08:39 start_time=27-Feb-2020 08:39 > end_time=01-Jan-1970 09:30 wait_time=01-Jan-1970 09:30 > db=0x7fb4880059a8 db_batch=(nil) batch_started=0 > wstore=0x7fb448035cd8 rstore=0x7fb448034c18 wjcr=(nil) > client=0x7fb44803ae18 reschedule_count=0 SD_msg_chan_started=0 > BDB=0x7fb4880059a8 db_name=bacula db_user=bacula connected=true > cmd="UPDATE Media SET InChanger=0, Slot=0 WHERE Slot=25 AND StorageId > IN (10) AND MediaId!=794" changes=1814 > RWLOCK=0x7fb4880059c0 w_active=0 w_wait=0 > threadid=0x7fb45f7fe700 JobId=0 JobStatus=C jcr=0x7fb3f400e148 > name=-Console-.2020-02-28_09.00.13_35 > use_count=1 killable=0 > JobType=U JobLevel=F > sched_time=28-Feb-2020 09:00 start_time=28-Feb-2020 09:00 > end_time=01-Jan-1970 09:30 wait_time=01-Jan-1970 09:30 > db=(nil) db_batch=(nil) batch_started=0 > wstore=0x7fb448035cd8 rstore=0x7fb448034c18 wjcr=(nil) > client=0x7fb44803ae18 reschedule_count=0 SD_msg_chan_started=0 > List plugins. Hook count=0 > > > > _______________________________________________ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
