At about 80GB of a ~300GB job the Bacula File Daemon v5.0.3 on a MAC OSX 10.5.8 PPC server died and of course the backup failed.
I have been running Bacula for several years on Linux, Mac, Windows and FreeBSD and have never seen the File Daemon die like this so I thought it might make sense to post it here. I am sure this is pretty irrelevant but the server is Bacula 5.0.3 on Gentoo Linux 64bit, no tapes, just file storage. Here is the related output: Since this email is going to wrap, here is a pastebin too: http://pastebin.com/y14u6YJq --[snip]-- bacula-dir and bacula-sd on Gentoo Linux, filestorage is being used, no tapes. The client in question (see /var/log/system.log below) in OSX10.5 server on PPC 20-Jun 21:30 backup-dir JobId 24: No prior Full backup Job record found. 20-Jun 21:30 backup-dir JobId 24: No prior or suitable Full backup found in catalog. Doing FULL backup. 20-Jun 21:30 backup-dir JobId 24: Start Backup JobId 24, Job=Fileserver_No_Students.2011-06-20_21.30.00_08 20-Jun 21:36 backup-dir JobId 24: Using Device "FileStorage" 20-Jun 21:36 backup-sd JobId 24: Volume "FileTape_0014" previously written, moving to end of data. 20-Jun 21:36 backup-sd JobId 24: Ready to append to end of Volume "FileTape_0014" size=5600916898 20-Jun 21:43 backup-sd JobId 24: User defined maximum volume capacity 10,737,418,240 exceeded on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 21:43 backup-sd JobId 24: End of medium on Volume "FileTape_0014" Bytes=10,737,362,180 Blocks=166,441 at 20-Jun-2011 21:43. 20-Jun 21:43 backup-dir JobId 24: Using Volume "FileTape_0017" from 'Scratch' pool. 20-Jun 21:43 backup-sd JobId 24: Labeled new Volume "FileTape_0017" on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 21:43 backup-sd JobId 24: Wrote label to prelabeled Volume "FileTape_0017" on device "FileStorage" (/opt/bacula/BaculaFileTapes) 20-Jun 21:43 backup-sd JobId 24: New volume "FileTape_0017" mounted on device "FileStorage" (/opt/bacula/BaculaFileTapes) at 20-Jun-2011 21:43. 20-Jun 22:03 backup-sd JobId 24: User defined maximum volume capacity 10,737,418,240 exceeded on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 22:03 backup-sd JobId 24: End of medium on Volume "FileTape_0017" Bytes=10,737,415,459 Blocks=166,441 at 20-Jun-2011 22:03. 20-Jun 22:03 backup-dir JobId 24: Using Volume "FileTape_0018" from 'Scratch' pool. 20-Jun 22:03 backup-sd JobId 24: Labeled new Volume "FileTape_0018" on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 22:03 backup-sd JobId 24: Wrote label to prelabeled Volume "FileTape_0018" on device "FileStorage" (/opt/bacula/BaculaFileTapes) 20-Jun 22:03 backup-sd JobId 24: New volume "FileTape_0018" mounted on device "FileStorage" (/opt/bacula/BaculaFileTapes) at 20-Jun-2011 22:03. 20-Jun 22:27 backup-sd JobId 24: User defined maximum volume capacity 10,737,418,240 exceeded on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 22:27 backup-sd JobId 24: End of medium on Volume "FileTape_0018" Bytes=10,737,377,329 Blocks=166,440 at 20-Jun-2011 22:27. 20-Jun 22:27 backup-dir JobId 24: Using Volume "FileTape_0019" from 'Scratch' pool. 20-Jun 22:27 backup-sd JobId 24: Labeled new Volume "FileTape_0019" on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 22:27 backup-sd JobId 24: Wrote label to prelabeled Volume "FileTape_0019" on device "FileStorage" (/opt/bacula/BaculaFileTapes) 20-Jun 22:27 backup-sd JobId 24: New volume "FileTape_0019" mounted on device "FileStorage" (/opt/bacula/BaculaFileTapes) at 20-Jun-2011 22:27. 20-Jun 22:49 backup-sd JobId 24: User defined maximum volume capacity 10,737,418,240 exceeded on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 22:49 backup-sd JobId 24: End of medium on Volume "FileTape_0019" Bytes=10,737,377,315 Blocks=166,440 at 20-Jun-2011 22:49. 20-Jun 22:49 backup-dir JobId 24: Using Volume "FileTape_0020" from 'Scratch' pool. 20-Jun 22:49 backup-sd JobId 24: Labeled new Volume "FileTape_0020" on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 22:49 backup-sd JobId 24: Wrote label to prelabeled Volume "FileTape_0020" on device "FileStorage" (/opt/bacula/BaculaFileTapes) 20-Jun 22:49 backup-sd JobId 24: New volume "FileTape_0020" mounted on device "FileStorage" (/opt/bacula/BaculaFileTapes) at 20-Jun-2011 22:49. 20-Jun 23:08 backup-sd JobId 24: User defined maximum volume capacity 10,737,418,240 exceeded on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 23:08 backup-sd JobId 24: End of medium on Volume "FileTape_0020" Bytes=10,737,377,363 Blocks=166,440 at 20-Jun-2011 23:08. 20-Jun 23:08 backup-dir JobId 24: Using Volume "FileTape_0021" from 'Scratch' pool. 20-Jun 23:08 backup-sd JobId 24: Labeled new Volume "FileTape_0021" on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 23:08 backup-sd JobId 24: Wrote label to prelabeled Volume "FileTape_0021" on device "FileStorage" (/opt/bacula/BaculaFileTapes) 20-Jun 23:08 backup-sd JobId 24: New volume "FileTape_0021" mounted on device "FileStorage" (/opt/bacula/BaculaFileTapes) at 20-Jun-2011 23:08. 20-Jun 23:29 backup-sd JobId 24: User defined maximum volume capacity 10,737,418,240 exceeded on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 23:29 backup-sd JobId 24: End of medium on Volume "FileTape_0021" Bytes=10,737,377,310 Blocks=166,440 at 20-Jun-2011 23:29. 20-Jun 23:29 backup-dir JobId 24: Using Volume "FileTape_0022" from 'Scratch' pool. 20-Jun 23:29 backup-sd JobId 24: Labeled new Volume "FileTape_0022" on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 23:29 backup-sd JobId 24: Wrote label to prelabeled Volume "FileTape_0022" on device "FileStorage" (/opt/bacula/BaculaFileTapes) 20-Jun 23:29 backup-sd JobId 24: New volume "FileTape_0022" mounted on device "FileStorage" (/opt/bacula/BaculaFileTapes) at 20-Jun-2011 23:29. 20-Jun 23:49 backup-sd JobId 24: User defined maximum volume capacity 10,737,418,240 exceeded on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 23:49 backup-sd JobId 24: End of medium on Volume "FileTape_0022" Bytes=10,737,377,256 Blocks=166,440 at 20-Jun-2011 23:49. 20-Jun 23:49 backup-dir JobId 24: Using Volume "FileTape_0023" from 'Scratch' pool. 20-Jun 23:49 backup-sd JobId 24: Labeled new Volume "FileTape_0023" on device "FileStorage" (/opt/bacula/BaculaFileTapes). 20-Jun 23:49 backup-sd JobId 24: Wrote label to prelabeled Volume "FileTape_0023" on device "FileStorage" (/opt/bacula/BaculaFileTapes) 20-Jun 23:49 backup-sd JobId 24: New volume "FileTape_0023" mounted on device "FileStorage" (/opt/bacula/BaculaFileTapes) at 20-Jun-2011 23:49. 21-Jun 00:12 backup-sd JobId 24: User defined maximum volume capacity 10,737,418,240 exceeded on device "FileStorage" (/opt/bacula/BaculaFileTapes). 21-Jun 00:12 backup-sd JobId 24: End of medium on Volume "FileTape_0023" Bytes=10,737,377,226 Blocks=166,440 at 21-Jun-2011 00:12. 21-Jun 00:12 backup-dir JobId 24: Using Volume "FileTape_0024" from 'Scratch' pool. 21-Jun 00:12 backup-sd JobId 24: Labeled new Volume "FileTape_0024" on device "FileStorage" (/opt/bacula/BaculaFileTapes). 21-Jun 00:12 backup-sd JobId 24: Wrote label to prelabeled Volume "FileTape_0024" on device "FileStorage" (/opt/bacula/BaculaFileTapes) 21-Jun 00:12 backup-sd JobId 24: New volume "FileTape_0024" mounted on device "FileStorage" (/opt/bacula/BaculaFileTapes) at 21-Jun-2011 00:12. 21-Jun 00:34 backup-sd JobId 24: Fatal error: append.c:223 Network error reading from FD. ERR=No data available 21-Jun 00:34 backup-sd JobId 24: Job write elapsed time = 02:57:41, Transfer rate = 8.173 M Bytes/second 21-Jun 00:34 backup-dir JobId 24: Fatal error: Network error with FD during Backup: ERR=No data available 21-Jun 00:34 backup-dir JobId 24: Fatal error: No Job status returned from FD. 21-Jun 00:34 backup-dir JobId 24: Error: Bacula backup-dir 5.0.3 (04Aug10): 21-Jun-2011 00:34:14 Build OS: x86_64-pc-linux-gnu gentoo JobId: 24 Job: Fileserver_No_Students.2011-06-20_21.30.00_08 Backup Level: Full (upgraded from Incremental) Client: "fileserver" 5.0.3 (04Aug10) powerpc-apple-darwin9.8.0,osx,9.8.0 FileSet: "Fileserver Except Students" 2011-06-20 11:19:54 Pool: "Full" (From Run FullPool override) Catalog: "MyCatalog" (From Client resource) Storage: "FileStorage" (From Job resource) Scheduled time: 20-Jun-2011 21:30:00 Start time: 20-Jun-2011 21:36:33 End time: 21-Jun-2011 00:34:14 Elapsed time: 2 hours 57 mins 41 secs Priority: 10 FD Files Written: 0 SD Files Written: 76,543 FD Bytes Written: 0 (0 B) SD Bytes Written: 87,140,870,350 (87.14 GB) Rate: 0.0 KB/s Software Compression: None VSS: no Encryption: no Accurate: no Volume name(s): FileTape_0014|FileTape_0017|FileTape_0018|FileTape_0019|FileTape_0020|FileTape_0021|FileTape_0022|FileTape_0023|FileTape_0024 Volume Session Id: 5 Volume Session Time: 1308575076 Last Volume Bytes: 9,460,362,197 (9.460 GB) Non-fatal FD errors: 0 SD Errors: 0 FD termination status: Error SD termination status: Error Termination: *** Backup Error *** $ sudo grep bacula /var/log/system.log Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0185000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0207000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: 21-Jun 04:34 fileserver: ABORTING due to ERROR in smartall.c:145 Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: Out of memory Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: 21-Jun 04:34 fileserver: ABORTING due to ERROR in smartall.c:145 Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: Out of memory Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0185000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0207000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: Bacula interrupted by signal 10: BUS error Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0207000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: 21-Jun 04:34 fileserver: ABORTING due to ERROR in smartall.c:145 Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: Out of memory Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: Kaboom! bacula-fd, fileserver got signal 10 - BUS error. Attempting traceback. Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: Kaboom! exepath=/usr/local/sbin/ Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0207000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0185000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: bacula-fd(5150,0xf0185000) malloc: *** mmap(size=2097152) failed (error code=12) Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** error: can't allocate region Jun 21 04:34:13 osxserver org.bacula.bacula-fd[5150]: *** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0207000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0207000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0207000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0103000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0207000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:13 osxserver bacula-fd[5150]: bacula-fd(5150,0xf0185000) malloc: *** mmap(size=2097152) failed (error code=12)\n*** error: can't allocate region\n*** set a breakpoint in malloc_error_break to debug Jun 21 04:34:14 osxserver com.apple.launchd[1] (org.bacula.bacula-fd[5150]): Exited with exit code: 1 --[snip]-- -- Bill Arlofski Reverse Polarity, LLC ------------------------------------------------------------------------------ EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
