Martin Simmons wrote:
> Also, you could attach gdb to each daemon and run the gdb command
>
> thread apply all bt
>
11th Nov 07-----------------------------------------------------------
Have attached /sbin/bacual-dir /sbin/bacula-fd and /sbin/bacula-sd to
gdb, run the commands
and will now wait until the error condition repeats.
Will post the output to [(gdb)info file] and [(gdb)thread apply all bt]
as soon as I have the error condition
as well as the dir/fd and sd status.
----------------------------------------------------------------------
The error has occurred again. I decided to start bacula with the init
scripts and attached gdb to the running process.
All the details descibing this problem are at the beginning of the thread.
As the job halted this was the output from the bconsole, about 200 lines
of roughly the same as below:-
...Orphaned buffer: backup-dir 8 bytes buf=9e1f010 allocated at
workq.c:167
Orphaned buffer: backup-dir 16 bytes buf=9e1eee0 allocated at jcr.c:247
Orphaned buffer: backup-dir 528 bytes buf=9e1f038 allocated at jcr.c:255
Orphaned buffer: backup-dir 528 bytes buf=9e23ab8 allocated at job.c:953
Orphaned buffer: backup-dir 528 bytes buf=9e23ce8 allocated at
job.c:1130
Orphaned buffer: backup-dir 6 bytes buf=9e23f50 allocated at
ua_server.c:105
Orphaned buffer: backup-dir 316 bytes buf=9e23f78 allocated at
ua_server.c:192
Orphaned buffer: backup-dir 804 bytes buf=9e24338 allocated at
bsock.c:429
Orphaned buffer: backup-dir 707 bytes buf=9e24c40 allocated at
mem_pool.c:198
Orphaned buffer: backup-dir 707 bytes buf=9e24680 allocated at
mem_pool.c:198
Orphaned buffer: backup-dir 24 bytes buf=9e1f268 allocated at
job.c:1153
Orphaned buffer: backup-dir 40 bytes buf=9e1f2a0 allocated at
alist.c:53...
From command: status all the only unusual output is:
Running Jobs:
JobId 166 Job holly.2007-12-06_23.25.09 is running.
Backup Job started: 07-Dec-07 01:41
Files=50,030 Bytes=12,066,088,479 Bytes/sec=407,211
Files Examined=66,825
Processing file: /etc/httpd/conf/httpd.conf
SDReadSeqNo=6 fd=7
Director connected at: 07-Dec-07 09:55
The sd status is:
backup-sd Version: 2.2.5 (09 October 2007) i686-pc-linux-gnu redhat
Enterprise release
Daemon started 06-Dec-07 11:42, 4 Jobs run since started.
Heap: heap=217,088 smbytes=160,745 max_bytes=161,943 bufs=124 max_bufs=133
Sizes: boffset_t=8 size_t=4 int32_t=4 int64_t=8
Running Jobs:
Writing: Full Backup job holly JobId=166 Volume="Thursday1"
pool="Thursday" device="LTO-2" (/dev/nst0)
spooling=0 despooling=0 despool_wait=0
Files=50,030 Bytes=12,073,480,104 Bytes/sec=404,607
FDReadSeqNo=646,525 in_msg=552497 out_msg=6 fd=8
The output from gdb thread apply all bt are as follows:-
1) bacula-dir
(gdb) thread apply all bt
Thread 8 (Thread -1210356848 (LWP 9317)):
#0 0x0038e402 in __kernel_vsyscall ()
#1 0x008e3051 in select () from /lib/libc.so.6
#2 0x0809a84f in bnet_thread_server (addrs=0x8e15b78, max_clients=20,
client_wq=0x80dcba0,
handle_client_request=0x8081f40 <handle_UA_client_request>) at
bnet_server.c:161
#3 0x08081f36 in connect_thread (arg=0x8e15b78) at ua_server.c:84
#4 0x003e245b in start_thread () from /lib/libpthread.so.0
#5 0x008ea24e in clone () from /lib/libc.so.6
Thread 7 (Thread -1220846704 (LWP 9318)):
#0 0x0038e402 in __kernel_vsyscall ()
#1 0x003e64dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#2 0x080b818f in watchdog_thread (arg=0x0) at watchdog.c:307
#3 0x003e245b in start_thread () from /lib/libpthread.so.0
#4 0x008ea24e in clone () from /lib/libc.so.6
Thread 6 (Thread -1231336560 (LWP 9320)):
#0 0x0038e402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x0809996d in read_nbytes (bsock=0x8e21460,
ptr=0xb69b4318
"(C\233�h*�\b`\024�\b\200*�\bXC\233��\037\b\b`\024�\b����c\\\f\b\b",
nbytes=4)
at bnet.c:82
#3 0x0809c136 in BSOCK::recv (this=0x8e21460) at bsock.c:381
#4 0x08081fba in handle_UA_client_request (arg=0x8e21460) at
ua_server.c:140
#5 0x080b87fc in workq_server (arg=0x80dcba0) at workq.c:357
#6 0x003e245b in start_thread () from /lib/libpthread.so.0
#7 0x008ea24e in clone () from /lib/libc.so.6
Thread 5 (Thread -1241826416 (LWP 9321)):
#0 0x0038e402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x0809996d in read_nbytes (bsock=0x8e22cf8,
ptr=0xb5fb3318
"(3��\220E�\b�,�\b�E�\bX3���\037\b\b�,�\b����c\\\f\b\b", nbytes=4) at
bnet.c:82
#3 0x0809c136 in BSOCK::recv (this=0x8e22cf8) at bsock.c:381
#4 0x08081fba in handle_UA_client_request (arg=0x8e22cf8) at
ua_server.c:140
#5 0x080b87fc in workq_server (arg=0x80dcba0) at workq.c:357
#6 0x003e245b in start_thread () from /lib/libpthread.so.0
#7 0x008ea24e in clone () from /lib/libc.so.6
Thread 4 (Thread -1252316272 (LWP 9322)):
#0 0x0038e402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x0809996d in read_nbytes (bsock=0x8e24950, ptr=0xb55b2318
"(#[��Q�\bPI�\b", nbytes=4)
at bnet.c:82
#3 0x0809c136 in BSOCK::recv (this=0x8e24950) at bsock.c:381
#4 0x0808208e in handle_UA_client_request (arg=0x8e24950) at
ua_server.c:140
#5 0x080b87fc in workq_server (arg=0x80dcba0) at workq.c:357
#6 0x003e245b in start_thread () from /lib/libpthread.so.0
#7 0x008ea24e in clone () from /lib/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 3 (Thread -1262806128 (LWP 9329)):
#0 0x0038e402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x0809996d in read_nbytes (bsock=0x8e447c8, ptr=0xb4bb1008 "~",
nbytes=4) at bnet.c:82
#3 0x0809c136 in BSOCK::recv (this=0x8e447c8) at bsock.c:381
#4 0x0805a4ab in bget_dirmsg (bs=0x8e447c8) at getmsg.c:109
#5 0x0804fda2 in wait_for_job_termination (jcr=0x8e42690) at backup.c:274
#6 0x08050534 in do_backup (jcr=0x8e42690) at backup.c:235
#7 0x0805dbb6 in job_thread (arg=0x8e42690) at job.c:290
#8 0x080607e4 in jobq_server (arg=0x80dca00) at jobq.c:464
#9 0x003e245b in start_thread () from /lib/libpthread.so.0
#10 0x008ea24e in clone () from /lib/libc.so.6
Thread 2 (Thread -1283785840 (LWP 9622)):
#0 0x0038e402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x0809996d in read_nbytes (bsock=0x8e421f0,
ptr=0xb37af128
"g\033�\b\214�z�\220&�\b\232�\v\b��z���\005\b�!�\b�", nbytes=4) at bnet.c:82
#3 0x0809c136 in BSOCK::recv (this=0x8e421f0) at bsock.c:381
#4 0x0805a4ab in bget_dirmsg (bs=0x8e421f0) at getmsg.c:109
#5 0x08065d50 in msg_thread (arg=0x8e42690) at msgchan.c:373
#6 0x003e245b in start_thread () from /lib/libpthread.so.0
#7 0x008ea24e in clone () from /lib/libc.so.6
Thread 1 (Thread -1208256256 (LWP 9314)):
#0 0x0038e402 in __kernel_vsyscall ()
#1 0x003e9616 in nanosleep () from /lib/libpthread.so.0
#2 0x08098754 in bmicrosleep (sec=60, usec=0) at bsys.c:71
#3 0x0806b274 in wait_for_next_job (one_shot_job_to_run=0x0) at
scheduler.c:130
#4 0x0804cd82 in main (argc=0, argv=0xbffc4e54) at dird.c:287
#0 0x0038e402 in __kernel_vsyscall ()
2) bacula-sd
(gdb) thread apply all bt
Thread 5 (Thread -1221436528 (LWP 7670)):
#0 0x00694402 in __kernel_vsyscall ()
#1 0x003e64dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#2 0x08096e2f in watchdog_thread (arg=0x0) at watchdog.c:307
#3 0x003e245b in start_thread () from /lib/libpthread.so.0
#4 0x0033a24e in clone () from /lib/libc.so.6
Thread 4 (Thread -1231926384 (LWP 7673)):
#0 0x00694402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x0807e18d in read_nbytes (bsock=0x8a7f460, ptr=0xb6924118
"H\027�\b\206\025\n\b\220", nbytes=4)
at bnet.c:82
#3 0x0807fbd6 in BSOCK::recv (this=0x8a7f460) at bsock.c:381
#4 0x0806032c in handle_connection_request (arg=0x8a7f460) at dircmd.c:211
#5 0x0809749c in workq_server (arg=0x80afd80) at workq.c:357
#6 0x003e245b in start_thread () from /lib/libpthread.so.0
#7 0x0033a24e in clone () from /lib/libc.so.6
Thread 3 (Thread -1210946672 (LWP 7943)):
#0 0x00694402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x0807e18d in read_nbytes (bsock=0x8a800b8, ptr=0xb7d26118
"H\027�\b\206\025\n\b\220", nbytes=4)
at bnet.c:82
#3 0x0807fbd6 in BSOCK::recv (this=0x8a800b8) at bsock.c:381
#4 0x0806032c in handle_connection_request (arg=0x8a800b8) at dircmd.c:211
#5 0x0809749c in workq_server (arg=0x80afd80) at workq.c:357
#6 0x003e245b in start_thread () from /lib/libpthread.so.0
#7 0x0033a24e in clone () from /lib/libc.so.6
Thread 2 (Thread -1263395952 (LWP 9621)):
#0 0x00694402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x0807e18d in read_nbytes (bsock=0x8a7bd10, ptr=0xb4b20e28
"h\016��\001", nbytes=4) at bnet.c:82
#3 0x0807fbd6 in BSOCK::recv (this=0x8a7bd10) at bsock.c:381
#4 0x080651c3 in do_fd_commands (jcr=0x8a81028) at fd_cmds.c:153
#5 0x08050d47 in do_append_data (jcr=0x8a81028) at append.c:270
#6 0x08065653 in append_data_cmd (jcr=0x8a81028) at fd_cmds.c:194
#7 0x08065241 in do_fd_commands (jcr=0x8a81028) at fd_cmds.c:165
#8 0x080657a2 in run_job (jcr=0x8a81028) at fd_cmds.c:128
#9 0x08065c1f in run_cmd (jcr=0x8a81028) at job.c:195
#10 0x080605bd in handle_connection_request (arg=0x8a7d380) at dircmd.c:229
#11 0x0809749c in workq_server (arg=0x80afd80) at workq.c:357
#12 0x003e245b in start_thread () from /lib/libpthread.so.0
#13 0x0033a24e in clone () from /lib/libc.so.6
Thread 1 (Thread -1208846112 (LWP 7667)):
#0 0x00694402 in __kernel_vsyscall ()
#1 0x00333051 in select () from /lib/libc.so.6
#2 0x0807f06f in bnet_thread_server (addrs=0x8a7b758, max_clients=41,
client_wq=0x80afd80,
handle_client_request=0x80601e0 <handle_connection_request(void*)>)
at bnet_server.c:161
---Type <return> to continue, or q <return> to quit---
#3 0x0804b354 in main (argc=<value optimized out>, argv=0x0) at
stored.c:265
#0 0x00694402 in __kernel_vsyscall ()
3) bacula-fd
(gdb) thread apply all bt
Thread 4 (Thread -1211143280 (LWP 7725)):
#0 0x00d7f402 in __kernel_vsyscall ()
#1 0x003e64dc in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#2 0x0807c18f in watchdog_thread (arg=0x0) at watchdog.c:307
#3 0x003e245b in start_thread () from /lib/libpthread.so.0
#4 0x0033a24e in clone () from /lib/libc.so.6
Thread 3 (Thread -1232122992 (LWP 7728)):
#0 0x00d7f402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x080633ad in read_nbytes (bsock=0x958bff0, ptr=0xb68f42f8
"��X\t�6\b\b\001", nbytes=4)
at bnet.c:82
#3 0x08065b76 in BSOCK::recv (this=0x958bff0) at bsock.c:381
#4 0x0805285b in handle_client_request (dirp=0x958bff0) at job.c:229
#5 0x0807c7fc in workq_server (arg=0x808d340) at workq.c:357
#6 0x003e245b in start_thread () from /lib/libpthread.so.0
#7 0x0033a24e in clone () from /lib/libc.so.6
Thread 2 (Thread -1221633136 (LWP 7941)):
#0 0x00d7f402 in __kernel_vsyscall ()
#1 0x003e8e1b in read () from /lib/libpthread.so.0
#2 0x080633ad in read_nbytes (bsock=0x958af88, ptr=0xb72f52f8
"��X\t�6\b\b\001", nbytes=4)
at bnet.c:82
#3 0x08065b76 in BSOCK::recv (this=0x958af88) at bsock.c:381
#4 0x0805285b in handle_client_request (dirp=0x958af88) at job.c:229
#5 0x0807c7fc in workq_server (arg=0x808d340) at workq.c:357
#6 0x003e245b in start_thread () from /lib/libpthread.so.0
#7 0x0033a24e in clone () from /lib/libc.so.6
Thread 1 (Thread -1209042720 (LWP 7724)):
#0 0x00d7f402 in __kernel_vsyscall ()
#1 0x00333051 in select () from /lib/libc.so.6
#2 0x0806428f in bnet_thread_server (addrs=0x958a4c8, max_clients=20,
client_wq=0x808d340,
handle_client_request=0x80526e0 <handle_client_request(void*)>) at
bnet_server.c:161
#3 0x0804b413 in main (argc=0, argv=0x0) at filed.c:227
#0 0x00d7f402 in __kernel_vsyscall ()
Can anyone can shed some light?
TIA Damian
--
Damian Brasher
Systems Admin/Prog
OMII-UK
Southampton University
-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
