On Fr, 29 Jul 2005, Kern Sibbald <[EMAIL PROTECTED]> wrote: > > What I see from this is that everything in the Director is normal. It thinks > that something like 5 jobs are running. The threads are all waiting on input > from one of the other daemons, and there is no mutex dead lock situation. So, > if everything is locked up, I suspect the problem is in one of the other > daemons. > > I recommend when it is in this state to do a "status" on all the Clients and > on the SD and see if there is anything interesting going on. Perhaps that > will tell us the right place to point the debugger.
Again, the director locked. This time it locked up at the first job
(Client Conc. Jobs = 1) and I was *not* able to connect with bconsole.
Therefore I couldn't get the status from sd or the clients.
This is what gdb of bacula-dir says:
(gdb) run -s -f -c /etc/bacula/bacula-dir.conf
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /usr/sbin/bacula-dir -s -f -c
/etc/bacula/bacula-dir.conf
[Thread debugging using libthread_db enabled]
[New Thread 1078020896 (LWP 29834)]
[New Thread 1086450608 (LWP 29837)]
[New Thread 1094839216 (LWP 29838)]
[New Thread 1103227824 (LWP 29857)]
backup-dir: dird.c:438 Director's configuration file reread.
[Thread 1103227824 (LWP 29857) exited]
[New Thread 1103227824 (LWP 30275)]
backup-dir: dird.c:438 Director's configuration file reread.
[Thread 1103227824 (LWP 30275) exited]
[New Thread 1103227824 (LWP 30574)]
[New Thread 1111620528 (LWP 30575)]
[New Thread 1120074672 (LWP 30577)]
[New Thread 1128463280 (LWP 30578)]
[New Thread 1136851888 (LWP 30580)]
[New Thread 1145240496 (LWP 30581)]
[New Thread 1153629104 (LWP 30582)]
[New Thread 1162017712 (LWP 30644)]
Program received signal SIGINT, Interrupt.
[Switching to Thread 1078020896 (LWP 29834)]
0x401a6436 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
(gdb) thread apply all bt
Thread 13 (Thread 1162017712 (LWP 30644)):
#0 0x401a4295 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#1 0x080959fc in rwl_writelock (rwl=0x80c5b80) at rwlock.c:231
#2 0x0808c8d2 in lock_jcr_chain () at jcr.c:544
#3 0x0808bd56 in new_jcr (size=1162017184, daemon_free_jcr=0xfffffffc)
at jcr.c:218
#4 0x0807458c in new_control_jcr (base_name=0xfffffffc <Address
0xfffffffc out of bounds>, job_type=-4)
at ua_server.c:90
#5 0x0807468e in handle_UA_client_request (arg=0x80e9d60) at
ua_server.c:122
#6 0x0809e4db in workq_server (arg=0x80c5920) at workq.c:347
#7 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#8 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 12 (Thread 1153629104 (LWP 30582)):
#0 0x401a6436 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1 0x401a3893 in _L_mutex_lock_26 () from /lib/tls/libpthread.so.0
#2 0x080c5b80 in jobs ()
#3 0x00000000 in ?? ()
#4 0x00000001 in ?? ()
#5 0x00000001 in ?? ()
#6 0x00000000 in ?? ()
#7 0x44c2fad8 in ?? ()
#8 0x0805b982 in jobq_server (arg=0x80c57a0) at jobq.c:675
#9 0x0805b982 in jobq_server (arg=0x80c57a0) at jobq.c:675
#10 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#11 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 11 (Thread 1145240496 (LWP 30581)):
#0 0x401a6436 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1 0x401a3893 in _L_mutex_lock_26 () from /lib/tls/libpthread.so.0
#2 0x080c5b80 in jobs ()
#3 0x00000000 in ?? ()
#4 0x00000001 in ?? ()
#5 0x00000001 in ?? ()
#6 0x00000000 in ?? ()
#7 0x4442fad8 in ?? ()
#8 0x0805b982 in jobq_server (arg=0x80c57a0) at jobq.c:675
#9 0x0805b982 in jobq_server (arg=0x80c57a0) at jobq.c:675
#10 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#11 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 10 (Thread 1136851888 (LWP 30580)):
#0 0x401a6436 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1 0x401a3893 in _L_mutex_lock_26 () from /lib/tls/libpthread.so.0
#2 0x080c5b80 in jobs ()
#3 0x00000000 in ?? ()
#4 0x00000001 in ?? ()
#5 0x00000001 in ?? ()
#6 0x00000000 in ?? ()
#7 0x43c2fad8 in ?? ()
#8 0x0805b982 in jobq_server (arg=0x80c57a0) at jobq.c:675
#9 0x0805b982 in jobq_server (arg=0x80c57a0) at jobq.c:675
#10 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#11 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 9 (Thread 1128463280 (LWP 30578)):
#0 0x401a4295 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#1 0x080959fc in rwl_writelock (rwl=0x80c5b80) at rwlock.c:231
#2 0x0808c8d2 in lock_jcr_chain () at jcr.c:544
#3 0x0805bea4 in jobq_server (arg=0x80c57a0) at jobq.c:582
#4 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#5 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 8 (Thread 1120074672 (LWP 30577)):
#0 0x401a66a1 in __read_nocancel () from /lib/tls/libpthread.so.0
#1 0x08084d4c in read_nbytes (bsock=0x80e1140, ptr=0x42c2f82c "@",
nbytes=4) at bnet.c:72
#2 0x08085067 in bnet_recv (bsock=0x80e1140) at bnet.c:175
#3 0x08055d88 in bget_dirmsg (bs=0x80e1140) at getmsg.c:79
#4 0x0805e508 in msg_thread (arg=0x80dcc48) at msgchan.c:235
#5 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#6 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 7 (Thread 1111620528 (LWP 30575)):
#0 0x401a66a1 in __read_nocancel () from /lib/tls/libpthread.so.0
#1 0x08084d4c in read_nbytes (bsock=0x80e5f20,
ptr=0x4241f08c "9Q\b\bHÌ\r\b _\016\bXòAB\210]\005\b
[EMAIL PROTECTED]<@[EMAIL PROTECTED]
", nbytes=4) at bnet.c:72
#2 0x08085067 in bnet_recv (bsock=0x80e5f20) at bnet.c:175
#3 0x08055d88 in bget_dirmsg (bs=0x80e5f20) at getmsg.c:79
#4 0x0804daf8 in wait_for_job_termination (jcr=0x80dcc48) at
backup.c:243
#5 0x0804da23 in do_backup (jcr=0x80dcc48) at backup.c:207
#6 0x08058946 in job_thread (arg=0x80dcc48) at job.c:215
#7 0x0805c08a in jobq_server (arg=0x80c57a0) at jobq.c:444
#8 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#9 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 6 (Thread 1103227824 (LWP 30574)):
#0 0x401a6436 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1 0x401a3893 in _L_mutex_lock_26 () from /lib/tls/libpthread.so.0
#2 0x080c5b80 in jobs ()
#3 0x080c70b8 in ?? ()
#4 0x00000001 in ?? ()
#5 0x00000001 in ?? ()
#6 0x00000000 in ?? ()
#7 0x41c1ead8 in ?? ()
#8 0x0805b982 in jobq_server (arg=0x80c57a0) at jobq.c:675
#9 0x0805b982 in jobq_server (arg=0x80c57a0) at jobq.c:675
#10 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#11 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 3 (Thread 1094839216 (LWP 29838)):
#0 0x401a6436 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1 0x401a3893 in _L_mutex_lock_26 () from /lib/tls/libpthread.so.0
#2 0x080c5b80 in jobs ()
#3 0x00000000 in ?? ()
#4 0x00000000 in ?? ()
#5 0x080e8f50 in ?? ()
#6 0x080e8f60 in ?? ()
#7 0x4141ea58 in ?? ()
#8 0x0808c9a8 in get_next_jcr (prev_jcr=0x80c5b80) at jcr.c:581
#9 0x0808c9a8 in get_next_jcr (prev_jcr=0x80c5b80) at jcr.c:581
#10 0x080590c8 in job_monitor_watchdog (self=0x80c5b80) at job.c:386
#11 0x0809dad6 in watchdog_thread (arg=0x0) at watchdog.c:257
#12 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#13 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 2 (Thread 1086450608 (LWP 29837)):
#0 0x4036ca27 in select () from /lib/tls/libc.so.6
#1 0x080877e0 in bnet_thread_server (addrs=0x40c1eb90,
max_clients=-514, client_wq=0x80c5920,
handle_client_request=0xfffffdfe) at bnet_server.c:154
#2 0x08074569 in connect_thread (arg=0xfffffdfe) at ua_server.c:79
#3 0x401a1b63 in start_thread () from /lib/tls/libpthread.so.0
#4 0x4037318a in clone () from /lib/tls/libc.so.6
Thread 1 (Thread 1078020896 (LWP 29834)):
#0 0x401a6436 in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#1 0x401a3893 in _L_mutex_lock_26 () from /lib/tls/libpthread.so.0
#2 0x00000006 in ?? ()
#3 0x00000069 in ?? ()
#4 0x00000005 in ?? ()
#5 0x000000d1 in ?? ()
#6 0xffffffff in ?? ()
#7 0x080e8f50 in ?? ()
#8 0xbffff958 in ?? ()
#9 0x0805afdb in jobq_add (jq=0x80c57a0, jcr=0x0) at jobq.c:240
#10 0x0805afdb in jobq_add (jq=0x80c57a0, jcr=0xffffffff) at jobq.c:240
#11 0x080585fb in run_job (jcr=0x80e8f50) at job.c:140
#12 0x0804b376 in main (argc=135171920, argv=0x80a0a58) at dird.c:241
I could run bacula-sd and bacula-fd on the client paris (at which
usually the jobs stop) under the gdb, too (now, that I have the debug
binaries available).
Regards
Volker
--
Volker Sauer * Alexanderstrasse 39/217 * 64283 Darmstadt
Telefon: 06151-154260 * Mobil: 0179-6901475 * ICQ#98164307
mailto:[EMAIL PROTECTED] * http://www.volker-sauer.de
PGPKey-Fingerprint: DB2611C7B12E0B2739992E4F7E354E4D5DD5D0E0
signature.asc
Description: Digital signature
