So when I find these jobs that are hung. They all do have Sleep state and a
longer time value. I'm just unsure what that means from a MySQL perspective.
The thread is in a sleep state. To my way of thinking that would mean that
Bacula has opened the thread but isn't using it.
What could the director be waiting on?
-Shon
On Thu, Feb 19, 2009 at 6:12 PM, Mingus Dew <shon.steph...@gmail.com> wrote:
> I checked mysql during one of these jobs thats just running. For one thing,
> I can see that other jobs start, run, complete, terminate all while this
> particular job is just hanging.
>
> Writing: Incremental Backup job Canopy_OLTPA_Lvl1_Tape JobId=22789
> Volume="B00046"
> pool="Canopy_Tapes" device="Ultrium-TD3" (/dev/rmt/0cbn)
> spooling=0 despooling=0 despool_wait=0
> Files=158 Bytes=50,216,398,373 Bytes/sec=5,228,151
> FDReadSeqNo=767,589 in_msg=767117 out_msg=5 fd=5
>
>
> r...@mt-back4: mysqladmin processlist
>
> +------+-------------+-----------+--------+---------+---------+-----------------------------------------------------------------------+------------------+
> | Id | User | Host | db | Command | Time |
> State |
> Info |
>
> +------+-------------+-----------+--------+---------+---------+-----------------------------------------------------------------------+------------------+
> | 2 | system user | | | Connect | 3134516 | Has read
> all relay log; waiting for the slave I/O thread to update it |
> | 6177 | bacula | localhost | bacula | Sleep | 20
> |
> | |
> | 6179 | bacula | localhost | bacula | Sleep | 44
> |
> | |
>
> +------+-------------+-----------+--------+---------+---------+-----------------------------------------------------------------------+------------------+
>
> So its got a long sleep time (6179). So what? That doesn't really
> illuminate anything. Its not like MySQL is starved for resources. I'm not
> buying that this is a MySQL issue though.
>
> -Shon
>
>
> On Thu, Feb 19, 2009 at 10:04 AM, Ryan Novosielski <novos...@umdnj.edu>wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> You should check it out with 'mysqladmin processlist' -- you may learn
>> that something is going on.
>>
>> =R
>>
>> Mingus Dew wrote:
>> > Sorry. I forgot to mention MySQL 4. Its still responding. I've tested it
>> > while the jobs were hung. Also, if I cancel the hung job, the next tape
>> > job in queue starts and completes just fine.
>> >
>> > -Shon
>> >
>> > On Wed, Feb 18, 2009 at 4:13 PM, Ryan Novosielski <novos...@umdnj.edu
>> > <mailto:novos...@umdnj.edu>> wrote:
>> >
>> > Mingus Dew wrote:
>> >> Hi all,
>> >> Been using Bacula 2.4.2 on Solaris 10_x86 for almost 2 years now.
>> >> Recently tape backups have been entering into a state that I can only
>> >> describe as "limbo".
>> >
>> >> If I check the status of the director, I may see something like
>> >
>> >> Running Jobs:
>> >> JobId Level Name Status
>> >> ======================================================================
>> >> 22649 Increme RMAN_A_Lvl1_Tape.2009-02-17_13.30.36 is running
>> >> 22650 Increme RMAN_B_Lvl1_Tape.2009-02-17_13.30.38 is waiting on max
>> >> Storage jobs
>> >> 22651 Increme RMAN_PROD_Lvl1_Tape.2009-02-17_14.00.40 is waiting on
>> >> max Storage jobs
>> >> 22652 Increme RMAN_BI_Lvl1_Tape.2009-02-17_14.00.42 is waiting
>> > on max
>> >> Storage jobs
>> >> 22653 Increme RMAN_COG_Lvl1_Tape.2009-02-17_14.00.44 is waiting
>> > on max
>> >> Storage jobs
>> >
>> >> If I check the status of the running jobid or the tape device, it will
>> >> show this:
>> >
>> >> Used Volume status:
>> >> B00046 on device "Ultrium-TD3" (/dev/rmt/0cbn)
>> >> Reader=0 writers=0 devres=0 volinuse=1
>> >> ====
>> >
>> >> Data spooling: 0 active jobs, 0 bytes; 80 total jobs,
>> > 47,799,329,608 max
>> >> bytes/job.
>> >> Attr spooling: 0 active jobs, 0 bytes; 80 total jobs, 40,616 max
>> > bytes.
>> >
>> >> Basically, tape is mounted and reserved, job is showing a "is running"
>> >> status, but nothing is happening. Because I lack any monitoring of how
>> >> long jobs have been running,
>> >> these have sat for as many as 3 days without changing status,
>> > erroring,
>> >> or completing. This backs up subsequent jobs that have been
>> > waiting for
>> >> the tape device.
>> >> The only commonality that I've seen is that they are tape jobs. Other
>> >> than that, the level, fileset, etc. are different.
>> >
>> >> On one occasion when I cancelled one of these long running jobs, I got
>> >> an error
>> >
>> >> Hostname : BUG!
>> >> Date : 2009-02-11 14:00:30
>> >> Severity : err
>> >
>> >> unregister_watchdog_unlocked called before start_watchdog
>> >
>> >
>> >> Hostname : BUG!
>> >> Date : 2009-02-11 14:00:30
>> >> Severity : err
>> >
>> >> bacula-dir[20200]: [ID 702911 daemon.error] backup4.director: ABORTING
>> >> due to ERROR in watchdog.c:206
>> >
>> >> If anyone has any advice on what might be happening, I would really
>> >> appreciate your responses.
>> >
>> > Check to see what, if anything, your backend database is doing. You
>> > don't tell us what it is, so I can't be any more specific.
>> >
>>
>> -
>>
>> ------------------------------------------------------------------------------
>> Open Source Business Conference (OSBC), March 24-25, 2009, San
>> Francisco, CA
>> - -OSBC tackles the biggest issue in open source: Open Sourcing the
>> Enterprise
>> - -Strategies to boost innovation and cut costs with open source
>> participation
>> - -Receive a $600 discount off the registration fee with the source
>> code: SFAD
>> http://p.sf.net/sfu/XcvMzF8H
>> _______________________________________________
>> Bacula-users mailing list
>> Bacula-users@lists.sourceforge.net
>> <mailto:Bacula-users@lists.sourceforge.net>
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>>
>>
>> > ------------------------------------------------------------------------
>>
>> >
>> ------------------------------------------------------------------------------
>> > Open Source Business Conference (OSBC), March 24-25, 2009, San
>> Francisco, CA
>> > -OSBC tackles the biggest issue in open source: Open Sourcing the
>> Enterprise
>> > -Strategies to boost innovation and cut costs with open source
>> participation
>> > -Receive a $600 discount off the registration fee with the source code:
>> SFAD
>> > http://p.sf.net/sfu/XcvMzF8H
>>
>>
>> > ------------------------------------------------------------------------
>>
>> > _______________________________________________
>> > Bacula-users mailing list
>> > Bacula-users@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>>
>> - --
>> ---- _ _ _ _ ___ _ _ _
>> |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Systems Programmer II
>> |$&| |__| | | |__/ | \| _| |novos...@umdnj.edu - 973/972.0922 (2-0922)
>> \__/ Univ. of Med. and Dent.|IST/CST - NJMS Medical Science Bldg - C630
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.9 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>>
>> iEYEARECAAYFAkmddP4ACgkQmb+gadEcsb5qMgCfYduk9xEwWstO45TzE4eYVDaZ
>> Ci8An1Q4nDRHjAdWIS/2Rg+z1leoP2ai
>> =6LS4
>> -----END PGP SIGNATURE-----
>>
>>
>> ------------------------------------------------------------------------------
>> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco,
>> CA
>> -OSBC tackles the biggest issue in open source: Open Sourcing the
>> Enterprise
>> -Strategies to boost innovation and cut costs with open source
>> participation
>> -Receive a $600 discount off the registration fee with the source code:
>> SFAD
>> http://p.sf.net/sfu/XcvMzF8H
>> _______________________________________________
>> Bacula-users mailing list
>> Bacula-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>>
>
------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
