Hello,
On 5/26/2006 11:05 PM, Bill Moran wrote:
Gregory Brauer <[EMAIL PROTECTED]> wrote:
At the end of our weekly cycle I would like to command
our loader to unload any tapes that are in the drive
to make for easy tape swapping. With versions of bacula
prior to 1.38 the director daemon was running as root,
so ejecting the tape was a simple matter of creating a
job that ran an "mtx unload" as a RunBeforeJob rule.
Now that bacula-dir is running as the bacula user for
security, the bacula-dir no longer has permission to run
mtx commands directly. I was wondering if there is a way
I can command bacula-sd to eject the tapes on behalf of the
director.
I use a combination of sudo and RunAfterJob to accomplish this. If there's
a better way, I'd love to hear it :)
Use the umount command from a Bacula shell?
(Also possible in a RunAfterJob like 'echo "umount XXX|bconsole -c ...'
in a shell.)
Or, perhaps, use the attached script if you run MySQL as catalog
database and want to make sure you have all vital data off-site.
Keep in mind that this script modifies your catalog, works your
autochanger and drive, and might break all of those (though I don't
think so). No warranties, no complaints, and all the usual disclaimer,
but of course I'd like to know if it works as expected...)
Arno
--
IT-Service Lehmann [EMAIL PROTECTED]
Arno Lehmann http://www.its-lehmann.de
#!/usr/bin/perl -w
# Version information
# 0.2.1 Fixed minor omission in selection of volumes to
# request (InChanger=0)
# 0.2 Test / evaluation for customer
# 0.1 Unpublished, trial runs
# 0 Development, not fully functional
# Basic volume management for Bacula.
#
# The following tasks are executed:
# - Determine which tapes can be removed from autochangers
# and brought off-site
# - Mark these volumes as "Used"
# - Print a report naming these volumes, ordered by storage
# device and slot number
# - if possible, move volumes so that a minimum number of
# magazines has to be handled
# - Determine which volumes that are not loaded can be used next
# - Print a report naming these volumes, ordered by pool they
# belong to
# - Unmount the storage devices we're working with
# - On command, scan the storage devices we're working with and
# mount them
# Currently known limitations:
# - Autochangers with more than one drive are probably not supported
# At least no testing of one has been done, and I'm pretty sure
# that there are some lines of codes that will break with such a
# setup.
# - MySQL is the only catalog database backend that is known to work.
# As this script contains some date / time arithmetic it's quite
# probable that other SQL backends won't work correctly.
# - This should have a better documentation.
# This script is copyright (C) 2006 IT-Service Arno Lehmann
# Arno Lehmann
use strict;
use DBI;
use Getopt::Long;
use Class::Date qw(date -DateParse);
use IPC::Open2;
my $version="0.2.1";
$0 =~ /.*\/([^\/]*)$/;
my $ME = $1;
my $COPY = '(C) 2006 IT-Service Lehmann, Arno Lehmann';
my (undef, $min, $hour, $mday, $mon, $year) = localtime();
my $DATE = sprintf('%4i-%02i-%02i %02i:%02i', $year+1900, $mon+1, $mday, $hour,
$min);
my %msgs = ('help'=>" This is $ME $version
$COPY
This program is distributed under the GNU public license Version 2.
For more information, see the source code or http://www.gnu.org/licenses/
Bacula is a Trademark and (C) 2000-2006 by Kern Sibbald. Please see
http://www.bacula.org for more information.
This script is used to help in managing Bacula volumes.
For a more detailed explanation, see \'perldoc $ME\'.
This script understands the following options. Options can be abbreviated
to uniqueness. Negating --option is done like --nooption. Defaults:
--help -h print this help
--host -H database host for catalog empty
--user -U database user bacula
--database -D database name of the catalog database bacula
--password -P database password empty
--database-type -T database interface to use mysql
--changer changer definitions empty
Try \"--changer help=\" for more information!
--bconsole -b bconsole prog. bconsole -c /etc/bacula/bconsole.conf
--var-file -f File to store information between program runs
/var/bacula/working/bac_vol_mgmt.state
--expunge -x look for volumes usable at time current time
--no-updates -n don't update volume status
--new-vols -i number of volumes per pools to import 8
--new-bypool -p print volumes to load ordered by pool no
--mount -m mount unmounted devices again and update
the slots
",
'changerhelp' => 'How to define changer devices.
The --changer options takes a parameter of the following form:
NAME=mtxprog|mtprog|device|control|offline|slots|scan|transfer|eject
Usually, when setting this through a shell, you will want to qoute the pipe
characters to prevent them being interpreted as program execution pipes. See
the example below.
The NAME is the name of the changer device you configure, it must match the
storage name inside Baculas catalog. The parts after the equal sign all define
separate aspects of this device. Each part can be left blank in which case a
default applies. Flags are case-insensitive, so eject is the Same as EJECT.
The fields have the following meaning an defaults:
Field meaning default
----------+------------------------------------------------------+------------
mtxprog The program to use to control this device. This could /usr/bin/mtx
be something like "mtx" or some wrapper around, for
example, FreeBSDs chio. The script builds a mtx
command using all the options you supply, so the
command you specify here MUST accept the mtx command
syntax and print similar output.
mtprog The mt program to control the tape drive. This is /usr/bin/mt
usually "mt" or some wrapper which supports at least
the rewoffl function of mt.
device This is the tape drive device to control. Currently, [none]
only one tape drive is used per storage device. As
the tape drive is used only to transfer tapes, this
should not be a problem.
control The changer control device, given to mtx. Usually you [none]
will use something like "/dev/sg0" here
offline Wether the tape drive needs an offline before the yes
tape can be unloaded.
You can use the following to indicate that offline
must be issued prior to unloading:
1 or yes or offline
To turn offlining off prior to unloading the tape:
0 or no or nooffline
slots Here, you set up which slots in the autochanger to 0-0
use for exporting tapes. Usually, you will use slots
all in one magazine or the mailslots. The format to
use is <slot number>-<slot number>, e.g. "1-8". To
only use one slot, give that slots number twice, as
in "1-1".
Note that there is no way to guarantee that all tapes
that should be removed can be put into that slot range.
scan Set up wether Bacula should actually load the tapes no
and read their labels after the tape exchange, or if
it can rely on the report from mtx. This corresponds
to the option scan in Baculas "upadte slots" command.
When not setting up to scan, an "mtx inventory" will
be issued before Bacula gets control of the device
again.
To turn on loading and reading, use:
1 or yes or scan
To turn it of, you set
0 or no or noscan
transfer This is used to tell Bacula if the autochanger can no
move tapes from slot to slot directly (i.e. if the mtx
command "transfer" can be used), or if tapes always
have to be moved through the tape drive.
Note that in the latter case, tape resorting is only
possible when at least one slot is empty.
To turn on the transfer capability, you can use these
keywords:
1 or yes or transfer
To turn this function off, you use
0 or no or load
eject Set up ejecting of the autochanger after tape no
reordering. Depending on the hardware you use, this
can, for example, result in ejecting a magazine or
opening the mail slots.
To turn on this feature, use
1 or yes or eject
To turn it off, you can use one of the following:
0 or no or noeject or keeploaded
Example: --changer HPDAT="mtx|/dev/st0|/dev/sg0|nooffline|4-6|scan||yes"
will set up a changer resource HPDAT to use "mtx -f /dev/sg0" for changer
control, "mt -f /dev/st0" for tape drive control, not issue an offline
command prior to unloading a tape, prefer slots 4 to 6 for tape exchange,
and Bacula will get an "update slots scan storage=HPDAT" command prior
to re-mounting this device. The autochanger will not be controlled with
"transfer" commands, but rather each tape that needs to be moved will be
loaded and unloaded again, to the new slot. The magazine of this device
will not be ejected after the work is done.
',
'header' => "This report was created $DATE\nby $ME $version
$COPY\n\n",
'report_rmv_head' => "The following tapes should be removed\nfrom
their storage devices and stored safely:",
'newsto_pre' => "\n==== ",
'newsto_post' => " ====\n\n",
'slot' => " Slot ",
'norem' => "--- No volumes need to be unloaded ---\n",
'noimp' => "*" x 72 . "
There are no volumes that can be (re)used. The next Backups will probably
have trouble!
" . '*' x 72 . "\n\n",
'noupdate' => '', #" > Volume not set to \"Used\"\n",
'load_from' => 'You should load these volumes for pool ',
'load_none' => 'No volumes to load. This might mean trouble when
the next jobs run!',
'load_all' => "The following volumes should be loaded:\n",
'changers' => 'Autochangers defined:',
'changer_err' => 'changer definition error:',
'changer_ign' => ' device ignored',
'nosuchdev' => 'Device not found: ',
'devbusy' => 'Device is busy and thus media will not be re-ordered:
',
'slotsprob' => 'No reordering due to problem with usable slots: ',
'slotsnoav' => 'No reordering due to no available slots in changer:
',
);
Getopt::Long::Configure("bundling");
my $showhelp = 0;
my $db_host = "";
my $db_user = "bacula";
my $db_database = "bacula";
my $db_password = "";
my $db_type = "mysql";
my $expunge_at = '';
my $dont_update = '';
my $new_vols = 8;
my $new_bypool = '';
my $debug = 0;
my %all_devices = ();
my $bconsole = 'bconsole -c /etc/bacula/bconsole.conf';
my $varfile = '/var/bacula/working/bac_vol_mgmt.state';
my $mount = '';
GetOptions("help|h"=>\$showhelp,
"debug|d=i"=>\$debug,
"host|H=s"=>\$db_host,
"user|U=s"=>\$db_user,
"database|D=s"=>\$db_database,
"password|P=s"=>\$db_password,
"database-type|T=s"=>\$db_type,
"bconsole|b=s"=>\$bconsole,
"expunge|x=s"=>\$expunge_at,
"no-updates|n"=>\$dont_update,
"new-vols|i=i"=>\$new_vols,
"new-bypool|p"=>\$new_bypool,
"changer=s%"=>\&process_changer,
"varfile|f=s"=>\$varfile,
"mount|m"=>\$mount
);
if ($debug >= 1000) {
my $dbtr = sprintf("%i", $debug / 1000);
$debug %= 1000;
DBI->trace("$dbtr");
debug_out(0, "Database tracing set to $dbtr");
}
debug_out(10,
"All Options:
showhelp: $showhelp
debug: $debug
host: $db_host
user: $db_user
database: $db_database
password: $db_password
database-type: $db_type
bconsole: $bconsole
expunge-at: $expunge_at
var-file: $varfile
mount: $mount
");
if ($debug >= 10) {
debug_out(10, $msgs{'changers'});
for $a (sort(keys(%all_devices))) {
debug_out(10, ' ' . $a);
for $b (sort(keys(%{$all_devices{$a}}))) {
debug_out(10, ' ' . $b . ' = ' . $all_devices{$a}->{$b});
}
}
}
if ($showhelp) {
show_help();
exit 0;
}
if ($mount) {
do_mount();
exit 0;
}
if ("" ne $expunge_at) {
$expunge_at = Class::Date->new($expunge_at);
} else {
$expunge_at = Class::Date->now;
}
debug_out(10, "expunge-at after parsing: $expunge_at\n");
my $dbconn = "dbi:" . $db_type. ":database=" . $db_database;
$dbconn .= ";host=" . $db_host if $db_host;
debug_out(40, "DBI connect with $dbconn");
my $h_db = DBI->connect($dbconn,
$db_user, $db_password,
{ PrintError => 0,
AutoCommit => 1 }
) || die DBI::errstr;
debug_out(10, "Have database connection $h_db");
my $h_q = $h_db->prepare('SELECT ClientId,FileSetId FROM Job WHERE Type="B" AND
JobStatus="T" GROUP BY ClientId,FileSetId;');
$h_q->execute();
my $db_res;
my @pairs;
while ($db_res=$h_q->fetchrow_hashref()) {
my %coll = ('Client'=>$db_res->{ClientId}, 'FileSet'=>$db_res->{FileSetId});
# Now we've got a list of all Client / FileSet pairs in the catalog
push @pairs, \%coll;
}
# using the list created above, we search for the jobs we need to consider
# we take the latest Full backup, the latest Differential one, and all
# incrementals after the most recent Full or Diff.
my ($i, $j, $catres);
my @jobs = ();
for $i (0 .. $#pairs) {
debug_out(20, "Unique Job $i before processing:");
for $j (keys %{$pairs[$i]}) {
debug_out(20,, " $j: $pairs[$i]{$j}");
}
$catres = $h_db->selectrow_hashref('SELECT JobId,StartTime FROM Job WHERE
Type="B" AND JobStatus="T" AND StartTime>"0-0-0 0:0:0" AND Level="F" AND
ClientId=' .
$pairs[$i]{'Client'} . ' AND FileSetId='
. $pairs[$i]{'FileSet'} .
' ORDER BY StartTime DESC LIMIT 1');
debug_out(20, " Full JobId: $catres->{'JobId'}\n StartTime:
$catres->{'StartTime'}");
$pairs[$i]{'After'}=$catres->{'StartTime'};
push @jobs, $catres->{'JobId'};
# now we've got the most current Full backup. Based on this, we search
# for Differentual ones.
$catres = $h_db->selectrow_hashref('SELECT JobId,StartTime FROM Job WHERE '
.
'Type="B" AND JobStatus="T" AND
StartTime>"' . $pairs[$i]->{'After'} .
'" AND Level="D" AND ClientId=' .
$pairs[$i]{'Client'} . ' AND FileSetId='
. $pairs[$i]{'FileSet'} .
' ORDER BY StartTime DESC LIMIT 1');
if ($catres) {
debug_out(20, " Diff JobId: $catres->{'JobId'}\n StartTime:
$catres->{'StartTime'}");
push @jobs, $catres->{'JobId'};
$pairs[$i]{'After'}=$catres->{'StartTime'};
} else {
debug_out(10, "No Differential Backups found. DBI::errstr: " .
((DBI::errstr) ? DBI::errstr : '*none*'));
}
# now we've got the most current Full and Diff backups. Based on this, we search
# for Incremental ones.
$catres = $h_db->selectall_hashref('SELECT JobId FROM Job WHERE ' .
'Type="B" AND JobStatus="T" AND
StartTime>"' . $pairs[$i]->{'After'} .
'" AND Level="I" AND ClientId=' .
$pairs[$i]{'Client'} . ' AND FileSetId='
. $pairs[$i]{'FileSet'} . ';',
'JobId');
if ($catres) {
debug_out(20, " Incr JobIds:");
my $a;
for $a (keys %{$catres}) {
debug_out(20, " JobId $a");
push @jobs, $a;
}
} else {
debug_out(10, "No Incremental Backups found. DBI::errstr: " .
((DBI::errstr) ? DBI::errstr : '*none*'));
}
debug_out(10, "Unique Job $i after processing:");
for $j (keys %{$pairs[$i]}) {
debug_out(10,, " $j: $pairs[$i]{$j}");
}
debug_out(10, "");
}
# Now we have a list of all jobs worth storing.
debug_out(10, "The following Jobs need to be handled:");
debug_out(10, @jobs);
my $all_jobs = join(',', @jobs);
my @volumes = ();
# the following query gives us the information we need to report.
$catres = $h_db->selectall_hashref('SELECT DISTINCT
Media.MediaId,VolumeName,Pool.Name AS Pool,Storage.Name AS Storage,Slot ' .
'FROM Media,JobMedia,Pool,Storage WHERE
JobId in (' . $all_jobs .
') AND Media.MediaId=JobMedia.MediaId ' .
'AND Media.StorageId=Storage.StorageId ' .
'AND Media.PoolId=Pool.PoolId ' .
'AND Media.InChanger=1 AND Media.Slot>0;',
'VolumeName');
if ($catres) {
debug_out(10, "Media to handle:");
my $a;
for $a (sort(keys %{$catres})) {
# all the media in one hash - should be ok because we don't expect
# thousands of volumes here...
debug_out(10, " Volume $a\n $catres->{$a}->{'Storage'}\n
$catres->{$a}->{'Slot'}\n $catres->{$a}->{'Pool'}");
push @volumes, $catres->{$a};
}
} else {
debug_out(10, "No Media: DBI::errstr: " .
((DBI::errstr) ? DBI::errstr : '*none*'));
}
print $msgs{'header'};
# next, we've got to check if we need to shuffle the media...
# We try to re-order tapes one device at a time...
for $a (keys(%all_devices)) {
reorder($a, $all_devices{$a}, [EMAIL PROTECTED]);
}
if ($#volumes >= 0) {
# we've got the volumes that should be stored.
# First, we sort by Storage device and Slot number to produce
# equally sorted reports.
@volumes = sort {
debug_out(30, "Comparing $a->{'VolumeName'} with $b->{'VolumeName'}");
$a->{'Storage'} cmp $b->{'Storage'} || $a->{'Slot'} <=> $b->{'Slot'}
} @volumes;
# The report itself is printed now.
print $msgs{'report_rmv_head'};
$j = '';
$h_q = $h_db->prepare('UPDATE Media SET VolStatus="Used" WHERE MediaId=?
AND VolStatus="Append";');
debug_out(10, 'Prepared statement to update volumes: ' .
$h_q->{'Statement'} .
' Result code: ' . ($h_db->errstr() ? $h_db->errstr() : "Ok"));
for $i (0 .. $#volumes) {
debug_out(20, "===\nVolume $volumes[$i]{'VolumeName'} from pool
$volumes[$i]{'Pool'}
Storage $volumes[$i]{'Storage'}
Slot $volumes[$i]{'Slot'}");
if ($j ne $volumes[$i]{'Storage'}) {
print $msgs{'newsto_pre'} . $volumes[$i]{'Storage'} .
$msgs{'newsto_post'};
$j = $volumes[$i]{'Storage'};
}
printf $msgs{'slot'} . '%4d - ' . $volumes[$i]{'VolumeName'} . ' (' .
$volumes[$i]{'Pool'} . ")\n", $volumes[$i]{'Slot'};
if ($dont_update) {
print $msgs{'noupdate'};
} else {
debug_out(20, $h_q->{'Statement'});
if ($h_q->execute(($volumes[$i]{'MediaId'}))) {
print 'Could not update Volume ' . $volumes[$i]{'VolumeName'} .
' because of ' . $h_db->errstr();
}
}
}
} else {
print $msgs{'norem'};
}
# now that we're done with the used media handling it's time
# to look for usable volumes...
undef(@volumes);
$catres = $h_db->selectall_hashref('SELECT MediaId,VolumeName,Pool.Name AS
Pool,' .
'FROM_UNIXTIME(UNIX_TIMESTAMP(Media.LastWritten)+' .
'Media.VolRetention) '.
'AS Expunges ' .
'FROM Media,Pool ' .
'WHERE ' .
'InChanger=0 AND ' .
'Pool.PoolId=Media.PoolId AND VolStatus IN '
.
'("Used","Append","Full","Recycle","Purged")
' .
'HAVING Expunges<="' .
$expunge_at . '";', 'VolumeName');
debug_out(20, 'SELECT MediaId,VolumeName,Pool.Name AS Pool,' .
'FROM_UNIXTIME(UNIX_TIMESTAMP(Media.LastWritten)+' .
'Media.VolRetention) '.
'AS Expunges ' .
'FROM Media,Pool ' .
'WHERE ' .
'Pool.PoolId=Media.PoolId AND VolStatus IN ' .
'("Used","Append","Full","Recycle","Purged") ' .
'HAVING Expunges<="' .
$expunge_at . '";');
if ($catres) {
debug_out(10, "\nMedia that can be loaded:");
my $a;
for $a (sort(keys %{$catres})) {
# all the media in one hash - see above...
debug_out(10, " Volume $a\n $catres->{$a}->{'Expunges'}\n
$catres->{$a}->{'Pool'}");
push @volumes, $catres->{$a};
}
} else {
debug_out(10, "No Media: DBI::errstr: " .
((DBI::errstr) ? DBI::errstr : '*none*'));
}
if ($#volumes >= 0) {
# we've got the volumes that can be loaded again.
# We sort them by expiration time.
@volumes = sort {
debug_out(30, "Comparing $a->{'VolumeName'} with $b->{'VolumeName'}");
Class::Date->new($a->{'Expunges'}) <=>
Class::Date->new($b->{'Expunges'})
} @volumes;
# and produce a nice report:
# Wee need only the first few volumes of each pool
my %pools;
for $i (0 .. $#volumes) {
debug_out(10, "===\nVolume $volumes[$i]{'VolumeName'} from pool
$volumes[$i]{'Pool'}
Expunges $volumes[$i]{'Expunges'}");
if (defined($pools{$volumes[$i]{'Pool'}})) {
$pools{$volumes[$i]{'Pool'}}++;
if (!defined($pools{$volumes[$i]{'Pool'} . '--CONT'})) {
$pools{$volumes[$i]{'Pool'} . '--CONT'} = ();
}
push(@{$pools{$volumes[$i]{'Pool'} . '--CONT'}},
$volumes[$i]{'VolumeName'});
} else {
$pools{$volumes[$i]{'Pool'}} = 1;
$pools{$volumes[$i]{'Pool'} . '--CONT'}->[0] =
($volumes[$i]{'VolumeName'});
}
}
if ($debug >= 10) {
for $a (sort(keys %pools)) {
if ($a !~ /.*--CONT$/) {
debug_out(10, "Pool $a - $pools{$a}\nContents:");
debug_out(10, @{$pools{$a . '--CONT'}});
}
}
}
if ($new_bypool) {
for $a (sort(keys %pools)) {
if ($a !~ /.*--CONT$/) {
print "\n" . $msgs{'load_from'} . $a . ":\n";
for($i=0; ($i < $new_vols) && ($i < $pools{$a}); $i++) {
print ' ' . $pools{$a . '--CONT'}[$i] . "\n";
}
if (0 == $i) {
print $msgs{'load_none'};
}
}
}
} else {
undef(@volumes);
for $a (sort(keys %pools)) {
if ($a !~ /.*--CONT$/) {
for($i=0; ($i < $new_vols) && ($i < $pools{$a}); $i++) {
push(@volumes, $pools{$a . '--CONT'}[$i]);
}
}
}
if ($#volumes) {
@volumes = sort(@volumes);
print "\n " . $msgs{'load_all'} . ' ' . join("\n ", @volumes) .
"\n";
} else {
print $msgs{'load_none'}
}
}
} else {
print $msgs{'noimp'};
}
sub do_mount {
# We read the varfile, and for each unique storage device we execute the
# update command and mount the device.
if ((-f $varfile) and (-r $varfile) and (-w $varfile)) {
debug_out(5, "Using $varfile for updating...");
if (open(VF, $varfile)) {
my %st;
while (<VF>) {
debug_out(10, "Got line: $_");
if (/^update .* storage=(.+)$/){
debug_out(15, "Match for storage=$1");
my $st = $1;
if (defined($st{$1})) {
$st{$1}++;
debug_out(20, "Storage $1 ignored - this is no.
$st{$1}");
} else {
debug_out(20, "Issuing $_...");
$st{$1} = 1;
my $po = get_prog_output($bconsole, $_);
debug_out(20, "Command got:\n$po");
debug_out(0, 'Problem updating:' .
"\n$po\n********\nTrying to continue.")
if ($po !~ /^330\d .*$/ms);
$po = get_prog_output($bconsole, "mount $st");
debug_out(15, "Mount command got:\n$po");
debug_out(0, 'Problem mounting:' .
"\n$po\n********\nTrying to continue.")
if ($po !~ /^(3905 .*|3001 Device .*)$/ms);
}
}
}
close(VF);
unlink($varfile);
} else {
debug_out(0, "$varfile could not be opened: $!");
}
} else {
debug_out(0, "$varfile doesn\'t exist, or is not read- and writeable.");
}
}
sub reorder {
# This function takes three arguments: A device name, a reference to a
# device-describing hash (see process_changer for details) and a
# reference to the global list is volumes to remove. It tries to
# shuffle volumes in the given storage so that they can be easily
# removed.
if ((!defined($_[0]) or (ref($_[1] ne 'HASH')) or (ref($_[2]) ne 'ARRAY')))
{
debug_out(0, 'Not called with a name, a hash and an array!');
return 1;
}
my $device = $_[0];
my %caps = %{$_[1]};
my $volumes = $_[2];
debug_out(45, "Shuffling in $device...");
my $out = get_prog_output($bconsole, ".storage");
debug_out(45, "Got storage defaults $out");
if ($out =~ /^1000 OK.*^$device$/ms) {
$out = get_prog_output($bconsole, '.defaults storage=' . $device);
if ($out =~ /^storage=$device.*device=(.*)(You have messages\.)?$/ms) {
my $subdev = $1;
$out = get_prog_output($bconsole, 'status storage=' . $device);
debug_out(50, "status storage got:\n$out");
if ($out =~ /^Autochanger \"$subdev\" with devices: *\n
*\"([^\n]+)\" \(/ms) {
$subdev = $1;
} else {
debug_out(0, "Couldn't find a drive for device $device /
$subdev.");
return;
}
debug_out(45, "Sub-device is $subdev");
$out =~ /^Running Jobs:(.*)\n====\n\nJobs.*/ms;
$out = $1;
debug_out(50, "Running jobs:\n$out");
if ($out !~ /^ +pool=.*device=\"\"$subdev\"/ms) {
debug_out(45, "$device ($subdev) is not busy.");
$out = get_prog_output($bconsole, 'unmount ' . $device);
store_remount($device, $caps{'scn'});
debug_out(45, "Unmounting:\n$out");
debug_out(0, "Problem unmounting $device ($subdev). trying to
continue...")
if ($out !~ /^3\d{3} Device \"$subdev\" .*unmounted.$/ms);
my $mtx_s = $caps{'mtx'} . ' -f ' . $caps{'ctr'} . ' status';
$out = get_prog_output($mtx_s);
debug_out(45, "First mtx status got:\n$out");
my $in_drive = undef;
my $slots_max = 0;
if ($out =~ /Storage Changer .*Drives, (\d+) Slots/ms) {
$slots_max = $1;
}
if ($out =~ /Data Transfer Element 0:Full \(Storage Element
(\d+) Loaded\)/ms) {
$in_drive = $1;
}
debug_out(45, "Slots: $slots_max Loaded: " .
(defined($in_drive) ? $in_drive : '*none*'));
$caps{'slo'} = $slots_max if ($caps{'slo'} > $slots_max);
$caps{'shi'} = $slots_max if ($caps{'shi'} > $slots_max);
$caps{'slo'} = 1 if ($caps{'slo'} < 1);
$caps{'shi'} = -1 if ($caps{'shi'} < 1);
if (0 < $slots_max) {
my @slots = ();
my %dummyvolume = ('VolumeName' => 'Dummy',
'Storage' => $device,
'Slot' => 999);
while ($out =~ m/Storage Element (\d+):(Full|Empty)/gms) {
debug_out(45, "Slot $1 is loaded");
my $vol = find_media($volumes, $device, $1);
if (defined($vol)) {
debug_out(45, "Found volume $vol->{'VolumeName'}");
if ($vol->{'Slot'} == $1) {
debug_out(50, "Volume $vol->{'VolumeName'} is
in slot $1");
$slots[$1] = $vol;
} else {
debug_out(0, "Slot $1 not correctly loaded. Not
sorting.");
return undef;
}
} elsif ('Empty' ne $2) {
$slots[$1] = \%dummyvolume;
} elsif ('Empty' eq $2) {
$slots[$1] = undef;
} else {
debug_out(0, "Seriously unexpected internal
problem: Slot $1, Status $2, Volume is $vol. Not re-ordering.");
return undef;
}
}
# array @slots is finally initialized with the volumes to re-order and dummies.
if ($debug >= 45) {
for my $s (1 .. $#slots) {
debug_out(45, "Slot $s: " .
(defined($slots[$s]->{'VolumeName'}) ?
$slots[$s]->{'VolumeName'} : '*empty*'));
}
}
my $keepslots = $slots_max - ($caps{'shi'}-$caps{'slo'}) -
1;
debug_out(45, "Slots left for storage: $keepslots");
my $emptyslots = 0;
foreach my $s (@slots[1 .. $slots_max]) {
$emptyslots++ if (!defined($s->{'VolumeName'}));
}
debug_out(45, "Slots empty: $emptyslots");
my $workslot = $emptyslots;
$workslot-- if (defined($in_drive));
$workslot++ if (1 == $caps{'trn'});
if ($workslot < 1) {
print $msgs{'slotsnoav'} . $device . "\n";
return undef;
}
debug_out(45, "Slots available for re-ordering: $workslot");
undef($workslot);
debug_out(45, "Limits: $caps{'shi'} $caps{'slo'}");
SRCH:
for(my $s = 1; $s<= $slots_max; $s++) {
debug_out(50, "Iter $s: " .
defined($slots[$s]->{'VolumeName'}) .
' Hi: ' . ($s > $caps{'shi'}) .
' Lo: ' . ($s < $caps{'slo'}));
if ((!defined($slots[$s]->{'VolumeName'})) and
(($s > $caps{'shi'}) or ($s < $caps{'slo'}))) {
$workslot = $s;
last SRCH;
}
}
debug_out(50, 'Hmm. Workslot not set.') if
(!defined($workslot));
$workslot = 1 if ((!defined($workslot)) and ($caps{'slo'} >
1));
$workslot = $slots_max if ((!defined($workslot)) and
($caps{'shi'} < $slots_max));
if (!defined($workslot)) {
debug_out(0, 'Couldn\'t find a slot for work. This
should never happen.');
return undef;
}
debug_out(45, "Working slot is $workslot");
my @moves = ();
my @temp = @slots;
for(my $s = $caps{'slo'}; $s <= $caps{'shi'}; $s++) {
if (!defined($temp[$s]->{'VolumeName'}) or
(('Dummy' eq $temp[$s]->{'VolumeName'}) and
(999 == $temp[$s]->{'Slot'}))) {
debug_out(45, "Volume $s should be replaced with" .
' a volume to export.');
for (my $t = 1; $t <= $slots_max; $t++) {
if ((($t < $caps{'slo'}) or ($t >
$caps{'shi'})) and ($t != $workslot) and (defined($temp[$t]->{'VolumeName'})
and (('Dummy' ne $temp[$t]->{'VolumeName'}) and (999 != $temp[$t]->{'Slot'}))))
{
debug_out(45, "Move from slot $t to slot
$s");
my $swp = $temp[$t];
$temp[$t] = $temp[$s];
$temp[$s] = $swp;
my %onemove = ('From'=>$t, 'To'=>$s);
push @moves, \%onemove;
last;
}
}
}
}
if ($debug >= 40) {
debug_out(40, 'Effective Tape moves:');
for my $a (@moves) {
debug_out(40, "From: $a->{'From'} To: $a->{'To'}");
}
}
# Finally, we have a list of all the tape moves we've got to do in effect.
# With a stupid autochanger without transfer abilities, we handle this as
follows:
# First, load the To: slot and unload it to the workslot.
# Then, the From: slot is loaded and unloaded to the To: slot.
# Finally, for bookkeeping, workslot becomes the From: slot.
# Loop until done
#
# With autoloaders that can transfer, we do it a little different.
# Instead of loading / unloading, we use the transfer command
# Whenever a To: slot is occupied, the tape is moved to the workslot and
# the From: slot becomes the workslot.
# If initially loaded, the drive is unloaded to the workslot.
my $po = '';
my $marker;
if (($caps{'trn'}) and
(defined($slots[$workslot]->{'VolumeName'}))) {
debug_out(40, "Will use transfers but have to free slot
$workslot first...");
$po = get_program_output($caps{'mtx'} . " -f
$caps{'ctr'} load $workslot");
$marker = 1;
$in_drive = $workslot;
$slots[$workslot] = undef;
if (length($po) > 0) {
debug_out(0, "Error: \n$po\n********\nCan\'t
continue reordering");
return undef;
}
} elsif (!$caps{'trn'} and
(defined($slots[$workslot]->{'VolumeName'}))) {
debug_out(0, "Can\'t use transfers but workslot is not
free. No reordering possible.");
return undef;
} elsif ((!$caps{'trn'}) and ($in_drive)) {
my $workslot2;
SRCH:
for(my $s = 1; $s<= $slots_max; $s++) {
debug_out(50, "Iter $s: " .
defined($slots[$s]->{'VolumeName'}) .
' Hi: ' . ($s > $caps{'shi'}) .
' Lo: ' . ($s < $caps{'slo'}));
if ((!defined($slots[$s]->{'VolumeName'})) and
(($s > $caps{'shi'}) or ($s < $caps{'slo'})) and
($s != $workslot)) {
$workslot2 = $s;
last SRCH;
}
}
if (!defined($workslot2)) {
debug_out(0, "No slot to empty drive. No
re-ordering.");
return undef;
}
if ($caps{'off'}) {
$po = get_program_output("$caps{'mt'} -f
$caps{'dev'} rewoffl");
if (length($po) > 0) {
debug_out(0, "Problem offlining drive:\n$po");
return undef;
}
}
my $po = get_prog_output("$caps{'mtx'} -f $caps{'ctr'}
unload $workslot2");
if (length($po) > 0) {
debug_out(0, "Problem unloading drive:\n$po");
return undef;
}
debug_out(50, "Drive free and workslot available. Ok.");
}
for my $m (@moves) {
$po = '';
if ($caps{'trn'}) {
if (defined($slots[$m->{'To'}]->{'VolumeName'})) {
debug_out(50, "$caps{'mtx'} -f $caps{'ctr'}
transfer $m->{'To'} $workslot");
$po = get_prog_output("$caps{'mtx'} -f
$caps{'ctr'} transfer $m->{'To'} $workslot");
$workslot = $m->{'From'};
}
debug_out(50, "$caps{'mtx'} -f $caps{'ctr'}
transfer $m->{'From'} $m->{'To'}");
$po = get_prog_output("$caps{'mtx'} -f $caps{'ctr'}
transfer $m->{'From'} $m->{'To'}");
if (length($po) > 0) {
debug_out(0, "Problem moving tape from slot
$m->{'From'} to slot $m->{'To'}:\n$po\n********\nCan\'t continue.");
return undef;
}
my $swap = $slots[$m->{'From'}];
my $gv = find_media($volumes, $device, $m->{'To'});
$gv->{'Slot'} = $workslot if ($gv);
$gv = find_media($volumes, $device, $m->{'From'});
$gv->{'Slot'} = $m->{'To'} if ($gv);
$slots[$m->{'From'}] = $slots[$m->{'To'}];
$slots[$m->{'To'}] = $swap;
} else {
# here we handle autoloaders without tranfer capability
debug_out(45, "Transfers in non-tranfering devices
not supported.");
if (defined($slots[$m->{'To'}]->{'VolumeName'})) {
$po = get_prog_output("$caps{'mtx'} -f
$caps{'ctr'} load $m->{'To'}");
if (length($po) > 0) {
debug_out(0, "Problem loading from slot
$m->{'To'}:\n$po\n********\nCan\'t continue reordering.");
return undef;
}
if ($caps{'off'}) {
$po = get_prog_output("$caps{'mt'} -f
$caps{'dev'} rewoffl");
if (length($po) > 0) {
debug_out(0, "Problem offlining $device
- trying to continue");
}
}
$po = get_prog_output("$caps{'mtx'} -f
$caps{'ctr'} unload $workslot");
if ($po !~ /^Unloading Data Transfer Element
into Storage Element ($workslot)...done$/) {
debug_out(0, "Problem unloading to slot
$workslot:\n$po\n********\nCan\'t continue reordering.");
return undef;
}
$workslot = $m->{'To'}
}
$po = get_prog_output("$caps{'mtx'} -f $caps{'ctr'}
load $m->{'From'}");
if (length($po) > 0) {
debug_out(0, "Problem loading from slot
$m->{'From'}:\n$po\n********\nCan\'t continue reordering.");
return undef;
}
$po = get_prog_output("$caps{'mtx'} -f $caps{'ctr'}
unload $m->{'To'}");
if ($po !~ /^Unloading Data Transfer Element into
Storage Element ($m->{'To'})...done$/) {
debug_out(0, "Problem unloading to slot
$m->{'To'}:\n$po\n********\nCan\'t continue reordering.");
return undef;
}
my $swap = $slots[$m->{'From'}];
my $gv = find_media($volumes, $device, $m->{'To'});
$gv->{'Slot'} = $workslot if ($gv);
$gv = find_media($volumes, $device, $m->{'From'});
$gv->{'Slot'} = $m->{'To'} if ($gv);
$slots[$m->{'From'}] = $slots[$m->{'To'}];
$slots[$m->{'To'}] = $swap;
}
}
if ($caps{'trn'} and $marker) {
if ($caps{'off'}) {
$po = get_program_output("$caps{'mt'} -f
$caps{'dev'} rewoffl");
debug_out(0, "Problem offlining drive:\n$po") if
(length($po) > 0);
}
$po = get_program_output("$caps{'mtx'} -f $caps{'ctr'}
unload $workslot");
debug_out(0, "Problem unloading drive:\n$po") if
(length($po) > 0);
}
if ($caps{'eje'}) {
$po = get_prog_output("$caps{'mtx'} -f $caps{'ctr'}
eject");
debug_out(0, "Problem unloading drive:\n$po") if
(length($po) > 0);
}
} else {
print $msgs{'slotsprob'} .
"Max: $slots_max Lo: $caps{'slo'} Hi: $caps{'shi'}\n";
}
} else {
print $msgs{'devbusy'} . "$device ($subdev)\n";
}
} else {
print $msgs{'nosuchdev'} . $device . " (sub)\n";
}
} else {
print $msgs{'nosuchdev'} . $device . "\n";
}
}
sub find_media {
# this function takes three arguments: a reference to the volumes
# array, a storage name and a slot. It returns a reference to the
# volume in the volumes array, or undef if the volume specified wasn't
# found.
if ('ARRAY' eq ref($_[0])) {
if ((defined($_[1])) and (defined($_[2]))) {
my $vols = $_[0];
my $stor = $_[1];
my $slot = $_[2];
debug_out(50, "Search for slot $slot in $stor");
foreach my $a (@$vols) {
debug_out(50, "Check $a->{'VolumeName'}");
return $a if (($slot == $a->{'Slot'}) and ($stor eq
$a->{'Storage'}));
}
return undef;
} else {
debug_out(0, 'Problem in find_media: storage or slot not
specified.');
return undef;
}
} else {
debug_out(0, 'Problem in find_media: volumes-array not supplied');
return undef;
}
}
sub get_prog_output {
if (defined($_[0]) and (defined($_[1]))) {
my $prog = shift;
my $params = join("\n", @_) . "\n";
debug_out(45, "Calling $prog and feed it $params");
my ($rdfh, $wrfh);
my $pid = open2($rdfh, $wrfh, $prog);
my $ret = '';
if ($pid) {
print $wrfh $params;
close($wrfh);
while (<$rdfh>) {
$ret .= $_;
if (length($ret) > 1024*1024) { # 2 MB of output should
definitely be enough...
debug_out(0, "Output from $prog got too long. Truncated.");
return $ret;
}
}
debug_out(45, "Read $ret");
return $ret;
} else {
debug_out(45, "Can't start $prog bcause of $!");
return '';
}
}
elsif (defined($_[0])) {
my $f = $_[0] . ' 2>&1 |';
debug_out(45, "File to open: $f");
my $ret = '';
if (open(my $p, $f)) {
while (<$p>) {
$ret .= $_;
if (length($ret) > 1024*1024) { # 2 MB of output should
definitely be enough...
close($p);
debug_out(0, "Output from $_[0] got too long. Truncated.");
return $ret;
}
}
close($p);
return $ret;
} else {
debug_out(0, "Error opening $f: $!");
return "";
}
} else {
debug_out(0, "get_prog_output called without program. This doesn't
work...");
return '';
}
}
sub process_changer {
# this function is called for the option --changer, once per occurence.
# it should return with die upon error because Getopt::Long handles this
# correctly as an option processing error.
# This is used as follows:
# --changer
HPDAT=/usr/bin/mtx|/usr/bin/mt|/dev/nst0|/dev/sg0|offl|4-6|transfer|eject
# This is called with three arguments:
# literal "changer"
# changer name, like "HPDAT"
# changer options, like
/usr/bin/mtx|/dev/nst0|/dev/sg0|offl|4-6|scan|transfer|eject
debug_out(45, "process_changer called with @_");
if ('changer' eq $_[0]) {
my $device = $_[1];
if ('help' eq $device) {
print($msgs{'changerhelp'});
exit 1;
}
my ($mtx, $mt, $dev, $ctr, $offline, $slots, $scan, $transfer, $eject) =
split(/\|/, $_[2]);
# the following line produces warnings when not all options are given.
# debug_out(55, "Got $mtx $dev $ctr $offline $slots $scan $transfer
$eject");
my $err = $msgs{'changer_err'};
my $fault = 0;
my ($slotlo, $slothi) = (0, 0);
if (defined($eject)) {
if ($eject =~ /^(1|yes|eject)$/i) {
$eject = 1;
} elsif ($eject =~ /^(0|no|noeject|keeploaded|)$/i) {
$eject = 0;
} else {
$err .= ' eject';
$fault = 1;
$eject = 0;
}
} else {
$eject = 0;
}
if (defined($transfer)) {
if ($transfer =~ /^(0|no|load|)$/i) {
$transfer = 0;
} elsif ($transfer =~ /^(1|yes|transfer)$/i) {
$transfer = 1;
} else {
$err .= ' transfer';
$fault = 1;
$transfer = 0;
}
} else {
$transfer = 0;
}
if (defined($scan)) {
if ($scan =~ /^(0|no|noscan|)$/i) {
$scan = 0;
} elsif ($scan =~ /^(1|yes|scan)$/i) {
$scan = 1;
} else {
$err .= ' scan';
$fault = 1;
$scan = 0;
}
} else {
$scan = 0;
}
if (defined($slots) && ($slots =~ /^(\d)+-(\d)+$/)) {
$slotlo = $1;
$slothi = $2;
} elsif ((defined($slots)) && ('' ne $slots)) {
$err .= ' slots';
$fault = 1;
}
if (defined($offline)) {
if ($offline =~ /^(1|yes|offline|)$/i) {
$offline = 1;
} elsif ($offline =~ /^(0|no|nooffline)$/i) {
$offline = 0;
} else {
$err .= ' offline';
$fault = 1;
}
} else {
$offline = 1;
}
if (!defined($ctr)) {
$err .= ' control-device';
$fault = 1;
}
if (!defined($dev)) {
$dev = '';
}
if (!defined($mt)) {
$mtx = '/usr/bin/mt';
}
if (!defined($mtx)) {
$mtx = '/usr/bin/mtx';
}
debug_out(45, "Got $mtx $mt $dev $ctr $offline ${slotlo}-${slothi}
$scan $transfer $eject");
if (!defined($device) or ('' eq $device)) {
$fault = 1;
$err .= ' no-device';
} else {
$err .= '. ' . $device;
}
die($err . $msgs{'changer_ign'}) if $fault;
my %this_changer;
$this_changer{'mtx'} = $mtx;
$this_changer{'mt'} = $mt;
$this_changer{'dev'} = $dev;
$this_changer{'ctr'} = $ctr;
$this_changer{'off'} = $offline;
$this_changer{'slo'} = $slotlo;
$this_changer{'shi'} = $slothi;
$this_changer{'scn'} = $scan;
$this_changer{'trn'} = $transfer;
$this_changer{'eje'} = $eject;
$all_devices{$device} = \%this_changer;
} else {
die("Internal Error. This is really bad but not dramatic \x{263a}.");
}
}
sub store_remount {
if ((defined($_[0])) and (defined($_[1]))) {
if (open(FH, ">>$varfile")) {
print FH 'update slots ' . ($_[1] ? 'scan ' : '') . 'storage=' .
$_[0] . "\n";
close(FH);
} else {
debug_out(0, "Can\'t open $varfile for appending. Error $!. Trying
to continue.
Please make sure the drives are remounted after tape changing!");
}
} else {
debug_out(0, "store_remount called with insufficient parameters.");
}
}
sub debug_out {
if ($debug >= shift) {
print STDERR "@_\n";
}
}
sub show_help {
print $msgs{'help'};
}
__END__
=head1 NAME
bac_vol_mgmt.pl - a script to simplify tape volume management with Bacula.
=head1 SYNTAX
B<bac_vol_mgmt.pl> B<--help|-h>
B<bac_vol_mgmt.pl> [B<--host>|B<-H> I<Hostname>] [B<--user>|B<-U> I<Username>]
[B<--database>|B<-D> I<Database>] [B<--database-type|-T> I<Database Type>]
[B<--password>|B<-P> I<Password>]
[B<--debug> I<Level>]
The long options can be abbreviated as long as they remain
unique. Short options (and values) can be grouped; for more
information, see B<perldoc Getopt::Long>.
=head1 DESCRIPTION
B<bac_vol_mgmt.pl> accesses the catalog used by the backup program
Bacula to do the following tasks:
=over 4
=item *
Determine which volumes are needed for a full recovery of all
known client / fileset combinations.
=item *
Mark these volumes as I<Used>.
=item *
Unmount the storage devices so that Bacula will not use them.
=item *
If possible and if the necessary options are present on the
command line, order the volumes in the autochanger devices in a
way that they can be easily removed.
=item *
If possible, eject the tape magazines or tapes from autochanger
devices.
=item *
Print a report about which volumes from which devices should be
removed for storage.
=item *
Print a report about which volumes are available for later use,
i.e. are pruned or will be pruned when needed.
=back
=head1 USAGE
For general instructions, please use the --help option to the program.
Usually, this program will be run in two steps. The first one does most of
the work, and when it's done you should be able to easily remove the tapes
you want to store.
Most of the options are intended for this step. A typical command line might
look
like this:
B<< C<<
bac_vol_mgmt.pl -P somepw -p --changer HPDAT="ssh goblin mtx|ssh goblin
mt|/dev/st1|/dev/sg5|nooffline|4-6|scan||yes" --changer DLT="ssh goblin mtx|ssh
goblin mt|/dev/st2|/dev/sg7|offline|4-5|scan|transfer|no" >> >>
Later, you only need the B<-m> switch to trigger slot scanning and mounting for
all
devices the first run unmounted and possibly reordered, or, in other words, all
storage devices where volumes might have changed with assistence of this script.
The latter step of the process can be done by a cron script or interactively.
For
example, you could set up an ssh key which only allows this command, and install
putty on a windows client to immediately log in using that key.
=head1 SECURITY CONSIDERATIONS
I strongly recommend you review the code of this script before deploying it in
a production environment.
Ensure proper operation in a test environment before using this script in your
production setup.
This script needs access to your Bacula catalog database. It even has to write
data into it (unless this is turned of using the B<-no-updates|-n> switch). In
an insecure environment - and which one isn't? - I recommend modifying the
script
to contain the user name and password for catalog access in the source code, or
have it read from a configuration file (not yet implemented). Otherwise, be
aware that information passed on the command line can be visible to any user
on the system.
That said, this script should not be able to damage any catalog or even backup
data.
=head1 CUSTOMIZING
As this script is supplied in the form of source code, you can easily modify it
so that you don't have to use the options all the time.
Unfortunately, the variables you might want to change are not all in one place.
Some of them are quite near the top of the file, where the global variables are
declared.
The changer setup is all done in the function C<process_changer> at the end
of the program code, this is where you'd change the defaults for the changer
setup.
Preferrably, though, would be to simply insert the necessary data into the
global changer variable. Just call the script with all the necessary parameters
and the debug level set sufficiently high and you see all the values as they
are assigned to the all_devices variable. That variable is a hash containing a
device name and a reference to a hash of variable-value-pairs. You could set
up something like
B<< C<<
my %all_devices = ('HPDAT' => \('ctr' => '/dev/sg5',
'dev' => '/dev/st1',
'eje' => 1,
'mt' => '/bin/mt',
'mtx' => '/bin/mtx',
'off' => 0,
'scn' => 1,
'shi' => 6, 'slo' => 4,
'trn' => 0 ))
>> >>
... or similar - I didn't test it.
=head1 AUTHOR and LICENSE
Arno Lehmann, IT-Service Lehmann, <[EMAIL PROTECTED]>
This program is copyrighted (German Urheberrecht) but everybody is
allowed to use, modify and distribute this program under the
following conditions:
=over 2
=item *
This license and the original copyright note must not be changed. If
you modify the script you will hold (the local corresponding
rights to) the copyright for your modified version, too.
=item *
The terms and the idea of the GPL version 2 apply. If you are unsure, ask the
copyright holder if your planned usage is ok.
=item *
No warranties, no promises. You are all on your own. This program
needs access to your Bacula catalog. If you don't like that idea,
don't use it or check the sourcecode!
=back
Although I give no warranties, in case of problems you can contact me.
I will help as good as possible.
Bacula is a Trademark and Copyright of Kern Sibbald. See L<www.bacula.org>
