Hello again Arno and friends ...
On 11/7/07, Arno Lehmann <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> 07.11.2007 19:12,, Hydro Meteor wrote::
> >
> > On Nov 7, 2007 1:26 AM, Arno Lehmann <[EMAIL PROTECTED]
> > <mailto:[EMAIL PROTECTED] >> wrote:
> >
> > Hello,
> >
> > 07.11.2007 12:17,, Hydro Meteor wrote::
> > > Hi all,
> > >
> > > I just recently set up a migration resources in my Director
> > > configuration file to migrate from File (hard drive) to DVD. The
> > simple
> > > test I tried worked fine (the DVD+RW was written to just
> perfectly).
> >
> > Good to hear - I'm planning a similar (test) setup here :-)
> >
> >
> > Credit goes to Richard Mortimer who had suggested this strategy in an
> > off-list email!
> >
> >
> >
> > > The Bacula User's Guide states:
> > >
> > > As part of this process, *the File catalog records associated
>
> > with
> > > the first backup job are purged*. In other words, Migration
> moves
> > > Bacula Job data from one Volume to another by reading the Job
> > data
> > > from the Volume it is stored on, writing it to a different
> > Volume in
> > > a different Pool, and then purging the database records for
> the
> > > first Job.
> > >
> > >
> > > Strangely, my original backup job (of the File on the hard drive)
> was
> > > not purged. Is not the purge supposed to take place once the
> > migration
> > > has succeeded (e.g., after the DVD was successfully written to)?
> >
> > Note the difference between File records and Job records - the File
> > records should be purged, the Job records remain in the catalog. At
> > least that's how I read the above manual paragraph.
> >
> > > I
> > > wonder why mine simple test situation indicated no purging took
> > place?
> >
> > Have you actually checked for the File records, or only the Job
> > records?
> >
> >
> > Hi Arno,
> >
> > I use the very nice Bacuview program to display the contents of the
> > Catalog. It shows all jobs saved in the Catalog and indeed you are
> > right, the Job records are not purged. But, interestingly enough,
> > Bacuview also gives me a view of the "media" records which has a table
> > that looks like this:
>
> Well, that table still doesn't hold any information about the file
> records :-)
Great point, I missed that. Thanks for pointing it out.
>
> > * Name Slot Status Jobs Bytes Expires Retention
> > Pool Name Pool Type Media *
> >
> > So, the file that was original saved (which should be purged) has the
> > name of "filetest" and it still exists in the Catalog (not purged), as
> in:
> >
> > Name: filetest1
> > Slot: -
> > Status: Used
> > Jobs: 1
> > Bytes: 54.3 MB
> > Expires: 2008-11-06
> > Retention: 365 days
> > Pool Name: thefullPool
> > Pool Type: Backup
> > Media: HardDisc
>
> This is a volume file. You need to check what the catalog knows about
> its contents. I don't use bacuview, but in bconsole, there is a query
> for this.
Ah yes, I like Bacuview but was becoming a little too reliant on it and I
can understand why Bacuview doesn't provide a view of Volume File contents
(it could be tens of thousands of lines long depending on how deep and wide
the file system resource tree is that was being backed up on the Client) --
probably too much to view in a web browser.
I had not previously used the *query* Bacula Console command, thank you for
calling my attention to it.
I suspect you'll find the original job is still known to be stored on
> this volume, the job will be marked as being migrated (status "M").
Yep, you are right. I ran another migration test and afterwards, "M" was the
value of the JobID's type, as in:
*list jobs
| jobid | name | starttime |
type | level | jobfiles | jobbytes | jobstatus |
*| 18 | Backup LeClient-fd | 2007-11-08 00:40:59 | M | F |
1,665 | 56,538,141 | T |*
Note: the number of job bytes in this example (where JobID was written to
another hard drive, not a tape or another other medium) is 56,538141 yet
after I successfully ran a migration job, the number of jobbytes for the
JobID 20 is slightly more than that of JobID 18:
| 20 | Backup LeClient-fd | 2007-11-08 00:40:59 | B | F |
*1,665* | *56,827,921* | T |
| 19 | migrate-volume | 2007-11-08 00:52:30 | g | F
| 0 | 0 | T |
Is this because JobID 20 was written to DVD+RW and there are additional
bytes of information when writing to DVD+RW (for example the DVD is split
into part files and so maybe there are bytes required to keep track of the
parts or other DVD-specific structures)? A difference from JobID 18 to JobID
20 of 289,780 bytes (283KB) is not necessarily insignificant.
Arno, another thing that is interesting is that JobID 20 must have been
spawned by the migration process (JobID 19) because manually, I only
executed JobID 19 (the migration) and not directly JobID 20. I guess this
makes sense in that JobID 19 (the migration) is sort of an intermediary
process and that a separate process (JobID 20) is necessary to write the
data to the target medium (in this case DVD+RW).
You will not find files associated with that original job in the
> catalog anymore - these have been purged.
Yes, conclusively, you are right. After JobID 19 (and its spawned child?
JobID 20) were finished and the messages in the console said that the
process was successful, I then attempted gain to query the list of files
(the contents of the Volume) for JobID 18. For example:
* query
Choose a query (1-16): 12
Enter JobId: 18
-->
*No results to list.*
Prior to running the migration job (JobID 19), the query choosing item 12
("List Files for a selected JobId") indeed show the file contents ( a very
long list of 1,665 files). And then when I tried the same query ("List Files
for a Selected JobID") for JobID 20, no problem, the list of 1,665 files. So
this example was very useful to run through to help better understand
especially a new person to migrations (and it helps to understand more about
what is inside the Catalog -- all those nice gems of information!).
By the way, what does the JobType of "g" mean for JobID 19? g = "good"? This
information is probably buried in the wonderful User's Guide somewhere and
my eyes missed it.
Only the files, neither the
> Job record itself nor the JobMedia records.
>
> > Note that the Jobs field is not the same as the JobID when presented by
> > Bacuview (there are separate views for the Jobs IDs).
> >
> > Maybe the file (in the example above named filetest1) has not been
> > purged because its retention is set for one year and therefore in a
> > situation whereby teh retention is set to some value, it overrides the
> > ability of the migration job to purge it from the Catalog?
>
> How all the retention periods work together in Bacula is described in
> the manual. I understand that a volume that has no more jobs on it,
> even if in its retention period, should get recycled. Most important
> is that recycling and purging only happen when actually needed.
Excellent reminder and reinforcing of the Law of Bacula (for those of us who
have to be practicing lawyers!).
> Also, the actual file on the hard drive where it is saved (the archive)
> > also still exists and was not deleted (I presume then that migration
> > merely copies the archive but does not delete the original archive
> > itself). This isn't a big deal because the original archive copy can be
> > deleted later.
>
> The volume file exists and is neither deleted and truncated... yes.
> That's how Bacula works.
Thank you for overarching confirmation.
>
> > By the way, the file "filetest1" was successfully migrated to "dvdtest1"
> > and an additional record shows up for the same table courtesy of
> Bacuview:
>
> There is not way to migrate files using Bacula.
>
> You can only migrate Jobs (as far as I know). You can't even migrate
> volumes.
>
> I suspect we've got a problem with the terminology, too :-)
Maybe a terminology problem, maybe not. There is a lot of information to
digest about Bacula and so the terms can sometimes be a little confusing to
a newcomer.
> Name: dvdtest1
> > Slot: -
> > Status: Append
> > Jobs: 1
> > Bytes: 54.3 MB
> > Expires: 2008-11-06
> > Retention: 365 days
> > Pool Name: DVDFullPool
> > Pool Type: Backup
> > Media: DVD
>
> This is volume information.
Right on.
>
> > If I understand what the User's Guide is saying, in theory then, using
> > the above examples, "filetest1" meta data should be entirely purged from
>
> > the Catalog as it is in essence deprecated in favor of "dvdtest1", true?
>
> No. Only the file entries in the catalog regarding the migrated jobs
> file be purged. Neither the volume information nor the job
> information, and also not (I think) the JobMedia records.
Confirmed (per example provided above and my testing of several migrations
today although I have only tested the migration of Full backups not
Incremental or Differential yet).
These are quite different things.
>
> > What if there is a bug or some problem in getting the Catalog to purge
> > the deprecated file meta data (let's say for example that perhaps file
> > retention length does not in fact override migration's ability to purge
> > and there is some other unknown reason as to why it is not purged even
> > though the User's Guide says it should be)? In such a case the purging
> > could and should be done manually true? For example, the original file
> > on the hard drive and the clone of the file on the DVD could have their
> > SHA1 checksums computed and compared. If the checksums match, then it
> > would be safe to both delete the original file on the hard drive and
> > delete its catalog meta data ( e.g., on the Bacula Console run a delete
> > --> Volume on the MediaID or the Volume name of the deprecated original
> > volume)? I suppose this is somewhat of an obvious question but migration
> > seems to be an interesting little animal with some new challenges.
>
> I think some of your worries come from not exactly understanding what
> information to expect in the catalog.
For sure. Thanks for allaying those worries -- I wasn't aware of how helpful
the console query command could be.
Volume information stays in the catalog as long as you don't delete
> the volume - not the volume file, the DVD, or the tape, but from the
> catalog: 'delete volume=xxx'.
>
> Job records are removed when pruning happens. They are updated on
> Migration.
>
> JobMedia records are automatically handled. In essence, they are
> removed when their corresponding jobs are removed.
>
> File records are removed when a job's file entries are purged. This
> can be triggered by several events: The Job being purged, the volume
> being purged, or the jobs file retention period being over during a
> pruning run.
>
> If there are bugs in Bacula, that might break, true. In such a case,
> you wouldn't trust manual workarounds, because you wouldn't believe
> that you can do manually what Bacula should do automatically...
>
> If you're worried about disk space consumption of your volume files on
> disk, just make sure they are recycled in time. And limit their pools
> size.
Ah, yes I forgot about the pool size limit option. Good call!
As far as I know, migration does not leave volumes unusable, and it
> does not unnecessarily clutter your catalog with data.
Migrations are quite a nice option to have. Thank you immensely for helping
me and hopefully others on the mailing list to understand!
Cheers!
-H
Arno
>
> > Cheers,
> >
> > -H
> >
> >
> >
> > Arno
> >
> > > Thanks for any suggestions,
> > >
> > > -H
> > >
> > >
> > >
> >
> ------------------------------------------------------------------------
> > >
> > >
> >
> -------------------------------------------------------------------------
> > > This SF.net email is sponsored by: Splunk Inc.
> > > Still grepping through log files to find problems? Stop.
> > > Now Search log events and configuration files using AJAX and a
> > browser.
> > > Download your FREE copy of Splunk now >> http://get.splunk.com/
> > >
> > >
> > >
> >
> ------------------------------------------------------------------------
> > >
> > > _______________________________________________
> > > Bacula-users mailing list
> > > [EMAIL PROTECTED]<Bacula-users@lists.sourceforge.net>.net
> <Bacula-users@lists.sourceforge.net>
> > <mailto:[EMAIL PROTECTED]<Bacula-users@lists.sourceforge.net>
> .net <Bacula-users@lists.sourceforge.net>>
>
> > >
> > https://lists.sourceforge.net<https://lists.sourceforge.net/lists/listinfo/bacula-users>/lists/listinfo/bacula-users
> <https://lists.sourceforge.net/lists/listinfo/bacula-users>
> >
> > --
> > Arno Lehmann
> > IT-Service Lehmann
>
> > www.its-lehmann.de <http://www.its-lehmann.de >
> >
> >
> -------------------------------------------------------------------------
> >
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems? Stop.
> > Now Search log events and configuration files using AJAX and a
> browser.
> > Download your FREE copy of Splunk now >> http://get.splunk.com/
> > _______________________________________________
> > Bacula-users mailing list
> > [EMAIL PROTECTED] <Bacula-users@lists.sourceforge.net>
> .net <Bacula-users@lists.sourceforge.net>
>
> > <mailto: [EMAIL PROTECTED]<Bacula-users@lists.sourceforge.net>
> .net <Bacula-users@lists.sourceforge.net>>
>
> >
> > https://lists.sourceforge.net<https://lists.sourceforge.net/lists/listinfo/bacula-users>
> /lists/listinfo/bacula-users<https://lists.sourceforge.net/lists/listinfo/bacula-users>
> >
> >
> >
> > ------------------------------------------------------------------------
>
> >
> >
> -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems? Stop.
> > Now Search log events and configuration files using AJAX and a browser.
> > Download your FREE copy of Splunk now >> http://get.splunk.com/
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Bacula-users mailing list
> > [EMAIL PROTECTED]
> > <Bacula-users@lists.sourceforge.net>.net<Bacula-users@lists.sourceforge.net>
> > https://lists.sourceforge.net<https://lists.sourceforge.net/lists/listinfo/bacula-users>
> /lists/listinfo/bacula-users<https://lists.sourceforge.net/lists/listinfo/bacula-users>
>
> --
> Arno Lehmann
> IT-Service Lehmann
> www.its-lehmann.de
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Bacula-users mailing list
> [EMAIL PROTECTED] <Bacula-users@lists.sourceforge.net>.net
> <Bacula-users@lists.sourceforge.net>
> https://lists.sourceforge.net<https://lists.sourceforge.net/lists/listinfo/bacula-users>
> /lists/listinfo/bacula-users<https://lists.sourceforge.net/lists/listinfo/bacula-users>
>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
