Returning to the original thread...
Just to make sure I'm being clear - my FileSet specification is:
FileSet {
Name = "u2LgFileList"
Include {
Options {
signature = MD5
}
File = </local/etc/u2LgFileList.list
}
}
The file /local/etc/u2LgFileList.list has 29K+ entries in it.
Note that this is /not/ an exclude list - it's explicitly listing the
files to be backed up.
The FD takes about 40 minutes to read in the file list.
The SD times out in 30 minutes waiting for the FD.
>From my reading of the manual there are directives that set the time
that the Director will wait for an FD to respond "FD Connect Timeout"
and the time that the FD will wait for the SD to respond "SD Connect
Timeout" as well as the "Heartbeat Interval" that will keep the
connection open during long backups.
I've not found a directive to modify the length of time that the SD will
wait for the FD to begin transferring data.
This is the error message from the failed backup message. Note that the
authorization rejection is /not/ the problem - a test backup that
succeeded was used to verify proper authorization and communication
between FD and SD.
29-Jan 16:33 athos-dir: Start Backup JobId 112,
Job=u2FullBackupJob.2007-01-29_16.33.15
29-Jan 17:21 u2-fd: u2FullBackupJob.2007-01-29_16.33.15 Fatal error:
Authorization key rejected by Storage daemon.
Please see http://www.bacula.org/rel-manual/faq.html#AuthorizationErrors
for help.
29-Jan 17:21 u2-fd: u2FullBackupJob.2007-01-29_16.33.15 Fatal error:
Failed to authenticate Storage daemon.
29-Jan 17:21 athos-dir: u2FullBackupJob.2007-01-29_16.33.15 Fatal error:
Socket error on Storage command: ERR=No data available 29-Jan 17:21
athos-dir: u2FullBackupJob.2007-01-29_16.33.15 Error: Bacula 1.39.22
(09Sep06): 29-Jan-2007 17:21:22
I think that the timeout is being specified in this code from
stored/job.c
gettimeofday(&tv, &tz);
timeout.tv_nsec = tv.tv_usec * 1000;
timeout.tv_sec = tv.tv_sec + 30 * 60; /* wait 30 minutes */
Dmsg1(100, "%s waiting on FD to contact SD\n", jcr->Job);
/*
* Wait for the File daemon to contact us to start the Job,
* when he does, we will be released, unless the 30 minutes
* expires.
*/
P(mutex);
for ( ;!job_canceled(jcr); ) {
errstat = pthread_cond_timedwait(&jcr->job_start_wait, &mutex,
&timeout);
if (errstat == 0 || errstat == ETIMEDOUT) {
break;
}
}
V(mutex);
----
Alan Davis
Senior Architect
Ruckus Network, Inc.
703.464.6578 (o)
410.365.7175 (m)
[EMAIL PROTECTED]
alancdavis AIM
> -----Original Message-----
> From: Kern Sibbald [mailto:[EMAIL PROTECTED]
> Sent: Monday, January 29, 2007 4:06 PM
> To: bacula-users@lists.sourceforge.net
> Cc: Alan Davis
> Subject: Re: [Bacula-users] Experience with extremely large fileset
> include lists?
>
> Hello,
>
> On Monday 29 January 2007 21:19, Alan Davis wrote:
> > Kern,
> >
> > Thanks for the fast response. To clarify a bit - the file list that
I
> > would be using would be individual files, not directories. There
would
> > be no exclude list as only the files that I need backed up would be
> > listed.
>
> Yes, my answer was based on that assumption.
>
> >
> > I have about 30TB of data files spread over several hundred
directories.
> > A true incremental backup will spend large amounts of time
determining
> > what files have been changed or added. The information about the
> > modified or new files is stored in a db as a side-effect of
processing
> > the files for release to production so building a file list is
trivial.
> > The only problem would be the FD's capability of handling a file
list of
> > 10K+ entries.
>
> All I can say is to try it, but I won't be surprised if it chews up a
lot
> of
> CPU.
>
> However, doing an equivalent of an incremental backup by means of an
> exclusion
> list doesn't seem possible to me.
>
> Bacula is really quite fast in traversing a very large filesystem
during
> an
> incremental backup.
>
> >
> > Thanks.
> >
> > ----
> > Alan Davis
> > Senior Architect
> > Ruckus Network, Inc.
> > 703.464.6578 (o)
> > 410.365.7175 (m)
> > [EMAIL PROTECTED]
> > alancdavis AIM
> >
> > > -----Original Message-----
> > > From: Kern Sibbald [mailto:[EMAIL PROTECTED]
> > > Sent: Monday, January 29, 2007 2:47 PM
> > > To: bacula-users@lists.sourceforge.net
> > > Cc: Alan Davis
> > > Subject: Re: [Bacula-users] Experience with extremely large
fileset
> > > include lists?
> > >
> > > On Monday 29 January 2007 18:17, Alan Davis wrote:
> > > > I understand that one of the projects is to incorporate features
> >
> > that
> >
> > > > will make very large exclude lists feasible, but does anyone
have
> > > > experience, good or bad, with very large include lists in a
fileset?
> > > >
> > > >
> > > >
> > > > I'm looking at the possibility of building a backup list from a
db
> >
> > query
> >
> > > > that has the potential to return tens of thousands of files
stored
> >
> > in
> >
> > > > hundreds of directories.
> > >
> > > For each file in the directories you specify (normally your whole
> > > filesystem),
> > > Bacula will do a linear search through the exclude list. Thus it
> >
> > could be
> >
> > > extremely CPU intensive. For a large list (more than 1000 files)
I
> > > believe
> > > it (the list) needs to be put into a hash tree, which is code that
> >
> > does
> >
> > > not
> > > exist.
> > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > Alan Davis
> > > >
> > > > Senior Architect
> > > >
> > > > Ruckus Network, Inc.
> > > >
> > > > 703.464.6578 (o)
> > > >
> > > > 410.365.7175 (m)
> > > >
> > > > [EMAIL PROTECTED]
> > > >
> > > > alancdavis AIM
> >
> >
------------------------------------------------------------------------
> -
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to
share
> > your opinions on IT & business topics through brief surveys - and
earn
> cash
> >
>
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
> > _______________________________________________
> > Bacula-users mailing list
> > Bacula-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/bacula-users
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
