On Tuesday 30 January 2007 16:43, Alan Davis wrote:
> Returning to the original thread...
>
> Just to make sure I'm being clear - my FileSet specification is:
>
> FileSet {
> Name = "u2LgFileList"
> Include {
> Options {
> signature = MD5
> }
> File = </local/etc/u2LgFileList.list
>
> }
> }
>
> The file /local/etc/u2LgFileList.list has 29K+ entries in it.
> Note that this is /not/ an exclude list - it's explicitly listing the
> files to be backed up.
OK, that is something that in principle could work.
>
> The FD takes about 40 minutes to read in the file list.
You didn't say how you were measuring this time. It is *very* unlikely that
it takes 40 minutes to read the file list. It should be read in in a matter
of seconds. During the time the FD is reading in the file list, unless I am
mistaken about what you are doing, it has not yet contacted the FD, which
would cause the problem you note below.
I suspect that something is wrong with the syntax and the FD is blocking. You
might first try this with a list of a few hundred files.
> The SD times out in 30 minutes waiting for the FD.
If it is really taking 40 minutes for the FD to initialize then this will
happen.
>
> >From my reading of the manual there are directives that set the time
> >From my reading of the manual there are directives that set the time
>
> that the Director will wait for an FD to respond "FD Connect Timeout"
> and the time that the FD will wait for the SD to respond "SD Connect
> Timeout" as well as the "Heartbeat Interval" that will keep the
> connection open during long backups.
>
> I've not found a directive to modify the length of time that the SD will
> wait for the FD to begin transferring data.
No, there is not one, and I don't plan on adding one since the FD should be up
and running in a matter of seconds, or at worse a minute or two.
>
> This is the error message from the failed backup message. Note that the
> authorization rejection is /not/ the problem - a test backup that
> succeeded was used to verify proper authorization and communication
> between FD and SD.
>
> 29-Jan 16:33 athos-dir: Start Backup JobId 112,
> Job=u2FullBackupJob.2007-01-29_16.33.15
> 29-Jan 17:21 u2-fd: u2FullBackupJob.2007-01-29_16.33.15 Fatal error:
> Authorization key rejected by Storage daemon.
> Please see http://www.bacula.org/rel-manual/faq.html#AuthorizationErrors
> for help.
> 29-Jan 17:21 u2-fd: u2FullBackupJob.2007-01-29_16.33.15 Fatal error:
> Failed to authenticate Storage daemon.
> 29-Jan 17:21 athos-dir: u2FullBackupJob.2007-01-29_16.33.15 Fatal error:
> Socket error on Storage command: ERR=No data available 29-Jan 17:21
> athos-dir: u2FullBackupJob.2007-01-29_16.33.15 Error: Bacula 1.39.22
> (09Sep06): 29-Jan-2007 17:21:22
>
> I think that the timeout is being specified in this code from
> stored/job.c
>
> gettimeofday(&tv, &tz);
> timeout.tv_nsec = tv.tv_usec * 1000;
> timeout.tv_sec = tv.tv_sec + 30 * 60; /* wait 30 minutes */
>
> Dmsg1(100, "%s waiting on FD to contact SD\n", jcr->Job);
> /*
> * Wait for the File daemon to contact us to start the Job,
> * when he does, we will be released, unless the 30 minutes
> * expires.
> */
> P(mutex);
> for ( ;!job_canceled(jcr); ) {
> errstat = pthread_cond_timedwait(&jcr->job_start_wait, &mutex,
> &timeout);
> if (errstat == 0 || errstat == ETIMEDOUT) {
> break;
> }
> }
> V(mutex);
Yes, that is how the SD timeouts out a FD that has not connected. As
mentioned above, I see no reason to change it. If it is taking 40 minutes to
start the FD, then something is terribly wrong and when the FD starts to walk
the filesystem, the delay will be hundreds or thousands of times longer.
As I told you, the current algorithm starts breaking down after about 1000
entries. To make it work with 29,000 is going to require some recoding. As
Rudolf indicated, it may be a trivial matter of changing the calls to use the
hash table instead of a dlist, but someone needs to program it ...
Regards,
Kern
>
>
>
> ----
> Alan Davis
> Senior Architect
> Ruckus Network, Inc.
> 703.464.6578 (o)
> 410.365.7175 (m)
> [EMAIL PROTECTED]
> alancdavis AIM
>
> > -----Original Message-----
> > From: Kern Sibbald [mailto:[EMAIL PROTECTED]
> > Sent: Monday, January 29, 2007 4:06 PM
> > To: bacula-users@lists.sourceforge.net
> > Cc: Alan Davis
> > Subject: Re: [Bacula-users] Experience with extremely large fileset
> > include lists?
> >
> > Hello,
> >
> > On Monday 29 January 2007 21:19, Alan Davis wrote:
> > > Kern,
> > >
> > > Thanks for the fast response. To clarify a bit - the file list that
>
> I
>
> > > would be using would be individual files, not directories. There
>
> would
>
> > > be no exclude list as only the files that I need backed up would be
> > > listed.
> >
> > Yes, my answer was based on that assumption.
> >
> > > I have about 30TB of data files spread over several hundred
>
> directories.
>
> > > A true incremental backup will spend large amounts of time
>
> determining
>
> > > what files have been changed or added. The information about the
> > > modified or new files is stored in a db as a side-effect of
>
> processing
>
> > > the files for release to production so building a file list is
>
> trivial.
>
> > > The only problem would be the FD's capability of handling a file
>
> list of
>
> > > 10K+ entries.
> >
> > All I can say is to try it, but I won't be surprised if it chews up a
>
> lot
>
> > of
> > CPU.
> >
> > However, doing an equivalent of an incremental backup by means of an
> > exclusion
> > list doesn't seem possible to me.
> >
> > Bacula is really quite fast in traversing a very large filesystem
>
> during
>
> > an
> > incremental backup.
> >
> > > Thanks.
> > >
> > > ----
> > > Alan Davis
> > > Senior Architect
> > > Ruckus Network, Inc.
> > > 703.464.6578 (o)
> > > 410.365.7175 (m)
> > > [EMAIL PROTECTED]
> > > alancdavis AIM
> > >
> > > > -----Original Message-----
> > > > From: Kern Sibbald [mailto:[EMAIL PROTECTED]
> > > > Sent: Monday, January 29, 2007 2:47 PM
> > > > To: bacula-users@lists.sourceforge.net
> > > > Cc: Alan Davis
> > > > Subject: Re: [Bacula-users] Experience with extremely large
>
> fileset
>
> > > > include lists?
> > > >
> > > > On Monday 29 January 2007 18:17, Alan Davis wrote:
> > > > > I understand that one of the projects is to incorporate features
> > >
> > > that
> > >
> > > > > will make very large exclude lists feasible, but does anyone
>
> have
>
> > > > > experience, good or bad, with very large include lists in a
>
> fileset?
>
> > > > > I'm looking at the possibility of building a backup list from a
>
> db
>
> > > query
> > >
> > > > > that has the potential to return tens of thousands of files
>
> stored
>
> > > in
> > >
> > > > > hundreds of directories.
> > > >
> > > > For each file in the directories you specify (normally your whole
> > > > filesystem),
> > > > Bacula will do a linear search through the exclude list. Thus it
> > >
> > > could be
> > >
> > > > extremely CPU intensive. For a large list (more than 1000 files)
>
> I
>
> > > > believe
> > > > it (the list) needs to be put into a hash tree, which is code that
> > >
> > > does
> > >
> > > > not
> > > > exist.
> > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > >
> > > > > ----
> > > > >
> > > > > Alan Davis
> > > > >
> > > > > Senior Architect
> > > > >
> > > > > Ruckus Network, Inc.
> > > > >
> > > > > 703.464.6578 (o)
> > > > >
> > > > > 410.365.7175 (m)
> > > > >
> > > > > [EMAIL PROTECTED]
> > > > >
> > > > > alancdavis AIM
>
> ------------------------------------------------------------------------
>
> > -
> >
> > > Take Surveys. Earn Cash. Influence the Future of IT
> > > Join SourceForge.net's Techsay panel and you'll get the chance to
>
> share
>
> > > your opinions on IT & business topics through brief surveys - and
>
> earn
>
> > cash
>
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
> V
>
> > > _______________________________________________
> > > Bacula-users mailing list
> > > Bacula-users@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/bacula-users
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
