Re: [Bacula-users] MySQL Indexes

2006-06-04 Thread Francisco Reyes
Dan Trainor writes: For millions of files try the directory restore option. 11: Enter a list of directories to restore for found JobIds<-- Significantly faster than trying to build the whole list of files. ___ Bacula-users mailing list Bacula-users

Re: [Bacula-users] MySQL Indexes

2006-05-26 Thread Peter Eriksson
On Fri, 26 May 2006, Martin Simmons wrote: > > > PID USERNAME THR PRI NICE SIZE RES STATETIMECPU COMMAND > > > 11507 root 3 590 5800K 2456K sleep 18:09 2.19% bacula-fd > > > 11613 daemon18 590 4368K 2256K sleep1:26 1.37% nfsmapid > > > 17097 root 1 5

Re: [Bacula-users] MySQL Indexes

2006-05-26 Thread Martin Simmons
> On Fri, 26 May 2006 16:09:18 +0200 (MEST), Peter Eriksson <[EMAIL > PROTECTED]> said: > > On Wed, 24 May 2006, Dan Trainor wrote: > > > I couldn't help but notice you mentioning that it had taken over two > > whole days to recover 360G of data. Are you kidding? > > Now the 2 days hav

Re: [Bacula-users] MySQL Indexes

2006-05-26 Thread Peter Eriksson
On Fri, 26 May 2006, Josh Fisher wrote: > Top shows over half of your 3 GB swap space being utilized. Almost as > much memory is in swap as is in physical RAM. Swap is perhaps being > thrashed. Not much I/O was going on so I don't think so. Ah! I was using the wrong version of top. A more accurat

Re: [Bacula-users] MySQL Indexes

2006-05-26 Thread Josh Fisher
Peter Eriksson wrote: On Wed, 24 May 2006, Dan Trainor wrote: I couldn't help but notice you mentioning that it had taken over two whole days to recover 360G of data. Are you kidding? Now the 2 days have passed and this time I was able to not make any mistakes and have started the ac

Re: [Bacula-users] MySQL Indexes

2006-05-26 Thread Peter Eriksson
On Wed, 24 May 2006, Dan Trainor wrote: > I couldn't help but notice you mentioning that it had taken over two > whole days to recover 360G of data. Are you kidding? Now the 2 days have passed and this time I was able to not make any mistakes and have started the actual recover process... Curre

Re: [Bacula-users] MySQL Indexes

2006-05-26 Thread Alan Brown
On Thu, 25 May 2006, Ryan Novosielski wrote: Note that it is _essential_ to have enough RAM or the machine will start banging on swap and slow down dramatically. How much is enough? Is there a metric for catalog size vs. memory? I have 2Gb ram. It's enough to cope with restore sets of 4-6 mi

Re: [Bacula-users] MySQL Indexes

2006-05-25 Thread Ryan Novosielski
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alan Brown wrote: > On Thu, 25 May 2006, Kern Sibbald wrote: > >>> On backups here with ~4 million files in them, the tree build takes less >>> than an hour. I usually just start the restore process and go do >>> something >>> else for a while. >> >>

Re: [Bacula-users] MySQL Indexes

2006-05-25 Thread Alan Brown
On Thu, 25 May 2006, Kern Sibbald wrote: On backups here with ~4 million files in them, the tree build takes less than an hour. I usually just start the restore process and go do something else for a while. Well that is nice to hear. :-) Note that it is _essential_ to have enough RAM or the

Re: [Bacula-users] MySQL Indexes

2006-05-25 Thread Kern Sibbald
> On Wed, 24 May 2006, Kern Sibbald wrote: > >> The problem isn't the 360G of data, it could be 1000G and the >> problem >> would be the same. It is the 5 million files. The current code is not >> very >> scalable beyond a million or two files when doing an interactive >> restore. >> It simply

Re: [Bacula-users] MySQL Indexes

2006-05-25 Thread Alan Brown
On Wed, 24 May 2006, Kern Sibbald wrote: The problem isn't the 360G of data, it could be 1000G and the problem would be the same. It is the 5 million files. The current code is not very scalable beyond a million or two files when doing an interactive restore. It simply takes too long to buil

Re: [Bacula-users] MySQL Indexes

2006-05-25 Thread Vicente Hernandez
We have the same problem. We have some servers that have more than 2 million files. Theses servers are taken a lot of time to backup, and the restores are taking a lot of time too. As we have seen, it points that the problem is in the inserts/selects to the mysql server. Vicente Hernández Ve

Re: [Bacula-users] MySQL Indexes

2006-05-25 Thread Rudolf Cejka
> (in bacula-dir) incore tree of files to restore for a 360GB partition > which is kind of annoying (especially when you make mistakes like I did > and have to restart the whole operation from the beginning...) How many files? For jobs with 5 millions of files, I wait at most 10 minut

Re: [Bacula-users] MySQL Indexes

2006-05-24 Thread Kern Sibbald
> Kern Sibbald wrote: >>>Peter Eriksson wrote: >>> Hi List! :-) I'm (again) trying to improve the performance of our Bacula installation. Currently it takes approx 2 full days for a full recover to generate the (in bacula-dir) incore tree of files to restore for

Re: [Bacula-users] MySQL Indexes

2006-05-24 Thread Dan Trainor
Kern Sibbald wrote: Peter Eriksson wrote: Hi List! :-) I'm (again) trying to improve the performance of our Bacula installation. Currently it takes approx 2 full days for a full recover to generate the (in bacula-dir) incore tree of files to restore for a 360GB partition which is kind of anno

Re: [Bacula-users] MySQL Indexes

2006-05-24 Thread Kern Sibbald
> Peter Eriksson wrote: >> Hi List! :-) >> >> I'm (again) trying to improve the performance of our Bacula >> installation. >> >> Currently it takes approx 2 full days for a full recover to generate the >> (in bacula-dir) incore tree of files to restore for a 360GB partition >> which is kind of ann

Re: [Bacula-users] MySQL Indexes

2006-05-24 Thread Peter Eriksson
> I couldn't help but notice you mentioning that it had taken over two > whole days to recover 360G of data. Are you kidding? Actually it hasn't even started to actual _recover_ process yet. It's busy building a tree (approx 1.3GB of RAM will be used for that) of what files to restore: >Select

Re: [Bacula-users] MySQL Indexes

2006-05-24 Thread Dan Trainor
Peter Eriksson wrote: Hi List! :-) I'm (again) trying to improve the performance of our Bacula installation. Currently it takes approx 2 full days for a full recover to generate the (in bacula-dir) incore tree of files to restore for a 360GB partition which is kind of annoying (especially when