Hello,
On 1/4/2006 12:12 AM, Joe Dollard wrote:
I've run into a performance problem when doing backups to tape that I
need some help in resolving. According to the output from btape test,
my tape drive can be written to at around 9,700 KB/s. I've also run a
test with the Windows file daemon and can backup to disk on my bacula
server at around 9,000 KB/s. Based upon these two figures, I would
assume that I should be able to do a backup from the Windows file daemon
to tape at 9,000 KB/s - which over my 100 megabit network I'd be very
happy with.
The basic asumptions sounds reasonable - windows client can deliver
data, and the tape could write it without holding the client. BUT the
figures you give are about the maximum throughput you can get over a
100M ethernet.
However with spooling disabled my backup to tape runs at
about 6700 KB/s (using the same job which gave me 9000 KB/s before).
With spooling enabled my backup runs at approx 4700 KB/s.
Unless I'm mistaken, the througput report witch spooling enabled is not
the figure you're interested in because it measures the overall data
rate: First, data is spooled from client to disk, then despooled from
disk to tape. In other words, the actual speed for each of the processes
might be much higer - in your case, I'd assume that the figures you give
above are a good estimate. 4700K/s, with moving each byte twice, would
be something like 9400K/s for each subprocess.
To solve the problem with direct client to tape data, you need to make
sure that there's no bottleneck in your whole setup. First, even if data
stalls for a short time, the tape drive will stop and has to reposition,
which can take quite long. During that phase, the network buffers will
run full, which, depending on your network and client setup, can even
lead to to a slowed client system.
In other words, writing the data to tape has to be the speed limiting
part of a network backup without spooling.
You can try to tune your network buffer setup - search the archives for
some more information - and you might even try to install a faster
network link between your backup server and the one delivering the data.
A dedicated network link can help a lot, especially if your network is
heavily used by other applications as well when the backup jobs run.
One of my servers has about 240GB of data that I need to run a full
backup on weekly, however my bacula server only has about 100GB of
available disk space. As I don't have enough disk space to spool the
entire job to disk first, the FD is going to be sitting idle while the
SD writes the first 100GB to disk, and then the process will be repeated
again, and again for the final 40GB.
Right, but some time in the future, that might change. Don't hold your
breath, though.
Is there anything I can do in
bacula to allow the FD to keep spooling data to the SD while the SD is
writing data to tape?
There have been different proposals how to handle this problem -
multiple smaller spool files per job are one solution. This might not be
the easiest way, because it requires a big change in the way the SD now
works, and would be limited by hard disk throughput.
The - theoretically - best solution I know about is first a BIG memory
buffer which holds data for the tape and is only written when it's
nearly filled, and which would be re-filled whenever possible. Behind
that, you'd need a fast disk setup with several dedicated disks, each
for one spool file, and of course each on it's controller. Each
controller would need it's own bus or dedicated link to the system, in turn.
In other words, a solution to really achieve maximum throughput requires
not only a major modification of Bacula, but also a really optimized
system it runs on.
Are there any other workaround I could use, or am
I going to have to buy a bigger hard drive for my backup server?
My experience tells me that installing a bigger hard disk spool area is
the best workaround in terms of cost and resulting speed improvement, yes.
Also, assuming that you have enough disk space, the planned (and already
started) development of job migration should allow a "real" D2D2T backup
setup with Bacula, which would allow not only higher data throughput,
but also more flexibility. Admittedly, that's something for the future,
but if you have the disk space now it should be a small modification of
your setup to use that space not as dedicated spool space, but as hard
disk volumes in a migration scheme.
I hope this explains your experiences, and perhaps helps a little when
deciding how to solve the current problem. And, of course, if you can
support development of the features I mentioned, I'm quite sure that
many Bacula users would be much impressed :-)
Arno
Thanks,
Joe
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log
files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
--
IT-Service Lehmann [EMAIL PROTECTED]
Arno Lehmann http://www.its-lehmann.de
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
