Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-20 Thread Phil Stracchino
On 03/20/16 14:43, Dimitri Maziuk wrote: > On 2016-03-19 14:41, Phil Stracchino wrote: > >> What's the best way to handle a removable-cartridge-drive technology >> like RDX in Bacula -- use the virtual changer...? > > "Without vchanger, Bacula has no right to claim it supports disk-based > backu

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-20 Thread Dimitri Maziuk
On 2016-03-19 14:41, Phil Stracchino wrote: > What's the best way to handle a removable-cartridge-drive technology > like RDX in Bacula -- use the virtual changer...? "Without vchanger, Bacula has no right to claim it supports disk-based backup". Vchanger is the only way to fly. LTO-6: ~$11/TB

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-20 Thread Josh Fisher
On 3/19/2016 3:41 PM, Phil Stracchino wrote: > On 03/19/16 10:56, Josh Fisher wrote: >> On 3/17/2016 8:48 AM, Alan Brown wrote: >>> . What's killed all these "smaller" >>> formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix. >>> That's despite even BDXL 120GB not being large en

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-19 Thread Cejka Rudolf
Alan Brown wrote (2016/03/17): > Caveat: BDXL is up to 120GB per disc (quad layer) and It _may_ be worth > investigating this format for backups, but bacula doesn't play nicely > with optical media. > > HVD development (6TB per disc) was abandoned in 2008. Ritek demonstrated > 250GB BDXL discs

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-19 Thread Kern Sibbald
On 03/20/2016 05:41 AM, Phil Stracchino wrote: > On 03/19/16 10:56, Josh Fisher wrote: >> On 3/17/2016 8:48 AM, Alan Brown wrote: >>> . What's killed all these "smaller" >>> formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix. >>> That's despite even BDXL 120GB not being large

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-19 Thread Phil Stracchino
On 03/19/16 10:56, Josh Fisher wrote: > On 3/17/2016 8:48 AM, Alan Brown wrote: >> . What's killed all these "smaller" >> formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix. >> That's despite even BDXL 120GB not being large enough capacity to hold a >> complete 4k video title. >

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-19 Thread Josh Fisher
On 3/17/2016 8:48 AM, Alan Brown wrote: > . What's killed all these "smaller" > formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix. > That's despite even BDXL 120GB not being large enough capacity to hold a > complete 4k video title. > RDX is a good choice for "smaller" format,

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-18 Thread Alan Brown
On 12/03/16 00:14, Heitor Faria wrote: > >> SSD is the only way to fly. After having tested with a PCIe NVMe drive, I'd >> say >> that's preferred, but a _fast_ SATA2/3 or SAS2 drive will work too (The old >> spool was a stripe of Intel SLC SSDs, the new one is a DC3700 card) > I never got this sp

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-14 Thread Alan Brown
On 13/03/16 21:48, Dan Langille wrote: >> As well as increasing max file size you need to boost the tape buffer size >> from the 64kB default. I use 2MB > This is a hardware setting? No, it's a bacula-sd setting > I tried Minimum block size & Maximum block size on my tape drive, but need to > t

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-14 Thread Cejka Rudolf
Dan Langille wrote (2016/03/13): > > As well as increasing max file size you need to boost the tape buffer size > > from the 64kB default. I use 2MB > > This is a hardware setting? Hello, it is software settings: * Increasing max file size is "Maximum File Size". I use 16 GB for LTO-5 and for

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-13 Thread Dan Langille
> On Mar 11, 2016, at 6:56 PM, Alan Brown wrote: > > On 11/03/16 20:14, Simon Templar wrote: >> In my case using spooling didn’t prevent shoe-shining; it just introduced >> long pauses while data was spooled. I think all this means is that I can >> read from my data sources faster than my tape

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-13 Thread Dan Langille
> On Mar 11, 2016, at 7:14 PM, Heitor Faria wrote: > > On 11/03/16 20:14, Simon Templar wrote: > In my case using spooling didn’t prevent shoe-shining; it just introduced > long pauses while data was spooled. I think all this means is that I can read > from my data sources faster than my tape c

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-11 Thread Heitor Faria
> On 11/03/16 20:14, Simon Templar wrote: >> In my case using spooling didn’t prevent shoe-shining; it just introduced >> long >> pauses while data was spooled. I think all this means is that I can read from >> my data sources faster than my tape can write. > Unless you are using DAT, do not use

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-11 Thread Alan Brown
On 11/03/16 20:14, Simon Templar wrote: > In my case using spooling didn’t prevent shoe-shining; it just > introduced long pauses while data was spooled. I think all this means > is that I can read from my data sources faster than my tape can write. Unless you are using DAT, do not use mechanical

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-11 Thread Dan Langille
> On Mar 11, 2016, at 3:14 PM, Simon Templar wrote: >> On Mar 10, 2016, at 11:09 PM, Kern Sibbald wrote: >> >> I have not tried this, but one thing that may help a lot is to turn on >> data spooling for the tape device. This will probably not speed up the >> process but should prevent that tape

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-11 Thread Simon Templar
In my case using spooling didn’t prevent shoe-shining; it just introduced long pauses while data was spooled. I think all this means is that I can read from my data sources faster than my tape can write. So far the only change I made to help with shoe-shining was to set Max File Size to a large

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-11 Thread Dan Langille
-- Dan Langille - BSDCan / PGCon d...@langille.org > On Mar 10, 2016, at 11:09 PM, Kern Sibbald wrote: > > Hello Dan, > > Copying from disk to tape with Bacula's current algorithm is virtually > guaranteed to be slower than using tar. This is for several reasons: > > 1. Bacula currently

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-10 Thread Kern Sibbald
Hello Dan, Copying from disk to tape with Bacula's current algorithm is virtually guaranteed to be slower than using tar. This is for several reasons: 1. Bacula currently is single threaded and reads a block from disk then stops to write the output block to tape. 2. After reading the block fr

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-10 Thread Dan Langille
> On Mar 10, 2016, at 6:35 PM, compdoc wrote: > >> Given LTO-4 can do 120MB/s, yeah, 94 is good enough > > Be sure to disable compression. I think I have it disabled two places, on Dir > and well as on the SD. Side note: I see compression mentioned at http://www.bacula.org/7.4.x-manuals/en/ma

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-10 Thread compdoc
> Given LTO-4 can do 120MB/s, yeah, 94 is good enough Be sure to disable compression. I think I have it disabled two places, on Dir and well as on the SD. I built a low-power, mini-itx system to house an LTO-4 drive. The controller is an LSI LSI20320IE Ultra320. All backup data is over the lan

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-10 Thread Dan Langille
> On Mar 9, 2016, at 6:52 PM, Heitor Faria wrote: > > I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 > and PostgreSQL 9.4 on FreeBSD 10.2 > > Everything is within one SD > > Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c > > The job summary

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-10 Thread Dan Langille
> On Mar 10, 2016, at 10:10 AM, Simon Templar wrote: > > Dan, > > I’m new to bacula, and arguably not very smart, but I’ve been struggling with > tape drive performance pretty much since the moment I got the configurations > to a functional state so I’ll share my learnings thus far. > > Can y

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-10 Thread Dan Langille
> On Mar 10, 2016, at 2:19 PM, Josh Fisher wrote: > > > Was job 232778 the only job on these inc volumes? Yes, and no. On 4 of the 12 volumes, part of the job was at the end of each volume. I have indicated yes/no for exclusive use of the volume, and the time Bacula spends reading from that

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-10 Thread Josh Fisher
Was job 232778 the only job on these inc volumes? More specifically, were multiple concurrent jobs writing interleaved blocks to these inc volumes at the same time job 232778 was? A copy job is essentially the same thing as a restore and could be slowed by interleaving. Bacula has to step thr

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-10 Thread Simon Templar
Dan, I’m new to bacula, and arguably not very smart, but I’ve been struggling with tape drive performance pretty much since the moment I got the configurations to a functional state so I’ll share my learnings thus far. Can you hear your tape drive? If so, do you hear lots of stops and starts wh

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

2016-03-09 Thread Heitor Faria
> I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 > and > PostgreSQL 9.4 on FreeBSD 10.2 > Everything is within one SD > Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c > The job summary: > Start time: 09-Mar-2016 19:41:54 > End time: 09-Mar-2