Hi J.R.,
> I was trying to make some processes faster which use tar a bit on
> large archives. I was able to use the command line options
> --use-compress-program pbzip2 and that helped a lot.
I think analysis of where the remaining delay is would be needed before
investing effort in removing it. There's a pipe between tar that's
reading the files, and pbzip2 that's writing the compressed tar file.
tar write(2)s down it 10 KiB at a time.
pipe(7) on Linux says the pipe's buffer is 64 KiB, but can be altered
with fcntl(2)'s F_SETPIPE_SZ. That 64 KiB allows the pipe writer to run
ahead of the reader: tar ahead of the compressor, a bigger pipe buffer
might stop the pipe reader being starved, if that's what's happening.
I thought I'd look at the 25/50/75-percentile of time taken to write(2)
down the pipe compared with reading from it.
$ LC_ALL=C strace -ffe %desc -tttT -o st \
tar --use-compress-program 'lzip -9' -cf foo.tar.lz
/boot/vmlinuz-linux
$
$ grep ' write(4, ' st.23321 |
> egrep -o ' <[0-9.]+>$' |
> tr -d ' <>' |
> stats
length min 25%ile median mean 75%ile max stddev
521 0.0000340 0.0000880 0.0000920 0.0001344 0.0000980 0.0197400
0.0008624761
$
$ sed -n '/boot\/vmlinuz-linux/,$p' st.23322 |
> grep 'read(0, ' |
> egrep -o ' <[0-9.]+>$' |
> tr -d ' <>' |
> stats
length min 25%ile median mean 75%ile max stddev
516 0.0000490 0.0002430 0.0002480 0.0002580 0.0002572 0.0017330
6.939183e-05
$
--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy
