It sounds like there are at least two issues involved.
The first could be a buffer cache starvation issue due to the load on
the filesystem from the tar. If the usb program is doing any filesystem
operation at all, even at low bandwidths, it could be hitting blockages
due to the disk intensive tar eating up available buffer cache buffers
(e.g. causing an excessive ratio of dirty buffers vs clean buffers).
This would NOT be a scheduler problem per-say, but instead a kernel
resource management problem.
The way to test this is to double-buffer or tripple-buffer the output
via shared memory. A pipe might not do the job if it gets stuck doing
direct transfers (I eventually gave up trying to optimize pipes in DFly
due to a similar problem and just run everything through a kernel buffer
now). Still, it may be possible to test against this particular problem
by having the program write to a pipe and another program or fork handle
the actual writing to the disk or filesystem.
Another way to test this is to comment out the writing in the usb program
entirely and see if things improve.
--
The second issue sounds more scheduler-related. Try running the
usb program at nice -20? You could even run it at a pseudo-realtime
priority using rtprio but nice -20 had better work properly against
a md5 or there is something seriously broken in the scheduler.
Dynamic priority handling is supposed to deal with this sort of thing
automatically, particularly if the usb program is not using a lot of
cpu, but sometimes it can't tell whether a newly-exec'd program is
going to be interactive or batch until after it has run for a while.
Tuning initial conditions after an exec for the scheduler is not an
easy task. Simply giving a program a more batch/bulk-run priority on
exec and letting the dynamic priority shift it more to interactive
operation tends to mess up interactive shells in the face of
cpu-intensive system operation, for example. Theoretically dynamic
priority handling should bump up the priority for the usb program well
beyond any initial conditions for exec once it has been running a while,
assuming it doesn't use tons of cpu.
--
An md5, or any single-file reading operation, would not overload the
buffer cache. File writing and/or multi-file operations (such as a
tar extraction or a tar-up) can create blockages in the buffer cache.
It takes a considerable amount of VM/buffer-cache tuning to get those
subsystems to pipeline properly and sometimes things can go stale and
stop pipelining properly for months without anyone realizing it.
-Matt
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
