On 6/19/2011 12:03 PM, Chris Angelico wrote:
On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessens<moky.m...@gmail.com> wrote:
My problem is that when FileToCopyTask raises an error, the program does not
stop.
In fact when the error is Disk Full, I want to stop the whole program
because I know that the next task will fail too.
If you're starting a thread for every file you're copying, you're
starting a huge number of threads that probably will just end up
fighting over the disk. To get a reasonably efficient early-abort, I'd
recommend having a fixed pool of worker threads (say, ten of them),
and have each thread (a) check if the early-abort flag is set, and
then (b) start copying the next file in queue. Once the queue's empty
or the early-abort flag is set, all ten threads will terminate when
they finish their current transfers.
(The ten threads figure is arbitrary. Optimum value for performance
will come by adjusting this.)
I an not convinced that much of anything is gained by having multiple
copying threads. It certainly adds complication -- hence this thread. I
would just use shutil to copy files or directories of files.
Note that copytree has an option to pass a filter function:
"If ignore is given, it must be a callable that will receive as its
arguments the directory being visited by copytree(), and a list of its
contents, as returned by os.listdir(). Since copytree() is called
recursively, the ignore callable will be called once for each directory
that is copied. The callable must return a sequence of directory and
file names relative to the current directory (i.e. a subset of the items
in its second argument); these names will then be ignored in the copy
process. ignore_patterns() can be used to create such a callable that
ignores names based on glob-style patterns."
--
Terry Jan Reedy
--
http://mail.python.org/mailman/listinfo/python-list
