On 7 Ott, 06:37, "Gabriel Genellina" <[EMAIL PROTECTED]> wrote: > En Mon, 06 Oct 2008 11:24:51 -0300, <[EMAIL PROTECTED]> escribió: > > > On 6 Ott, 15:24, oyster <[EMAIL PROTECTED]> wrote: > >> my code is not right, can sb give me a hand? thanx > > >> for example, I have 1000 urls to be downloaded, but only 5 thread at > >> one time > > I would restructure my code with someting like this ( WARNING: the > > following code is > > ABSOLUTELY UNTESTED and shall be considered only as pseudo-code to > > express my idea of > > the algorithm (which, also, could be wrong:-) ): > > Your code creates one thread per url (but never more than MAX_THREADS > alive at the same time). Usually it's more efficient to create all the > MAX_THREADS at once, and continuously feed them with tasks to be done. A > Queue object is the way to synchronize them; from the documentation: > > <code> > from Queue import Queue > from threading import Thread > > num_worker_threads = 3 > list_of_urls = ["http://foo.com";, "http://bar.com";, > "http://baz.com";, "http://spam.com";, > "http://egg.com";, > ] > > def do_work(url): > from time import sleep > from random import randrange > from threading import currentThread > print "%s downloading %s" % (currentThread().getName(), url) > sleep(randrange(5)) > print "%s done" % currentThread().getName() > > # from this point on, copied almost verbatim from the Queue example > # at the end ofhttp://docs.python.org/library/queue.html > > def worker(): > while True: > item = q.get() > do_work(item) > q.task_done() > > q = Queue() > for i in range(num_worker_threads): > t = Thread(target=worker) > t.setDaemon(True) > t.start() > > for item in list_of_urls: > q.put(item) > > q.join() # block until all tasks are done > print "Finished" > </code> > > -- > Gabriel Genellina
Agreed. I was trying to do what the OP was trying to do, but in a way that works. But keeping the thread alive and feeding them the URL is a better design, definitly. And no, I don't think its 'premature optimization': it is just cleaner. Ciao ------ FB -- http://mail.python.org/mailman/listinfo/python-list
