On 5/20/2017 1:19 AM, dieter wrote:
If your (590) pages are linked together (such that you must fetch
a page to get the following one) and page fetching is the limiting
factor, then this would limit the parallelizability.
The pages are not linked together. The URL requires a page number. If I
page requester separately from the rest of the script and
> the end value set to 590, each page request takes 1.5 seconds.
That's very slow to fetch a page.
> If I use a thread pool of 16 threads, each request takes 0.1
> seconds. (Higher thread numbers will result in the server for
he
end value set to 590, each page request takes 1.5 seconds.
If I use a thread pool of 16 threads, each request takes 0.1 seconds.
(Higher thread numbers will result in the server forcibly closing the
connection.)
I'm trying to figure out how I would find the sentinel text by using a
Hi all,
I hope someone can help me with this issue
I see that i can't start a thread Pool from another thread, why?
running python 2.6.4 windowsXP
>>> import multiprocessing.dummy as threads
>>> def makePool():
threads.Pool(3)
>>> makePool()
>&g
On Sun, Aug 30, 2009 at 1:06 PM, John Haggerty wrote:
> twisted? I don't get it
>
Twisted. Big library / framework for network'd applications. Specifically,
asynchronous network'd applications, really.
--S
--
http://mail.python.org/mailman/listinfo/python-list
twisted? I don't get it
On Sun, Aug 30, 2009 at 1:55 PM, Stephen Hansen wrote:
>
> On Sun, Aug 30, 2009 at 9:56 AM, Vitaly Babiy wrote:
>
>> Hey,
>> Any one know of a good thread pool library. I have tried a few but they
>> don't seem to clean up after the
On Sun, Aug 30, 2009 at 9:56 AM, Vitaly Babiy wrote:
> Hey,
> Any one know of a good thread pool library. I have tried a few but they
> don't seem to clean up after them selfs well.
>
> Thanks,
> Vitaly Babiy
>
>
As obscene as it seems, I actually use twisted&#
Hey,
Any one know of a good thread pool library. I have tried a few but they
don't seem to clean up after them selfs well.
Thanks,
Vitaly Babiy
--
http://mail.python.org/mailman/listinfo/python-list
Hi, all:
Here is a thread pool for sending as a client, all blocked at
Semaphore.
How can I choose these threads that is idle for some time and
then send heartbeat messages to the server thru sockets of these
threads?
Thanks in advance.
Best wishes
--
http://mail.python.org/mailman
In article <[EMAIL PROTECTED]>,
Alex <[EMAIL PROTECTED]> wrote:
>
>In order to understand the concept of threading pool in python I'm
>working on a simple single-site web crawler.
You can also compare your code against
http://www.pythoncraft.com/OSCON2001/ThreadPoolSpider.py
--
Aahz ([EMAIL PR
On May 17, 2:23 pm, Jeff <[EMAIL PROTECTED]> wrote:
> Your worker threads wait around forever because there is no place for
> them to exit. Queue.get() by default blocks until there is an item in
> the queue available. You can do something like this to cause the
> worker to quit when the queue is
Your worker threads wait around forever because there is no place for
them to exit. Queue.get() by default blocks until there is an item in
the queue available. You can do something like this to cause the
worker to quit when the queue is empty. Just make sure that you fill
the queue before start
Hi all.
In order to understand the concept of threading pool in python I'm
working on a simple single-site web crawler.
I would like to stop the program when the threading pool have
downloaded all internal links from a web site, but now my program keep
waiting forever even if there are no more lin
If the task is I/O bound (something like a web spider?), seems
>they'd still be useful...
Yes, I use a thread pool for that. But async I/O may be a more efficient
solution (e.g. Twisted).
--
René Pijlman
Wat wil jij leren? http://www.leren.nl
--
http://mail.python.org/mailman/listinfo/python-list
> There isn't a thread pool module in the standard library, but I'm sure
> many have been written by people in the python community.
> Anyone have a favorite? Is there one particular implementation that's
> recommended?
>
> Not looking for anything fancy, jus
David Hirschfield wrote:
> There isn't a thread pool module in the standard library, but I'm sure
> many have been written by people in the python community.
> Anyone have a favorite? Is there one particular implementation that's
> recommended?
Because of the GIL, thr
I believe "Python in a Nutshell" has a couple of clear examples using
Queue and Threading, including one with a pool of worker threads that
wait for entries in one queue and place results in another.
Also you should look at the Python Cookbook, which probably includes
the same or similar examples
I think you want the Queue module in standard lib. Haven't started a
thread yet without it :)
regards
/rune
--
http://mail.python.org/mailman/listinfo/python-list
There isn't a thread pool module in the standard library, but I'm sure
many have been written by people in the python community.
Anyone have a favorite? Is there one particular implementation that's
recommended?
Not looking for anything fancy, just something that lets me queue
19 matches
Mail list logo
