Ok sorry for the mail,
I found the solution by using deque.count(url) that's available
starting from python 2.7
On Fri, Oct 12, 2012 at 2:40 PM, Christophe Vandeplas
<christo...@vandeplas.com> wrote:
> Hello,
>
> I have a question about deque and thread-safety.
>
> My application has multiple threads running concurrently and doing the
> same action (downloading pages)
> To know what has already been downloaded I created the variable:
> seen = deque('', 1000) (keeps list of max 1000 urls in memory)
>
> In one place of the code I do: seen.append(url)
>
> And another place:
> def seenPage()
> if url in seen:
> return True
> return False
>
> From the documentation I understand that deques are thread-safe:
>> Deques are a generalization of stacks and queues (the name is pronounced
>> “deck”
>> and is short for “double-ended queue”). Deques support thread-safe, memory
>> efficient appends and pops from either side of the deque with approximately
>> the
>> same O(1) performance in either direction.
>
> It seems that appending to deques is indeed thread-safe, but not
> iterating over them.
> I've seen a similar, but different, situation here:
> http://comments.gmane.org/gmane.comp.python.devel/85487
>
> Forgetting the above url, and considering my situation this behavior
> screws up the concept I need:
> - Keeping a thread-safe collection of seen urls,
> - Being able to check if something is in that collection
> - No need to clean the collection to prevent the memory from filling up
>
> So I know I could work around this problem by using a lock.
> But then I don't only need to use the lock around the iterator, but
> also around the append(), but that defeats the purpose of deque being
> thread-safe.
>
> In short, what's your advice:
>
> 1/ build a lock around the .append() and the iterator. Using the
> already-existing lock in the deque. But HOW?
>
> 1/ simply build a lock around the .append() and the iterator.
> Defeating the build-in thread-safety.
>
> 2/ use another collection that does what I need
>
>
> Thanks for your expertise.
>
> Christophe
--
http://mail.python.org/mailman/listinfo/python-list
