Glenn Langford added the comment:
In abstract, I like the namedtuple interface for sqlite3 as well. One caution
is that the approach suggested at
http://peter-hoffmann.com/2010/python-sqlite-namedtuple-factory.html
can have a dramatic impact on performance. For one DB-intensive application
Glenn Langford added the comment:
...if I understand the proposed caching scheme, then repeated executions of the
query
SELECT a,b,c FROM table
would result in cache hits, since the column names remain the same. I'm
guessing this would resolve the performance problem in the app I saw, b
Glenn Langford added the comment:
> My workload is launching two subprocess in parallel, and whenever one is
> ready, launches another one.
Since you have timeout=15.0, wait() should return at least every 15s. Can you
determine if the wait is being repeatedly called in the while loop,
Glenn Langford added the comment:
> the wait is actually returning after the 15 seconds, although nothing is
> reported as finished...What kind of debug information from the futures would
> be useful?
What is the state of the pending Futures that wait() is stuck on? (e.g. display
Glenn Langford added the comment:
> Any ideas how to debug this further?
Wherever the cause of the problem might live, and to either work around it or
gain additional information, here is one idea to consider.
Do you need to submit your Futures just two at a time, and tightly loop every
Glenn Langford added the comment:
Under the hood, the behaviour of as_completed is quite different. So there is
no guarantee it would behave the same.
In any event, with millions of tasks you might consider Celery (I haven't used
it myself):
http://www.celeryprojec
Changes by Glenn Langford :
--
nosy: -glangford
___
Python tracker
<http://bugs.python.org/issue20319>
___
___
Python-bugs-list mailing list
Unsubscribe:
Changes by Glenn Langford :
--
nosy: -glangford
___
Python tracker
<http://bugs.python.org/issue20516>
___
___
Python-bugs-list mailing list
Unsubscribe:
Changes by Glenn Langford :
--
nosy: -glangford
___
Python tracker
<http://bugs.python.org/issue13299>
___
___
Python-bugs-list mailing list
Unsubscribe:
Changes by Glenn Langford :
--
nosy: -glangford
___
Python tracker
<http://bugs.python.org/issue20369>
___
___
Python-bugs-list mailing list
Unsubscribe:
Changes by Glenn Langford :
--
nosy: -glangford
___
Python tracker
<http://bugs.python.org/issue20297>
___
___
Python-bugs-list mailing list
Unsubscribe:
New submission from Glenn Langford:
as_completed() calls _create_and_install_waiters() over all given Futures, even
if some (or all) of the Futures have already completed.
The following fragment of the as_completed code (from 3.3.3):
with _AcquireFutures(fs):
finished = set
New submission from Glenn Langford:
concurrent.futures.wait() can get into a state where it blocks forever on
waiter.event.wait(), even when the underlying Futures have completed.
This is demonstrated in a stress test where a large number of wait() calls are
run in multiple threads
Glenn Langford added the comment:
See also http://bugs.python.org/issue20319
--
___
Python tracker
<http://bugs.python.org/issue20297>
___
___
Python-bugs-list m
Glenn Langford added the comment:
Uploading proposed new version of as_completed() for review. Note the following
changes:
- does not install waiters for Futures which are completed
- locks only one Future at a time to improve concurrency (rather than locking
all Futures at once); traverses
Changes by Glenn Langford :
--
nosy: +bquinlan
___
Python tracker
<http://bugs.python.org/issue20297>
___
___
Python-bugs-list mailing list
Unsubscribe:
Changes by Glenn Langford :
--
nosy: +bquinlan
___
Python tracker
<http://bugs.python.org/issue20319>
___
___
Python-bugs-list mailing list
Unsubscribe:
Glenn Langford added the comment:
The same bug also exists in concurrent.futures.as_completed(). The minimal fix
suggested here also works, but the bigger fix suggested in issue #20297 is
recommended for as_completed().
--
___
Python tracker
<h
Changes by Glenn Langford :
--
nosy: +mark.dickinson
___
Python tracker
<http://bugs.python.org/issue20297>
___
___
Python-bugs-list mailing list
Unsubscribe:
Glenn Langford added the comment:
@Victor: Would like to give a patch but I am not a core developer, and I don't
version control set up yet. The proposed fix is based on reading the
distribution source code.
--
___
Python tracker
Glenn Langford added the comment:
Comments on @Victor's comments - I will have a look at the dev guide. :-)
I think you have identified another bug in the current code.
"Future._waiters is a list and so accept duplicated waiters." What this means
is that the following
Changes by Glenn Langford :
Removed file: http://bugs.python.org/file33656/test_dupfuture.py
___
Python tracker
<http://bugs.python.org/issue20297>
___
___
Python-bug
Changes by Glenn Langford :
Added file: http://bugs.python.org/file33657/test_dupfuture.py
___
Python tracker
<http://bugs.python.org/issue20297>
___
___
Python-bug
Changes by Glenn Langford :
--
nosy: +haypo, mark.dickinson, tim.peters
___
Python tracker
<http://bugs.python.org/issue20367>
___
___
Python-bugs-list mailin
New submission from Glenn Langford:
concurrent.futures.as_completed([f,f]) will yield f twice, then fail with a
KeyError for a Future f which is not completed.
If the Future has already completed, as_completed([f,f]) will yield f once and
does not trigger an exception.
What is the correct
Glenn Langford added the comment:
There is a subtlety in the as_completed() code which explains a lot - note that
"finished" starts off as a set in the _AcquireFutures block. So if a Future f
has already completed,
as_completed( [f,f] )
will only yield f once, because f appears o
New submission from Glenn Langford:
For a Future f which has already completed,
wait( [f,f], return_when=ALL_COMPLETED )
blocks forever.
This is because the test in wait():
if len(done) == len(fs)
is comparing the length of a set to the length of a list.
If f has not completed, wait
Glenn Langford added the comment:
> - you replaced the _AcquireFutures context manager on all futures with an
> >individual lock. In my opinion, it should be changed in a different patch. I
> >don't know which option is better, but if it is changed, it should be
> cha
Glenn Langford added the comment:
Proposed patch...please treat with an appropriate level of suspicion since this
is my first patch submission. :-)
A corresponding change will be made to as_completed() for #20367. Suggestions
welcome.
--
keywords: +patch
Added file: http
Glenn Langford added the comment:
Proposed patch for as_completed(). #20369 fixes wait(), and behaviour is
consistent between the two.
--
keywords: +patch
Added file: http://bugs.python.org/file33684/issue20367.patch
___
Python tracker
<h
Glenn Langford added the comment:
> Could you please try to write a unit test.
Revised patch with unit test for as_completed().
--
Added file: http://bugs.python.org/file33685/issue20367.patch
___
Python tracker
<http://bugs.python.org/issu
Glenn Langford added the comment:
Thanks for the feedback. The new patch is modified for PEP8 with naming
consistent with other concurrent tests. assertEqual I think is clearer by
checking list length, so it is not changed. The docstring is updated.
I suggest asyncio be handled separately
Glenn Langford added the comment:
Updated patch with a test case, and added a minor note to the docstring to
clarify behaviour.
The use of sleep() in the test is not great, but it is the most obvious way to
test and it is consistent with the approach used in other concurrent test cases
Glenn Langford added the comment:
> It seems more plausible that the locks around the removals are fixing the bug
> but I don't see how. I'll look into it some more.
It is the locks around the waiter removals that matter; I think there are only
formatting changes elsewhere i
Glenn Langford added the comment:
@Brian - Ah, I see now what you are referring to. The patch has changes to
_create_and_install_waiters() which should not be there. The only code that
needs to change is waiter removal as I originally suggested. I am set up with a
dev environment now and will
Glenn Langford added the comment:
This patch shows the minimal desired outcome. It is not elegant in its current
form, but does only what is necessary. Ultimately I think as_completed() should
go its own way and not lock all Futures at once (#20297).
--
Added file: http
Glenn Langford added the comment:
Ah...ok, here is a patch that includes an update to
Doc/library/concurrent.futures.rst as well.
--
Added file: http://bugs.python.org/file33728/issue20367.patch
___
Python tracker
<http://bugs.python.org/issue20
Glenn Langford added the comment:
@Victor: Thank you, and I appreciate all your advice! I am still learning the
dev environment but hope to be helpful. :)
--
___
Python tracker
<http://bugs.python.org/issue20
Glenn Langford added the comment:
Revised patch; I don't think there is a need to sort the keys when waiters are
being removed since only one lock is acquired at a time. Stress tests on both
wait() and as_completed() work with this approach.
--
Added file: http://bugs.pytho
Glenn Langford added the comment:
Updated patch with change to Doc/library/concurrent.futures.rst.
--
Added file: http://bugs.python.org/file33751/issue20369.patch
___
Python tracker
<http://bugs.python.org/issue20
Glenn Langford added the comment:
An idea for a different possible fix - rather than cleaning up waiters in
wait() and as_completed(), could they be removed in Future.set_result() and
Futures.set_exception() ?
I'm not certain if any waiter should ever be notified twice; if not, pe
New submission from Glenn Langford:
The current approach taken in as_completed() and wait() is to immediately lock
the entire set of given Futures simultaneously. This limits concurrency,
particularly when the number of futures is large, and also requires the
complete set of Futures to be
Glenn Langford added the comment:
> if the future is cancelled or finished, control is yielded back to the caller
> with the condition's lock held.
Hmmm...I don't agree. I assume you are looking at this code:
with f._condition: # Lock the Future; yield if completed o
Glenn Langford added the comment:
> Actually, I think what you're seeing here is the context manager being
garbage collected
Yes indeed, I am with you now. Thanks for identifying the problem, and for your
explanation. The future should not be yielded whil
Glenn Langford added the comment:
Updated patch: Fixed bug where quick yield in as_completed() did not release
lock on future.
--
Added file: http://bugs.python.org/file33999/futures-base-v2.patch
___
Python tracker
<http://bugs.python.
New submission from Glenn Langford:
In asyncio, tasks.py as_completed() appears to trigger adding and removing
callbacks multiple times for the pending set of futures, each time a single
future completes.
For example, to wait for 5 futures which complete at different times:
- as_completed
Changes by Glenn Langford :
Added file: http://bugs.python.org/file34000/test_thrash.py
___
Python tracker
<http://bugs.python.org/issue20566>
___
___
Python-bugs-list m
Changes by Glenn Langford :
Added file: http://bugs.python.org/file34001/output.txt
___
Python tracker
<http://bugs.python.org/issue20566>
___
___
Python-bugs-list mailin
Glenn Langford added the comment:
> Glenn, what led you to discover this?
It was found by code inspection. I recently started looking at
concurrent.futures, really just curious as to how futures were implemented.
Because one of the concurrent.futures bugs I raised also applied to asyncio
Glenn Langford added the comment:
> OK, code is ready for review at
> http://code.google.com/p/tulip/source/detail?r=674355412f33
This looks like a link to an old revision.
--
___
Python tracker
<http://bugs.python.org/i
50 matches
Mail list logo
