Brian Quinlan added the comment:
I'll try to take a look at this before the end of the week, but I'm currently
swamped with other life stuff :-(
--
___
Python tracker
<https://bugs.python.o
Brian Quinlan added the comment:
New changeset 884eb89d4a5cc8e023deaa65001dfa74a436694c by Brian Quinlan in
branch 'master':
bpo-39205: Tests that highlight a hang on ProcessPoolExecutor shutdown (#18221)
https://github.com/python/cpython/commit/884eb89d4a5cc8e023deaa65001dfa
Change by Brian Quinlan :
--
keywords: +patch
pull_requests: +17601
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/18221
___
Python tracker
<https://bugs.python.org/issu
New submission from Brian Quinlan :
```
from concurrent.futures import ProcessPoolExecutor
import time
t = ProcessPoolExecutor(max_workers=3)
t.map(time.sleep, [1,2,3])
t.shutdown(wait=False)
```
Results in this exception and then a hang (i.e. Python doesn't terminate):
```
Excepti
Brian Quinlan added the comment:
Can I add "needs backport to 3.8" and "needs backport to 3.7" labels now or
do I have to use cherry_picker at this point?
On Mon, Jul 1, 2019 at 3:55 PM Ned Deily wrote:
>
> Ned Deily added the comment:
>
> > I don't
Brian Quinlan added the comment:
I don't know what the backport policy is. The bug is only theoretical AFAIK
i.e. someone noticed it through code observation but it has not appeared in
the wild.
On Mon, Jul 1, 2019 at 3:25 PM Ned Deily wrote:
>
> Ned Deily added the comment
Change by Brian Quinlan :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Brian Quinlan added the comment:
New changeset 242c26f53edb965e9808dd918089e664c0223407 by Brian Quinlan in
branch 'master':
bpo-31783: Fix a race condition creating workers during shutdown (#13171)
https://github.com/python/cpython/commit/242c26f53edb965e9808dd918089e6
Change by Brian Quinlan :
--
resolution: -> duplicate
stage: -> resolved
status: open -> closed
superseder: -> function changed when pickle bound method object
___
Python tracker
<https://bugs.python
Change by Brian Quinlan :
--
stage: -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue37208>
___
___
Python-bugs-list
Change by Brian Quinlan :
--
resolution: -> duplicate
superseder: -> picke cannot dump Exception subclasses with different super()
args
___
Python tracker
<https://bugs.python.org/i
Change by Brian Quinlan :
--
title: picke cannot dump exceptions subclasses with different super() args ->
picke cannot dump Exception subclasses with different super() args
___
Python tracker
<https://bugs.python.org/issu
New submission from Brian Quinlan :
$ ./python.exe nopickle.py
TypeError: __init__() missing 1 required positional argument: 'num'
The issue is that the arguments passed to Exception.__init__ (via `super()`)
are collected into `args` and then serialized by pickle e.g.
>>&
Brian Quinlan added the comment:
That's a super interesting bug! It looks like this issue is that your exception
can't be round-tripped using pickle i.e.
>>> class PoolBreaker(Exception):
... def __init__(self, num):
... super().__init__()
...
Brian Quinlan added the comment:
Joshua, I'm closing this since I haven't heard from you in a month. Please
re-open if you use case isn't handled by `initializer` and `initargs`.
--
assignee: -> bquinlan
resolution: -> out of date
stage: -> resolve
Brian Quinlan added the comment:
We can bike shed over the name in the PR ;-)
--
___
Python tracker
<https://bugs.python.org/issue36780>
___
___
Python-bug
Change by Brian Quinlan :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Brian Quinlan added the comment:
My understanding is that tracebacks have a pretty large memory profile so I'd
rather not keep them alive. Correct me if I'm wrong about that.
--
___
Python tracker
<https://bugs.python.o
Brian Quinlan added the comment:
After playing with it for a while, https://github.com/python/cpython/pull/6375
seems reasonable to me.
It needs tests and some documentation.
Antoine, are you still -1 because of the complexity increase
Brian Quinlan added the comment:
When I first wrote and started using ThreadPoolExecutor, I had a lot of code
like this:
with ThreadPoolExecutor(max_workers=500) as e:
e.map(download, images)
I didn't expect that `images` would be a large list but, if it was, I wanted
all o
Brian Quinlan added the comment:
Was this fixed by https://github.com/python/cpython/pull/3895 ?
--
___
Python tracker
<https://bugs.python.org/issue30
Brian Quinlan added the comment:
Brian, I was looking for an example where the current executor isn't sufficient
for testing i.e. a useful test that would be difficult to write with a real
executor but would be easier with a fake.
Maybe you have such an example from your
Change by Brian Quinlan :
--
pull_requests: +13087
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue31783>
___
___
Python-bugs-list mai
Brian Quinlan added the comment:
I think that ProcessPoolExecutor might have a similar race condition - but not
in exactly this code path since it would only be with the queue management
thread (which is only started once).
--
___
Python tracker
Brian Quinlan added the comment:
Great report Steven!
I was able to reproduce this with the attached patch (just adds some sleeps and
prints) and this script:
from threading import current_thread
from concurrent.futures import ThreadPoolExecutor
from time import sleep
pool
Change by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<https://bugs.python.org/issue31783>
___
___
Python-bugs-list mailing list
Unsubscrib
Brian Quinlan added the comment:
How did the experiment go? Are people still interested in this?
--
___
Python tracker
<https://bugs.python.org/issue22
Brian Quinlan added the comment:
Ben, do you still think that your patch is relevant or shall we close this bug?
--
___
Python tracker
<https://bugs.python.org/issue22
Brian Quinlan added the comment:
Do the `initializer` and `initargs` parameters deal with this use case for you?
https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
--
___
Python tracker
<ht
Brian Quinlan added the comment:
If we supported this, aren't we promising that we will always materialize the
iterator passed to map?
I think that we'd need a really strong use-case for this to be worth-while.
--
___
Python track
Brian Quinlan added the comment:
Hey Brian,
I understand the non-determinism. I was wondering if you had a non-theoretical
example i.e. some case where the non-determinism had impacted a real test that
you wrote?
--
___
Python tracker
<ht
Brian Quinlan added the comment:
Hey Hrvoje,
I agree that #1 is the correct approach. `disown` might not be the best name -
maybe `allow_shutdown` or something. But we can bike shed about that later.
Are you interested in writing a patch
Brian Quinlan added the comment:
Hey Ethan, I'm really sorry about dropping the ball on this. I've been burnt
out on Python stuff for the last couple of years.
When we left this, it looked like the -1s were in the majority and no one new
has jumped on to support `filter`.
If you
Brian Quinlan added the comment:
Using a default executor could be dangerous because it could lead to deadlocks.
For example:
mylib.py
def my_func():
tsubmit(...)
tsubmit(...)
tsubmit(somelib.some_func, ...)
somelib.py
--
def some_func():
tsubmit(...) # Potential
Brian Quinlan added the comment:
So you actually use the result of ex.submit i.e. use the resulting future?
If you don't then it might be easier to just create your own thread.
--
___
Python tracker
<https://bugs.python.org/is
Brian Quinlan added the comment:
Do you have a example that you could share?
I can't think of any other fakes in the standard library and I'm hesitant to be
the person who adds the first one ;-)
--
___
Python tracker
<https://bu
Change by Brian Quinlan :
--
stage: -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue26374>
___
___
Python-bugs-list
Brian Quinlan added the comment:
Can we close this bug then?
--
___
Python tracker
<https://bugs.python.org/issue26374>
___
___
Python-bugs-list mailin
Brian Quinlan added the comment:
Hey Brian, why can't you use threads in your unit tests? Are you worried about
non-determinism or resource usage? Could you make a ThreadPoolExecutor with a
single worker?
--
___
Python tracker
&
Change by Brian Quinlan :
--
keywords: +patch
pull_requests: +13045
stage: needs patch -> patch review
___
Python tracker
<https://bugs.python.org/issu
Brian Quinlan added the comment:
OK, I completely disagree with my statement:
"""If you added this as an argument to shutdown() then you'd probably also have
to add it as an option to the constructors (for people using Executors as
context managers). But, if you h
Brian Quinlan added the comment:
BTW, the 61 process limit comes from:
63 - -
--
___
Python tracker
<https://bugs.python.org/issue26903>
___
___
Python-bug
Change by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<https://bugs.python.org/issue26903>
___
___
Python-bugs-list mailing list
Unsubscrib
Brian Quinlan added the comment:
If no one has short-term plans to improve multiprocessing.connection.wait, then
I'll update the docs to list this limitation, ensure that ProcessPoolExecutor
never defaults to >60 processes on windows and raises a ValueError if the user
explicitly
Brian Quinlan added the comment:
>> The current behavior is explicitly documented, so presumably
>> it can't be (easily) changed
And it isn't clear that it would be desirable to change this even if it were
possible - doing structured resource clean-up seems consi
Brian Quinlan added the comment:
Actually it immediately converts the iterable into a list. Recall:
def filter(self, fn, iterable, timeout=None):
l = list(iterable) # iterable => list
return (item for (item, keep) in zip(l, self.map(fn, l, timeout)) if k
Brian Quinlan added the comment:
Ethan: I'm -0 so I'd happily go with the consensus. Does anyone have a strong
opinion?
Ram: What did you mean by "Possible issues"? Did you mean that to be an example
using the exec
Brian Quinlan added the comment:
This feature seems unnecessary to me but couldn't the implementation be
simplified to work in terms of map? i.e. (pseudocode):
def filter(self, fn, iterable, timeout=None):
l = list(iterable)
return (item for (item, keep) in zip(l, self.map(fn, l, ti
Brian Quinlan added the comment:
Committed, thanks for the fix!
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/i
Brian Quinlan added the comment:
The patch looks good to me too. I'll commit it soon.
--
assignee: -> bquinlan
___
Python tracker
<http://bugs.python.org
Brian Quinlan added the comment:
Oops, no. That was junk due to my sloppiness. I’ll fix it in a minute.
On Jan 31, 2014, at 5:03 PM, STINNER Victor wrote:
>
> STINNER Victor added the comment:
>
>> New changeset 0bcf23a52d55 by Brian Quinlan in branch 'defau
Brian Quinlan added the comment:
Thanks very much for the patch Glenn!
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue20319>
___
_
Changes by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<http://bugs.python.org/issue20319>
___
___
Python-bugs-list mailing list
Unsubscrib
Brian Quinlan added the comment:
I'm looking at "futures.patch".
I don't understand why these blocks are helpful -"_create_and_install_waiters"
has two call sites and both ("as_completed" and "wait") call
"_create_and_install_
Brian Quinlan added the comment:
I'm not currently working on concurrent.futures but I do look at patches and
bug reports. I'll take a look at this and Issue20297 sometime this week.
--
___
Python tracker
<http://bugs.python.o
Brian Quinlan added the comment:
Can't you accomplish what you want using add_done_callback?
e.g.
# Pseudocode
class MyExecutor(ThreadPoolExecutor):
def __init__(self):
self._count = 0
def _decrement(self):
with self._some_lock:
self._count -= 1
def submit(self, fn,
Brian Quinlan added the comment:
Hi Victor,
I don't understand your problem. Could you be very specific in your description?
--
___
Python tracker
<http://bugs.python.org/is
Brian Quinlan added the comment:
Take a look at http://bugs.python.org/issue11161 and its fix. Do you think that
just saying that ProcessPoolExecutor will not work from the interactive shell
is sufficient?
--
nosy: +bquinlan
___
Python tracker
Brian Quinlan added the comment:
The only thing that I've used .running() for is for UI reasons. But it is
easier to just iterative over your futures and call a method than to install a
callback and handle the state yourself.
--
nosy: +bqu
Brian Quinlan added the comment:
Any progress on this or can I close the bug?
--
___
Python tracker
<http://bugs.python.org/issue13785>
___
___
Python-bugs-list m
Changes by Brian Quinlan :
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue11161>
___
___
Python-bugs-list
Brian Quinlan added the comment:
I think that we are happy to not fix this.
--
resolution: accepted -> wont fix
status: open -> closed
___
Python tracker
<http://bugs.python.org/i
Brian Quinlan added the comment:
OK, working as intended.
--
resolution: -> invalid
___
Python tracker
<http://bugs.python.org/issue7200>
___
___
Python-
Changes by Brian Quinlan :
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue11271>
___
___
Python-bugs-list mailing list
Unsubscri
Brian Quinlan added the comment:
I'm closing this since the filer hasn't specified exactly what they want.
--
status: open -> closed
___
Python tracker
<http://bugs.pyth
Changes by Brian Quinlan :
--
stage: -> needs patch
___
Python tracker
<http://bugs.python.org/issue13785>
___
___
Python-bugs-list mailing list
Unsubscri
Brian Quinlan added the comment:
This has come up before. Did you actually bang into this? Because the fix seems
pretty ugly to me and the work-around (using functools.partial) is pretty easy.
But, if people are actually hitting this, then your is probably the best that
we can do
Brian Quinlan added the comment:
The concurrent.futures stuff looks good to me.
Could you add a comment explaining why the delete is necessary? And, as Antoine
said, the test should be CPython only.
--
___
Python tracker
<http://bugs.python.
Brian Quinlan added the comment:
My current thinking is that adding a "state" property probably isn't worth it
given the fact that you can construct this yourself and the requirement for a
"history" property is too specialized.
The callback idea seams reasonable an
Brian Quinlan added the comment:
I've had people request that they be able control the order of processed work
submissions. So a more general way to solve your problem might be to make the
two executors take an optional Queue argument in their constructors.
You'd have to explain in
Brian Quinlan added the comment:
The queue that you identified i.e.
self._call_queue = multiprocessing.Queue(self._max_workers +
EXTRA_QUEUED_CALLS)
does not get considered during submit() - are you sure that it somehow causes
submit() to block
Brian Quinlan added the comment:
Thanks for the patch!
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/i
Brian Quinlan added the comment:
Hey Nam,
I'm not sure that I understand. You want ThreadPoolExecutor.submit to block if
there are too many work items in the queue? Are you sure that this happens
currently with ProcessPoolExecutor? I can't see wh
Changes by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<http://bugs.python.org/issue15015>
___
___
Python-bugs-list mailing list
Unsubscri
Brian Quinlan added the comment:
I guess the question is: why do you need to know the state in that form?
--
___
Python tracker
<http://bugs.python.org/issue13
Changes by Brian Quinlan :
--
nosy: +bquinlan
___
Python tracker
<http://bugs.python.org/issue14148>
___
___
Python-bugs-list mailing list
Unsubscribe:
Brian Quinlan added the comment:
I'm closing this since tbrink didn't respond to pitrou's comments.
--
resolution: -> out of date
___
Python tracker
<http://bugs.
Changes by Brian Quinlan :
--
assignee: -> bquinlan
nosy: +bquinlan
___
Python tracker
<http://bugs.python.org/issue14119>
___
___
Python-bugs-list mai
Brian Quinlan added the comment:
Could you run just the test_concurrent_futures test, hit ctrl-C at the point
where it hangs, and send the traceback here?
--
___
Python tracker
<http://bugs.python.org/issue14
Brian Quinlan added the comment:
Thanks for the patch!
1. The fetching the state feature seems reasonable but I think that explaining
the difference between CANCELLED and CANCELLED_AND_NOTIFIED is going to be
hard. Maybe you could look at how Doc/library/concurrent.futures.rst would need
to
Brian Quinlan added the comment:
You'll probably get more traction if you file a new bug.
--
___
Python tracker
<http://bugs.python.org/issue5788>
___
___
Brian Quinlan added the comment:
On my crappy computer, ProcessPoolExecutor.map adds <3ms of added execution
time per call using your test framework.
What is your use case where that is too much?
That being said, I don't have any objections to making improvements.
If you want to pur
Changes by Brian Quinlan :
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue11864>
___
___
Python-bugs-list
Brian Quinlan added the comment:
> Killed by the user, or by an automatic device (such as the Linux OOM
> killer), or crashed.
Crashed would be bad - it would indicate a bug in the
ProcessPoolExecutor code.
>
>> If the user kills a child then maybe all we want to do is raise
Brian Quinlan added the comment:
Under what circumstances do we expect a ProcessPoolExecutor child process to be
killed outside of the control of the ProcessPoolExecutor?
If the user kills a child then maybe all we want to do is raise an exception
rather than deadlock as a convenience
Brian Quinlan added the comment:
It looks like the timing is too sensitive.
--
assignee: -> bquinlan
___
Python tracker
<http://bugs.python.org/issu
Changes by Brian Quinlan :
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue11777>
___
___
Python-bugs-list
Brian Quinlan added the comment:
Nice catch. I hadn't noticed that the docs are lying :-)
--
___
Python tracker
<http://bugs.python.org/issue11777>
___
___
Brian Quinlan added the comment:
I think that it surprising behavior, especially considering that asking for the
*first* element in the iterator causes *all* of the futures to be created.
--
___
Python tracker
<http://bugs.python.org/issue11
New submission from Brian Quinlan :
from concurrent import futures
with futures.ThreadPoolExecutor(max_workers=5) as e:
e.map(print, range(10))
# No output
--
assignee: bquinlan
components: Library (Lib)
messages: 133104
nosy: bquinlan
priority: normal
severity: normal
status: open
Brian Quinlan added the comment:
Filing a new bug might have been a mistake. Once the investigation in issue
10517 isolated the failure as being in a different module, I thought it best to
file a new bug with a minimal repro case.
Fill free to cleanup
Brian Quinlan added the comment:
This is the expected behavior.
Future.cancel() returns a bool indicating if the future was cancelled or not.
If it returns False then there is no way to stop it.
The behavior that you are seeing for shutdown is documented as:
"""
shutdown(wai
Brian Quinlan added the comment:
No, I wasn't able to replicate.
--
___
Python tracker
<http://bugs.python.org/issue10632>
___
___
Python-bugs-list m
Brian Quinlan added the comment:
Sorry, I didn't read an error message very carefully. When I apply your patch I
see:
>>> from concurrent.futures import *
>>> from time import *
>>> t = ThreadPoolExecutor(5)
>>> t.submit(sleep, 100)
>>>
Erro
Brian Quinlan added the comment:
Your approach seems workable but your patch allows the interpreter to exit
while work items are still being processed. See the comment at the top of
concurrent/futures/thread.py.
--
___
Python tracker
<h
Brian Quinlan added the comment:
I would suggest that you base your patch on 3.3/default.
--
___
Python tracker
<http://bugs.python.org/issue11635>
___
___
Pytho
Brian Quinlan added the comment:
ProcessPoolExecutor is not expected to work with any interactive shell on
Windows because it uses the multiprocessing module behind the scenes, which has
that limitation.
I'm reclassifying this as a documentation bug since either a reference to
Brian Quinlan added the comment:
Sorry for taking so long to reply - I was on holidays until today.
This is an incompatible API change (since people may be providing "fn" by
keyword) so we should probably hold off until 3.3.
I also don't really like that the signature for sub
Brian Quinlan added the comment:
Good point! I'd suggest functools.partial.
--
___
Python tracker
<http://bugs.python.org/issue10918>
___
___
Python-bugs-l
Brian Quinlan added the comment:
Arian,
This seems like such an unimportant edge case that I don't want to mess with
the API just to accommodate it.
If you really need to pass an "fn" keyword argument, use:
.submit(foo, 1, 2, **{'fn': bar})
--
resolution:
1 - 100 of 152 matches
Mail list logo
