[issue26203] nesting venv in virtualenv segfaults
New submission from André Caron: When trying to create a new virtual environment using Python 3.5's venv package from a virtual environment created by the popular 3rd-party virtualenv package, I get a segfault. Nested virtual environments work fine both with venv and virtualenv, but using one inside the other doesn't work. This suggests a subtle incompatibility between the two implementations. I'm not sure whether compatibility between the implementations is a goal or not at the moment, but there is extensive tooling that uses the 3rd party virtualenv implementation. For example, I run tests with Tox, which uses virtualenv. I cannot test any package that uses the standard venv because everything crashes. As a result of this, I will avoid moving to the standard implementation. I put up some sample scripts up on GitHub to show how to reproduce the issue. I hope they can help explain the issue. https://github.com/AndreLouisCaron/nested-venv-bug Cheers, André -- components: Library (Lib) messages: 258923 nosy: André Caron priority: normal severity: normal status: open title: nesting venv in virtualenv segfaults type: crash versions: Python 3.5 ___ Python tracker <http://bugs.python.org/issue26203> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25833] pyvenv: venvs cannot be moved because activate scripts hard-code paths
Changes by André Caron : -- nosy: +André Caron ___ Python tracker <http://bugs.python.org/issue25833> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25887] awaiting on coroutine more than once should be an error
André Caron added the comment:
Hi there!
I've just stumbled upon this behavior and I was also surprised by the fact that
the second await simply returns None.
After fiddling around for a while, I noticed that if I wrap the coroutine
object using asyncio.ensure_future() or asyncio.get_event_loop().create_task(),
the same result/exception is returned by multiple await expressions.
I haven't looked at the patch, but the intent to make the 2nd await raise a
RuntimeError seems strange for several reasons:
- it's inconsistent with the Future/Task interface;
- it's quite common to await a 2nd time to get the coroutine result after
calling asyncio.wait(...) using ALL_COMPLETED or FIRST_EXCEPTION;
- as mentioned in the mailing list the await keyword in C#/Hack/JS which
inspired the await keyword (as per PEP492) returns the result/exception
multiple times.
I put up a Gist that shows the inconsistency:
https://gist.github.com/AndreLouisCaron/db2965aae095f5c85dd5
Here's an example of asyncio.wait() I was referencing:
async def main()
f1 = foo()
f2 = bar()
asyncio.wait([f1, f2], return_when=asyncio.FIRST_EXCEPTION)
print('1:', await f1)
print('2:', await f2)
I also noticed that there seems to be some intent to avoid making a distinction
between a normal function returning a future and a coroutine function from the
point of view of the caller.
If the patch is merged as is, I will always need to use asyncio.ensure_future()
on all coroutine calls before asyncio.wait() because the result is inconsistent
depending on the implementation of foo() and bar(): if they return futures, I'm
OK, but if any of them is a proper coroutine function, I might get RuntimeError
exceptions.
Any chance you can consider changing the patch to make awaiting a coroutine's
result multiple times a valid pattern?
--
nosy: +André Caron
___
Python tracker
<http://bugs.python.org/issue25887>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25887] awaiting on coroutine more than once should be an error
André Caron added the comment: > I believe you're not using the asyncio.task() function correctly. I assume you meant asyncio.wait(). I just updated my gist with a variant of my example that uses the (done, pending) pair returned by asyncio.wait() as you suggested. The set of done futures that is returned does not help in this case because you need to test objects by set membership using something like "if f1 in done:", which will never be true because f1 is a coroutine object and the set contains a wrapper returned by asyncio.ensure_future(). So we're back to square 1. Because you cannot await multiple times on a coroutine object (in contrast to futures), you will need to explicitly call asyncio.ensure_future() on everything you pass to asyncio.wait() if you care about the future's result -- which is more often than not in my experience. > There are many reasons as to why we don't want coroutines to be awaitable many times. One of them is that we don't want low-level coroutine objects to hold references to return values. Coroutines in Python are exhaustible resources (like generators). I understand concerns for efficiency and management of scarce resources. However, I don't understand your comment. Maybe this has something to do with CPython/asyncio internals, which I know nothing about. If my code keeps a reference to the coroutine object, it can't be released right? I don't see how adding a reference to the return value or wrapping the coroutine object in a Task wrapper (in order to recover the return value) will affect the allocation of coroutine resources. > Well, coroutines are much more lower level than Future/Tasks. >From an implementer's point of view, maybe. From a user's point of vue, the >value of having an "awaitable" concept is that I can use different objects in >the same way. If the semantics of await expressions vary depending on the >awaitable's type, then the value of the abstraction is reduced. If we go back to the core of the issue, the real problem here is the fact that the current behaviour of multiple awaits on a coroutine object is surprising: nobody expects it to return None the 2nd time around. Raising a RuntimeError instead of returning None is still surprising behaviour IMO. In addition to that, the error message that was agreed upon in this issue suggests a programming error (you're not _supposed_ to await twice), but what I'm getting to here with my example is that asyncio.wait() is a common/legitimate use case of this. Anyways, if you insist on considering this a programming error, you should probably reject coroutine objects in calls to asyncio.wait() since there will be effectively no way of recovering the return value after the call to asyncio.wait(). -- ___ Python tracker <http://bugs.python.org/issue25887> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26357] asyncio.wait loses coroutine return value
New submission from André Caron: When the asyncio.wait() function with coroutine objects as inputs, it is impossbible to extract the results reliably. This issue arises because asyncio.wait() returns an unordered set of futures against which membership tests of coroutine objects always fail. The issue is made even worse by the fact that coroutine objects cannot be awaited multiple times (see https://bugs.python.org/issue25887). Any await expression on the coroutine object after the call to asyncio.wait() returns None, regardless of the coroutine's return value. (See attached `asyncio-wait-coroutine.py` for an example of both these issues.) In the worst case, multiple inputs are coroutine objects and the set of returned futures contains return values but there is no way to determine which result corresponds to which coroutine call. To work around this issue, callers need to explicitly use asyncio.ensure_future() on coroutine objects before calling asyncio.wait(). (See comment in `asyncio-wait-coroutine.py` for an example "fix"). Note that, in general, it is not possible to know ahead of time whether all inputs to asyncio.wait() are coroutines or futures. Furthermore, the fact that a given third-party library function is implemented as a regular function that returns a Future or a proper coroutine is an implementation decision which may not be part of the public interface. Even if it is, the inputs to asyncio.wait() may come from complex code paths and it may be difficult to verify that all of them end up producing a Future. As a consequence, the only reliable way to recover all results from asyncio.wait() is to explicitly call asyncio.ensure_future() on each of the inputs. When doing so, both the membership test against the `done` set and the await expressions work as expected. Quickly, there are several possible solutions: - allow programs to await coroutine multiple times; - make the set membership test of a coroutine object succeed; or - change support for coroutine objects as inputs to asyncio.wait(): ** update documentation for asyncio.wait() to explain this limitation; or ** explicitly reject coroutine objects; or ** warn when passing coroutine objects as inputs -- unless wrapped. Related issue: https://bugs.python.org/issue25887 proposes a patch to change the behaviour of awaiting a coroutine object multiple times in order to produce a RuntimeError on all awaits after the 1st. While that change in behaviour would make it easier to diagnose the loss of the return value, it does not fix this issue. -- components: asyncio files: asyncio-wait-coroutine.py messages: 260250 nosy: André Caron, gvanrossum, haypo, yselivanov priority: normal severity: normal status: open title: asyncio.wait loses coroutine return value versions: Python 3.5 Added file: http://bugs.python.org/file41914/asyncio-wait-coroutine.py ___ Python tracker <http://bugs.python.org/issue26357> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25887] awaiting on coroutine more than once should be an error
André Caron added the comment: After thinking about this some more, I think my problem with asyncio.wait() is a bit bigger than the simple fact that coroutine objects cannot be awaited multiple times. It seems to me like asyncio.wait() is completely broken for coroutine objects as inputs and that multiple awaits on coroutine objects only make that problem worse. While I still think there is a major benefit API-wise to have await expressions produce the same behaviour for both Futures and coroutine objects, I'm moving that discussion to issue #26357. http://bugs.python.org/issue26357 -- ___ Python tracker <http://bugs.python.org/issue25887> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26357] asyncio.wait loses coroutine return value
André Caron added the comment: Hi Guido, Thanks for the quick reply :-) AFAICT, there seem to be three possible directions regarding this issue -- for both wait() and as_completed(): 1) remove the need for ensure_future(): make the membership test succeed and allow multiple await expressions on the same coroutine; 2) fail fast: reject coroutine objects as inputs to wait() and reject multiple await expressions on coroutine objects (see issue #25887); 3) clarify API usage: deprecate couroutine objects and enhance docs for wait(), as_completed() and ensure_future(). >From a pure API standpoint, #1 makes the API more uniform and less surprising, >#2 makes it easier to detect and fix incorrect usage and #3 accelerates >troubleshooting when people get bitten by this quirk in the API. You're right that technically, the case of side-effect-only coroutine invocations is a use case that's currently supported by wait() and that removing this (e.g. by adopting #2) would be a compatibility break. I personally doubt that this is a common use case as both wait() and as_completed() seem specifically designed to recover the results, but I'm not an asyncio expert either. Asyncio is still young and is undergoing adoption, so I would like to see this issue resolved without resorting to #3. Any chance #1 or #2 can be considered? -- ___ Python tracker <http://bugs.python.org/issue26357> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
