[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio
Jason Fried added the comment: Yeah I have to agree at this point, from a naive point of view it looks awesome, but its just as bad as loosing gil guarantees. On Mon, Aug 26, 2019 at 03:04 Andrew Svetlov wrote: > > Andrew Svetlov added the comment: > > The solution produces subtle and error-prone code. It can live in third > party library but not good enough for stdlib I think. > > -- > > ___ > Python tracker > <https://bugs.python.org/issue33523> > ___ > -- ___ Python tracker <https://bugs.python.org/issue33523> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38415] @asynccontextmanager decorated functions are not callable like @contextmanager
New submission from Jason Fried : The standard contextmanager decorator produces a wrapper that itself can be used as a decorator ``` @contextmanager def some_context(): ... yield @some_context() def some_function(): # we are inside a with some_context() now. ... ``` When I created a version of asynccontextmanager internally before it was available in the stdLib I copied this behavior and I have people internally to facebook using this behavior, Was there a reason this behavior was not replicated to asynccontextmanager? I have a diff an tests to add -- components: Library (Lib) messages: 354232 nosy: fried, yselivanov priority: normal severity: normal status: open title: @asynccontextmanager decorated functions are not callable like @contextmanager type: enhancement versions: Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue38415> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38415] @asynccontextmanager decorated functions are not callable like @contextmanager
Change by Jason Fried : -- keywords: +patch pull_requests: +16250 stage: -> patch review pull_request: https://github.com/python/cpython/pull/16667 ___ Python tracker <https://bugs.python.org/issue38415> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38857] AsyncMock issue with awaitable return_value/side_effect/wraps
New submission from Jason Fried : If you are trying to use AsyncMock to mock a coroutine that returns awaitable objects, AsyncMock awaits on those objects instead of returning them as is. Example: mock = AsyncMock(return_value=asyncio.Future()) v = await mock() # blocks on trying to await the future Expected: mock = AsyncMock(return_value=asyncio.Future()) v = await mock() assert isisnstance(v, asyncio.Future) This problem affects side_effects and wraps. -- components: Library (Lib) messages: 357000 nosy: fried, lisroach priority: normal severity: normal status: open title: AsyncMock issue with awaitable return_value/side_effect/wraps versions: Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue38857> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38857] AsyncMock issue with awaitable return_value/side_effect/wraps
Change by Jason Fried : -- type: -> behavior ___ Python tracker <https://bugs.python.org/issue38857> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38859] AsyncMock says it raises StopIteration but that is Impossible
New submission from Jason Fried : If an AsyncMock uses a side_effect that is an Iterable, if called more than items exist its suppose to raise StopIteration according to the docs but PEP 479 says that is impossible. My Suggestion is that we update the docs and the code to Raise a StopAsyncIteration since it will not be converted to a RuntimeError -- components: Library (Lib) messages: 357008 nosy: fried, lisroach priority: normal severity: normal status: open title: AsyncMock says it raises StopIteration but that is Impossible type: behavior versions: Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue38859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38859] AsyncMock says it raises StopIteration but that is Impossible
Change by Jason Fried : -- keywords: +patch pull_requests: +16762 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17269 ___ Python tracker <https://bugs.python.org/issue38859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38857] AsyncMock issue with awaitable return_value/side_effect/wraps
Change by Jason Fried : -- keywords: +patch pull_requests: +16761 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17269 ___ Python tracker <https://bugs.python.org/issue38857> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38857] AsyncMock issue with awaitable return_value/side_effect/wraps
Jason Fried added the comment:
Its not possible to have it both ways. Also it stinks too much of trying to
guess.
The root of your issue is you want a normal MagicMock not an AsyncMock. Its the
automatic behavior of patch to pick AsyncMock vs MagicMock that is the heart of
your issue. This bug fix doesn't involve that behavior at all, and AsyncMock
was measurably broken without the fix, in an unavoidable way. Your breakage is
avoidable by changes to how you patch.
with patch("bar", return_value=42)
To still do it the old way you would have to pass new_callable=MagicMock to
patch.
--
___
Python tracker
<https://bugs.python.org/issue38857>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35767] unittest loader doesn't work with partial test functions
Change by Jason Fried : -- stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue35767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35767] unittest loader doesn't work with partial test functions
Change by Jason Fried : -- resolution: -> fixed ___ Python tracker <https://bugs.python.org/issue35767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35767] unittest loader doesn't work with partial test functions
New submission from Jason Fried : https://github.com/python/cpython/blob/3.7/Lib/unittest/loader.py#L232 fullName = '%s.%s' % (testCaseClass.__module__, testFunc.__qualname__) Instead we should probably replace testFunc.__qualname__ with attrname I ran into this while running a test suite that built up test functions using partials and added them to the TestCase class with setattr. This works in 3.6.3 -- messages: 333926 nosy: fried, lisroach, lukasz.langa priority: normal severity: normal status: open title: unittest loader doesn't work with partial test functions versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35767] unittest loader doesn't work with partial test functions
Jason Fried added the comment: Oh this is broken in 3.7 trunk -- ___ Python tracker <https://bugs.python.org/issue35767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35767] unittest loader doesn't work with partial test functions
Jason Fried added the comment: working on a pull request -- ___ Python tracker <https://bugs.python.org/issue35767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35767] unittest loader doesn't work with partial test functions
Change by Jason Fried : -- keywords: +patch, patch pull_requests: +11316, 11317 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35767] unittest loader doesn't work with partial test functions
Change by Jason Fried : -- keywords: +patch pull_requests: +11316 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35767] unittest loader doesn't work with partial test functions
Change by Jason Fried : -- keywords: +patch, patch, patch pull_requests: +11316, 11317, 11318 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30395] deadlocked child process after forking on pystate.c's head_mutex
Changes by Jason Fried : -- pull_requests: +1825 ___ Python tracker <http://bugs.python.org/issue30395> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio
New submission from Jason Fried : At Facebook and Instagram we have large interconnected codebases without clear boundaries of ownership. As we move more and more services to utilize asyncio we are finding that once blocking (but fast) code paths, are now cropping up with asyncio code using run_until_complete(). Now this is fine for all the blocking callers, but some times we have async callers to that blocking code path and now it doesn't work. So we have two options revert the change to not use asyncio deep in the dep tree or Convert all functions in the stack to be asyncio. Both are not possible and engineers have solved them in two crazy ways. 1. Nested Event Loops, when you hit a run_until_complete, create a new eventloop and do the async and return the result. 2. Like the first, but each library creates its own eventloop, and swaps it with the running loop for the duration of the run_until_complete, restoring the original loop when its done. Both of these ultimately have the same problem, everything on the primary event loop stops running until the new loop is complete. What if we could instead start servicing the existing eventloop from the run_until_complete. This would insure that tasks don't timeout. This would allow us to convert things to asyncio faster without having to have absolute knowledge of a codebase and its call graph, and not have to have all engineers completely synchronized. -- components: asyncio messages: 316678 nosy: asvetlov, fried, yselivanov priority: normal severity: normal status: open title: loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue33523> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio
Change by Jason Fried : -- nosy: +lukasz.langa ___ Python tracker <https://bugs.python.org/issue33523> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio
Change by Jason Fried : -- keywords: +patch pull_requests: +6540 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue33523> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33544] Asyncio Event.wait() and Condition.wait() is a hold over from before awaitable, and should be awaitable
New submission from Jason Fried : wait is a very overloaded word in asyncio. Events and Conditions are not consistent with the rest of asyncio. Why don't Future and Task have wait() methods? well because they are awaitable Some subjective reasoning: Every time I go to use one of these things, I attempt to await them like everything else in the Asyncio world and get a nice exception for it. await event vs await event.wait() I propose we make conditions and events awaitable and deprecate the .wait or at-least remove it from the documentation. -- components: asyncio messages: 316842 nosy: asvetlov, fried, lukasz.langa, yselivanov priority: normal severity: normal status: open title: Asyncio Event.wait() and Condition.wait() is a hold over from before awaitable, and should be awaitable type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue33544> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33544] Asyncio Event.wait() is a hold over from before awaitable, and should be awaitable
Jason Fried added the comment: Removed Condition from this request, because it has an __await__ method for supporting the the deprecated pattern with async cond: I'll open a different bug, for Condition behavior for 3.9 when we can remove the deprecated pattern. -- title: Asyncio Event.wait() and Condition.wait() is a hold over from before awaitable, and should be awaitable -> Asyncio Event.wait() is a hold over from before awaitable, and should be awaitable ___ Python tracker <https://bugs.python.org/issue33544> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33544] Asyncio Event.wait() is a hold over from before awaitable, and should be awaitable
Change by Jason Fried : -- keywords: +patch pull_requests: +6584 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue33544> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33546] asyncio.Condition should become awaitable in 3.9
New submission from Jason Fried : In 3.9 we can remove the deprecated pattern for accepting __enter__ and __exit__ for locks. This will free up __await__ for Condition to use for replacing .wait() which is wart from before awaitables. My new proposed behavior is await cond which would be equivalent of: async with cond: await cond.wait() -- components: asyncio messages: 316848 nosy: asvetlov, fried, lukasz.langa, yselivanov priority: normal severity: normal status: open title: asyncio.Condition should become awaitable in 3.9 ___ Python tracker <https://bugs.python.org/issue33546> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio
Jason Fried added the comment: For loops not supporting this throwing NotImplmentedError from the method to enable reentrancy seems appropriate. "You should convert all your stack to async functions..." That may not be practical for large code bases in transition to asyncio. The fixes for reentrancy that I find in reality are not adding async logic through out the call stack but instead its one of the two I listed. -- ___ Python tracker <https://bugs.python.org/issue33523> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33864] collections.abc.ByteString does not register memoryview
New submission from Jason Fried : Looking at the typing Module docs in the section for ByteString This type represents the types bytes, bytearray, and memoryview. But collections.abc.ByteString does not have memoryview registered. Is it because memoryview doesn't support .index()? -- assignee: docs@python components: Documentation messages: 319557 nosy: docs@python, fried, lukasz.langa priority: normal severity: normal status: open title: collections.abc.ByteString does not register memoryview type: behavior versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue33864> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27194] Tarfile superfluous truncate calls slows extraction.
New submission from Jason Fried: With large tar file extracts I noticed that tarfile was slower than it should be. Seems in linux that for large files (10MB) truncate is not always a free operation even when it should be a no-op. ex: File is already 10mb seek to end and truncate. I created a script to test the validity of this patch. It generates two random tar archives containing 1024 files of 10mb each. The files are randomized so disk caching should not interfere. So to extract those 1g tar files the following was observed Time Delta for TarFile: 148.23699307441711 Time Delta for FastTarFile: 107.71058106422424 Time Diff: 40.52641201019287 0.27338932859929255 -- components: Library (Lib) files: truncate.patch keywords: patch messages: 267035 nosy: asvetlov, fried, lukasz.langa priority: normal severity: normal status: open title: Tarfile superfluous truncate calls slows extraction. type: performance versions: Python 3.5 Added file: http://bugs.python.org/file43138/truncate.patch ___ Python tracker <http://bugs.python.org/issue27194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27194] Tarfile superfluous truncate calls slows extraction.
Jason Fried added the comment: I ran this on Linux ext4. I didn't see much improvement on OSX with SSD. -- Added file: http://bugs.python.org/file43139/test.py ___ Python tracker <http://bugs.python.org/issue27194> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27199] TarFile expose copyfileobj bufsize to improve throughput
New submission from Jason Fried: The default of 16k while good for memory usage it is not well suited for all cases. if we increased this to 4MB we saw a pretty large improvement to tar file creation and extraction on linux servers. For a 1gb tar file containing 1024 random files each of 10MB in size. Time Delta for TarFile: 146.3240258693695 Time Delta for FastTarFile 4MB copybufsize: 102.76440262794495 Time Diff: 43.55962324142456 0.2976928975444698 -- components: Library (Lib) files: buftest.py messages: 267134 nosy: asvetlov, fried, lukasz.langa priority: normal severity: normal status: open title: TarFile expose copyfileobj bufsize to improve throughput type: performance versions: Python 3.5 Added file: http://bugs.python.org/file43158/buftest.py ___ Python tracker <http://bugs.python.org/issue27199> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27199] TarFile expose copyfileobj bufsize to improve throughput
Changes by Jason Fried : -- keywords: +patch Added file: http://bugs.python.org/file43159/copybufsize.patch ___ Python tracker <http://bugs.python.org/issue27199> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
