[issue38248] inconsistency in asyncio.Task between cancellation while running vs. cancellation immediately after it finishes

[Nathaniel Smith]

New submission from Nathaniel Smith :

Just noticed this while looking at the code to asyncio.Task.__step

asyncio Futures have two different error states: they can have an arbitrary 
exception set, or they can be marked as cancelled.

asyncio Tasks handle this by detecting what exception was raised by the task 
code: if it's a CancelledError, then the mark the Future as cancelled, and if 
it's anything else, they set that as the Future's exception.

There is also a special case handled inside the 'except StopIteration' clause 
in Task.__step. If we request that a Task be cancelled, but then the task exits 
normally before we have a chance to throw in a CancelledError, then we also 
want mark the Future as cancelled. BUT, this special case is handled 
incorrectly: it does Future.set_exception(CancelledError()), instead of 
Future.cancel(). Normally it's impossible for a task to end up with its 
exception set to CancelledError, but it does happen in this one weird edge 
case, basically as a race condition.

Here's some sample code to illustrate the problem (tested on 3.7):

--

import asyncio

# This gets cancelled normally
async def cancel_early():
asyncio.current_task().cancel()
await asyncio.sleep(1)

async def cancel_late():
asyncio.current_task().cancel()
# No sleep, so CancelledError doesn't get delivered until after the
# task exits

async def main():
t_early = asyncio.create_task(cancel_early())
await asyncio.sleep(0.1)
print(f"t_early.cancelled(): {t_early.cancelled()!r}")

t_late = asyncio.create_task(cancel_late())
await asyncio.sleep(0.1)
print(f"t_late.cancelled(): {t_late.cancelled()!r}")
print(f"t_late.exception(): {t_late.exception()!r}")

asyncio.run(main())


--

Output:

t_early.cancelled(): True
t_late.cancelled(): False
t_late.exception(): CancelledError()

The obvious fix would be to modify the 'except StopIteration' branch to handle 
this case by calling super().cancel() instead of super().set_exception(...).

Alternatively, I could see an argument that asyncio.Task should always preserve 
the CancelledError, so that e.g. you can get a proper traceback. In that case 
we'd need to change the 'except CancelledError' branch to call 
super().set_exception(...) instead of super().cancel(). This would also need 
some more changes, like overriding .cancelled() to check for a stored exception 
and then return isinstance(stored_exc, CancelledError), and maybe others... I'm 
not sure of the full consequences.

But handling these two cases differently is definitely wrong, that part I'm 
sure of :-)

--
messages: 352964
nosy: asvetlov, njs, yselivanov
priority: normal
severity: normal
status: open
title: inconsistency in asyncio.Task between cancellation while running vs. 
cancellation immediately after it finishes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38249] Optimize out Py_UNREACHABLE in the release mode

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

Py_UNREACHABLE is used to indicate that a specific point in the program cannot 
be reached, even if the compiler might otherwise think it can. This is exact 
the case for __builtin_unreachable in GCC and Clang. I propose to extend 
Py_UNREACHABLE() to __builtin_unreachable() in the release mode. This will 
allow the compiler to generate more efficient code.

If there are circumstances in which Py_UNREACHABLE() is reachable, then it is 
improper use of Py_UNREACHABLE(). It should be replaced with raising an 
appropriate exception (like TypeError, ValueError, RuntimeError or SystemError) 
or, in extreme cases, with explicit Py_FatalError()

--
components: Interpreter Core
messages: 352965
nosy: barry, serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: Optimize out Py_UNREACHABLE in the release mode
type: performance
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38249] Optimize out Py_UNREACHABLE in the release mode

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +15905
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/16329

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38209] Simplify dataclasses.InitVar by using __class_getitem__()

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset b4d0b39a9b4cd203bcc5b236dc96456e9658119a by Serhiy Storchaka in 
branch 'master':
bpo-38209: Simplify dataclasses.InitVar by using __class_getitem__(). (GH-16255)
https://github.com/python/cpython/commit/b4d0b39a9b4cd203bcc5b236dc96456e9658119a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38250] enum.Flag should be more set-like

[John Belmonte]

New submission from John Belmonte :

I would like Flag class instances to have more set-like abilities:
  1. iteration, to walk through each set bit of the value
  2. len corresponding to #1
  3. subset operator

I may be told "implement it yourself as instance methods", or that #3 has an 
idiom (a & b is b).  Ideally though, every Flag user should be able to rely on 
these being implemented consistently.

When trying to implement #1 without devolving into bit fiddling, naturally one 
might try to use the class iterator.  Unfortunately the semantics of that 
enumeration include 0, aliases, and compound values.  I've used Flag in several 
situations and projects, and so far there hasn't been a case where that was the 
desired semantics.  Interesting though, if #1 were implemented in the standard 
library, then we could enumerate all bits of the Flag via iteration of 
`~MyFlag(0)`... though that's obscuring things behind another idiom.

Thank you for considering.

--
components: Library (Lib)
messages: 352967
nosy: John Belmonte
priority: normal
severity: normal
status: open
title: enum.Flag should be more set-like
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38251] urllib.request memory leak / overflow

[Ofer Sadan]

New submission from Ofer Sadan :

running `urllib.request.urlopen` multiple times causes the memory usage to 
increase with each run, even after calling `close()` on the request or using 
`del` on the result

To recreate this problem, run code:

import urllib.request
def ip():
r = urllib.request.urlopen('https://api.ipify.org')
b = r.read()
r.close()
print(len(b))
return b

for i in range(1000):
result = ip()
del result

Even though `len(b)` has a maximum value of 15 (for this url at least), the 
memory increases with each run by 200KB - 1MB

--
components: Library (Lib)
messages: 352968
nosy: Ofer Sadan
priority: normal
severity: normal
status: open
title: urllib.request memory leak / overflow
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38251] urllib.request memory leak / overflow

[Ofer Sadan]

Ofer Sadan  added the comment:

Just to note: I'm using Python 3.7.4 on Windows 10 (64bit), the issue exists on 
both the 32bit/64bit python versions I could test on.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38252] micro-optimize ucs1lib_find_max_char in Windows 64-bit build

[Ma Lin]

New submission from Ma Lin :

C type `long` is 4-byte integer in 64-bit Windows build. [1]

But `ucs1lib_find_max_char()` function [2] uses SIZEOF_LONG, so it loses a 
little performance in 64-bit Windows build.

Below is the benchmark of using SIZEOF_SIZE_T and this change:

-   unsigned long value = *(unsigned long *) _p;
+   sizt_t value = *(sizt_t *) _p;

D:\dev\cpython\PCbuild\amd64\python.exe -m pyperf timeit -s "b=b'a'*10_000_000; 
f=b.decode;" "f('latin1')"

before: 5.83 ms +- 0.05 ms
after : 5.58 ms +- 0.06 ms

[1] https://stackoverflow.com/questions/384502

[2] 
https://github.com/python/cpython/blob/v3.8.0b4/Objects/stringlib/find_max_char.h#L9

Maybe there can be more optimizations, so I didn't prepare a PR for this.

--
components: Interpreter Core
messages: 352970
nosy: Ma Lin, inada.naoki, serhiy.storchaka, sir-sigurd
priority: normal
severity: normal
status: open
title: micro-optimize ucs1lib_find_max_char in Windows 64-bit build
type: performance
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38101] Update devguide triaging keywords

[Ido Michael]

Ido Michael  added the comment:

Hi Lisa,

I can take this (first commit), where can I edit the dev guide?

I could see someone did update the easy keyword in it?

Ido

--
nosy: +Ido Michael

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36947] [Good first issue] Fix 3.3.3.1 Metaclasses Documentation

[Ido Michael]

Ido Michael  added the comment:

Hey,

Is someone working on this issue? 
Can I take it?

Ido

--
nosy: +Ido Michael

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27873] multiprocessing.pool.Pool.map should take more than one iterable

[Ido Michael]

Ido Michael  added the comment:

Hey,

Is it still open?
What else needs to be done?

Ido

--
nosy: +Ido Michael

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15436] __sizeof__ is not documented

[Ido Michael]

Ido Michael  added the comment:

I can add those changes if someone didn't take it already?

Ido

--
nosy: +Ido Michael

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38251] urllib.request memory leak / overflow

[Kumar Akshay]

Kumar Akshay  added the comment:

JFYI, Not reproducible on macOS.

--
nosy: +kakshay

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38251] urllib.request memory leak / overflow

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Is this related to https://bugs.python.org/issue35941 given that it's related 
to windows. Similar report https://bugs.python.org/issue37498

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38251] urllib.request memory leak / overflow

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38101] Update devguide triaging keywords

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I think devguide has it's own repo that accepts PRs 
https://github.com/python/devguide

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38251] urllib.request memory leak / overflow

[Kumar Akshay]

Kumar Akshay  added the comment:

Yes, https://bugs.python.org/issue37498 is indeed the same issue

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38101] Update devguide triaging keywords

[Ido Michael]

Ido Michael  added the comment:

Thanks Karthikeyan!

Found it, so for the last one pep3121, would you like to describe PEP in 
general or Extension Module Initialization and Finalization?

Ido

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38251] urllib.request memory leak / overflow

[hongweipeng]

Change by hongweipeng :


--
nosy: +hongweipeng

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36274] http.client cannot send non-ASCII request lines

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I should say, though, this issue is a long-standing regression from Python 3.0. 
Although intentional, the inability for a client to override the encoding of 
the request line does make it impossible without replacing all of .putrequest 
to override that behavior, so although it would be a feature, it might justify 
a backport to supported bugfix Pythons to facilitate migration from Python 2.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38253] Fix typo of Py_SET_ERANGE_IF_OVERFLOW in pyport.h

[hai shi]

New submission from hai shi :

Py_SET_ERANGE_ON_OVERFLOW should be changed to Py_SET_ERANGE_IF_OVERFLOW in 
pyport.h.

--
assignee: docs@python
components: Documentation
messages: 352981
nosy: docs@python, shihai1991
priority: normal
severity: normal
status: open
title: Fix typo of Py_SET_ERANGE_IF_OVERFLOW in pyport.h
versions: Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38253] Fix typo of Py_SET_ERANGE_IF_OVERFLOW in pyport.h

[hai shi]

Change by hai shi :


--
keywords: +patch
pull_requests: +15906
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/16230

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38216] Fix for issue30458 (HTTP Header Injection) prevents crafting invalid requests

[Jason R. Coombs]

Jason R. Coombs  added the comment:

In https://github.com/cherrypy/cherrypy/pull/1807, I discovered that there is 
already a fairly straightforward means for a third-party package to override 
the putrequest character validation (just monkeypatch 
http.client._contains_disallowed_url_pchar_re with a less strict pattern; I 
used `[\n]`). This approach is barely worse than the proposal I made in the PR, 
the main differences being that a monkeypatch of that global variable is global 
(not selective to specific instances or subclasses of HTTPConnection) and 
doesn't have any tests in the CPython test suite to protect that as a supported 
mechanism.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38216] Fix for issue30458 (HTTP Header Injection) prevents crafting invalid requests

[Jason R. Coombs]

Jason R. Coombs  added the comment:

Also, with the CherryPy approach, the Python 2.7 story is more complicated. I 
haven't yet addressed that in the CherryPy 17 maintenance branch (which 
supports Python 2.7).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38172] Python 3.8 Segfult with Bandersnatch pytest Suite

[Cooper Lees]

Cooper Lees  added the comment:

Happy to close, just was not 100% sure if the fix is merged into the 3.8 
branch. I mainly opened this bug to ensure that has been done.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38254] Pickle not deserializing an OSError exception as expected

[David Parks]

New submission from David Parks :

Below is a minimum reproducible test case of what appears to be an issue in 
pickle. 

Before pickling the exception ClientConnectionError, from aiohttp, the property 
ClientConnectionError._os_error is a PermissionError object (a subclass of 
OSError). After deserialization ClientConnectionError._os_error appears to be a 
string based on the exception that is produced. 

I'm not familiar enough with pickle to make much headway here.

```
import aiohttp
import pickle


connection_key = aiohttp.client_reqrep.ConnectionKey
ose = OSError(1, 'unittest')
cce = aiohttp.client_exceptions.ClientConnectorError(connection_key, ose)
cce_pickled = pickle.dumps(cce)
pickle.loads(cce_pickled)
```

```
Traceback (most recent call last):
  File "/opt/.pycharm_helpers/pydev/pydevd.py", line 1758, in 
main()
  File "/opt/.pycharm_helpers/pydev/pydevd.py", line 1752, in main
globals = debugger.run(setup['file'], None, None, is_module)
  File "/opt/.pycharm_helpers/pydev/pydevd.py", line 1147, in run
pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/opt/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, 
in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/project_neural_mouse/src/ARCHIVE/scratch/scratch_13.py", line 11, in 

  File "/usr/local/lib/python3.6/dist-packages/aiohttp/client_exceptions.py", 
line 133, in __init__
super().__init__(os_error.errno, os_error.strerror)
AttributeError: 'str' object has no attribute 'errno'
```

--
components: Library (Lib)
messages: 352985
nosy: davidparks21
priority: normal
severity: normal
status: open
title: Pickle not deserializing an OSError exception as expected
type: crash
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38254] Pickle not deserializing an aiohttp ClientConnectorError exception as expected

[David Parks]

Change by David Parks :


--
title: Pickle not deserializing an OSError exception as expected -> Pickle not 
deserializing an aiohttp ClientConnectorError exception as expected

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38172] Python 3.8 Segfult with Bandersnatch pytest Suite

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Victor, am I correct in thinking that this should be closed as either 
intermittent, lacking in sufficient information for us to do anything, or 
possibly a 3rd party issue?

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38248] inconsistency in asyncio.Task between cancellation while running vs. cancellation immediately after it finishes

[Yury Selivanov]

Yury Selivanov  added the comment:

> The obvious fix would be to modify the 'except StopIteration' branch to 
> handle this case by calling super().cancel() instead of 
> super().set_exception(...).

Yeah, I think this is the solution we should do in 3.8.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38248] inconsistency in asyncio.Task between cancellation while running vs. cancellation immediately after it finishes

[Yury Selivanov]

Change by Yury Selivanov :


--
keywords: +patch
pull_requests: +15907
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/16330

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38216] Fix for issue30458 (HTTP Header Injection) prevents crafting invalid requests

[Ned Deily]

Ned Deily  added the comment:

I am certainly not a domain expert but, at a high level, I think the approach 
in PR 16321 is a reasonable compromise and I would support merging it to 3.7 
and 3.6 (I'll let Larry and Benjamin speak for 3.5 and 2.7) assuming there are 
no review objections.  Since we seem to be getting close to a comprehensive 
solution here, I am going to hold off on tagging 3.7.5r1 for a few more days in 
the hopes we can reach agreement on this approach.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38251] urllib.request memory leak / overflow

[Steve Dower]

Steve Dower  added the comment:

Closing as a duplicate of issue35941

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> ssl.enum_certificates() regression

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38216] Fix for issue30458 (HTTP Header Injection) prevents crafting invalid requests

[Ned Deily]

Ned Deily  added the comment:

Also, besides the normal news entry (via blurb), there should probably be a 
"Notable changes in x.y.z" entry added to the end of each affected release's 
Doc/whatsnew/x.y.rst file.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com