[issue39758] StreamWriter.wait_closed() can hang indefinately.
New submission from Tom Christie :
Raising an issue that's impacting us on `httpx`.
It appears that in some cases SSL unwrapping can cause `.wait_closed()` to hang
indefinately.
Trio are particularly careful to work around this case, and have an extensive
comment on it:
https://github.com/python-trio/trio/blob/31e2ae866ad549f1927d45ce073d4f0ea9f12419/trio/_ssl.py#L779-L829
Originally raised via https://github.com/encode/httpx/issues/634
Tested on:
* Python 3.7.6
* Python 3.8.1
```
import asyncio
import ssl
import certifi
hostname = 'login.microsoftonline.com'
context = ssl.create_default_context()
context.load_verify_locations(cafile=certifi.where())
async def main():
reader, writer = await asyncio.open_connection(hostname, 443, ssl=context)
print('opened')
writer.close()
print('close started')
await writer.wait_closed()
print('close completed')
asyncio.run(main())
```
--
components: asyncio
messages: 362688
nosy: asvetlov, tomchristie, yselivanov
priority: normal
severity: normal
status: open
title: StreamWriter.wait_closed() can hang indefinately.
type: behavior
versions: Python 3.7, Python 3.8
___
Python tracker
<https://bugs.python.org/issue39758>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14042] json.dumps() documentation is slightly incorrect.
New submission from Tom Christie :
json.dumps() documentation is slightly incorrect.
http://docs.python.org/library/json.html#json.dumps
Reads:
"If ensure_ascii is False, then the return value will be a unicode instance."
Should read:
"If ensure_ascii is False, then the return value MAY BE be a unicode
instance."
(Without the caps of course)
bash: python
Python 2.7.1 (r271:86832, Jun 16 2011, 16:59:05)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> type(json.dumps({'a': 1}, ensure_ascii=False))
Tested against 2.6 and 2.7.
--
components: Library (Lib)
messages: 153558
nosy: tomchristie
priority: normal
severity: normal
status: open
title: json.dumps() documentation is slightly incorrect.
type: behavior
versions: Python 2.6, Python 2.7
___
Python tracker
<http://bugs.python.org/issue14042>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36709] Asyncio SSL keep-alive connections raise errors after loop close.
Tom Christie added the comment:
> From my understanding, the correct code should close all transports and wait
> for their connection_lost() callbacks before closing the loop.
Ideally, yes, although we should be able to expect that an SSL connection that
hasn't been gracefully closed wouldn't loudly error on teardown like that.
In standard sync code, the equivelent would running something like this...
```python
session = requests.Session()
session.get('https://example.com/')
```
We wouldn't expect a traceback to be raised on exiting. (Even though the user
*hasn't* explicitly closed the session, and even though a keep alive SSL
connection will be open at the point of exit.)
--
___
Python tracker
<https://bugs.python.org/issue36709>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36709] Asyncio SSL keep-alive connections raise errors after loop close.
Tom Christie added the comment: Right, and `requests` *does* provide both those styles. The point more being that *not* having closed the transport at the point of exit shouldn't end up raising a hard error. It doesn't raise errors in sync-land, and it shouldn't do so in async-land. Similarly, we wouldn't expect an open file resource to cause errors to be raised at the point of exit. -- ___ Python tracker <https://bugs.python.org/issue36709> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36709] Asyncio SSL keep-alive connections raise errors after loop close.
New submission from Tom Christie :
If an asyncio SSL connection is left open (eg. any kind of keep-alive
connection) then after closing the event loop, an exception will be raised...
Python:
```
import asyncio
import ssl
import certifi
async def f():
ssl_context = ssl.create_default_context()
ssl_context.load_verify_locations(cafile=certifi.where())
await asyncio.open_connection('example.org', 443, ssl=ssl_context)
loop = asyncio.get_event_loop()
loop.run_until_complete(f())
loop.close()
```
Traceback:
```
$ python example.py
Fatal write error on socket transport
protocol:
transport: <_SelectorSocketTransport fd=8>
Traceback (most recent call last):
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py",
line 868, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
Fatal error on SSL transport
protocol:
transport: <_SelectorSocketTransport closing fd=8>
Traceback (most recent call last):
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py",
line 868, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/sslproto.py",
line 676, in _process_write_backlog
self._transport.write(chunk)
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py",
line 872, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py",
line 681, in _fatal_error
self._force_close(exc)
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py",
line 693, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py",
line 677, in call_soon
self._check_closed()
File
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py",
line 469, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
```
It looks to me like the original "OSError: [Errno 9] Bad file descriptor"
probably shouldn't be raised in any case - if when attempting to tear down the
SSL connection, then we should probably pass silently in the case that the
socket has already been closed uncleanly.
Bought to my attention via: https://github.com/encode/httpcore/issues/16
--
assignee: christian.heimes
components: SSL, asyncio
messages: 340764
nosy: asvetlov, christian.heimes, tomchristie, yselivanov
priority: normal
severity: normal
status: open
title: Asyncio SSL keep-alive connections raise errors after loop close.
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue36709>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36709] Asyncio SSL keep-alive connections raise errors after loop close.
Tom Christie added the comment: This appears somewhat related: https://bugs.python.org/issue34506 As it *also* logs exceptions occuring during `_fatal_error` and `_force_close`. -- ___ Python tracker <https://bugs.python.org/issue36709> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33366] `contextvars` documentation incorrectly refers to "non-local state".
New submission from Tom Christie : The `contextvars` documentation, at https://docs.python.org/3.7/library/contextvars.html starts with the following: "This module provides APIs to manage, store, and access non-local state." I assume that must be a documentation bug, right. The module isn't for managing non-local state, it's for managing state that *is* local. I'd assume it ought to read... "This module provides APIs to manage, store, and access context-local state." (ie. for managing state that is transparently either thread-local or task-local depending on the current execution context.) -- assignee: docs@python components: Documentation, asyncio messages: 315792 nosy: asvetlov, docs@python, tomchristie, yselivanov priority: normal severity: normal status: open title: `contextvars` documentation incorrectly refers to "non-local state". type: enhancement versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue33366> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33366] `contextvars` documentation incorrectly refers to "non-local state".
Tom Christie added the comment: Refs: https://github.com/python/cpython/pull/6617 -- ___ Python tracker <https://bugs.python.org/issue33366> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22767] `separators` argument to json.dumps() behaves unexpectedly across 2.x vs 3.x
New submission from Tom Christie:
This is one of those behavioural issues that is a borderline bug.
The seperators argument to `json.dumps()` behaves differently across python 2
and 3.
* In python 2 it should be provided as a bytestring, and can cause a
UnicodeDecodeError otherwise.
* In python 3 it should be provided as unicode,and can cause a TypeError
otherwise.
Examples:
Python 2.7.2
>>> print json.dumps({'snowman': '☃'}, separators=(':', ','),
ensure_ascii=False)
{"snowman","☃"}
>>> print json.dumps({'snowman': '☃'}, separators=(u':', u','),
ensure_ascii=False)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1:
ordinal not in range(128)
And:
Python 3.4.0
>>> print(json.dumps({'snowman': '☃'}, separators=(':', ','),
ensure_ascii=False))
{"snowman","☃"}
>>> print(json.dumps({'snowman': '☃'}, separators=(b':', b','),
ensure_ascii=False))
<...>
TypeError: sequence item 2: expected str instance, bytes found
Technically this isn't out of line with the documentation - in both cases it
uses `separators=(':', ',')` which is indeed the correct type in both v2 and
v3. However it's unexpected behaviour that it changes types between versions,
without being called out.
Working on a codebase with `from __future__ import unicode_literals` this is
particularly unexpected because we get a `UnicodeDecodeError` when running code
that otherwise looks correct.
It's also slightly awkward to fix because it's a bit of a weird branch
condition.
The fix would probably be to forcibly coerce it to the correct type regardless
of if it is supplied as unicode or a bytestring, or at least to do so for
python 2.7.
Possibly related to http://bugs.python.org/issue22701 but wasn't able to
understand if that ticket was in fact a different user error.
--
messages: 230274
nosy: Tom.Christie
priority: normal
severity: normal
status: open
title: `separators` argument to json.dumps() behaves unexpectedly across 2.x vs
3.x
type: behavior
versions: Python 2.7, Python 3.4
___
Python tracker
<http://bugs.python.org/issue22767>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22767] `separators` argument to json.dumps() behaves unexpectedly across 2.x vs 3.x
Tom Christie added the comment: Not too fussed if this is addressed or not, but I think this is closed a little prematurely. I don't think there's a problem under Python 3, that's entirely reasonable. However under Python 2, `json.dumps()` will normally handle *either* bytes or unicode transparently for you (just altering the return type accordingly). If you happen to be using unicode separators, then the normally lax behaviour of "either unicode or bytes" that stops being the case. -- ___ Python tracker <http://bugs.python.org/issue22767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22767] `separators` argument to json.dumps() behaves unexpectedly across 2.x vs 3.x
Tom Christie added the comment:
> But only if you use non-ascii in the binary input, in which case you get an
> encoding error, which is a correct error.
Kind of, except that this (python 2.7) works just fine:
>>> data = {'snowman': '☃'}
>>> json.dumps(data, ensure_ascii=False)
'{"snowman": "\xe2\x98\x83"}'
Whereas this raises an exception:
>>> json.dumps(data, separators=(u':', u','), ensure_ascii=False)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1:
ordinal not in range(128)
If it was the same in both cases then I wouldn't consider it a problem.
As it is, introducing the `seperators` parameter changes the behaviour.
Anyways, I'll get off my high horse now. :)
--
___
Python tracker
<http://bugs.python.org/issue22767>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22767] `separators` argument to json.dumps() behaves unexpectedly across 2.x vs 3.x
Tom Christie added the comment: > So, as soon as (but only as soon as) you mix unicode with your non-ascii > data, your program blows up. Indeed. For context tho my example of running into this the unicode literals used as seperators weren't even in the same package as the non-ASCII binary strings. (JSONRenderer in Django REST framework, being excersized by some third party test code. End result non-obvious exception. Anyways, okay with this resolution, although I am now using a compat branch to ensure that we use binary seperators in py2 to continue to get the more lax rendering style. -- ___ Python tracker <http://bugs.python.org/issue22767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18290] json encoder does not support JSONP/JavaScript safe escaping
Tom Christie added the comment: I believe the status of this should be reassessed and that python should default to escaping '\u2028' and '\u2029'. *Strictly* speaking this isn't a bug and is per the JSON spec. *However* this *is* a bug in the JSON spec - which *should* be a strict subset of JSON. Given that both escaped and unescaped are valid, ensuring that those two characters *are* always escaped would clearly be more user-friendly behavior on our part, and *would* lead to less bugs in, say web frameworks that use the JSON module and then pass the output to template (eg populating a javscript variable with some JSON output). -- nosy: +tomchristie ___ Python tracker <http://bugs.python.org/issue18290> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18290] json encoder does not support JSONP/JavaScript safe escaping
Tom Christie added the comment: > There is explicit note in the documentation about incompatibility with > JavaScript. That may be, but we're still unnecessarily making for a poorer user experience. There's no good reason why we shouldn't just treat \u2028 and \u2029 as control characters - it's only going to making things better for developers using the json module. It is an unnecessary usability bug as it stands. Just because JSON has a bug in its spec wrt those two characters, doesn't mean we can't help our users avoid ever having to know about that or work around it in user code. -- ___ Python tracker <http://bugs.python.org/issue18290> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1043134] Add preferred extensions for MIME types
Tom Christie added the comment: Confirming that I've also bumped into this for Python 3.5. A docs update would seem to be the lowest-cost option to start with. Right now `mimetypes.guess_extension()` isn't terribly useful, and it'd be better to at least know that upfront. -- nosy: +Tom.Christie ___ Python tracker <http://bugs.python.org/issue1043134> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
