date:20200420



Serhiy Storchaka  added the comment:

Reducing the import time is not a goal, it is just a measurable side effect. 
The goal is to reduce the number of imported modules unneeded for the 
particular test. Importing every module can have side effects which affects the 
tested behavior. It would be nice to minimize it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40332] RegEx for numbers in documentation (easy fix - solution provided)



Serhiy Storchaka  added the comment:

I concur with Mark. It is just an example. The regular expression which would 
support all possible forms of numbers (including exponent, underscores, 
non-decimal numbers) would be more complex and would distract from the main 
goal of the example.

--
nosy: +serhiy.storchaka
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40335] Regression in multiline SyntaxError offsets



Change by Pablo Galindo Salgado :


--
keywords: +patch
pull_requests: +18949
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19619

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40336] Refactor typing._SpecialForm



New submission from Serhiy Storchaka :

The proposed PR refactors the implementation of typing._SpecialForm.

Different special forms are different by name, docstring, and the behavior of 
__getitem__. Instead of special casing by name in __getitem__, instances are 
now parametrized by the implementation of __getitem__. _SpecialForm is now used 
as a decorator and takes the name and the docstring from the decorated 
function. It looks nicer to me now.

Also removed implementations of some methods:

* Inheriting from _Immutable no longer needed because __reduce__ makes instance 
atomic for copying.

* __new__ was only needed when _SpecialForm was a subclass of type. It 
prevented using it as a metaclass. Now it is no longer needed.

* Docstring is now assigned directly to the __doc__ slot. Previously it 
conflicted with the class docstring, but now the class docstring was converted 
into the class comment. _SpecialForm is not a public class, and pydoc shows the 
class comment if there is no a class docstring.

* __eq__ and __hash__ are no longer needed. Instances are singletons, and 
comparing by identity works as well as comparing by name. They could be needed 
before implementing __reduce__().

* __call__ no longer needed. Instances are not callable, and the error message 
is good enough. The bonus -- callable() now returns False.

--
components: Library (Lib)
messages: 366817
nosy: gvanrossum, levkivskyi, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Refactor typing._SpecialForm
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40336] Refactor typing._SpecialForm



Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +18950
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19620

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

2020-04-20 Thread Mark Shannon



Mark Shannon  added the comment:

Eddie,
How did you compare the branching vs. non-branching implementations?  I have 
read the code in the PR. What is the non-branching version that you used to 
compare it against?

Why not use the sign bit as the immortal bit? It simplifies the test 
considerably, making it faster, and doesn't need any assembly code.

Are you suggesting that making a set of common objects immortal will make 
things faster? Have you tested it?
(I would expect that the negative impact on branch predictability would easily 
outweigh the cost of the memory write (A guaranteed L1 hit).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39983] test.regrtest: test marked as failed (env changed), but no warning: test_multiprocessing_forkserver



STINNER Victor  added the comment:

Another example: test_asyncio on AMD64 Fedora Stable Clang 3.x
https://buildbot.python.org/all/#/builders/226/builds/630

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17986] Alternative async subprocesses (pep 3145)



STINNER Victor  added the comment:

Well, this issue is basically inactive for 5 years. It doesn't sound like a 
common feature request. I close it.

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34037] asyncio: BaseEventLoop.close() shutdowns the executor without waiting causing leak of dangling threads



STINNER Victor  added the comment:

I reopen the issue.

> /usr/lib64/python3.9/asyncio/events.py:254: in shutdown_default_executor
> raise NotImplementedError

That means that the project (python-anyio) implements its own event loop which 
inherits from AbstractEventLoop, it doesn't use BaseEventLoop which implements 
shutdown_default_executor().

AbstractEventLoop documentation says:

https://docs.python.org/dev/library/asyncio-eventloop.html#asyncio.AbstractEventLoop

"The Event Loop Methods section lists all methods that an alternative 
implementation of AbstractEventLoop should have defined."

It points to the following documentation which lists shutdown_asyncgens():

It points to 
https://docs.python.org/dev/library/asyncio-eventloop.html#asyncio-event-loop


> I'm not all that familiar with asyncio, but it seems to me that all event 
> loops inherited from BaseEventLoop must be updated to implement this new 
> method to correctly work with run() in Python 3.9. Is that right?

Yes

> If it is, this needs at least a much more prominent What's New entry. Or the 
> hard NotImplementedError should turn into a warning.

Raising NotImplementedError is a deliberate design choice.

Documentation the new requirement in the following section sounds like a good 
idea.

https://docs.python.org/dev/whatsnew/3.9.html#changes-in-the-python-api

Kyle: Can you please add a short sentence there to document the new requirement?

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40284] Add mapping methods to types.SimpleNamespace



STINNER Victor  added the comment:

Raymond started a thread on python-dev:
https://mail.python.org/archives/list/python-...@python.org/message/JOMND56PJGRN7FQQLLCWONE5Z7R2EKXW/

It seems like most core developers are against modifying types.SimpleNamespace, 
the trend is more about adding a different class.

About a different class, so far, I see no consensus on an API. I suggest to 
close this issue (which is about types.SimpleNamespace) and continue the 
discussion on python-dev.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40275] test.support has way too many imports



STINNER Victor  added the comment:

Vinay Sajip: "What is the practical impact on the time taken for test runs?"

My first concern is that our "unit tests" are not "unit" anymore. 
test_threading should be restricted to test the threading module: it should not 
test "indirectly" the logging module.

If someone wants to test how logging behaves on fork, test_logging should get a 
new test. (I didn't check if it already has such test.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows



Steve Dower  added the comment:

I posted a workaround right above your post for when you want to resolve 
executables against PATH instead of using Windows's normal rules.

I'm not sure we reached any good approach for launching sys.executable in a 
venv and automatically bypassing the redirector.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting



Steve Dower  added the comment:

> I would expect that the negative impact on branch predictability would easily 
> outweigh the cost of the memory write (A guaranteed L1 hit)

If that were true then Spectre and Meltdown wouldn't have been so interesting :)

Pipelining processors are going to speculatively execute both paths, and will 
skip the write much more quickly than by doing it, and meanwhile nobody should 
have tried to read the value so it hasn't had to block for that path. I'm not 
aware of any that detect no-op writes and skip synchronising across cores - the 
dirty bit of the cache line is just set unconditionally.

Benchmarking already showed that the branching version is faster. It's possible 
that "refcount += (refcount & IMMORTAL) ? 0 : 1" could generate different code 
(should be mov,test,lea,cmovz rather than mov,and,add,mov or 
mov,and,jz,add,mov), but it's totally reasonable for a branch to be faster than 
unconditionally modifying memory.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting



Antoine Pitrou  added the comment:

> Benchmarking already showed that the branching version is faster.

But micro-benchmarks may tell you things which are not true in the real-world 
(for example an easily-predicted branch).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows



Steve Dower  added the comment:

Thanks! I deleted the NEWS file (we've already got one for this issue in this 
release) and skipped the check instead. Once CI passes, I'll merge it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting



Steve Dower  added the comment:

> But micro-benchmarks may tell you things which are not true in the real-world 
> (for example an easily-predicted branch)

This is true, though this branch should be more easily predictable because 
INCREF/DECREF are inlined. If there were only one function doing the branch 
it'd be different.

Wouldn't even surprise me if PGO made different optimisations to this in 
certain functions. Anything that normally operates on a code object, for 
example, is almost always going to take the branch when everything is imported 
early and before immortalisation.

We also have the real world app that is Instagram. If they say that this has 
shown an improvement in their app, I'm inclined to believe that it's not 
reliant on a microbenchmark (other controlled circumstances, sure, but not a 
microbenchmark ;) ).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40337] builtins.RuntimeError: Caught RuntimeError in pin memory thread for device 0.

2020-04-20 Thread shawn


New submission from shawn :

 File "D:\yolov3\train.py", line 430, in 
  train() # train normally
 File "D:\yolov3\train.py", line 236, in train
  for i, (imgs, targets, paths, _) in pbar: # batch 
-
File "D:\Programs\Python\Python38\Lib\site-packages\tqdm\std.py", line 1127, in 
__iter__
  for obj in iterable:
File 
"D:\Programs\Python\Python38\Lib\site-packages\torch\utils\data\dataloader.py", 
line 345, in __next__
  data = self._next_data()
File 
"D:\Programs\Python\Python38\Lib\site-packages\torch\utils\data\dataloader.py", 
line 856, in _next_data
  return self._process_data(data)
File 
"D:\Programs\Python\Python38\Lib\site-packages\torch\utils\data\dataloader.py", 
line 881, in _process_data
  data.reraise()
File "D:\Programs\Python\Python38\Lib\site-packages\torch\_utils.py", line 394, 
in reraise
  raise self.exc_type(msg)

builtins.RuntimeError: Caught RuntimeError in pin memory thread for device 0.
Original Traceback (most recent call last):
File 
"D:\Programs\Python\Python38\lib\site-packages\torch\utils\data\_utils\pin_memory.py",
 line 31, in _pin_memory_loop
data = pin_memory(data)
File 
"D:\Programs\Python\Python38\lib\site-packages\torch\utils\data\_utils\pin_memory.py",
 line 55, in pin_memory
return [pin_memory(sample) for sample in data]
File 
"D:\Programs\Python\Python38\lib\site-packages\torch\utils\data\_utils\pin_memory.py",
 line 55, in 
return [pin_memory(sample) for sample in data]
File 
"D:\Programs\Python\Python38\lib\site-packages\torch\utils\data\_utils\pin_memory.py",
 line 47, in pin_memory
return data.pin_memory()
 ... (truncated)

--
messages: 366829
nosy: shawn
priority: normal
severity: normal
status: open
title: builtins.RuntimeError: Caught RuntimeError in pin memory thread for 
device 0.
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40337] builtins.RuntimeError: Caught RuntimeError in pin memory thread for device 0.

2020-04-20 Thread Eric V. Smith



Eric V. Smith  added the comment:

This appears to be a problem in a third-party library: torch.

You didn't include any code example that triggers the problem. Without some way 
to duplicate this issue, there's not much we can do about it.

--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40333] Request for multi-phase initialization API to run code after importlib init



STINNER Victor  added the comment:

The current workaround is to use PyConfig._init_main=0, call 
Py_InitializeFromConfig(), tune Python, and then call _Py_InitializeMain() to 
finish the "main" initialization:

https://docs.python.org/dev/c-api/init_config.html#multi-phase-initialization-private-provisional-api

You can execute code between init_importlib() and init_importlib_external().

I understand that it's not enough for you. As I wrote, it's a workaround.

--

Well, your use case is to fully control the "Python Path Configuration":
https://docs.python.org/dev/c-api/init_config.html#path-configuration

This part was completely put aside in PEP 587 on purpose (to be able to put the 
done part in Python 3.8), but it's part of PEP 432.

It was proposed to rewrite getpath.c and getpathc.c in pure Python to let 
embedders to fully override this code with their own Python implementation. But 
nobody is available to implement this feature.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40333] Request for multi-phase initialization API to run code after importlib init



Change by STINNER Victor :


--
nosy: +eric.snow, ncoghlan, steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40338] [Security] urllib and anti-slash (\) in the hostname


New submission from STINNER Victor :

David Schütz reported the following urllib vulnerability to the PSRT at 
2020-03-29.

He wrote an article about a similar vulnerability in Closure (Javascript):
https://bugs.xdavidhu.me/google/2020/03/08/the-unexpected-google-wide-domain-check-bypass/

David was able to bypass a wildcard domain check in Closure by using the "\" 
character in the URL like this:

  https://xdavidhu.me\test.corp.google.com

Example in Python:

>>> from urllib.parse import urlparse
>>> urlparse("https://xdavidhu.me\\test.corp.google.com";)
ParseResult(scheme='https', netloc='xdavidhu.me\\test.corp.google.com', 
path='', params='', query='', fragment='')

urlparse() currently accepts "\" in the netloc.

This could present issues if server-side checks are used by applications to 
validate a URLs authority.

The problem emerges from the fact that the RFC and the WHATWG specifications 
differ, and the RFC does not mention the "\":

* RFC: https://tools.ietf.org/html/rfc3986#appendix-B
* WHATWG: https://url.spec.whatwg.org/#relative-state

This specification difference might cause issues, since David do understand 
that the parser is implemented by the RFC, but the WHATWG spec is what the 
browsers are using, who will mainly be the ones opening the URL.

--
components: Library (Lib)
messages: 366832
nosy: vstinner
priority: normal
severity: normal
status: open
title: [Security] urllib and anti-slash (\) in the hostname
type: security
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40338] [Security] urllib and anti-slash (\) in the hostname



STINNER Victor  added the comment:

(The first message is basically David's email rephrased. Here is my reply ;-))

> This could present issues if server-side checks are used by applications to 
> validate a URLs authority.

Which kind of application would be affected by this vulnerability?

It's unclear to me if urllib should be modified to explicitly reject \ in 
netloc, or if only third-party code should pay attention to this corner case 
(potential vulnerability).


The urllib module has _parse_proxy() and HTTPPasswordMgr.reduce_uri() code 
which use an "authority" variable.

Example:
---
from urllib.parse import urlsplit, _splitport, _splittype, _splituser,
_splitpasswd

def _parse_proxy(proxy):
"""Return (scheme, user, password, host/port) given a URL or an authority.

If a URL is supplied, it must have an authority (host:port) component.
According to RFC 3986, having an authority component means the URL must
have two slashes after the scheme.
"""
scheme, r_scheme = _splittype(proxy)
if not r_scheme.startswith("/"):
# authority
scheme = None
authority = proxy
else:
# URL
if not r_scheme.startswith("//"):
raise ValueError("proxy URL with no authority: %r" % proxy)
# We have an authority, so for RFC 3986-compliant URLs (by ss 3.
# and 3.3.), path is empty or starts with '/'
end = r_scheme.find("/", 2)
if end == -1:
end = None
authority = r_scheme[2:end]
userinfo, hostport = _splituser(authority)
if userinfo is not None:
user, password = _splitpasswd(userinfo)
else:
user = password = None
return scheme, user, password, hostport


def reduce_uri(uri, default_port=True):
"""Accept authority or URI and extract only the authority and path."""
# note HTTP URLs do not have a userinfo component
parts = urlsplit(uri)
if parts[1]:
# URI
scheme = parts[0]
authority = parts[1]
path = parts[2] or '/'
else:
# host or host:port
scheme = None
authority = uri
path = '/'
host, port = _splitport(authority)
if default_port and port is None and scheme is not None:
dport = {"http": 80,
 "https": 443,
 }.get(scheme)
if dport is not None:
authority = "%s:%d" % (host, dport)
return authority, path

def test(uri):
print(f"{uri} => reduce_uri: {reduce_uri(uri)}")
print(f"{uri} => _parse_proxy: {_parse_proxy(uri)}")

test(r"https://www.example.com";)
test(r"https://u...@www.example.com";)
test(r"https://xdavidhu.me\test.corp.google.com";)
test(r"https://user:passw...@xdavidhu.me\test.corp.google.com";)
---

Output on Python 3.9:
---
https://www.example.com => reduce_uri: ('www.example.com:443', '/')
https://www.example.com => _parse_proxy: ('https', None, None,
'www.example.com')
https://u...@www.example.com => reduce_uri: ('u...@www.example.com:443', '/')
https://u...@www.example.com => _parse_proxy: ('https', 'user', None,
'www.example.com')
https://xdavidhu.me\test.corp.google.com => reduce_uri:
('xdavidhu.me\\test.corp.google.com:443', '/')
https://xdavidhu.me\test.corp.google.com => _parse_proxy: ('https',
None, None, 'xdavidhu.me\\test.corp.google.com')
https://user:passw...@xdavidhu.me\test.corp.google.com => reduce_uri:
('user:passw...@xdavidhu.me\\test.corp.google.com:443', '/')
https://user:passw...@xdavidhu.me\test.corp.google.com =>
_parse_proxy: ('https', 'user', 'password',
'xdavidhu.me\\test.corp.google.com')
---

It seems to behave as expected, no?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37006] Add top level await statement support for doctest

2020-04-20 Thread Batuhan Taskaya



Change by Batuhan Taskaya :


--
nosy: +BTaskaya
versions: +Python 3.9 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting



Antoine Pitrou  added the comment:

> We also have the real world app that is Instagram.

I don't think Instagram is a single app. What is Python used for at Instagram?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows



Steve Dower  added the comment:


New changeset 9b0b5d2baebd0b6a545317200c313a6a7408731e by Barry in branch 
'master':
bpo-40260: Revert breaking changes made in modulefinder (GH-19595)
https://github.com/python/cpython/commit/9b0b5d2baebd0b6a545317200c313a6a7408731e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows



Change by miss-islington :


--
pull_requests: +18951
pull_request: https://github.com/python/cpython/pull/19621

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

2020-04-20 Thread Rémi Lapeyre


Rémi Lapeyre  added the comment:

>> We also have the real world app that is Instagram.

> I don't think Instagram is a single app. What is Python used for at Instagram?

According to their engineering blog 
(https://instagram-engineering.com/static-analysis-at-scale-an-instagram-story-8f498ab71a0c):

> Our server app is a monolith, one big codebase of several million lines and a 
> few thousand Django endpoints [1], all loaded up and served together. A few 
> services have been split out of the monolith, but we don’t have any plans to 
> aggressively break it up.

--
nosy: +remi.lapeyre

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting



Antoine Pitrou  added the comment:

Wow.  I see, thank you.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39562] Asynchronous comprehensions don't work in asyncio REPL



STINNER Victor  added the comment:

Is there an update of this *release blocker* issue? Should we revert the commit 
9052f7a41b90f2d34011c8da68f9a4facebc8a97?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39651] Exceptions raised by EventLoop.call_soon_threadsafe



Chris Meyer  added the comment:

Is this related to bpo-39010 too?

--
nosy: +cmeyer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError



Change by Chris Meyer :


--
nosy: +cmeyer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37373] Configuration of windows event loop for libraries



Change by Chris Meyer :


--
nosy: +cmeyer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39651] Exceptions raised by EventLoop.call_soon_threadsafe

2020-04-20 Thread Ben Darnell



Ben Darnell  added the comment:

No, this is unrelated to bpo-39010.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34037] asyncio: BaseEventLoop.close() shutdowns the executor without waiting causing leak of dangling threads



Change by Chris Meyer :


--
nosy: +cmeyer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39562] Asynchronous comprehensions don't work in asyncio REPL

2020-04-20 Thread Batuhan Taskaya



Batuhan Taskaya  added the comment:

> Is there an update of this *release blocker* issue? Should we revert the 
> commit 9052f7a41b90f2d34011c8da68f9a4facebc8a97?

I dont think we can gain much by reverting 
9052f7a41b90f2d34011c8da68f9a4facebc8a97 because it doesn't introduce a new 
issue, it just adds "another" path to notice it. 

Test:
>>> await __import__('asyncio').sleep(1)

Expected (and the behavior of 3.6):

(.venv) [  6:09PM ]  [ isidentical@threeheadedgiant:~ ]
 $ python3.6 z.py   
**
File "doctest.txt", line 4, in doctest.txt
Failed example:
await __import__('asyncio').sleep(1)
Exception raised:
Traceback (most recent call last):
  File "/usr/lib/python3.6/doctest.py", line 1330, in __run
compileflags, 1), test.globs)
  File "", line 1
await __import__('asyncio').sleep(1)
   ^
SyntaxError: invalid syntax
**
1 items had failures:
   1 of   1 in doctest.txt
***Test Failed*** 1 failures.
result: TestResults(failed=1, attempted=1

Current master (+ reverted GH 18968):
(.venv) [  6:09PM ]  [ isidentical@threeheadedgiant:~ ]
 $ ./cpython/python z.py
/home/isidentical/cpython/Lib/doctest.py:1336: RuntimeWarning: coroutine 
'' was never awaited
  exec(compile(example.source, filename, "single",
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
result: TestResults(failed=0, attempted=1)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39849] Compiler warninig: warning: variable ‘res’ set but not used [-Wunused-but-set-variable]

2020-04-20 Thread hai shi



Change by hai shi :


--
pull_requests: +18952
pull_request: https://github.com/python/cpython/pull/19623

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows



miss-islington  added the comment:


New changeset 81de3c225774179cdc82a1733a64e4a876ff02b5 by Miss Islington (bot) 
in branch '3.8':
bpo-40260: Revert breaking changes made in modulefinder (GH-19595)
https://github.com/python/cpython/commit/81de3c225774179cdc82a1733a64e4a876ff02b5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39562] Asynchronous comprehensions don't work in asyncio REPL

2020-04-20 Thread Batuhan Taskaya



Batuhan Taskaya  added the comment:

GH 19230:
(.venv) [  6:17PM ]  [ isidentical@threeheadedgiant:~ ]
 $ ./cpython/python z.py
**
File "/home/isidentical/doctest.txt", line 4, in doctest.txt
Failed example:
await __import__('asyncio').sleep(1)
Exception raised:
Traceback (most recent call last):
  File "/home/isidentical/cpython/Lib/doctest.py", line 1336, in __run
exec(compile(example.source, filename, "single",
  File "", line 1
SyntaxError: 'await' outside function
**
1 items had failures:
   1 of   1 in doctest.txt
***Test Failed*** 1 failures.
result: TestResults(failed=1, attempted=1)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection



Allan Feldman  added the comment:

Thanks for the explanation! I agree that "about to be finalized" is unclear in 
the docs :)

I still believe that having different behavior for the ordering of finalizers 
versus weakref callbacks depending on whether the path is through gc versus 
reference counting is a bug.

The callback should be able to assume that when it's running, the referent is 
actually dead. The execution of a weakref callback in our case results in items 
being dropped from a WeakValueDictionary prematurely (the object is still 
referenced, accessible, and alive at the time the weakref callback runs).

I've attached a patch that would cause weakref callbacks to run consistently 
after finalizers. With the patch applied, all tests in cpython appear to pass, 
but the code examples above now work consistently.

--
keywords: +patch
Added file: 
https://bugs.python.org/file49076/0001-Run-finalizers-before-invoking-weakref-callbacks.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection



Allan Feldman  added the comment:

Also I just noticed this statement:

> In current CPython, for your ForeverObject(False), `del o` does not make the 
> object trash "for real".  __del__ runs immediately (due to deterministic, 
> synchronous reference counting) and resurrects it.  That cuts off the "about 
> to have its memory destroyed and recycled" part, so the callback doesn't run.

The problem is the callback _does_ run even though the object is resurrected! :)

(Only if going through gc)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40338] [Security] urllib and anti-slash (\) in the hostname

2020-04-20 Thread hai shi



Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23117] Properly codesign Mac python 2.7.9.pkg so it can work thru OS X firewall

2020-04-20 Thread Ned Deily



Ned Deily  added the comment:

Thanks to the additional requirements of Gatekeeper in macOS 10.15 Catalina, 
the binaries included in current python.org installers for macOS are now 
codesigned as of 3.8.2, 3.7.7, and 2.7.18.

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed
versions: +Python 3.7, Python 3.8, Python 3.9 -Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36890] python-3.7.3-macosx10.6.pkg verification error on macOS 10.6 Snow Leopard

2020-04-20 Thread Ned Deily



Ned Deily  added the comment:

Beginning in 2020, we have stopped providing the 10.6+ installer variant for 
macOS since all evidence that we have is that it was used very little and there 
are a grower number of problems with trying to use it on very old systems, as 
evidenced in this issue.  Since there has been no feedback yet about not 
providing the 10.6+ installer variant, I am closing this issue as not to be 
fixed. (We continue to provide the 10.9 64-bit-only installer variant.)  By the 
way, this does not mean we are intentionally removing or breaking support for 
building and running on 10.6 systems.  It's just that we are no longer 
providing pre-built binaries for them.

--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection



Pablo Galindo Salgado  added the comment:

I think this is a good summary of what you are referring to:

>>> import gc, weakref
>>> class Lazarus:
...def __del__(self):
...   global x
...   x = self
...
>>> def callback(*args):
... print("DEAD")
...

# No gc dead:

>>> x = None
>>> a = Lazarus()
>>> w = weakref.ref(a, callback)
>>> del a # Notice that the callback has not been executed
>>> del x
DEAD

# Gc code:

>>> a = Lazarus()
>>> x = None
>>> cycle = []
>>> cycle.append(cycle)
>>> cycle.append(a)
>>> w = weakref.ref(a, callback)
>>> del a,cycle
>>> gc.collect()
DEAD
>>> x. #The callback has executed but the variable is still alive
<__main__.Lazarus object at 0x1020e9910>


The "problem" is that when going through the gc, the callback is executed when 
the object is resurrected but going through normal refcount the callback is 
executed only when the object dies.

Notice that in both cases the fundamental property still holds: the callback is 
executed *once*. What you are discussing now is a matter of *when*. 

Not sure if we want to homogenize both code paths, but changing the one that 
goes through the GC is going to be challenging to say the least because as Tim 
mentioned, the weakrefs semantics are *very* delicate.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection



Pablo Galindo Salgado  added the comment:

One thing we could do is call the weakref callbacks *after* we call 
`finalize_garbage` and only on the "final_unreachable" set (the objects that do 
not resurrect). Notice that doing this still has one difference: the callback 
will be executed AFTER the finalizer while in the normal refcount path is 
executed BEFORE the finalizer.

I have not given enough thought to the correctness of doing this but my gut 
feeling tells me something can go wrong if we do that. What do you think about 
this path, Tim? 

Here is the diff I am refering to for clarity:

diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 5727820f09..498ff927ab 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -1252,9 +1252,6 @@ collect(PyThreadState *tstate, int generation,
 }
 }

-/* Clear weakrefs and invoke callbacks as necessary. */
-m += handle_weakrefs(&unreachable, old);
-
 validate_list(old, collecting_clear_unreachable_clear);
 validate_list(&unreachable, collecting_set_unreachable_clear);

@@ -1267,6 +1264,9 @@ collect(PyThreadState *tstate, int generation,
 PyGC_Head final_unreachable;
 handle_resurrected_objects(&unreachable, &final_unreachable, old);

+/* Clear weakrefs and invoke callbacks as necessary. */
+m += handle_weakrefs(&final_unreachable, old);
+
 /* Call tp_clear on objects in the final_unreachable set.  This will cause
 * the reference cycles to be broken.  It may also cause some objects
 * in finalizers to be freed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39562] Asynchronous comprehensions don't work in asyncio REPL



STINNER Victor  added the comment:

Either the regression should be fixed, or the commits which introduced the 
regression should be reverted.

I'm talking about python-gmpy2 doctest which pass on Python 3.8: msg365311.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39562] Asynchronous comprehensions don't work in asyncio REPL



Pablo Galindo Salgado  added the comment:

> Either the regression should be fixed, or the commits which introduced the 
> regression should be reverted.

If you do that we will have now two problems:

* The asyncio repr will not work correctly.
* The flags in compile still have a clash.

The solution is to merge https://github.com/python/cpython/pull/19230 or some 
version of it, which should fix the issue in  python-gmpy2 doctest .

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows



Change by Steve Dower :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

2020-04-20 Thread Mark Shannon



Mark Shannon  added the comment:

On 20/04/2020 2:33 pm, Steve Dower wrote:
> 
> Steve Dower  added the comment:
> 
>> I would expect that the negative impact on branch predictability would 
>> easily outweigh the cost of the memory write (A guaranteed L1 hit)
> 
> If that were true then Spectre and Meltdown wouldn't have been so interesting 
> :)

I really don't follow your logic here.

> 
> Pipelining processors are going to speculatively execute both paths, and will 
> skip the write much more quickly than by doing it, and meanwhile nobody 
> should have tried to read the value so it hasn't had to block for that path. 
> I'm not aware of any that detect no-op writes and skip synchronising across 
> cores - the dirty bit of the cache line is just set unconditionally.

Processors only execute the one path, speculatively or otherwise, that's 
why branch prediction is so important.
(And is important for the Spectre attack. If what you said were true, 
the Spectre attack wouldn't need to pre-bias the branch predictor.)
I'm assuming a modern Intel processor.

> 
> Benchmarking already showed that the branching version is faster. It's 
> possible that "refcount += (refcount & IMMORTAL) ? 0 : 1" could generate 
> different code (should be mov,test,lea,cmovz rather than mov,and,add,mov or 
> mov,and,jz,add,mov), but it's totally reasonable for a branch to be faster 
> than unconditionally modifying memory.

It hasn't been shown that it is faster. I'm not claiming that it isn't, 
just that I haven't seen the evidence.

> 
> --
> 
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection

2020-04-20 Thread Tim Peters



Tim Peters  added the comment:

Allan, we don't (at least not knowingly) write tests that rely on order of 
end-of-life actions, because the _language_ defines nothing about the order.  
So you can permute the order and it's unlikely any standard tests would fail.

The only reason your example "works" outside of cyclic gc is because, as 
already noted, CPython (the implementation you're using, not Python the 
language) primarily relies on reference counting.  That guarantees __del__ is 
called first, but that's not a language guarantee, it's a consequence of 
reference counting invoking __del__ IMMEDIATELY upon the refcount falling to 0. 
 The _language_ not only doesn't guarantee that, it doesn't even guarantee that 
__del__ will ever be called.

For CPython (this implementation), it's important that we don't introduce 
gratuitous breakage of programs that are "working" because of implementation 
details.  How do you know that no program currently relies on the 
implementation detail that cyclic gc happens to run callbacks before finalizers 
now?  CPython has done that for many years, and I personally wouldn't risk 
breaking "working" programs by flipping that order now, not without truly 
compelling reason (example: stopping a segfault would be compelling).

Pablo, as above, I'm inclined to leave things alone unless we can "prove" no 
current code could possibly be relying (even by accident) on that gc currently 
runs callbacks before finalizers.  Which may be the case!  I don't know ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration



Serhiy Storchaka  added the comment:

I afraid that this is a part of the larger issue (see also issue31165). Seems 
the cause is that the GIL can be released when allocate or reallocate the 
memory. Does your project use some memory managing hooks?

Using sys.modules.copy().items() can help a lot, because you allocate memory 
for new dict only once. In case of list(sys.modules.items()) you allocate 
memory for every key-value pair, and may allocate memory several times to 
resize a list.

This reduces the probability of occurring the problem in many times, but not to 
zero. There is still a chance that the size of the original dict increase when 
you allocate a memory for a new dict, so copying the data to a new dict can 
fail or even crash or spoil the memory.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39849] Compiler warninig: warning: variable ‘res’ set but not used [-Wunused-but-set-variable]

2020-04-20 Thread Dong-hee Na



Dong-hee Na  added the comment:

Thanks hai shi, now the issue is fixed :)

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39849] Compiler warninig: warning: variable ‘res’ set but not used [-Wunused-but-set-variable]

2020-04-20 Thread Dong-hee Na



Dong-hee Na  added the comment:


New changeset 5dd21f5d1c9b5a9316deca4535932675f04efeee by Hai Shi in branch 
'master':
bpo-39849: Enable assertions in _testcapimodule.c and _testinternalcapi.c 
(GH-19623)
https://github.com/python/cpython/commit/5dd21f5d1c9b5a9316deca4535932675f04efeee


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection



Pablo Galindo Salgado  added the comment:

> Pablo, as above, I'm inclined to leave things alone unless we can "prove" no 
> current code could possibly be relying (even by accident) on that gc 
> currently runs callbacks before finalizers.  Which may be the case!  I don't 
> know ;-)

I very much agree with this. Also, (I think you already mentioned this) 
over-specifying the order of things in the gc may be a great way to shoot 
ourselves in the foot if we need to fix bugs or some odd behaviour during 
finalization/destruction (think for instance the latest bug regarding tp_clear 
and weakref callbacks).

I think we could at least improve somehow the docs, to say at least that the 
order is not specified so people that look at them do not rely on it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40288] atexit module should not be loaded more than once per interpreter

2020-04-20 Thread Dong-hee Na



Change by Dong-hee Na :


--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40339] Instead of skipping, IPV6 test(s) fail on a non-IPV6 machine

2020-04-20 Thread M T



New submission from M T :

I have no use for IPv6 and, when recompiling my OS, disable the feature 
completely. Python compiles nicely despite of this, but the IPv6-related tests 
fail instead of being skipped:

ERROR: test_create_server_ipv6 
(test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests)
--
Traceback (most recent call last):
  File 
"/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/test/test_asyncio/test_base_events.py",
 line 1149, in test_create_server_ipv6
self.loop.run_until_complete(main())
  File 
"/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/asyncio/base_events.py", 
line 488, in run_until_complete
return future.result()
  File 
"/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/asyncio/coroutines.py", 
line 129, in throw
return self.gen.throw(type, value, traceback)
  File 
"/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/test/test_asyncio/test_base_events.py",
 line 1141, in main
lambda: None, '::1', 0, loop=self.loop)
  File 
"/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/asyncio/streams.py", 
line 119, in start_server
return (yield from loop.create_server(factory, host, port, **kwds))
  File 
"/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/asyncio/base_events.py", 
line 1041, in create_server
infos = yield from tasks.gather(*fs, loop=self)
  File 
"/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/asyncio/base_events.py", 
line 990, in _create_server_getaddrinfo
flags=flags, loop=self)
  File 
"/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/concurrent/futures/thread.py",
 line 56, in run
result = self.fn(*self.args, **self.kwargs)
  File "/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/socket.py", line 
745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] hostname nor servname provided, or not known

==
FAIL: test_ipaddr_info (test.test_asyncio.test_base_events.BaseEventTests)
--
Traceback (most recent call last):
  File 
"/spare/usr/ports/lang/python36/work/Python-3.6.10/Lib/test/test_asyncio/test_base_events.py",
 line 104, in test_ipaddr_info
base_events._ipaddr_info('::3', 1, INET6, STREAM, TCP))
AssertionError: (, https://bugs.python.org/file49077/config.log

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection



Allan Feldman  added the comment:

I definitely understand the possibility that some code is relying on the 
current gc behavior of weakref callbacks being invoked after finalizers.

That being said, the behavior is currently inconsistent between gc and 
reference counted paths. The language doesn't have to define the explicit 
ordering but the internal inconsistency was definitely unexpected for us.

The result is that behavioral consistency becomes more difficult in application 
code when using language provided structures such as WeakValueDictionary.

cache = weakref.WeakValueDictionary()

class Bar:
pass

class Foo:
def __init__(self):
self._self = self
self.value = Bar()
cache[id(self.value)] = self.value

def __del__(self):
# the cache may or may not have self.value at this point
# even though self.value is strongly referenced!
print(list(cache.items()))


>From the weakref docs:
> Entries in the dictionary will be discarded when no strong reference to the 
> value exists any more.

But doesn't the code above imply that the entry is discarded even though there 
are strong references to the value?

In any case, I definitely appreciate all the eyes on this Tim + Pablo! At the 
very least, documentation updates do sound like a good idea if we're moving 
forward with leaving the behavior of weakrefs as currently specified. In 
particular, it would be worth pointing out that weakrefs callbacks can run even 
when the object is referenced.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40286] Add randbytes() method to random.Random



Raymond Hettinger  added the comment:

The randbytes() method needs to depend on genrandbits().  It is documented that 
custom generators can supply there own random() and genrandbits() methods and 
expect that the other downstream generators all follow.  See the attached 
example which demonstrates that randbytes() bypasses this framework pattern.

Also, I don't want randbytes() in the C extension.  We're tried to keep as much 
of the code as possible in pure Python and only have the MersenneTwister 
specific code in the C module.  The improves maintainability and makes the code 
more accessible to a broader audience.

Also, please don't change the name of the genrand_int32() function.  It was a 
goal to change as little as possible from the official, standard version of the 
C code at 
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html .  For 
the most part, we just want to wrap that code for Python bindings, but not 
modify it.

--
status: closed -> open
Added file: https://bugs.python.org/file49078/run_random.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40286] Add randbytes() method to random.Random



Raymond Hettinger  added the comment:

Direct link to MT code that I would like to leave mostly unmodified:  
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/CODES/mt19937ar.c

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection



Pablo Galindo Salgado  added the comment:

> The result is that behavioral consistency becomes more difficult in 
> application code when using language provided structures such as 
> WeakValueDictionary.


Well, you are already in tricky territory using finalizers. Note this sentence 
from the docs 
(https://docs.python.org/3/reference/datamodel.html#object.__del__):

> It is not guaranteed that __del__() methods are called for objects that still 
> exist when the interpreter exits.


So CPython does not even promise that __del__ will be called always!


> But doesn't the code above imply that the entry is discarded even though 
> there are strong references to the value?

No, because if there would be strong references then the refcount won't be 0 
and the object would not have been finalized. If the object is finalized is 
because nobody has more strong references. A WeakValueDictionary holds weak 
references in the values, that is why is called WeakValueDictionary ;) 

> In particular, it would be worth pointing out that weakrefs callbacks can run 
> even when the object is referenced.

That can be true but with a big caveat. There are two cases:

- Normal refcount falls to 0: the callback is executed when the deallocator is 
going to run, so nobody has strong references.

- GC: The gc calls the callback *before* the finalizer but at that point 
(before the finalizer) is unreachable from the outside, so all the references 
are from internal objects. Is still referenced but from unreachable objects 
that (unless resurrected) they are going to be destroyed by the gc. Notice that 
at the point the callback runs, all those objects are unreachable, so they 
cannot be referenced by anything in your program except themselves.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40314] python code order of magnitude faster than equivalent CPython code for simple import statement

2020-04-20 Thread Zachary Ware



Zachary Ware  added the comment:

Are you quite sure you're converting your times correctly, particularly in the 
C code?  The units in the Python version should be seconds; does the C version 
actually take 23 seconds to execute?  What kind of timing do you get if you 
time both programs with the same external program?

--
nosy: +zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40330] ShareableList size guard incorrect for str data



miss-islington  added the comment:


New changeset eba9f6155df59c9beed97fb5764c9f01dd941af0 by Antoine Pitrou in 
branch 'master':
bpo-40330: Fix utf-8 size check in ShareableList (GH-19606)
https://github.com/python/cpython/commit/eba9f6155df59c9beed97fb5764c9f01dd941af0


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection



Allan Feldman  added the comment:

Yup agree with all the points above, just wanted to point out that I think 
self.value is strongly referenced (even though it's just internal references 
and will be collected by the gc) during Foo.__del__ execution (annotated code 
below), yet the WeakValueDictionary entry is cleared:

import gc
import sys
import weakref

cache = weakref.WeakValueDictionary()


class Bar:
pass


class Foo:
def __init__(self):
self._self = self
self.value = Bar()
cache[id(self.value)] = self.value

def __del__(self):
print(sys.getrefcount(self.value))  # -> 2
# the cache may or may not have self.value at this point
# even though self.value is strongly referenced!
print(list(cache.items()))  # -> []


o = Foo()
del o
gc.collect()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40330] ShareableList size guard incorrect for str data



Change by Antoine Pitrou :


--
pull_requests: +18953
pull_request: https://github.com/python/cpython/pull/19625

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40339] Instead of skipping, IPV6 test(s) fail on a non-IPV6 machine

2020-04-20 Thread Karthikeyan Singaravelan



Karthikeyan Singaravelan  added the comment:

See also https://bugs.python.org/issue37199 for related work.

--
nosy: +ZackerySpytz, asvetlov, xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40237] Test code coverage (C) job of Travis CI fails on test_distutils which creates _configtest.gcno file

2020-04-20 Thread Brett Cannon



Brett Cannon  added the comment:

The historical background is code coverage of C code seemed like a good idea, 
so we set it up. :) Not much else to it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40330] ShareableList size guard incorrect for str data



Antoine Pitrou  added the comment:


New changeset 887ff8e37e238fbce18c647e588283904f38ab24 by Antoine Pitrou in 
branch '3.8':
[3.8] bpo-40330: Fix utf-8 size check in ShareableList (GH-19606) (GH-19625)
https://github.com/python/cpython/commit/887ff8e37e238fbce18c647e588283904f38ab24


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40330] ShareableList size guard incorrect for str data



Change by Antoine Pitrou :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40286] Add randbytes() method to random.Random



Raymond Hettinger  added the comment:

When a new method gets added to a module, it should happen in a way that is in 
harmony with the module's design.

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40340] Programming FAQ about "How do I convert a string to a number?" contains a typo



New submission from Dominik V. :

The paragraph about [How do I convert a string to a 
number?](https://docs.python.org/3/faq/programming.html#how-do-i-convert-a-string-to-a-number)
 contains the following sentence:

> By default, these interpret the number as decimal, so that `int('0144') == 
> 144` and `int('0x144')` raises ValueError.

The first part however doesn't raise an error. Most likely octal notation was 
meant, i.e. `int('0o144') == 144`.

For consistency with the `int('0x144')` part one could also omit the equality 
comparison, i.e. just write `int('0o144')`.

In order to emphasize that the "and" is not part of the code (though this 
should be displayed by the browser) once could also write:

> [...] so that _both_ `int('0o144')` and `int('0x144')` raise ValueError.

(emphasis added)

--
assignee: docs@python
components: Documentation
messages: 366870
nosy: Dominik V., docs@python
priority: normal
severity: normal
status: open
title: Programming FAQ about "How do I convert a string to a number?" contains 
a typo
type: enhancement
versions: Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40340] Programming FAQ about "How do I convert a string to a number?" contains a typo

2020-04-20 Thread Cajetan Rodrigues



Cajetan Rodrigues  added the comment:

> these interpret the number as decimal

I'm no linguist, but I feel both expressions in the subordinate clause have 
their purposes, with respect to the the main clause:

* `int('0144') == 144` to show what works
* `int('0x144')` to show what does not work

I don't believe the octal notation was meant in the first example, as the 
equality wouldn't then hold, even if somehow the typecast worked (which it 
doesn't). The point was to positively reinforce the idea that the content of 
the string would be interpreted as a decimal. So something like 
`int('00144') == 144` would still hold True. Negative reinforcement is 
provided in the second example.

But I agree that the sentence itself does not clearly separate the two 
examples, so I would suggest adding a comma after the first example, like so:

> By default, these interpret the number as decimal, so that `int('0144') == 
> 144`, and `int('0x144')` raises ValueError.

--
nosy: +cajetan.rodrigues

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39942] Making instance of `TypeVar` fails because of missing `name`

2020-04-20 Thread Ivan Levkivskyi



Ivan Levkivskyi  added the comment:


New changeset a25a04fea5446b1712cde0cff556574be139285a by HongWeipeng in branch 
'master':
bpo-39942:Fix failure in `TypeVar` when missing `__name__` (GH-19616)
https://github.com/python/cpython/commit/a25a04fea5446b1712cde0cff556574be139285a


--
nosy: +levkivskyi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39942] Making instance of `TypeVar` fails because of missing `name`



Change by miss-islington :


--
pull_requests: +18955
pull_request: https://github.com/python/cpython/pull/19627

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39942] Making instance of `TypeVar` fails because of missing `name`



Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 3.0 -> 4.0
pull_requests: +18954
pull_request: https://github.com/python/cpython/pull/19626

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40312] Weakref callbacks running before finalizers in GC collection

2020-04-20 Thread Tim Peters



Tim Peters  added the comment:

Well, the refcounts on _everything_ cyclic gc sees are greater than 0.  Because 
if an object's refcount ever falls to 0 in CPython, reference counting deals 
with it immediately, and so it doesn't survive to participate in cyclic gc.

IOW, absolutely everything cyclic gc deals with is "strongly referenced" in the 
sense you're using:  absolutely everything cyclic gc deals with has a nonzero 
refcount.  Indeed, in a debug build, gc asserts early on that everything it 
sees has a refcount > 0.

It can be trash anyway.  Finding those "just internal references" is the 
_point_ of cyclic gc.  In your last example, the instant after you did `del o`, 
the instance of Foo, and the instance of Bar, _both_ became trash, period, 
despite that they both retained positive refcounts.

So you may as well object that the refcount of `self` is also greater than 0 
inside __del__.  Indeed, add

print(sys.getrefcount(self))

inside your __del__, and you'll see it prints 4.  So it's _twice_ as "strongly 
referenced" as self.value ;-)  It's being destroyed anyway.

Refcounts are simply irrelevant at this point, and so also is the concept of 
"strong reference" on its own.  All that matters is whether a strong reference 
exists from _outside_ the generation being collected.  `self` and `self.value` 
are in exactly the same boat in this regard:  there are no strong references to 
either from outside, but there are strong references to both from inside.  
They're both trash.

There is _a_ principled way to proceed that would get you what you want in this 
example, but CPython never pursued it because it would add major complexity for 
almost no apparent practical gain:  build a graph of all the cyclic trash, 
compute the strongly connected components, build the induced DAG composed of 
the SCCs, then run end-of-life actions in order of a topological sort of that 
DAG.  Then __del__ would run first (self is in a self-cycle SCC with no 
predecessors in the DAG), and the weakref callback second (the Bar instance 
would be in its own SCC, with the `self` SCC as its predecessor in the DAG).

But I'm not sure any real gc system does that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39942] Making instance of `TypeVar` fails because of missing `name`



miss-islington  added the comment:


New changeset 694a95ff437613e6295f531336b5bc7cab9e3713 by Miss Islington (bot) 
in branch '3.7':
bpo-39942:Fix failure in `TypeVar` when missing `__name__` (GH-19616)
https://github.com/python/cpython/commit/694a95ff437613e6295f531336b5bc7cab9e3713


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40340] Programming FAQ about "How do I convert a string to a number?" contains a typo



Dominik V.  added the comment:

Indeed, thanks for clarifying. It seems I misunderstood the example, and 
perhaps that calls for a better separation. What about adding

> By default, these interpret the number as decimal, so that `int('0144') == 
> 144` holds true and `int('0x144')` raises ValueError.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39942] Making instance of `TypeVar` fails because of missing `name`



miss-islington  added the comment:


New changeset 41660cac63c1a216e43335007e329e213054100e by Miss Islington (bot) 
in branch '3.8':
bpo-39942:Fix failure in `TypeVar` when missing `__name__` (GH-19616)
https://github.com/python/cpython/commit/41660cac63c1a216e43335007e329e213054100e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40198] macOS Python builds from Python.org ignore DYLD_LIBRARY_PATH due to hardened runtime

2020-04-20 Thread dgelessus



dgelessus  added the comment:

I can confirm that the newly released Python 2.7.18 has the 
.allow-dyld-environment-variables entitlement:

$ ./python2.7 --version
Python 2.7.18
$ codesign --display --entitlements=:- python2.7
Executable=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

http://www.apple.com/DTDs/PropertyList-1.0.dtd";>


com.apple.security.cs.allow-dyld-environment-variables

com.apple.security.cs.disable-library-validation

com.apple.security.cs.disable-executable-page-protection




and accepts DYLD_LIBRARY_PATH again:

$ DYLD_LIBRARY_PATH=tests/objc ./python2.7 -c 'import os; 
print(os.environ.get("DYLD_LIBRARY_PATH"))'
tests/objc

Thank you for the fix!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40327] list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration



Change by Raymond Hettinger :


--
keywords: +patch
pull_requests: +18956
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19628

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39942] Making instance of `TypeVar` fails because of missing `name`

2020-04-20 Thread Ivan Levkivskyi



Change by Ivan Levkivskyi :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40341] Programming FAQ includes actively discouraged solutions; Should those be removed?


New submission from Dominik V. :

The Programming FAQ contains multiple solutions (examples) which it describes 
as "shouldn't be used". The question is why are these included in the first 
place? Some of them are complicated in a way that a (new) programmer is 
unlikely to discover them by themselves.

Below I include all the relevant parts, since I wasn't sure whether to open a 
new issue for each them.

# [How do I write a function with output parameters (call by 
reference)?](https://docs.python.org/3/faq/programming.html#how-do-i-write-a-function-with-output-parameters-call-by-reference)

Among others it contains these two list items:

> 2. By using global variables. This isn’t thread-safe, and is not recommended.
> [...]
> 5. Or bundle up values in a class instance: [...] There’s almost never a good 
> reason to get this complicated.

Especially number (5) is a pretty obscure way of solving that particular 
problem (even more so since a perfectly viable solution exists with (1), which 
is again recommended at the end of the paragraph). This additional 
recommendation of (1) at the end of the paragraph feels only necessary to draw 
attention away from the above do-not-use-solutions.

Also solutions (3) and (4) are basically equivalent in a sense that they rely 
on mutable builtin objects. So either they could be merged in one example or 
the list version could be left out altogether.

# [How do I use strings to call 
functions/methods?](https://docs.python.org/3/faq/programming.html#how-do-i-use-strings-to-call-functions-methods)

The last bullet point:

> Use locals() or eval() to resolve the function name: [...] Note: Using eval() 
> is slow and dangerous. If you don’t have absolute control over the contents 
> of the string, someone could pass a string that resulted in an arbitrary 
> function being executed.

This solution proposes to use `eval` and then actively discourages its use 
later on. Instead it could mention `globals` as an analogy to `locals` with the 
example of retrieving a globally defined function in another namespace.

# [How can I sort one list by values from another 
list?](https://docs.python.org/3/faq/programming.html#how-can-i-sort-one-list-by-values-from-another-list)

The second part of the paragraph speaks about using a `for` loop with repeated 
`append` instead of using the previously mentioned list comprehension. It then 
speaks about the many reasons why the usage of a `for` loop is discouraged 
here, so it seems strange that it was mentioned in first place. Also, right 
now, 50% of the whole section are devoted to an analysis about what not to do 
for the last step which distracts from the actual solution. IMO it is 
sufficient to just show the list comprehension.

--
assignee: docs@python
components: Documentation
messages: 366878
nosy: Dominik V., docs@python
priority: normal
severity: normal
status: open
title: Programming FAQ includes actively discouraged solutions; Should those be 
removed?
type: enhancement
versions: Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39939] PEP 616: Add str methods to remove prefix or suffix



STINNER Victor  added the comment:

The documentation should explain well the difference between 
removeprefix()/removesuffix() and lstrip()/strip()/rstrip(), since it is the 
rationale of the PEP ;-)

An example that can be used to explain the difference:

>>> "Monty Python".removesuffix(" Python")
'Monty'
>>> "Monty Python".strip(" Python")
'M'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40342] Programming FAQ about "How do I apply a method to a sequence of objects?" should include the option of an explicit for-loop



New submission from Dominik V. :

Right now the question is simply answered with:

> result = [obj.method() for obj in mylist]

However this is only appropriate if the result of the method is needed (i.e. if 
it's some kind of transformation).

There are many cases where it's not and the method is meant to update the 
object in place. Here it's better to use a for loop:

for obj in mylist:
obj.update()

Sometimes people use a one-way list comprehension hack because it saves one 
line:

[obj.update() for obj in mylist]

However this is discouraged for multiple reasons (builds a superfluous list, 
obfuscates the actual intent, ...).

So I feel like the Programming FAQ should actively mention this scenario and 
promote the usage of a for loop here.

--
assignee: docs@python
components: Documentation
messages: 366880
nosy: Dominik V., docs@python
priority: normal
severity: normal
status: open
title: Programming FAQ about "How do I apply a method to a sequence of 
objects?" should include the option of an explicit for-loop
type: enhancement
versions: Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40341] Programming FAQ includes actively discouraged solutions; Should those be removed?



Raymond Hettinger  added the comment:

I rather like the discussion in [How do I write a function with output 
parameters].  Some variant of each of those solutions arise in code reviews 
over and over again.  Rarely do the occur in simplistic examples, but they do 
occur.  The simple examples given are the easiest way to learn about the 
anti-patterns.

In the section [How do I use strings to call functions/methods?], let's 
eliminate the eval() variant which isn't really helpful.  The locals() 
technique does occur in practice and is worth showing.

In the section [How can I sort one list by values from another list], I concur 
that we can completely eliminate "alternative" and its explanatory text.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39939] PEP 616: Add str methods to remove prefix or suffix



STINNER Victor  added the comment:

When, I even expect that some people use .strip() whereas their intent was to 
use .lstrip():

>>> "Python vs Monty Python".strip("Python")
' vs Monty '

Again, strip() is used with a string whereas the real intent was to use 
removesuffix() which didn't exist ;-)

A note should be added to lstrip(), strip() and rstrip() documentation to point 
to removeprefix() and/or removesuffix().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40120] Undefined C behavior going beyond end of struct via a [1] arrays.



Antoine Pitrou  added the comment:

Another possibility yet would be:

typedef struct {
PyObject_VAR_HEAD
Py_hash_t ob_shash;
char ob_sval;
} PyBytesObject;

#define PyBytes_AS_STRING(op) (assert(PyBytes_Check(op)), \
&(((PyBytesObject *)(op))->ob_sval))

Not sure whether that would be UB...

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40343] Programming FAQ about "How do I call a method defined in a base class from a derived class that overrides it?" should mention the no-arguments-version of `super`



New submission from Dominik V. :

Right now it contains the following example:

class Derived(Base):
def meth(self):
super(Derived, self).meth()

`super()` without arguments is beneficial for multiple reasons, so it should be 
used in the example.

Also the paragraph speaks about versions prior 3.0 which seems strange because

1. the page is served at https://docs.python.org/3/faq/programming.html i.e. 
corresponding to version Python 3
2. Python 2 maintenance has been finally dropped.

The provided example is still useful though, for example in multiple 
inheritance scenarios (though these are very specific and `super()` of course 
also works if base classes are compatible). So perhaps it's better left out?

--
assignee: docs@python
components: Documentation
messages: 366884
nosy: Dominik V., docs@python
priority: normal
severity: normal
status: open
title: Programming FAQ about "How do I call a method defined in a base class 
from a derived class that overrides it?" should mention the 
no-arguments-version of `super`
type: enhancement
versions: Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40342] Programming FAQ about "How do I apply a method to a sequence of objects?" should include the option of an explicit for-loop



Raymond Hettinger  added the comment:

The current answer seems reasonable to me.  It addresses the question that most 
people want to have answered.  Also, it isn't common to loop over methods that 
don't return values, because those typically do in-place mutations.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39939] PEP 616: Add str methods to remove prefix or suffix



Raymond Hettinger  added the comment:

Please add an underscore to the names:  remove_prefix(). and remove_suffix().

The latter method causes a mental hiccup when first reading as removes-uffix, 
forcing mental backtracking to get to remove-suffix.

We had a similar problem with addinfourl initially being read as add-in-four-l 
before mentally backtracking to add-info-url.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40344] Programming FAQ about "What is the most efficient way to concatenate many strings together?" -- Improving the example



New submission from Dominik V. :

The section mentions the usage of `str.join` and contains the following example:

chunks = []
for s in my_strings:
chunks.append(s)
result = ''.join(chunks)

Since `join` accepts any iterable the creation of the `chunks` list in a for 
loop is superfluous. If people just copy & paste from this FAQ they'll even end 
up with less performant code.

The example could be improved by providing an example list such as:

strings = ['spam', 'ham', 'eggs']
meal = ', '.join(strings)

Arguably this isn't a particularly long list of strings, so one more example 
could be added using e.g. `range(100)`:

numbers = ','.join(str(x) for x in range(100))

This also emphasizes the fact that `join` takes any iterable rather than just 
lists.

--
assignee: docs@python
components: Documentation
messages: 366887
nosy: Dominik V., docs@python
priority: normal
severity: normal
status: open
title: Programming FAQ about "What is the most efficient way to concatenate 
many strings together?" -- Improving the example
type: enhancement
versions: Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40344] Programming FAQ about "What is the most efficient way to concatenate many strings together?" -- Improving the example



Dominik V.  added the comment:

Here's the link to the relevant section: 
https://docs.python.org/3/faq/programming.html#what-is-the-most-efficient-way-to-concatenate-many-strings-together

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40345] Programming FAQ about "How do I iterate over a sequence in reverse order?" should be more precise about `reversed`

New submission from Dominik V. :

https://docs.python.org/3/faq/programming.html#how-do-i-iterate-over-a-sequence-in-reverse-order

It contains the following example:

for x in reversed(sequence):
... # do something with x ...

With the note:

> This won’t touch your original sequence, but build a new copy with reversed
> order to iterate over.

The part about "build a new copy" is not correct in a sense that `reversed`
just returns an iterator over the original sequence. This has mainly two
consequences:

1. It can't be indexed, i.e. `reversed(sequence)[0]` doesn't work.
2. Changing the original sequence after `r = reversed(sequence)` has been
constructed, is reflected in `r` when iterating over it.

So the sentence should be changed into something like:

> This creates an iterator object that can be used to iterate over the original
> sequence in reverse order.

Then for the second example about `sequence[::-1]` it would be good to mention
the difference to `reversed`, namely that this version *does* create a copy of
the original list (in reverse order). It could also be used as an opportunity
to show how to reverse a string, since that is a very popular question on
StackOverflow.

Also the various mentions of Python versions 2.3 and 2.4 seem strange since
this is documentation about Python 3 and those version are anyway very old. So
they should be left out as well.

--
assignee: docs@python
components: Documentation
messages: 366889
nosy: Dominik V., docs@python
priority: normal
severity: normal
status: open
title: Programming FAQ about "How do I iterate over a sequence in reverse
order?" should be more precise about `reversed`
type: enhancement
versions: Python 3.8, Python 3.9

___
Python tracker

___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39939] PEP 616: Add str methods to remove prefix or suffix



STINNER Victor  added the comment:

> Please add an underscore to the names:  remove_prefix(). and remove_suffix().

The PEP 616 was approved with removeprefix() and removesuffix() names. The 
rationale for the names can be even found in the PEP:
https://www.python.org/dev/peps/pep-0616/#alternative-method-names

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40344] Programming FAQ about "What is the most efficient way to concatenate many strings together?" -- Improving the example



Raymond Hettinger  added the comment:

Dominik, can you please limit your tracker issues to a handful on entries that 
you really care about?  This is turning into a stream of consciousness dump 
onto our tracker.  You really don't need to rewrite every entry you see, 
especially when we haven't had user complaints about the existing entries.

Also, it seems to me that you're missing the point of the simplified examples 
in the docs.  Yes, of course, the for-loop in chunks is superfluous; however, 
that for-loop is very common pattern, but it typically does other work in the 
loop.  For example:

blocks = []
while True:
block = s.recv(4096)
if not block:
break
blocks.append(block)
page = b''.join(blocks)

The problem with that example is that it shifts focus to tcp clients rather 
than the core topic to how to join strings.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40346] Redesign random.Random class inheritance