[issue41586] Allow to set pipe size on subprocess.Popen.

[Ruben Vorderman]

Change by Ruben Vorderman :


--
keywords: +patch
pull_requests: +21035
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/21921

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41584] Clarify documentation for binary arithmetic operation subclass __r*__ precedence

[Eric Wieser]

Change by Eric Wieser :


--
nosy: +Eric Wieser

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39345] Py_Initialize Hangs on Windows 10

[Fieschi]

Fieschi  added the comment:

Personally, I have the same problem of Py_Initialize() hanging indefinitely.

Here is the context in which it happens : I am developing an application in 
Java, in which I use the library jep (https://github.com/ninia/jep), that 
enables me to get a Python interpreter from Java, and I am developing and 
testing it on Windows. My Python version is 3.8.2 and I am on Windows 10 - 
version 1903.
When I test this library outside my app in a simple Java project, everything 
works fine, and the interpreter works. 
But when I try to use it in the app, it hangs indefinitely when I create the 
interpreter. When I digged into the code of the library, I found out that it 
occurs in the native code of jep, during the call to Py_Initialize(). I posted 
an issue on the github of jep, and they brought me here. I bet this is related 
to stdin and stdout when I see what dhamilton posted.
My Java's stdout is normal and writes in the console. I tried to reset or 
redirect Java's stdin an stdout, but it doesn't change nothing.

And when I try to do this on Linux (my application is also on Linux), on Ubuntu 
16, everything works fine and it doesn't hang indefinitely. So this only 
happens on Windows.

About capturing a process dump, all I can get is a message displayed on the 
Java console when I close the app (because it hangs indefinitely) :
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x302b9d8f, pid=11960, 
tid=0x3f98
#
# JRE version: Java(TM) SE Runtime Environment (8.0_241-b07) (build 
1.8.0_241-b07)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.241-b07 mixed mode 
windows-amd64 compressed oops)
# Problematic frame:
# C  0x302b9d8f
#
# Failed to write core dump. Minidumps are not enabled by default on client 
versions of Windows

# The crash happened outside the Java Virtual Machine in native code.

All of this is just my personal case, and it's probably not the same for 
dhamilton. I hope it helped you.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34410] itertools.tee not thread-safe; can segfault interpreter when wrapped iterator releases GIL

[Irit Katriel]

Irit Katriel  added the comment:

This seems resolved, can it be closed?

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41574] Python[11231:143796] WARNING: running implicitly; please run panels using NSSavePanel rather than NSApplication.

[Ronald Oussoren]

Ronald Oussoren  added the comment:

the version of Tk in /System is not relevant, as I wrote the Tkinter in the 
Python.org installer uses a copy of Tcl/Tk that's installed by the Python.org 
installer (somewhere in /Library/Frameworks/Python.framework).

This can be checked using the otool(1) command:

$ otool -vL 
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_tkinter.cpython-38-darwin.so
 
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_tkinter.cpython-38-darwin.so:
/Library/Frameworks/Python.framework/Versions/3.8/lib/libtcl8.6.dylib 
(compatibility version 8.6.0, current version 8.6.8)
/Library/Frameworks/Python.framework/Versions/3.8/lib/libtk8.6.dylib 
(compatibility version 8.6.0, current version 8.6.8)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
version 1197.1.1)


As you can see _tkinter is linked with Tcl/Tk 8.6 inside the Python framework. 

I'm also using the Python.org installer (although the system I tested on has a 
slightly out of date copy of Python 3.8) and don't get this warning. That might 
be because I've been testing something different from what you're doing. 

Could you attach a script that demonstrates the problem?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[李超然]

New submission from 李超然 :

We find an issue while using shared memory. When opening another process to 
overwrite the shared memory, the memory of this process will increase to about 
the size of this shared memory. So when several processes try to read or write 
the shared memory, there will be N times memory usage.

```
import os, psutil
import numpy as np
from multiprocessing import Process, shared_memory

SHM_SIZE = 10 * 30 * 20


def f(name):
print('[Sub] (Before) Used Memory of {}: {} MiB'.format(
os.getpid(),
psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024,
))
shm = shared_memory.SharedMemory(name=name)
b = np.ndarray(shape=(SHM_SIZE, 1), dtype=np.float64, buffer=shm.buf)
for i in range(SHM_SIZE):
b[i, 0] = 1
print('[Sub] (After) Used Memory of {}: {} MiB'.format(
os.getpid(),
psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024,
))


def main():
print('[Main] Used Memory of {}: {} MiB'.format(
os.getpid(),
psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024,
))
shm = shared_memory.SharedMemory(create=True, size=8*SHM_SIZE, name='shm')
p = Process(target=f, args=('shm',))
p.start()
p.join()
print('[Main] Used Memory of {}: {} MiB'.format(
os.getpid(),
psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024,
))
input()
shm.close()
shm.unlink()


if __name__ == '__main__':
main()
```

--
components: Library (Lib)
messages: 375642
nosy: seraphlivery
priority: normal
severity: normal
status: open
title: Potential memory leak while using shared memory
type: resource usage
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41574] Python[11231:143796] WARNING: running implicitly; please run panels using NSSavePanel rather than NSApplication.

[Prudent]

Prudent  added the comment:

Sorry, maybe I missed to share a critical point that I wrote Python codes using 
“VS Code” software, rather than IDEL. That warning only showed in VS Code 
software when I ran it. After I re-opened my code in IDEL and ran it again. No 
any warnings or errors existed at all.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41574] Python[11231:143796] WARNING: running implicitly; please run panels using NSSavePanel rather than NSApplication.

[Ronald Oussoren]

Ronald Oussoren  added the comment:

Weird. Maybe VS Code uses a different Python installation? You can add "import 
sys; print(sys.prefix)" to your script to check.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41584] Clarify documentation for binary arithmetic operation subclass __r*__ precedence

[Vedran Čačić]

Vedran Čačić  added the comment:

Yes, there is a corner case here. If A derives from B and B derives from C, A 
doesn't implement __rsub__ and B does, A()-C() will call B's __rsub__.

--
nosy: +veky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41584] Clarify documentation for binary arithmetic operation subclass __r*__ precedence

[Vedran Čačić]

Vedran Čačić  added the comment:

Sorry, of course I meant C()-A(). The point is that the class A _doesn't_ have 
to implement __rsub__.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41588] Potential Memory leak with concurrent.futures.ThreadPoolExecutor's map

[Or Yahalom]

New submission from Or Yahalom :

I've been debugging a high memory consumption in one of my scripts and traced 
it back to the `concurrent.futures.ThreadPoolExecutor`.

When further investigating and playing around, I found out that when using 
`concurrent.futures.ThreadPoolExecutor` with the map function, and passing a 
dictionary to the map's function as an argument, the memory used by the pool 
won't be freed and as a result the total memory consumption will continue to 
rise. (Seems like it also happens when passing a list and maybe even other 
types).

Here is an example of a code to recreate this issue:

```
#!/usr/bin/env python3

import os
import time
import psutil
import random
import concurrent.futures

from memory_profiler import profile as mem_profile

p = psutil.Process(os.getpid())

def do_magic(values):
return None

@mem_profile
def foo():
a = {i: chr(i) for i in range(1024)}
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
proccessed_data = pool.map(do_magic, a)

def fooer():
while True:
foo()
time.sleep(1)

fooer()
```

--
components: Extension Modules
messages: 375647
nosy: or12
priority: normal
severity: normal
status: open
title: Potential Memory leak with concurrent.futures.ThreadPoolExecutor's map
type: resource usage
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[Eric V. Smith]

Eric V. Smith  added the comment:

It's likely that the same memory is being counted by both processes, to the 
output is misleading. Shared memory is notoriously difficult to allocate 
per-process. For example, it's definitely true that the shared memory is 
consuming virtual address space in both processes. That's often the value 
that's reported. But since the actual memory is shared, it's not consuming more 
physical memory than expected.

--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[李超然]

李超然  added the comment:

You mean if I have a machine that has 16GB RAM, and the maximum shared memory 
size is 8GB. I then create two processes to write to this shared memory, and 
the system won't run out of memory. Is it right?
I can try this experiment later on. But I can not understand this. I can see 
this memory increment in `htop`. I think the memory stat from `htop` is 
correct, otherwise, any system monitor software could deliver a fake warning 
message about the memory.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[Eric V. Smith]

Eric V. Smith  added the comment:

You'll have to play with it. I'm just saying that it's a very complicated 
subject, and not as simple as asking how much memory an individual process is 
using. For example, see 
https://www.howtogeek.com/659529/how-to-check-memory-usage-from-the-linux-terminal/
 for the many statistics that are involved.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41513] Scale by power of two in math.hypot()

[Tim Peters]

Tim Peters  added the comment:

Here's an amusing cautionary tale:  when looking at correct-rounding failures 
for the fsum approach, I was baffled until I realized it was actually the 
_decimal_ method that was failing.  Simplest example I have is 9 instances of 
b=4.999, which is 1 ulp less than 5. Of course the best-possible 
hypot is 3*b rounded, but that's not what default decimal precision computes 
(when rounded back).  In fact, it bounces between "off by 1" and "correct" 
until decimal precision exceeds 96:

>>> pfailed = []
>>> for p in range(28, 300):
... with decimal.localcontext() as c:
... c.prec = p
... if float(sum([decimal.Decimal(b) ** 2] * 9).sqrt()) != 3*b:
... pfailed.append(p)
>>> pfailed
[28, 29, 30, 31, 34, 36, 37, 39, 41, 42, 43, 44, 45, 57, 77, 96]

In a nutshell, it takes in the ballpark of 50 decimal digits to represent b 
exactly, then twice as many to represent its square exactly, then another to 
avoid losing a digit when summing 9 of those.  So if decimal prec is less than 
about 100, it can lose info.

So the fsum method is actually more reliable (it doesn't lose any info until 
the fsum step).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[李超然]

李超然  added the comment:

Okay. I know this is complicated. So how can I make sure this is not an issue? 
Can you provide some steps or a bash script to prove that the memory increment 
issue does not exist? I'm now not being persuaded because I don't know how to 
prove there is no issue about the memory. Will the experiment in the last 
message prove it? Or maybe you can provide an simple example showing there is 
no memory issue. Thank you.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41585] policy.max_line_length is incorrectly assumed to never be None

[Irit Katriel]

Irit Katriel  added the comment:

Are you able to post a script that reproduces the issue?

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40553] Python 3.8.2 Mac freezing/not responding when saving new programs

[Alexander Boeckmann]

Alexander Boeckmann  added the comment:

Hey! I have the exact same thing as described happening! Python freezes when I 
try to save new files which sucks. 
I'm on Catalina: 10.15.6
IDLE: 3.8.5
Brand new MacBook Air
As said earlier the only way out is to force quit. Would taking a video help?
I'll make one anyways

--
nosy: +aboeckmann

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[Eric V. Smith]

Eric V. Smith  added the comment:

Sorry, I don't have any particular suggestion other than accounting for all 
virtual, shared, and physical memory of all types, and seeing how they're being 
used and allocated per-process by the various tools.

There are probably guides for this on the internet, but I haven't been able to 
find any with a quick search.

I'm not saying there isn't a problem, but I'm saying it's going to require more 
analysis before we can verify that a problem does exist.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41568] test_zoneinfo leaked [84, 84, 84] references

[STINNER Victor]

STINNER Victor  added the comment:

https://buildbot.python.org/all/#/builders/84 is back to green, I close the 
issue.

--
nosy: +vstinner
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41025] C implementation of ZoneInfo cannot be subclassed

[STINNER Victor]

STINNER Victor  added the comment:

> test_zoneinfo leaked [84, 84, 84] references, sum=252

Reported as bpo-41568 and fixed there.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[李超然]

李超然  added the comment:

Sorry, I don't know what I can do about it. In my perspective, I think there is 
a memory leak because monitor software have provided proof for me to believe 
that. 
I have provided a script to reproduce this issue. I think that is enough for 
developers to conduct other tests to verify whether this issue does exist. I 
don't think I can provide any more information since you can reproduce this 
problem. 
As a user, I just need some advice about how to fix this issue or confirmation 
about this issue, or some explanations that make me believe that there is no 
issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[Eric V. Smith]

Eric V. Smith  added the comment:

Okay. We'll see if someone else can provide more info.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41589] Strange behavior with sparse.dok_matrix decimal is cast to integer

[browser.365]

New submission from browser.365 :

import numpy as np
from scipy import sparse
import decimal
D = decimal.Decimal

Al = sparse.dok_matrix((10, 10), dtype=np.dtype(D))
Al.astype(D)

Al[1,1] = D('0.1')
print(Al[1,1])
print(type(Al[1,1]))

Al[0,0] = D('0')
print(Al[0,0])
print(type(Al[0,0]))



z = decimal.Decimal('0')

print(z)
print(type(z))
print(' - ')


Running the above code gives:

0.1

0

0

 - 

All of the elements should be decimal.

--
messages: 375660
nosy: browser.365
priority: normal
severity: normal
status: open
title: Strange behavior with sparse.dok_matrix  decimal is cast to integer
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40204] Docs build error with Sphinx 3.0 due to invalid C declaration

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 8f88190af529543c84d5dc78f19abbfd73335cf4 by Victor Stinner in 
branch '3.9':
[3.9] bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) (GH-21901)
https://github.com/python/cpython/commit/8f88190af529543c84d5dc78f19abbfd73335cf4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41590] "zip()" very slowly for this

[email0.ya]

New submission from email0.ya :

https://docs.python.org/3.9/tutorial/datastructures.html
Nested List Comprehensions

--
assignee: docs@python
components: Documentation
files: nested_lists.py
messages: 375662
nosy: docs@python, email0.ya
priority: normal
severity: normal
status: open
title: "zip()" very slowly for this
type: performance
versions: Python 3.9
Added file: https://bugs.python.org/file49405/nested_lists.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26332] OSError: exception: access violation writing <...> (Windows 10 x64, Python 3.5.1)

[Irit Katriel]

Irit Katriel  added the comment:

Access violation is a very general error which means that a bad pointer was 
dereferenced in c++. This can happen in many ways. You need to provide a way to 
reproduce this error in order to make this an actual bug report.

In your case, it is happening when you call a function in a third party 
library, which could mean that this library has a bug or that you are passing 
bad parameters to it.

If you cannot provide more information than what is here, this ticket should be 
closed on the basis that there is not enough information do anything about the 
issue.

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41589] Strange behavior with sparse.dok_matrix decimal is cast to integer

[Eric V. Smith]

Eric V. Smith  added the comment:

This seems like a scipy or numpy issue, not a Python bug. You might have better 
luck asking about this behavior on a scipy or numpy forum of some kind, or 
maybe on Stackoverflow.

--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41591] Comprehensions documentation

[Walid Taha]

New submission from Walid Taha :

The documentation for list comprehensions contains the following phrase:

"As we saw in the previous section, the nested listcomp is evaluated in the 
context of the for that follows it, so this example is equivalent to:"

This should be corrected, as it currently contradicts what was said previously, 
which is that list comprehensions and the conditional they contain are scoped 
in the same order as they appear (rather than the reverse).

This issue can be found on this page: 
https://docs.python.org/3/tutorial/datastructures.html

It also seems to appear in the most recent version:
https://docs.python.org/3.10/tutorial/datastructures.html

To confirm that the first (and not the second statement) is correct, you may 
consider the following code:

l=[]
for x in range(0,3):
  for y in range (0,x+1):
l.append((x,y))
print(l)

l=[(x,y) for x in range (0,3) for y in range (0,x+1)]
print(l)

Which run on 3.7.5 produces the following output

[(0, 0), (1, 0), (1, 1), (2, 0), (2, 1), (2, 2)]
[(0, 0), (1, 0), (1, 1), (2, 0), (2, 1), (2, 2)]

--
assignee: docs@python
components: Documentation
messages: 375665
nosy: docs@python, wtaha
priority: normal
severity: normal
status: open
title: Comprehensions documentation
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41590] "zip()" very slowly for this

[Steven D'Aprano]

Steven D'Aprano  added the comment:

What are you actually reporting? What part of the documentation do you think 
should be changed, why should it be changed, and what should it be changed to?

It is normal for different algorithms to perform with different speed. I'm not 
sure what the purpose of the "nested_lists" file is. You have discovered that 
different programs perform differently. Okay. What do you want us to do?

One last comment: using time() as you did is unreliable, especially for small 
code snippets that are very fast. The best way to time small snippets of Python 
code is to use the timeit module.

--
nosy: +steven.daprano

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[Tim Peters]

Tim Peters  added the comment:

There's no evidence of a Python issue here, so I recommend closing this. It's 
not the Python bug tracker's job to try to make sense of platform-specific 
reporting tools, which, as already explained, can display exceedingly confusing 
numbers. We (the Python project) didn't write them, have no control over what 
they display, and have no special insight into them.

Here's a similar simple program that creates a 4 GiB block of shared memory, 
and passes it to 16 processes. I'm running this on Windows 10, on a machine 
with 4 physical cores and 16 GiB of RAM. If the memory weren't actually being 
shared, Windows would have killed the job (there's no "over allocation" allowed 
at all on Windows), because there's nowhere near 4 * 17 = 68 GiB of RAM 
available.

Instead, the program brings the machine to its knees, but because it created 16 
processes each of which is 100% CPU- and memory-bound. The Windows task manager 
detailed view shows less than a MiB of non-shared memory in use by each of the 
worker processes, with the 4 GiB in the "memory in use that can be shared with 
other processes" column for each worker process.  I'm running lots of other 
stuff too, and Windows still says there's over 5 GiB of my 16 GiB of RAM free 
for use.

Windows reporting has its own quirks. For example, the shared memory block 
doesn't show up at all in the workers at first. Instead the amount reported 
steadily increases, as the write loop in each worker process forces the OS to 
materialize shared pages in the worker's virtual address space (that is, 
Windows reports pages actually in use by a process, not virtual address space 
reservations).

from multiprocessing import Process, shared_memory

SHM_SIZE = 2 ** 32 # 4 GiB

def f(name):
shm = shared_memory.SharedMemory(name=name)
print(f"shm of size {shm.size:,}")
buf = shm.buf
for i in range(shm.size):
buf[i] = 123
shm.close()

def main():
shm = shared_memory.SharedMemory(create=True, size=SHM_SIZE, name='shm')
ps = []
for i in range(16):
p = Process(target=f, args=('shm',))
p.start()
ps.append(p)
for p in ps:
p.join()
shm.close()
shm.unlink()

if __name__ == '__main__':
main()

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41584] Clarify documentation for binary arithmetic operation subclass __r*__ precedence

[Brett Cannon]

Brett Cannon  added the comment:

Right, there is a `lhs.__rsub__ is not rhs.__rsub__` check which needs to be 
clearly communicated in that note.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41587] Potential memory leak while using shared memory

[李超然]

李超然  added the comment:

Thank you Tim Peters for replying to me. I tried your demo and it is a proof 
exactly I want. It did prove that there is no memory issue. And I tried to 
modify my own code and showed the same result. I will close this issue.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40204] Docs build error with Sphinx 3.0 due to invalid C declaration

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +21036
pull_request: https://github.com/python/cpython/pull/21924

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41592] Make _SubParsersAction public

[Eric Pederson]

New submission from Eric Pederson :

ArgumentParser.add_subparsers() returns a _SubParsersAction.  This requires 
user code using type annotations to use a protected type which causes type 
checkers like PyCharm to complain.  For example:

def add_subparser(name: str, subparser: _SubParsersAction, subparsers: dict) -> 
ArgumentParser:
parser = subparser.add_parser(name)
parser.add_argument('-v', '--verbose', action='store_true')
subparsers[name] = parser
return parser

You can't use plain Action because Action doesn't have the add_parser() method.

--
components: Library (Lib)
messages: 375670
nosy: sourcedelica
priority: normal
severity: normal
status: open
title: Make _SubParsersAction public
type: enhancement
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41502] Option for Colored Logging in http.server Module

[Mika Hawkins]

Mika Hawkins  added the comment:

Yes I totally agree that it would be useful if the http.server module had an 
option for colored logging, That would be extremly helpful/useful. Thank you 
for putting that in...

--
nosy: +Mika_Hawkins

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41590] "zip()" very slowly for this

[email0.ya]

email0.ya  added the comment:

Above is a link to part of the tutorial. An example with a for statement at the 
beginning is good. I am not saying that it needs to be replaced with something. 
The above example reads: "A list comprehension consists of brackets containing 
an expression followed by a for clause, then zero or more for or if clauses".

def transpose1_3_1 (a: list) -> list:
"" "Transpose matrix version 1.3.1" ""
max_j = len (max (a))
return [
[row [j] if j 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41590] "zip()" very slowly for this

[Vedran Čačić]

Vedran Čačić  added the comment:

First, what you wrote with if...else is _not_ "if clause". It's a conditional 
expression. You'll recognize them easily if you remember that "if clause" in 
comprehensions never has an "else", while conditional expression always does.

Second, speed is just one of many, many criteria by which we evaluate programs. 
If only speed matters to you, you probably shouldn't be writing in Python at 
all. There are much faster languages.

--
nosy: +veky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41590] "zip()" very slowly for this

[Vedran Čačić]

Vedran Čačić  added the comment:

Third, where did you get the idea that transposing a "matrix" consisting of 
lists of varying length is a sensible operation? Your reference (Wikipedia) 
tells nothing about that case. I've never seen [[],[1]] called "a matrix".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41550] SimpleQueues put blocks if feeded with strings greater than 2**16-13 chars

[Raphael Grewe]

Raphael Grewe  added the comment:

I used SimpleQueue on purpose because I only need the functions get and put. 
Of course I could also use Queue but that would be just a workaround for me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41550] SimpleQueues put blocks if feeded with strings greater than 2**16-13 chars

[Irit Katriel]

Irit Katriel  added the comment:

The get and put functions of Queue have the optional 'block' and 'timeout' args 
that you need, and SimpleQueue doesn't.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41593] pathlib PermissionError problem

[leonyxz]

New submission from leonyxz :

python 3.x
FreeBSD 11.3-RELEASE-p12 with custom settings mentioned later.
On vanilla configuration of FreeBSD there is no issue.
Installation of FreeBSD in question has mac_bsdextended turned on 
(https://www.freebsd.org/cgi/man.cgi?query=mac_bsdextended&sektion=4&apropos=0&manpath=FreeBSD+12.1-RELEASE+and+Ports)
Using ugidfw 
(https://www.freebsd.org/cgi/man.cgi?query=ugidfw&sektion=8&manpath=freebsd-release-ports),
 the following rule has been set:
---
0 subject not uid root gid nobody object gid wheel type d mode sx
---
The following code shows the minimal reproducible example:
---
from pathlib import Path
BASE_DIR = Path(__file__).resolve(strict=True).parent.parent
print(BASE_DIR)
---
Running the code directly works:
---
$ python test.py
/usr/home/example
---
But importing it as a module does not:
---
$ python -m test
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
  File "/usr/home/example/django_3136test/test.py", line 2, in 
BASE_DIR = Path(__file__).resolve(strict=True).parent.parent
  File "/usr/local/lib/python3.6/pathlib.py", line 1141, in resolve
s = self._flavour.resolve(self, strict=strict)
  File "/usr/local/lib/python3.6/pathlib.py", line 346, in resolve
return _resolve(base, str(path)) or sep
  File "/usr/local/lib/python3.6/pathlib.py", line 330, in _resolve
target = accessor.readlink(newpath)
  File "/usr/local/lib/python3.6/pathlib.py", line 440, in readlink
return os.readlink(path)
PermissionError: [Errno 13] Permission denied: '/usr'
---
The issue seems to be comnected with following piece code
https://github.com/python/cpython/blob/master/Lib/pathlib.py#L342-L346
---
try:
target = accessor.readlink(newpath)
except OSError as e:
if e.errno != EINVAL and strict:
raise
# Not a symlink, or non-strict mode. We just leave the path
# untouched.
path = newpath
---
With ugidfw enabled, `os.readlink` raises a `PermissionError`, which is 
unhandled by pathlib.

--
components: FreeBSD, Library (Lib)
messages: 375677
nosy: koobs, leonyxz
priority: normal
severity: normal
status: open
title: pathlib PermissionError problem
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41566] Include much faster DEFLATE implementations in Python's gzip and zlib libraries. (isa-l)

[Julia Frances]

Julia Frances  added the comment:

welcome to python.org in this website there are many ways to do python coding. 
You need to follow the guide and the rules on how to do python coding.

--
hgrepos: +392
keywords: +patch
message_count: 8.0 -> 9.0
nosy: +jfrances21
nosy_count: 3.0 -> 4.0
pull_requests: +21037
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18233] SSLSocket.getpeercertchain()

[JustAnotherArchivist]

Change by JustAnotherArchivist :


--
nosy: +JustAnotherArchivist

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41590] "zip()" very slowly for this

[Steven D'Aprano]

Steven D'Aprano  added the comment:

If you know about timeit, why aren't you using it?

In any case, I have just ran your nested_lists.py file, and on my 
computer, the last version with zip is the fastest version:

0 transpose1_0(lT) Time =  1.0627508163452149e-05
7 zip(*lT) Time =  1.5511512756347657e-06

1.55e-6 is smaller than 1.06e-5:

py> 1.55e-6 < 1.06e-5
True

Smaller times are faster, and the first version transpose1_0 takes 
nearly seven times longer to run than the last version with zip.

As for the other comments, the purpose of the tutorial is to teach the 
language in the simplest way possible. It is aimed at beginners, not 
intermediate or expert programmers. The purpose of this section is to 
demonstrate the basic features of list comprehensions, not to overwhelm 
the beginner with complicated details.

Making the example more complicated so that it is a tiny bit faster 
would not a good tradeoff for the tutorial, but in fact all the more 
complicated versions are slower, not faster.

You are also confused about the structure of comprehensions. The 
comprehension consists of 

expression-part for-part (optional for- and if-parts)

There is no else-part in the comprehension, and the for-part always 
comes before any if-part. Your code:

row[j] if j < len(row) else 0

is not an "if statement" as you call it, neither is it part of the 
comprehension syntax. It is the expression-part of the comprehension, 
containing a ternary if-expression. It is not part of the structure of 
the comprehension.

See the full language specification, in particular the grammar:

https://docs.python.org/3.9/reference/grammar.html

We could re-write the if-expression like this:

(j < len(row)) and row[j] or 0

That doesn't mean that comprehensions consist of an "and" part followed 
by an "or" part followed by a for-part. The "and" and "or" are just part 
of the expression part of the comprehension.

Adding this level of technical detail and complexity to an introductory 
tutorial aimed at beginners would not be a good idea.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41594] Intermittent failures of loop.subprocess_exec() to capture output

[Kyle Meyer]

New submission from Kyle Meyer :

I've been debugging an intermittent test failure in code that calls
`loop.subprocess_exec` with an `asyncio.SubprocessProtocol` subclass.
Here's a minimal example that I hope captures the issue.  It's based
closely off of the `DateProtocol` [example][1] in the asyncio protocol
documentation.

import asyncio

class Protocol(asyncio.SubprocessProtocol):
def __init__(self, exit_future):
self.exit_future = exit_future
self.output = bytearray()

def pipe_data_received(self, fd, data):
self.output.extend(data)

def process_exited(self):
self.exit_future.set_result(True)

async def get_stdout():
loop = asyncio.get_running_loop()
exit_future = asyncio.Future(loop=loop)
transport, protocol = await loop.subprocess_exec(
lambda: Protocol(exit_future),
"printf", "ok", stdin=None, stderr=None)
await exit_future
transport.close()
return bytes(protocol.output)

The attached script adds some debugging statements to what's above and
then repeats the specified number of calls to
`asyncio.run(get_stdout())`, aborting if `asyncio.run(get_stdout())`
returns an empty byte string rather then the expected `b'ok'`.  With
Python 3.7.3 on a Debian system, I see occasional aborts.  For
example, after two runs of the script with no unexpectedly empty
output, I saw

$ python3 reproducer.py 500
Iteration:  32 of 500
Failed on iteration 32
logfile: /tmp/asyncio-debug-bqehu8f3.log

$ tail /tmp/asyncio-debug-bqehu8f3.log
DEBUG: process_exited() called
DEBUG: Starting iteration 31
DEBUG: Using selector: EpollSelector
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: process_exited() called
DEBUG: Starting iteration 32
DEBUG: Using selector: EpollSelector
DEBUG: process_exited() called
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: Failed on iteration 32

Based on the debug statements, it looks like `pipe_data_received` does
get called with the output for the run that comes up empty, but only
after `process_exited` is called.

On my first try with a python built from a recent commit (0be7c216e1),
I triggered the failure:

$ python -V
Python 3.10.0a0

$ python reproducer.py 500
Iteration:   8 of 500
Failed on iteration 8
logfile: /tmp/asyncio-debug-m5fba4ga.log

$ tail /tmp/asyncio-debug-m5fba4ga.log
DEBUG: process_exited() called
DEBUG: Starting iteration 7
DEBUG: Using selector: EpollSelector
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: process_exited() called
DEBUG: Starting iteration 8
DEBUG: Using selector: EpollSelector
DEBUG: process_exited() called
DEBUG: pipe_data_received(): fd=1, data=b'ok'
DEBUG: Failed on iteration 8

As I'm following the example from the documentation closely, I hope
I'm not committing a silly error that leads to an expected race here.


[1]: 
https://docs.python.org/3/library/asyncio-protocol.html#loop-subprocess-exec-and-subprocessprotocol

--
components: asyncio
files: reproducer.py
messages: 375680
nosy: asvetlov, kyleam, yselivanov
priority: normal
severity: normal
status: open
title: Intermittent failures of loop.subprocess_exec() to capture output
type: behavior
versions: Python 3.7
Added file: https://bugs.python.org/file49406/reproducer.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41595] curses' window.chgat does not change color when specifying curses.A_COLOR

[Maarten]

New submission from Maarten :

Or'ing curses.A_COLOR in the `attr` argument of curses.window.chgat ends of 
with the line not showing.

Actual Problem:
The text is invisible/hidden/not shown.
Using only the attribute curses.A_BLINK is fine.

Expected:
The color of the line is changed +the text starts blinking.


I have attached a small python reproducer and a small c source that shows the 
correct behavior.
The c example should be compiled as `gcc chgat.c -lncurses -o chgat`.

This is using ncurses 6.1

The c code is from https://tldp.org/HOWTO/NCURSES-Programming-HOWTO/attrib.html,
and the python script is a translation of it.

--
components: Library (Lib)
files: chgat.py
messages: 375681
nosy: maarten
priority: normal
severity: normal
status: open
title: curses' window.chgat does not change color when specifying curses.A_COLOR
versions: Python 3.7
Added file: https://bugs.python.org/file49407/chgat.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41595] curses' window.chgat does not change color when specifying curses.A_COLOR

[Maarten]

Maarten  added the comment:

Equivalent c script of chgat.py

Compile with:
gcc chgat.c -lncurses -o chgat

--
Added file: https://bugs.python.org/file49408/chgat.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41595] curses' window.chgat does not change color when specifying curses.A_COLOR

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
nosy: +Jeffrey.Kintscher

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41566] Include much faster DEFLATE implementations in Python's gzip and zlib libraries. (isa-l)

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests:  -21037

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2889] curses for windows (alternative patch)

[Maarten]

Maarten  added the comment:

Current ncurses master is buildable on Visual Studio, using msys2.

It should be possible to create a vcxproject and integrate it in the Visual 
Studio build.

--
nosy: +maarten

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41566] Include much faster DEFLATE implementations in Python's gzip and zlib libraries. (isa-l)

[Gregory P. Smith]

Gregory P. Smith  added the comment:

fwiw, no PEP is needed for things like this.  it'd just be an alternative 
library implementing the core of the zlib and/or gzip modules behind the scenes.

normally this kind of thing would be done using a check for the availability of 
the library in configure.ac with conditional compilation via #ifdef's from the 
configure output that controls what is set in pyconfig.h in the zlib and gzip 
modules itself.

Within the stdlib, I'd focus only on using things that can be used in a 100% 
api compatible way with the existing modules.

Otherwise creating a new module and putting it up on PyPI to expose the 
functionality from the libraries you want makes sense and will be easier to 
make available to everyone on existing Python versions rather than waiting on 
getting something into CPython.

FWIW I tend to avoid software provided by Intel given any other choice.

Look instead at Chromium's zlib as discussed in 
https://github.com/madler/zlib/issues/346 or possibly at a project like 
https://github.com/zlib-ng/zlib-ng.  These are much more likely to be drop in 
zlib replacements making this simply a build time thing which is more a matter 
of making sure we get the relevant compiler flags correct when the faster 
libraries are detected by configure.ac as available.

There is a caveat to using any of these: how well maintained and security 
vetted are all of the code paths in the implementation?  zlib proper gets 
massive security attention.  Its low rate of change and staleness are a 
feature.  Chromium's zlib also gets proper security attention.  No idea about 
zlib-ng and even less about Intels self serving arm-ignoring oddity.

--
nosy: +gregory.p.smith
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34215] streams.py:IncompleteReadError message is unclear when expected is None

[Zackery Spytz]

Change by Zackery Spytz :


--
components: +asyncio
nosy: +ZackerySpytz
title: streams.py:IncompleteReadError is message is unclear when expected is 
None -> streams.py:IncompleteReadError message is unclear when expected is None
versions: +Python 3.10 -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34215] streams.py:IncompleteReadError message is unclear when expected is None

[Zackery Spytz]

Change by Zackery Spytz :


--
keywords: +patch
pull_requests: +21038
pull_request: https://github.com/python/cpython/pull/21925

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41593] pathlib PermissionError problem

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
nosy: +Jeffrey.Kintscher

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41585] policy.max_line_length is incorrectly assumed to never be None

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
nosy: +Jeffrey.Kintscher

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41513] Scale by power of two in math.hypot()

[Tim Peters]

Tim Peters  added the comment:

Just FYI, if the "differential correction" step seems obscure to anyone, here's 
some insight, following a chain of mathematical equivalent respellings:

result + x / (2 * result) =
result + (sumsq - result**2) / (2 * result) =
result + (sumsq - result**2) / result / 2 =
result + (sumsq / result - result**2 / result) / 2 =
result + (sumsq / result - result) / 2 =
result + sumsq / result / 2 - result / 2 =
result / 2 + sumsq / result / 2 =
(result + sumsq / result) / 2

I hope the last line is an "aha!" for you then:  it's one iteration of Newton's 
square root method, for improving a guess "result" for the square root of 
"sumsq". Which shouldn't be too surprising, since Newton's method is also 
derived from a first-order Taylor expansion around 0.

Note an implication: the quality of the initial square root is pretty much 
irrelevant, since a Newton iteration basically doubles the number of "good 
bits".  Pretty much the whole trick relies on computing "sumsq - result**2" to 
greater than basic machine precision.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41560] pathlib.Path.glob fails on empty string

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
nosy: +Jeffrey.Kintscher

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41564] Cannot access member "hex" for type "ByteString"

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
nosy: +Jeffrey.Kintscher
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25229] distutils doesn't add "-Wl, " prefix to "-R" on Linux if the C compiler isn't named 'gcc'

[Kenta Tsubouchi]

Kenta Tsubouchi  added the comment:

This patch seems good for proprietary compilers in linux.
Because such compilers need gcc-style link option.
So I'll make PR.

--
nosy: +ketsubouchi
type:  -> enhancement
versions:  -Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 
3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41574] Python[11231:143796] WARNING: running implicitly; please run panels using NSSavePanel rather than NSApplication.

[Prudent]

Prudent  added the comment:

It is also very weird to me. I checked the python version used in VS code, the 
version in use was 3.8. But it seems the used Tcl/Tk package still linked to 
the system-embedded Tck/Tk 8.5.9 on macOS. 

I found this description from "https://www.python.org/download/mac/tcltk/";, 
which may help us know more. Anyway, this warning doesn't affect running codes 
too much.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41566] Include much faster DEFLATE implementations in Python's gzip and zlib libraries. (isa-l)

[Ruben Vorderman]

Ruben Vorderman  added the comment:

> Within the stdlib, I'd focus only on using things that can be used in a 100% 
> api compatible way with the existing modules.

> Otherwise creating a new module and putting it up on PyPI to expose the 
> functionality from the libraries you want makes sense and will be easier to 
> make available to everyone on existing Python versions rather than waiting on 
> getting something into CPython.

Agreed. 100% backwards compatibility is more important than speed. And getting 
it available to users is faster as a module than implementing it in CPython. 

I already have a PR open in the xopen module to implement the use of the igzip 
program. Implementing a module like the gzip module in CPython will be more 
work, but I will certainly consider it. Thanks for the suggestion! 

> There is a caveat to using any of these: how well maintained and security 
> vetted are all of the code paths in the implementation?  zlib proper gets 
> massive security attention.  Its low rate of change and staleness are a 
> feature.

I didn't consider that. So I looked at the CVE page for ZLIB. The latest issues 
are from 2017. Before that 2005. This is the 2017 report: 
https://wiki.mozilla.org/images/0/09/Zlib-report.pdf. 
Note how it states that the old compiler support etc. are a basis for 
vulnerabilities. Precisely zlib-ng did get rid of these parts. On the other 
hand, Mozilla notes that Zlib is a great candidate for periodic security 
audits, precisely for the same reasons you mention.

> FWIW I tend to avoid software provided by Intel given any other choice.

I know the feeling. They rightfully have a very bad reputation for things they 
did in the past. But this particular code is open source and compilable with 
open source compilers. Part of it is written in Assembly, to get the speed 
advantage. I benchmarked it on my AMD processor and I too get enormous speed 
advantages out of it.

>  even less about Intels self serving arm-ignoring oddity.

They *do* provide instructions to build for arm. Right on their README. 
https://github.com/intel/isa-l#building-isa-l. I think it is very unfair to be 
so dismissive just because Intel pays the developers. This is good work, which 
speeds up bioinformatics workloads, which in turn helps us to help more 
patients.

On the whole I think the arguments to make a module are very strong. So I think 
that is the appropriate way forward. I'd love everyone to switch to more 
efficient deflate algorithms, but CPython may not be the right project to drive 
this change. At least this discussion is now here as a reference for other 
people who are curious about improving this performance aspect.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com