[issue36085] Enable better DLL resolution

[mattip]

mattip  added the comment:

Correct me if I'm wrong, but once SetDefaultDllDirectories() is used, there is 
no going back: PATH no longer can change the search path no matter what flags 
are used with LoadLibrary* calls (see the recent Anaconda issue when they did 
this and broke NumPy). Assuming that is true, then

> add sys._adddlldirectory() and sys._removedlldirectory() as CPython-specific 
> functions for extending the search path (for use by packages currently 
> modifying PATH at runtime)

Why is this CPython-specific and "private"? It seems like
* it should be a public interface, used by all implementations consistently, as 
a policy decision for the win32 platform and documented as such
* located in os, much like os.environ (not critical)

There should be some kind of debugging tool for when LoadLibraryEx fails, to 
indicate what might have gone wrong, perhaps os.getdlldirectory() would be 
helpful

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35704] On AIX, test_unpack_archive_xztar fails with default MAXDATA settings

[Michael Felt]

Change by Michael Felt :


--
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34373] test_time errors on AIX

[Michael Felt]

Change by Michael Felt :


--
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11192] test_socket error on AIX

[Michael Felt]

Change by Michael Felt :


--
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34720] Fix test_importlib.test_bad_traverse for AIX

[Michael Felt]

Change by Michael Felt :


--
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35892] Fix awkwardness of statistics.mode() for multimodal datasets

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset fc06a192fdc44225ef1cc879f615a81931ad0a85 by Raymond Hettinger in 
branch 'master':
bpo-35892: Fix mode() and add multimode() (#12089)
https://github.com/python/cpython/commit/fc06a192fdc44225ef1cc879f615a81931ad0a85


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35892] Fix awkwardness of statistics.mode() for multimodal datasets

[Raymond Hettinger]

Change by Raymond Hettinger :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36216] urlsplit does not handle NFKC normalization

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Sample buildbot log of print statement in testcase causing rerun of test : 
https://buildbot.python.org/all/#/builders/101/builds/364/steps/4/logs/stdio

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Inada Naoki]

Inada Naoki  added the comment:


New changeset f2a186712bfe726d54723eba37d80c7f0303a50b by Inada Naoki in branch 
'master':
bpo-30040: new empty dict uses key-sharing dict (GH-1080)
https://github.com/python/cpython/commit/f2a186712bfe726d54723eba37d80c7f0303a50b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Inada Naoki]

Change by Inada Naoki :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.8 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36229] Linear-time list, set, and bytearray ops.

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

This is an interesting idea. But I have two concerns.

1. It is hard to implement refcount-based optimization on Python 
implementations which do not use reference counting (i.e. PyPy). If the effect 
of this optimization will be significant, people will become writing a code 
that depends on it, and this will cause problems on other implementations.

2. Currently list1 + list2 + list3 returns a list which allocates the exact 
amount of memory needed to contain its content. But with the proposed changes 
the result list could preallocate more memory. If the result is a long living 
object, this can cause to wasting of memory.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36085] Enable better DLL resolution

[Paul Moore]

Paul Moore  added the comment:

> Also, if I'm understanding your intention, loading an extension may fail when 
> Python is embedded if the process is using the legacy DLL search path. So, 
> like with ctypes, we'll be forcing embedding applications to update how they 
> load DLLs in order to comply with us, else they'll have to accept that some 
> packages won't work without the SetDefaultDllDirectories call.

This bothers me - how will backward compatibility work in that case?
There are applications (for example, Vim) that can embed Python, and
it's possible to pick the Python version at compile time. Would Vim
need additional code (possibly guarded by some sort of "If this is
Python 3.8 or later" flag, which from my knowledge of the Vim code
would not be particularly easy to add in a backward compatible way) to
handle this change?

Actually, as a more general point, I have been following this
discussion, but I really have no feel as to what practical impact
there would be on an embedded application. I get that this is OS
functionality, and therefore it's not Python's place to explain the
details to users, but IMO it *is* Python's responsibility to explain
how embedding applications will need to change if we change how we
configure things. Even if users are currently using an approach that
is no longer encouraged (which is *I think* what you're saying about
putting DLLs on PATH) they are still using something that works right
now, and we're breaking it - so we need to describe what changed,
*why* we felt we should break their code, and what they need to do,
both to switch to the new model, and (if they have a requirement to do
so) to support the old and new models simultaneously in their code
(very few people, not even embedders, can suddenly say "we're dropping
support for Python 3.7 and older, we only allow 3.8+ from now on").

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35132] python-gdb error: Python Exception Type does not have a target

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 047f8f25b93e2649d234fa565a59383fceb40e16 by Victor Stinner (Miss 
Islington (bot)) in branch '3.7':
bpo-35132: Fixes missing target in gdb pep0393 check. (GH-11848) (GH-12284)
https://github.com/python/cpython/commit/047f8f25b93e2649d234fa565a59383fceb40e16


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35132] python-gdb error: Python Exception Type does not have a target

[STINNER Victor]

STINNER Victor  added the comment:

Thanks Dylan Cali for the bug report. Lisa Roach fixed the bug in 3.7 and 
master branches. Until a new release is published, you can copy manually the 
file from:
https://github.com/python/cpython/blob/3.7/Tools/gdb/libpython.py

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.7, Python 3.8 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31784] Implementation of the PEP 564: Add time.time_ns()

[Ned Deily]

Change by Ned Deily :


--
pull_requests: +12266

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36205] Python 3.7 and 3.8 process_time is not reported correctly when built on older macOS versions

[Ned Deily]

Change by Ned Deily :


--
keywords: +patch
pull_requests: +12267
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36206] re.match() not matching escaped hyphens

[Ned Deily]

Change by Ned Deily :


--
pull_requests: +12265

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36206] re.match() not matching escaped hyphens

[Ned Deily]

Change by Ned Deily :


--
pull_requests:  -12265

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36206] re.match() not matching escaped hyphens

[Ned Deily]

Change by Ned Deily :


--
pull_requests: +12268

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36205] Python 3.7 and 3.8 process_time is not reported correctly when built on older macOS versions

[Ned Deily]

Ned Deily  added the comment:

Further investigation shows that several time related functions were added to 
macOS at 10.12 including clock_gettime. For older systems, timemodule.c falls 
back to using getrusage. With Python 3.6.x, that fallbacks correctly but it 
appears that refactoring introduced with the implementation of PEP 564 
(bpo-31784, #3989) broke that for 3.7.x (and master/3.8).  I've attached a WIP 
PR that at the moment just turns Alexander's test into a potential test case.

Since this problem has been around since 3.7.0, I am lowering the priority to 
"deferred blocker" to not hold up 3.7.3rc1.

Victor, I'd appreciate it if you could take a look at this. Thanks!

--
nosy:  -lukasz.langa
priority: release blocker -> deferred blocker
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36206] re.match() not matching escaped hyphens

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
pull_requests:  -12268

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36266] Which module could not be found?

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
components: +Windows
nosy: +paul.moore, steve.dower, tim.golden, zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35746] [ssl][CVE-2019-5010] TALOS-2018-0758 Denial of Service

[STINNER Victor]

STINNER Victor  added the comment:

Yes, I close the issue.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[STINNER Victor]

STINNER Victor  added the comment:

Changing the default is trivial and safe. Let's do that. Additional doc 
wouldn't hurt.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36252] update to Unicode 12

[STINNER Victor]

STINNER Victor  added the comment:

Would you mind to explain how you update Unicode in Python?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22107] tempfile module misinterprets access denied error on Windows

[Jim Maloy]

Jim Maloy  added the comment:

This issue persists as of today (March 2019), in Python 3.7.2 (64 bit) running 
on Windows 10.  I gather from the comments that fixing it is no trivial matter, 
although I don't fully understand why.  The hang of "several seconds" that was 
originally described is at least 30 seconds on that platform -- I'm not sure 
when it would clear, as I didn't have the patience to wait it out.

https://stackoverflow.com/questions/55109076/python-tempfile-temporaryfile-hangs-on-windows-when-no-write-privilege

It seems to me that a genuine naming collision would be pretty rare -- at least 
for the (fairly common) use case I'm dealing with, trying to check where the 
directory is writeable or not.  Would it make sense to at least set a default 
number of collision-retries that is significantly lower?  Say, between 3 and 
10, instead of the system max?

--
nosy: +JDM

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36269] post install in setup.py does not work when executed through pip

[Saba Kauser]

New submission from Saba Kauser :

Hello,

I have a post install class that looks like this:
if('darwin' in sys.platform):
class PostInstall(install):
""" Post installation - run install_name_tool on Darwin """
def run(self):
clipath = os.getenv('IBM_DB_HOME', '@loader_path/clidriver')
print("in PostInstall with {}".format(clipath))
for so in glob.glob(r'build/lib*/ibm_db*.so'):
os.system("install_name_tool -change libdb2.dylib 
{}/lib/libdb2.dylib {}".format(clipath, so))
install.run(self)
cmd_class = dict(install = PostInstall)


And I pass cmd_class to setup(..) as:
setup(..
  include_package_data = True,
   cmdclass = cmd_class,
   **extra
 )

When I execute setup.py as "python setup.py install", then the PostInstall 
operation is executed after the ibm_db.so is built and installed and I see the 
intended result. 
However, when I run "pip install ibm_db" or "pip install .",
the execution order looks like this:
warnings.warn(notifyString)
running install
in PostInstall with /Applications/dsdriver/  ==> this is my post install 
script
running build
running build_py
creating build  
creating build/lib.macosx-10.9-x86_64-3.7==> I need to traverse to this 
folder to find my shared library

I would expect it to be run post the ibm_db is installed, not before it gets 
built. 

Can you please let me know how can this be fixed. Is this a bug with pip?

--
components: Build
messages: 337736
nosy: sabakauser
priority: normal
severity: normal
status: open
title: post install in setup.py does not work when executed through pip
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4356] Add "key" argument to "bisect" module functions

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

Hi, there has been renewed interest from @alexchamberlain and @f6v on GitHub 
for this feature. Would it be possible to get the patch reviewed so we can add 
it in 3.8?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35859] Capture behavior depends on the order of an alternation

[Ma Lin]

Change by Ma Lin :


--
pull_requests: +12269

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36159] Modify Formatter Class to handle arbitrary objects

[Ross Biro]

Ross Biro  added the comment:

I'm currently writing a language translator between two domain specific
computer languages.  Because some expressions occur repeatedly, but in
slightly different contexts, I make multiple passes.  The first pass
reduces everything it can and leaves place holder objects for things it
can't reduce.  Later passes replace the objects with their final expression
in the new language.  The final expression varies by context, so every time
it's reevaluated, it could change.  I would really like to handle things
like a + b as

"{a} + {b}".format(a=a, b=b)

This works great when a and b are strings.  But when they are place holder
objects, I wasn't able to find a good solution.  Although the Formatter
class came so close that I thought I would suggest the change.  What I
ended up doing was replacing objects with unique strings so that I could
use format and then using regular expressions on the output string to split
it into an array and replace the string identifiers with the original
objects.  The change I've suggested to the Formatter class would have
allowed me to skip the regular expressions.

Ross

On Mon, Mar 11, 2019 at 11:47 PM Lisa Roach  wrote:

>
> Lisa Roach  added the comment:
>
> Can you give an example use case for this? F-strings are the newer method
> of string interpolation, I'm not sure it's worth putting effort into adding
> features to string.Formatter.
>
> --
> nosy: +lisroach
>
> ___
> Python tracker 
> 
> ___
>

-- 
*Ross Biro* | CTO
___

O: 240-380-2231|  F: 240-556-0361 <(240)%20556-0361>
The Interface Financial Group 

CONFIDENTIALITY NOTICE: This email message, including any attachments, is
for the sole use of the intended recipient/s and may contain confidential &
privileged information. Any unauthorized review, use, disclosure, or
distribution is prohibited. If you are not the intended recipient, please
contact the sender by reply email and destroy all copies of the original
messages and any attachments.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35859] Capture behavior depends on the order of an alternation

[Ma Lin]

Change by Ma Lin :


--
pull_requests: +12270

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35859] Capture behavior depends on the order of an alternation

[Ma Lin]

Change by Ma Lin :


--
pull_requests: +12271

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35859] Capture behavior depends on the order of an alternation

[Ma Lin]

Change by Ma Lin :


Added file: https://bugs.python.org/file48204/t.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35859] Capture behavior depends on the order of an alternation

[Ma Lin]

Ma Lin  added the comment:

> Could you please create and run some microbenchmarks to measure
> possible performance penalty of additional MARH_PUSHes? I am 
> especially interesting in worst cases.

Besides the worst case, I prepared two solutions.

Solution_A (PR12288):
Fix the bugs, I can find test-case for every changes.
I feel JUMP_MIN_UNTIL_3 should MARK_PUSH() as well, but I really can't find a 
test-case to prove this feel.
Commit_12 is a safety-check, if JUMP_MIN_UNTIL_3 or other JUMPs should be 
protected, maybe there will be a bug report come from millions of users.

Solution_B (PR12289):
Based on Solution_A, protects JUMP_MIN_UNTIL_3.

Worst (PR12290):
Based on Solution_B, protects in everywhere, this should the worst performance.

Legend of this table:
* No: no protection
* L : LASTMARK_SAVE()
* Mu: MARK_PUSH() unconditionally
* Mr: MARK_PUSH() if in a repeat

  unpatched  Solution_A  Solution_B   Worst
JUMP_MAX_UNTIL_1 No No  No->  L/Mu
JUMP_MAX_UNTIL_2 L/Mu   L/MuL/Mu  L/Mu
JUMP_MAX_UNTIL_3 No No  No->  L/Mu

JUMP_MIN_UNTIL_1 No No  No->  L/Mu
JUMP_MIN_UNTIL_2 L->L/MrL/Mr  ->  L/Mu
JUMP_MIN_UNTIL_3 No No ->   L/Mu  ->  L/Mu

JUMP_REPEAT_ONE_1L->L/MrL/Mr  ->  L/Mu
JUMP_REPEAT_ONE_2L->L/MrL/Mr  ->  L/Mu

JUMP_MIN_REPEAT_ONE  L->L/MrL/Mr  ->  L/Mu

JUMP_BRANCH  L/Mr   L/MrL/Mr  ->  L/Mu

JUMP_ASSERT  No No  No->  L/Mu

JUMP_ASSERT_NOT  No   ->L/MrL/Mr  ->  L/Mu

🔶 I made a benchmark tool for Python RE engines.
https://github.com/animalize/re_benchmarks

re100mb.py was extracted from a real case, process 100MB real data in 16 rounds 
(with 16 patterns).
This table picks best of 4 results from each round:

 unpatched  ABWorstregex 2019.3.9
test 01 best:  0.6470.6290.6250.6350.102
test 02 best:  3.9804.0464.0524.0054.530
test 03 best:  0.7300.7080.7090.7060.433
test 04 best:  0.7630.7430.7400.7360.799
test 05 best:  2.5662.6932.6922.6540.981
test 06 best:  0.8830.8650.8590.8740.872
test 07 best:  2.8472.7732.7972.8901.202
test 08 best:  3.6443.6643.7403.6991.139
test 09 best:  0.3440.3480.3430.3450.378
test 10 best:  0.3410.3470.3430.3440.407
test 11 best:  4.4904.6554.5204.4401.340
test 12 best:  0.2640.2630.2620.2640.271
test 13 best:  0.2300.2300.2310.2330.281
test 14 best:  0.9320.9250.9190.9430.334
test 15 best:  1.8371.8151.8041.8660.683
test 16 best:  1.6911.7081.6762.0423.805

sum of above: 26.189   26.412   26.312   26.676   17.557

There seems no significant changes.

🔶 Below are some micro benchmarks, run t.py with the benchmark tool testit.py

SRE_OP_MAX_UNTIL a few repeats
unpatched: 744.85 nsec
SolutionA: 755.86 nsec
SolutionB: 745.14 nsec
Worst: 843.56 nsec
regex: 2.45 usec

SRE_OP_MAX_UNTIL many repeats
unpatched: 393.24 msec
SolutionA: 321.16 msec
SolutionB: 323.24 msec
Worst: 498.48 msec
regex: 240.73 msec

--
SRE_OP_MIN_UNTIL a few repeats
unpatched: 702.75 nsec
SolutionA: 701.90 nsec
SolutionB: 730.81 nsec
Worst: 873.67 nsec
regex: 1.84 usec

SRE_OP_MIN_UNTIL many repeats
unpatched: 210.60 msec
SolutionA: 174.20 msec
SolutionB: 323.93 msec
Worst: 491.73 msec
regex: 195.11 msec

--
SRE_OP_REPEAT_ONE short string
unpatched: 466.56 nsec
SolutionA: 468.85 nsec
SolutionB: 466.04 nsec
Worst: 463.86 nsec
regex: 1.13 usec

SRE_OP_REPEAT_ONE long string
unpatched: 2.19 msec
SolutionA: 2.19 msec
SolutionB: 2.19 msec
Worst: 2.19 msec
regex: 3.23 msec

--
SRE_OP_MIN_REPEAT_ONE short string
unpatched: 563.76 nsec
SolutionA: 566.68 nsec
SolutionB: 561.92 nsec
Worst: 601.69 nsec
regex: 1.12 usec

SRE_OP_MIN_REPEAT_ONE long string
unpatched: 11.16 msec
SolutionA: 11.27 msec
SolutionB: 10.55 msec
Worst: 15.97 msec
regex: 7.24 msec

--
SRE_OP_BRANCH
unpatched: 419.34 nsec
SolutionA: 422.07 nsec
SolutionB: 418.25 nsec
Worst: 423.56 nsec
regex: 1.34 usec

--
SRE_OP_ASSERT
unpatched: 320.58 nsec
SolutionA: 326.29 nsec
SolutionB: 320.81 nsec
Worst: 333.22 nsec
regex: 1.14 usec

SRE_OP_ASSERT_NOT
unpatched: 440.18 nsec
SolutionA: 440.93 nsec
SolutionB: 434.44 nsec
Worst: 446.33 nsec
regex: 1.00 usec

🔶 reduce the size of match_context struct

In stack-consuming scenes, Solution_A and Solution_B are better than unpatched.
This is because commit 10 and commit 11, they reduced the size of match_context 
struct, the stack uses this struct to 

[issue36270] DOC: Add link to sys.exc_info for "Reference Manual"

[Cheryl Sabella]

New submission from Cheryl Sabella :

In the docs for `sys.exc_info()`, the description of traceback says:

traceback gets a traceback object (see the ---> Reference Manual <---) which 
encapsulates the call stack at the point where the exception originally 
occurred.

It might be nice if Reference Manual had a link.

Assigning to @Mariatta for the PyCon sprint.

--
assignee: Mariatta
components: Documentation
messages: 337740
nosy: Mariatta, cheryl.sabella
priority: normal
severity: normal
stage: needs patch
status: open
title: DOC: Add link to sys.exc_info for "Reference Manual"
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4356] Add "key" argument to "bisect" module functions

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I'll look at it this weekend.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15749] cgitb prints html for text when display disabled.

[Cheryl Sabella]

Cheryl Sabella  added the comment:

I can still recreate this issue under 3.8.  Should this patch be converted to a 
GitHub pull request?

--
nosy: +cheryl.sabella
versions: +Python 3.7, Python 3.8 -Python 2.7, Python 3.2, Python 3.3, Python 
3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36225] Lingering subinterpreters should be implicitly cleared on shutdown

[Joannah Nanjekye]

Joannah Nanjekye  added the comment:

I have been wondering where the regression to test this can be put..in 
test__xxsubinterpreters.py may be?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36218] .sort() segfaults consistently on crafted input

[Ned Deily]

Ned Deily  added the comment:

Thanks for the analysis and the suggested PR which is now awaiting review.  
While segfaults are nasty, I don't see how this problem would be likely 
exploitable as a DoS without direct access to the interpreter.  So I'm 
downgrading it to "deferred blocker" for now to unblock 3.7.3rc1.

--
priority: release blocker -> deferred blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36271] '_io.TextIOWrapper' object has no attribute 'mode'

[Dávid Nemeskey]

New submission from Dávid Nemeskey :

TextIOWrapper objects returned by gzip.open() or bz2.open() do not have the 
'mode' data member. Those returned by io.open() do. It would be nice if it did.

--
components: Library (Lib)
messages: 337745
nosy: nemeskeyd
priority: normal
severity: normal
status: open
title: '_io.TextIOWrapper' object has no attribute 'mode'
versions: Python 3.5, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36271] '_io.TextIOWrapper' object has no attribute 'mode'

[Dávid Nemeskey]

Dávid Nemeskey  added the comment:

Note that this is not the same as GZipFile.mode, which is 1 or 2 (READ or 
WRITE), instead of the more informative "[raw][bt ]".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36272] Recursive logging crashes Interpreter in Python 3

[Saim Raza]

New submission from Saim Raza :

Following code logs an error and calls itself leading to stack overflow and 
eventually core dump in Python 3.6.

>>> import logging
>>> def rec():
... logging.error("foo")
... rec()
>>> rec()

[1]101641 abort (core dumped)  python3
FTR, this doesn't crash Python 2.7.

Attaching the error (condensed) in Python 3.6:

ERROR:root:foo
...
--- Logging error ---
Traceback (most recent call last):
...
RecursionError: maximum recursion depth exceeded in comparison
...
Fatal Python error: Cannot recover from stack overflow.
...
[1]101641 abort (core dumped)  python3
Python 2.7:

RuntimeError: maximum recursion depth exceeded
But no core dump in Python 2.7.

FTR, the error above with Python 3.6 will come into play if the log level is 
set to logging.ERROR. Similarly for other log levels.

--
components: Library (Lib)
messages: 337747
nosy: Saim Raza
priority: normal
severity: normal
status: open
title: Recursive logging crashes Interpreter in Python 3
type: crash
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36272] Recursive logging crashes Interpreter in Python 3

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I am not sure this is related to logging and looks similar to issue35542 except 
stack (depends on OS) is exhausted without setrecursionlimit(). What does below 
return?

def rec():
rec()

rec()

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36085] Enable better DLL resolution

[Steve Dower]

Steve Dower  added the comment:

> That will require rewriting many scripts and packages that use ctypes or cffi
> to load DLLs. It would also break DLLs that internally rely on modifying PATH
> for a delayed load, though I hope that's uncommon. I think it's easier for
> everyone else if we implement this just for extension-module loading with the
> LoadLibraryExW flags.

Only if they're loading them via PATH. If they're using full paths they'll be 
fine, and if they're using system DLLs they'll be fine. In both cases, the fix 
will work (better) with existing versions.

> Also, if I'm understanding your intention, loading an extension may fail when
> Python is embedded if the process is using the legacy DLL search path.

That's true. "import" will always use the secure flags, and so if you were 
relying on PATH to locate dependencies of the extension module (note that 
extension modules themselves are loaded by full path, so it doesn't apply to 
them), you need to stop doing that.

> Also, at runtime we can raise a SystemError if AddDllDirectory isn't found via
> GetProcAddress. This supports portable Python installations.

This adds a lot of complexity for very old Windows 7 installs. I'm not inclined 
to care that much for them - installing security updates isn't a big ask for a 
nearly EOL operating system.

> Correct me if I'm wrong, but once SetDefaultDllDirectories() is used, there is
> no going back: PATH no longer can change the search path no matter what flags
> are used with LoadLibrary* calls

Correct. But this is already the case if your sysadmin has enabled certain 
policies or if you're running via Store Python. So you can't rely on PATH 
anyway.

> Why is this CPython-specific and "private"? It seems like
> * it should be a public interface, used by all implementations consistently,
> as a policy decision for the win32 platform and documented as such

Not every implementation has to support Windows. We can certainly recommend 
that those that do copy it, but I'm not going to force MicroPython to declare 
an exception from a standard Python API.

> * located in os, much like os.environ (not critical)

Fair point. It can go into os. :)

> There should be some kind of debugging tool for when LoadLibraryEx fails, to
> indicate what might have gone wrong, perhaps os.getdlldirectory() would be
> helpful

I'd love to have this. Now someone just has to invent one that can be used from 
Python :) It's unrelated to this discussion - in fact, this change will make it 
so that you get the failure on _all_ machines, not just on some random user's 
machine.

We can't retrieve the true search path, only the ones that were added through 
an API that we control and making assumptions based on the documentation. I 
think this would be more of a distraction. The best way to diagnose actual 
LoadLibrary failures is to use a debugger, at which point simply getting the 
search paths doesn't add anything.

> This bothers me - how will backward compatibility work in that case?

The new search order is compatible with the old search order, so you can update 
all your layouts to have DLL dependencies in suitable locations and you'll be 
fine.

And if you're still writing code for Windows 7 with no security updates 
installed, Python 3.8 isn't going to save you anyway.

> I really have no feel as to what practical impact there would be on an
> embedded application.

Since we're not going to change the default search directories for the entire 
process when embedding, the only practical impact is that your extension 
modules need to have their dependent DLLs either:
* in the system directory
* adjacent to the .pyd file
* in a directory added using AddDllDirectory

And if the embedding application is already calling SetDefaultDllDirectories, 
as has been recommended for years, then they're already experiencing this 
change and won't have to update a thing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36264] os.path.expanduser should not use HOME on windows

[Steve Dower]

Steve Dower  added the comment:

I like the patch, but I'm not sure all the tests are properly preserving the 
real value of USERPROFILE.

Modifying this value could have a real impact on the rest of the process, so we 
should be very careful to undo it regardless of test result. (Modifying HOME is 
not a as big a deal since, as you point out, it's not "real" ;) )

Also, it's probably better to get the current value and check against that, 
rather than setting it to a known value. One day we might do the right thing 
and switch expanduser() from the environment variables to the correct API, 
which would break the test.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I don't think this should have been done.  Conceptually, there is no basis for 
presuming key-sharing for new empty dicts -- you can't know what they would 
share with.  This patch essentially undoes the entire reason for having a 
pre-allocated minsize dict.  If it were deemed to be the norm that applications 
typically had huge numbers of empty dicts that were never populated, then the 
correct solution would be a NULL pointer to the table field (as dicts do).

FWIW, the macro benchmarks aren't very informative here because they don't 
exercise much of this code.  I think there is an over-prioritization of small 
space savings at the expense of the intended use cases for dicts.  This patch 
just forces every dict that gets used to have to convert back from a 
key-sharing dict and do a memory allocation and memset(0) in the process.  The 
whole point of the minsize dict was to avoid that cost in the common case of 
small dicts (i.e. passing keyword arguments).

--
nosy: +rhettinger, tim.peters
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36216] urlsplit does not handle NFKC normalization

[Steve Dower]

Change by Steve Dower :


--
pull_requests: +12272

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36264] os.path.expanduser should not use HOME on windows

[Anthony Sottile]

Anthony Sottile  added the comment:

> Modifying this value could have a real impact on the rest of the process, so 
> we should be very careful to undo it regardless of test result. (Modifying 
> HOME is not a as big a deal since, as you point out, it's not "real" ;) )

Agreed, though I'd like to write it off as an existing problem (if it is a 
problem, it looks to me like the tests are doing proper environment cleanup).  
If they are broken, they'd already be leaking environment variables on posix 
since these tests run on both

Auditing the tests I modified:

- test_netrc: uses `with support.EnvironmentVarGuard() as environ:` (OK!)
- test_ntpath: uses `with support.EnvironmentVarGuard() as env:` (OK!)
- test_dist: `MetadataTestCase` extends `support.EnvironGuard` (OK!)
- test_config: `BasePyPIRCCommandTestCase` extends `support.EnvironGuard` (OK!)

so actually, looks like this is already good to go on that front!

Definitely agree on using the true value in the long term -- I'd like to defer 
that until that's actually happening though as it'll be a much more involved 
patch!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36216] urlsplit does not handle NFKC normalization

[Steve Dower]

Steve Dower  added the comment:

Thanks. Just posted a PR that puts the print behind a verbose flag (on Python 3 
it uses subtest so that we see which character caused the failure).

If that doesn't work for whatever reason, I'll just leave it out and hope that 
we never have to debug it on Python 2 :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36264] os.path.expanduser should not use HOME on windows

[Steve Dower]

Steve Dower  added the comment:

Great, thanks for checking! I'll merge.

--
assignee:  -> steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36264] os.path.expanduser should not use HOME on windows

[Steve Dower]

Steve Dower  added the comment:


New changeset 25ec4a45dcc36c8087f93bd1634b311613244fc6 by Steve Dower (Anthony 
Sottile) in branch 'master':
bpo-36264: Don't honor POSIX HOME in os.path.expanduser on Windows (GH-12282)
https://github.com/python/cpython/commit/25ec4a45dcc36c8087f93bd1634b311613244fc6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36266] Which module could not be found?

[Steve Dower]

Steve Dower  added the comment:

I agree. Unfortunately, the operating system does not provide this information.

The best I can offer is to run Process Monitor [1] and watch its logs. It 
should show the paths it attempts to access.

[1]: http://technet.microsoft.com/en-us/sysinternals/bb896645

--
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36264] os.path.expanduser should not use HOME on windows

[Zachary Ware]

Zachary Ware  added the comment:

If we're going ahead with this, it's worth a mention in whatsnew as this is 
going to break things for some users.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36205] Python 3.7 and 3.8 process_time is not reported correctly when built on older macOS versions

[STINNER Victor]

STINNER Victor  added the comment:

Sorry, I don't understand the problem.

Can someone please give the result of these commands on Python 3.6 and 3.7 on 
macOS?

Example on Linux (Fedora 29):

$ python3
>>> import platform, time
>>> platform.platform()
'Linux-4.20.13-200.fc29.x86_64-x86_64-with-fedora-29-Twenty_Nine'
>>> for clock in ('monotonic', 'perf_counter', 'process_time', 'thread_time', 
>>> 'time'): print(f'clock: {time.get_clock_info(clock)}')
... 
clock: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, 
resolution=1e-09)
clock: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, 
resolution=1e-09)
clock: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_PROCESS_CPUTIME_ID)', monotonic=True, 
resolution=1e-09)
clock: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_THREAD_CPUTIME_ID)', monotonic=True, 
resolution=1e-09)
clock: namespace(adjustable=True, 
implementation='clock_gettime(CLOCK_REALTIME)', monotonic=False, 
resolution=1e-09)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36205] Python 3.7 and 3.8 process_time is not reported correctly when built on older macOS versions

[STINNER Victor]

STINNER Victor  added the comment:

Ah, or maybe use test.pythoninfo:

$ ./python -m test.pythoninfo|grep -E '^(platform|time)'
platform.architecture: 64bit ELF
platform.libc_ver: glibc 2.28
platform.platform: Linux-4.20.13-200.fc29.x86_64-x86_64-with-glibc2.28
platform.python_implementation: CPython
time.altzone: -7200
time.daylight: 1
time.get_clock_info(clock): namespace(adjustable=False, 
implementation='clock()', monotonic=True, resolution=1e-06)
time.get_clock_info(monotonic): namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, 
resolution=1e-09)
time.get_clock_info(perf_counter): namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, 
resolution=1e-09)
time.get_clock_info(process_time): namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_PROCESS_CPUTIME_ID)', monotonic=True, 
resolution=1e-09)
time.get_clock_info(thread_time): namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_THREAD_CPUTIME_ID)', monotonic=True, 
resolution=1e-09)
time.get_clock_info(time): namespace(adjustable=True, 
implementation='clock_gettime(CLOCK_REALTIME)', monotonic=False, 
resolution=1e-09)
time.time: 1552405487.361025
time.timezone: -3600
time.tzname: ('CET', 'CEST')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36085] Enable better DLL resolution

[Steve Dower]

Steve Dower  added the comment:

Since I just dug enough to find it, the best way to diagnose problems with 
dependent DLLs not being found is probably to run Process Monitor [1] while 
doing the import and checking its logs. It should show the paths that were 
attempted to be accessed.

[1]: http://technet.microsoft.com/en-us/sysinternals/bb896645

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35218] decompressing and then re-compressing zipfiles with Python 3 zipfile loses flag_bits

[keeely]

keeely  added the comment:

Can you please get on and fix this if you're going to deprecate Python 2.0.  
It's unkind on your users to deprecate the old version before you've fixed 
basic regressions like this in the new one.  Just to be clear, I am attaching 
the code that reproduces this (tmp.py).   Run it with Python 2 and 3 to see the 
difference.  I will not fill in your agreement form, and no I will not provide 
my reasons, but you don't need it, it's a one-line fix.

--
Added file: https://bugs.python.org/file48205/tmp.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36272] Recursive logging crashes Interpreter in Python 3

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +vinay.sajip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35854] EnvBuilder and venv symlinks do not work on Windows on 3.7.2

[Steve Dower]

Steve Dower  added the comment:

Yep. Done

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36264] os.path.expanduser should not use HOME on windows

[Steve Dower]

Steve Dower  added the comment:

> If we're going ahead with this, it's worth a mention in whatsnew

Good call. I'll do it

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35218] decompressing and then re-compressing zipfiles with Python 3 zipfile loses flag_bits

[STINNER Victor]

STINNER Victor  added the comment:

> I will not fill in your agreement form, and no I will not provide my reasons, 
> but you don't need it, it's a one-line fix.

Ok. It's up to you, but in that case, we cannot merge your change. Contributors 
have to sign it. Your test is larger than a single line.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Raymond Hettinger]

Change by Raymond Hettinger :


--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36085] Enable better DLL resolution

[Paul Moore]

Paul Moore  added the comment:

> > This bothers me - how will backward compatibility work in that case?
>
> The new search order is compatible with the old search order, so you can 
> update all your layouts to have DLL dependencies in suitable locations and 
> you'll be fine.

OK, cool. But one thing I'm not clear on, will this change just affect
the embedded Python, or will it affect the whole process - which would
mean that supporting an embedded Python 3.8 interpreter would mean
potentially reorganising the application layout. That may be quite a
cost, in some applications.

Note that this is all on the basis of "I don't understand the
implications, they should be documented" rather than being a specific
problem that I know will happen. My particular scenario, though, is an
application like Vim, that provides optional support for an "embedded
scripting" which may be any one of a number of Python versions, or
even other languages. In an application like that, costs for
supporting Python 3.8 may simply result in no (or delayed) support for
Python 3.8, rather than the application getting fixed.

> And if you're still writing code for Windows 7 with no security updates 
> installed, Python 3.8 isn't going to save you anyway.

Nobody's suggesting that it will. But maintaining *existing* code that
supports older Windows versions, while still allowing Python 3.8 to be
used as an embedded scripting language on systems that support it, is
an entirely reasonable proposal.

> > I really have no feel as to what practical impact there would be on an
> > embedded application.
>
> Since we're not going to change the default search directories for the entire 
> process when embedding

OK, if that's the case, then that alleviates most of my concerns. But
it really wasn't obvious to me, and it's something that I think should
be made clear in the docs, if only to reassure embedding applications
that Python isn't making global changes. The docs for SetDllDirectory
seem to imply that there *is* a global impact - "The SetDllDirectory
function affects all subsequent calls to the LoadLibrary and
LoadLibraryEx functions" (note - *all* subsequent calls, which implies
that behaviour will change for the embedding application once Python
has been loaded).

> the only practical impact is that your extension modules need to have their 
> dependent DLLs either:
> * in the system directory
> * adjacent to the .pyd file
> * in a directory added using AddDllDirectory

That seems fine, so let's just state that and keep things simple for
embedders to understand.

> And if the embedding application is already calling SetDefaultDllDirectories, 
> as has been recommended for years, then they're already experiencing this 
> change and won't have to update a thing.

Sadly, in my experience, an awful lot of projects (specifically, open
source projects that write mostly cross-platform code, with the
minimum of OS-specific differences) don't follow recommendations like
this. They use LoadLibrary without digging too deeply into the
implications or complexities, as long as it does what they want. And I
don't think MS helped themselves much here, either - the whole
business with SxS installs and assemblies was (IMO) *way* too much
complexity for most cross platform projects to bother with, and went
ignored. Even once things got simpler again, there remained a sense of
"don't go there, just get something that works". (And to be clear, I'm
not bashing on MS here - I find the Linux machinery around all of this
to be just as complex and confusing).

Anyhow, if as you say the only impact is that when a pyd file depends
on a DLL, that DLL needs to be located in one of three places, all of
which are equally valid on Python <=3.7, and there's no impact on the
non-Python part of the embedded application, then it's not a big deal.
Let's make the change, write up those points in What's New (at least),
and leave it at that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Inada Naoki]

Inada Naoki  added the comment:

> I don't think this should have been done.  Conceptually, there is no basis 
> for presuming key-sharing for new empty dicts -- you can't know what they 
> would share with.  
This patch essentially undoes the entire reason for having a pre-allocated 
minsize dict.  If it were deemed to be the norm that applications typically had 
huge numbers of empty dicts that were never populated, then the correct 
solution would be a NULL pointer to the table field (as dicts do).

It increases massive NULL checks, and possibly cause bugs.

> FWIW, the macro benchmarks aren't very informative here because they don't 
> exercise much of this code.  I think there is an over-prioritization of small 
> space savings at the expense of the intended use cases for dicts.  This patch 
> just forces every dict that gets used to have to convert back from a 
> key-sharing dict and do a memory allocation and memset(0) in the process.  
> The whole point of the minsize dict was to avoid that cost in the common case 
> of small dicts (i.e. passing keyword arguments).

Note that there is _PyDict_NewPresized().

When kwargs is empty, this patch increase dict creation speed.
When kwargs is not empty, temporary key-sharing table is not used because we 
use _PyDict_NewPresized() is used.

$ ./py.edict bm_edict.py --compare-to ./py.master
py.master: . 192 ns +- 7 ns
py.edict: . 175 ns +- 4 ns
Mean +- std dev: [py.master] 192 ns +- 7 ns -> [py.edict] 175 ns +- 4 ns: 1.10x 
faster (-9%)

py.master: . 269 ns +- 4 ns
py.edict: . 273 ns +- 2 ns
Mean +- std dev: [py.master] 269 ns +- 4 ns -> [py.edict] 273 ns +- 2 ns: 1.02x 
slower (+2%)

So I don't think net performance of kwargs doesn't get worse.

--
nosy:  -Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36236] Python crash on macOS when CWD is invalid

[STINNER Victor]

STINNER Victor  added the comment:

_Py_wgetcwd() call has been introduced by the following commit:

commit ee3784594b33c72c3fdca6a71892d22f14045ab6
Author: Nick Coghlan 
Date:   Sun Mar 25 23:43:50 2018 +1000

bpo-33053: -m now adds *starting* directory to sys.path (GH-6231) (#6236)

Historically, -m added the empty string as sys.path
zero, meaning it resolved imports against the current
working directory, the same way -c and the interactive
prompt do.

This changes the sys.path initialisation to add the
*starting* working directory as sys.path[0] instead,
such that changes to the working directory while the
program is running will have no effect on imports
when using the -m switch.

(cherry picked from commit d5d9e02dd3c6df06a8dd9ce75ee9b52976420a8b)

I don't think that it's correct to fail with a fatal error if the current 
directory no longer exist. Would it be possible to not add it to sys.path if it 
doesn't exist? Silently ignore the error.

@Nick: What do you think?

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36236] Python crash on macOS when CWD is invalid

[STINNER Victor]

Change by STINNER Victor :


--
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Inada Naoki]

Change by Inada Naoki :


--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35218] decompressing and then re-compressing zipfiles with Python 3 zipfile loses flag_bits

[keeely]

keeely  added the comment:

You can take out the test.  It wasn't there before so who's going to care?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35661] Store the venv prompt in pyvenv.cfg

[STINNER Victor]

STINNER Victor  added the comment:

This issue broke x86 Gentoo Installed with X 3.x buildbot:

https://buildbot.python.org/all/#/builders/103/builds/2208

ERROR: test_prompt (test.test_venv.BasicTest)
--
Traceback (most recent call last):
  File 
"/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.8/test/test_venv.py",
 line 123, in test_prompt
builder.create(self.env_dir)
  File 
"/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.8/venv/__init__.py",
 line 70, in create
self.setup_scripts(context)
  File 
"/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.8/venv/__init__.py",
 line 278, in setup_scripts
self.install_scripts(context, path)
  File 
"/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.8/venv/__init__.py",
 line 354, in install_scripts
with open(dstfile, 'wb') as f:
PermissionError: [Errno 13] Permission denied: 
'/buildbot/tmp/tmpdir/tmph7_oob5e/bin/Activate.ps1'

Please either fix or revert the change.

--
nosy: +vstinner
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36216] urlsplit does not handle NFKC normalization

[STINNER Victor]

STINNER Victor  added the comment:

Commit e37ef41289b77e0f0bb9a6aedb0360664c55bdd5 introduced a regression on 
AMD64 Windows8.1 Refleaks 2.7:

test_urlparse leaked [114, 114, 114] references, sum=342

https://buildbot.python.org/all/#/builders/33/builds/532

It is fixed by PR 12291.

You can test:

python -m test -R 3:3 test_urlparse

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36264] os.path.expanduser should not use HOME on windows

[Steve Dower]

Change by Steve Dower :


--
pull_requests: +12273

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35661] Store the venv prompt in pyvenv.cfg

[Steve Dower]

Steve Dower  added the comment:

Presumably the issue is in the test and not the main change itself.

Is this another case where Linux can't run venv tests inside a venv?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36273] test_thread leaks a core dump on PPC64 AIX 3.x

[STINNER Victor]

New submission from STINNER Victor :

PPC64 AIX 3.x:
https://buildbot.python.org/all/#/builders/10/builds/2224

0:12:47 [160/420/1] test_threading failed (env changed)
...
Ran 158 tests in 12.023s

OK (skipped=1)
Warning -- files was modified by test_threading
  Before: []
  After:  ['core']


I don't know if it's a regression or if it occurred in the past.

Michael: Can you please try to reproduce the issue? Any idea if it's new or not?

You can try to identify which test creates a core dump using:

$ ./python -m test.bisect_cmd --fail-env-changed test_thread -v

The command is supposed to find which test creates the "core" file.

--
components: Tests
messages: 337772
nosy: Michael.Felt, vstinner
priority: normal
severity: normal
status: open
title: test_thread leaks a core dump on PPC64 AIX 3.x
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36216] urlsplit does not handle NFKC normalization

[Steve Dower]

Steve Dower  added the comment:

If it's fixed by not printing to the console, then it must be a refleak in 
print. Or perhaps in the test failure/repeat.

That PR is going to be merged as soon as AppVeyor finishes, so nothing to worry 
about here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36205] Python 3.7 and 3.8 process_time is not reported correctly when built on older macOS versions

[Ned Deily]

Ned Deily  added the comment:

***
tip of master built on current macOS 10.14 - correct results:

3.8.0a2+ (heads/master:f45813df52, Mar 12 2019, 12:25:58)
[Clang 10.0.0 (clang-1000.11.45.5)]
macOS-10.14.3-x86_64-i386-64bit
monotonic: namespace(adjustable=False, implementation='mach_absolute_time()', 
monotonic=True, resolution=1e-09)
perf_counter: namespace(adjustable=False, 
implementation='mach_absolute_time()', monotonic=True, resolution=1e-09)
process_time: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_PROCESS_CPUTIME_ID)', monotonic=True, 
resolution=1.0002e-06)
thread_time: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_THREAD_CPUTIME_ID)', monotonic=True, 
resolution=1e-09)
time: namespace(adjustable=True, 
implementation='clock_gettime(CLOCK_REALTIME)', monotonic=False, 
resolution=1.0002e-06)
***

***
v3.8.0a2 python.org installer built on macOS 10.9 - INCORRECT results:

3.8.0a2 (v3.8.0a2:23f4589b4b, Feb 25 2019, 10:59:08)
[Clang 6.0 (clang-600.0.57)]
macOS-10.14.3-x86_64-i386-64bit
monotonic: namespace(adjustable=False, implementation='mach_absolute_time()', 
monotonic=True, resolution=1e-09)
perf_counter: namespace(adjustable=False, 
implementation='mach_absolute_time()', monotonic=True, resolution=1e-09)
process_time: namespace(adjustable=False, 
implementation='getrusage(RUSAGE_SELF)', monotonic=True, resolution=1e-06)
thread_time: not available
time: namespace(adjustable=True, implementation='gettimeofday()', 
monotonic=False, resolution=1e-06)
***

***
v3.6.8 python.org installer built on macOS 10.9 - correct results:

3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
Darwin-18.2.0-x86_64-i386-64bit
monotonic: namespace(adjustable=False, implementation='mach_absolute_time()', 
monotonic=True, resolution=1e-09)
perf_counter: namespace(adjustable=False, 
implementation='mach_absolute_time()', monotonic=True, resolution=1e-09)
process_time: namespace(adjustable=False, 
implementation='getrusage(RUSAGE_SELF)', monotonic=True, resolution=1e-06)
thread_time: not available
time: namespace(adjustable=True, implementation='gettimeofday()', 
monotonic=False, resolution=1e-06)
***

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36273] test_thread leaks a core dump on PPC64 AIX 3.x

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

See also https://bugs.python.org/issue35828#msg337076 where 
test_multiprocessing_fork seemed to leave a core dump. Maybe one test core 
dumps and leaves the report to the other causing env changed?

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34713] csvwriter.writerow()'s return type is undocumented

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

csvwriter.writerows() does not return anything.

The return value of csv.writecsv() is the return value of the write() method of 
the file object given as parameter in csv.writer(). I'm not sure it's safe to 
document it, should we return None instead?

--
nosy: +remi.lapeyre

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35661] Store the venv prompt in pyvenv.cfg

[Cheryl Sabella]

Cheryl Sabella  added the comment:

Sorry about that.  I'll work on a fix and test it on the buildbots first.  

There are some tests with pyvenv.cfg that are skipped if it happens in a venv 
(@skipInVenv decorator), but the test right above this one (test_defaults) does 
not use the decorator and it checks pyvenv.cfg, so I think test_prompt might 
just need to do some cleanup first.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Inada Naoki]

Inada Naoki  added the comment:

Another micro benchmark:

$ ./py.edict.opt -m perf timeit --compare-to ./py.master.opt '{}' --duplicate=10
py.master.opt: . 26.3 ns +- 0.5 ns
py.edict.opt: . 13.0 ns +- 0.1 ns

Mean +- std dev: [py.master.opt] 26.3 ns +- 0.5 ns -> [py.edict.opt] 13.0 ns +- 
0.1 ns: 2.02x faster (-51%)

$ ./py.edict.opt -m perf timeit --compare-to ./py.master.opt 'd={}; 
d["a"]=None' --duplicate=10
py.master.opt: . 58.1 ns +- 0.9 ns
py.edict.opt: . 64.1 ns +- 0.9 ns

Mean +- std dev: [py.master.opt] 58.1 ns +- 0.9 ns -> [py.edict.opt] 64.1 ns +- 
0.9 ns: 1.10x slower (+10%)


Hmm, while 2x faster temporal empty dict is nice, 10% slower case can be 
mitigated.
I will try more micro benchmarks and optimizations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10948] Trouble with dir_util created dir cache

[Ivan Tashev]

Ivan Tashev  added the comment:

distutils.dir_util is easily found in the documentation. If this behaviour is 
not fixed, at least the docs should state dir_util is not recommended for 
public use.

--
nosy: +ivtashev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36229] Linear-time list, set, and bytearray ops.

[Brandt Bucher]

Brandt Bucher  added the comment:

Thank for the input, Serhiy. On point (1):

It's a valid concern... but I don't think it's necessarily a good enough reason 
to withhold a simple, yet *significant* performance increase for a common use 
case.

This reminds me of past CPython implementation details, such as ordered 
dictionaries in 3.6. Some users may errantly adopt problematic idioms that 
aren't strictly portable between implementations, but if you ask me the 
performance improvement (again, from quadratic to linear) for three common 
built-in types (I admit I'm being a bit generous to bytearray here ;) is well 
worth it.

Regarding (2): 

I believe that CPython lists only overallocate by something like 1/8, which in 
my opinion is a fairly conservative value. If the lists are large enough for 
this small preallocation to be a real memory burden, I would imagine that 
making the extra copies would be even worse.

And, if you ask me, list overallocation is a feature, not a bug!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36264] os.path.expanduser should not use HOME on windows

[Eryk Sun]

Eryk Sun  added the comment:

> `os.path.expanduser` in `ntpath` uses `HOME` in preference to 
> `USERPROFILE` / `HOMEDRIVE\\HOMEPATH`

Guido intentionally added support for HOME in ntpath.expanduser way back in 
Python 1.5 (circa 1997), and now we're removing it over 20 years later. I 
expect this to break some deployments, but it's not a serious problem. 

My feeling here is "meh". Practically all use of expanduser() is dubious in 
Windows, varying from going against convention to being completely wrong. We're 
long due for a module in the standard library that abstracts access to 
platform-specific configuration and special directories, but attempts to adopt 
one keep losing momentum. I guess it's too prone to bike shedding and arguments.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Brett Cannon]

Brett Cannon  added the comment:

It's sounding like it might be worth the effort then to make lib2to3's parser 
not be a "hidden" thing in lib2to3, break it out as a new parser module (I have 
no stance on name), and then deprecate the old parser module. I think this was 
discussed at the last language summit when Christian proposed deprecating 
lib2to3 and everyone said the parser is too useful to lose.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36264] os.path.expanduser should not use HOME on windows

[Steve Dower]

Steve Dower  added the comment:

> Guido intentionally added support for HOME in ntpath.expanduser way back in 
> Python 1.5 (circa 1997), and now we're removing it over 20 years later.

Wouldn't be the first thing to be removed :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35760] test_asyncio: test_async_gen_asyncio_gc_aclose_09() race condition

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I have modified the test with help of Victor to have lower sleep delay to 
reproduce the bug. Since sleep calls call_later with the delay where the Timer 
object is created time.monotonic() + delay. I have added a print statement to 
see when the sleep is scheduled.

Modified test with 0.01 for run and 0.001 for finally : 

def test_async_gen_asyncio_gc_aclose_09(self):
DONE = 0
import time

async def gen():
nonlocal DONE
try:
while True:
yield 1
finally:
print(time.monotonic(), "finally sleeps 1 ms")
await asyncio.sleep(0.001)
print(time.monotonic(), "finally sleeps 1 more ms")
await asyncio.sleep(0.001)
DONE = 1
print(time.monotonic(), "finally: DONE = 1--")

async def run():
g = gen()
await g.__anext__()
await g.__anext__()
print(time.monotonic(), "del g")
del g

print(time.monotonic(), "run sleeps 10 ms...")
await asyncio.sleep(0.01)
print(time.monotonic(), "run exit")

self.loop.run_until_complete(run())
self.assertEqual(DONE, 1)


./python.exe -X dev -m test -j4 -F test_asyncgen -m 
test_async_gen_asyncio_gc_aclose_09

On running the test I can see that sometimes sometimes DONE is set before run 
exit.

0:00:00 load avg: 1.87 [  1] test_asyncgen passed
0.738210762 del g
0.739222722 run sleeps 10 ms...
0.762559912001 0.01
0.757210842 finally sleeps 1 ms
0.76050695 0.001
0.762445572 finally sleeps 1 more ms
0.765119049 0.001
0.765694536 run exit
0.76704106 finally: DONE = 1--

sometimes DONE is set after run exit.

0:00:00 load avg: 1.87 [  2] test_asyncgen passed
0.739718657 del g
0.740647619 run sleeps 10 ms...
0.764014802 0.01
0.759427252 finally sleeps 1 ms
0.762613547 0.001
0.764573796 finally sleeps 1 more ms
0.767242878 0.001
0.767722044 run exit
0.769133638 finally: DONE = 1--


There are cases where run exit is called before second sleep is even called 
causing failure.

0:00:03 load avg: 1.96 [ 16/1] test_asyncgen failed
0.685123806 del g
0.685635626 run sleeps 10 ms...
0.698605025 0.01
0.69112447 finally sleeps 1 ms
0.700476131 0.001
0.701259052 run exit
0.702733814 finally sleeps 1 more ms
0.70500493 0.001
Task was destroyed but it is pending!
source_traceback: Object created at (most recent call last):

Sometimes the second sleep scheduled time and run exit time are almost the same 
by 1 ms.

0:00:04 load avg: 1.96 [ 18/2] test_asyncgen failed
0.701280462 del g
0.701810041 run sleeps 10 ms...
0.714906478 0.01
0.707617751 finally sleeps 1 ms
0.713339727 0.001
0.714532967 finally sleeps 1 more ms
0.716712187001 0.001
0.716623958 run exit
Task was destroyed but it is pending!


I am not sure at this point if I should further debug this at Lib/asyncio/ or 
at _asynciomodule.c regarding the control flow change related code when an 
object is awaited. I suspect there is a race condition here since the test 
seems to set DONE=1 with different control flows and also where DONE=1 is not 
even called. This is more easily reproducible at lower resolution of time like 
using 0.01 and 0.001 for this test so this feels more like the race condition 
is exposed at lower time levels and thus increasing the time difference between 
the test doesn't seem to be a good solution. 

The above test also fails in 3.6 but takes a long time before below trace 
occurs : 

0:00:42 load avg: 4.26 [159] test_asyncgen passed
16775.541152592 del g
16775.541369282 run sleeps 10 ms...
16775.542167311 finally sleeps 1 ms
16775.544090629 finally sleeps 1 more ms
16775.546002851 finally: DONE = 1--
16775.552762612 run exit
0:00:43 load avg: 4.26 [160/1] test_asyncgen failed
16775.529271686 del g
16775.529405 run sleeps 10 ms...
16775.529833951 finally sleeps 1 ms
16775.539453811 finally sleeps 1 more ms
16775.539972874 run exit
Task was destroyed but it is pending!
task:  wait_for=>
test test_asyncgen failed -- Traceback (most recent call last):
  File 
"/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_asyncgen.py",
 line 693, in test_async_gen_asyncio_gc_aclose_09
self.assertEqual(DONE, 1)
AssertionError: 0 != 1


Yury and Andrew, your thoughts would be helpful in debugging this issue and 
also to check the actual behavior to be tested in this case.

--
nosy: +asvetlov, yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35760] test_asyncio: test_async_gen_asyncio_gc_aclose_09() race condition

[Yury Selivanov]

Yury Selivanov  added the comment:

Can you change "await asyncio.sleep(0.01)" to "await asyncio.sleep(1)" and 
check if the test still fails?  (IOW see if there's a bug in Python's async gen 
implementation or this is simply caused by slow CI bots).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Xavier Combelle]

Xavier Combelle  added the comment:

never used the parser module nor lib2to3. Does they have any advantage over 
ast.parse and ast module ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35661] Store the venv prompt in pyvenv.cfg

[Steve Dower]

Steve Dower  added the comment:

It may also be that the directory exists already - the previous test explicitly 
removes it before calling create().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35661] Store the venv prompt in pyvenv.cfg

[Steve Dower]

Steve Dower  added the comment:

Maybe you can just call builder.create_configuration() from the test and pass 
in enough context that it doesn't have to create a real venv? That would also 
make the test faster.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35760] test_asyncio: test_async_gen_asyncio_gc_aclose_09() race condition

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I changed sleep(0.01) to sleep(1) and even after 350 runs I couldn't see any 
failure in test. I guess it fails only for less difference in time like 0.01 
and 0.001.

$ ./python.exe -X dev -m test -j4 -F test_asyncgen -m 
test_async_gen_asyncio_gc_aclose_09
[snip]
0:02:46 load avg: 3.71 [356] test_asyncgen passed
0.660945396 del g
0.661572575 run sleeps 1000 ms...
1.664627785 1
0.666234736 finally sleeps 1 ms
0.667970371 0.001
0.669262596 finally sleeps 1 more ms
0.671174578 0.001
0.671972246 finally: DONE = 1--
1.668627272 run exit

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36085] Enable better DLL resolution

[Eryk Sun]

Eryk Sun  added the comment:

> will this change just affect the embedded Python, or will it affect 
> the whole process

SetDefaultDllDirectories affects the whole process and cannot be reverted back 
to the legacy search path that includes "%SystemRoot%",  "%SystemRoot%\System" 
(the old 16-bit directory), the current working directory, and the PATH 
directories. Also, there's no LoadLibraryExW flag to use the legacy search 
path, either, so scripts and packages that use ctypes or cffi will have to be 
updated if they depend on PATH or changing the working directory. 

The alternative is to modify Python's importer to use the secure LoadLibraryExW 
flags (i.e. LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | 
LOAD_LIBRARY_SEARCH_DEFAULT_DIRS), without affecting the rest of the process. 

LOAD_LIBRARY_SEARCH_DEFAULT_DIRS includes the application directory, the user 
directories added with AddDlldirectory or SetDllDirectoryW, and the System32 
directory. LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR prepends the directory of the 
target DLL, which must be passed as a fully-qualified path.

> The docs for SetDllDirectory seem to imply that there *is* a global
> impact

SetDllDirectoryW still works after calling SetDefaultDllDirectories, as long as 
we include either LOAD_LIBRARY_SEARCH_USER_DIRS or 
LOAD_LIBRARY_SEARCH_DEFAULT_DIRS. It only allows adding a single directory, so 
it's of limited use anyway.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

What about creating small dict using a dict display? E.g. {'a': None}.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35760] test_asyncio: test_async_gen_asyncio_gc_aclose_09() race condition

[Yury Selivanov]

Yury Selivanov  added the comment:

> I guess it fails only for less difference in time like 0.01 and 0.001.

Yeah, that's what I was thinking.  0.1 seconds is still too tight, a simple GC 
pause can skew the timers and break the test logic.

Maybe we should just increase the sleep to 0.5 or 1 second and hope that it 
will eliminate the race.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36085] Enable better DLL resolution

[Steve Dower]

Steve Dower  added the comment:

> The alternative ...

Is what I proposed in the first place. Adding the SetDefaultDllDirectories call 
to Py_Main (that is, NOT for embedders) is to ensure consistency throughout the 
process. It'll only affect extension module dependencies that do their own 
delay loading and currently rely on unsupported resolve paths.

Since everyone seems to have misunderstood at least part of the proposal, I'm 
not going to explain any more until I have a patch. Excluding boilerplate and 
docs, it'll be about ten lines of code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> Hmm, while 2x faster temporal empty dict is nice, 
> 10% slower case can be mitigated.
> I will try more micro benchmarks and optimizations.

I hate to see you working so hard to make this work.  IMO, the effort is 
futile.  Dicts were intentionally designed to do a single memory allocation and 
memset(0) to be fully ready to store data.  This PR is undoing that decision 
and will make Python worse rather than better.  The PR is optimizing for the 
wrong thing.  The space used by empty dicts is something we care much less 
about than the cost of the first assignment.

No "mitigations" are going to help.  If there is a second malloc and we incur 
the cost of switching from sharing-to-non-sharing, that is additional work that 
will have be done by every single dictionary that actually gets used.  Nothing 
will get that lost work back.

FWIW, if we were going to optimize for space used by empty dicts, the table 
pointer could just be set to NULL.  That would be better than key sharing which 
is not only slower but also conceptually wrong (outside the context of instance 
creation, dicts will never be shared).

> For the record, legitimate case when many empty dicts are 
> created, and few are populated, is the collections-free 
> approach to defaultdict(dict):

While it is helpful to know that there is some possible application that would 
benefit, we don't optimize for the rare case, we optimize for the common case 
where dicts get used.  A substantial fraction of the language is implemented 
using dicts. For the most part, we use NULL values when the dict isn't actually 
needed; so, the normal case is that dicts do get used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35661] Store the venv prompt in pyvenv.cfg

[Brett Cannon]

Change by Brett Cannon :


--
assignee: brett.cannon -> cheryl.sabella

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36085] Enable better DLL resolution

[Paul Moore]

Paul Moore  added the comment:

OK, I don't really follow enough of the details here to comment properly. But 
clearly Steve and Eryk are not yet in agreement.

My personal view is that this is something where we should be trying *very* 
hard to preserve backward compatibility. The proposal here is intended to solve 
the problem of making it easier for .pyd files to reliably load helper DLLs 
from shared locations. That's fine, and while it's an important use case (AIUI, 
it matters for a lot of the scientific stack) IMO it's *not* important enough 
to warrant breaking working scripts or embedding applications (particularly as 
this is a fairly obscure detail of how Windows works, so it's unlikely that 
people carefully follow "best practices" here).

I'm very concerned that comments I've seen here, specifically

>> That will require rewriting many scripts and packages that use ctypes or cffi
>> to load DLLs. It would also break DLLs that internally rely on modifying PATH
>> for a delayed load, though I hope that's uncommon. I think it's easier for
>> everyone else if we implement this just for extension-module loading with the
>> LoadLibraryExW flags.
>
> Only if they're loading them via PATH. If they're using full paths they'll be 
> fine, and if they're using system DLLs they'll be fine. In both cases, the 
> fix will work (better) with existing versions.
>
>> Also, if I'm understanding your intention, loading an extension may fail when
>> Python is embedded if the process is using the legacy DLL search path.
>
> That's true. "import" will always use the secure flags, and so if you were 
> relying on PATH to locate dependencies of the extension module (note that 
> extension modules themselves are loaded by full path, so it doesn't apply to 
> them), you need to stop doing that.

imply that it's OK to break working code "because they are doing things 
wrongly". That's not how backward compatibility works - we should avoid 
breaking *any* working code, no matter how ill-advised it seems to be.

If it's necessary to break code that (say) uses ctypes to load a DLL via PATH, 
or an embedding application that relies on getting DLLs using PATH, then we 
need to follow PEP 387 and go through a deprecation cycle for the existing 
behaviour.

For the ctypes case I assume we can detect where we found the DLL being loaded, 
so warning that behaviour will change is certainly possible.

For the embedding case, we could (for example) add an API 
Py_UseSecureSearchPath(bool) that embedders should call to opt into the new 
search semantics. With an explicit opt-in, we can then migrate that to be the 
default over time - have the Python API warn for a release if called without 
the opt-in, and then switch the default to be the secure search path, with 
applications that want to use the old search path being able to opt out using 
Py_UseSecureSearchPath(FALSE) for a release or two.

That proposal is very much off the top of my head. But the point is that it's 
not impossible to make the transition follow the normal backward compatibility 
rules, and so we should do so.

Of course, far simpler would be to choose a solution which *doesn't* break 
existing code :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36085] Enable better DLL resolution

[Paul Moore]

Paul Moore  added the comment:

> Since everyone seems to have misunderstood at least part of the proposal, I'm 
> not going to explain any more until I have a patch. Excluding boilerplate and 
> docs, it'll be about ten lines of code.

+1 on that. Code is much harder to misunderstand :-)

Paul

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30040] new empty dict can be more small

[Mark Shannon]

Mark Shannon  added the comment:

In general, I agree with Raymond that this is likely to counter-productive.

But let's not guess, let's measure :)

I expect that there are few live empty dicts at any time for most programs. In 
which case there is no point in any change that attempts to save memory use for 
empty dicts.

But I could be wrong. If there commonly are lots of live empty dicts,
then some sort of optimisation could be appropriate.


I should also add that dict.clear() uses a key-sharing dict to avoid 
allocation, because PyDict_Clear() is a void function so there is no way to 
handle an allocation failure.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com