Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On Thu, May 4, 2017 at 4:19 AM, Terry Reedy wrote: > On 5/3/2017 7:13 PM, Koos Zevenhoven wrote: >> [...] >> Shutil was among the most important to be updated, IMO. >> >> I had made some sort of list of affected modules elsewhere [1]: >> ntpath, posixpath, os.scandir, os.[other stuff], DirEntry (tempted to >> say os.DirEntry, but that is >> not true), shutil.[stuff], (io.)open, fileinput, filecmp, zipfile, >> tarfile, tempfile (for the 'dir' keyword arguments), maybe even glob >> and fnmatch (are the patterns paths?) > > > What did not get done for 3.6 should be proposed for 3.7. > Anyone, feel free. The nightmare part is done, so this could be a case where a PR actually pays off in terms of being able to use the feature. There's no need for any unnecessary masochism (should there ever be?). [...] > > Enhancing public APIs in normal (non-provisional) modules in bugfix releases > has turned out to be a bad thing to do. Hence the policy to not do that. > The few exceptions have been necessary to fix a bug that needed to be fixed, > and could not reasonably be fixed otherwise. Such exceptions can of course more easily be made when the adoption of a version is still small, and almost all users will never see X.Y.0 or X.Y.1. The fraction of 3.6 users is probably super tiny right now, and even those users are likely to eagerly update to bugfix releases. For instance, are there any major (LTS?) linux distros that already come with 3.6.0 or 3.6.1? Well OK, 3.6.2 may be too late for some. —Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale
On 4 May 2017 at 12:24, INADA Naoki wrote: > [PEP 538] >> * PEP 540 proposes to entirely decouple CPython's default text encoding from >> the C locale system in that case, allowing text handling inconsistencies to >> arise between CPython and other locale-aware components running in the same >> process and in subprocesses. This approach aims to make CPython behave less >> like a locale-aware application, and more like locale-independent language >> runtimes like the JVM, .NET CLR, Go, Node.js, and Rust > > https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html says: > >> Every instance of the Java virtual machine has a default charset, which may >> or may not be one of the standard charsets. The default charset is >> determined during virtual-machine startup and typically depends upon the >> locale and charset being used by the underlying operating system. > > I don't know about .NET runtime on Unix much. (mono and .NET Core). > "Go, Node.js and Rust" seems enough examples. I'll push an update to drop the JVM and .NET from the list of examples. >> New build-time configuration options >> [snip] > In case of (b), while warning about C locale is not shown, warning > about coercion > is still shown. So when people don't want to see warning under C > locale and there is no > (C.UTF-8, C.utf8, UTF-8) locales, there are three ways: > > * Set PYTHONUTF=1 (if PEP 540 is accepted) > * Set PYTHONCOERCECLOCALE=0. > * Use both of ``--without-c-locale-coercion`` and > ``--without-c-locale-warning`` > configure options. > > Is my understanding right? Yes, that sounds right. > BTW, I prefer PEP 540 provides ``--with-utf8mode`` option which > enables UTF-8 mode > by default. And if it is added, there are too few use cases for > ``--without-c-locale-warning``. > > There are some use cases people want to use UTF-8 by default in system > wide. (e.g. > container, webserver in Cent OS, etc...) > > On the other hand, most of C locale usage are "per application" basis, > rather than "system wide." > configure option is not suitable for such per application setting, off course. Yeah, in addition to Barry requesting such an option in one of the earlier linux-sig reviews, my main rationale for including it is that providing both config options offers a quick compatibility fix for any distro where emitting the coercion and/or C locale warning on stderr causes problems. The only one of those that Fedora encountered in the F26 alpha was deemed a bug in the affected application (something in autotools was checking for "no output on stderr" instead of "subprocess exit code is 0", and the fix was to switch it to check the subprocess exit code), but there are enough Linux distros and BSD variants out there that I'm a lot more comfortable shipping the change with straightforward "off" switches for the stderr output. > But I don't propose removing the option from PEP 538. > We can discuss about reducing configure options later. +1. >> On platforms where they would have no effect (e.g. Mac OS X, iOS, Android, >> Windows) these preprocessor variables would always be undefined. > > Why ``--with[out]-c-locale-coercion`` have no effect on macOS, iOS and > Android? On these three, we know the system encoding is UTF-8, so we never interpreted the C locale as meaning "ascii" in the first place. > On Android, locale coercion fixes readline. Do you mean locale > coercion happen always > regardless this configuration option? Right, the change for Android is that we switch to calling 'setlocale(LC_ALL, "C.UTF-8")' during interpreter startup instead of 'setlocale(LC_ALL, "")'. That change is guarded by "#ifdef __ANDROID__", rather than either of the new conditionals. > On macOS, ``LC_ALL=C python`` doesn't make Python's stdio to > ``ascii:surrogateescape``? Similar to Android, CPython itself is hardcoded to assume UTF-8 on Mac OS X, since that's a platform API guarantee that users can't change. > Even so, locale coercion may fix libraries like readline, curses. > While C locale is less common on macOS, I don't understand any > reason to disable it on macOS. My understanding is that other libraries and applications also automatically use UTF-8 for system interfaces on Mac OS X and iOS. It could be that that understanding is wrong, and locale coercion would provide a benefit there as well. (Checking the draft implementation, it turns out I haven't actually implemented the configure logic to make those config settings platform dependent yet - they're currently only undefined on Windows by default, since that doesn't use the autotools based build system) > > I know almost nothing about iOS, but it's similar to Android or macOS > in my expectation. > > >> Improving the handling of the C locale >> -- >> > ... >> locale settings for locale-aware operations. Both the JVM and the .NET CLR >> use UTF-16-LE as their primary encoding for
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On 4 May 2017 at 04:15, Brett Cannon wrote: > My allergies have hit me hard so I'm not thinking at full capacity, but did > we ever decide if supporting os.PathLike in the stdlib was viewed as an > enhancement or bugfix? Specifically I'm thinking of > https://bugs.python.org/issue30218 for adding support to > shutil.unpack_archive() and whether it should be backported to 3.6. I'd treat it as an enhancement, similar to adding native context management support to various APIs. Otherwise, as Terry pointed out, we're setting folks up for cases where testing on the latest maintenance release seems fine, but their app breaks when deployed on a slightly older version. It's one thing for that to happen when the app is encountering a genuine standard library bug, but something else entirely when they're instead inadvertently relying on an enhancement that we could have deferred to the next feature release. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On 03.05.17 21:15, Brett Cannon wrote: My allergies have hit me hard so I'm not thinking at full capacity, but did we ever decide if supporting os.PathLike in the stdlib was viewed as an enhancement or bugfix? Specifically I'm thinking of https://bugs.python.org/issue30218 for adding support to shutil.unpack_archive() and whether it should be backported to 3.6. I concur with Terry, Victor and Nick. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale
On Thu, 4 May 2017 11:24:27 +0900 INADA Naoki wrote: > Hi, Nick and all core devs who are interested in this PEP. > > I'm reviewing PEP 538 and I want to accept it in this month. > It will reduces much UnicodeError pains which server-side OPs facing. > Thank you Nick for working on this PEP. > > If you have something worrying about this PEP, please post a comment > soon. If you don't have enough time to read entire this PEP, feel free to > ask a question about you're worrying. >From my POV, it is problematic that the behaviour outlined in PEP 538 (see Abstract section) varies depending on the adoption of another PEP (PEP 540). If we want to adopt PEP 538 before pronouncing on PEP 540, then PEP 538 should remove all points conditional on PEP 540 adoption, and PEP 540 should later be changed to adopt those removed points as PEP 540-specific changes. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] "make touch" replaced with "make regen-all"
Hi, tl;dr Are you ok to backport my change replacing "make touch" with "make regen-all"? (commit a5c62a8e) Since the creation of CPython, generated files were regenerated depending on file modification time. For development, that's a convenient feature. But in practice, it caused a long list of pain points. It caused me many issues in my experience: * On Solaris, Python failed to regenerated the AST because only the system Python was Python 2.6 and the script required Python 2.7 or newer. The "make touch" workaround didn't help, also because of the old version of the system Python. * On FreeBSD, generated files require "python" but only "python2.7" or "python3.6" programs are available. In The build system was enhanced to try pythonX.Y and then "python". * "make touch" workaround requires Mercurial, but also a specific version of Mercurial: more than once, "make touch" failed because my Mercurial version was too old. * Since CPython migrated to Git, "make touch" doesn't work anymore. Sorry, I didn't check why exactly, but I would prefer to not depend on Git *and* Mercurial. For all these reasons, it was decided to modify the CPython (UNIX/BSD) build system to not regenerate generated files based on file modification time anymore, but require an explicit action: "make regen-all". I already pushed my change to the master branch: https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5e102b --- commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b Author: Victor Stinner Date: Wed May 3 18:21:48 2017 +0200 bpo-23404: make touch becomes make regen-all (#1405) Don't rebuild generated files based on file modification time anymore, the action is now explicit. Replace "make touch" with "make regen-all". Changes: * Remove "make touch", Tools/hg/hgtouch.py and .hgtouch * Add a new "make regen-all" command to rebuild all generated files * Add subcommands to only generate specific files: - regen-ast: Include/Python-ast.h and Python/Python-ast.c - regen-grammar: Include/graminit.h and Python/graminit.c - regen-importlib: Python/importlib_external.h and Python/importlib.h - regen-opcode: Include/opcode.h - regen-opcode-targets: Python/opcode_targets.h - regen-typeslots: Objects/typeslots.inc * Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN * pgen is now only built by by "make regen-grammar" * Add $(srcdir)/ prefix to paths to source files to handle correctly compilation outside the source directory Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make" default target building Python. --- See the issue for the full rationale: http://bugs.python.org/issue23404 My commit fixed the two remaining FreeBSD buildbots which were broken because of broken "make touch". All FreeBSD buildbots are repaired on master! Ok, now the question is: are you ok to backport this change to Python 2.7, 3.5 and 3.6? I started with a backport to 3.6: https://github.com/python/cpython/pull/1461 See also "Test somehow that generated files are up to date: run make regen-all" issue: http://bugs.python.org/issue30259 Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
Thanks for the feedback, everyone. I'll just consider it an enhancement then. On Wed, 3 May 2017 at 15:07 Terry Reedy wrote: > On 5/3/2017 2:15 PM, Brett Cannon wrote: > > My allergies have hit me hard so I'm not thinking at full capacity, but > > did we ever decide if supporting os.PathLike in the stdlib was viewed as > > an enhancement or bugfix? Specifically I'm thinking of > > https://bugs.python.org/issue30218 for adding support to > > shutil.unpack_archive() and whether it should be backported to 3.6. > > On the face of it, that particular issue looks like an enhancement that > should have gone into 3.6 (if ever), but did not. I notice that > https://www.python.org/dev/peps/pep-0519/#implementation > did not include "Update shutil", so it was not done, at least not > completely. > > Was shutil updated at all? Probably not explicitly. A lot of support just fell through thanks to updating os and os.path to support path-like objects. > Is unpack_archive the only shutil function > not updated? Not sure. I opened https://bugs.python.org/issue30235 to track finding out. -Brett If so, I could see the omission as a bug. > If the patch for 30218 were applied in 3.6, would the doc > https://docs.python.org/3/library/shutil.html#shutil.unpack_archive > need to be changed, with a note "Added in 3.6.2: filename can be any > pathlike object"? If so, it is an enhancement. > > -- > Terry Jan Reedy > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On 5/4/2017 10:43 AM, Koos Zevenhoven wrote: On Thu, May 4, 2017 at 4:19 AM, Terry Reedy wrote: What did not get done for 3.6 should be proposed for 3.7. Anyone, feel free. The nightmare part is done, so this could be a case where a PR actually pays off in terms of being able to use the feature. There's no need for any unnecessary masochism (should there ever be?). I have no idea what this means, but don't need to. Enhancing public APIs in normal (non-provisional) modules in bugfix releases has turned out to be a bad thing to do. Hence the policy to not do that. The few exceptions have been necessary to fix a bug that needed to be fixed, and could not reasonably be fixed otherwise. Such exceptions can of course more easily be made when the adoption of a version is still small, and almost all users will never see X.Y.0 or X.Y.1. This is not an allowed excuse for breaking the policy. The x.y language is defined when x.y.0 is released. Please stop. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "make touch" replaced with "make regen-all"
I see no issue backporting since I don't think we have any compatibility promises when it comes to Makefile commands. Plus if the perf changes to add PGO support could be backported then I don't see why this shouldn't be allowed. On Thu, 4 May 2017 at 10:15 Victor Stinner wrote: > Hi, > > tl;dr Are you ok to backport my change replacing "make touch" with > "make regen-all"? (commit a5c62a8e) > > > Since the creation of CPython, generated files were regenerated > depending on file modification time. For development, that's a > convenient feature. But in practice, it caused a long list of pain > points. It caused me many issues in my experience: > > * On Solaris, Python failed to regenerated the AST because only the > system Python was Python 2.6 and the script required Python 2.7 or > newer. The "make touch" workaround didn't help, also because of the > old version of the system Python. > > * On FreeBSD, generated files require "python" but only "python2.7" or > "python3.6" programs are available. In The build system was enhanced > to try pythonX.Y and then "python". > > * "make touch" workaround requires Mercurial, but also a specific > version of Mercurial: more than once, "make touch" failed because my > Mercurial version was too old. > > * Since CPython migrated to Git, "make touch" doesn't work anymore. > Sorry, I didn't check why exactly, but I would prefer to not depend on > Git *and* Mercurial. > > > For all these reasons, it was decided to modify the CPython (UNIX/BSD) > build system to not regenerate generated files based on file > modification time anymore, but require an explicit action: "make > regen-all". I already pushed my change to the master branch: > > > https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5e102b > > --- > commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b > Author: Victor Stinner > Date: Wed May 3 18:21:48 2017 +0200 > > bpo-23404: make touch becomes make regen-all (#1405) > > Don't rebuild generated files based on file modification time > anymore, the action is now explicit. Replace "make touch" > with "make regen-all". > > Changes: > > * Remove "make touch", Tools/hg/hgtouch.py and .hgtouch > * Add a new "make regen-all" command to rebuild all generated files > * Add subcommands to only generate specific files: > > - regen-ast: Include/Python-ast.h and Python/Python-ast.c > - regen-grammar: Include/graminit.h and Python/graminit.c > - regen-importlib: Python/importlib_external.h and Python/importlib.h > - regen-opcode: Include/opcode.h > - regen-opcode-targets: Python/opcode_targets.h > - regen-typeslots: Objects/typeslots.inc > > * Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN > * pgen is now only built by by "make regen-grammar" > * Add $(srcdir)/ prefix to paths to source files to handle correctly > compilation outside the source directory > > Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make" > default target building Python. > --- > > > See the issue for the full rationale: > >http://bugs.python.org/issue23404 > > My commit fixed the two remaining FreeBSD buildbots which were broken > because of broken "make touch". All FreeBSD buildbots are repaired on > master! > > Ok, now the question is: are you ok to backport this change to Python > 2.7, 3.5 and 3.6? > > I started with a backport to 3.6: > > https://github.com/python/cpython/pull/1461 > > > See also "Test somehow that generated files are up to date: run make > regen-all" issue: > http://bugs.python.org/issue30259 > > Victor > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "make touch" replaced with "make regen-all"
Yes. It is perfectly reasonable to backport improvements to the tooling as long as it doesn't break anyone's existing build process. Sent from my iPhone > On May 4, 2017, at 10:13 AM, Victor Stinner wrote: > > tl;dr Are you ok to backport my change replacing "make touch" with > "make regen-all"? (commit a5c62a8e) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On Wed, May 3, 2017 at 9:15 PM, Brett Cannon wrote: > My allergies have hit me hard so I'm not thinking at full capacity, but did > we ever decide if supporting os.PathLike in the stdlib was viewed as an > enhancement or bugfix? Specifically I'm thinking of > https://bugs.python.org/issue30218 for adding support to > shutil.unpack_archive() and whether it should be backported to 3.6. We've already backported a few patches that improves the PEP 519 support in the stdlib with the permission from the release manager of 3.6. I'd ask Ned whether bpo-30218 qualifies for backporting to 3.6. --Berker ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On 04.05.17 21:01, Berker Peksağ wrote: We've already backported a few patches that improves the PEP 519 support in the stdlib with the permission from the release manager of 3.6. I'd ask Ned whether bpo-30218 qualifies for backporting to 3.6. AFAIK it was before releasing 3.6.1. Some users avoid using x.0 versions of software. For 3.6.2 we can have stronger rule. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "make touch" replaced with "make regen-all"
FWIW this will also make cross-compiling a lot easier, since you can't accidentally overwrite the cross-compiled pgen as easily. On Thu, May 4, 2017 at 12:13 PM, Victor Stinner wrote: > Hi, > > tl;dr Are you ok to backport my change replacing "make touch" with > "make regen-all"? (commit a5c62a8e) > > > Since the creation of CPython, generated files were regenerated > depending on file modification time. For development, that's a > convenient feature. But in practice, it caused a long list of pain > points. It caused me many issues in my experience: > > * On Solaris, Python failed to regenerated the AST because only the > system Python was Python 2.6 and the script required Python 2.7 or > newer. The "make touch" workaround didn't help, also because of the > old version of the system Python. > > * On FreeBSD, generated files require "python" but only "python2.7" or > "python3.6" programs are available. In The build system was enhanced > to try pythonX.Y and then "python". > > * "make touch" workaround requires Mercurial, but also a specific > version of Mercurial: more than once, "make touch" failed because my > Mercurial version was too old. > > * Since CPython migrated to Git, "make touch" doesn't work anymore. > Sorry, I didn't check why exactly, but I would prefer to not depend on > Git *and* Mercurial. > > > For all these reasons, it was decided to modify the CPython (UNIX/BSD) > build system to not regenerate generated files based on file > modification time anymore, but require an explicit action: "make > regen-all". I already pushed my change to the master branch: > > https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5e102b > > --- > commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b > Author: Victor Stinner > Date: Wed May 3 18:21:48 2017 +0200 > > bpo-23404: make touch becomes make regen-all (#1405) > > Don't rebuild generated files based on file modification time > anymore, the action is now explicit. Replace "make touch" > with "make regen-all". > > Changes: > > * Remove "make touch", Tools/hg/hgtouch.py and .hgtouch > * Add a new "make regen-all" command to rebuild all generated files > * Add subcommands to only generate specific files: > > - regen-ast: Include/Python-ast.h and Python/Python-ast.c > - regen-grammar: Include/graminit.h and Python/graminit.c > - regen-importlib: Python/importlib_external.h and Python/importlib.h > - regen-opcode: Include/opcode.h > - regen-opcode-targets: Python/opcode_targets.h > - regen-typeslots: Objects/typeslots.inc > > * Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN > * pgen is now only built by by "make regen-grammar" > * Add $(srcdir)/ prefix to paths to source files to handle correctly > compilation outside the source directory > > Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make" > default target building Python. > --- > > > See the issue for the full rationale: > >http://bugs.python.org/issue23404 > > My commit fixed the two remaining FreeBSD buildbots which were broken > because of broken "make touch". All FreeBSD buildbots are repaired on > master! > > Ok, now the question is: are you ok to backport this change to Python > 2.7, 3.5 and 3.6? > > I started with a backport to 3.6: > > https://github.com/python/cpython/pull/1461 > > > See also "Test somehow that generated files are up to date: run make > regen-all" issue: > http://bugs.python.org/issue30259 > > Victor > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://refi64.com/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On Thu, May 4, 2017 at 8:30 PM, Terry Reedy wrote: > On 5/4/2017 10:43 AM, Koos Zevenhoven wrote: >> On Thu, May 4, 2017 at 4:19 AM, Terry Reedy wrote: >>> Enhancing public APIs in normal (non-provisional) modules in bugfix >>> releases >>> has turned out to be a bad thing to do. Hence the policy to not do that. >>> The few exceptions have been necessary to fix a bug that needed to be >>> fixed, >>> and could not reasonably be fixed otherwise. >> >> Such exceptions can of course more easily be made when the adoption of >> a version is still small, and almost all users will never see X.Y.0 or >> X.Y.1. > > This is not an allowed excuse for breaking the policy. The x.y language is > defined when x.y.0 is released. Please stop. > Don't worry, I didn't even start :) I do think it can cause problems if most of a stdlib module supports PathLike and some parts do not. People will write code in 3.7 believing it works on 3.6, while it doesn't. Anyway, I'm completely happy if the policy outweighs this issue, and I have absolutely no need to argue about the decision. —Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale
On Sat, Mar 4, 2017 at 11:50 PM, Nick Coghlan wrote: > > Providing implicit locale coercion only when running standalone > --- > > Over the course of Python 3.x development, multiple attempts have been made > to improve the handling of incorrect locale settings at the point where the > Python interpreter is initialised. The problem that emerged is that this is > ultimately *too late* in the interpreter startup process - data such as > command > line arguments and the contents of environment variables may have already > been > retrieved from the operating system and processed under the incorrect ASCII > text encoding assumption well before ``Py_Initialize`` is called. > > The problems created by those inconsistencies were then even harder to > diagnose > and debug than those created by believing the operating system's claim that > ASCII was a suitable encoding to use for operating system interfaces. This > was > the case even for the default CPython binary, let alone larger C/C++ > applications that embed CPython as a scripting engine. > > The approach proposed in this PEP handles that problem by moving the locale > coercion as early as possible in the interpreter startup sequence when > running > standalone: it takes place directly in the C-level ``main()`` function, even > before calling in to the `Py_Main()`` library function that implements the > features of the CPython interpreter CLI. > > The ``Py_Initialize`` API then only gains an explicit warning (emitted on > ``stderr``) when it detects use of the ``C`` locale, and relies on the > embedding application to specify something more reasonable. > It feels like having a short section on the caveats of this approach would help to introduce this section. Something that says that this PEP can cause a split in how Python behaves in non-sandalone applications (mod_wsgi, IDEs where libpython is compiled in, etc) vs standalone (unless the embedders take similar steps as standalone python is doing). Then go on to state that this approach was still chosen as coercing in Py_Initialize is too late, causing the inconsistencies and problems listed here. -Toshio ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "make touch" replaced with "make regen-all"
On 5 May 2017 at 03:40, Brett Cannon wrote: > I see no issue backporting since I don't think we have any compatibility > promises when it comes to Makefile commands. Plus if the perf changes to add > PGO support could be backported then I don't see why this shouldn't be > allowed. For the benefit of Linux distros attempting to ensure they're doing full "from source" builds, it would be good to note this in a "Notable changes in maintenance releases", akin to the existing ones for 3.4 and 2.7 (perhaps retitling the latter accordingly). The note just needs to say that folks that care about doing "complete" builds need to adjust their command sequence to be "./configure && make regen-all && make install", rather than the previous pattern of "./configure && make install". This should also make bootstrapping easier, since bootstrap builds can skip the "make regen-all" step and instead rely on the checked in pre-generated files. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On 4 May 2017 at 08:25, Victor Stinner wrote: > If you start to backport support for the fspath protocol, be prepared > to have to backport it *many* places. I expect that slowly in the near > future, many functions will be patched to support the fspath protocol. > > I suggest to only do that in master. It's not that hard to cast > manually to a string: it's just str(path), no? For 3.6+, "os.fspath(path)" would be more robust - unlike str(), it will still error out immediately if you pass it something completely inappropriate. That improved casting mechanism and the implicit support in the low level APIs is the main benefit I see in PEP 519, and if we were talking about an os module API that was missing os.path support, I'd be more likely to be on the "it's a bug" side of things. It's only higher level APIs like shutil that I put into the same category as third party libraries for now: Python 3.6 users shouldn't expect implicit use of the fspath protocol to be universal yet, so they may still need some explicit calls to os.fspath() in their own code to get different APIs to work well together. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale
On 5 May 2017 at 02:25, Antoine Pitrou wrote: > On Thu, 4 May 2017 11:24:27 +0900 > INADA Naoki wrote: >> Hi, Nick and all core devs who are interested in this PEP. >> >> I'm reviewing PEP 538 and I want to accept it in this month. >> It will reduces much UnicodeError pains which server-side OPs facing. >> Thank you Nick for working on this PEP. >> >> If you have something worrying about this PEP, please post a comment >> soon. If you don't have enough time to read entire this PEP, feel free to >> ask a question about you're worrying. > > From my POV, it is problematic that the behaviour outlined in PEP 538 > (see Abstract section) varies depending on the adoption of another PEP > (PEP 540). > > If we want to adopt PEP 538 before pronouncing on PEP 540, then PEP 538 > should remove all points conditional on PEP 540 adoption, and PEP 540 > should later be changed to adopt those removed points as PEP > 540-specific changes. While I won't be certain until I update the PEP and reference implementation, I'm pretty sure Inada-san's suggestion to replace the call to Py_SetStandardStreamEncoding with defaulting to surrogateescape on the standard streams in the C.UTF-8 locale will remove this current dependency between the PEPs as well as making the "C.UTF-8 locale" and "C locale coerced to C.UTF-8" behaviour indistinguishable at runtime (aside from the stderr warning in the latter case). It will then be up to Victor to state in PEP 540 how locale coercion will interact with Python UTF-8 mode (with my recommendation being the one currently in PEP 538: it should implicitly set the environment variable, so the mode activation is inherited by subprocesses) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "make touch" replaced with "make regen-all"
Le 5 mai 2017 6:31 AM, "Nick Coghlan" a écrit : The note just needs to say that folks that care about doing "complete" builds need to adjust their command sequence to be "./configure && make regen-all && make install", rather than the previous pattern of "./configure && make install". Hum, you only need to regenerate files if you modify Grammar, AST or opcodes. Do you mean that Linux distro make such downstream changes? Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "make touch" replaced with "make regen-all"
On 5 May 2017 at 16:01, Victor Stinner wrote: > Le 5 mai 2017 6:31 AM, "Nick Coghlan" a écrit : > > The note just needs to say that folks that care about doing "complete" > builds need to adjust their command sequence to be "./configure > && make regen-all && make install", rather than the previous > pattern of "./configure && make install". > > > Hum, you only need to regenerate files if you modify Grammar, AST or > opcodes. Do you mean that Linux distro make such downstream changes? No, but at least some of them do have a policy that it be possible to rebuild the project from its "original sources" in the distro's build system without relying on cached artifacts provided by the upstream project. In CPython's case, that means distros with such a policy should ensure they're running the "regen-all" step in addition to the C build steps. To be 100% clear, I agree it makes sense to change the default behaviour to be to re-use our checked in generated files, and to make that policy consistent across all active branches. The request for a note is just to make it clear to redistributors that CPython now has two potential tiers of build dependencies (those required to run "make" and those required to run "make regen-all"), so if a redistributor cares about that kind of thing, they're going to have to update their build processes accordingly. If they don't make they change, they may inadvertently lose the ability to run the "regen-all" step at some point in the future (as an extreme example: to avoid a tricky bootstrapping problem, while still improving the tools available for standard library maintenance, we may someday decide to allow cffi and/or cython as file regeneration dependencies, while continuing to disallow them as conventional build dependencies). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
