[Python-Dev] subprocess, buffered files, pipes and broken pipe errors
Hi, => I propose to ignore BrokenPipeError on Popen.__exit__(), what do you think? A recent issue fixed subprocess.Popen.__exit__() to read the exit status of the child process, even if stdin.close() raised a BrokenPipeError: http://bugs.python.org/issue21619 When I saw the issue, I was surprised that Popen.__exit__() doesn't ignore BrokenPipeError. Popen.communicate() is designed to handle corner cases and it ignores BrokenPipeError. Popen.__exit__() only returns when the process exited. The method starts by closing all pipes. Usually the child process notices that stdin was closed and exit (programs used as "pipes" like grep, cat, wc, etc.). The problem is that Popen uses a buffered writer for Popen.stdin with a default buffer size of 8 KB. We may buffer some data and so stdin.close() will call stdin.write(). If the child process exited or closed stdin, the pipe is broken, and the write in the parent process (stdin.close() in Popen.__exit__) will raise BrokenPipeError. I propose to ignore BrokenPipeError in Popen.__exit__, as done in communicate(), for convinience: http://bugs.python.org/issue23570 Serhiy wants to keep BrokenPipeError, he wrote that file.close() should not ignore write errors (read the issue for details). I consider that BrokenPipeError on a pipe is different than a write error on a regular file. EPIPE and SIGPIPE are designed to notify that the pipe is closed and that it's now inefficient to continue to write into this pipe. Ignoring BrokenPipeError in Popen.__exit__() respects this constrain because the method closes stdin and only returns when the process exited. So the caller will not write anything into stdin anymore. What do you think? Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess, buffered files, pipes and broken pipe errors
On 06.03.15 14:53, Victor Stinner wrote:
I propose to ignore BrokenPipeError in Popen.__exit__, as done in
communicate(), for convinience:
http://bugs.python.org/issue23570
Serhiy wants to keep BrokenPipeError, he wrote that file.close()
should not ignore write errors (read the issue for details).
I rather said about file.__exit__.
I consider that BrokenPipeError on a pipe is different than a write
error on a regular file.
EPIPE and SIGPIPE are designed to notify that the pipe is closed and
that it's now inefficient to continue to write into this pipe.
And into the file like open('/dev/stdout', 'wb').
Ignoring BrokenPipeError in Popen.__exit__() respects this constrain
because the method closes stdin and only returns when the process
exited. So the caller will not write anything into stdin anymore.
And the caller will not write anything into the file after calling
file.__exit__.
I don't see large difference between open('file', 'wb') and Popen('cat
>file', stdin=PIPE), between sys.stdout with redirecting stdout and
running external program with Pipe().
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 488: elimination of PYO files
Over on the import-sig I proposed eliminating the concept of .pyo files
since they only signify that *some* optimization took place, not
*what* optimizations
took place. Everyone on the SIG was positive with the idea so I wrote a
PEP, got positive feedback from the SIG again, and so now I present to you
PEP 488 for discussion.
There is no patch yet, but this is not a complicated change and I could get
it done at the sprints at PyCon if necessary (I suspect updating the test
suite will take the most work).
There are currently two open issues, although one is purely a bikeshed
topic on formatting of file names so I don't really consider it open for
change from what is proposed in the PEP without Guido saying he hates my
preference or someone having a really good argument for some alternative.
The second open issue on the common case file name is something to
reasonably debate and come to consensus on.
===
PEP: 488
Title: Elimination of PYO files
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Feb-2015
Post-History:
2015-03-06
Abstract
This PEP proposes eliminating the concept of PYO files from Python.
To continue the support of the separation of bytecode files based on
their optimization level, this PEP proposes extending the PYC file
name to include the optimization level in bytecode repository
directory (i.e., the ``__pycache__`` directory).
Rationale
=
As of today, bytecode files come in two flavours: PYC and PYO. A PYC
file is the bytecode file generated and read from when no
optimization level is specified at interpreter startup (i.e., ``-O``
is not specified). A PYO file represents the bytecode file that is
read/written when **any** optimization level is specified (i.e., when
``-O`` is specified, including ``-OO``). This means that while PYC
files clearly delineate the optimization level used when they were
generated -- namely no optimizations beyond the peepholer -- the same
is not true for PYO files. Put in terms of optimization levels and
the file extension:
- 0: ``.pyc``
- 1 (``-O``): ``.pyo``
- 2 (``-OO``): ``.pyo``
The reuse of the ``.pyo`` file extension for both level 1 and 2
optimizations means that there is no clear way to tell what
optimization level was used to generate the bytecode file. In terms
of reading PYO files, this can lead to an interpreter using a mixture
of optimization levels with its code if the user was not careful to
make sure all PYO files were generated using the same optimization
level (typically done by blindly deleting all PYO files and then
using the `compileall` module to compile all-new PYO files [1]_).
This issue is only compounded when people optimize Python code beyond
what the interpreter natively supports, e.g., using the astoptimizer
project [2]_.
In terms of writing PYO files, the need to delete all PYO files
every time one either changes the optimization level they want to use
or are unsure of what optimization was used the last time PYO files
were generated leads to unnecessary file churn. The change proposed
by this PEP also allows for **all** optimization levels to be
pre-compiled for bytecode files ahead of time, something that is
currently impossible thanks to the reuse of the ``.pyo`` file
extension for multiple optimization levels.
As for distributing bytecode-only modules, having to distribute both
``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
of code obfuscation and smaller file deployments.
Proposal
To eliminate the ambiguity that PYO files present, this PEP proposes
eliminating the concept of PYO files and their accompanying ``.pyo``
file extension. To allow for the optimization level to be unambiguous
as well as to avoid having to regenerate optimized bytecode files
needlessly in the `__pycache__` directory, the optimization level
used to generate a PYC file will be incorporated into the bytecode
file name. Currently bytecode file names are created by
``importlib.util.cache_from_source()``, approximately using the
following expression defined by PEP 3147 [3]_, [4]_, [5]_::
'{name}.{cache_tag}.pyc'.format(name=module_name,
cache_tag=sys.implementation.cache_tag)
This PEP proposes to change the expression to::
'{name}.{cache_tag}.opt-{optimization}.pyc'.format(
name=module_name,
cache_tag=sys.implementation.cache_tag,
optimization=str(sys.flags.optimize))
The "opt-" prefix was chosen so as to provide a visual separator
from the cache tag. The placement of the optimization level after
the cache tag was chosen to preserve lexicographic sort order of
bytecode file names based on module name and cache tag which will
not vary for a single interpreter. The "opt-" prefix was chosen over
"o" so as to be somewhat self-documenting. The "opt-" prefix was
chosen over "O" so as to not have any confusi
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2015-02-27 - 2015-03-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open4804 (+16) closed 30555 (+45) total 35359 (+61) Open issues with patches: 2260 Issues opened (47) == #22079: Ensure in PyType_Ready() that base class of static type is sta http://bugs.python.org/issue22079 reopened by doko #23538: New Windows installer in 3.5.0a1 breaks compatibility with Win http://bugs.python.org/issue23538 opened by Link Mauve #23539: Content-length not set for HTTP methods expecting body when bo http://bugs.python.org/issue23539 opened by demian.brecht #23540: Proposal for asyncio: SubprocessTransport.detach() to detach a http://bugs.python.org/issue23540 opened by martius #23542: Update PEP 476 for using urllib2.build_opener() http://bugs.python.org/issue23542 opened by Shakeel Mohamed #23543: encoding error trying to save string to file http://bugs.python.org/issue23543 opened by Gravitania #23544: IDLE hangs when selecting Stack View with debug active http://bugs.python.org/issue23544 opened by Andrew.Harrington #23545: Turn on extra warnings on GCC http://bugs.python.org/issue23545 opened by serhiy.storchaka #23546: windows, IDLE and pep 397 http://bugs.python.org/issue23546 opened by Liam.Marsh #23549: heapq docs should be more precise about how to access the smal http://bugs.python.org/issue23549 opened by eli.bendersky #23550: Add to unicodedata a function to query the "Quick_Check" prope http://bugs.python.org/issue23550 opened by Hammerite #23551: IDLE to provide menu options for using PIP http://bugs.python.org/issue23551 opened by rhettinger #23552: Have timeit warn about runs that are not independent of each o http://bugs.python.org/issue23552 opened by rhettinger #23553: Reduce the number of comparisons for range checking. http://bugs.python.org/issue23553 opened by rhettinger #23554: EchoServerClientProtocol class should be called EchoServerProt http://bugs.python.org/issue23554 opened by gc #23555: behavioural change in argparse set_defaults in python 2.7.9 http://bugs.python.org/issue23555 opened by doko #23556: Scope for raise without argument is different in Python 2 and http://bugs.python.org/issue23556 opened by a3nm #23557: Misc/SpecialBuilds.txt contains outdated information about PYM http://bugs.python.org/issue23557 opened by Jakub Klama #23560: Group the docs of similar methods in stdtypes.rst http://bugs.python.org/issue23560 opened by ezio.melotti #23564: Patch fixing sanity check for ordered fd sequence in _posixsub http://bugs.python.org/issue23564 opened by Hisham Muhammad #23565: local_clear walks the list of threads without holding head_loc http://bugs.python.org/issue23565 opened by stutzbach #23566: RFE: faulthandler.register() should support file descriptors http://bugs.python.org/issue23566 opened by haypo #23567: os.stat() tuple access vs named attribute access int vs float http://bugs.python.org/issue23567 opened by gregory.p.smith #23568: unittest.mock.MagicMock doesn't support __rdivmod__ http://bugs.python.org/issue23568 opened by zkrynicki #23570: Change "with subprocess.Popen():" (context manager) to ignore http://bugs.python.org/issue23570 opened by haypo #23571: Raise SystemError if a function returns a result with an excep http://bugs.python.org/issue23571 opened by haypo #23572: functools.singledispatch fails when "not BaseClass" is True http://bugs.python.org/issue23572 opened by Sergio Pascual #23573: Avoid redundant allocations in str.find and like http://bugs.python.org/issue23573 opened by serhiy.storchaka #23574: datetime: support leap seconds http://bugs.python.org/issue23574 opened by haypo #23575: MIPS64 needs ffi's n32.S http://bugs.python.org/issue23575 opened by Simon Hoinkis #23577: Add tests for wsgiref.validate http://bugs.python.org/issue23577 opened by ashkop #23578: struct.pack error messages do not indicate which argument was http://bugs.python.org/issue23578 opened by alynn #23579: Amazon.com links http://bugs.python.org/issue23579 opened by Edrie Ddrie #23581: unittest.mock.MagicMock doesn't support matmul (@) operator http://bugs.python.org/issue23581 opened by zkrynicki #23583: IDLE: printing unicode subclasses broken (again) http://bugs.python.org/issue23583 opened by mjpieters #23584: test_doctest lineendings fails in verbose mode http://bugs.python.org/issue23584 opened by jbeck #23585: patchcheck doesn't depend on all http://bugs.python.org/issue23585 opened by rbcollins #23587: asyncio: use the new traceback.TracebackException class http://bugs.python.org/issue23587 opened by haypo #23588: Errno conflicts in ssl.SSLError http://bugs.python.org/issue23588 opened by Ben.Darnell #23589: Redundant sentence in FAQ http://bugs.python.org/issue23589 opened by fossilet #23591: Add IntFla
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 9:34 AM, Brett Cannon wrote: > Not specifying the optimization level when it is at 0 > - > > It has been suggested that for the common case of when the > optimizations are at level 0 that the entire part of the file name > relating to the optimization level be left out. This would allow for > file names of ``.pyc`` files to go unchanged, potentially leading to > less backwards-compatibility issues (although Python 3.5 introduces a > new magic number for bytecode so all bytecode files will have to be > regenerated regardless of the outcome of this PEP). > > It would also allow a potentially redundant bit of information to be > left out of the file name if an implementation of Python did not > allow for optimizing bytecode. This would only occur, though, if the > interpreter didn't support ``-O`` **and** didn't implement the ast > module, else users could implement their own optimizations. The presence of the "opt-0" part in the filename could imply that there are other supported optimization levels, even when there aren't. So leaving that out when optimizations aren't supported may be a good idea. Perhaps add sys.implementation.supports_optimization or something like that? Then only leave "opt-0" off if the implementation does not support any optimization. --eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On 06/03/15 16:34, Brett Cannon wrote: Over on the import-sig I proposed eliminating the concept of .pyo files since they only signify that /some/ optimization took place, not /what/ optimizations took place. Everyone on the SIG was positive with the idea so I wrote a PEP, got positive feedback from the SIG again, and so now I present to you PEP 488 for discussion. [snip] Historically -O and -OO have been the antithesis of optimisation, they change the behaviour of the program with no noticeable effect on performance. If a change is to be made, why not just drop .pyo files and be done with it? Any worthwhile optimisation needs to be done at runtime or involve much more than tweaking bytecode. Cheers, Mark. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > > On 06/03/15 16:34, Brett Cannon wrote: > > Over on the import-sig I proposed eliminating the concept of .pyo files > > since they only signify that /some/ optimization took place, not > > /what/ optimizations took place. Everyone on the SIG was positive with > > the idea so I wrote a PEP, got positive feedback from the SIG again, and > > so now I present to you PEP 488 for discussion. > > > [snip] > > Historically -O and -OO have been the antithesis of optimisation, they > change the behaviour of the program with no noticeable effect on > performance. > If a change is to be made, why not just drop .pyo files and be done with > it? > I disagree with your premise that .pyo files don't have a noticeable effect on performance. If you don't use asserts a lot then there is no effect, but if you use them heavily or have them perform expensive calculations then there is an impact. And the dropping of docstrings does have an impact on memory usage when you use Python at scale. You're also assuming that we will never develop an AST optimizer that will go beyond what the peepholer can do based on raw bytecode, or something that involves a bit of calculation and thus something you wouldn't want to do at startup. > > Any worthwhile optimisation needs to be done at runtime or involve much > more than tweaking bytecode. > I disagree again. If you do something like whole program analysis and want to use that to optimize something, you will surface that through bytecode and not editing the source. So while you are doing "much more than tweaking bytecode" externally to Python, you still have to surface to the interpreter through bytecode. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > > > On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > >> >> On 06/03/15 16:34, Brett Cannon wrote: >> > Over on the import-sig I proposed eliminating the concept of .pyo files >> > since they only signify that /some/ optimization took place, not >> > /what/ optimizations took place. Everyone on the SIG was positive with >> > the idea so I wrote a PEP, got positive feedback from the SIG again, and >> > so now I present to you PEP 488 for discussion. >> > >> [snip] >> >> Historically -O and -OO have been the antithesis of optimisation, they >> change the behaviour of the program with no noticeable effect on >> performance. >> If a change is to be made, why not just drop .pyo files and be done with >> it? >> > > I disagree with your premise that .pyo files don't have a noticeable > effect on performance. If you don't use asserts a lot then there is no > effect, but if you use them heavily or have them perform expensive > calculations then there is an impact. And the dropping of docstrings does > have an impact on memory usage when you use Python at scale. > > You're also assuming that we will never develop an AST optimizer that will > go beyond what the peepholer can do based on raw bytecode, or something > that involves a bit of calculation and thus something you wouldn't want to > do at startup. > I don't want to speak for him, but you're going to get the best results optimizing ASTs at runtime, which is what I thought he was suggesting. Trying to optimize Python at compile time is setting your sights really low. You have so little information then. > > >> >> Any worthwhile optimisation needs to be done at runtime or involve much >> more than tweaking bytecode. >> > > I disagree again. If you do something like whole program analysis and want > to use that to optimize something, you will surface that through bytecode > and not editing the source. So while you are doing "much more than tweaking > bytecode" externally to Python, you still have to surface to the > interpreter through bytecode. > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar wrote: > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > >> >> >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: >> >>> >>> On 06/03/15 16:34, Brett Cannon wrote: >>> > Over on the import-sig I proposed eliminating the concept of .pyo files >>> > since they only signify that /some/ optimization took place, not >>> > /what/ optimizations took place. Everyone on the SIG was positive with >>> > the idea so I wrote a PEP, got positive feedback from the SIG again, >>> and >>> > so now I present to you PEP 488 for discussion. >>> > >>> [snip] >>> >>> Historically -O and -OO have been the antithesis of optimisation, they >>> change the behaviour of the program with no noticeable effect on >>> performance. >>> If a change is to be made, why not just drop .pyo files and be done with >>> it? >>> >> >> I disagree with your premise that .pyo files don't have a noticeable >> effect on performance. If you don't use asserts a lot then there is no >> effect, but if you use them heavily or have them perform expensive >> calculations then there is an impact. And the dropping of docstrings does >> have an impact on memory usage when you use Python at scale. >> >> You're also assuming that we will never develop an AST optimizer that >> will go beyond what the peepholer can do based on raw bytecode, or >> something that involves a bit of calculation and thus something you >> wouldn't want to do at startup. >> > > I don't want to speak for him, but you're going to get the best results > optimizing ASTs at runtime, which is what I thought he was suggesting. > Trying to optimize Python at compile time is setting your sights really > low. You have so little information then. > OK, I don't want to derail the discussion of the PEP into one over how best to optimize CPython's performance relative to bytecode vs. runtime like PyPy. The point is that we have -O and -OO and people do have uses for those flags. People can also do custom optimizations thanks to the flexibility of loaders. All of this leads to wanting different bytecode files for different optimization levels to make sure you're actually executing your code with the optimizations you expect. If people think that optimizing code and surfacing it in bytecode files is a waste and want to suggest either dropping .pyo files entirely or dropping -O and only having -OO that's fine, but that is not what this PEP is proposing nor a PEP I want to bother writing. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On 03/06/2015 08:34 AM, Brett Cannon wrote: > Over on the import-sig I proposed eliminating the concept of .pyo files since > they only signify that /some/ optimization > took place, not /what/ optimizations took place. Everyone on the SIG was > positive with the idea so I wrote a PEP, got > positive feedback from the SIG again, and so now I present to you PEP 488 for > discussion. +1 overall, comments in-line. > Implementation > == > > importlib > - > > As ``importlib.util.cache_from_source()`` is the API that exposes > bytecode file paths as while as being directly used by importlib, it > requires the most critical change. Not sure what that sentence is supposed to say -- maybe "as well as" and not "as while as" ? > The ``debug_override`` parameter will be deprecated. As the parameter > expects a boolean, the integer value of the boolean will be used as > if it had been provided as the argument to ``optimization`` (a > ``None`` argument will mean the same as for ``optimization``). A > deprecation warning will be raised when ``debug_override`` is given a > value other than ``None``, but there are no plans for the complete > removal of the parameter as this time (but removal will be no later > than Python 4). "at this time" not "as this time" > Rest of the standard library > > > The various functions exposed by the ``py_compile`` and > ``compileall`` functions will be updated as necessary to make sure > they follow the new bytecode file name semantics [6]_, [1]_. The CLI > for the ``compileall`` module will not be directly affected (the > ``-b`` flag will be implicitly as it will no longer generate ``.pyo`` > files when ``-O`` is specified). "will be implicit" not "will be implicitly" -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
Thanks! All suggestions applied to my local copy. On Fri, Mar 6, 2015 at 1:55 PM Ethan Furman wrote: > On 03/06/2015 08:34 AM, Brett Cannon wrote: > > Over on the import-sig I proposed eliminating the concept of .pyo files > since they only signify that /some/ optimization > > took place, not /what/ optimizations took place. Everyone on the SIG was > positive with the idea so I wrote a PEP, got > > positive feedback from the SIG again, and so now I present to you PEP > 488 for discussion. > > +1 overall, comments in-line. > > > Implementation > > == > > > > importlib > > - > > > > As ``importlib.util.cache_from_source()`` is the API that exposes > > bytecode file paths as while as being directly used by importlib, it > > requires the most critical change. > > Not sure what that sentence is supposed to say -- maybe "as well as" and > not "as while as" ? > > > > The ``debug_override`` parameter will be deprecated. As the parameter > > expects a boolean, the integer value of the boolean will be used as > > if it had been provided as the argument to ``optimization`` (a > > ``None`` argument will mean the same as for ``optimization``). A > > deprecation warning will be raised when ``debug_override`` is given a > > value other than ``None``, but there are no plans for the complete > > removal of the parameter as this time (but removal will be no later > > than Python 4). > > "at this time" not "as this time" > > > > Rest of the standard library > > > > > > The various functions exposed by the ``py_compile`` and > > ``compileall`` functions will be updated as necessary to make sure > > they follow the new bytecode file name semantics [6]_, [1]_. The CLI > > for the ``compileall`` module will not be directly affected (the > > ``-b`` flag will be implicitly as it will no longer generate ``.pyo`` > > files when ``-O`` is specified). > > "will be implicit" not "will be implicitly" > > -- > ~Ethan~ > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015, at 13:34, Brett Cannon wrote: > On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar > wrote: > > > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > > > >> > >> > >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > >> > >>> > >>> On 06/03/15 16:34, Brett Cannon wrote: > >>> > Over on the import-sig I proposed eliminating the concept of .pyo files > >>> > since they only signify that /some/ optimization took place, not > >>> > /what/ optimizations took place. Everyone on the SIG was positive with > >>> > the idea so I wrote a PEP, got positive feedback from the SIG again, > >>> and > >>> > so now I present to you PEP 488 for discussion. > >>> > > >>> [snip] > >>> > >>> Historically -O and -OO have been the antithesis of optimisation, they > >>> change the behaviour of the program with no noticeable effect on > >>> performance. > >>> If a change is to be made, why not just drop .pyo files and be done with > >>> it? > >>> > >> > >> I disagree with your premise that .pyo files don't have a noticeable > >> effect on performance. If you don't use asserts a lot then there is no > >> effect, but if you use them heavily or have them perform expensive > >> calculations then there is an impact. And the dropping of docstrings does > >> have an impact on memory usage when you use Python at scale. > >> > >> You're also assuming that we will never develop an AST optimizer that > >> will go beyond what the peepholer can do based on raw bytecode, or > >> something that involves a bit of calculation and thus something you > >> wouldn't want to do at startup. > >> > > > > I don't want to speak for him, but you're going to get the best results > > optimizing ASTs at runtime, which is what I thought he was suggesting. > > Trying to optimize Python at compile time is setting your sights really > > low. You have so little information then. > > > > OK, I don't want to derail the discussion of the PEP into one over how > best > to optimize CPython's performance relative to bytecode vs. runtime like > PyPy. The point is that we have -O and -OO and people do have uses for > those flags. People can also do custom optimizations thanks to the > flexibility of loaders. I think it would be preferable deprecate -O and -OO and replace them with flags like --no-docstrings or --no-asserts. Ideally, "optimization" levels shouldn't change program semantics. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Sat, Mar 7, 2015 at 6:09 AM, Benjamin Peterson wrote: > I think it would be preferable deprecate -O and -OO and replace them > with flags like --no-docstrings or --no-asserts. Ideally, "optimization" > levels shouldn't change program semantics. Plenty of C compilers have optimization levels that can change behaviour (replacing division with mult-by-reciprocal and such), so I don't see any reason for Python to object. The removal of docstrings will be a problem to only a handful of programs (eg [1]) which use them for more than introspection. And the removal of assert shouldn't be a semantic change to a well-written program. [1] https://github.com/Rosuav/LetMeKnow/blob/master/letmeknow.py ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 2:09 PM Benjamin Peterson wrote: > > > On Fri, Mar 6, 2015, at 13:34, Brett Cannon wrote: > > On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar > > wrote: > > > > > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > > > > > >> > > >> > > >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > > >> > > >>> > > >>> On 06/03/15 16:34, Brett Cannon wrote: > > >>> > Over on the import-sig I proposed eliminating the concept of .pyo > files > > >>> > since they only signify that /some/ optimization took place, not > > >>> > /what/ optimizations took place. Everyone on the SIG was positive > with > > >>> > the idea so I wrote a PEP, got positive feedback from the SIG > again, > > >>> and > > >>> > so now I present to you PEP 488 for discussion. > > >>> > > > >>> [snip] > > >>> > > >>> Historically -O and -OO have been the antithesis of optimisation, > they > > >>> change the behaviour of the program with no noticeable effect on > > >>> performance. > > >>> If a change is to be made, why not just drop .pyo files and be done > with > > >>> it? > > >>> > > >> > > >> I disagree with your premise that .pyo files don't have a noticeable > > >> effect on performance. If you don't use asserts a lot then there is no > > >> effect, but if you use them heavily or have them perform expensive > > >> calculations then there is an impact. And the dropping of docstrings > does > > >> have an impact on memory usage when you use Python at scale. > > >> > > >> You're also assuming that we will never develop an AST optimizer that > > >> will go beyond what the peepholer can do based on raw bytecode, or > > >> something that involves a bit of calculation and thus something you > > >> wouldn't want to do at startup. > > >> > > > > > > I don't want to speak for him, but you're going to get the best results > > > optimizing ASTs at runtime, which is what I thought he was suggesting. > > > Trying to optimize Python at compile time is setting your sights really > > > low. You have so little information then. > > > > > > > OK, I don't want to derail the discussion of the PEP into one over how > > best > > to optimize CPython's performance relative to bytecode vs. runtime like > > PyPy. The point is that we have -O and -OO and people do have uses for > > those flags. People can also do custom optimizations thanks to the > > flexibility of loaders. > > I think it would be preferable deprecate -O and -OO and replace them > with flags like --no-docstrings or --no-asserts. Ideally, "optimization" > levels shouldn't change program semantics. > OK, but that doesn't influence the PEP's goal of dropping .pyo files. Are you suggesting that the tag be changed to be less specific to optimizations and more free-form? Like `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff like this gets baked into the .pyc file itself instead of the file name, but I don't think we should just drop the ability to switch off asserts and docstrings like Mark seemed to be suggesting. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, 06 Mar 2015 18:11:19 + Brett Cannon wrote: > And the dropping of docstrings does have an impact on > memory usage when you use Python at scale. What kind of "scale" are you talking about? Do you have any numbers about such impact? > You're also assuming that we will never develop an AST optimizer No, the assumption is that we don't have such an optimizer *right now*. Having command-line options because they might be useful some day is silly. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote: > On Fri, 06 Mar 2015 18:11:19 + > Brett Cannon wrote: > > And the dropping of docstrings does have an impact on > > memory usage when you use Python at scale. > > What kind of "scale" are you talking about? Do you have any numbers > about such impact? > > > You're also assuming that we will never develop an AST optimizer > > No, the assumption is that we don't have such an optimizer *right now*. > Having command-line options because they might be useful some day is > silly. Quoting the PEP: This issue is only compounded when people optimize Python code beyond what the interpreter natively supports, e.g., using the astoptimizer project [2]_. Brett, I'm a very strong +1 on the PEP. It's well-written and gives a good explanation for why such a thing is needed. The current behaviour of re-using the same .pyo file for two distinct sets of bytecode is out-and-out buggy: [steve@ando ~]$ python3.3 -O -c "import dis; print(dis.__doc__[:32])" Disassembler of Python byte code [steve@ando ~]$ python3.3 -OO -c "import dis; print(dis.__doc__[:32])" Disassembler of Python byte code The second should fail, since doc strings should be removed under -OO optimization, but because the .pyo file already exists it doesn't. Even if CPython drops -O and -OO altogether, this PEP should still be accepted to allow third party optimizers like astoptimizer to interact without getting in each other's way. (And for the record, I'm an equally strong -1 on dropping -O and -OO.) Thank you. -- Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Sat, 7 Mar 2015 09:34:20 +1100 Steven D'Aprano wrote: > On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote: > > On Fri, 06 Mar 2015 18:11:19 + > > Brett Cannon wrote: > > > And the dropping of docstrings does have an impact on > > > memory usage when you use Python at scale. > > > > What kind of "scale" are you talking about? Do you have any numbers > > about such impact? > > > > > You're also assuming that we will never develop an AST optimizer > > > > No, the assumption is that we don't have such an optimizer *right now*. > > Having command-line options because they might be useful some day is > > silly. > > Quoting the PEP: > > This issue is only compounded when people optimize Python > code beyond what the interpreter natively supports, e.g., > using the astoptimizer project [2]_. The astoptimizer project is not part of Python. It's third-party software that has no relationship to .pyo files. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015, at 15:13, Chris Angelico wrote: > On Sat, Mar 7, 2015 at 6:09 AM, Benjamin Peterson > wrote: > > I think it would be preferable deprecate -O and -OO and replace them > > with flags like --no-docstrings or --no-asserts. Ideally, "optimization" > > levels shouldn't change program semantics. > > Plenty of C compilers have optimization levels that can change > behaviour (replacing division with mult-by-reciprocal and such), so I > don't see any reason for Python to object. Yes, but in the C world, you have pass scary flags like -ffast-math. Just -O2 generally won't reach outside of standard semantics. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote: > > OK, but that doesn't influence the PEP's goal of dropping .pyo files. Correct. > > Are you suggesting that the tag be changed to be less specific to > optimizations and more free-form? Like > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff > like this gets baked into the .pyc file itself instead of the file name, > but I don't think we should just drop the ability to switch off asserts > and > docstrings like Mark seemed to be suggesting. Basically, though the filename strings could perhaps be more compact. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 3:37 PM Antoine Pitrou wrote: > On Fri, 06 Mar 2015 18:11:19 + > Brett Cannon wrote: > > And the dropping of docstrings does have an impact on > > memory usage when you use Python at scale. > > What kind of "scale" are you talking about? Do you have any numbers > about such impact? > I know YouTube at least uses -OO and I don't have numbers to share (numbers I were last shown were years ago and I wouldn't be authorized to share anyway, but I do know they still use -OO). > > > You're also assuming that we will never develop an AST optimizer > > No, the assumption is that we don't have such an optimizer *right now*. > Having command-line options because they might be useful some day is > silly. > I'm not talking about changing any command-line option in the PEP so I don't know what you're referring to. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 5:47 PM Antoine Pitrou wrote: > On Sat, 7 Mar 2015 09:34:20 +1100 > Steven D'Aprano wrote: > > > On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote: > > > On Fri, 06 Mar 2015 18:11:19 + > > > Brett Cannon wrote: > > > > And the dropping of docstrings does have an impact on > > > > memory usage when you use Python at scale. > > > > > > What kind of "scale" are you talking about? Do you have any numbers > > > about such impact? > > > > > > > You're also assuming that we will never develop an AST optimizer > > > > > > No, the assumption is that we don't have such an optimizer *right now*. > > > Having command-line options because they might be useful some day is > > > silly. > > > > Quoting the PEP: > > > > This issue is only compounded when people optimize Python > > code beyond what the interpreter natively supports, e.g., > > using the astoptimizer project [2]_. > > The astoptimizer project is not part of Python. It's third-party > software that has no relationship to .pyo files. > Directly, no. But the point is that the PEP enables the astoptimizer project to write out .pyc files specifying different optimizations that won't clash with -O or -OO .pyc files. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 6:49 PM Benjamin Peterson
wrote:
>
>
> On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote:
> >
> > OK, but that doesn't influence the PEP's goal of dropping .pyo files.
>
> Correct.
>
> >
> > Are you suggesting that the tag be changed to be less specific to
> > optimizations and more free-form? Like
> > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff
> > like this gets baked into the .pyc file itself instead of the file name,
> > but I don't think we should just drop the ability to switch off asserts
> > and
> > docstrings like Mark seemed to be suggesting.
>
> Basically, though the filename strings could perhaps be more compact.
>
That's fine. Do you have a file name format you want to propose then
instead of "opt-{}" (which is what I'm assuming your "basically" is
referring to)?
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On 03/06/2015 11:34 AM, Brett Cannon wrote: There are currently two open issues, although one is purely a bikeshed topic on formatting of file names so I don't really consider it open for change from what is proposed in the PEP without Guido saying he hates my preference or someone having a really good argument for some alternative. The second open issue on the common case file name is something to reasonably debate and come to consensus on. === PEP: 488 Title: Elimination of PYO files +1 And... Have you considered doing this by having different magic numbers in the .pyc file for standard, -O, and -O0 compiled bytecode files? Python already checks that number and recompiles the files if it's not what it's expected to be. And it wouldn't require any naming conventions or new cache directories. It seems to me it would be much easier to do as well. Cheers, Ron ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
